Evaluation of Feature Attribution Methods in Interpretable Machine Learning

The question

A classifier that scores well on a medical-image benchmark may have learned the disease, or it may have learned an artifact that travels with the label: a scanner's text overlay, a ward's marker. Saliency maps, class activation maps among them, are supposed to reveal which, by highlighting the pixels a prediction depended on. But an explanation method is itself a chain of mathematical operations with assumptions of its own, and on a real dataset there is no ground truth for what the model used, so there is no way to score whether the map is right. My honours thesis asked how well fiveiclass activation map (CAM) explanation methods comparedSource: Resume, revised 2026-09-11 CAM methods (Grad-CAM, Grad-CAM++, Layer-CAM, Eigen-CAM and FullGrad) localize the evidence a near-perfect classifier actually relies on, once that evidence is known.

Inducing ground truth

The way to know is to plant it. The design is not mine: it comes from Yilun Zhou, Serena Booth, Marco Tulio Ribeiro and Julie Shah's paper "Do Feature Attribution Methods Correctly Attribute Features?" (AAAI 2022) and the Feature Attribution Evaluation repository that accompanies it, both of which the thesis cites. I applied it to chest X-rays. Starting from the public COVID-19 Radiography Database of chest X-rays, I reassigned every image's label at random, so nothing in the original image could predict its new label, and then added a text watermark of random size and position to the positive class only, saving its rectangle as a binary mask. A model that reaches high validation accuracy on this data can only have done so by reading the watermark. A CAM that explains that model correctly should highlight the watermark and nothing else, and the mask makes the comparison measurable. The ground truth is induced, not annotated: no radiologist marked anything.

1. 1Start from public chest X-raysImages from the COVID-19 Radiography Database, split into training, validation and test sets. Each experiment takes a pair of the original classes.
2. 2Reassign every label at randomEach image lands in the positive or negative class with equal odds, so nothing in the original image predicts its new label.
3. 3Watermark the positive class onlyA text watermark of random size and position is added to every positive image. Its rectangle is saved as a binary mask: the ground truth.
4. 4Train a classifier from scratchResNet-34 or VGG-16, randomly initialized. High validation accuracy is only possible by reading the watermark, because nothing else is informative.
5. 5Explain, threshold, scoreEach CAM method explains the model's prediction on watermarked test images. The map is binarized at a threshold and compared with the mask by intersection over union.

Method

I trained ResNet-34 and VGG-16 from randomly initialized weights in PyTorch on Google Colab, then generated CAMs from each method for watermarked test images with the pytorch-grad-cam library, targeting each network's final convolutional layer. Each CAM assigns every pixel a score; I binarized the map at thresholds of 50–95%ithresholds at which each CAM was binarized before scoring, in steps of five pointsThesis §3.7, p. 19. Pixels scoring above the threshold form the prediction mask.Source: Honours thesis, SFU, 2022 and scored each binary mask against the ground truth by intersection over union, averaged over teniwatermarked test images per experiment over which IoU was averagedThesis §3.7, p. 19.Source: Honours thesis, SFU, 2022 watermarked test images per experiment. Paired t-tests then compared FullGrad's per-image scores with the runner-up method's at each threshold, at a significance level of 5%isignificance level (alpha) of the paired t-testsThesis §3.8, p. 21; FullGrad against the runner-up method at each threshold.Source: Honours thesis, SFU, 2022. No new metric was defined; the point was to apply a plain, defensible one where ground truth existed.

What the scores showed

• Grad-CAM
• Eigen-CAM
• Grad-CAM++
• Layer-CAM
• FullGrad

00.20.40.6

FullGrad peak 0.530 at 90%

50%55%60%65%70%75%80%85%90%95%

threshold

Values as a table

Table 4.6: average IoU by threshold, ResNet-34

Method	50%	55%	60%	65%	70%	75%	80%	85%	90%	95%
Grad-CAM	0.079	0.094	0.115	0.146	0.187	0.235	0.286	0.321	0.292	0.136
Eigen-CAM	0.125	0.150	0.187	0.227	0.265	0.304	0.338	0.328	0.236	0.075
Grad-CAM++	0.091	0.110	0.137	0.176	0.222	0.271	0.318	0.340	0.274	0.091
Layer-CAM	0.091	0.109	0.136	0.175	0.222	0.271	0.318	0.341	0.277	0.095
FullGrad	0.075	0.104	0.134	0.170	0.214	0.286	0.378	0.489	0.530	0.384

Explaining ResNet-34 (Fig. 2), FullGrad's maps were the sharpest and most targeted on the watermark, but they carried a dispersed checkerboard pattern that no other method showed. Thresholding removed it: FullGrad's average IoU rose with the threshold to 0.53ibest average IoU for FullGrad explaining ResNet-34, at the 90% thresholdTable 4.6, p. 31, prints .530 (the text on p. 24 rounds it to 0.53 and once to 0.54).Source: Honours thesis, SFU, 2022, the best score in the experiment, while every other method peaked lower and earlier. Eigen-CAM led at low thresholds; Grad-CAM++ and Layer-CAM were nearly indistinguishable at every threshold, in the maps and in the scores.

• Grad-CAM
• Eigen-CAM
• Grad-CAM++
• Layer-CAM
• FullGrad

00.20.40.6

FullGrad peak 0.560 at 65%

50%55%60%65%70%75%80%85%90%95%

threshold

Values as a table

Table 4.7: average IoU by threshold, VGG-16

Method	50%	55%	60%	65%	70%	75%	80%	85%	90%	95%
Grad-CAM	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000
Eigen-CAM	0.115	0.121	0.126	0.132	0.139	0.148	0.158	0.164	0.152	0.068
Grad-CAM++	0.079	0.085	0.088	0.088	0.088	0.088	0.085	0.076	0.061	0.034
Layer-CAM	0.132	0.147	0.164	0.180	0.192	0.198	0.186	0.148	0.105	0.0578
FullGrad	0.411	0.475	0.535	0.560	0.553	0.505	0.418	0.306	0.195	0.0731

Explaining VGG-16 (Fig. 3), FullGrad almost exclusively highlighted the watermark, with no dispersion, and reached 0.560ibest average IoU for FullGrad explaining VGG-16, at the 65% thresholdTable 4.7, p. 31.Source: Honours thesis, SFU, 2022, among the highest averages in any experiment. Grad-CAM assigned no attribution at all in that run. The t-tests supported both pictures: wherever FullGrad led, its advantage over the runner-up was statistically significant, except at VGG-16's highest thresholds, where thresholding had removed too many pixels from every mask.

The failures matter as much as the winners. In one ResNet-34 experiment, four of the fiveiCAM methods that appeared to highlight the border between spine and lung instead of the planted watermark, in one ResNet-34 runThesis §4.2, Fig. 4.4 and Table 4.3, pp. 25–27. The watermark sat at the side of the region FullGrad highlighted (p. 26).Source: Honours thesis, SFU, 2022 methods appeared to highlight the border between spine and lung instead of the watermark; FullGrad was not centred on the watermark either, though the watermark sat at the edge of the region it highlighted, and every score fell to near zero. In a VGG-16 run that never learned the task, most maps were noise, though FullGrad still found the watermark. A CAM can only be as good as the model it explains, and a method's behaviour changes with the architecture: the same FullGrad produced different maps for the two networks.

Why it matters

In clinics, a deep network that reads X-rays will be judged by whether people can trust it, and saliency maps are the tool offered for that trust. My results say the tool must itself be tested: some methods missed the planted evidence entirely, a convincing-looking map can be wrong, and the right threshold depends on the method. FullGrad, the only method tested that satisfies local and global attribution at once, was the most robust, precise and accurate in these experiments, and thresholding its output made it better. That is a reason to evaluate explanation methods against induced ground truth before deploying them, not a reason to trust any of them unexamined. It is also the habit I have kept since: build the measuring instrument before trusting the thing being measured.

What carries beyond this setting is the approach, not the result. Plant a feature you control, so that there is something to score an explanation against. Report the runs where every method failed alongside the ones that produced a winner. Expect an explanation method's behaviour to change with the system it explains. What does not carry is the evidence. These were convolutional image classifiers reading chest X-rays, not language models, and a class activation map says nothing about the internals of a transformer. Whether the same approach holds up on other kinds of model is a question this thesis does not answer.

The thesis

The full thesis is linked in the facts column, as a copy with its signed approval page removed. I wrote it in the SFU Multimedia Lab under the supervision of Dr. Ivan V. Bajić, whose suggested experiments and revisions shaped it.