Leaderboards
OpenXAI provides leaderboards for systematic evaluation and comparison of various explanation methods. Below, you can see one leaderboard per dataset and model.
Step-by-Step instructions
To participate in the OpenXAI leaderboard, please follow these step-by-step instructions:
- Use the OpenXAI benchmark dataloader to retrieve a given dataset.
- Use the OpenXAI LoadModel to load a pre-trained model.
- Use the OpenXAI Explainer to load a post hoc explanation method.
- Quantify the performance of the explanation method using OpenXAI evaluation metrics.
- Submit your scores using this FORM.
We invite submissions to any one or multiple benchmarks in OpenXAI. To be included in the leaderboard, please fill out this FORM, include results of your explanation method and provide the full implementation of your algorithm with proper documentations and GitHub link.
Explore Leaderboards
Faithfulness
| Method | PGI PGI measures the difference in prediction probability that results from perturbing the features deemed as influential by a given post hoc explanation. Higher values are better. | PGU PGU measures the difference in prediction probability that results from perturbing the features deemed as unimportant by a given post hoc explanation. Lower values are better. |
|---|---|---|
|
LIME
|
0.42 | 0.68 |
|
SHAP
|
0.44 | 0.64 |
|
Vanilla Gradient
|
0.39 | 0.68 |
|
SmoothGrad
|
0.49 | 0.66 |
|
Integrated Gradient
|
0.40 | 0.68 |
|
Gradient x Input
|
0.45 | 0.62 |
Stability
| Method | RIS RIS measures the maximum change in explanation relative to changes in the inputs. Values closer to zero are better (log 1 ~ 0). | ROS ROS measures the maximum change in explanation relative to changes in the output prediction probabilities. Values closer to zero are better (log 1 ~ 0). | RRS RRS measures the maximum change in explanation relative to changes in the internal representations learned by the model. Values closer to zero are better (log 1 ~ 0). |
|---|---|---|---|
|
Integrated Gradient
|
0.62 | 2.88 | 6.83 |
|
SHAP
|
1.28 | 4.46 | 11.46 |
|
Vanilla Gradient
|
1.35 | 4.44 | 11.34 |
|
Gradient x Input
|
-1.90 | 1.32 | 8.10 |
|
SmoothGrad
|
2.97 | 6.17 | 13.10 |
|
LIME
|
6.35 | 9.57 | 16.50 |
Fairness
| Method | Fairness@PGI Computes the absolute difference in PGI scores between the majority and the minority subgroups. Lower disparities are better. | Fairness@PGU Computes the absolute difference in PGU scores between the majority and the minority subgroups. Lower disparities are better. | Fairness@RIS Computes the absolute difference in RIS scores between the majority and the minority subgroups. Lower disparities are better. | Fairness@ROS Computes the absolute difference in ROS scores between the majority and the minority subgroups. Lower disparities are better. | Fairness@RRS Computes the absolute difference in RRS scores between the majority and the minority subgroups. Lower disparities are better. |
|---|---|---|---|---|---|
|
Gradient x Input
|
0.001 | 0.056 | 0.270 | 0.093 | 0.245 |
|
Integrated Gradient
|
0.009 | 0.056 | 0.070 | 0.214 | 0.137 |
|
SmoothGrad
|
0.035 | 0.085 | 0.011 | 0.321 | 0.026 |
|
Vanilla Gradient
|
0.045 | 0.079 | 0.273 | 0.093 | 0.255 |
|
SHAP
|
0.102 | 0.009 | 0.021 | 0.361 | 0.032 |
|
LIME
|
0.138 | 0.095 | 0.013 | 0.32 | 0.062 |
Faithfulness
| Method | FA FA computes the fraction of top-K features that are common between a given post hoc explanation and the corresponding ground truth explanation. Higher values are better. | RA RA measures the fraction of top-K features that are not only common between a given explanation and the corresponding ground truth explanation, but also have the same position in the respective rank orders. Higher values are better. | SA SA computes the fraction of top-K features that are not only common between a given explanation and the corresponding ground truth explanation, but also share the same sign in both the explanations. Higher values are better | SRA SRA computes the fraction of top-K features that are not only common between a given explanation and the corresponding ground truth explanation, but also share the same feature attribution sign and rank in both the explanations. Higher values the better. | RC RC computes the Spearman’s rank correlation coefficient to measure the agreement between feature rankings provided by a given explanation and the corresponding ground truth explanation. Higher values the better. | PRA PRA captures if the relative ordering of every pair of features is the same for a given explanation as well as the corresponding ground truth explanation and computes the fraction of feature pairs for which the relative ordering is the same between the two explanations. Higher values the better. | PGI PGI measures the difference in prediction probability that results from perturbing the features deemed as influential by a given post hoc explanation. Higher values are better. | PGU PGU measures the difference in prediction probability that results from perturbing the features deemed as unimportant by a given post hoc explanation. Lower values are better. |
|---|---|---|---|---|---|---|---|---|
|
LIME
|
0.938 | 0.810 | 0.938 | 0.814 | 0.998 | 0.989 | 0.156 | 0.169 |
|
SHAP
|
0.130 | 0.007 | 0.112 | 0.006 | -0.053 | 0.488 | 0.135 | 0.180 |
|
Vanilla Gradient
|
0.950 | 0.950 | 0.846 | 0.846 | 1.000 | 1.000 | 0.149 | 0.174 |
|
SmoothGrad
|
0.950 | 0.950 | 0.606 | 0.606 | 1.000 | 1.000 | 0.202 | 0.124 |
|
Integrated Gradient
|
0.950 | 0.950 | 0.846 | 0.846 | 1.000 | 1.000 | 0.148 | 0.173 |
|
Gradient x Input
|
0.785 | 0.161 | 0.382 | 0.071 | 0.890 | 0.875 | 0.171 | 0.161 |
Stability
| Method | RIS RIS measures the maximum change in explanation relative to changes in the inputs. Values closer to zero are better (log 1 ~ 0). | ROS ROS measures the maximum change in explanation relative to changes in the output prediction probabilities. Values closer to zero are better (log 1 ~ 0). | RRS RRS measures the maximum change in explanation relative to changes in the internal representations learned by the model. Values closer to zero are better (log 1 ~ 0). |
|---|---|---|---|
|
SHAP
|
-0.230 | 10.056 | 10.056 |
|
LIME
|
-0.698 | 9.397 | 9.397 |
|
Gradient x Input
|
-0.906 | 9.437 | 9.437 |
|
Vanilla Gradient
|
-1.384 | 5.241 | 5.241 |
|
Integrated Gradient
|
-2.004 | 4.560 | 4.560 |
|
SmoothGrad
|
-4.780 | 4.931 | 4.931 |
Fairness
| Method | Fairness@FA Computes the absolute difference in FA scores between the majority and the minority subgroups. Lower disparities are better. | Fairness@RA Computes the absolute difference in RA scores between the majority and the minority subgroups. Lower disparities are better. | Fairness@SA Computes the absolute difference in SA scores between the majority and the minority subgroups. Lower disparities are better. | Fairness@SRA Computes the absolute difference in SRA scores between the majority and the minority subgroups. Lower disparities are better. | Fairness@RC Computes the absolute difference in RC scores between the majority and the minority subgroups. Lower disparities are better. | Fairness@PRA Computes the absolute difference in PRA scores between the majority and the minority subgroups. Lower disparities are better. | Fairness@PGI Computes the absolute difference in PGI scores between the majority and the minority subgroups. Lower disparities are better. | Fairness@PGU Computes the absolute difference in PGU scores between the majority and the minority subgroups. Lower disparities are better. | Fairness@RIS Computes the absolute difference in RIS scores between the majority and the minority subgroups. Lower disparities are better. | Fairness@ROS Computes the absolute difference in ROS scores between the majority and the minority subgroups. Lower disparities are better. | Fairness@RRS Computes the absolute difference in RRS scores between the majority and the minority subgroups. Lower disparities are better. |
|---|---|---|---|---|---|---|---|---|---|---|---|
|
SmoothGrad
|
0.018 | 0.014 | 0.011 | 0.014 | 0.005 | 0.006 | 0.045 | 0.028 | 0.200 | 0.104 | 0.104 |
|
Vanilla Gradient
|
0.019 | 0.014 | 0.142 | 0.081 | 0.053 | 0.064 | 0.016 | 0.029 | 0.172 | 0.014 | 0.014 |
|
Integrated Gradient
|
0.021 | 0.016 | 0.141 | 0.080 | 0.053 | 0.064 | 0.010 | 0.032 | 0.363 | 0.140 | 0.140 |
|
LIME
|
0.027 | 0.017 | 0.014 | 0.016 | 0.011 | 0.012 | 0.056 | 0.051 | 0.238 | 0.050 | 0.050 |
|
SHAP
|
0.050 | 0.039 | 0.038 | 0.039 | 0.016 | 0.020 | 0.012 | 0.017 | 0.279 | 0.067 | 0.067 |
|
Gradient x Input
|
0.057 | 0.044 | 0.042 | 0.044 | 0.017 | 0.022 | 0.012 | 0.028 | 0.219 | 0.015 | 0.015 |
Faithfulness
| Method | PGI PGI measures the difference in prediction probability that results from perturbing the features deemed as influential by a given post hoc explanation. Higher values are better. | PGU PGU measures the difference in prediction probability that results from perturbing the features deemed as unimportant by a given post hoc explanation. Lower values are better. |
|---|---|---|
|
LIME
|
0.26 | 0.24 |
|
SHAP
|
0.26 | 0.23 |
|
Vanilla Gradient
|
0.24 | 0.26 |
|
SmoothGrad
|
0.33 | 0.13 |
|
Integrated Gradient
|
0.24 | 0.26 |
|
Gradient x Input
|
0.25 | 0.23 |
Stability
| Method | RIS RIS measures the maximum change in explanation relative to changes in the inputs. Values closer to zero are better (log 1 ~ 0). | ROS ROS measures the maximum change in explanation relative to changes in the output prediction probabilities. Values closer to zero are better (log 1 ~ 0). | RRS RRS measures the maximum change in explanation relative to changes in the internal representations learned by the model. Values closer to zero are better (log 1 ~ 0). |
|---|---|---|---|
|
LIME
|
1.19 | 3.33 | 13.25 |
|
SHAP
|
1.57 | 3.80 | 9.40 |
|
Gradient x Input
|
2.46 | 4.28 | 9.56 |
|
SmoothGrad
|
-3.11 | -0.95 | 4.60 |
|
Integrated Gradient
|
3.16 | 5.17 | 9.53 |
|
Vanilla Gradient
|
4.06 | 5.86 | 10.80 |
Fairness
Fairness evaluation is not applicable for the HELOC dataset as it does not include any protected attributes.
Faithfulness
| Method | FA FA computes the fraction of top-K features that are common between a given post hoc explanation and the corresponding ground truth explanation. Higher values are better. | RA RA measures the fraction of top-K features that are not only common between a given explanation and the corresponding ground truth explanation, but also have the same position in the respective rank orders. Higher values are better. | SA SA computes the fraction of top-K features that are not only common between a given explanation and the corresponding ground truth explanation, but also share the same sign in both the explanations. Higher values are better | SRA SRA computes the fraction of top-K features that are not only common between a given explanation and the corresponding ground truth explanation, but also share the same feature attribution sign and rank in both the explanations. Higher values the better. | RC RC computes the Spearman’s rank correlation coefficient to measure the agreement between feature rankings provided by a given explanation and the corresponding ground truth explanation. Higher values the better. | PRA PRA captures if the relative ordering of every pair of features is the same for a given explanation as well as the corresponding ground truth explanation and computes the fraction of feature pairs for which the relative ordering is the same between the two explanations. Higher values the better. | PGI PGI measures the difference in prediction probability that results from perturbing the features deemed as influential by a given post hoc explanation. Higher values are better. | PGU PGU measures the difference in prediction probability that results from perturbing the features deemed as unimportant by a given post hoc explanation. Lower values are better. |
|---|---|---|---|---|---|---|---|---|
|
LIME
|
0.932 | 0.671 | 0.929 | 0.670 | 0.994 | 0.982 | 0.156 | 0.104 |
|
SHAP
|
0.586 | 0.054 | 0.269 | 0.024 | 0.384 | 0.645 | 0.121 | 0.136 |
|
Vanilla Gradient
|
0.957 | 0.957 | 0.469 | 0.469 | 1.000 | 1.000 | 0.129 | 0.131 |
|
SmoothGrad
|
0.957 | 0.957 | 0.274 | 0.274 | 1.000 | 1.000 | 0.054 | 0.177 |
|
Integrated Gradient
|
0.957 | 0.957 | 0.469 | 0.469 | 1.000 | 1.000 | 0.127 | 0.132 |
|
Gradient x Input
|
0.582 | 0.049 | 0.255 | 0.022 | 0.390 | 0.641 | 0.123 | 0.135 |
Stability
| Method | RIS RIS measures the maximum change in explanation relative to changes in the inputs. Values closer to zero are better (log 1 ~ 0). | ROS ROS measures the maximum change in explanation relative to changes in the output prediction probabilities. Values closer to zero are better (log 1 ~ 0). | RRS RRS measures the maximum change in explanation relative to changes in the internal representations learned by the model. Values closer to zero are better (log 1 ~ 0). |
|---|---|---|---|
|
Gradient x Input
|
1.036 | 8.964 | 8.964 |
|
Integrated Gradient
|
1.403 | 5.776 | 5.776 |
|
SHAP
|
1.458 | 9.331 | 9.331 |
|
Vanilla Gradient
|
2.124 | 6.336 | 6.336 |
|
SmoothGrad
|
-3.724 | 3.507 | 3.507 |
|
LIME
|
4.534 | 8.833 | 8.833 |
Fairness
Fairness evaluation is not applicable for the HELOC dataset as it does not include any protected attributes.
Faithfulness
| Method | PGI PGI measures the difference in prediction probability that results from perturbing the features deemed as influential by a given post hoc explanation. Higher values are better. | PGU PGU measures the difference in prediction probability that results from perturbing the features deemed as unimportant by a given post hoc explanation. Lower values are better. |
|---|---|---|
|
LIME
|
0.67 | 0.26 |
|
SHAP
|
0.53 | 0.33 |
|
Vanilla Gradient
|
0.29 | 0.64 |
|
SmoothGrad
|
0.70 | 0.12 |
|
Integrated Gradient
|
0.29 | 0.64 |
|
Gradient x Input
|
0.58 | 0.26 |
Stability
| Method | RIS RIS measures the maximum change in explanation relative to changes in the inputs. Values closer to zero are better (log 1 ~ 0). | ROS ROS measures the maximum change in explanation relative to changes in the output prediction probabilities. Values closer to zero are better (log 1 ~ 0). | RRS RRS measures the maximum change in explanation relative to changes in the internal representations learned by the model. Values closer to zero are better (log 1 ~ 0). |
|---|---|---|---|
|
SmoothGrad
|
-1.70 | -0.95 | 6.22 |
|
LIME
|
1.79 | 2.45 | 9.77 |
|
SHAP
|
1.98 | 2.68 | 9.78 |
|
Gradient x Input
|
3.25 | 3.08 | 8.25 |
|
Integrated Gradient
|
4.32 | 4.33 | 11.74 |
|
Vanilla Gradient
|
5.01 | 4.58 | 9.86 |
Fairness
| Method | Fairness@PGI Computes the absolute difference in PGI scores between the majority and the minority subgroups. Lower disparities are better. | Fairness@PGU Computes the absolute difference in PGU scores between the majority and the minority subgroups. Lower disparities are better. | Fairness@RIS Computes the absolute difference in RIS scores between the majority and the minority subgroups. Lower disparities are better. | Fairness@ROS Computes the absolute difference in ROS scores between the majority and the minority subgroups. Lower disparities are better. | Fairness@RRS Computes the absolute difference in RRS scores between the majority and the minority subgroups. Lower disparities are better. |
|---|---|---|---|---|---|
|
SmoothGrad
|
0.006 | 0.040 | 0.024 | TBD | 0.017 |
|
LIME
|
0.024 | 0.006 | 0.014 | TBD | 0.012 |
|
Gradient x Input
|
0.034 | 0.016 | 0.021 | TBD | 0.042 |
|
Integrated Gradient
|
0.036 | 0.018 | 0.012 | TBD | 0.005 |
|
Vanilla Gradient
|
0.037 | 0.013 | 0.015 | TBD | 0.037 |
|
SHAP
|
0.058 | 0.017 | 0.011 | TBD | 0.005 |
Faithfulness
| Method | FA FA computes the fraction of top-K features that are common between a given post hoc explanation and the corresponding ground truth explanation. Higher values are better. | RA RA measures the fraction of top-K features that are not only common between a given explanation and the corresponding ground truth explanation, but also have the same position in the respective rank orders. Higher values are better. | SA SA computes the fraction of top-K features that are not only common between a given explanation and the corresponding ground truth explanation, but also share the same sign in both the explanations. Higher values are better. | SRA SRA computes the fraction of top-K features that are not only common between a given explanation and the corresponding ground truth explanation, but also share the same feature attribution sign and rank in both the explanations. Higher values the better. | RC RC computes the Spearman’s rank correlation coefficient to measure the agreement between feature rankings provided by a given explanation and the corresponding ground truth explanation. Higher values the better. | PRA PRA captures if the relative ordering of every pair of features is the same for a given explanation as well as the corresponding ground truth explanation and computes the fraction of feature pairs for which the relative ordering is the same between the two explanations. Higher values the better. | PGI PGI measures the difference in prediction probability that results from perturbing the features deemed as influential by a given post hoc explanation. Higher values are better. | PGU PGU measures the difference in prediction probability that results from perturbing the features deemed as unimportant by a given post hoc explanation. Lower values are better. |
|---|---|---|---|---|---|---|---|---|
|
LIME
|
0.869 | 0.697 | 0.858 | 0.689 | 0.921 | 0.913 | 0.420 | 0.269 |
|
SHAP
|
0.601 | 0.105 | 0.133 | 0.009 | 0.379 | 0.655 | 0.391 | 0.205 |
|
Vanilla Gradient
|
0.923 | 0.921 | 0.138 | 0.136 | 1.000 | 1.000 | 0.297 | 0.391 |
|
SmoothGrad
|
0.923 | 0.923 | 0.741 | 0.741 | 1.000 | 1.000 | 0.485 | 0.082 |
|
Integrated Gradient
|
0.923 | 0.923 | 0.138 | 0.138 | 1.000 | 1.000 | 0.297 | 0.392 |
|
Gradient x Input
|
0.567 | 0.075 | 0.070 | 0.003 | 0.281 | 0.580 | 0.395 | 0.193 |
Stability
| Method | RIS RIS measures the maximum change in explanation relative to changes in the inputs. Values closer to zero are better (log 1 ~ 0). | ROS ROS measures the maximum change in explanation relative to changes in the output prediction probabilities. Values closer to zero are better (log 1 ~ 0). | RRS RRS measures the maximum change in explanation relative to changes in the internal representations learned by the model. Values closer to zero are better (log 1 ~ 0). |
|---|---|---|---|
|
LIME
|
1.715 | 7.899 | 7.899 |
|
SHAP
|
1.857 | 7.883 | 7.883 |
|
Gradient x Input
|
2.307 | 7.543 | 7.543 |
|
SmoothGrad
|
-3.007 | 2.941 | 2.941 |
|
Integrated Gradient
|
3.492 | 6.843 | 6.843 |
|
Vanilla Gradient
|
4.092 | 6.928 | 6.928 |
Fairness
| Method | Fairness@FA Computes the absolute difference in FA scores between the majority and the minority subgroups. Lower disparities are better. | Fairness@RA Computes the absolute difference in RA scores between the majority and the minority subgroups. Lower disparities are better. | Fairness@SA Computes the absolute difference in SA scores between the majority and the minority subgroups. Lower disparities are better. | Fairness@SRA Computes the absolute difference in SRA scores between the majority and the minority subgroups. Lower disparities are better. | Fairness@RC Computes the absolute difference in RC scores between the majority and the minority subgroups. Lower disparities are better. | Fairness@PRA Computes the absolute difference in PRA scores between the majority and the minority subgroups. Lower disparities are better. | Fairness@PGI Computes the absolute difference in PGI scores between the majority and the minority subgroups. Lower disparities are better. | Fairness@PGU Computes the absolute difference in PGU scores between the majority and the minority subgroups. Lower disparities are better. | Fairness@RIS Computes the absolute difference in RIS scores between the majority and the minority subgroups. Lower disparities are better. | Fairness@ROS Computes the absolute difference in ROS scores between the majority and the minority subgroups. Lower disparities are better. | Fairness@RRS Computes the absolute difference in RRS scores between the majority and the minority subgroups. Lower disparities are better. |
|---|---|---|---|---|---|---|---|---|---|---|---|
|
Gradient x Input
|
0.001 | 0.001 | 0.002 | 0.001 | 0.001 | 0.001 | 0.001 | 0.002 | 0.020 | 0.399 | 0.399 |
|
SHAP
|
0.001 | 0.000 | 0.001 | 0.000 | 0.000 | 0.001 | 0.004 | 0.001 | 0.106 | 0.200 | 0.200 |
|
SmoothGrad
|
0.002 | 0.002 | 0.001 | 0.001 | 0.001 | 0.001 | 0.003 | 0.004 | 0.018 | 0.102 | 0.102 |
|
Integrated Gradient
|
0.002 | 0.002 | 0.007 | 0.005 | 0.004 | 0.005 | 0.009 | 0.004 | 0.140 | 0.010 | 0.010 |
|
Vanilla Gradient
|
0.004 | 0.003 | 0.009 | 0.006 | 0.005 | 0.006 | 0.009 | 0.006 | 0.079 | 0.167 | 0.167 |
|
LIME
|
0.005 | 0.003 | 0.002 | 0.003 | 0.001 | 0.002 | 0.003 | 0.006 | 0.055 | 0.113 | 0.113 |
Faithfulness
| Method | PGI PGI measures the difference in prediction probability that results from perturbing the features deemed as influential by a given post hoc explanation. Higher values are better. | PGU PGU measures the difference in prediction probability that results from perturbing the features deemed as unimportant by a given post hoc explanation. Lower values are better. |
|---|---|---|
|
LIME
|
0.25 | 0.35 |
|
SHAP
|
0.32 | 0.27 |
|
Vanilla Gradient
|
0.31 | 0.29 |
|
SmoothGrad
|
0.41 | 0.13 |
|
Integrated Gradient
|
0.31 | 0.29 |
|
Gradient x Input
|
0.31 | 0.29 |
Stability
| Method | RIS RIS measures the maximum change in explanation relative to changes in the inputs. Values closer to zero are better (log 1 ~ 0). | ROS ROS measures the maximum change in explanation relative to changes in the output prediction probabilities. Values closer to zero are better (log 1 ~ 0). | RRS RRS measures the maximum change in explanation relative to changes in the internal representations learned by the model. Values closer to zero are better (log 1 ~ 0). |
|---|---|---|---|
|
SmoothGrad
|
1.11 | -1.07 | 5.31 |
|
LIME
|
5.53 | 3.40 | 9.86 |
|
SHAP
|
5.96 | 3.95 | 10.39 |
|
Gradient x Input
|
6.80 | 3.46 | 7.22 |
|
Vanilla Gradient
|
7.58 | 4.24 | 8.02 |
|
Integrated Gradient
|
8.14 | 5.53 | 11.60 |
Fairness
Fairness evaluation is not applicable for the Synthetic dataset as it does not include any protected attributes.
Faithfulness
| Method | FA FA computes the fraction of top-K features that are common between a given post hoc explanation and the corresponding ground truth explanation. Higher values are better. | RA RA measures the fraction of top-K features that are not only common between a given explanation and the corresponding ground truth explanation, but also have the same position in the respective rank orders. Higher values are better. | SA SA computes the fraction of top-K features that are not only common between a given explanation and the corresponding ground truth explanation, but also share the same sign in both the explanations. Higher values are better. | SRA SRA computes the fraction of top-K features that are not only common between a given explanation and the corresponding ground truth explanation, but also share the same feature attribution sign and rank in both the explanations. Higher values the better. | RC RC computes the Spearman’s rank correlation coefficient to measure the agreement between feature rankings provided by a given explanation and the corresponding ground truth explanation. Higher values the better. | PRA PRA captures if the relative ordering of every pair of features is the same for a given explanation as well as the corresponding ground truth explanation and computes the fraction of feature pairs for which the relative ordering is the same between the two explanations. Higher values the better. | PGI PGI measures the difference in prediction probability that results from perturbing the features deemed as influential by a given post hoc explanation. Higher values are better. | PGU PGU measures the difference in prediction probability that results from perturbing the features deemed as unimportant by a given post hoc explanation. Lower values are better. |
|---|---|---|---|---|---|---|---|---|
|
LIME
|
0.930 | 0.763 | 0.928 | 0.762 | 0.995 | 0.986 | 0.154 | 0.180 |
|
SHAP
|
0.775 | 0.145 | 0.377 | 0.068 | 0.873 | 0.860 | 0.171 | 0.160 |
|
Vanilla Gradient
|
0.950 | 0.950 | 0.464 | 0.464 | 1.000 | 1.000 | 0.171 | 0.162 |
|
SmoothGrad
|
0.950 | 0.950 | 0.456 | 0.456 | 1.000 | 1.000 | 0.227 | 0.063 |
|
Integrated Gradient
|
0.950 | 0.950 | 0.464 | 0.464 | 1.000 | 1.000 | 0.171 | 0.163 |
|
Gradient x Input
|
0.785 | 0.161 | 0.382 | 0.071 | 0.890 | 0.875 | 0.171 | 0.161 |
Stability
| Method | RIS RIS measures the maximum change in explanation relative to changes in the inputs. Values closer to zero are better (log 1 ~ 0). | ROS ROS measures the maximum change in explanation relative to changes in the output prediction probabilities. Values closer to zero are better (log 1 ~ 0). | RRS RRS measures the maximum change in explanation relative to changes in the internal representations learned by the model. Values closer to zero are better (log 1 ~ 0). |
|---|---|---|---|
|
Gradient x Input
|
0.405 | 3.422 | 3.422 |
|
SmoothGrad
|
5.249 | 9.419 | 9.419 |
|
SHAP
|
5.673 | 8.751 | 8.751 |
|
Integrated Gradient
|
5.957 | 9.022 | 9.022 |
|
Vanilla Gradient
|
6.133 | 6.144 | 6.144 |
|
LIME
|
9.355 | 13.564 | 13.564 |
Fairness
Fairness evaluation is not applicable for the Synthetic dataset as it does not include any protected attributes.
Faithfulness
| Method | PGI PGI measures the difference in prediction probability that results from perturbing the features deemed as influential by a given post hoc explanation. Higher values are better. | PGU PGU measures the difference in prediction probability that results from perturbing the features deemed as unimportant by a given post hoc explanation. Lower values are better. |
|---|---|---|
|
LIME
|
0.232 | 0.247 |
|
SHAP
|
0.274 | 0.194 |
|
Vanilla Gradient
|
0.24 | 0.24 |
|
SmoothGrad
|
0.324 | 0.106 |
|
Integrated Gradient
|
0.24 | 0.238 |
|
Gradient x Input
|
0.254 | 0.216 |
Stability
| Method | RIS RIS measures the maximum change in explanation relative to changes in the inputs. Values closer to zero are better (log 1 ~ 0). | ROS ROS measures the maximum change in explanation relative to changes in the output prediction probabilities. Values closer to zero are better (log 1 ~ 0). | RRS RRS measures the maximum change in explanation relative to changes in the internal representations learned by the model. Values closer to zero are better (log 1 ~ 0). |
|---|---|---|---|
|
SmoothGrad
|
-0.59 | -1.39 | 3.15 |
|
Gradient x Input
|
3.81 | 3.19 | 7.16 |
|
SHAP
|
4.24 | 3.72 | 8.27 |
|
LIME
|
4.49 | 3.54 | 8.36 |
|
Integrated Gradient
|
5.91 | 5.29 | 9.27 |
|
Vanilla Gradient
|
6.29 | 5.54 | 9.02 |
Fairness
| Method | Fairness@PGI Computes the absolute difference in PGI scores between the majority and the minority subgroups. Lower disparities are better. | Fairness@PGU Computes the absolute difference in PGU scores between the majority and the minority subgroups. Lower disparities are better. | Fairness@RIS Computes the absolute difference in RIS scores between the majority and the minority subgroups. Lower disparities are better. | Fairness@ROS Computes the absolute difference in ROS scores between the majority and the minority subgroups. Lower disparities are better. | Fairness@RRS Computes the absolute difference in RRS scores between the majority and the minority subgroups. Lower disparities are better. |
|---|---|---|---|---|---|
|
Gradient x Input
|
0.052 | 0.082 | 0.647 | 0.786 | 0.392 |
|
LIME
|
0.054 | 0.083 | 0.275 | 0.279 | 0.062 |
|
SHAP
|
0.066 | 0.084 | 0.229 | 0.182 | 0.023 |
|
Vanilla Gradient
|
0.076 | 0.086 | 0.399 | 0.663 | 0.175 |
|
Integrated Gradient
|
0.087 | 0.085 | 0.177 | 0.225 | 0.090 |
|
SmoothGrad
|
0.119 | 0.096 | 0.094 | 0.090 | 0.169 |
Faithfulness
| Method | FA FA computes the fraction of top-K features that are common between a given post hoc explanation and the corresponding ground truth explanation. Higher values are better. | RA RA measures the fraction of top-K features that are not only common between a given explanation and the corresponding ground truth explanation, but also have the same position in the respective rank orders. Higher values are better. | SA SA computes the fraction of top-K features that are not only common between a given explanation and the corresponding ground truth explanation, but also share the same sign in both the explanations. Higher values are better. | SRA SRA computes the fraction of top-K features that are not only common between a given explanation and the corresponding ground truth explanation, but also share the same feature attribution sign and rank in both the explanations. Higher values the better. | RC RC computes the Spearman’s rank correlation coefficient to measure the agreement between feature rankings provided by a given explanation and the corresponding ground truth explanation. Higher values the better. | PRA PRA captures if the relative ordering of every pair of features is the same for a given explanation as well as the corresponding ground truth explanation and computes the fraction of feature pairs for which the relative ordering is the same between the two explanations. Higher values the better. | PGI PGI measures the difference in prediction probability that results from perturbing the features deemed as influential by a given post hoc explanation. Higher values are better. | PGU PGU measures the difference in prediction probability that results from perturbing the features deemed as unimportant by a given post hoc explanation. Lower values are better. |
|---|---|---|---|---|---|---|---|---|
|
LIME
|
0.838 | 0.738 | 0.836 | 0.735 | 0.986 | 0.983 | 0.108 | 0.107 |
|
SHAP
|
0.619 | 0.281 | 0.527 | 0.267 | 0.469 | 0.692 | 0.128 | 0.087 |
|
Vanilla Gradient
|
0.857 | 0.857 | 0.790 | 0.791 | 1. | 1. | 0.109 | 0.105 |
|
SmoothGrad
|
0.857 | 0.857 | 0.299 | 0.299 | 1. | 1. | 0.148 | 0.054 |
|
Integrated Gradient
|
0.857 | 0.857 | 0.790 | 0.791 | 1. | 1. | 0.109 | 0.104 |
|
Gradient x Input
|
0.639 | 0.286 | 0.507 | 0.264 | 0.535 | 0.690 | 0.129 | 0.086 |
Stability
| Method | RIS RIS measures the maximum change in explanation relative to changes in the inputs. Values closer to zero are better (log 1 ~ 0). | ROS ROS measures the maximum change in explanation relative to changes in the output prediction probabilities. Values closer to zero are better (log 1 ~ 0). | RRS RRS measures the maximum change in explanation relative to changes in the internal representations learned by the model. Values closer to zero are better (log 1 ~ 0). |
|---|---|---|---|
|
SmoothGrad
|
-1.10 | -2.93 | -2.93 |
|
Gradient x Input
|
2.98 | 1.78 | 1.78 |
|
SHAP
|
3.12 | 2.35 | 2.35 |
|
LIME
|
4.24 | 2.28 | 2.28 |
|
Integrated Gradient
|
4.33 | 1.41 | 1.41 |
|
Vanilla Gradient
|
4.77 | 1.82 | 1.82 |
Fairness
| Method | Fairness@FA Computes the absolute difference in FA scores between the majority and the minority subgroups. Lower disparities are better. | Fairness@RA Computes the absolute difference in RA scores between the majority and the minority subgroups. Lower disparities are better. | Fairness@SA Computes the absolute difference in SA scores between the majority and the minority subgroups. Lower disparities are better. | Fairness@SRA Computes the absolute difference in SRA scores between the majority and the minority subgroups. Lower disparities are better. | Fairness@RC Computes the absolute difference in RC scores between the majority and the minority subgroups. Lower disparities are better. | Fairness@PRA Computes the absolute difference in PRA scores between the majority and the minority subgroups. Lower disparities are better. | Fairness@PGI Computes the absolute difference in PGI scores between the majority and the minority subgroups. Lower disparities are better. | Fairness@PGU Computes the absolute difference in PGU scores between the majority and the minority subgroups. Lower disparities are better. | Fairness@RIS Computes the absolute difference in RIS scores between the majority and the minority subgroups. Lower disparities are better. | Fairness@ROS Computes the absolute difference in ROS scores between the majority and the minority subgroups. Lower disparities are better. | Fairness@RRS Computes the absolute difference in RRS scores between the majority and the minority subgroups. Lower disparities are better. |
|---|---|---|---|---|---|---|---|---|---|---|---|
|
SmoothGrad
|
0.048 | 0.036 | 0.024 | 0.029 | 0.019 | 0.023 | 0.088 | 0.073 | 0.036 | 0.046 | 0.046 |
|
Vanilla Gradient
|
0.046 | 0.034 | 0.011 | 0.006 | 0.001 | 0.002 | 0.069 | 0.069 | 0.068 | 0.023 | 0.023 |
|
Integrated Gradient
|
0.046 | 0.034 | 0.011 | 0.006 | 0.001 | 0.001 | 0.067 | 0.069 | 0.056 | 0.008 | 0.008 |
|
LIME
|
0.045 | 0.048 | 0.043 | 0.052 | 0.037 | 0.037 | 0.063 | 0.067 | 0.028 | 0.011 | 0.011 |
|
SHAP
|
0.029 | 0.016 | 0.060 | 0.045 | 0.032 | 0.026 | 0.070 | 0.072 | 0.058 | 0.036 | 0.036 |
|
Gradient x Input
|
0.026 | 0.019 | 0.063 | 0.048 | 0.035 | 0.029 | 0.071 | 0.069 | 0.079 | 0.058 | 0.058 |
Faithfulness
| Method | PGI PGI measures the difference in prediction probability that results from perturbing the features deemed as influential by a given post hoc explanation. Higher values are better. | PGU PGU measures the difference in prediction probability that results from perturbing the features deemed as unimportant by a given post hoc explanation. Lower values are better. |
|---|---|---|
|
LIME
|
0.083 | 0.181 |
|
SHAP
|
0.113 | 0.153 |
|
Vanilla Gradient
|
0.079 | 0.183 |
|
SmoothGrad
|
0.184 | 0.084 |
|
Integrated Gradient
|
0.073 | 0.185 |
|
Gradient x Input
|
0.107 | 0.154 |
Stability
| Method | RIS RIS measures the maximum change in explanation relative to changes in the inputs. Values closer to zero are better (log 1 ~ 0). | ROS ROS measures the maximum change in explanation relative to changes in the output prediction probabilities. Values closer to zero are better (log 1 ~ 0). | RRS RRS measures the maximum change in explanation relative to changes in the internal representations learned by the model. Values closer to zero are better (log 1 ~ 0). |
|---|---|---|---|
|
LIME
|
1.25 | 3.31 | 7.82 |
|
SHAP
|
1.27 | 3.06 | 6.93 |
|
Gradient x Input
|
1.89 | 3.54 | 6.67 |
|
SmoothGrad
|
-3.03 | -1.02 | 3.20 |
|
Integrated Gradient
|
3.07 | 4.76 | 7.82 |
|
Vanilla Gradient
|
3.91 | 5.63 | 8.46 |
Fairness
Fairness evaluation is not applicable for the Give me some credit dataset as it does not include any protected attributes.
Faithfulness
| Method | FA FA computes the fraction of top-K features that are common between a given post hoc explanation and the corresponding ground truth explanation. Higher values are better. | RA RA measures the fraction of top-K features that are not only common between a given explanation and the corresponding ground truth explanation, but also have the same position in the respective rank orders. Higher values are better. | SA SA computes the fraction of top-K features that are not only common between a given explanation and the corresponding ground truth explanation, but also share the same sign in both the explanations. Higher values are better. | SRA SRA computes the fraction of top-K features that are not only common between a given explanation and the corresponding ground truth explanation, but also share the same feature attribution sign and rank in both the explanations. Higher values the better. | RC RC computes the Spearman’s rank correlation coefficient to measure the agreement between feature rankings provided by a given explanation and the corresponding ground truth explanation. Higher values the better. | PRA PRA captures if the relative ordering of every pair of features is the same for a given explanation as well as the corresponding ground truth explanation and computes the fraction of feature pairs for which the relative ordering is the same between the two explanations. Higher values the better. | PGI PGI measures the difference in prediction probability that results from perturbing the features deemed as influential by a given post hoc explanation. Higher values are better. | PGU PGU measures the difference in prediction probability that results from perturbing the features deemed as unimportant by a given post hoc explanation. Lower values are better. |
|---|---|---|---|---|---|---|---|---|
|
LIME
|
0.828 | 0.585 | 0.828 | 0.576 | 0.926 | 0.911 | 0.060 | 0.264 |
|
SHAP
|
0.530 | 0.155 | 0.462 | 0.143 | 0.156 | 0.556 | 0.113 | 0.225 |
|
Vanilla Gradient
|
0.900 | 0.900 | 0.886 | 0.886 | 1.000 | 1.000 | 0.066 | 0.259 |
|
SmoothGrad
|
0.900 | 0.900 | 0.369 | 0.369 | 1.000 | 1.000 | 0.277 | 0.056 |
|
Integrated Gradient
|
0.900 | 0.900 | 0.886 | 0.886 | 1.000 | 1.000 | 0.066 | 0.259 |
|
Gradient x Input
|
0.553 | 0.144 | 0.455 | 0.138 | 0.233 | 0.562 | 0.098 | 0.232 |
Stability
| Method | RIS RIS measures the maximum change in explanation relative to changes in the inputs. Values closer to zero are better (log 1 ~ 0). | ROS ROS measures the maximum change in explanation relative to changes in the output prediction probabilities. Values closer to zero are better (log 1 ~ 0). | RRS RRS measures the maximum change in explanation relative to changes in the internal representations learned by the model. Values closer to zero are better (log 1 ~ 0). |
|---|---|---|---|
|
SHAP
|
1.44 | 0.09 | 0.09 |
|
Gradient x Input
|
2.06 | 0.15 | 0.15 |
|
LIME
|
2.56 | 1.81 | 1.81 |
|
Integrated Gradient
|
3.12 | 0.59 | 0.59 |
|
SmoothGrad
|
-3.20 | -3.56 | -3.56 |
|
Vanilla Gradient
|
3.96 | 1.38 | 1.38 |
Fairness
Fairness evaluation is not applicable for the Give me some credit dataset as it does not include any protected attributes.