Project ongoing · Artificial intelligence
ExplAIn
We build tools to understand how an artificial intelligence decides.
- still trust the model, after seeing its explanation
- 3 out of 27
- programs analysed, half malware and half clean
- 420
- risk levels in the European AI Act
- 4
- the right to an explanation
- Art. 86
For people who come from computer science, mathematics, philosophy, law
Why explain
An artificial intelligence model gives an answer but almost never says why. If the answer is right for the wrong reason, nobody notices until the wrong reason stops working.
Explanation methods do not change the model: they place a diagnostic tool alongside it, which shows which parts of the input decided the answer. This is how the errors that are not visible get found.
The wolf and the snow
In 2016 Ribeiro, Singh and Guestrin deliberately trained a wrong model: in all the example photos the wolves were on snow and the huskies were not. The model learned the snow: it says wolf if there is snow, husky if there is not, whichever animal is actually there.
Their method, LIME, switches groups of pixels on and off and watches how the answer changes, then builds a simple model that imitates the real one close to that image. The explanation highlights the snow.
They showed the answers to 27 students with at least one machine learning course. Before the explanation 10 trusted the model; afterwards, only 3.
The numbers in the chart
| People | before the explanation | after |
|---|---|---|
| Trust the model | 10 | 3 |
| Point to the snow as a clue | 12 | 25 |
Integrated Gradients
Another method starts from a neutral input, for example a black image, and reaches the real input in small steps. At every step it measures how sensitive the answer is to each piece of the input, and sums the contributions.
IG_i(x) = (x_i − x′_i) · ∫ from 0 to 1 ∂F(x′ + α·(x − x′)) / ∂x_i dα
x the real input x′ the neutral input
F the model α the step, from 0 to 1
in practice: from 20 to 300 steps, and the sum of the contributions must give F(x) − F(x′)- 01
Completeness
The summed contributions give exactly the difference between the real answer and the neutral one.
- 02
Sensitivity
If changing a piece changes the answer, that piece receives a contribution.
- 03
Independence from the code
Two networks that compute the same function have the same explanations.
- 04
Symmetry
Interchangeable, equal pieces receive the same contribution.
The malware and the bytes that do not count
Some malware detectors read the bytes of Windows programs directly, and work very well. But what do they base their decisions on? The study by Perasso and colleagues, presented at Ital-IA 2025, measures this with Integrated Gradients on two networks, MalConv and BBDNN, with 210 malware samples from seven families and 210 clean Windows 11 programs.
In a program there are areas that should not count: the DOS header, which is kept only for compatibility; the padding between the sections; the bytes added at the end. And there is the real code, the .text section. Both networks give most of the relevance to the code, but a significant share ends up in the areas that do not count, especially in the padding of the clean programs.
The numbers in the chart
| Network and programs | Average relevance |
|---|---|
| MalConv (clean programs) | 0,26 |
| MalConv (malware) | 0,09 |
| BBDNN (clean programs) | 0,41 |
| BBDNN (malware) | 0,11 |
Translation from Turkish
In Turkish the pronoun "o" stands for both him and her. A study published in Science in 2017 showed that when translating "O bir doktor. O bir hemşire." Google Translate wrote "He is a doctor. She is a nurse.": him the doctor, her the nurse. The gender was not in the sentence, the model added it, learning it from the texts.
Since December 2018, for short sentences from Turkish, Google shows both translations. The bias was in the data; spotting it took looking inside the answers.
The European rules
The AI Act, European regulation 2024/1689, has been in force since 1 August 2024 and divides artificial intelligence systems into four risk levels. For the high-risk ones it requires transparency, human oversight, accuracy and robustness; and anyone affected by a decision taken with their help has the right to an explanation, under Article 86.
- Unacceptable risk banned: social scoring, manipulation, emotion recognition at school and at work
- High risk education, employment, essential services, critical infrastructure, medical devices
- Transparency risk chatbots and generated content must be declared
- Minimal risk no obligations: spam filters, video games
The numbers in the chart
| Livello | |
|---|---|
| Unacceptable risk | banned: social scoring, manipulation, emotion recognition at school and at work |
| High risk | education, employment, essential services, critical infrastructure, medical devices |
| Transparency risk | chatbots and generated content must be declared |
| Minimal risk | no obligations: spam filters, video games |
- 21 April 2021The European Commission proposes the regulation.
- 1 August 2024Regulation 2024/1689 enters into force.
- 2 February 2025The prohibitions apply.
- 2 August 2025The rules on general-purpose models apply.
- 27 July 2026Regulation 2026/1744 enters into force, which shifts the deadlines for high risk.
- 2 December 2027Obligations for high-risk systems in the areas of Annex III, such as education and employment.
- 2 August 2028Obligations for artificial intelligence inside regulated products, such as medical devices.
Where we are
- June 2025At Ital-IA 2025, in Trieste, the study on spurious correlations in malware detection: it is one of the three cases in the project.
- April 2026We presented the project at Palazzo della Borsa, with three cases: images, cybersecurity and translation.
- Next stepA toolkit that anyone can reuse, ready for the European rules on artificial intelligence.
Photos
People on the project
Who we do it with
Sources
- Ribeiro, Singh and Guestrin, "Why Should I Trust You?", KDD 2016.
- Sundararajan, Taly and Yan, Axiomatic Attribution for Deep Networks, ICML 2017.
- Perasso, Lozza, Ponte, Demetrio, Oneto and Roli, Empirical Quantification of Spurious Correlations in Malware Detection, Ital-IA 2025, CEUR Workshop Proceedings 4121.
- Caliskan, Bryson and Narayanan, Science, 2017; Google, Reducing gender bias in Google Translate, 2018.
- Regulation (EU) 2024/1689 and regulation (EU) 2026/1744.
- Presentation at the Deep-Tech Showcase, Palazzo della Borsa, 21 April 2026.
Do you want to work on it?
You do not need experience and you do not need a CV. Write to us: we invite you to the next meeting, where you meet the team.