Explainable AI (XAI) is the practice of building AI systems whose individual outputs come with reasons that the people affected and the people deciding can understand, check against the facts and challenge, rather than a bare score or label.
What counts as a good explanation
NIST's Four Principles of Explainable Artificial Intelligence (NISTIR 8312) gives a useful checklist. An explainable system should meet all four.
- Explanation. The system gives evidence or reasons with its outputs.
- Meaningful. The reasons make sense to the person receiving them. A partner, a candidate and an engineer need different explanations of the same score.
- Explanation accuracy. The reasons truly reflect how the system produced the output. A plausible story that does not match the model fails this test.
- Knowledge limits. The system works only within the conditions it was designed for, and says so when it is outside them or unsure.
The UK Information Commissioner's Office and the Alan Turing Institute, in Explaining decisions made with AI, split explanations into types: rationale, responsibility, data, fairness, safety and performance, and impact. Most teams start with rationale (why this output) and responsibility (who to ask).
How systems are made explainable
- Interpretable by design. A points-based scorecard or a small linear model can be read directly. Each factor's contribution is visible.
- Post-hoc attribution. For complex models, techniques estimate how much each input moved a particular output. These are approximations, so they need checking against the explanation accuracy principle.
- Counterfactuals. "The score would have been 7 instead of 5 if the company had a second paying customer." Easy for non-specialists to use.
- Evidence citation. Each part of a score points to the specific fact it relied on, as in evidence-based scoring. The reader checks the source rather than trusting the model.
The term took hold partly through the DARPA Explainable AI (XAI) program, which funded research into models that can explain themselves to human users.
Why explainability matters in hiring and investing
A person who cannot see why a system ranked them low cannot correct a mistake in their data or argue that the criterion was wrong. That is why explanation shows up in law. Under the EU AI Act, Article 86 gives people affected by decisions based on certain high-risk systems, including recruitment tools, a right to clear and meaningful explanations of the AI system's role and the main elements of the decision. The Act's high-risk rules apply from 2 December 2027, so check the current text for when this right takes effect. This is a summary, not legal advice.
For decision makers, explanations are how human-in-the-loop review becomes real. A partner can only overrule a score they understand. In AI candidate scoring, a hiring manager who sees the reasons can spot when the tool rewarded the wrong thing.
Worked example: two ways to show the same score
Northwind Ventures, a fictional seed fund, scores Ledgerly, a fictional fintech, 6.8 out of 10 for thesis fit. Here are two versions of what the partner sees.
| Version A: bare score | Version B: explained score |
|---|---|
| Thesis fit: 6.8 | Thesis fit: 6.8 |
| Confidence: high | Raised the score: B2B payments infrastructure (company site, product page); two founders with prior exits (founder profiles) |
| Lowered the score: no revenue evidence found; customer list not public | |
| Limit: score based on public information only; no data room access |
With Version B, the partner asks the founders one question about revenue and rescoring takes a minute. With Version A, the partner either trusts 6.8 or ignores it. Version B meets all four NIST principles; Version A meets none.
Explainability vs interpretability
| Interpretability | Explainability | |
|---|---|---|
| Focus | How the model works inside | Why this particular output came out |
| Audience | Mostly builders and validators | Users, affected people, reviewers |
| Typical example | Reading a linear model's coefficients | "Ranked lower because no revenue evidence was found" |
| Possible without the other? | Yes, a readable model can still lack user-facing reasons | Yes, via post-hoc methods, but accuracy must be checked |
The terms overlap and many writers use them interchangeably. Explanations also need keeping: auditability is what lets an explanation be checked months later.
Common mistakes
- Explanations that do not match the model. A generic reason generated after the fact can mislead more than no reason.
- One explanation for every audience. Feature weights help an engineer, not a candidate.
- Listing twenty factors. Show the few that moved the result most.
- Hiding the limits. If the data was thin, say so next to the score.
- Treating explanation as the whole of governance. It is one control inside AI governance, alongside testing, oversight and records.
How ScoringFactory approaches it
ScoringFactory shows the receipts. Every score is cited to the line in the record it came from, so a partner or hiring manager can check the reasons, find gaps and disagree. The team makes the decision. Data handling and access practices are on the trust page.
Frequently asked questions
What is explainable AI?
Explainable AI is AI built so that each output comes with reasons people can understand and check. Instead of a bare score, the system shows which facts raised or lowered it, where those facts came from, and where its knowledge runs out. The aim is to let users and affected people verify and challenge results.
Why does explainability matter in hiring and lending decisions?
Both decide access to jobs or money, so mistakes cost people directly. An explanation lets the person correct wrong data and lets the decision maker spot a bad criterion. Law reflects this: the EU AI Act gives people a right to an explanation of the AI's role in decisions made with certain high-risk systems, including recruitment tools.
What is the difference between explainability and interpretability?
Interpretability is about understanding how a model works inside, such as reading its coefficients. Explainability is about giving a reason for a particular output to the person who needs it. A model can be interpretable to its builders yet give users no reasons, and a complex model can offer explanations through post-hoc methods.
Are post-hoc explanations reliable?
Sometimes. Methods that estimate each input's effect on an output are approximations and can disagree with each other or with the model. Test them: change the inputs an explanation says mattered and check that the output moves as claimed. NIST calls this explanation accuracy. Where stakes are high, prefer models whose reasoning can be read directly.
Sources
- Four Principles of Explainable Artificial Intelligence (NISTIR 8312), NIST, 2021
- Explaining decisions made with AI, UK Information Commissioner's Office, current text
- Regulation (EU) 2024/1689 (Artificial Intelligence Act), official text, EUR-Lex, Official Journal of the European Union
- DARPA's explainable AI (XAI) program: a retrospective (Gunning, Vorm, Wang and Turek, 2021), Applied AI Letters