AI Explainability: How to Avoid Rubber-Stamping Recommendations

For the fourth year in a row, MIT Sloan Management Review and Boston Consulting Group (BCG) have assembled an international panel of AI experts that includes academics and practitioners to help us understand how responsible artificial intelligence (RAI) is being implemented across organizations worldwide. In the spring of 2025, we also fielded a global executive survey yielding 1,221 responses to learn the degree to which organizations are addressing responsible AI. In our first article this year, we focused on accountability for general-purpose AI producers. In this piece, we examine the relationship between explainability and human oversight in providing accountability for AI systems.

In the context of AI governance, explainability refers to the ability to provide humans with clear, understandable, and meaningful explanations for why an AI system made a certain decision. It is closely related to, but broader than, the more technical notion of interpretability, which focuses on understanding the inner workings of how a model’s inputs influence its outputs. Both concepts seek to improve the transparency of increasingly complex and opaque AI systems and are also reflected in recent efforts to regulate them. For example, the European Union’s AI Act requires that high-risk AI systems be designed to enable effective human oversight and grants individuals a right to receive “clear and meaningful explanations” from the entity deploying the system. South Korea’s comprehensive AI law introduces similar requirements for “high-impact” AI systems (in sectors like health care, energy, and public services) to explain the reasoning behind AI-generated decisions. Companies are responding to these requirements by launching commercial governance solutions, with the explainability market alone projected to reach $16.2 billion by 2028.

The growing focus on explainability and oversight leads to the natural question of whether one can exist without the other, which is why we asked our panel to react to the following provocation: Effective human oversight reduces the need for explainability in AI systems. A clear majority (77%) disagree or strongly disagree with the statement, arguing that explainability and human oversight are complementary, not competing, aspects of AI accountability — the presence of one does not diminish the need for the other.

Beyond the practical role that explainability plays in helping humans exercise control over AI outputs, our experts also emphasize that it promotes deeper societal values, such as trust, transparency, fairness, and due process. Without explainability, they caution, human overseers are reduced to rubber-stamping decisions made by machines, raising a threat to those values. Still, they acknowledge that explainability has its limits in practice. Below, we share insights from our panelists and draw on our own RAI experience to recommend how organizations can bolster AI accountability through meaningful explainability and human oversight.

Explainability and human oversight are complementary, not competing, aspects of AI accountability. The majority of our panelists believe this and consequently think that effective human oversight does not reduce the need for explainability. As Bruno Bioni, Data Privacy Brasil’s founding director, puts it, “Explainability and human oversight constitute complementary and intersecting safeguards within AI governance frameworks.”

He further explains, “Their interrelation does not render them mutually exclusive, nor does the presence of one negate or diminish the relevance of the other.” National University of Singapore’s Simon Chesterman echoes this view: “Oversight and explainability are not competing values [but] mutually reinforcing pillars of responsible AI governance.” Similarly, Renato Leite Monteiro of e& emphasizes that “human oversight and explainability operate as complementary forces rather than substitutes.” Or, as Intel Labs’s Elizabeth Anne Watkins sums it up, “They work in tandem.”

This complementarity stems from the fact that human oversight and explainability have distinct uses. ForHumanity’s Ryan Carrier explains, “Human oversight serves immensely different purposes than explainability, and therefore they should not be considered as trade-offs or substitutions for each other.” United Nations University’s Tshilidzi Marwala similarly contends that “human oversight and explainability are distinct yet powerfully synergistic concepts.”

Alberta Machine Intelligence Institute’s Alyssa Lefaivre Škopac notes, “Human oversight is essential for making sure AI is used safely and responsibly, but that doesn’t take away the need for explainability.” She further explains, “If the person overseeing the system doesn’t understand how the AI is making decisions, it becomes much harder to guide or adjust it in the right way.”

Australian National University’s Belona Sonna underscores that explainability “facilitates effective human oversight by providing insights into how the model operates, which is paramount for identifying and addressing potential sources of bias, discrimination, or inequity.” EnBW’s Rainer Hoffmann reinforces this point and goes further, noting that explainability helps “developers to debug systems, regulators to ensure compliance, and business leaders to manage risk” — in other words, explainability solves for more than system outputs.

Ultimately, explainability enables humans to understand why machines make the decisions they do, while human oversight ensures outputs of AI systems are accurate, free from unforeseen risks, and consistent with organizational values. As a result, H&M Group’s Linda Leopold argues that “rather than reducing the need for explainability, effective human oversight actually relies on it,” and she describes human oversight as “making sure that systems behave as intended.”

Carrier echoes this view, noting that human oversight is “necessary for intervenability, control, alignment, and risk management.” Stanford’s Riyanka Roy Choudhury also highlights the connection between explainability and informed judgment, stating, “Understanding AI reasoning allows informed human judgment to trust or override” decisions. IAG’s Ben Dias adds that explainability helps AI end users and system operators “understand the outputs and more easily identify outliers and errors,” while “human oversight is crucial to ensure that AI systems operate as intended and to mitigate the risk of unforeseen consequences.”

Explainability promotes deeper societal values to avoid humans merely rubber-stamping machines. In addition to its practical benefits, our panelists also believe that explainability supports deeper societal values, such as trust, transparency, fairness, and due process, that cannot be provided by human oversight alone. As Monteiro puts it, explainability “serves critical functions beyond facilitating human oversight and ultimately reflects deeper societal values of autonomy, fairness, due process, and accountability that transcend operational considerations.”

Marwala similarly emphasizes that “while oversight acts as a crucial immediate safeguard against operational risks, explainability providers a deeper, more enduring value [because it] fosters transparency, builds trust, and underpins sound governance.” RAIght.ai’s Richard Benjamins gives an example in the health care context, saying a doctor should not accept “a diagnosis from an AI system without being able to understand the reason behind the diagnosis,” since AI systems are error-prone. “Accepting that error rate without understanding the reason behind the diagnosis does not feel professionally correct,” he says. Nasdaq’s Douglas Hamilton similarly notes that humans are seeking “satisfying explanations” that “match our gut feel or our expertise as to why things occur,” which is different from the strictly computational explanations that AI systems often give.

This intuitive link between explainability, transparency, and trust is further articulated by several panelists, including Škopac, who notes, “Transparency around how decisions are made helps to build trust.” Dias agrees, stating, “Explainability assists end users in comprehending the outcomes of their interactions with an AI system.” He adds that systems “must be explainable in order to establish trust with their end users.”

Idoia Salazar of OdiseIA also underscores the importance of explainability for trust in consequential decision-making, asserting that “trust in AI systems cannot rely solely on human monitoring [but] requires transparent, interpretable systems that enable informed human judgment.” She notes that this is particularly important in “domains like health care, finance, or criminal justice.” Benjamins echoes this point, highlighting the risks in AI “systems that provide medical diagnoses; give or deny access to essential services, like financial services or school enrollment; and estimate the probability of social service fraud.” Choudhury says this relationship is “reinforced by regulations like the EU AI Act, which pushes against black-box systems in high-risk applications.”

The heightened need for AI explainability in high-stakes contexts is also rooted in fundamental notions of fairness and due process. Bioni says, “Lacking explainability by design can severely compromise both meaningful human intervention and what we increasingly refer to as informational due process in automated decision-making.” Monteiro adds that explainability “empowers individuals to understand and challenge automated decisions affecting them — a fundamental right in most privacy and data protection frameworks” that helps “create the foundation for meaningful trust in automated systems, rather than merely supporting oversight functions.” Finally, Carrier appeals to explainability requirements, saying, “The requirement to describe in clear and plain language the basic logic of an AI system [and its] potential consequences” is “the right thing to do.”

In fact, several experts emphasize that human oversight without explainability can only ever be superficial. DBS Bank’s Sameer Gupta warns, “Without clear insight into how and why an AI system reaches its conclusions, oversight becomes superficial, reducing human involvement to a rubber stamp rather than acting as a critical check.” He adds, “Explainability ensures that AI decisions can be understood, evaluated, and corrected when needed — especially vital when outcomes are unexpected or problematic.”

Chesterman echoes this, noting that “opacity over time can encourage a rubber-stamping role for any human only notionally ‘in the loop.’” Salazar believes that without explainability, “oversight becomes superficial and reactive rather than proactive and corrective,” while Hoffmann cautions it can lead to “misplaced trust.” Chow argues that without explainability, human oversight creates “a dangerous illusion of control” and that “increasing human oversight of AI systems may actually heighten rather than diminish the need for explainability.”

In practice, AI explainability has limits. Despite the practical and principled importance of explainability, our panelists acknowledge that it is not always feasible or necessary in every context. Standard Chartered Bank’s David R. Hardoon believes the need for explainability is context-dependent, stressing that “it is important to be specific in what needs to be explained, to whom, and for what purpose” rather than appeal to some abstract notion of explainability.

Chesterman notes that transparency and explainability come with “trade-offs in complexity and performance.” University of Helsinki’s Teemu Roos agrees that explainability can “come in many shapes and forms.” Moreover, he concedes that while explainability may sometimes involve “clearly describing a system’s behavior,” in other cases “modern AI systems have become way too complex for us to cling to this type of explainability.”

Apollo Global Management’s Katia Walsh goes further, saying, “AI explainability is a fallacy,” contending that “AI reasoning systems have reached a level of sophistication and complexity that rivals the human brain.” Finally, Škopac emphasizes that “there are times when explainability just isn’t possible,” and in those cases, “strong oversight and domain expertise are critical for keeping things on track” instead.

Recommendations

In sum, we offer the following recommendations for organizations seeking to provide meaningful accountability through human oversight and the explainability of AI systems:

1. AI systems should be designed to enable effective human oversight. To enable human oversight, require AI systems to produce evidence for or against results; include detailed audit and change logs; monitor and compare reject rates in practice against those observed in testing; flag decisions for escalation; and present outputs with contextual indicators or confidence scores; among other measures.

2. Human oversight competencies need to be cultivated. Looking beyond system design, be sure system users are thoroughly trained in AI generally and in using specific systems to develop human oversight competencies. Content or process expertise is insufficient for effective human oversight of AI systems. Users must also understand AI limitations, biases, and failure modes to make informed judgments, intervene appropriately, and ensure accountability in AI-assisted decision-making.

3. Human oversight of AI system use must address many types of explainability. In some use cases, effective human oversight must always involve reviewing explanations from AI systems (such as a counterintuitive medical diagnosis). In other cases, effective human oversight might be less frequent (such as for inventory management or pricing recommendations). Consider: Under what conditions should effective human oversight demand a review of AI system decision-making explanations? How can your organization avoid “explainability theater”?

4. Meaningful accountability must avoid the illusion of control. Explainability and human oversight only matter when they lead to real understanding and meaningful accountability. If practical constraints limit explainability, stressing it too much can create a false sense of control. Likewise, superficial oversight can give the illusion of accountability without substance. Instead of rigidly adhering to formal ideals, organizations should evaluate how explainability and oversight actually work in context and adjust their approach to match what’s most meaningful in practice.