The perils of blind trust in AI – why human accountability must not be automated away
03 Jul 2024
Blog
This blog covers:
Air Canada generated unwanted headlines in February after the airline’s chatbot misled a grieving customer into purchasing full-price flight tickets. The customer, seeking bereavement fares after the death of their grandmother, followed the chatbot’s ill-informed advice and ended up paying significantly more than they should have.
In a stunning attempt to evade responsibility, Air Canada argued that its chatbot was “a separate legal entity” accountable for its actions. The adjudicator, understandably astounded by this claim, firmly rejected the airline’s argument, stating: “It should be obvious to Air Canada that it is responsible for all the information on its website. It makes no difference whether the information comes from a static page or a chatbot.”
This case is a stark reminder of the perils of blind trust in artificial intelligence (AI) and the importance of maintaining human accountability in an increasingly automated world.
Beware AI worship
As AI becomes more sophisticated and ubiquitous, business leaders are more likely to fall into the trap of viewing it as something capable of solving every problem and making every decision. This mindset, which we might call “AI worship,” is misguided and actively dangerous. It’s exactly 20 years since Little Britain’s receptionist Carol Beer coined the catchphrase “computer says no,” but now this reality is no laughing matter.
When organizations place blind faith in AI, they risk abdicating human responsibility and judgment. Employees may feel pressure to defer to the algorithm, even when its recommendations seem questionable or counterintuitive. This pressure – and perhaps systematic laziness – can lead to a lack of accountability, as individuals and teams disclaim ownership of decisions by arguing that they were following the system’s advice.
Moreover, as hinted above, an over-reliance on AI can breed complacency and erode critical thinking skills. When algorithms are trusted to make decisions, the incentive for humans to engage in independent analysis and reasoning diminishes. Over time, this can lead to an atrophying of the skills and capabilities essential for sound judgment and effective oversight.
The importance of human oversight
To mitigate these risks, business leaders must ensure that human oversight and accountability remain firmly in place, even as AI takes on a greater role in decision-making. AI will increasingly make decisions for us but the decision criteria, materiality threshold and the consequences of these decisions need to be firmly established and understood.
Admittedly, in supply chain management, for instance, AI has the potential to revolutionize demand forecasting and inventory optimization. It’s useful, because it can analyze vast amounts of data – from historical sales figures to weather patterns or material/ commodity pricing – and provide valuable insights and recommendations to inform inventory management decisions.
AI has its limits, though. Hence, a thoughtful, proactive approach to AI governance that prioritizes transparency, explainability, and human control is needed. Rather than simply deferring to the algorithm, organizations should insist on AI systems that provide clear, understandable rationales for their recommendations. This approach allows human decision-makers to interrogate the logic behind the AI’s suggestions and identify potential biases or errors.
Establishing clear governance frameworks is also vital. This process involves developing policies and guidelines around AI development, deployment, monitoring, and designating specific roles and responsibilities for overseeing AI initiatives. By creating a clear chain of command and accountability, organizations can ensure that AI is used responsibly and ethically and that humans remain firmly in the loop.
Fostering a culture of healthy skepticism
Ultimately, the success of AI governance depends on an organization’s culture and mindset. This means encouraging employees to ask probing questions about AI recommendations, not dismissing them outright but truly understanding the logic and data behind them. Teams should strive to uncover what’s included in the historical baseline that informs the AI’s projections rather than overriding the system based on intuition alone.
At the same time, the culture must accommodate the reality that AI insights may diverge from expectations or the “answer we thought we should get.” Employees need an environment where they feel empowered to surface and explore these divergences, seeing them as opportunities for learning and improvement rather than inconvenient anomalies to suppress.
Plans often have multiple inputs, and a curious, challenging mindset is key to continuously refining those inputs and the AI systems they feed. The skepticism that's needed is not a blanket distrust of AI, but rather an open-minded rigor in understanding how it arrives at its recommendations.
Fostering this nuanced culture requires commitment and intentional change management. Leaders must model the right mindset, communicate openly about the role and limitations of AI, and create forums for employees to ask hard questions and surface concerns without fear. With the right balance of healthy challenge and receptivity to data-driven insights, organizations can unlock the full potential of AI while keeping human judgment firmly in the driver’s seat.
Striking the balance
The Air Canada chatbot case is just one example of the challenges and risks that arise when organizations place too much trust in AI. Other shocking news stories will follow this path in the coming weeks and months. Similar concerns have emerged in fields ranging from healthcare – are you ready for a digital doctor to diagnose you now? – to aviation, where many travelers remain uncomfortable with fully autonomous planes. As another example, there is a reason the brakes have been pumped on driverless cars.
As you navigate your organization’s AI journey, it’s crucial to keep these risks in mind and work to mitigate them actively. Here are some core questions to consider:
- Have we established clear governance frameworks and accountability measures to ensure that humans remain in control of critical decisions?
- Has the decision criteria and the correct materiality threshold been established and when is acceptable for AI to make decisions?
- Are we insisting on transparent, explainable AI systems allowing human oversight and interrogation?
- Are we fostering a healthy skepticism and critical thinking culture, where employees feel empowered to question AI recommendations and raise concerns?
By proactively addressing these issues and striking the right balance between human and artificial intelligence, business leaders can harness the power of AI while safeguarding against its pitfalls. The alternative – blindly automating away human judgment and accountability – is a risk no organization can afford to take.
Author(s)
-
