Artificial intelligence (AI) is in the process of becoming an integral part of many industries, permeating even highly regulated sectors such as pharmaceuticals, manufacturing, and healthcare. These fields demand rigorous transparency, trustworthiness and accountability, making explainable AI (XAI) a fundamental requirement for effective adoption. The crucial question in the process of adopting AI-based algorithms in the industry is the distribution of responsibility between the human operator and the machine. Depending on the role of the human operator we can distinguish several types of processes:
- Human-in-the-Loop – the human operator is actively involved in the decision-making process, intervening in every decision or critical step.
- Human-on-the-Loop – the human operator is overseeing the process, intervening only when necessary (e.g. when certain pre-defined circumstances are met, under which the reliability of the algorithms is decreased)
- Human-in-Command – the human is responsible for overarching decisions and governance of the process, but does not directly monitor or control every operation (i.e. the human control required is managerial rather than operational)
The Driving Automation Levels developed by the Society of Automotive Engineers represent an exhaustive example of a well-established and accepted reference framework, governing the process of AI adoption in Autonomous Driving. It delimits six distinct stages in the process, at each stage specifying the distribution of responsibility, and the features required of the system.
What Is Trustful AI?
Trustful AI encompasses a holistic approach to AI systems that ensures stakeholders (developers, users, and regulators) can rely on their functionality, fairness, and safety. It represents a crucial milestone in meeting industry-compliance requirements and ensuring accountability. In edge environments, where real-time decision-making is paramount, understanding the rationale behind AI-driven actions becomes even more critical. Whether it’s an AI model detecting anomalies in manufacturing equipment, or making split-second decisions in autonomous vehicles, explainability ensures confidence and mitigates risks in time-sensitive scenarios. Building trust in AI involves three overlapping areas:
- Explainability – the ability to understand how an AI system arrives at a specific result. It encompasses both local explanations (for individual predictions) and global explanations (for the system’s overall behavior). Meanwhile developers focus on interpretability, a more technical subset of explainability, to diagnose and improve models, as well as to predict possible circumstances under which the reliability of the models might be compromised.
- Transparency – provides comprehensive visibility into the entire lifecycle of an AI system, including how it was designed, trained, and deployed. It requires the documentation of critical processes, such as data sourcing, model development, testing and validation.
- Traceability – tracks the history of data and decisions made by the AI system. It enables stakeholders to audit decisions, ensures accountability, and provides a mechanism for debugging or improving the model’s reliability.
The Challenges Trustful AI Faces
Adopting AI-based technologies in regulated industries comes with its own set of unique challenges. The complexity of modern AI models makes it difficult to provide clear, human-understandable justification for outcomes, which is a critical component for trust. Moreover, there is no universal solution to fit every kind of AI model. Despite that, explainability methods are continuously being investigated that have the potential to bridge the gap and demystify AI models. These include model-specific methods, tailored to specific model types (e.g. gradient-based saliency maps, layer-wise relevance propagation), and model-agnostic methods, which provide local explanations and can be applied across different architectures (e.g. Local Interpretable Model-Agnostic Explanations (LIME) and Shapley Additive Explanations (SHAP)). Another challenge is ensuring the robustness of the model. This implies the search for vulnerabilities before deployment, i.e. mechanisms that can be used to confuse AI models (both intentionally and unintentionally). One method that achieves this scope is through the computation of the minimum variation to the input that transforms a correct prediction into a wrong one. Finally, in addition to being data hungry, the AI model training process can be cumbersome, involving multiple iterations and complex hyperparameter optimization procedures. In this context it is paramount to maintain clear documentation as well as model and data versioning, to ensure that results can be reproduced, and that the performance of the model from one iteration to the next is not degrading.
The journey towards trustworthy AI is ongoing, and it is the shared responsibility of all involved parties, from academia and industry leaders to regulators and policymakers to create a clear and efficient framework of guidelines and good practices covering the entirety of the AI Lifecycle.
Blog signed by: COGNI team