The Challenge: The Economic Safety Gap
Technological innovation, particularly in fields like AI, robotics, and automation, is advancing at an unprecedented pace. However, safety engineering has struggled to keep up. This disparity has created an Economic Safety Gap – the gap between what is technologically possible and what can be safely deployed (Paper: Mind the economic safety gap). For safety-critical systems, this gap poses significant economic challenges, particularly for small and medium-sized enterprises (SMEs), which often lack the resources to meet complex safety demands.
The Consequences
As new technologies become more complex, ensuring their safety becomes increasingly difficult and costly. Without significant innovation in safety methods, companies may need to choose between two undesirable outcomes:
- Slowing Innovation: Prioritizing safety but leaving vast technological and economic potential untapped.
- Accepting Risk: Deploying systems with potentially insufficient safety measures, which could lead to safety incidents and regulatory non-compliance.
Clearly, neither option is sustainable. A viable path forward instead is to accelerate innovation in safety engineering.
The Solution: Generative AI for Safety Engineering
To address this challenge, researchers at Fraunhofer IKS propose using Generative Artificial Intelligence (genAI) to automate aspects of the safety engineering workflow. Specifically, genAI-powered tools can support engineers during activities like Hazard and Risk Analysis (HARA) or Safety Requirements Engineering – both critical steps in safety-critical system development.
A Prototype Assistant
Fraunhofer IKS has developed a proof-of-concept genAI assistant for HARA. This tool allows safety engineers to interact with a Large Language Model (LLM) to identify potential hazards and assess their risks. Using this tool makes it possible to:
- Reduce Time and Cost: Automating parts of the process can cut safety assurance timelines from months to weeks.
- Enable Complexity: Make safety assurance feasible for more complex systems.
- Broaden Accessibility: Help SMEs afford rigorous safety analyses, reducing reliance on niche expertise.
- Strengthen Safety Cases: Provide more thorough and consistent analyses by leveraging data from past projects.
Research Challenges
Although promising, the use of genAI in safety-critical applications comes with challenges which Fraunhofer IKS addresses in the scope of the DSgenAI project:
- Trustworthiness: Current AI models can produce “hallucinations” – outputs that are factually incorrect or misleading. Techniques like retrieval-augmented generation and attribution are being explored to improve reliability.
- Collaboration: Engineers must remain in the loop to validate AI-generated outputs. Iterative workflows and user-friendly interfaces are essential to facilitate effective human-AI collaboration.
- Specialized Models: Fine-tuning LLMs for safety engineering tasks may improve the quality of output for critical tasks. This often requires application-specific, realistic datasets. Collaboration across industries could help create standardized, domain-specific safety models.
The Path Forward
Fraunhofer IKS is committed to advancing genAI tools for safety-critical applications. By innovating safety engineering processes, we can ensure that technological progress is both safe and economically viable.
Author: Andreas Kreutz and Florian Geissler