AI Safety Testing Framework for LLM and Agentic Systems
This case study describes the framework and its public deliverable types. The playbook, templates, case materials, source files, and any research article are not published here.
Bridging principles and executable tests
AI governance and risk documents describe properties such as robustness, transparency, accountability, and human oversight, but teams still need a consistent way to turn those expectations into testable system requirements. I led this independent project to organize that translation into one end-to-end evaluation structure for language-model and agentic systems.
A four-layer assurance structure
The framework connects governance context to technical evaluation and operational decisions. Public guidance informed the structure, while the project defined its own testing and evidence model.
- Mapped the EU AI Act, NIST AI RMF, ISO/IEC 42001 and 23894, and UN principles to operational testing needs.
- Defined critical capability levels, risk tiers, seven trustworthiness dimensions, and adversarial test categories.
- Connected safety and deployment levels to evidence requirements and follow-up verification.
From risk framing to mitigation verification
The workflow begins with system context and risk analysis, moves through test selection and evidence collection, and ends with a structured decision record. Findings are not treated as the end of the process: mitigation actions return to the test plan so controls can be checked again under the same evaluation logic.
- Risk analysis and threat-model definition
- Test planning across capability, safety, and misuse concerns
- Multidimensional scoring and evidence review
- Safety-case assembly, deployment decision, and mitigation re-test
A toolkit for repeatable review
The project produced a testing playbook, compliance scorecard, safety-case template, risk-analysis template, and web prototype. I piloted the workflow on three case-study systems to examine whether the structure remained usable across different evaluation contexts.
Independent methodology, not certification
The framework is an independent research and design project. References to laws, standards, and public frameworks describe mapping and methodological alignment; they do not represent endorsement, certification, a formal conformity assessment, or legal advice.