Skip to content

SanityOps: What It Does, Who It Compares To, and What It Compares ​


Table of Contents ​


1. Why This Document Exists ​

An enterprise may have already procured AI development platforms to build Agents, deployed security products to intercept attack traffic, and introduced testing tools to run test cases. However, when Agent counts grow from dozens to thousands, enterprises discover: No existing system covers the complete closed loop of "Logic Artifact defects → security risk validation → service quality assessment → root-cause traceability."

  • Security products can intercept attacks that have already occurred, but cannot discover static defects in System Prompts, Skills, and Tool Schemas before production deployment;
  • Testing tools can produce pass rates, but cannot answer "which Logic Artifact is the root cause of failed test cases, and which test cases need regression after remediation";

SanityOps was designed precisely to fill this gap — it is a quality, security, and governance framework for enterprise-grade AI Agents. It does not handle building, orchestrating, or running Agents, but rather provides systematic inspection, audit, and quality measurement capabilities for Logic Artifacts that have already been built and their runtime results. It does not promise that Agent governance will always be correct or absolutely secure, but systematically reduces the uncertainty of Agent behavior and its associated business and security risks.


2. What SanityOps Can Do ​

SanityOps operates around four core activities, each answering a key question:

DomainCore QuestionWhat You Can DoWhat You Get
InspectDo Logic Artifact definitions or cross-artifact relationships contain static defects?Static analysis of System Prompts, Skills, Tool Schemas, and cross-artifact consistencyDefect ID (QD), static evidence, severity level, remediation recommendations
RiskDo Logic Artifacts contain explicit risks, or can security risk behaviors be triggered in controlled environments?Static audit of explicit risks (Explicit), dynamic validation of implicit risks in shadow sandboxes (Implicit)Risk classification (EX), impact type label (D1/D2/D3), severity grade (Explicit·P0/P1/P2), termination status (A/B/C/D), validation evidence
QualityDoes the Agent's actual task or user-visible service meet defined quality requirements?Controlled blind testing of RAG-Agents and Tool-Agents, with test rigor configured by business risk tierReliability metrics, test case results, quality Gate decisions, regression conclusions
RelevanceWhich attack surfaces, validation strategies, failure modes, and regression requirements may static defects be associated with?Map defects discovered by Inspect to candidate attack surfaces, failure modes, and test recommendationsAttack Surface (AS), Failure Mode (FM), defect chains, mapping recommendations, candidate root causes

Governance Closed-Loop Overview ​

The four activities are not isolated checkpoints, but a traceable closed-loop chain:

Logic Artifacts and their versions
    ↓
Inspect: Discover static defects (QD)
    ↓
Relevance: Map candidate attack surfaces, failure modes, test and remediation directions
    ↓
Risk: Confirm explicit risks or dynamic behavior evidence in controlled environments
    ↓
Quality: Assess actual tasks, outputs, and service quality
    ↓
Human Review: Determine root cause, control effectiveness, remediation scope, and residual risk
    ↓
Remediation, re-inspection, regression, release, or exception decision

Relevance serves as the connector: from defects (QD), candidate attack surfaces (AS) can be derived, Risk audit directions recommended, quality failure modes (FM) derived, defect chains identified, and post-remediation validation and regression scope recommended. However, correlation does not equal causation — mapping conclusions belong to risk correlation and validation recommendations, and do not automatically prove that attacks have succeeded or quality has failed.

Current Coverage Boundaries ​

SanityOps v1.0 has primarily defined: Inspect (including Tool / Prompt / Skill / Cross / Permission subsets), Risk Explicit, Risk Implicit, Quality Tool-Agent, Quality RAG-Agent, and Inspect defect impact and linkage mapping. Production runtime monitoring, incident response, and Knowledge / Memory / Identity / A2A governance are directions for subsequent expansion; the above directions should not be characterized as currently fully covered specialized specification capabilities.

Delivery Form ​

SanityOps provides two complementary delivery forms:

  • Framework (Open Source Specification): Complete methodology documentation across Inspect, Risk, Quality, Relevance, and Core, released under CC BY-SA 4.0, enabling enterprises to build their own toolchain
  • Platform (Tooling): The set of tools that implement the specification:
    • 🟢 Open source — the standalone Inspector CLI (Apache 2.0): full Inspect coverage including Permission, plus Risk Explicit; integrates into CI/CD and can act as a client of the SanityOps Platform
    • 🔵 Commercial — the full feature suite (Defect Inspector, Risk Scanner, Quality Evaluator), offered as SaaS / self-hosted

Both forms share the same governance language and evaluation standards; enterprises can choose their adoption path based on their circumstances.


3. Who It Compares To, and What It Compares ​

3.1 Framework Level: Positioning Comparison with Protocols / Standards / Methodologies ​

The SanityOps Framework does not replace any of the following categories, but complements them — they address problems at different layers.

#Comparison CategoryWhat It SolvesWhat SanityOps Adds
1OWASP LLM Top 10 / Agentic AI Top 10Risk classification language and awareness elevationProvides mapping from static defects to attack surfaces (AS-01→AS-09), with QD IDs linking to repairable Logic Artifact-level defects
2NIST AI RMFOrganizational-level AI risk management frameworkProvides executable inspection, validation, and measurement specifications for Agent Logic Artifacts
3ISO/IEC 42001AI management system certification standardsProvides technical-level governance assets (evidence chains, version baselines, gate decisions) to support certification audits
4Agent Communication Protocols (MCP / A2A)Inter-Agent communication and interoperabilityDoes not involve the communication layer; A2A governance listed as future expansion direction
5Traditional SAST / DASTSource code and application-layer security testingFocuses on Logic Artifact layer (Prompt / Skill / Tool Schema), which traditional tools do not cover
6Model Evaluation Benchmarks (MMLU / HELM, etc.)General model capability measurementAssesses quality and risk of Logic Artifacts in Agent systems, not the model itself
7Red Teaming MethodologiesAdversarial testing methodsAssociates attack generation with Logic Artifact defects, enabling root-cause localization rather than just Pass/Fail
8Prompt Engineering Best PracticesWriting guides and rules of thumbProvides systematic defect classification (QD), quantitative scoring, and auditable evidence

Core Distinction: Protocols/standards define "what should be done," methodologies describe "how to test," and SanityOps provides an executable governance closed loop for Agent Logic Artifacts — from defect discovery to risk validation, quality assessment, root-cause traceability, and regression closure, with versioned evidence retained at every step.

3.2 Platform Level: Comparison Navigation with Specialized Tools ​

The following three specialized tools have technical value in their respective domains and do not constitute a same-level substitution relationship with SanityOps, but rather can be combined complementarily.

Comparison DimensionPromptfooRAGASNVIDIA SkillSpector
PositioningStandalone Prompt evaluation and Red Team toolStandalone open-source RAG evaluation librarySkill security scanner
Core QuestionWas the attack payload defended (Pass/Fail)?How semantically similar is generated content to reference answers?Security issues in Skill definitions
SanityOps Corresponding SubsetRisk Implicit (dynamic validation + root-cause localization)Quality (governance framework-level quality assessment)Inspect + Risk (defect inspection + cross-artifact consistency)
Governance Closed Loop and Evidence ChainNo endogenous linkage; statistical dimensions at plugin/strategy levelOnly covers "assessment" phase; does not constitute "assess→localize→remediate→re-assess" closed loopSecurity scanning; does not constitute governance closed loop
Root-Cause TraceabilityDoes not answer "which Logic Artifact was wrong"Scoring results not directly linked to underlying Logic Artifact defectsDoes not provide cross-artifact defect chain mapping
Complementary RelationshipCan serve as external attack generation component for Risk ImplicitCan serve as candidate technical component for Quality evaluatorCan complement Inspect with security scanning capabilities

Key Principle: The above tools have technical value within their design scope and can serve as candidate technical components for corresponding SanityOps phases. However, the two do not constitute a same-level substitution relationship — specialized tools answer "did this test pass?" while SanityOps answers "can this Agent be released, where is the problem, how to sustain quality, and is the evidence complete?"


4. Detailed Comparison Entry Points ​

The following three detailed comparison documents respectively depart from three SanityOps subsets to conduct detailed technical comparisons with corresponding specialized tools. Each document focuses on "what SanityOps can do vs. what the specialized tool can do," and explains combinable complementary paths.

DocumentComparison ObjectSanityOps SubsetCore Comparison Dimensions
vs. PromptfooPromptfooRisk ImplicitRoot-cause localization, artifact-driven attack generation, shadow sandbox isolation
vs. RAGASRAGASQualityGovernance framework vs. evaluation library, critical-item independent gating, test case assetization
vs. NVIDIA SkillSpectorSkillSpectorInspect + RiskGovernance closed loop vs. security scanning, cross-artifact consistency

5. Join the Community ​

SanityOps' goal is not to replace your existing security products or testing tools, but to fill the missing closed-loop capabilities for your Agent governance.

If you are seeking answers to the following questions, welcome to join the discussion:

  • Before Agent production deployment, how to systematically discover defects in Prompts / Skills / Tool Schemas?
  • After security testing produces Pass/Fail results, how to localize root causes to specific Logic Artifacts?
  • How do quality assessment results support release admission decisions?

Repository: SanityOps GitHub
Discussion Channels: Comparisons · Q&A · Announcements


SanityOps does not promise absolute security, but promises that every step of governance is documented, evidenced, and traceable.