License & Deployment Mix: 14 tools – 9 OSS (OSI-approved), 1 source-available (Arize Phoenix, Elastic-2.0), 4 commercial platforms. 5 ship a SaaS offering and 13 are self-hostable. (OSS and SaaS overlap: several open tools also ship a hosted edition; the four commercial platforms and Phoenix are the SaaS tail.)

Theme: AI assurance – measuring and constraining what a model says, not building the model.

What Is LLM Evaluation & Guardrails?

This category covers the tooling that sits around an LLM or AI agent to make its output trustworthy: it evaluates quality (offline tests, RAG metrics, LLM-as-judge), red-teams for failure modes (jailbreaks, injection, leakage), grounds and verifies claims against a source of truth (hallucination detection), and guards inputs and outputs at runtime (schema, policy, and safety rails). The buyers are AI/platform engineers shipping LLM features, not data scientists training models.

The unifying problem is that a language model is fluent but not reliable: it will produce a plausible, well-formed answer that is wrong, unsafe, or off-policy. Everything here exists to catch that before it reaches a user or an action – whether as a CI gate, an inline filter, or a deterministic ground-truth check.

Distinct from neighbouring categories

The line matters because all three touch “AI”:

• AI Coding Assistants (AI Coding Assistants) are CLI/IDE clients that call cloud LLM APIs to write code. This category evaluates and guards LLM output; it does not consume an LLM to do developer work.
• AI / ML Platforms (AI / ML Platforms) train, serve, and operate models (MLOps, feature stores, serving). This category starts where a model is already serving and asks whether to trust a given response.
• Code Sandboxes (Code Sandboxes (Untrusted / AI-Gen Code)) isolate the execution of untrusted / AI-generated code. Here we assess the text an agent produces, not the blast radius of code it runs. The two compose: sandbox the action, guard the reasoning.

Capabilities typically present

• Offline / CI evaluation – assertion suites, LLM-as-judge, and metric scoring run as tests against a prompt/agent/RAG matrix
• RAG-specific metrics – faithfulness, answer/context relevancy, context precision and recall
• Red-team probing – adversarial scans for jailbreaks, prompt injection, data leakage, toxicity, and hallucination
• Grounding / hallucination verification – scoring or proving a claim against retrieved context or an authoritative source of truth
• Runtime guardrails – inline input/output validation: schema, PII, topic/policy rails, and safety classification
• Observability – tracing, span capture, and online eval of live traffic (OpenTelemetry-native in the open tools)
• Agent / MCP integration – verifiers and rails exposed to reasoning agents (e.g. via MCP) rather than only batch scoring

The information on these pages was researched by a combination of human review and large language models. To suggest an addition or correction, please contact us. Prepared by Rhodium Systems Inc., author of the ResorsIT platform — a unified IT operations management platform for IT teams and MSPs that integrates a curated suite of open-source, commercial, and SaaS applications into a single system with shared identity, single sign-on, access control, and a common audit trail. Use this catalogue only as a starting point for your own research, and review any tool carefully against your own requirements before relying on it. Catalogue data version 2026.197.

Comparison

Cross-tool comparison for the LLM Evaluation & Guardrails category: the tooling that evaluates, red-teams, grounds, and guards the output of LLMs and AI agents.

All facts below are point-in-time and derived from each tool’s per-tool evaluation file. Commercial vendors often do not publish SSO, deployment, or protocol specifics, so several cells are marked unconfirmed rather than asserted. Capability ratings are a coarse editorial read, not a benchmark.

Licensing & Cost Model

DeepEval’s Confident AI, Vectara’s hosted HHEM API, and the Arize AX SaaS are commercial companions to the open tools above and are scoped out of those files (noted in prose, not the frontmatter). Patronus’s Lynx models + HaluBench are open; the platform is not.

Capability Matrix

Legend: ● primary ◐ partial / secondary ○ minimal or none.

Columns: Eval offline/CI scoring; RAG retrieval-specific metrics; RedT adversarial red-team probing; Grnd grounding / hallucination verification; Guard runtime input/output rails; Obs tracing / online observability; Agent agent- or MCP-native integration.

The commercial platforms score broadly because they bundle several segments; the open tools are sharper single-purpose components you compose. Reverify and Vectara HHEM are the two purest grounding tools – Reverify deterministic against binary ground truth, HHEM a learned CPU groundedness scorer.

SSO & Deployment

* unconfirmed: the vendor advertises enterprise SSO but does not publish the specific protocol / SCIM support. The nine OSS libraries and CLIs have no auth surface of their own – they run in-process or as a local CLI; where the open guardrail projects ship an optional server (Guardrails, NeMo), it is fronted by the proxy / Authentik, not by built-in SSO. Only Arize Phoenix has native OIDC, which is why it is the sole tool here that fits the Authentik-fronted internal-infra pattern.

Choosing

• A CI eval gate over prompts/agents – start with Promptfoo (config/CLI) or DeepEval (pytest); add Ragas for RAG-specific metrics.
• Runtime protection around an agent action – Guardrails AI for schema/validator I/O, NeMo Guardrails for programmable dialog/topic rails.
• A ground-truth check before trusting a claim – Reverify for binary/RE facts (deterministic, MCP), Vectara HHEM for cheap groundedness scoring of RAG answers.
• Pre-deployment red-team – Garak.
• Self-hosted observability + eval – Arize Phoenix (mind the Elastic-2.0 managed-service restriction).
• A single managed platform – Galileo, Braintrust, Patronus, or Fiddler, traded against cost and the OSS building blocks above.

In short: compose the open building blocks – guardrails + grounding verifiers at runtime, the eval frameworks as CI gates, and Phoenix for self-hosted observability – or adopt a single managed platform, traded against cost.

Tools

14 tools.

Arize Phoenix

OSS AI-observability platform by Arize: OTel-based LLM/agent tracing, an eval framework with LLM-as-judge templates, datasets and experiments, served from a self-hostable web UI with OIDC auth.

License: Elastic-2.0 (source-available) · Kind: web · Deploy: docker, package, saas · SSO: OIDC · Tags: llm, observability, tracing, evaluation, llm-as-judge, opentelemetry, rag, self-hosted, python

Website · Source

Braintrust

Commercial LLM evaluation and observability platform for building, scoring, and monitoring AI products, with open-source client SDKs and a proprietary hosted back end.

License: Proprietary (proprietary) · Kind: web · Deploy: saas, docker, k8s · SSO: OIDC, SAML · Tags: llm, evaluation, observability, experiments, saas

Website

DeepEval

Pytest-style Python framework for evaluating LLM apps: 40+ metrics (G-Eval, hallucination, RAG faithfulness/relevancy, safety) run as unit tests in CI, plus red-teaming and dataset tooling.

License: Apache-2.0 (OSS) · Kind: library · Deploy: package, native · SSO: none · Tags: llm, evaluation, rag, testing, ci, pytest, python

Website · Source

Fiddler AI

Enterprise AI observability, evaluation, and security platform for ML and LLM applications, with in-VPC/on-prem deployment and low-latency runtime guardrails.

License: Proprietary (proprietary) · Kind: web · Deploy: saas, docker, k8s · SSO: OIDC, SAML · Tags: llm, observability, guardrails, ml-monitoring, security, saas

Website

Galileo

Commercial LLM/agent evaluation and observability platform that turns offline evals into runtime guardrails, powered by low-latency Luna evaluator models.

License: Proprietary (proprietary) · Kind: web · Deploy: saas, docker, k8s · SSO: OIDC · Tags: llm, evaluation, observability, guardrails, agents, saas

Website

Garak

NVIDIA’s LLM vulnerability scanner: a red-team CLI that probes a model for jailbreaks, prompt injection, data leakage, toxicity, and hallucination, then scores each weakness with paired detectors.

License: Apache-2.0 (OSS) · Kind: cli · Deploy: package, native · SSO: none · Tags: llm, red-team, security, safety, vulnerability-scanner, python, cli

Website · Source

Guardrails AI

Apache-2.0 Python framework that wraps LLM calls in input/output Guards – pluggable validators from the Guardrails Hub that detect, quantify, and mitigate risks and enforce structured output.

License: Apache-2.0 (OSS) · Kind: library · Deploy: package, native, docker · SSO: none · Tags: llm, guardrails, safety, validation, structured-output, python, library

Website · Source

NeMo Guardrails

NVIDIA’s Apache-2.0 toolkit for programmable LLM guardrails: input, output, dialog, retrieval, and execution rails defined in the Colang modeling language, wrapping a conversational model at runtime.

License: Apache-2.0 (OSS) · Kind: library · Deploy: package, native, docker · SSO: none · Tags: llm, guardrails, safety, colang, dialog, rag, python, library

Website · Source

Patronus AI

Commercial LLM evaluation and safety platform for scoring hallucination and other failures at scale; also publishes the open-source Lynx hallucination detection models and HaluBench.

License: Proprietary (proprietary) · Kind: web · Deploy: saas · SSO: OIDC · Tags: llm, evaluation, hallucination, safety, guardrails, saas

Website

Promptfoo

Developer-first LLM eval and red-teaming toolkit: declarative test cases run prompts, agents, and RAG pipelines against many providers, with assertions, scoring, and a local results viewer.

License: MIT (OSS) · Kind: cli · Deploy: package, native · SSO: none · Tags: llm, evaluation, red-teaming, rag, testing, ci, typescript, cli

Website · Source

Ragas

Python library for evaluating RAG and LLM pipelines: reference-free metrics for faithfulness, answer relevancy, and context precision/recall, plus synthetic test-set generation.

License: Apache-2.0 (OSS) · Kind: library · Deploy: package, native · SSO: none · Tags: llm, evaluation, rag, groundedness, testing, python

Website · Source

Reverify

Deterministic grounding and verification layer for AI reverse-engineering: a model proposes claims about a binary, local tools check them against the actual bytes and return VERIFIED / REFUTED / INCONCLUSIVE with evidence.

License: MIT (OSS) · Kind: cli · Deploy: package, native · SSO: none · Tags: llm, anti-hallucination, grounding, verification, reverse-engineering, binary-analysis, mcp, python, cli

Website · Source

TruLens

MIT Python library for evaluating and tracking LLM apps and agents via feedback functions – programmatic scorers (the RAG Triad and more) that measure groundedness, relevance and quality of instrumented runs.

License: MIT (OSS) · Kind: library · Deploy: package, native · SSO: none · Tags: llm, evaluation, feedback-functions, rag-triad, observability, opentelemetry, python, library

Website · Source

Vectara HHEM

Apache-2.0 open hallucination-evaluation model (HHEM-2.1-Open): a small FLAN-T5 cross-encoder that scores 0-1 whether a hypothesis is grounded in given evidence, runnable on CPU for fast RAG groundedness checks.

License: Apache-2.0 (OSS) · Kind: library · Deploy: package · SSO: none · Tags: llm, hallucination-detection, groundedness, rag, nli, cross-encoder, transformers, model, python

Website · Source