Implementing LLM Guardrails: A Practical Guide to NVIDIA NeMo and Llama Guard

Enterprises are shipping LLM and agent-based applications faster than they’re securing them, and the 2026 data on this is consistent across analysts. Gartner projects that 40% of enterprise applications will include task-specific AI agents by the end of 2026, up from under 5% at the start of the year. Separately, more than 40% of these agent projects are expected to fail by 2027, largely due to runaway costs, unclear business value, and agents that behave in ways that violate policy or create risk.

The World Economic Forum’s Global Cybersecurity Outlook 2026, produced with Accenture, found that 94% of surveyed leaders identify AI as the most significant driver of cybersecurity change in 2026, and 87% flagged AI-related vulnerabilities as the fastest-growing cyber risk in 2025. Deloitte’s 2026 State of AI in the Enterprise survey of 3,235 IT and business leaders found the same governance gap from the inside: only 21% of organizations report having a mature governance model in place for agentic AI, even as agentic AI development usage scales quickly across the same organizations.

That gap between how fast LLM and agent systems are being deployed and how well they’re being governed is exactly what runtime guardrail frameworks like NeMo Guardrails and content-safety classifiers like Llama Guard are built to close.

What are LLM guardrails?

LLM guardrails are programmable checks that sit between a user, an LLM application, and any tools or data sources it can access, inspecting, blocking, or modifying inputs and outputs against defined safety and business policies before anything reaches production. Guardrails don’t replace a model’s own safety training; they enforce policy at request time, regardless of what the underlying model was aligned to do.

Guardrail frameworks typically operate across several distinct stages:

• Input rails—screen user messages before they reach the LLM (jailbreak detection, prompt injection filtering, PII masking).
• Dialog rails—keep multi-turn conversations on-topic and redirect the model away from restricted subjects.
• Retrieval rails—validate and filter retrieved context in a RAG pipeline before it’s injected into the prompt.
• Execution rails—gate which tools, APIs, or actions an LLM-based agent is allowed to invoke.
• Output rails—vet the model’s generated response for toxicity, hallucination, policy violations, or leaked sensitive data before it reaches the user.

This maps closely onto the risk categories in the OWASP Top 10 for Large Language Model Applications, the community-driven reference for the most critical LLM-specific security risks, which places prompt injection and improper output handling among its top concerns. Two of the most widely used tools for implementing these rails in practice are NVIDIA’s NeMo Guardrails and Meta’s Llama Guard, and they solve different parts of the problem.

NVIDIA NeMo Guardrails: The Orchestration Layer

NeMo Guardrails acts as an intermediary between application code and LLM requests and responses. Once integrated, every LLM inference request is first checked by Guardrails to ensure it’s safe and not malicious. It’s open source, Apache 2.0 licensed, and maintained by NVIDIA.

What sets it apart from a simple input/output filter is Colang, a domain-specific language purpose-built for defining conversational flows. Dialog rails model the intended flow of a conversation using Colang, keeping the LLM on track across multiple turns if a user tries to steer the conversation off-topic or into restricted territory; dialog rails redirect it back. NeMo Guardrails supports five rail types in total: input, dialog, retrieval, execution, and output, and integrates with frameworks like LangChain, LangGraph, and LlamaIndex, supports multi-agent deployments, and can call NVIDIA’s own Nemotron safety models or third-party classifiers as part of its rails.

Two things worth knowing before you start:

• NeMo Guardrails is async-first—its core mechanics use Python’s async model, with both sync and async versions of its public methods (e.g., generate and generate_async).
• Configuration lives in two places: a config.yml file that declares your models and which Rails are active, and one or more Colang .co files that define the actual dialog logic.

Meta Llama Guard: The Content-Safety Classifier

Where NeMo Guardrails orchestrates, Llama Guard classifies. Llama Guard 4 is a natively multimodal safety classifier with 12 billion parameters, trained jointly on text and images, and can classify both LLM inputs (prompt classification) and LLM outputs (response classification). It functions as an LLM itself, generating text that indicates whether a given prompt or response is safe or unsafe. If unsafe, it lists the content categories that were violated, based on the standardized MLCommons hazard taxonomy.

A few practical details matter for implementation:

• Llama Guard 4 can run on a single GPU and is compatible with the Llama 3 model line, working as a drop-in replacement for the earlier Llama Guard 3 8B and 11B-vision models for both text-only and multimodal use cases.
• Meta also ships a separate, much smaller model for a narrower job: Llama Prompt Guard 2 comes in 86M and 22M parameter versions, focused specifically on detecting prompt injections and jailbreaks with a simplified benign-vs-malicious classification, which makes it a lighter option when you only need injection detection rather than full content-safety classification.
• Llama Guard isn’t infallible against determined adversaries. One evaluation using adversarial, disguised jailbreak prompts found that over four in ten obfuscated harmful prompts successfully evaded detection, which is a good reminder that a classifier is one layer in a defense-in-depth setup, not a complete solution on its own.

NeMo Guardrails vs. Llama Guard: How They Actually Compare

The “vs.” in most searches for this topic is a bit misleading—in production, these two are usually deployed together rather than as alternatives.

Dimension	NVIDIA NeMo Guardrails	Meta Llama Guard
What it is	Orchestration/middleware toolkit	Content-safety classifier model
Core mechanism	CoLang-defined rails and dialog flows	Fine-tuned LLM outputting safe/unsafe + category
Scope	Input, dialog, retrieval, execution, and output rails	Input and/or output content classification only
Conversation/topic control	Yes — dialog rails keep multi-turn conversations on track	No — classifies single prompts/responses, no dialog state
Tool/action gating	Yes — execution rails control what agents can invoke	No
RAG-aware filtering	Yes — retrieval rails validate retrieved chunks	No
Typical role in a stack	The policy engine that decides what to check and when	One of the classifiers NeMo Guardrails calls
Deployment	Python library or containerized microservice	Hosted or self-hosted behind an OpenAI-compatible endpoint (e.g., vLLM)

LlamaGuard 3 is a fine-tuned classifier model that outputs a safe/unsafe label with a hazard category, while NeMo Guardrails is an orchestration layer that can call LlamaGuard 3 or any other classifier as one of its rail components; they’re complementary, with LlamaGuard handling binary classification while NeMo Guardrails handles the broader policy enforcement workflow: routing, PII redaction, topic boundaries, and multi-turn dialog state.

Implementing NeMo Guardrails with Llama Guard: A Step-By-Step Example

The pattern below wires Llama Guard 4 into NeMo Guardrails as the content-safety model for both input and output rails. Treat this as an illustrative starting configuration. NeMo Guardrails move fast, so confirm exact argument names and the current Colang version against the official NeMo Guardrails documentation before pinning anything in production.

1. Install the toolkit and serve Llama Guard.

pip install nemoguardrails

# Serve Llama Guard 4 behind an OpenAI-compatible endpoint, e.g. with vLLM:

# vllm serve meta-llama/Llama-Guard-4-12B –port 8000

2. Declare your models in config.yml

The models section defines the LLM providers and models used by NeMo Guardrails — a required type: main entry for your application LLM, plus additional typed entries (such as type: content_safety) for models used only by specific rails:

# config/config.yml

models:

  – type: main

    engine: openai

    model: gpt-4o

  – type: content_safety

    engine: openai

    parameters:

      base_url: “http://localhost:8000/v1”   # your vLLM endpoint serving Llama Guard

      model_name: “meta-llama/Llama-Guard-4-12B”

rails:

  input:

    flows:

      – content safety check input $model=content_safety

  output:

    flows:

      – content safety check output $model=content_safety

3. Define the safety-check prompts in prompts.yml

NeMo Guardrails provides an off-the-shelf is_content_safe output parser for content-safety checks: if the classifier’s response includes “safe,” the content is considered safe, and if it includes “unsafe,” the content is blocked.

# config/prompts.yml

prompts:

  – task: content_safety_check_input $model=content_safety

    content: |

      Check if this content is safe according to our safety policy:

      {{ user_input }}

    output_parser: is_content_safe

    max_tokens: 50

  – task: content_safety_check_output $model=content_safety

    content: |

      Check if this response is safe according to our safety policy:

      {{ bot_response }}

    output_parser: is_content_safe

    max_tokens: 50

4. Add a dialog rail for topic boundaries

This is where Colang’s canonical-form pattern comes in — a user message block defines a canonical form along with example utterances that fit it, which NeMo Guardrails encodes into a semantic vector space to match new user input against the closest known intent:

# config/rails.co

define user ask about competitors

  “What do you think of [CompetitorX]?”

  “Is [CompetitorX] better than your product?”

define flow ask about competitors

  user ask about competitors

  bot refuse to discuss competitors

define bot refuse to discuss competitors

  “I’m not able to comment on other products, but I’m happy to help with anything about our platform.”

5. Wire it up and test

from nemoguardrails import LLMRails, RailsConfig

config = RailsConfig.from_path(“./config”)

rails = LLMRails(config)

response = rails.generate(messages=[

    {“role”: “user”, “content”: “How do I reset my account password?”}

])

print(response)

Run a mix of benign requests, known jailbreak phrasings, and off-topic questions through this pipeline before deploying. One published comparison testing input moderation against a set of prompt-injection-style questions found that NeMo Guardrails’ dialog rails blocked a higher share of unwanted requests than Llama Guard’s classification alone. This is less an argument for one tool over the other than a demonstration of why dialog-level control and classifier-level content safety catch different things.

Extending guardrails to RAG and agents

Guardrails matter even more once an LLM is connected to a knowledge base or given the ability to act.

For retrieval-augmented generation, retrieval rails filter and validate knowledge base results before they reach the LLM, reducing the risk that an attacker-controlled or poisoned document gets treated as trusted context. A distinct concern from the standard prompt-injection case. Anyone building secure retrieval-augmented generation should treat retrieval rails as a required layer, not an optional one, since a RAG pipeline is only as trustworthy as the content it retrieves. SoluLab’s RAG development services build this filtering into the retrieval architecture itself, rather than bolting it on afterward.

For agentic systems, execution rails validate tool inputs and outputs before and after invocation and integrate with frameworks like LangGraph for multi-agent safety—the core mechanism behind any credible AI agent security framework. This matters because an ungoverned agent isn’t just a content-safety problem; it’s an agent that can take real actions. SoluLab’s work on agentic AI in cybersecurity and custom AI agent development is built around exactly this principle: agents need execution-level guardrails, human oversight checkpoints, and audit trails before they’re given real autonomy, not after.

Common Implementation Mistakes

• Treating a classifier as a complete solution. Llama Guard alone doesn’t manage dialog state, retrieval, or tool access—pairing it with an orchestration layer like NeMo Guardrails closes that gap.
• Skipping output rails because input rails passed. A benign-looking prompt can still produce an unsafe, hallucinated, or policy-violating response; check both directions.
• Under-testing against adversarial phrasing. Straightforward test prompts miss the disguised, multi-turn, or encoded attacks that actually show up in production—obfuscated prompts specifically designed to evade detection succeed at meaningfully higher rates than direct, naive attempts.
• Ignoring latency budgets. Running a full classifier synchronously on every request adds real latency; co-locating the classifier with your main model and batching classifier calls keeps overhead manageable.
• Assuming model-level alignment is enough. RLHF and safety fine-tuning reduce harmful output at training time, but they aren’t a runtime policy enforcement layer—guardrails apply regardless of what the underlying model was trained to refuse.

Conclusion

Implementing large language model guardrails isn’t a single tool decision; it’s an architecture decision. NeMo Guardrails gives you the orchestration layer: dialog control, retrieval filtering, execution gating, and a place to plug in whichever classifier fits your policy. Llama Guard (or a comparable classifier) gives you the actual safety judgment call on a given prompt or response. Used together, they cover far more of the OWASP LLM risk surface than either does alone, and given how far agentic AI adoption has outpaced governance maturity across the enterprises surveyed by Gartner, WEF, and Deloitte this year, that combination is quickly becoming a baseline expectation rather than a nice-to-have.

This is also squarely where a specialized development partner adds value beyond wiring together two open-source tools. SoluLab is an LLM development company that builds guardrail architecture directly into custom LLM and agent deployments from the start, including enterprise AI security and governance frameworks that define accountability, risk controls, and audit processes across the AI lifecycle, not just at the model layer. That includes RAG pipelines built with retrieval-level filtering, agents built with execution-level gating, and the ongoing testing needed to catch the adversarial phrasing that simple test suites miss.

FAQs