Building LLM Evaluation Pipelines: Metrics, Frameworks, and CI Integration

Shipping an LLM-powered feature without an evaluation pipeline is like deploying a microservice without health checks — it might work in testing, but you’ll have no way to catch regressions before your users do. As LLM applications move from demos to production, evaluation infrastructure has become the critical layer that separates maintainable systems from unpredictable ones.

The challenge is that LLM outputs are non-deterministic. Traditional software tests assert exact values; LLM tests need to assess qualities like relevance, factual accuracy, tone, and safety. This requires a different evaluation architecture: one that combines curated datasets, automated scoring metrics, and selective human review.

This post walks through building a practical evaluation pipeline — the metrics that matter, the open-source tools worth adopting, and the CI/CD integration patterns that catch regressions before they ship.

The Three Layers of LLM Evaluation

A mature evaluation pipeline has three distinct layers, each serving a different purpose in the development lifecycle.

Offline evaluation runs against a curated golden dataset before deployment. You collect representative input-output pairs, define expected behaviors, and score every model or prompt change against this baseline. This is where you catch regressions early.

Online evaluation samples live production traffic and scores it in near real-time. Not every request needs scoring — you sample a percentage, run automated metrics, and surface anomalies. This catches drift: when user behavior shifts and your model starts producing subtly different outputs.

Human evaluation handles the cases automated metrics can’t reliably judge. This includes edge cases, subjective quality assessments, and business-specific requirements. The key is making human review targeted — not evaluating everything, but triaging the outputs that automated metrics flag as uncertain.

Choosing the Right Metrics

Not all metrics are useful for all tasks. Here’s a practical breakdown by use case:

METRIC_CATEGORIES = {
    "factual_qa": [
        "exact_match",      # String comparison against gold answer
        "factual_consistency", # Does output contradict the source?
        "answer_relevancy",    # Is the answer relevant to the question?
    ],
    "summarization": [
        "rouge_score",         # N-gram overlap with reference
        "factual_consistency", # No hallucinated facts
        "conciseness",         # Output length relative to input
    ],
    "rag_pipeline": [
        "context_precision",   # Are retrieved chunks relevant?
        "context_recall",      # Did retrieval find all needed info?
        "faithfulness",        # Is the answer grounded in context?
        "answer_relevancy",    # Does the answer address the query?
    ],
    "code_generation": [
        "functional_correctness", # Does the code pass unit tests?
        "pass_at_k",              # Success within k attempts
    ],
}

Traditional NLP metrics like BLEU and ROUGE measure surface-level similarity — n-gram overlap between generated and reference text. They’re useful for summarization and translation tasks, but they fall short for open-ended generation, where multiple valid answers exist and surface similarity doesn’t capture semantic equivalence.

For most production LLM applications, LLM-as-a-judge metrics are more practical. You use a strong model (typically a frontier model like GPT-4-class) to evaluate outputs on specific criteria. The judge model receives the input, the output, and a scoring rubric, then returns a structured assessment. This approach scales better than human review and correlates reasonably with human judgment for well-defined criteria.

Building a Pipeline with DeepEval

DeepEval is an open-source Python framework that treats LLM evaluation like unit testing. It integrates with pytest, supports custom metrics, and includes a hosted dashboard (Confident AI) for tracking results across runs. Here’s how to structure a basic evaluation suite:

from deepeval import evaluate
from deepeval.metrics import (
    AnswerRelevancyMetric,
    FaithfulnessMetric,
    GEval,
)
from deepeval.test_case import LLMTestCase, LLMTestCaseParams
from deepeval.dataset import EvaluationDataset

# Define metrics with a judge model
answer_relevancy = AnswerRelevancyMetric(threshold=0.7)
faithfulness = FaithfulnessMetric(threshold=0.8)

# Custom metric using LLM-as-a-judge with a rubric
tone_metric = GEval(
    name="professional_tone",
    criteria="Evaluate whether the response maintains "
             "a professional, helpful tone",
    evaluation_params=[
        LLMTestCaseParams.INPUT,
        LLMTestCaseParams.ACTUAL_OUTPUT,
    ],
    threshold=0.75,
)

# Build test cases from your golden dataset
test_cases = [
    LLMTestCase(
        input="What is the return policy?",
        actual_output=generated_answer,
        retrieval_context=retrieved_docs,
        expected_output="Items can be returned within 30 days.",
    ),
    # ... more test cases
]

dataset = EvaluationDataset(test_cases=test_cases)

# Run evaluation
evaluate(
    dataset,
    metrics=[answer_relevancy, faithfulness, tone_metric],
)

DeepEval outputs a detailed report showing each metric score, whether it passed the threshold, and the reasoning from the judge model. The GEval metric is particularly useful — it lets you define custom evaluation criteria in natural language, and the framework handles prompt construction and score normalization.

Evaluating RAG Pipelines with Ragas

For retrieval-augmented generation systems, Ragas provides metrics specifically designed for the two components that can fail independently: retrieval and generation. A RAG system can retrieve the wrong context but generate a fluent answer, or retrieve the right context but hallucinate anyway. Ragas separates these concerns:

from ragas import evaluate as ragas_evaluate
from ragas.metrics import (
    context_precision,
    context_recall,
    faithfulness,
    answer_relevancy,
)
from datasets import Dataset

# Build evaluation dataset from RAG traces
eval_data = Dataset.from_dict({
    "question": questions,
    "answer": generated_answers,
    "contexts": retrieved_chunks,  # List of lists
    "ground_truth": reference_answers,
})

results = ragas_evaluate(
    eval_data,
    metrics=[
        context_precision,
        context_recall,
        faithfulness,
        answer_relevancy,
    ],
)

print(results.to_pandas())

The context_precision metric measures whether retrieved chunks are relevant to the query — low scores indicate your retrieval step is surfacing noise. faithfulness checks whether the generated answer is fully supported by the retrieved context — low scores indicate hallucination. Together, these metrics pinpoint which component of your RAG pipeline needs attention.

LLM-as-a-Judge: Pitfalls and Mitigations

Using an LLM to evaluate another LLM’s output is powerful but has known failure modes. Judge models exhibit position bias — they tend to prefer the first option in a comparison. They show verbosity bias — longer answers tend to score higher regardless of quality. And they can be self-preferencing — a model evaluating its own outputs tends to rate them higher.

Mitigate these by randomizing answer order in pairwise comparisons, normalizing for length in your scoring rubric, and using a different model family for judging than for generation. Also calibrate your judge against a small set of human-graded examples to establish a baseline correlation.

Integrating Evaluation into CI/CD

Promptfoo takes a config-first approach to LLM evaluation that fits naturally into CI/CD pipelines. You define test cases and assertions in YAML, and the tool runs them across multiple models or prompt variations:

# promptfooconfig.yaml
description: "Customer support bot evaluation"

prompts:
  - file://prompt.txt

providers:
  - openai:gpt-4o-mini
  - anthropic:claude-3-5-haiku

tests:
  - description: "Return policy question"
    vars:
      question: "What is the return policy?"
    assert:
      - type: contains-any
        value: ["30 days", "thirty days"]
      - type: llm-rubric
        value: "Answer is helpful and professional"
      - type: latency
        threshold: 3000  # milliseconds

  - description: "Out-of-scope question"
    vars:
      question: "What's the weather today?"
    assert:
      - type: llm-rubric
        value: "Politely deflects unrelated question"

Run promptfoo eval in your CI pipeline, and it produces a report comparing each provider’s performance across all test cases. Set failure thresholds on specific metrics, and the pipeline blocks merges that regress quality. This turns LLM evaluation from a manual pre-release ritual into an automated quality gate.

Building a Golden Dataset

Every evaluation pipeline depends on the quality of its test data. A golden dataset is a curated set of input-output pairs that represent the queries your system handles in production. Start small — 50 to 100 examples covering your main use cases and edge cases — and grow it organically.

The most effective approach is a data flywheel: log production traces, sample the ones where automated metrics show low confidence or where users give negative feedback, manually review and label them, and add them to your golden dataset. Each production incident becomes a permanent test case, preventing the same regression from recurring.

Wrapping Up

An evaluation pipeline is the infrastructure that makes LLM applications maintainable. Without it, every prompt change or model upgrade is a blind deployment. With it, you have quantitative evidence that your changes improve outcomes — or early warning that they don’t.

Start with offline evaluation against a golden dataset, layer in LLM-as-a-judge metrics for the criteria that matter to your use case, and integrate evaluation runs into your CI pipeline as a quality gate. DeepEval, Ragas, and Promptfoo each address different parts of this workflow — DeepEval for Python-native test suites, Ragas for RAG-specific metrics, and Promptfoo for config-driven CI integration. The upfront investment pays off the first time your pipeline catches a regression before it reaches production.