RAGAS and DeepEval both score RAG systems: faithfulness, context precision, contextual relevancy. The metrics overlap 80%. What separates them is where and how you use them.

RAGAS was born from a research paper. DeepEval was born from a CI/CD need. It shows in both APIs, and it determines which one you should install first.

On my AI Evaluation Lab, a teaching project built up progressively (heuristics → LLM Judge → RAG → RAGAS → DeepEval, 14 steps), I deliberately started by coding the metrics by hand before installing the frameworks. The reason: when you use a tool without understanding what it computes, you read scores without knowing what they measure. This section covers what came out of that.

A key point: how RAGAS and DeepEval actually compute a score

Before getting into the details of each metric, one thing to keep in mind at all times: neither RAGAS nor DeepEval compute their scores with fixed rules or pattern matching. Both rely on LLM-as-a-judge: you give an LLM (GPT-4o, Claude, Llama depending on the config) a prompt that precisely describes an evaluation criterion, and ask it to score a response against that criterion — exactly like asking a human reviewer to fill out a scoring grid, except the reviewer is a language model.

Concretely, for faithfulness, the judge receives a prompt that roughly says: « Here is a context, here is a response. Break the response down into individual claims. For each claim, say whether it’s supported by the context. Answer in JSON format. » The LLM executes that instruction, returns its list of validated or rejected claims, and the framework computes the final score (proportion of supported claims). For answer relevancy, the judge reconstructs likely questions from the response, then an embedding model measures their similarity to the original question — a mix of LLM judge and vector similarity depending on the metric.

Two direct consequences worth keeping in mind:

The judge can be wrong. An LLM evaluating another LLM inherits the same limitations as an LLM answering directly: it can misread a negation, be too lenient on an ambiguous phrasing, or simply hallucinate its own verdict.

The score depends on the prompt and the judge model chosen, not just on the response being evaluated. Swapping the model used as judge (GPT-4o vs. Claude vs. a smaller open-source model) can shift the score on the exact same data, because each model has its own sensitivity to the evaluation prompt. This mechanic is what explains the relativity of scores discussed further down.

RAGAS and DeepEval automate and industrialize this principle — optimized evaluation prompts, output JSON parsing, score aggregation — but the underlying mechanism stays the same in both cases: one model judging another.

The hidden cost: a judge burns tokens

The point that’s easy to forget: every score isn’t free. A metric like faithfulness doesn’t make a single LLM call — it makes several per evaluated case: one call to break the response down into individual claims, then one call (sometimes one per claim) to check each claim against the context. Multiply that by several metrics (faithfulness, answer relevancy, context precision, context recall) and by the size of the dataset, and a « complete » evaluation can add up to dozens of LLM calls for a single tested response. On a dataset of a few hundred questions with 3-4 metrics, that quickly climbs into the thousands of calls — and therefore a real token cost, not counting runtime.

That cost isn’t symmetric between RAGAS and DeepEval depending on how you use them: running RAGAS once on a full dataset to compare two versions of a pipeline is a one-off, accepted cost. Running DeepEval as a gate on every pull request, if the test suite isn’t kept in check, can quickly blow up the API bill on an active repo.

A few concrete levers to avoid paying more than necessary:

Sample instead of evaluating everything. In CI, a representative, stable subset (a few dozen critical cases) is enough to catch a regression. Reserve exhaustive evaluation of the full dataset for offline runs with RAGAS, not every commit.

Use a cheaper judge model for the gate, a more capable one for the audit. A lightweight model (Llama 8B-class or GPT-4o-mini) as judge in CI for a fast, cheap first filter; a stronger model reserved for deeper, occasional analyses where judgment accuracy matters more.

Trim the number of metrics to the strict minimum. Faithfulness and answer relevancy already cover most of the hallucination and off-topic risk. Systematically adding context precision and recall to every run doubles the cost for marginal gain if the retriever hasn’t changed.

Don’t rerun generation on every evaluation. Separate the « generate responses » phase from the « evaluate them » phase to avoid paying for the application LLM twice when only the evaluation logic changed.

Cache results for unchanged cases. If a test case hasn’t moved since the last run and neither has the pipeline being evaluated, there’s no reason to send it back through the judge.

None of these levers are specific to RAGAS or DeepEval — both call an LLM judge the same way, so both incur the same cost, and the same strategies apply to both.

What these metrics actually measure

The starting point to keep in mind: RAGAS doesn’t compute a single global score. It splits a RAG pipeline into two distinct stages and checks each one independently — because a pipeline can fail in two different places, for two different reasons.

Stage 1 — retrieval: did we pull the right documents?

Stage 2 — generation: did the LLM actually use those documents well?

A retriever that misses documents and an LLM that hallucinates are two different bugs, in two different parts of the code. Conflating the two amounts to saying « the final answer is bad » without knowing whether to fix the chunking, the embeddings, or the generation prompt. That’s exactly why RAGAS splits its metrics into two families instead of producing a single score.

Stage 1: retrieval metrics

Context Precision — Of the documents pulled by the retriever, what proportion is actually useful for answering? A counter-intuitive example: for the query « What is the price? », a chunk describing a beautiful duplex with a terrace can score a higher cosine similarity (0.78) than a chunk that literally contains the price (0.61) — because the embedding captures the semantic domain (« real estate »), not the intent of the question. Retrieval then prioritizes the chunk that doesn’t answer the question. Context precision is precisely the metric that catches this: not « is this chunk semantically close, » but « does this chunk let you actually answer. »

Context Recall — Was everything needed to answer actually retrieved? It’s checked against a reference answer (ground truth): every element of that reference must have a trace in the retrieved context. Example: the reference answer for « What are the charges on this apartment? » is « €85 in monthly co-ownership fees, plus €450 in annual property tax. » If the retriever only surfaces the chunk mentioning the €85 in fees, and never brings back the document with the property tax, recall drops — even if the one chunk retrieved was perfectly relevant and clean. A low recall means the retriever missed relevant information, not that what it did retrieve was bad.

Neither of these two metrics ever looks at the LLM’s response. They only evaluate what comes out of the retriever, before generation even starts.

Stage 2: generation metrics

Take a concrete case from the dataset: the model receives this context about a studio apartment in Lyon —

Studio in Lyon 69003, 28 sqm, listed at €120,000. Monthly co-ownership fees: €85. Close to public transport.

Question asked: « What is the annual property tax? » The property tax is mentioned nowhere. A good answer says it’s not available. A hallucination looks like: « The property tax is approximately €800 per year. » — invented, plausible, false. And this is exactly the kind of error that doesn’t look like an error. It looks like an answer.

Faithfulness — Does the response rely solely on the retrieved context, without inventing anything? An LLM judge breaks the response down into individual claims, then checks each one against the provided context. Back to the Lyon studio: if the generated response is « This 28 sqm studio is sold for €120,000, with a parking space included, » the judge splits it into three claims — « 28 sqm, » « €120,000, » « parking space included » — and checks each one separately against the context. The first two are confirmed. The third isn’t: no parking space is mentioned anywhere. The faithfulness score reflects that proportion — 2 out of 3 claims confirmed, not a global « true » or « false » verdict. RAGAS and DeepEval both rely on this same sentence-by-sentence decomposition principle, even if the metric names differ slightly between tools.

Answer Relevancy — Does the response actually answer the question asked, independent of whether it’s true? The principle: the LLM is asked to generate several likely questions from the produced answer, then their similarity to the original question is measured. Example: to the question « What is the price of this studio? », the model answers « This 28 sqm studio is close to public transport and has moderate co-ownership fees. » Every claim in that response is true and grounded — nothing is invented — but none of it mentions the price. The questions reconstructed from that response (« What’s the square footage? », « Is it well served by transport? ») drift away from the question actually asked, and the answer relevancy score drops even though faithfulness stays high. This is exactly the kind of faithful-but-off-topic answer that this metric catches, and that faithfulness alone would let through.

Neither of these two metrics ever looks at what the retriever pulled upstream. They only evaluate the relationship between the provided context and the produced response.

Why this split matters

A concrete example: low recall + high faithfulness means the retriever missed an important document, but the LLM didn’t invent anything to compensate — it likely answered « information not available, » which is correct but incomplete. Conversely, high recall + low faithfulness means the retriever did its job correctly, but the LLM hallucinated despite having sufficient context. The fix isn’t the same in both cases: rework the chunking and embeddings in the first, rework the generation prompt in the second. A single global score would never let you tell these two situations apart.

Grounded doesn’t mean true — and it isn’t the same as complete

Two distinctions that trip up almost everyone at first, and that change how you read RAGAS/DeepEval scores.

Correctness ≠ Grounding. A chunk mentions an address and a south-facing terrace, without giving the square footage. The model still answers « 78 sqm » — and turns out to be right, if you check the actual listing. That doesn’t make it grounded: nothing in the provided context lets you trace that information. It’s a correct hallucination, by accident. The direct backend analogy: a function that returns the right value not because it read the database, but because it guessed and hardcoded it. The test passes. The behavior is wrong — and the day the real data changes, the function will keep returning the old value without ever flagging it.

Grounded ≠ explicit citation. « €320,000. » and « According to the documents, the price is €320,000. » are equally grounded responses if the context confirms that price. The first simply isn’t formally traceable. Conflating the two produces a very concrete false diagnosis: a correct, grounded response gets flagged as if the model ignored the context, purely because it doesn’t cite its source.

Faithfulness (RAGAS) and FaithfulnessMetric (DeepEval) measure the first distinction. Neither measures the second — explicit citation is a traceability signal, not a truthfulness one.

Metrics aren’t objective — they’re relative to the evaluator

A direct consequence of the LLM-as-a-judge principle mentioned above: a faithfulness score of 0.92 doesn’t mean « the response is 92% true. » It means: this judge, with this prompt, at this moment, considers the response to look strongly grounded. On the same set of responses, a heuristic evaluator (lexical matching) might flag a 40% hallucination rate, while a semantic LLM judge considers 80% of the responses grounded — both are right by their own criteria. Swap the judge model — Llama 3.1 8B vs. Claude — and the same set of responses can produce noticeably different scores. Neither score is « the truth. » That’s true for RAGAS as much as for DeepEval, since both rely on the same underlying principle.

RAGAS: the exploratory mode

With RAGAS, you build a dataset — questions, generated answers, retrieved contexts, reference answers — then run an evaluation function that goes through all of it and returns a table of scores. No dashboard, no built-in tracking over time. It’s built for comparing two versions of a pipeline side by side, not for running continuously.

On my AI Evaluation Lab, the RAGAS integration was used to recompute those 4 metrics and compare the results against what I’d first coded by hand. RAGAS doesn’t do anything conceptually new — it automates, with embeddings and an LLM judge, what you can first understand by coding it naively yourself.

Where RAGAS is weak: no native CI/CD integration, no flexible synthetic dataset generation (heavy dependency on LangChain/LlamaIndex).

DeepEval: the gate mode

DeepEval approaches the problem from the opposite angle: if you know how to write a pytest test, you know how to write a DeepEval test. You define a test case — the question asked, the response generated, the retrieved context — then attach it to one or more metrics with a tolerance threshold. The test fails if the score drops below that threshold, exactly like a regular assertion.

deepeval test run in a GitHub Actions pipeline, and a PR that drops faithfulness below the threshold doesn’t pass. On the AI Evaluation Lab, wiring DeepEval up with a Groq model instead of OpenAI required writing a small adapter — DeepEval expects an OpenAI-compatible LLM by default. That detail illustrates the difference in nature from RAGAS well: DeepEval is designed as a test framework meant to be integrated into a pipeline, with its own integration constraints, not as a library you call once on a dataset.

DeepEval also goes further than RAGAS in scope: multi-turn evaluation (ConversationalTestCase), agent evaluation, customizable dataset synthesis without dependency on any particular retrieval framework.

The pattern that keeps coming up

RAGAS to explore offline, DeepEval to lock in CI. You change your chunking strategy, compare with RAGAS on a few hundred questions, validate the new threshold, then lock it in with DeepEval so a future PR can’t silently regress it.

Both share the same limitation: they’re LLM-as-judge metrics, so they’re sensitive to the judge model chosen. Neither tells you what happens once you’re in production — that’s the role of an observability layer, not an eval framework. That’ll be the subject of the next article.

To choose quickly

• First RAG, no CI established yet: RAGAS is enough to start measuring.
• Pipeline moving past prototype stage, CI already in place: add DeepEval as a gate as soon as possible.
• You’re iterating on retrieval (chunking, embeddings): RAGAS to compare versions on a dataset.
• You want to block a merge that degrades quality: DeepEval, no question.

A RAGAS or DeepEval score only means something if you’ve first defined what « a good answer » means for your use case — grounded, correct, complete, traceable aren’t the same requirement. Both tools measure against a definition. They don’t provide it.