How to know if your LLM is actually working

LLMs are unreliable in ways traditional software isn’t. A REST API returns the same response for the same input. An LLM doesn’t. It might work perfectly for 100 requests, then silently produce wrong answers on request 101. The model provider might update the model and subtly change behavior. Your prompt might drift as you add context.

You need eval infrastructure before you ship. Here’s what I built across multiple production LLM systems.

The three evaluation layers

Layer 1: Regression tests (deterministic)

Basic assertions on structured output. For Dealer’s Dash, every calculation routes through a deterministic math engine. The LLM never does arithmetic. This means I can write standard pytest assertions on the math output. If revenue_trend["q1_to_q2"] doesn’t equal the expected value, the test fails. No LLM involved in the assertion.

Similarly, if your LLM outputs JSON (which it should, for anything production), validate the schema. Every time. pydantic models with strict validation catch malformed outputs before they reach users.

Layer 2: Eval scoring (LLM-judged)

For tasks where correctness is subjective, use a separate LLM call to score the output. This is the “LLM-as-judge” pattern.

The RankOps SEO agent uses this. After Claude generates new page metadata, a separate Claude call scores it on relevance to page content (does the title match what the page is about?), click-through potential (would a human click this in search results?), and length constraints (is the title under 60 characters?).

Outputs scoring below threshold are regenerated. This adds one extra LLM call per generation but prevents bad output from shipping.

Key insight: the judge LLM should be a different model than the generator LLM. Claude judges GPT outputs better than it judges its own. Self-evaluation bias is real and well-documented.

Layer 3: Production monitoring (observability)

Evals catch what you can test. Monitoring catches what you can’t predict. At minimum:

  • Token usage per request. Spikes indicate prompt injection or unexpected context growth.
  • Latency P50/P95/P99. Degradation means model changes or infrastructure issues.
  • Output length distribution. Sudden shifts (all responses getting shorter) indicate prompt drift.
  • Error rate by category. Parse errors vs. rate limits vs. content filter blocks each need different handling.

I use Helicone for LLM observability (tracks cost, latency, token counts per model) plus Sentry for application-level errors. The combination covers both “the LLM did something wrong” and “the LLM didn’t run at all.”

The eval loop pattern

The most powerful pattern I’ve used: run evals continuously, feed results back into prompts.

  1. Every LLM output gets scored by a judge LLM (async, post-response)
  2. Low scores are stored with the input/output/judge-reasoning
  3. Weekly: review patterns in low scores
  4. Update prompts to address systematic failures
  5. Measure if scores improve after prompt update

This is what powers both RankOps and the Voice Call Center. The systems get better over time because there’s a measurement and improvement loop. Without it, you’re just hoping.

What breaks in production

The failures I’ve actually seen, in order of frequency:

  1. Rate limits. Provider throttles your requests mid-batch. Need exponential backoff and a dead-letter queue.
  2. Content filters. Completely fine requests get blocked. Need fallback models and graceful degradation.
  3. Silent model updates. Provider updates the model, your prompt behaves differently. Need eval tests that run on every deploy.
  4. Token cost explosion. A new user pastes a 50-page document into the chat. Need input length limits and per-user cost monitoring.
  5. Output format changes. Model starts returning slightly different JSON structure. Need strict schema validation at the API boundary.

None of these are “the AI became sentient.” All of them are boring infrastructure problems. Solve the boring problems first.

The minimum viable eval stack

If you’re building an LLM-powered product today, here’s what you need before launch:

  1. Schema validation on all LLM outputs (pydantic/zod)
  2. At least 10 regression test cases for your core prompts, run in CI
  3. LLM-as-judge scoring for one critical quality dimension, run on every response
  4. Token cost tracking per request, per user, per model
  5. Error rate dashboard with alerting above 2%

That’s it. You don’t need a perfect eval system. You need one that catches the worst failures before users do. Start there and iterate.

← All writing