BYOB: Bring Your Own Benchmark

The landscape of AI and LLMs is fast-moving. Literally every week there is at least one new model being released. So how do you navigate safely this field, if what you are searching for is just the model which performs the best?

Sure, every model is shipped with a plethora of benchmarks it has been evaluated against, but these are quite wide-scoped and might not fit how the model will perform on your specific task. This is the same reason my Data Science Insights notes keep returning to one theme: evaluation only becomes useful when it reflects the decision you are actually making.

The problem

You might argue, if these benchmarks are not useful, why do they exist in the first place? Well, there is quite a lot of hype around AI that in turn pushed for a race to be top of the rankings with your new model, because that means publicity and traction for your project.

They can be informative anyway; they give an indication of how well a model might perform, but they shouldn’t just be taken at face value, as they are weak proxies for what your users will actually experience using your AI agent.

The solution? BYOB

OK, I am not suggesting any alcoholic solution here. I just recycled this acronym from the hospitality world (I first saw this sign years ago on the window of one of my favourite Indian restaurants in London), but in this case it means:

    Bring
    Your
    Own 
    Benchmark

And what does that mean? If you want to ensure that the AI agent you spent weeks (months?) building is performing as it should, you need to have in mind how the users will interact with it.

Why does a custom benchmark tell you more about production behaviour?

The idea behind why this approach is more effective at evaluating your agent is tied to the fact that if the test cases resemble how the users are interacting with your system, then the failures you will spot will also resemble the ones the users will experience in production. This gives the benchmark predictive value for the use case you care about, rather than a more generic high-level academic performance.

A concrete example: benchmarking a Text-to-SQL agent

Let’s make it more tangible. Suppose you are building a Text-to-SQL agent that answers questions over a set database.

The setup

Let’s build a simple schema for the sake of the example:

• customers(id, name, country)
• orders(id, customer_id, amount, created_at)
• products(id, name, category)
• order_items(order_id, product_id, quantity)

Even with a tiny schema, there are joins, aggregations, and time filters — enough to surface real failure modes.

The benchmark dataset

Now let’s construct a small but representative benchmark of user questions.

Each example includes:

• a natural language query
• the ground-truth SQL
• optional notes about ambiguity or edge cases

Example cases:

Case 1 — basic aggregation

User query:

“How many orders were placed in 2024?”

Ground truth:

SELECT COUNT(*) AS n_orders
  FROM orders 
 WHERE created_at >= '2024-01-01' 
   AND created_at < '2025-01-01';

Case 2 — join + grouping
User query:

“Total revenue by country”

Ground truth:

  SELECT c.country, SUM(o.amount) AS tot_revenue
    FROM orders o
    JOIN customers c
      ON o.customer_id = c.id
GROUP BY c.country;

Case 3 — filtering + ranking

User query:

“Top 3 product categories by sales volume”

Ground truth:

  SELECT p.category, SUM(oi.quantity) AS tot_quantity
    FROM order_items oi
    JOIN products p ON oi.product_id = p.id
GROUP BY p.category
ORDER BY tot_quantity DESC
   LIMIT 3;

Case 4 — ambiguous phrasing (edge case)

User query:

“Best customers last year”

Ground truth (one acceptable interpretation):

  SELECT c.id, c.name, SUM(o.amount) AS tot_spent
    FROM orders o
    JOIN customers c 
      ON o.customer_id = c.id
   WHERE o.created_at >= '2024-01-01'
     AND o.created_at < '2025-01-01'
GROUP BY c.id, c.name
ORDER BY tot_spent DESC
   LIMIT 10;

Note: this case is intentionally ambiguous — “best” could mean highest spend, most orders, or highest margin.

What counts as correct?

For Text-to-SQL, exact string match would just be too brittle. Two queries can differ syntactically and still be equivalent.

Instead, you define multiple scoring layers:

• Execution accuracy (primary metric) Run both the generated SQL and the ground truth against the database.
  Score = 1 if results match, 0 otherwise.

• Semantic equivalence (fallback) If results differ due to ordering or formatting, normalize outputs (e.g., sort rows, ignore column aliases).

• Constraint checks (pass/fail)

  • No invalid tables or columns
    
  • No dangerous queries (e.g., DROP, DELETE)
    
  • Must include required filters (e.g., time constraints when specified)

• Partial credit (optional)
  For complex queries:

  • Correct joins: +0.3
    
  • Correct aggregation: +0.3
    
  • Correct filters: +0.4

This gives you signal even when the final query is not fully correct.

Evaluating the agent

If your system is an agent, you may also track:

• Tool use correctness: Did it query the right tables?
• Iteration quality: Did it recover from an initial error?
• Latency: Time to final answer
• Cost: Number of model calls or tokens used

For example, an agent that produces the right SQL after three failed attempts might pass on accuracy but fail on efficiency.

For classic supervised models, the same “what mistake matters?” question shows up in accuracy, precision, recall, and F1. I built Signal vs Noise as an interactive way to explore those trade-offs.

Why this works

This benchmark is small, but it reflects real usage:

• users ask vague questions
• joins are required
• aggregation is common
• and ambiguity is unavoidable

Because the test cases resemble production, the failures will resemble production too. That’s what gives the benchmark predictive value.

What should go into your benchmark?

If you look at the example above, the important part is not the SQL — it’s the selection of cases.

A useful benchmark is not a random sample of tasks. It is a curated slice of real test cases.

In practice, that means including:

• common requests (the bulk of your traffic)
• edge cases (where things usually break)
• ambiguous queries (where interpretation matters)
• failure cases you have already observed
• “annoying” inputs users actually write
• anything that has triggered a support ticket

A good rule of thumb is that your benchmark should look less like a demo and more like your support queue.

What counts as correct?

Some tasks have exact answers. Many don’t.

So instead of asking “does this match the gold output?”, you ask:

• does it produce the right result?
• does it respect constraints?
• is it acceptable for a user?

This is where concepts like execution-based evaluation, partial credit, and pass/fail constraints become more useful than exact matching.

Are you evaluating the answer or the system?

The moment you move from a model to an agent, the unit of evaluation changes.

In the example, we didn’t just care about the final SQL query. We also cared about:

• how many attempts it took
• whether it used the right tables
• whether it recovered from errors
• how long it took
• how expensive it was

Two systems can produce the same final answer and still be very different products.

A benchmark that only evaluates outputs will miss that difference.

Where this can go wrong

Building your own benchmark is not a silver bullet.

It can still fail you if it is:

• too small to be representative
• too clean compared to real inputs
• too static while your product evolves
• too easy to overfit against
• disconnected from actual user traffic

A misleading custom benchmark is still misleading — just more convincingly so.

For the third point specifically, there is a way to make sure the tests evolve with your product, and that’s creating a feedback loop. That means capturing how customers are interacting with the AI agent in near real-time using this signal to direct the effort (e.g. do the user ask more about join-heavy queries, do they care more about aggregated figures?)

“Isn’t this too much work?”

Yes, it is more work than watching a leaderboard.

But you do not need hundreds of examples to get value.

A small benchmark of 20–50 high-signal cases is often enough to compare models meaningfully, catch regressions - and guide iterations.

The deeper point

A benchmark does not just measure your system, it forces you to answer the following questions:

• what does “good” actually mean?
• what failures are acceptable?
• what should never happen?
• what trade-offs are we willing to make?

In that sense, the benchmark is a compressed version of your product spec.

What you gain

Once you have a benchmark like this, decisions become easier and less subjective.

You can:

• compare models based on your actual use case
• detect regressions before users do
• understand trade-offs (accuracy vs latency vs cost)
• iterate faster with clear feedback loops

Most importantly, you stop relying on vibes.

NoteBuild your own benchmark

Ready to evaluate your AI system against real user behaviour?

Download the free BYOB Benchmark Starter Kit—a practical template for defining test cases, scoring outputs, tracking regressions, and comparing models.

Download the free starter kit

The shift

In conclusion, adopting this approach means moving from:

“Is this model good?”

to:

“Is this system good at my job, under my constraints, for my users?”