Evals Engineer

Also known as: AI Evaluation Engineer, LLM Evaluation Engineer, Model Evaluation Engineer, AI Quality Engineer, ML Evaluation Engineer.

What does an Evals Engineer do?

An Evals Engineer is an engineer who builds the quality evaluation system for AI: the datasets, graders and checks the team uses to decide whether a model got better or worse. Part software engineer, part researcher, part statistician. It is rarely a separate title, but without this work every model change ships blind:

• builds evaluation datasets from real production requests;
• writes automated graders and calibrates LLM-as-judge with domain experts;
• builds regression checks that run before every release;
• works out how many examples are needed for a reliable result;
• shows the team where the model fails and what to fix first.

When does a company need an Evals Engineer?

A company needs an Evals Engineer when its AI product already serves users and the team changes models, prompts and pipelines faster than it can check the results. Typical triggers: users complain about quality after a model update, "better or worse" arguments are settled on three examples, and in legal, financial and medical products quality has to be proven to a client or a regulator. The role isn't needed at the prototype stage before there are real requests: then the AI / LLM Engineer builds quality evaluation themselves.

Evals Engineer vs QA Engineer: what's the difference?

An Evals Engineer differs from a QA Engineer by measuring the probabilistic quality of model answers on a sample rather than checking deterministic code behaviour against test cases. What sets them apart from an ML researcher is the focus on evaluation infrastructure for the product rather than new methods; the closest comparison is an infrastructure engineer who owns quality.

Why is it hard to hire an Evals Engineer?

It is hard to hire an Evals Engineer because the role is only now taking shape as a profession, and the combination of skills it needs is rare.

• The title is rare: evals usually appears as a skill inside ML or AI engineer job posts, so candidates are found by experience, not by job title.
• The role needs engineering, statistics and domain understanding at once; most candidates have one or two of the three.
• The work is almost invisible from outside: internal eval suites are private, and in a CV it reads as "improved model quality".
• The strongest signal is a public eval suite with a documented methodology, and few people have one.

What to decide before hiring an Evals Engineer?

• Who evaluates quality today, and on how many examples? The answer shows whether the system is built from scratch or extended, and those are different candidates.
• Can the Evals Engineer block a release? If their verdict can be ignored, the role turns into reports nobody reads.
• How much domain-expert time is available for labelling and calibration? Without experts the graders can't be calibrated, and the evaluation will be inaccurate.
• Which model errors are critical for the business? This decides what to measure first and whether the candidate needs experience in your domain.

Where to find an Evals Engineer?

Evals Engineers most often come from quality teams of legal and financial AI products, from ML engineers in search and ranking, where quality has long been measured on samples, and from maintainers of open-source model evaluation frameworks. The signal in a profile is a public eval suite or a paper with a methodology, and experience calibrating a judge model with experts. NEWHR finds such candidates by the results of their work, not by applications to a job post.

How to assess an Evals Engineer in an interview?

An Evals Engineer is assessed with a practical task: design an evaluation for one real company scenario, including the sample size and the judge model's agreement with humans. If the candidate has public eval work, it is reviewed separately. On the company side, the Head of AI or a lead ML engineer takes part and, where possible, a domain expert.

Strong signals:

• starts by asking which errors matter to the user;
• calculates the sample size and talks about the margin of error;
• checks the judge model's agreement with humans before trusting it.

Red flags:

• proposes one general metric for every scenario;
• trusts LLM-as-judge without calibration;
• can't explain how their evaluation affected a release decision.

How long does it take to hire an Evals Engineer?

We present the first candidates for the Evals Engineer role within 5–7 days of the brief.

Related roles

• QA Engineer — checks code behaviour against test cases, not the probabilistic quality of a model.
• AI / LLM Engineer — builds LLM features; the Evals Engineer checks whether they got better.
• Agentic AI Engineer — builds agents, which need especially rigorous quality evaluation.
• Data Scientist — also works with statistics and samples, but usually for product decisions rather than model releases.