AI Infrastructure / Inference Engineer

Also known as: Inference Engineer, ML Inference Engineer, AI Infrastructure Engineer, LLM Serving Engineer, ML Systems Engineer.

What does an AI Infrastructure / Inference Engineer do?

An AI Infrastructure / Inference Engineer is an engineer responsible for models in production running fast and cheaply: sets up serving, manages GPU resources and tracks the cost of every answer. It is an engineering role; running a GPU cloud or cluster is a separate role, the AI / GPU Infrastructure Lead. Typical tasks:

• chooses and tunes model serving frameworks (vLLM, TensorRT-LLM, Triton Inference Server);
• sets up batching, caching and quantisation for the required latency and throughput;
• manages GPU resources and autoscaling under load;
• profiles inference and removes bottlenecks;
• calculates the cost per answer and helps choose between own GPUs, the cloud and external APIs.

When does a company need an AI Infrastructure / Inference Engineer?

A company needs an AI Infrastructure / Inference Engineer when it runs models on its own GPUs or in the cloud rather than only through external APIs, and the inference bill or response latency becomes a business problem. Most often these are companies with their own or open-source models in production, with high request volumes, or with data requirements that rule out sending data to an external provider. The role isn't needed if all models are called through a provider's API at modest volume: an AI / LLM Engineer and an MLOps / LLMOps Engineer are enough.

AI Infrastructure / Inference Engineer vs MLOps Engineer: what's the difference?

An AI Infrastructure / Inference Engineer differs from an MLOps Engineer by squeezing performance out of specific hardware (latency, throughput, cost), while an MLOps Engineer builds the process: deployment, model versions, monitoring. A Neural Network Optimization Engineer works one level lower, on the model's weights, kernels and compilers.

Why is it hard to hire an AI Infrastructure / Inference Engineer?

It is hard to hire an AI Infrastructure / Inference Engineer because deep experience optimising inference on GPUs is rare, and the title means different things to different companies.

• The title is vague: an "AI Infrastructure Engineer" is DevOps for ML in one company and someone who profiles CUDA kernels in another.
• Most ML engineers have never gone below the default settings of a serving framework.
• The combination is rare: systems programming, an understanding of GPU architecture and knowledge of how modern models work.
• Strong candidates are concentrated in AI labs, GPU clouds and large companies with their own models, where they are valued and hard to lure away.

What to decide before hiring an AI Infrastructure / Inference Engineer?

• What matters most: latency, throughput or cost? Each target calls for different methods, and the candidate's experience should match yours.
• Own GPUs, the cloud or external APIs? An engineer who has only worked with managed cloud services may not know how to run your own hardware.
• Which models, and what request volume? Serving one large LLM and a dozen small computer vision models are different jobs.
• Who owns the GPU cluster itself? If there is no separate infrastructure team, this person will have to look after the hardware too.

Where to find an AI Infrastructure / Inference Engineer?

AI Infrastructure / Inference Engineers most often come from GPU clouds, AI labs and companies with their own models, as well as from HPC and CUDA development. The signal in a profile is a specific optimisation with before-and-after numbers (latency, throughput, cost) and contributions to open-source inference tools. NEWHR finds such candidates by the results of their work, not by applications to a job post.

How to assess an AI Infrastructure / Inference Engineer in an interview?

An AI Infrastructure / Inference Engineer is assessed by a walkthrough of an optimisation the candidate did (what the bottleneck was, metrics before and after, how much was saved) and a practical task on profiling and speeding up inference. On the company side, the Head of AI or CTO takes part, along with a lead ML infrastructure engineer.

Strong signals:

• starts with profiling rather than a ready-made recipe;
• talks in numbers: latency, throughput, cost per request;
• explains the trade-off between latency and throughput on a concrete example.

Red flags:

• knows only a framework's default settings;
• can't explain where the bottleneck was in their past optimisation;
• doesn't check how quantisation affected model quality.

How long does it take to hire an AI Infrastructure / Inference Engineer?

We present the first candidates for the AI Infrastructure / Inference Engineer role within 5–7 days of the brief.

Related roles

• Neural Network Optimization Engineer — optimises the model and the computation itself: weights, precision, kernels.
• MLOps / LLMOps Engineer — builds the deployment and monitoring process for models, not their performance.
• AI / GPU Infrastructure Lead — leads GPU clusters and the team at platform level.
• AI / LLM Engineer — builds the product on the models the inference engineer makes fast.
• SRE — owns the reliability of the general platform, not GPU inference.