What does an Inference Engineer do?

Nobody in this job trains anything. The weights arrive finished, and the work is making them serve fast, cheap, and reliably.

August 20, 2026

An Inference Engineer takes a model that already exists and makes it serve — fast enough, cheaply enough, and reliably enough to sit under production traffic. Nobody in this job trains anything. The weights arrive finished; the work starts there.

It is the newest distinct discipline in the AI stack, and it exists for a boring commercial reason. Once frontier models moved out of demos and into products, serving them became one of the largest line items a software company had ever seen, and somebody had to own that number. That turned out to require a skill set that neither model researchers nor ordinary backend engineers had.

The job is four numbers

Almost everything an Inference Engineer does can be traced back to moving one of four measurements, and postings are usually explicit about which one they care about:

Time to first token (TTFT). How long the user stares at nothing. Dominated by prompt processing and by how long the request waited in a queue.
Inter-token latency. How fast text streams once it starts. This is what makes a chat interface feel quick or laboured.
Throughput per GPU. Tokens per second the fleet sustains under load. Determines how much hardware the traffic needs.
Cost per million tokens. The one finance cares about, and downstream of the other three.

These trade against each other, which is the whole difficulty of the job. Batching more requests together raises throughput and lowers cost per token — and makes every individual request wait longer. Almost every technique below is a way of buying one of these numbers, and the engineering judgement is knowing which one the product actually needs.

Why a request has two halves

The single most important structural fact about serving a language model is that one request runs in two phases with completely different bottlenecks.

Schematic, not measured data — the proportions are illustrative. Prefill reads the whole prompt in one pass and saturates the GPU’s arithmetic units. Decode then generates one token at a time, and each step must re-read the model weights and the growing KV cache from memory, so it is limited by memory bandwidth rather than by compute.

This split explains most of the toolkit. Techniques that help prefill do nothing for decode, and vice versa. A posting that talks about long-context or document processing is a prefill problem; one that talks about streaming chat latency is a decode problem. Being able to say which one you are looking at is roughly the first interview question.

The three layers you can work at

Inference roles differ enormously in how deep they go, and this is the axis that separates two postings sharing the same title.

The serving stack. Running and tuning vLLM, SGLang, TensorRT-LLM, or TGI against a real traffic shape: continuous batching, KV-cache management and prefix reuse, prefill/decode disaggregation, request scheduling, autoscaling a GPU fleet without leaving it idle or saturated. Plenty of good jobs live entirely here and never touch a kernel.
The model. Changing the artefact itself so it serves cheaper — quantizing weights and activations to FP8, INT8 or INT4 and measuring honestly what the compression cost in quality; speculative decoding, where a small draft model proposes tokens the big model verifies in a batch; distillation and pruning; choosing how to shard across GPUs with tensor, pipeline, or expert parallelism.
The kernel. Writing or tuning the GPU code underneath: CUDA, or Triton — meaning OpenAI’s kernel language, not NVIDIA’s Triton Inference Server, a name collision that trips up candidates constantly. This is attention kernels, fused operations, memory-layout work, and reading a profile to find where the bandwidth actually went.

Postings rarely say which layer they mean in the title. They give it away in the requirements: named serving frameworks without any mention of CUDA is layer one; “CUDA” or “kernel” anywhere in the requirements is layer three, and the interview will reflect that.

What the job is not

The boundary questions come up constantly, and all three have clean answers:

Not training. Pretraining, fine-tuning, RLHF, and research are a different discipline with different skills. The dividing line is whether the weights are an input to your work or an output of it.
Not MLOps. Pipelines, feature stores, experiment tracking, and model registries are platform engineering for the ML lifecycle. An Inference Engineer is measured on latency and cost, not on the reliability of a workflow.
Not general backend. The API in front of the model is ordinary service engineering. The job here starts where the GPU does.

Reading a posting

Four questions separate a real inference role from one wearing the vocabulary:

Does it name a metric? TTFT, p99, tokens per second per GPU, or cost per million tokens in the responsibilities means somebody owns a number. Postings without one are often platform roles that touch inference occasionally.
Does it name a serving stack? vLLM, SGLang, TensorRT-LLM, TGI. Their absence in a role claiming to be about serving is a genuine warning sign.
Does it say CUDA or Triton? That is the layer-three tell, and it roughly doubles the depth of the interview.
Whose model is it? Serving open-weight models you host is a different job from serving a frontier model you trained, which is different again from building a platform other people serve on.

Who hires for this

Four groups, and the work differs between them: model labs serving their own frontier models; inference providers and GPU clouds whose entire product is serving somebody else’s; chip vendors who need the software stack to make the hardware look good; and, increasingly, ordinary application companies whose inference bill grew large enough to justify a specialist. That last group is the one expanding fastest, and it is where most of the roles that don’t require kernel work will come from.

Titles are unreliable here even by the standards of this industry. Inference Engineer, Model Serving Engineer, ML Performance Engineer, GPU Engineer, Kernel Engineer, and “Member of Technical Staff, Inference” frequently describe the same job — which is exactly why this board classifies Inference Engineer roles by reading the description rather than the title.

A note on pay

We would rather publish nothing than a number we cannot stand behind, and inference is a case where we have nothing yet: the role was added to this board’s taxonomy days before this piece was written, and it has no live postings to draw a salary distribution from. When it does, the figures will appear in the salary report, which currently covers the five adjacent deployed-engineering roles and is a reasonable proxy for the market these roles compete in.

What can be said without data: inference sits close to Applied AI Engineering in the org chart and nowhere near it in required skills. If you are weighing the two, the comparison with ML and AI engineering is the more useful read.

Browse Forward Deployed Engineer jobs

Get new deployed-engineering roles in your inbox each week: