DeployedJobs

Inference Engineer Jobs

Inference Engineers make an already-trained model serve fast, cheap, and reliably in production — serving stacks, GPU kernels, quantization, and the latency and cost numbers the business is held to. The hiring sits at the frontier labs, the inference providers, and the chip and cloud vendors building the serving layer underneath them. Every posting here is scored for how deep into inference infrastructure it actually goes, which the title on its own never tells you.

53 verified roles
← PrevPage 3 of 3

What an Inference Engineer actually does

An Inference Engineer owns what happens after training finishes. The model exists; the job is to serve it under real traffic without the latency, the GPU bill, or the reliability falling over. That work runs from the serving stack down to the kernel, and it is measured in numbers the rest of the company can read: time to first token, p99 latency, tokens per second per GPU, cost per million tokens.

It is the one role on this board that is not customer-facing, and it is here because it is the other half of the same shift. As frontier models moved from demonstrations into production systems, somebody had to make them affordable at scale — and that turned out to be a distinct discipline from both model research and ordinary backend engineering.

Titles vary more than the work does. Inference Engineer, Model Serving Engineer, ML Performance Engineer, GPU or Kernel Engineer, and “Member of Technical Staff, Inference” frequently describe the same job. Training, pretraining, and feature-store work are a different discipline and are not included here.

Day to day

  • Tuning a serving stack — vLLM, SGLang, or TensorRT-LLM — against a real traffic shape
  • Profiling a model to find where the latency and the GPU memory actually go
  • Quantizing a model and measuring what the compression cost in quality
  • Writing or tuning CUDA and Triton kernels when the framework leaves throughput behind
  • Running the batching, caching, and autoscaling that keep a GPU fleet busy but not saturated

What hiring teams screen for

  • Python
  • CUDA, Triton, or C++ for kernel work
  • PyTorch internals
  • A serving stack (vLLM, SGLang, TensorRT-LLM, TGI)
  • Quantization and compression (FP8, INT8/INT4, AWQ, GPTQ)
  • GPU architecture and memory hierarchy
  • Distributed inference — tensor, pipeline, and expert parallelism
  • Profiling, benchmarking, and honest measurement

Common questions

How is this different from an ML engineer?
An ML engineer usually trains or fine-tunes models; an Inference Engineer takes a finished model and makes it serve. The skills barely overlap — this side is systems and performance work, closer to low-level backend engineering than to data science.
Do I need to write CUDA?
For a growing share of these roles, yes, at least enough to read a profile and tune a kernel. Plenty of postings sit entirely at the serving-stack level and never touch a kernel, so read the posting — every one here carries an inference and serving depth score, and kernel-level work is what pushes it above 80.
Why is a non-customer-facing role on a deployed engineering board?
Because the same companies are hiring for both, and the two roles are ends of one pipeline: an FDE gets a model into a customer's workflow, an Inference Engineer makes serving it affordable. Every posting is scored for customer-facing time, so the distinction stays visible rather than blurred.
Who is hiring for this?
Model labs, inference providers and GPU clouds, chip vendors, and increasingly any application company whose inference bill has become a line item worth an engineer. The employer list on this page is generated from live postings.

Related roles