Senior AI Inference Engineer
About StackYak
StackYak is building the infrastructure layer for AI.
We are an early-stage, funded company building software that brings compute, GPU infrastructure, networking, and inference together into one product. We are building for real production workloads from the start.
This is not internal IT or a slow-moving infrastructure team maintaining someone else's platform. The infrastructure is the product.
We are a small, senior team with very little bureaucracy. This is a founding role in its discipline. You will have substantial ownership over how our inference systems are designed, deployed, and operated.
Treat this description as a starting point rather than a boundary. We value people who take ownership early, make well-reasoned technical decisions, and execute without waiting for perfectly defined requirements.
The Role
We need someone who can take a model, a set of GPUs, and a production requirement and determine how that model should actually run.
Given a frontier open-weight model — dense or mixture-of-experts, tens to hundreds of billions of parameters — you should be able to determine the right precision, quantization, tensor/pipeline/expert parallelism, memory strategy, batching, concurrency, topology, and serving runtime. Then defend those decisions, measure the results, and operate the system in production.
Beyond operating inference infrastructure, we want to turn that expertise into software. Decisions that currently require an experienced engineer should become reproducible, testable, and eventually automated.
This is not a research position or an architecture-only role. You will benchmark, deploy, debug, tune, automate, and operate real inference systems with customers depending on them.
What You Will Own
- Model placement, parallelism, precision, quantization, and memory strategy across single-GPU, multi-GPU, and multi-node deployments.
- Selection, evaluation, and operation of serving runtimes including vLLM, SGLang, TensorRT-LLM, and emerging alternatives.
- Evidence-based concurrency limits, latency and time-to-first-token (TTFT) targets, throughput expectations, and capacity commitments.
- Benchmarking methodology, tooling, reproducibility, and performance analysis.
- Production model lifecycle: loading, startup, health, upgrades, capacity management, and failure handling.
- Architecture and operational boundaries between multi-tenant and dedicated inference.
- Incident response and root-cause analysis across inference runtimes, GPUs, drivers, hosts, and networks.
- Software that automates inference configuration, placement, capacity planning, and deployment decisions.
What Success Looks Like
Given a model, a hardware target, and a workload profile, you can quickly produce a deployment that is sensible, measurable, reproducible, and production-ready.
You can answer which hardware can serve a model, how to distribute it across GPUs or nodes, which runtime and precision to use, what concurrency is sustainable, and how to balance latency, throughput, and cost.
Then you help StackYak answer those questions programmatically instead of requiring an expert for every deployment.
How We Work
- Small, senior team with direct access to the founders.
- Strong opinions, loosely held. Everyone participates in technical decisions.
- Shared responsibility for production reliability and on-call.
- Engineers move between design, implementation, debugging, and operations.
- We value demonstrated accomplishments over degrees, certifications, and credentials.
- We debate technical decisions, commit to a direction, and execute.
- Remote, distributed team working across time zones.
- We are hiring across inference, compute infrastructure, and networking. Boundaries are intentionally flexible.
- Early-stage startup environment with substantial ownership, fast execution, and evolving priorities.
Required Experience
We care about what you have already built and operated, not just what you know in theory. You should have direct experience with most of the following:
- Operating large language models in production under real customer workloads.
- Serving large models across multiple GPUs and multiple nodes.
- Sizing models against available hardware, including GPU memory and topology constraints.
- Selecting precision and quantization strategies with measurable implications for quality, performance, and cost.
- Tuning batching, concurrency, and memory utilization beyond default configurations.
- Measuring and interpreting latency, TTFT, throughput, utilization, and inference cost.
- Working with NVIDIA and/or AMD GPU inference environments.
- Building production Python software, infrastructure tooling, and automation.
- Debugging Linux and systems problems below the container boundary.
- Diagnosing inference failures originating in GPU hardware, drivers, hosts, networking, or other infrastructure rather than the serving runtime itself.
- Taking ownership of production systems, including operational reliability, troubleshooting, and incident response.
Especially Valuable Experience
- Operating inference infrastructure at an AI company, inference provider, GPU cloud, hyperscaler, or substantial internal AI platform.
- Working across multiple GPU generations and vendors, with practical understanding of how hardware differences affect deployment decisions.
- Understanding distributed inference and the networking implications of multi-node serving.
- Benchmarking and comparing vLLM, SGLang, TensorRT-LLM, or other serving runtimes.
- Making and explaining architectural trade-offs rather than relying on vendor recipes or framework defaults.
- Building automated deployment systems, schedulers, placement engines, or capacity planners.
- Turning specialized operational knowledge into repeatable software and automation.
- Independently exploring, building, and testing systems beyond assigned work.
This Role Is Probably Not a Fit If
- Your experience is primarily model training or ML research rather than production inference.
- You have deployed a model behind an API but have not worked directly with GPU memory, parallelism, topology, and concurrency.
- You treat default serving framework configurations as a complete architecture.
- You enjoy benchmarking but do not want responsibility for operating production systems.
- You prefer kernel optimization without interest in reliability, infrastructure, or productization.
- You prefer documenting recommendations over implementing and operating solutions.
- You want narrowly defined responsibilities or need an extended ramp-up before taking ownership.
- You would rather remain the person with specialized knowledge than build software that makes that knowledge broadly accessible.
Compensation
Competitive compensation plus meaningful equity. Exact structure depends on location, engagement model, and experience. Employment versus contracting arrangements are determined based on location.
Apply
https://jobs.gem.com/stackyak/am9icG9zdDrauilGE5SD2AfJ04PacNuT
A Note for Agencies
We are not accepting submissions from external recruiting agencies for this role. Unsolicited candidate submissions do not create any fee obligation.
