Sr. Principal Machine Learning Software Engineer - Inference Serving Engine
Ampere Computing
Job details
Pay
$225,500 – $338,500 / yearly
Job type
Full-time
Work setting
Remote / Hybrid
Location
Portland, OR
Posted
Posted today
Benefits
Pulled from the full job description.
Full job description
Ampere is a semiconductor design company developing SoCs and platform solutions focused on high-performance, energy-efficient AI compute, part of the SoftBank Group.
Lead the architecture and development of high-performance AI inference-serving software for Ampere accelerators - working across model serving, frameworks, compilers, runtimes, and optimized kernels to enable efficient deployment of modern AI workloads including generative AI and LLMs. Hands-on technical leadership role.
What you'll achieve:
- Architect and build an inference-serving engine that optimizes state-of-the-art AI models on Ampere deep learning accelerators
- Drive end-to-end serving performance: throughput, latency, tail latency, memory efficiency, scalability, and cost per inference
- Develop batching and scheduling, KV-cache management, model loading, memory planning, quantization, and multi-device/distributed inference
- Optimize model execution via graph compilation, operator fusion, precision optimization, and efficient attention
- Influence future accelerator architecture through hardware/software co-design
- Mentor engineers and lead technical decisions across cross-functional initiatives
Hybrid role in Portland, OR. Benefits include premium medical/dental/vision, 401K, unlimited flextime, and 10+ paid holidays.
Qualifications
- Experience with LLM, transformer, multimodal, or generative-AI inference
- Experience with PyTorch, oneDNN, SYCL, OpenCL, OpenMP, MPI, CUDA, ROCm, or similar
- Experience with ML compilers, graph optimization, code generation, kernel development, or device runtimes
- Familiarity with distributed inference, cloud-native serving, containers, Kubernetes, observability
- Experience with mixed precision, INT8/INT4/FP8 quantization, sparsity, or efficient-attention methods
- Contributions to open-source AI, compiler, runtime, or HPC projects
Requirements
- Bachelor's in CS/CE/Math/EE or related field with 12+ years of relevant experience; master's with 8+ years; or Ph.D. with 5+ years
- Experience developing or optimizing production inference runtimes, serving engines, deployment toolkits, or high-performance AI systems
- Experience with vLLM, llama.cpp, TensorRT/TensorRT-LLM, ONNX Runtime, OpenVINO, Triton Inference Server, TVM, XLA, MLIR, or comparable
- Expert proficiency in modern C++ and strong C experience, including concurrent programming and systems-level performance optimization
- Strong understanding of AI inference performance: batching, scheduling, latency/throughput tradeoffs, KV-cache behavior, quantization
- Demonstrated ability to lead complex cross-functional technical initiatives
Ready to apply?
Apply on the company siteYou’ll be taken to an external site to complete your application.