Hiring.Camp

MLOps Senior Lead Engineer

Nvidia

·

Today

Location
Santa Clara, CA,US, US · Seattle, WA,US, US
Workplace
Remote
Type
Full-time
Department
Engineering
Seniority
Lead
Source
Eightfold

Description

NVIDIA has been transforming computer graphics, PC gaming, and accelerated computing for more than 25 years. It’s a unique legacy of innovation that’s fueled by great technology—and amazing people. Today, we’re tapping into the unlimited potential of AI to define the next era of computing. An era in which our GPU acts as the brains of computers, robots, and self-driving cars that can understand the world. Doing what’s never been done before takes vision, innovation, and the world’s best talent. As an NVIDIAN, you’ll be immersed in an inclusive, supportive environment where everyone is inspired to do their best work. Come join the team and see how you can make a lasting impact on the world.

The Managed AI Research Superclusters (MARS) organization at NVIDIA deploys and operates modern GPU high performance computing systems for NVIDIA researchers. We are seeking a senior engineer for our MLOps team: expert in HPC and computational machine learning who work alongside researchers to dig into their requirements, workloads and infrastructure needs, and turn recurring problems into platform improvements.

What you will be doing:

  • Set the technical direction for the MLOps team. Keep the team passionate about customer efficiency. Serve as the senior hands-on engineer others rely on in complex customer and performance situations.
  • Work 1-to-1 with researchers on LLM, autonomous-vehicle and generative-model workloads to diagnose training and inference performance problems and repeated crashes across compute, network and storage.
  • Debug large-scale distributed training end to end: NCCL collectives, high-speed interconnects and remote direct memory access, parallel storage, containers (Enroot, Pyxis), and Slurm and Kubernetes schedulers.
  • Find recurring problems across customers, work with MARS peers, partners and platform teams to fix them at scale, and translate customer needs into platform feature requests.
  • Test the platform: run what customers run, and shift testing earlier so issues are caught before capacity is handed over.
  • Produce trainings, guidance and reference tests; share what we learn through blog posts, talks and papers.
  • Raise the engineering bar: mentor engineers at all levels, pair on bring-ups, run design reviews and keep decisions written down with a clear owner.

What we need to see:

  • BS or MS in Computer Science, Engineering or a related field, or equivalent experience.
  • 8+ years designing, operating or supporting large-scale GPU or HPC compute infrastructure, with experience in distributed training and inference as an infrastructure engineer or advanced user.
  • Strong hands-on experience with Slurm or Kubernetes for model development and prediction, and with containers (Docker, Enroot, Apptainer or Singularity).
  • Solid understanding of at least one distributed training framework (PyTorch, NeMo or Megatron, JAX) and inference servers (Triton, TorchServe, Ray Serve).
  • Experience profiling distributed training performance with tools such as the PyTorch profiler or flight recorder, Nsight Systems and Nsight Compute, and a proven grasp of NCCL collectives such as all-gather and all-to-all.
  • Fluency in Python and at least one systems language (Go, Rust or C++); familiarity with infrastructure as code such as Terraform.
  • Ability to explain how to isolate a performance problem to compute, network or storage, and to debug several kinds of complex systems.
  • Data-driven judgment, and the ability to influence without authority across research, platform and partner teams. Excellent written and verbal communication.

Ways to stand out from the crowd:

  • Experience leading bring-up or acceptance testing of GPU clusters with thousands of nodes, including MLPerf or similar benchmarking.
  • Deep knowledge of InfiniBand and RDMA, and of fast distributed storage such as Lustre or GPFS for AI and HPC workloads.
  • CUDA programming background and a clear picture of how GPUs, networks and storage interact during training.
  • Experience with observability for ML workloads (OpenTelemetry, Grafana, ClickHouse) and with goodput or efficiency metrics.
  • Track record of technical leadership: mentoring engineers, shaping a team's engineering practices, and engaging partners and vendors.

Widely considered to be one of the technology world’s most desirable employers, NVIDIA offers highly competitive salaries and a comprehensive benefits package. As you plan your future, see what we can offer to you and your family www.nvidiabenefits.com/

Your base salary will be determined based on your location, experience, and other objective job-related factors. The expected base salary ranges for this position are:

187,500 USD – 364,000 USD for Level 4; and 232,500 USD – 442,000 USD for Level 5.

You will also be eligible for equity and benefits.

Applications for this job will be accepted at least until November 23, 2026.

This posting is for an existing vacancy.

NVIDIA uses AI tools in its recruiting processes.

NVIDIA is committed to fostering an inclusive work environment and proud to be an equal opportunity employer. As we highly value diversity in our current and future employees, we do not discriminate (including in our hiring and promotion practices) on the basis of race, religion, color, national origin, gender, gender expression, sexual orientation, age, marital status, veteran status, disability status or any other characteristic protected by law.

Skills

PythonRustDockerKubernetesTerraformClickHouseMachine LearningPyTorchCUDAGoLLM

Similar Jobs

1

Senior Consultant / Manager – MLOps & LLMOps Lead (H/F/X)

Wavestone·Puteaux, Île-de-France·Onsite

1d ago