Member of Technical Staff - ML Training Infrastructure

 About Pantheon

Pantheon's mission is to build and deploy billions of useful robots to automate physical labor, starting with bi-manual manipulators for industry. Industrial labor makes up almost $1T of U.S. spend, and yet over 439,000 jobs remain unfilled. Pantheon is building a future where labor is as cheap as energy and raw materials.

The approach is full-stack: owning hardware, ML, and deployment infrastructure is all required to reach 99.9%+ robotic reliability. Robotics is an engineering, operational, and infrastructure problem as much as it is a research one.

 

The Opportunity

This is Pantheon's first dedicated training infrastructure hire. You will own the compute the research team runs on and build most of it from scratch — GPU clusters, data clusters, scheduling, orchestration, and the training stack itself.

Pantheon is pretraining a robotics foundation model now, with runs scaling to hundreds of GPUs in the coming weeks. Your decisions about cluster architecture, job scheduling, and training reliability will directly set how fast the research team can iterate. You will inherit a live pretraining program, not a greenfield roadmap — the first 90 days are about hardening smoke-run infrastructure into runs at scale, owning the scale-up, and writing the playbook that makes large runs routine.

 

What You'll Do

• •      Own distributed training end-to-end: parallelism strategy, multi-node performance, and scaling efficiency, including evaluating, qualifying, and operating GPU capacity from cloud providers
• •      Build the data path from storage to GPU: high-throughput loaders, pre-encoding pipelines, and sampling infrastructure that keep hundreds of GPUs fed without stalls
• •      Design fault tolerance for long-running training: checkpointing, checkpoint durability, automatic failure detection and restart
• •      Build the evaluation harness that runs automatically against every checkpoint — probes, calibration checks, dashboards — so research always knows whether a run is working
• •      Own observability and reproducibility: experiment tracking, alerting, pinned environments, and profiling to find and eliminate bottlenecks across compute, networking, and storage
• •      Translate research requirements into production-grade systems and build infrastructure that makes experimentation fast

 

You Should Have

• •      Experience training or operating multi-node distributed training jobs at 100+ GPUs, including debugging NCCL timeouts, GPU starvation, and checkpoint corruption on runs you were responsible for
• •      Strong software engineering and distributed systems fundamentals
• •      Experience operating production systems where reliability and performance matter — you care about observability and reproducibility by reflex
• •      Comfort debugging complex issues spanning hardware, networking, storage, data pipelines, and ML systems

 

Nice to Have

• •      Pretraining experience at an AI lab, or training-platform experience at a robotics or AV company
• •      Bare-metal storage cluster experience
• •      Familiarity with Slurm, Kubernetes, Ray, or similar orchestration tools; WebDataset or similar sharded-loader stacks
• •      Experience with dataset versioning, manifests, or experimentation platforms
• •      Experience working directly with researchers to improve model performance through infrastructure