Hiring.Camp

Member of Techincal Staff

Trainy

·

Today

Salary
$140k – $250k/yr
Location
San Francisco, California · San Mateo, California
Workplace
Remote
Type
Full-time
Source
Y Combinator

Description

About the role
Trainy operates the infrastructure behind some of the world's leading foundation model labs. Every day, our platform keeps more than 10,000 GPUs healthy, scheduled, and productive across multiple clouds and bare-metal clusters, where a single bad node can stall a training run that costs thousands of dollars an hour. We're looking for an early-career engineer to join our operations team and help keep that fleet running. You'll work on Konduktor, our Kubernetes-native orchestration platform, with a focus on the systems that detect failing hardware, surface problems before customers notice them, and automate the fixes. You'll also spend time onsite with our customers in the Bay Area, working directly with the teams training frontier models. This is a rare chance to work hands-on with frontier-scale AI infrastructure from day one.
What you'll be doing

  • Supporting day-to-day operations of large GPU clusters for foundation model labs, triaging node failures, job disruptions, and customer escalations alongside senior engineers.
  • Visiting customer sites in the Bay Area twice a week to work alongside their infrastructure and research teams, gather feedback, and resolve issues in person.
  • Building and extending Konduktor's health controller to detect GPU, network, and node faults, then automatically cordon, drain, and remediate affected machines.
  • Designing and shipping TypeScript front ends that give our team and our customers real-time visibility into cluster health, job status, and fleet utilization.
  • Turning recurring operational work into automation, replacing manual runbooks with tooling and controllers.
  • Writing clear incident reports and runbooks, and feeding what you learn from production back into the platform.

What we need to see

  • BS in Computer Science, Computer Engineering, or a related field, or equivalent practical experience (internships, research, open source, or a homelab count).
  • Working proficiency in TypeScript and at least one of Python or Go. You'll write front-end visualizations as well as backend tooling and automation.
  • Experience building web front ends with a modern framework (React or similar), and an instinct for turning raw telemetry into views that make problems obvious. Familiarity with Grafana or Prometheus is a plus.
  • Hands-on familiarity with Linux and the command line, including reading logs, inspecting processes, and diagnosing basic system and networking issues.
  • Foundational knowledge of Kubernetes concepts such as pods, nodes, deployments, taints, and labels, and comfort using kubectl across multiple clusters and contexts.
  • A methodical approach to troubleshooting: forming hypotheses, gathering evidence, and writing up clear findings and runbooks.
  • Strong written communication and the ability to work directly with customers' ML and infrastructure teams during incidents.
  • Ability to be onsite at customer locations in the San Francisco Bay Area twice a week.

Ways to stand out from the crowd

  • Fluency with agentic software development, such as using tools like Claude Code, Cursor, or Codex to plan, implement, test, and review changes, along with good judgment about when to trust, verify, or rewrite their output.
  • Familiarity with ML frameworks (PyTorch, JAX, Megatron/NeMo) and distributed HPC infrastructure (Slurm, Lustre, InfiniBand/RoCE), including common failure modes like NCCL hangs and straggler nodes.
  • Exposure to GPU infrastructure and NVIDIA tooling such as DCGM, nvidia-smi, and XID error interpretation.
  • Experience writing Kubernetes controllers or operators (controller-runtime, kubebuilder), especially for automated node health detection, cordoning, and remediation.
  • Experience with charting and data visualization libraries (D3, Recharts, ECharts, or similar) for real-time or time-series data.
  • Familiarity with job queueing and batch scheduling on Kubernetes (Kueue, MultiKueue, or Volcano).
  • Experience with GitOps workflows (FluxCD or Argo CD) and multi-cloud environments (GKE, OKE, EKS).
  • Contributions to open-source infrastructure projects or a portfolio of tools you've built to monitor or automate systems.

Skills

PythonTypeScriptReactKubernetesLinuxPyTorch

Similar Jobs

30

Founding Member of Techincal Staff - Agents

Relixir·Mountain View, California

2mo ago

Member of Technical Staff, FA Files

Pure Storage·Mumbai, Maharashtra

Today

Member of Technical Staff, FA Files

Pure Storage·Mumbai, Maharashtra

Today

Member of Technical Staff - Flash Array (Files)

Pure Storage·Mumbai, Maharashtra

Today

Senior Member of Technical Staff

Brainbase Labs·San Francisco, California·Onsite

1d ago

Member of Technical Staff, Corporate Engineer (IT)

Arca·New York City, New York·Onsite

1d ago

Member of Technical Staff, Infrastructure & Operations

Flox·Remote·Remote

1d ago

Member of Technical Staff

Flox·Remote·Remote

1d ago

Member of Technical Staff

Clera·San Francisco, California·Onsite

1d ago

Founding Member of Technical Staff

Clera·San Francisco, California·Onsite

1d ago

Member of Technical Staff - Simulation Researcher

Moonlake·San Francisco, California·Onsite

1d ago

Member of Technical Staff, Evaluations

XDOF·San Mateo Hybrid·Hybrid

1d ago

Member of Technical Staff (Index Engineer, Search Crawler)

Perplexity·Belgrade, Central Serbia +2

1d ago

Member of Technical Staff, Translational Science

LatchBio·San Francisco, California·Onsite

1d ago

Member of Technical Staff, Molecular Design

LatchBio·San Francisco, California·Onsite

1d ago

Member of Technical Staff, Medicinal Chemistry

LatchBio·San Francisco, California·Onsite

1d ago

Member of Technical Staff, Experimental Biology & Assay Development

LatchBio·San Francisco, California·Onsite

1d ago

Member of Technical Staff, Clinical Operations

LatchBio·San Francisco, California·Onsite

1d ago

Member of Technical Staff, Clinical Development

LatchBio·San Francisco, California·Onsite

1d ago

Member of Technical Staff, Clinical Data & Statistical Programming

LatchBio·San Francisco, California·Onsite

1d ago

Member of Technical Staff, Chemistry

LatchBio·San Francisco, California·Onsite

1d ago

Member of Technical Staff, Biostatistics & Trial Design

LatchBio·San Francisco, California·Onsite

1d ago

Member of Technical Staff - Forward Deployed Engineer

Phylo·South San Francisco, California +1·Onsite

1d ago

Member of Technical Staff, AI

Pathos·Remote·Remote

1d ago

Member of Technical Staff - Pre-Training Evals Infra (Inference)

Reflectionai·Global Office·Onsite

2d ago

Member of Technical Staff (Perplexity Frontier Research)

Perplexity·London, England +3·Hybrid

2d ago

Member of Technical Staff (Cybersecurity Solutions Architect)

Perplexity·Belgrade, Central Serbia +4·Hybrid

2d ago

Member of Technical Staff (Enterprise Security Engineer)

Perplexity·Belgrade, Central Serbia +4·Hybrid

2d ago

Member of Technical Staff – VM Product & Infra

Lyceum·Zürich, Zurich·Onsite

2d ago

Member of Technical Staff – Security

Lyceum·Zürich, Zurich·Onsite

2d ago