- Location
- San Francisco, CA · Toronto, ON · Pittsburgh, PA · Remote US & Canada
- Workplace
- Hybrid, Remote
- Type
- Full-time
- Department
- Software
- Seniority
- Manager
- Source
- Lever
Description
You will…
-
Lead and grow the ML Dataset Platform team as both its technical lead and people manager: set the technical direction, define the roadmap, hire, and mentor engineers.
-
Define the APIs, tools and storage formats for Waabi's ML datasets, balancing maximal reuse of data and tooling against the specific needs of each training workload.
-
Build the tooling to manage datasets at scale, including versioning and efficient incremental updates without full rebuilds.
-
Design the APIs that let teams express training-time transformations along the continuum between fully reproducible ahead-of-time ETL and dynamic transformation within data loaders.
-
Own the common dataloading tools used across workloads, including observability into bottlenecks and the throughput of training jobs.
-
Streamline data access for multi-cloud training so that datasets are available as needed with predictable cost and latency.
-
Make data lineage a first-class, fully automatic property of every dataset, from raw data through to the model trained on them.
-
Partner with the ML Training Platform team on the boundary between data and training infrastructure, and with the ML development teams on transitioning shared-dataset tooling into the new platform.
-
Drive adoption: work directly with model teams to migrate their training data onto the platform and feed what you learn back into your team’s roadmap.
Qualifications:
-
Deep ML dataset experience: You have designed and built large-scale data infrastructure for ML training (storage formats, dataset versioning, ETL, dataloading) and understand the performance characteristics of petabyte-scale sensor or multimodal data.
-
Strong software engineering and implementation: You have very strong Python skills and are comfortable in PyTorch training loops; you have strong software-engineering fundamentals and experience with distributed systems, cloud storage, and data processing frameworks (e.g., Spark, Ray, Beam).
-
Proven technical leadership: You have led a platform or infrastructure effort used by multiple teams, defined its APIs, and driven adoption across an organization.
-
People leadership: You have managed or tech-led a team of engineers, including hiring and mentoring, and can balance hands-on technical contribution with roadmap ownership and stakeholder management.
-
Team player: You communicate clearly with researchers and engineers, build consensus across teams with different needs, and deliver in a fast-paced environment.
Bonus:
-
Experience with ML data infrastructure in autonomous driving or robotics (multi-sensor logs, LiDAR, camera, HD maps).
-
Experience with columnar and tensor storage formats (e.g., Parquet, Lance, WebDataset, Zarr) and dataset versioning tools.
-
Experience with multi-cloud or hybrid training environments and the data-movement challenges that come with them.
-
Experience with data lineage, metadata catalogs, or feature-store systems.