Hiring.Camp

Inference Software Engineer

Etchedai

·

Jan 16, 2025

Salary
$2k+
Location
Cupertino, CA · Cupertino, California, United States
Workplace
Remote
Department
Software
Source
Greenhouse

Description

About Etched

Etched is building AI chips that are hard-coded for individual model architectures. Our first product (Sohu) only supports transformers, but has an order of magnitude more throughput and lower latency than a B200. With Etched ASICs, you can build products that would be impossible with GPUs, like real-time video generation models and extremely deep & parallel chain-of-thought reasoning agents.

Key responsibilities

  • Contribute to the architecture and design of the Sohu host software stack
  • Implement high-performance, modular code across the complete Etched software stack, consisting of a mix of Rust, C++ and Python.
  • Interface with firmware and drivers teams delivering highest-performance HW/SW stack.
  • Work with AI model researchers and product-facing teams building out the Etched serving front-end.

Representative projects

  • Build scheduling logic for handling continuous batching and real time inference
  • Implement inference-time acceleration techniques such as speculative decoding, tree search, KV cache sharing, etc.
  • Implement distributed networking primitives for efficient multi-server inference

You may be a good fit if you have

  • Experience with C++ and Python
  • Familiarity with transformer model architectures and inference serving stacks (vLLM, SGLang, etc.) or experience working in distributed inference/training environments
  • Experience working cross-functionally in large software and hardware organizations

Strong candidates may also have

  • Experience with Rust
  • Familiarity with GPU kernels, the CUDA compilation stack and related tools, or other hardware accelerators
  • Understanding of distributed systems, networking, and parallel programming

Benefits

  • Full medical, dental, and vision packages, with 100% of premium covered
  • Housing subsidy of $2,000/month for those living within walking distance of the office
  • Daily lunch and dinner in our office
  • Relocation support for those moving to Cupertino

How we’re different

Etched believes in the Bitter Lesson. We think most of the progress in the AI field has come from using more FLOPs to train and run models, and the best way to get more FLOPs is to build model-specific hardware. Larger and larger training runs encourage companies to consolidate around fewer model architectures, which creates a market for single-model ASICs.

We are a fully in-person team in Cupertino, and greatly value engineering skills. We do not have boundaries between engineering and research, and we expect all of our technical staff to contribute to both as needed.

Skills

PythonRust

Similar Jobs

30

Software Engineer, Inference

Sierra·San Francisco, California·Onsite

1mo ago

Software Engineer, Inference

Thinkingmachines·New York City, New York +1·Onsite

1mo ago

Software Engineer, Inference

Pulse·San Francisco, California·Onsite

1y+ ago

Inference Software Engineer

Etched·San Jose, California·Remote

1y+ ago

Software Engineer, Inference

Pulse·San Francisco, California·Onsite

Software Engineer - Inference Product

Baseten·New York City, New York +1·Hybrid

2d ago

Senior Software Engineer, Inference Performance

Tensordyne·Berlin, State of Berlin +1

1w ago

Senior/Staff Back-End Software Engineer, Inference

Cohere·Montréal, Quebec +4·Remote

1mo ago

Software Engineer- Inference Platform

Baseten·Montréal, Quebec +4·Hybrid

1mo ago

Software Engineer- Inference Performance

Baseten·Montréal, Quebec +4·Hybrid

1mo ago

Senior Software Engineer, Inference

Hewlett Packard Enterprise (HP)·Denver, Colorado +2·Remote, Hybrid

2mo ago

Senior Software Engineer, Inference

Hewlett Packard Enterprise (HP)·Denver, Colorado +2·Remote, Hybrid

2mo ago

Senior Software Engineer, Inference

Hewlett Packard Enterprise (HP)·Denver, Colorado +2·Remote, Hybrid

2mo ago

Senior Lead Software Engineer- Inference platform engineer

JP Morgan Chase·Seattle, Washington

2mo ago

Senior Lead Software Engineer- Inference platform engineer

JP Morgan Chase·Seattle, Washington

2mo ago

Software Engineer, Inference (AI Data Engineering)

Spacex·Palo Alto, California

3mo ago

Staff + Senior Software Engineer, Inference

Anthropic·Ontario, California·Remote

3mo ago

Software Engineer, Inference Runtime

LM Studio·New York City, New York·Hybrid

3mo ago

Senior Software Engineer, Inference

Pika·Palo Alto, California·Onsite

5mo ago

Staff+ Software Engineer, Inference Runtime

Anthropic·New York, US +1·Onsite

5mo ago

Staff + Senior Software Engineer, Inference

Anthropic·San Francisco, California·Onsite

5mo ago

Staff Software Engineer, Inference

Coreweave·Bellevue, Washington +1

Staff Software Engineer, Inference Infrastructure

Cohere·London, England +4·Remote, Hybrid

10mo ago

Backend Software Engineer, ML Inference

Prior Labs·Berlin, State of Berlin +2·Onsite

3d ago

Senior Software Engineer - Akamai Inference Cloud (Remote/Poland)

Akamai·Poland·Remote

3d ago

Principal Software Engineer - Akamai Inference Cloud - Remote/Poland

Akamai·Poland·Remote

3d ago

Sr. Software Engineer, Training/Inference (C++)

xAI·Palo Alto, California +1

2w ago

Senior Software Engineer — ML Inference & Numerics

Tensordyne·Berlin, State of Berlin +1

2w ago

Staff Software Engineer, LLM Inference Platform

"Zenlayer, Inc"

3w ago

Software Engineer Intern, Inference

Rhoda Ai·Mountain View, California

1mo ago