Hiring.Camp

Senior AI System Engineer - LLM Training System

Huawei Technologies Canada Co., Ltd.

·

2 weeks ago

Salary
$127k – $225k
Location
Vancouver, British Columbia, Canada
Workplace
Onsite
Type
Full-time
Department
Computing Data Application Acceleration Lab
Seniority
Mid
Experience
3+ years
Education
PhD
Source
Recruitee

Description

Huawei Canada has an immediate permanent opening for a Senior Engineer.

About the team: 

The Computing Data Application Acceleration Lab aims to create a leading global data analytics platform organized into three specialized teams using innovative programming technologies. This team focuses on full-stack innovations, including software-hardware co-design and optimizing data efficiency at both the storage and runtime layers. This team also develops next-generation GPU architecture for gaming, cloud rendering, VR/AR, and Metaverse applications. One of the goals of this lab are to enhance algorithm performance and training efficiency across industries, fostering long-term competitiveness.

About the job:

As a Senior System Engineer, you will drive core compute operator development and distributed scaling performance across foundation model pre-training and post-training workloads on Ascend NPUs. You will identify computational and communication bottlenecks, implement high-performance kernels using AscendC, Triton, and TileLang, and optimize distributed acceleration suites (MindSpeed / open-source training frameworks).

  • Ascend High-Performance Kernel Development: Design, implement, and tune core operators (FlashAttention variants, FP8/MXFP8/W4A8 GEMM, Fused RMSNorm, dynamic MoE routing/dispatch) using AscendC, with explorations in Triton (Triton-Ascend backend) and TileLang.

  • Pre-training & Post-training System Optimization: Architect, scale, and optimize distributed pre-training and RL post-training pipelines using MindSpeed (MindSpeed-LLM / MindSpeed-MM) alongside upstream frameworks (Megatron-LM / Megatron-Core, PyTorch FSDP, veRL, vLLM, Ray).

  • Distributed Parallelism & Training Efficiency: Tune multi-dimensional hybrid parallelism (TP, PP, CP, EP, ZeRO/FSDP) and RL rollout/actor pipelining on Ascend NPU clusters, achieving optimal MFU and compute-communication overlap.

  • Low-Precision & Numerical Parity: Implement low-precision training/inference schemes (FP8, MXFP8, mixed-precision), guaranteeing numerical stability, batch-invariance, and strict precision parity between training and serving runtimes.

  • Profiling & Hardware-Aware Tuning: Conduct deep performance profiling on Ascend hardware using Ascend Profiler (msprof), msprobe, and PyTorch Profiler to eliminate pipeline bubbles, AICore/AIVector stalls, and HCCL communication overhead.

  • Open Source & Ecosystem Enablement: Contribute high-performance operators, bug fixes, and acceleration features to open-source Ascend communities and upstream AI acceleration projects.

    The total target annual compensation ranges from $127,000 to $225,000 depending on education, experience, and demonstrated expertise.

Skills

PythonPyTorch

Similar Jobs

30

Senior/Principal System Validation Engineer (AI Fabric Switches)

Astera Labs·Los Angeles, California +1

Today

Sr System Development Engineer, Mfg Bring Up and Test, Edge & High Performance Accelerator Servers for AI/ML

Amazon Company

Sr. System Development Engineer, Edge & AI/ML Accelerator Team

Amazon Company

Senior System Software Engineer - AI Factory Automation

Nvidia·New York City, New York +1·Remote

1d ago

Senior System Architect Autonomous Networks for Data, Analytics and AI

Swisscom·Lausanne, Vaud +1·Hybrid

2d ago

Applied AI Health Data System Engineer-Senior Manager

Pwc·Atlanta, Georgia +57·Hybrid

1w ago

Senior System Software Architect, AI and GPU Networking

Nvidia·Jerusalem, Israel +1

1w ago

Senior System Software Architect, AI and GPU Networking

Nvidia·Illinois, US

1w ago

Applied AI Health System Engineer [Health Services-Payer] – Senior Manager

Pwc·Atlanta, Georgia +57·Hybrid

1w ago

Senior System Software Engineer, DGX Cloud AI Infrastructure

Nvidia·Santa Clara, California

2w ago

Senior System Software Engineer, DGX Cloud AI Infrastructure

Nvidia·Santa Clara, California

2w ago

Senior AI-Native Designer & Builder, Design System

Zscaler·Santa Clara, California·Remote, Hybrid

2w ago

Senior Software System Architect, AI Factory Management

Nvidia·Rotterdam, South Holland·Remote

2w ago

Senior Software System Architect, AI Factory Management

Nvidia·The Netherlands·Remote

2w ago

Senior Scientist (m/f/x) AI-Driven Optical System Development

Job Market·Jena, Thuringia +1

2w ago

Senior Technologist, System Architecture - AI Adoption & Development

Western Digital·Rochester, Minnesota·Onsite

3w ago

Lead Engineer, Senior-System Performance Engineer (NSP, AI/ML, CPU, DDR)

Qualcomm·Bengaluru, Karnataka

3w ago

Senior AI Platform/System Engineer, Coding Agents and MCP

"Pixalate, Inc."·Lahore, Punjab·Remote

4w ago

Senior Software System Architect, AI Factory Management

Nvidia·Ra'anana, Central District

1mo ago

Senior Software System Architect, AI Factory Management

Nvidia·Jerusalem, Israel

1mo ago

Senior System Software Engineer - Local AI

Nvidia·Pune, Maharashtra

1mo ago

Senior System Software Engineer - Local AI

Nvidia·Pune, Maharashtra

1mo ago

Sr Product Designer, Design System (AI)

Sinch·Stockholm, Sweden

1mo ago

Senior System Software Engineer - Local AI Automation

Nvidia·Pune, Maharashtra

1mo ago

Senior System Software Engineer - Local AI Automation

Nvidia·Pune, Maharashtra

1mo ago

Senior System Engineer - AI Program

Israel·Israel·Onsite

1mo ago

Senior System Engineer - AI Program

Kla·Israel

1mo ago

Senior System Thermal Engineer - AI & Compute Server Development

Synnex·Fremont, California +1

1mo ago

AI Model System Software Performance Optimization Engineer - Senior Engineer

Qualcomm·Bengaluru, Karnataka

1mo ago

Mid - Senior Level System Engineer (AI)

Belay Technologies·Maryland, US

2mo ago