RoCE Network Engineer (GPU Infrastructure)

Job Summary:

No two days are the same at Microsoft. We engage in solving challenging and complex problems, we collaborate and share learnings with each other and innovate on solutions to enable our customers make the most of Microsoft products. We are rapidly expanding, and we are looking for talented people with a growth mindset, a passion for solving complex issues and obsession to customer happiness. Besides technical talent, we also look for engineers with a run for the fire behavior - people who thrive under difficult and challenging circumstances, who loves to get involved in difficult situations and at the same time learn.

Must Have skils

RoCEv2/RDMA , NVIDIA networking (Spectrum-X,ConnectX,Bluefild), Congestion Control, Cisco , Nexus , Juniper , TCP/IP

Requirements

•   Bachelor's degree in Computer Science, Electrical Engineering or related field, or equivalent experience.

•    5+ years in data center networking, with 3+ years hands-on with RDMA/RoCEv2 in production.

•    Deep knowledge of lossless Ethernet: PFC (802.1Qbb), ECN, DCQCN, ETS, DSCP/CoS marking and buffer management.

•    Strong L2/L3 fundamentals: BGP, ECMP, VXLAN/EVPN, LACP, MTU and jumbo frames.

•    Experience with leaf-spine Clos fabrics at scale (1,000+ ports).

•    Hands-on with NVIDIA/Mellanox ConnectX NICs and tools: mlnx_qos, mlxconfig, ethtool, rdma and ibv_devinfo.

•    Proven troubleshooting of RDMA performance issues using perftest, packet captures and switch counters.

•     Linux administration skills (RHEL/Ubuntu), including driver and firmware (MLNX_OFED / DOCA) management.

•      Scripting in Python or Bash for automation and diagnostics.

•     Clear written and verbal communication for incident updates and RCAs.

Preferred qualifications

•     Experience with NVIDIA Spectrum-X (Spectrum-4 switches + BlueField-3 SuperNICs), adaptive routing and congestion control.

•    Background running GPU clusters (DGX/HGX H100, H200, B200, GB200, GB300) for AI training workloads.

•     Familiarity with NCCL internals, GPUDirect RDMA, collective communication patterns and rail-optimized topologies.

•     Working knowledge of InfiniBand and UFM, and the trade-offs between IB and RoCE fabrics.

•      Experience with Ultra Ethernet Consortium (UEC) concepts or emerging AI Ethernet standards.

•      Network automation and observability: Ansible, NetBox, gNMI/OpenConfig, Prometheus, Grafana.

•      Prior work in managed services or LiveSite operations for hyperscale customers.

•      Certifications such as NVIDIA Certified Professional (InfiniBand/Ethernet), CCNP/CCIE Data Center or Arista ACE.

Responsibilities

• Build leaf-spine and rail-optimized back-end fabrics for GPU clusters, including port maps, cabling plans and switch configs on NVIDIA Spectrum-X, Arista and Cisco platforms.
• Configure and tune PFC, ECN, ETS and DSCP-to-priority mapping, plus switch buffer profiles, xoff/xon thresholds and headroom per port speed and cable length.
• Optimize DCQCN parameters (Kmin/Kmax, Pmax, CNP interval), enable adaptive routing and packet spraying, and apply PFC watchdog to prevent pause storms and deadlocks.
• Manage ConnectX-7 / BlueField-3 firmware and MLNX_OFED / DOCA drivers at scale, and validate RoCE mode, trust DSCP, MTU, PCIe/NUMA affinity and GPUDirect RDMA paths.
• Isolate QP errors, retransmits and out-of-sequence drops using NIC counters, switch telemetry and RoCEv2 packet captures (UDP 4791), and diagnose optics faults via FEC, BER and DOM data.
• Run perftest, NCCL all-reduce and all-to-all tests for cluster acceptance, fix ECMP imbalance and bottlenecks, and tune NCCL variables for RoCE.
• Track PFC pauses, ECN marks, CNP rates, buffer occupancy and link errors through streaming telemetry, Prometheus and Grafana, with auto-drain of bad links and nodes.
• Use Python, Ansible and switch APIs (gNMI/REST) for config deployment, health checks, ZTP and OS upgrades with pre/post validation under change management.
• Design and deploy RoCEv2 back-end fabrics (leaf-spine, rail-optimized) for GPU clusters from hundreds to thousands of GPUs.
• Configure host-side RDMA on ConnectX-7 / BlueField-3 NICs (QoS trust mode, PFC priorities, CNP settings, MTU).
• Act as escalation point for RoCE fabric incidents: link flaps, PFC storms, pause-frame deadlocks, congestion hotspots and NCCL slowdowns.
• Write and maintain runbooks, SOPs and troubleshooting guide for operations team.
• Participate in on-call rotation and meet incident SLAs for customer-facing clusters.
• Drive rootcause analysis and write RCAs and post-incident actions.
• Mentor junior engineers and contribute to internal GPU networking training.