Lead Data Engineer (Databricks, MLops, AWS)

Role Overview 

We are seeking a Senior/Lead Platform Engineer who will take ownership of the design, implementation and operation of our core data, analytics and ML infrastructure. This role spans across platform architecture, DevSecOps, DataOps, and ML infrastructure, and requires a combination of strategic thought leadership and hands-on execution. You will build, integrate and operate platforms on AWS and Databricks, enabling scalable, secure, production-grade ML/AI solutions. 

Key Responsibilities 

• Architect and implement end-to-end data and ML platforms: data lakes, warehouses, streaming and batch pipelines, model training/deployment infrastructure, on AWS + Databricks. 

• Lead DevSecOps and DataOps practices: infrastructure as code (IaC), CI/CD pipelines for data & ML workflows, secure multi-account/multi-region cloud operations. 

• Integrate AWS services (e.g., S3, Redshift, Kinesis, Lambda, EKS/ECS) with Databricks runtime, Delta Lake, Unity Catalog etc to build scalable, performant pipelines. 

• Build and operate ML infrastructure: training clusters, model versioning, MLOps toolchain (e.g., MLflow), model monitoring and observability, automatic retraining workflows. 

• Establish data governance, lineage, quality, observability standards across data pipelines and ML workflows. 

• Mentor engineering teams, define architectural best practices and guide implementation of high-scale data/ML systems. 

• Optimize system performance, cost and scalability; diagnose and resolve large-scale production issues. 

• Continuously evaluate new tools and technologies in the areas of cloud, data platform, DevSecOps, ML infrastructure and apply them to drive innovation.