Databricks Engineer

Job description

A leading organization is seeking an experienced Databricks Engineer to design, build, and guide the development of an enterprise data platform based on Databricks Lakehouse architecture. The Databricks Engineer will combine hands-on data engineering with platform architecture, governance, performance optimization, and technical leadership.

This position requires current development experience. The Databricks Engineer must be comfortable writing, debugging, reviewing, and optimizing production PySpark and SQL code in addition to making broader architectural decisions. The work includes establishing engineering standards, designing scalable data patterns, implementing governance and security controls, and helping engineering teams adopt consistent development practices.

Responsibilities

• Design scalable Databricks Lakehouse architectures using medallion architecture, Delta Lake, data mesh, and modern data engineering patterns.
• Remain hands-on with PySpark and SQL development, debugging, query optimization, and production data pipelines.
• Lead technical code reviews and provide guidance on code quality, performance, maintainability, and engineering standards.
• Define Databricks platform architecture, including workspace design, storage patterns, compute strategy, and enterprise integrations.
• Design and implement governance using Unity Catalog, including catalogs, schemas, permissions, data lineage, identity controls, and secure data access.
• Develop CI/CD and automation approaches for Databricks using Git, automated testing, deployment pipelines, and infrastructure as code.
• Establish enterprise data modeling standards, including schema patterns, slowly changing dimensions, change data capture, and data quality practices.
• Optimize Databricks workloads through partitioning, Z-ordering, liquid clustering, caching, materialized views, autoscaling, and appropriate compute selection.
• Design integration patterns for APIs, databases, streaming sources, analytics platforms, and cloud services.
• Support workload isolation, capacity planning, platform monitoring, and cost optimization.
• Partner with engineering, data, and business teams to translate requirements into practical technical designs.
• Mentor engineers on Databricks development, architectural patterns, troubleshooting, and engineering best practices.
• Evaluate evolving Databricks capabilities across data engineering, streaming, analytics, and AI and machine learning use cases.
• Maintain technical documentation and architectural decision records to support knowledge sharing and platform consistency.

Required experience and skills

• 3+ years of experience in data engineering, data architecture, or data platform engineering.
• 2+ years of experience focused on Databricks platform architecture.
• Current hands-on development experience with production PySpark and SQL.
• Advanced Python, PySpark, and SQL skills, including complex transformations, debugging, query optimization, and data pipeline development.
• Experience conducting code reviews and explaining unfamiliar code from both technical and business perspectives.
• Strong experience with Databricks technologies, including Unity Catalog, Delta Lake, Delta Live Tables, Workflows, SQL Warehouses, MLflow, and Databricks SQL.
• Deep understanding of Unity Catalog governance, including permissions, lineage, metastore design, catalog structures, and multi-workspace patterns.
• Experience designing enterprise-scale lakehouse architectures using medallion, data mesh, domain-driven, or data product patterns.
• Cloud architecture experience with Microsoft Azure, Amazon Web Services, or Google Cloud Platform, including storage, identity, networking, and security.
• Hands-on experience with infrastructure as code using Terraform or comparable technologies.
• Strong understanding of Git, CI/CD, automated testing, and DevOps practices for data platforms.
• Experience with data modeling, slowly changing dimensions, change data capture, and data quality frameworks.
• Experience with performance tuning, capacity planning, observability, and cost optimization for large-scale data platforms.
• Strong communication skills with the ability to explain architectural and technical concepts to both technical and non-technical stakeholders.
• Ability to provide technical mentorship, lead engineering standards, and drive alignment across teams.