Senior Data Engineer (Databricks + PySpark + ADF + Unity Catalog)

Job Description

We are seeking a highly skilled Senior Data Engineer with hands-on experience in Databricks, Unity Catalog, and Azure Data Factory (ADF) to support a large-scale enterprise data modernization initiative. The ideal candidate will be responsible for designing, developing, and maintaining scalable data pipelines, enabling secure data governance, and supporting cloud-based analytics solutions in Azure.

 

This role will collaborate closely with architects, business stakeholders, and cross-functional engineering teams to build reliable, high-performance data platforms and accelerate data-driven decision-making.

Responsibilities

Your Impact

• Design, develop, and optimize ETL/ELT pipelines using Azure Data Factory and Databricks.
• Build and maintain scalable data ingestion frameworks for structured and unstructured data sources.
• Develop data transformation processes using PySpark, Spark SQL, and Databricks notebooks.
• Implement and manage Unity Catalog for enterprise data governance, security, and access controls.
• Create and optimize Delta Lake architectures for analytics and reporting workloads.
• Monitor pipeline performance and troubleshoot production issues.
• Collaborate with data architects to implement enterprise data models and best practices.
• Ensure data quality, integrity, security, and compliance standards are met.
• Support CI/CD deployment processes and DevOps practices for data engineering solutions.
• Work closely with business and analytics teams to understand requirements and deliver scalable solutions.

Qualifications

Your Skills & Experience

• 4-7+ years of experience in Data Engineering or Big Data development.
• Strong hands-on experience with Azure Databricks.
• Proficiency in PySpark, Spark SQL, and distributed data processing.
• Experience implementing and managing Unity Catalog.
• Strong experience with Azure Data Factory (ADF).
• Experience working with Delta Lake and modern Lakehouse architectures.
• Solid understanding of data modeling, ETL/ELT, and data warehousing concepts.
• Experience with Azure cloud services and ecosystem.
• Proficiency with Git, version control, and CI/CD processes.
• Strong analytical and problem-solving skills.
• Excellent communication and stakeholder management abilities.

Preferred Qualifications

• Experience with Azure Synapse Analytics.
• Familiarity with Collibra or other data governance platforms.
• Experience with Infrastructure as Code (Terraform, ARM Templates).
• Databricks Certification (Data Engineer Associate/Professional).
• Microsoft Azure Data Engineer Associate Certification.

Nice to Have

• Experience supporting enterprise-scale data migration projects.
• Knowledge of healthcare, retail, or consumer products data environments.
• Exposure to real-time data processing and streaming technologies.

Additional Information

• An inclusive workplace that promotes diversity and collaboration.
• Access to ongoing learning and development opportunities.
• Competitive compensation and benefits package.
• Flexibility to support work-life balance.
• Comprehensive health benefits for you and your family.
• Generous paid leave and holidays.
• Wellness program and employee assistance.

As part of our dedication to an inclusive and diverse workforce, Publicis Sapient is committed to Equal Employment Opportunity without regard for race, color, national origin, ethnicity, gender, protected veteran status, disability, sexual orientation, gender identity, or religion. We are also committed to providing reasonable accommodations for qualified individuals with disabilities and disabled veterans in our job application procedures. If you need assistance or an accommodation due to a disability, you may contact us at [email protected]