Senior Data Engineer (Azure Data Factory/Databricks/PySpark)

Company Description

Publicis Sapient is a digital transformation partner helping established organizations get to their future, digitally enabled state, both in the way they work and the way they serve their customers. We help unlock value through a start-up mindset and modern methods, fusing strategy, consulting, and customer experience with agile engineering and problem-solving creativity. United by our core values and our purpose of helping people thrive in the brave pursuit of next, our 20,000+ people in 53 offices around the world combine experience across technology, data sciences, consulting, and customer obsession to accelerate our clients’ businesses through designing the products and services their customers truly value.

Job Description

We’re looking for a Senior Data Engineer with strong hands-on expertise in building scalable data pipelines and cloud-native data solutions on Azure. This role focuses on designing and implementing real production systems using Azure Data Factory, Azure Databricks, and modern big data technologies. You will work across distributed data platforms, integrating cloud and on-premises environments, and delivering robust, enterprise-grade data solutions aligned with industry best practices.

 

Responsibilities

Your Impact

• Design and build scalable data pipelines using Azure Data Factory (ADF) and Azure Databricks (ADB)
• Develop PySpark-based transformation logic for large-scale data processing, including joins, aggregations, and window functions
• Architect and implement hybrid data integrations between cloud and on-premises systems
• Enable secure connectivity from Databricks to on-prem databases using enterprise-grade patterns
• Build and optimize data models across Cosmos DB and Snowflake for different workloads
• Implement monitoring, logging, and error-handling mechanisms to ensure reliability and performance
• Collaborate with cross-functional teams to define standards, patterns, and best practices for data engineering solutions

 

 

Qualifications

 

Skills & Experience

• Strong experience building data pipelines with Azure Data Factory and Azure Databricks
• Hands-on expertise in PySpark for distributed data processing
• Advanced knowledge of SQL, including complex joins and performance optimization
• Experience working with Cosmos DB and Snowflake
• Solid programming skills in Python, Scala, or Java
• Experience with Kafka or event-driven architectures
• Strong understanding of data architecture and distributed systems
• Proven ability to deliver production-grade data solutions in complex environments

Technical Requirements 

Candidates must demonstrate solid, hands-on knowledge across the following three areas:

1. PySpark (Production-Level Coding)

• DataFrame transformations: select, filter, groupBy, agg, withColumn
• Window functions: rank, row_number, lag, lead
• Joins: inner, left, broadcast joins and usage scenarios
• Reading/writing data: Parquet, Delta, CSV
• UDFs: syntax, registration, and performance tradeoffs
• Ability to produce production-ready code without pseudo-code

2. Azure Databricks (Architecture & Platform Expertise)

• Cluster types, auto-scaling, and cluster policies
• Notebook orchestration, workflows, and job scheduling
• Delta Lake: ACID, schema evolution, time travel, OPTIMIZE, VACUUM
• Unity Catalog: governance, lineage, and access control
• Integration with Azure Data Factory

3. Hybrid Data Architecture (On-Prem to Cloud Integration)

• JDBC/ODBC connectivity from Databricks
• Secure credential management with Azure Key Vault and secret scopes
• Network architecture: VNet injection, private endpoints, Self-Hosted IR
• End-to-end pipeline design from on-prem to cloud
• Performance optimization and error handling in hybrid environments

Set Yourself Apart With

• Experience designing end-to-end data architectures in Azure ecosystems
• Knowledge of Delta Lake features (ACID transactions, schema evolution, time travel)
• Experience with Databricks Workflows and orchestration
• Understanding of data governance (Unity Catalog)
• Experience with hybrid cloud/on-prem integration patterns
• Exposure to performance optimization at scale

 

 

Additional Information

• An inclusive workplace that promotes diversity and collaboration.
• Access to ongoing learning and development opportunities.
• Competitive compensation and benefits package.
• Flexibility to support work-life balance.
• Comprehensive health benefits for you and your family.
• Generous paid leave and holidays.
• Wellness program and employee assistance.

As part of our dedication to an inclusive and diverse workforce, Publicis Sapient is committed to Equal Employment Opportunity without regard for race, color, national origin, ethnicity, gender, protected veteran status, disability, sexual orientation, gender identity, or religion. We are also committed to providing reasonable accommodations for qualified individuals with disabilities and disabled veterans in our job application procedures. If you need assistance or an accommodation due to a disability, you may contact us at [email protected]