Lead Data Engineer

Role and Responsibilities:

• Interface with business stakeholders, understand their data and analytics needs, establish requirement in alignment with data product owners and align on delivery plan.
• Lead discovery of the current reporting and data landscape: inventory sources, ETL processes, upstream/downstream dependencies, and existing metrics/definition
• Translate stakeholder requirements and current-state findings into a target pipeline design (batch and near-real time where applicable), including data flow diagrams, lineage, and logical/physical model
• Develop and maintain end to end data pipelines: author, tune, and optimize Python and SQL code for extraction, transformation, and loading (ETL/ELT), including incremental loads, partitioning, and CDC patterns
• Build reusable Python libraries and parameterized pipelines to accelerate delivery and ensure consistency
• Implement data quality, validation, and reconciliation processes: automated tests, reconciliation scripts, anomaly detection, and monitoring/alerting to ensure trust in the data
• Define and codify metric logic, mapping, and lineage; implement validation tests and documentation for each metric to support governance and auditability
• Own UAT for new/changed data assets: create test plans, run tests, document findings, and provide pass/fail recommendations to IT and engineering partners
• Prepare rollout materials and lead change adoption: runbooks, operational runbooks, deployment guides, knowledge transfer sessions, and training for analysts and business users
• Support table retirement and technical debt reduction by coordinating deprecation plans and ensuring no regression in reporting
• Collaborate in an Agile environment with peers and stakeholders; participate in planning, standups, and retrospectives to deliver iterative value
• Rigorously create and maintain document such that anyone can understand the design, logic, code, acceptance testing, data quality testing, and outcomes. Document prototypes in detail, including learnings of what doesn't work.
• Effectively communicate analysis, reports, project status, milestones etc. in coordination with data product owners, program managers and scrum maters.
• Implement straight through, exception-based data product designs meeting data governance standards, including reporting. Perform necessary data quality checking tasks, prepare data quality reports, establish necessary data security measures to prepare model ready data.

Candidate Profile:

• Bachelor's/master's degree in computer science/engineering, operations research, or related analytics areas
• Deep expertise in Python and SQL, including complex query authoring and performance tuning.
• Hands on experience building and maintaining end to end data pipelines (batch and near real time).
• Proficient in data transformation/ETL/ELT: incremental loads, partitioning, CDC.
• Metric development: define logic, lineage, and validation tests.
• Implement data quality checks, automated tests, and reconciliation scripts.
• Knowledge of data warehousing and logical modeling
• Highly proficient in Microsoft Word, Excel, PowerPoint
• Exposure to Agile planning concepts: Kanban, Sprint.
• Able to work in a fast paced and continuously evolving environment with the ability to take-up challenging tasks.
• Can understand cross cultural differences and can collaborate with clients across the globe.
• Outstanding written and verbal communication skills