Healthcare Data Engineer, Incremental Processing

Job description

A leading healthcare organization is seeking a Healthcare Data Engineer to evaluate and modernize provider data pipelines for incremental processing. The Data Engineer will assess existing architecture, data lineage, source dependencies, downstream usage, and data quality to determine how full-refresh workloads can transition to a more scalable incremental processing model.

The position requires advanced SQL, ETL engineering, change data capture, healthcare data experience, and the ability to turn technical analysis into clear architecture and implementation recommendations.

Responsibilities

• Analyze provider data architecture, end-to-end data flows, dependencies, and current processing patterns.
• Determine the technical feasibility of transitioning provider datasets from full-refresh to incremental processing.
• Evaluate source systems for change data capture (CDC) capabilities, timestamps, and other mechanisms that support incremental loading.
• Trace data lineage and document source-to-target relationships across healthcare data pipelines.
• Assess potential impacts of processing changes on downstream applications, analytics, and reporting.
• Design recommended incremental processing patterns and implementation approaches.
• Develop reconciliation and validation strategies for comparing incremental outputs against established full-refresh results.
• Collaborate with data architecture and modeling teams on future-state data designs.
• Identify technical debt, pipeline performance bottlenecks, and opportunities to improve scalability and maintainability.
• Perform data profiling, troubleshooting, and data quality analysis.
• Produce architecture diagrams, technical documentation, findings, and actionable engineering recommendations.
• Participate in Agile planning, backlog refinement, technical design reviews, and collaborative engineering sessions.

Required experience and skills

• Advanced SQL development and data analysis skills.
• Experience designing and implementing incremental data processing and CDC solutions.
• Strong knowledge of ETL pipeline development and performance optimization.
• Experience conducting data lineage analysis and documenting source-to-target mappings.
• Experience working with large-scale healthcare datasets.
• Familiarity with healthcare data domains such as provider, claims, enrollment, or master data.
• Experience with Databricks, Spark, or comparable distributed data processing platforms.
• Strong data profiling, troubleshooting, reconciliation, and data quality analysis capabilities.
• Experience evaluating downstream dependencies for analytics and reporting platforms.
• Ability to create technical documentation, architecture diagrams, and implementation recommendations.
• Experience supporting healthcare analytics and reporting environments.
• Experience working within Agile teams and using Jira for delivery management.