Senior Data Engineer

Job description

A leading organization is seeking a Senior Data Engineer to design and develop cloud-based data solutions using Python, SQL, Spark, and Microsoft Azure technologies. The Senior Data Engineer will build scalable data pipelines, support modern data architectures, and help deliver reliable data solutions for analytics and artificial intelligence initiatives.

The ideal candidate has approximately 5 to 7 years of relevant experience and a strong foundation in cloud data engineering, distributed data processing, and production-grade pipeline development. Experience with generative AI, retrieval-augmented generation (RAG), agentic AI, or related technologies is preferred.

Responsibilities

• Design and develop scalable cloud-based data engineering solutions.
• Build, enhance, and support production data pipelines and ETL/ELT workflows.
• Develop data processing solutions using Python, SQL, PySpark, and Apache Spark.
• Work with Microsoft Azure data technologies, including Azure Databricks, Azure Data Factory, ADLS Gen2, Synapse Analytics, or Microsoft Fabric.
• Support data modeling, distributed computing, and lakehouse architecture initiatives.
• Work with SQL and NoSQL data stores to support data processing and application requirements.
• Implement testing, monitoring, error handling, and operational support for production data pipelines.
• Apply Git, CI/CD, and cloud DevOps practices throughout data solution development.
• Collaborate across technical teams to troubleshoot issues and deliver reliable data solutions.
• Contribute to data solutions that support artificial intelligence use cases.

Required experience and skills

• Approximately 5 to 7 years of relevant experience.
• Strong experience designing and developing cloud-based data engineering solutions.
• Hands-on experience with Python, SQL, PySpark, and Apache Spark.
• Experience with Microsoft Azure data services such as Azure Databricks, Azure Data Factory, ADLS Gen2, Synapse Analytics, or Microsoft Fabric.
• Strong understanding of ETL/ELT, data modeling, distributed computing, and lakehouse architectures.
• Experience working with SQL and NoSQL data stores.
• Experience building production-grade data pipelines with testing, monitoring, error handling, and operational support.
• Working knowledge of Git, CI/CD, and cloud DevOps practices.
• Strong analytical, troubleshooting, communication, and collaboration skills.
• Experience with artificial intelligence technologies.
• Experience with generative AI, RAG, agentic AI, or enterprise search solutions is preferred.
• Familiarity with LLM concepts such as embeddings, vector search, prompt engineering, grounding, evaluation, and model inference is preferred.
• Familiarity with Kafka, Event Hubs, or similar streaming technologies is preferred.
• Experience with Docker, Kubernetes or AKS, APIs, and microservices is preferred.
• Experience with data governance, metadata management, data lineage, and access controls is preferred.
• Relevant Microsoft Azure certifications are a plus.