Senior Engineer - IT

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks and PySpark
• Build robust ETL/ELT workflows for batch and near real‑time data processing
• Write optimized and efficient SQL queries for data transformation, validation, and analysis
• Work with large-scale distributed datasets using Spark-based processing frameworks
• Perform data ingestion from multiple sources (APIs, files, databases) into data lake/lakehouse architecture
• Ensure data quality, integrity, and consistency through validations and monitoring
• Optimize pipeline performance and cost (partitioning, caching, query tuning, etc.)
• Collaborate with cross-functional teams (Data Ops, Business, Analytics) to deliver data solutions
• Troubleshoot pipeline failures and implement proactive monitoring and alerting
• Maintain proper documentation for data models, transformations, and workflows

Required Skills & Experience

• Strong hands-on experience in:
  • Databricks (Lakehouse platform, notebooks, jobs, clusters)
  • PySpark / Apache Spark
  • Advanced SQL (joins, window functions, performance tuning)
• Solid understanding of ETL concepts, data warehousing, and data modeling
• Experience building scalable data pipelines for structured and semi-structured data
• Familiarity with data lake / delta lake architecture
• Good understanding of performance optimization techniques in Spark and SQL
• Experience with version control (Git) and deployment practices
• Strong problem-solving and analytical skills

Good to Have

• Experience with cloud platforms (AWS preferably)
• Exposure to CI/CD pipelines (Jenkins)
• Knowledge of Delta Lake, Unity Catalog, or similar governance tools
• Understanding of data quality frameworks / monitoring tools
• Experience working in Agile / Scrum teams