M22 - Data Engineer

Overview

We are looking for a Data Engineer (Associate Consultant/Consultant) to design, develop, and maintain scalable data engineering solutions that support business intelligence, analytics, and data-driven applications. You will work closely with Product Owners, Project Managers, Data Architects, Business Analysts, and developers to build reliable data pipelines, data warehouses, and backend services while ensuring data quality, governance, and security. 

Responsibilities

• Design, develop, deploy, and maintain scalable data engineering solutions, including data pipelines, data warehouses, data lakes, operational data stores, and data marts.
• Build and optimize ETL/ELT processes for data extraction, transformation, loading, and web scraping where required.
• Develop and maintain large-scale batch and real-time data pipelines using modern data processing frameworks.
• Design and develop backend APIs and database solutions to support business applications.
• Integrate data from multiple sources while ensuring scalability, reliability, and compliance.
• Ensure data quality, governance, security, and adherence to organisational standards.
• Collaborate with Product Owners, Project Managers, Data Architects, Business Analysts, Data Analysts, and Frontend Developers to deliver scalable data solutions.
• Support the development of reports, dashboards, and data visualisations for business users.
• Provide operational support, troubleshoot incidents, and resolve service requests within agreed SLAs.
• Participate in Agile development practices, including sprint planning, code reviews, pair programming, and CI/CD.
• Maintain technical documentation and support project audits and knowledge-sharing activities.

Requirements

• Experience in data engineering, data warehousing, or business intelligence projects.
• Proficient in SQL, Python, Pandas, or similar data processing technologies.
• Hands-on experience building ETL/ELT pipelines using technologies such as SSIS, AWS Glue, AWS DMS, Lambda, ECS, EventBridge, or Spring.
• Strong knowledge of relational and NoSQL databases, including PostgreSQL, MySQL, MongoDB, Cassandra, Athena, S3, and SQLite.
• Experience with cloud platforms such as AWS, Azure, or Google Cloud.
• Good understanding of data modelling, data warehouses, data lakes, data marts, and data virtualisation.
• Familiarity with REST APIs, backend development, and web protocols.
• Knowledge of big data technologies such as Spark, Hadoop, Kafka, or RabbitMQ is an advantage.
• Experience with web scraping tools such as Selenium, BeautifulSoup, or Node.js is an advantage.
• Understanding of data governance, security, and access control best practices.
• Comfortable working in both Windows and Linux environments.
• Knowledge of system design, data structures, and algorithms.
• Strong analytical, problem-solving, communication, and stakeholder management skills.
• Experience working in an Agile development environment.