Data Engineer
Asia Select Inc.
·Today
- Location
- PH
- Workplace
- Hybrid
- Type
- Full-time
- Department
- IT
- Seniority
- Mid
- Experience
- 3+ years
- Closing date
- Today
- Source
- Vincere
Description
JOB RESPONSIBILITIES:
- Design and develop data pipelines that will ingest the data from the respective source systems into the data warehouse
- Support and optimize existing data pipelines to ensure that the downstream SLAs are met
- Responsible in attending agile ceremonies as part of the Agile Tribe set-up and delivering the different demands of the Tribe related to data ingestions to the data warehouse platform
- Ensures data quality within the data warehouse tables
- Ensures the data engineering standards and practices are followed like code version history, readability, functionality, extendibility, test runs, deployments, and post deployment support
- Supports L2 Data Operations team and act as L3 support
- Manage data storage, computing unit requirements, and data security within the data warehouse
- Gather data requirements coming from Tribe demands
- Supports other data engineers on other Tribes or domains if necessary
- Demonstrate technical excellence when it comes to the data warehouse platform
- Supports stakeholders of the data warehouse platform by solving data related issues and conducting data investigations
- Data Delivery Documentations – Data Dictionary, Data Mapping Documents, Data Models
- Optimize Data Pipelines
- Project delivery and operations support
JOB QUALIFICATION:
- At least has 3 to 6 years of experience in IT project delivery or data development delivery particularly in Data Warehousing or Business Intelligence.
- Graduate of any IT related courses like BS Computer Science, Information Technology or the likes
- Strong SQL and ETL development experience, preferably related to Data Warehousing, Business Intelligence, or other Data Delivery projects.
- Proficiency in optimizing data pipelines in order to meet SLA requirements and save cost
- Experience in Big Data, Hadoop, EMR, Hive, and cloud environments is a plus
- Proficiency in Python, R, Java, Scala is a plus
- Proficiency in data modeling, data integrations, data quality, and data architecture is a plus
- AWS data integrations (Glue, Athena)
- Spark, Databricks
- Delta lake, Data Lake
- Kafka, Confluent, Real-time processing, Data Streaming
- Azure Fabric (experience in ETL, Azure Data Factory, Azure blob storage)
- Python (numpy, pandas, sklearn) works with PySpark and Scala technology
- MustApache Airflow for Orchestration, can work with Azure Fabric pipelines and Azure DevOp
- Must know these DE frameworks: Data Ingestion and working with API calls (see Azure Key vault), Data Transformation (table joins, aggregations, CTEs)
Work Setup: Hybrid: 1-2x a week onsite