Data Engineer

Aboitiz Data Innovation (ADI) is one of the leading up and coming start-ups in the field of Data Science and Artificial Intelligence. We believe data can drive change for a better world by advancing businesses across industries and communities. We are seeking talented and motivated Data Engineer to join our team, who are passionate about defining and implementing information management solutions to address operational needs within businesses.

Responsibilities:

• Move data from various sources to AWS S3 (data landing zone) using tools such as AWS Database Management Service (DMS), Python, or PySpark. Jobs will be orchestrated using Airflow or run in Databricks.
• Ingest raw data into the Data Lake (RAW zone) for processing.
• Prepare data in the Data Lake through the TRUSTED and CURATED zones using DBT and SQL to ensure data quality and readiness for analysis.
• Design and implement data models that meet business requirements and optimize performance.
• Work closely with data scientists, analysts, and other stakeholders to identify data requirements and provide solutions.
• Monitor data pipelines and workflows, troubleshoot issues, and optimize performance for scalability and reliability.

Requirements:

• Bachelor’s degree in Computer Science, Engineering, or a related field.
• Proven experience as a Data Engineer or in a similar role.
• Strong analytical and problem-solving skills.
• Proficient in Python and SQL.
• Experience with PySpark, DBT, and PyDeequ.
• Familiarity with Databricks.
• Strong knowledge of AWS services, including Database Management Service and S3.
• Experience with Apache Airflow for workflow management.
• Understanding of Data Lake, Data Warehouse, Data Modeling, and Medallion Architecture.
• Experience with APIs/Web Services, API Gateway, and AWS Lambda.
• Knowledge of Docker and EKS (Elastic Kubernetes Service).
• Familiarity with Poetry for Python dependency management.
• Excellent communication skills and ability to work collaboratively in a team environment.