Offer summary

Qualifications:

Bachelor's degree in IT or Computer Science, 2-5 years of experience in data engineering, Strong expertise in MS SQL and PostgreSQL, Proficiency in Python and Spark.

Key responsabilities:

Design and implement data pipelines using Azure Data Factory

Develop and manage ETL processes within Azure Databricks

Job description

Job brief

We are hiring a Data Engineer for a leading company in the financial services industry, committed to delivering innovative solutions and driving data-driven decision-making. This is a unique opportunity to join a fast-paced and dynamic team focused on leveraging advanced technologies to support organizational growth and excellence in data management.

Responsibilities

Data Integration and Orchestration: Design and implement data pipelines using Azure Data Factory to move data from various sources into Azure storage solutions.
ETL Processing and Transformation: Develop and manage efficient ETL processes within Azure Databricks clusters using Spark (PySpark/Scala).
Performance Optimization: Continuously optimize data pipelines and storage solutions for efficiency and scalability.
CI/CD Pipelines: Implement and enhance CI/CD pipelines for data infrastructure using Azure DevOps services, automating testing and deployment.
Azure Infrastructure Management: Provision, configure, and manage Azure cloud infrastructure for data pipelines and applications.
Security and Compliance: Ensure secure deployment of data pipelines, adhering to data privacy and governance standards.
Monitoring and Troubleshooting: Set up monitoring tools to track pipeline health and resolve infrastructure issues.
Scripting: Automate tasks using Azure Functions and Azure CLI for efficient pipeline management.

Requirements

Bachelors degree in IT, Computer Science, or a related field.
2-5 years of experience in data engineering or a related role.
Strong expertise in MS SQL, PostgreSQL, CDC, and data streaming applications (e.g., Apache Kafka, Qlik Replicate).
Proficiency in Python and familiarity with Spark (PySpark/Scala) for data processing.
Experience with Azure DevOps, GitHub tools, and cloud-native applications.
Understanding of data governance, cloud security, and best practices in Azure environments.
Excellent communication skills in English, with the ability to work independently in a fast-paced environment.

Benefits & Perks

Competitive salary: RM 8,000 - RM 12,000 per month, based on experience and qualifications.
Contributions to KWSP and SOCSO.
Comprehensive health insurance and medical benefits.
5-day work week and flexible working arrangements.
Opportunities for professional development and career advancement.

Required profile