Job description

Design, develop, and implement scalable data pipelines using Azure Data Factory (ADF) to automate data integration and transformation processes.

Utilize Azure Databricks for advanced data processing and analytics, leveraging Spark to handle large datasets efficiently.

Ensure the reliability, availability, and performance of data platforms and services.

Implement data quality and validation processes to ensure data accuracy and integrity.

Work closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions that meet business needs.

Collaborate with IT and business teams to integrate data solutions with existing systems and applications.

Optimize data processing workflows for performance and cost-efficiency.

Troubleshoot and resolve data pipeline issues in a timely manner.

Maintain comprehensive documentation of data architectures, processes, and best practices.

Stay updated with the latest industry trends and Azure technologies to continuously improve data engineering practices.

Must Have

Proven experience as a Data Engineer with a focus on Azure Data Factory and Databricks.

Strong proficiency in SQL and experience with data modeling and ETL processes.

Solid understanding of Azure ecosystem (e.g., Azure Storage, Azure SQL Database, Azure Synapse Analytics).

Familiarity with other Azure services like Azure Functions, Logic Apps, and Azure DevOps.

Experience with programming languages such as Python or Scala.

Excellent problem-solving skills and attention to detail.

Strong communication and collaboration skills.

Required profile