Senior Data Engineer

We are seeking a highly skilled and motivated Data Engineer with experience in Databricks, Azure Data Explorer (ADX), Azure IoT Hub, real-time data streams, Kusto Query Language (KQL) and Microsoft Fabric. The successful candidate will play a key role in building and maintaining scalable data pipelines, working on real-time streaming solutions, and ensuring efficient data storage and querying across our cloud platform.

Key Responsibilities:

• Design, develop, and maintain data pipelines that ingest, process, and transform data from various real-time sources (IoT devices, sensors, etc.) into Databricks delta lake.
• Integrate and manage real-time streaming data from Azure IoT Hub into the data platform using Azure Stream Analytics or other real-time data processing services.
• Build and manage real-time data pipelines for use cases requiring immediate analytics and processing.
• Write complex and optimized Kusto Query Language (KQL) scripts for querying and analysing data in ADX.
• Leverage Azure Databricks for data transformation, machine learning, and advanced analytics workflows.
• Collaborate with cross-functional teams, including data scientists, software engineers, and product managers, to deliver data-driven solutions.
• Ensure data quality, reliability, and consistency by implementing appropriate validation and monitoring tools.
• Implement and manage Microsoft Fabric solutions to create unified and scalable data environments for efficient data exploration and analysis.
• Optimize data storage, retrieval, and processing, ensuring that all pipelines are efficient, scalable, and cost-effective.
• Work with Azure DevOps for version control, CI/CD pipeline setup, and automation for deployment of data solutions.

Required Skills and Qualifications:

• Bachelor’s degree in computer science, Data Engineering.
• 6+ years of experience as a Data Engineer working with cloud-based data platforms.
• Strong experience with Python, Azure Data Explorer (ADX) and proficiency in Kusto Query Language (KQL).
• Experience in setting up and managing Azure IoT Hub for real-time streaming data ingestion.
• Solid understanding of real-time streaming data architectures and tools like Azure Stream Analytics, Event Hubs, or Apache Kafka.
• Hands-on experience with Azure Databricks for building scalable data transformation pipelines and advanced analytics workflows.
• Familiarity with Microsoft Fabric for data integration, pipeline management, and optimization and real-time dashboards.
• Strong knowledge of SQL, Python, data modelling, and data warehousing principles.
• Familiarity with Azure Synapse Analytics, Azure Data Factory, and other Azure data services.
• Experience with CI/CD pipelines and Azure DevOps for version control and automation.
• Excellent problem-solving skills and attention to detail.
• Strong communication and collaboration skills.

Preferred Skills:

• Experience with machine learning models and AI integrations in Databricks.
• Familiarity with Scala, or R for data processing.
• Knowledge of data security best practices and experience implementing data governance policies.
• Certifications in Azure Data Engineering or related fields are a plus.

If you are passionate about working with cutting-edge cloud data technologies and thrive in fast-paced environments, we encourage you to apply and join our dynamic team!