JD:
• Role: Sr / Lead Data Engineer
• Experience: 5 to 8 years
• Job Location : Hyderabad, Work from Office
• Mode of employment: Permanent
• Notice period: Immediate to 15 days ONLY.
Skills and Responsibilities.
• Strong programming skills in Python, Scala, or SQL, as well as Pyspark.
• Architect and implement data pipelines using Apache Spark for efficient data processing and transformation.
• Develop and maintain data workflows using Apache Airflow to automate data pipeline orchestration and scheduling.
• Create and optimize SQL queries for data extraction, transformation, and loading (ETL) processes.
• Harness the power of Python for data manipulation, scripting, and automation.
• Manage and optimize data storage and processing on AWS EMR, Redshift, and Azure Synapse.
• Collaborate with teams to design and maintain Azure Data Factory (ADF) pipelines for data integration.
• Leverage DBT for data modelling and version control, ensuring data accuracy and consistency.
• Integrate data from diverse sources using Fivetran, ensuring seamless data ingestion.
• Continuously improve data quality, performance, and reliability across the data ecosystem.
• Monitor and troubleshoot data pipelines, ensuring data availability and integrity.
• Implement data security best practices and ensure compliance with data privacy regulations.