Job Description: Senior Data Engineer (Hive, Spark, Scala, and Python).
We are seeking a highly skilled data engineer to join our team. In this role, you will be responsible for designing and implementing scalable data pipelines to process large amounts of data. This will include extracting data from a variety of sources, transforming and cleaning the data, and loading it into Hive or Spark for analysis. You will also be responsible for optimizing the performance of these pipelines and ensuring their reliability. As a Senior Data Engineer, you will play a critical role in the design, implementation, and maintenance of highperformance big data infrastructure, systems, and processing pipelines. You will work with a team of data engineers to scale our systems to handle billions of records of structured and unstructured data.
Key Responsibilities:
Design and implement big data infrastructure, systems, and processing pipelines
Troubleshoot and resolve technical issues related to data pipelines
Oversee assigned programs and guide team members
Explore new technologies and learn new techniques to solve business problems creatively
Build Flask and Rest APIs for data processing and analysis
Experience in Data Modelling concepts and SQL script
Handson experience with Spark Streaming or other live stream processing technologies
Any Cloud development (AWS, Azure, GCP) would be a plus and added advantage
Required Skills and Experience:
Strong experience with Hive, Spark, Scala, Python, Linux, and Shell scripting
Strong understanding of data modelling concepts and SQL script
Experience in building and maintaining highperformance big data infrastructure and processing pipelines
Experience with Spark Streaming or other live stream processing technologies
Any Cloud development (AWS, Azure, GCP) would be a plus and added advantage
Strong problemsolving and analytical skills
Excellent communication and team management skills
This is an exciting opportunity for an experienced Senior Data Engineer to take a lead role in a dynamic organization. If you are passionate about data engineering and want to work with a team of talented individuals, we would love to hear from you.
