Senior Data Engineer

Hashlist is a platform for projects within the automotive industry.

We establish supplier relationships with automotive companies and provide a one-stop shop for candidates interested in working in the industry.

Are you ready to be a part of that journey?

Our client is now looking for a Senior Data Engineer.

By applying for this role, you will simultaneously tap into the Hashlist network and access automotive projects across the whole industry.

____________

Available locations: This role is open for candidates in Brazil and India (Bengaluru, Chennai & Pune).

____________

Responsibilities:

• Design and build a scalable data platform that can efficiently ingest, store, manage, and stream large volumes of data, simplifying analysis and enabling rapid development of high-quality data products and services.
• Implement and test low-latency, real-time data pipelines using cutting-edge big data technologies to support data-as-a-service products.
• Collaborate with cross-functional teams to clean, curate, and build end-to-end data pipelines based on requirements.
• Design and implement privacy and security controls into the CI/CD of each data pipeline.
• Ensure operational excellence by incorporating error handling, restart ability, logging, monitoring, and alerting into the pipelines to maintain data consistency.
• Drive continuous improvements in reliability, performance, scalability, and quality, owning the associated KPIs.
• Provide on-call support for the platform and troubleshoot any issues that arise.

Qualifications:

• 5+ years of experience in developing and operating cloud-native data streaming systems, with a focus on scalability, low latency, data privacy, and quality control.
• Experience with cloud-native real-time streaming data technologies such as Spark, Flink, Kinesis, Lambda, Kafka, EMR/EKS platforms, and Lakehouse platforms (e.g., Delta.io).
• 5+ years of experience in optimizing AWS data system architecture, including Infrastructure as Code, deployment automation, and security.
• Proven experience in deploying and monitoring AWS infrastructure, particularly with MSK and EKS clusters in production environments.
• Proficiency in programming languages such as Spark-Scala, Spark, Python, or Java.
• Strong knowledge of relational databases, SQL, SQL tuning, and NoSQL big data technologies.
• Experience with geospatial data processing is a plus.
• Experience in developing and deploying containerized applications using Docker, Kubernetes,
• and Helm.
• Proven ability to build systems that monitor data quality and prevent data losses.
• Strong understanding of data structures, algorithms, and operational excellence principles.
• Curiosity and passion for learning, with a strong bias for action.

____________

Next steps:

• Press "Apply"
• We will review your application
• If qualified, you will be accepted into the network, and can be considered for this and similar projects