Senior Data Engineer

Hashlist is a platform for projects within the automotive industry.

We establish supplier relationships with automotive companies and provide a one-stop shop for candidates interested in working in the industry.

Are you ready to embark on this journey?

Our client is seeking a Senior Data Engineer. By applying, you gain access to the Hashlist network, unlocking opportunities in automotive projects across the industry.

Available locations: This position is open to candidates in Brazil and India (Bengaluru, Chennai & Pune).

Responsibilities:

• Design and build a scalable data platform that can efficiently ingest, store, manage, and stream large volumes of data, simplifying analysis and enabling rapid development of high-quality data products and services.
• Implement and test low-latency, real-time data pipelines using cutting-edge big data technologies to support data-as-a-service products.
• Collaborate with cross-functional teams to clean, curate, and build end-to-end data pipelines based on requirements.
• Design and implement privacy and security controls into the CI/CD of each data pipeline.
• Ensure operational excellence by incorporating error handling, restartability, logging, monitoring, and alerting into the pipelines to maintain data consistency.
• Drive continuous improvements in reliability, performance, scalability, and quality, owning the associated KPIs.
• Provide on-call support for the platform and troubleshoot any issues that arise.

Qualifications:

• 5+ years of experience in developing and operating cloud-native data streaming systems, with a focus on scalability, low latency, data privacy, and quality control.
• Experience with cloud-native real-time streaming data technologies such as Spark, Flink, Kinesis, Lambda, Kafka, EMR/EKS platforms, and Lakehouse platforms (e.g., Delta.io).
• 5+ years of experience in optimizing AWS data system architecture, including Infrastructure as Code, deployment automation, and security.
• Proven experience in deploying and monitoring AWS infrastructure, particularly with MSK and EKS clusters in production environments.
• Proficiency in programming languages such as Spark-Scala, Py-Spark, Python, or Java.
• Strong knowledge of relational databases, SQL, SQL tuning, and NoSQL big data technologies.
• Experience with geospatial data processing is a plus.
• Experience in developing and deploying containerized applications using Docker, Kubernetes, and Helm.
• Proven ability to build systems that monitor data quality and prevent data losses.
• Strong understanding of data structures, algorithms, and operational excellence principles.
• Curiosity and passion for learning, with a strong bias for action.

Next Steps:

• Click "Apply."
• We will review your application.
• If qualified, you’ll be accepted into the Hashlist network and considered for this and other relevant projects.