Key Facts

Remote From:

Full time

Senior (5-10 years)

English

Hard Skills

Apache Spark Apache Airflow Scala (Programming Language) Data Architecture Data Engineering Geospatial Analysis Kubernetes Data Partitioning AWS Cloud Services SQL (Programming Language) +19 more

Roles & Responsibilities

4+ years of experience in Data Engineering
2+ years hands-on experience with Scala (must) or Java
Hands-on experience with Spark / PySpark (must)
Experience with Airflow (must)

Requirements:

Design and develop the big data processing engine and heavy ETL pipelines, including geo-location analysis
Build scalable data pipelines across cloud and on-prem environments
Ensure performance, reliability, and data quality of ETL processes
Collaborate with product, data science, and engineering teams to implement data-driven solutions

Commit

About Commit

Commit is a global tech services company with offices in New York, Israel, and Europe. The company was founded in 2005 and has over 700 multi-disciplinary innovation experts who serve a broad range of companies, from small startups to large enterprises in multiple business sectors. Commit specializes in advanced technologies and applications with dedicated practices in Software, IoT, Big Data, Cloud, Cyber, Collaboration, Data center migration projects, and more. Commit offers innovative, end-to-end technology solutions by developing custom software and IoT platforms for clients looking to build their next-gen products within the modern ICT world. Commit’s complete and comprehensive engineering powerhouse of resources, and proprietary Flexible R&D methodology helps transform its clients’ technology visions into high-quality products while reducing costs and improving time-to-market.

Company type: SME

Founded: 2018

Company size: 501 - 1000

Website LinkedIn See all jobs →

Job description

Description

Company provides connected car manufacturers and mobile network operators with a unique toolset for analyzing and improving the connected journey experience. Company analyzes voice and data usage patterns for all subscribers, 24/7, flagging adverse network experiences that affect subscribers and connected vehicles. Using AI and advanced Machine Learning, it correlates all the important parameters and establishes the impact of these experiences on subscribers.

We are looking for a Big Data / Data Engineer to join our growing team. In this role, you will be responsible for the design and development of our big data processing engine — a heavy ETL pipeline handling complex logic including geo-location analysis, running across both cloud and on-prem environments.

Requirements:

4+ years of experience in Data Engineering
2+ years of hands-on experience with Scala (must) or Java (optional)
2+ years of experience with SQL and NoSQL DBs
Hands-on experience with Spark & PySpark (must)
Experience with Airflow (must)
Experience with AWS and/or GCP
Experience with Kubernetes (k8s) and on-prem deployments
Solid understanding of distributed systems design, pipeline architecture, and partitioning strategies
Deep understanding of Agile principles, practices and values
Solid understanding of design patterns, software development techniques, and clean code practices

Requirements

Advantage:

Experience with Apache Iceberg, Helm Charts, or data catalog tools
Experience with Spring Framework
Experience with microservice architecture
Experience working with DevOps and CI/CD practices
Customer-facing experience or comfort working directly with clients
Bachelor's, BSc or Master's degree in Computer Science or Engineering from a leading academic institution