Job description

Intuition Machines uses AI/ML to build enterprise security products. We apply our research to systems that serve hundreds of millions of people, with a team distributed around the world. You are probably familiar with our best-known product, the hCaptcha security suite. Our approach is simple: low overhead, small teams, and rapid iteration.

We are seeking a Senior Data Engineer to maintain, extend, and improve our existing data/ML workflows and implement new ones. The ideal candidate will work with ML engineers and researchers to build datasets on demand, influence data storage and processing, and collaborate with various teams to enhance our data platform.

What will you do:

Maintain, extend, and improve existing data/ML workflows, and implement new ones to handle high-velocity data.
Provide interfaces and systems that enable ML engineers and researchers to build datasets on demand.
Influence data storage and processing strategies.
Collaborate with the ML team, as well as frontend and backend teams, to build out our data platform.
Reduce time-to-deployment for dashboards and ML models.
Establish best practices and develop pipelines that enable ML engineers and researchers to efficiently build and use datasets.
Work with large datasets under performance constraints comparable to those at the largest companies.
Iterate quickly, with a focus on shipping early and often, ensuring that new products or features can be deployed to millions of users.

What we are looking for:

Thoughtful, conscientious, and self-directed.
Experience working with data engineering services on major cloud providers.
Minimum of 3 years of experience in a data role involving designing and building data stores, feature engineering, and building reliable data pipelines that handle high loads.
Proven ability to make independent decisions regarding data processing strategy and architecture.
At least 2 years of professional software development experience in a role other than data engineering.
Significant experience coding and developing in Python.
Experience in building and maintaining distributed data pipelines.
Experience working with Kafka infrastructure and applications.
Deep understanding of SQL and NoSQL databases (preferably Clickhouse).
Familiarity with public cloud providers (AWS or Azure).
Experience with CI/CD and orchestration platforms: Kubernetes, containerization, and microservice design. Familiarity with distributed systems and architectures.

Nice to Have:

Experience acting as an intermediary between ML and backend/frontend teams.
Exposure to machine learning fundamentals such as model training, model inference, and frameworks like PyTorch and TensorFlow.