Data Platform Engineer

What you will be doing:

Are you a highly skilled Data Platform Engineer with a passion for cutting-edge technology and a desire to make a real impact? Join our innovative team at ComplyAdvantage, where you'll design, manage, monitor, and optimise a diverse array of data systems running on AWS and GCP. Your expertise will ensure the seamless operation of SQL, NoSQL, Kafka, and Spark-based data layers, supporting our global client base.

Key Responsibilities:

• Innovative Design and Maintenance: Develop and sustain data layer orchestration systems, managing self-managed and PaaS data layer technologies, and ensure data consistency while designing databases.
• Performance and Availability: Oversee data layer components on AWS and GCP, guaranteeing top performance and reliability. Conduct regular performance tuning and troubleshoot issues.
• Backup and Cost Management: Configure and deploy robust backup systems for databases, Kafka, and Spark, ensuring data integrity and enabling disaster recovery. Implement cost management strategies to optimise resource utilisation.
• Documentation and Collaboration: Maintain thorough documentation for all Data Infrastructure engineering processes. Work closely with various teams to integrate data sources and support application requirements, adhering to best practices.
• Capacity and Project Planning: Monitor and manage data infrastructure capacity, planning for future growth and scalability. Collaborate with stakeholders to define and prioritise engineering projects, setting realistic timelines and expectations.
• Technical Support: Provide guidance and support to engineering team members on database and data layer architecture and technologies. Collaborate with Software Engineers to develop resilient data sharding strategies.

Our Tech Stack:

• Cloud-Based Infrastructure: Fully cloud-based with a Kubernetes-focused tech stack. Compute workloads run in Kubernetes clusters across multiple regions.
• Infrastructure Management: Heavy use of Terraform and Helm, adhering to GitOps paradigms for managing cloud infrastructure and Kubernetes applications.
• Core Technologies: Extensive use of Kafka, distributed PostgreSQL and Cassandra QL, Elasticsearch, and Databricks/Spark. Development of inter-cloud failover options to support multi-cloud plans.
• Wide Array of Applications: Teams build and release containerised applications for low latency APIs, machine learning models, and data processing pipelines.

About You:

• Experience: Significant expertise with AWS and/or GCP core services, data systems (Apache Kafka, Apache Spark, Elasticsearch, PostgreSQL, Cassandra), and Kubernetes. Experience codifying infrastructure in Terraform and/or Helm charts.
• Cost and Performance: Awareness of cost and performance considerations in designing data systems architectures. Proficiency with logging, monitoring, and tracing tools.
• CI/CD and Security: Familiarity with CI tooling, test pipelines, and asset generation (e.g., Docker images, Helm charts). Understanding of security considerations in data systems.
• Data Modelling: Experience with data modelling, sharding, and indexing strategies for large-scale databases.

Nice to Have:

• Experience running distributed databases within Kubernetes.
• Familiarity with distributed SQL and NoSQL databases such as Yugabyte, Cockroach, Spanner, HBase, or CouchDB.

Education:

• BSc/BA degree in computer science, engineering, or related discipline OR equivalent experience in required skills

What’s in it for you? 

• Equity as we want you to have a part of what we are building 
• Unlimited Time Off Policy- A work-life balance and focus on our well-being are critical to keeping us performing at our best 
• We embrace a hybrid approach that requires employees to be in the office for two days a week. We strongly believe that this approach fosters collaboration and enables the building of meaningful relationships
• You will also get a new starter budget to kit out your home office 
• Opportunity to work on innovative projects with smart-minded people keen to share their knowledge and continuously improve 
• Annual learning budget (prorated based on start date) to drive your performance and career development