Match score not available

Cloud Data Administrator (Spark)

72% Flex

Remote:

Full Remote

Contract:

Full time

Experience:

Mid-level (2-5 years)

Work from:

Poland

Offer summary

Qualifications:

Min. 4 years Apache Spark experience, Proficiency in Spark SQL, Python, and monitoring tools.

Key responsabilities:

Administrate Spark Clusters (PaaS) using Terraform
Optimize resource management and performance

NATEK Information Technology & Services SME https://natek.eu/

501 - 1000 Employees

See more NATEK offers

Job description

Your missions

Job Description

BI Development Team is in the process of moving into Azure Cloud. One of the main areas is to administrate Spark Clusters (PaaS). Join us and be part of the Global BI Admin Team.

You’ll be working with a team located in Germany, but the entire project is conducted remotely. Communication is in English.

Your Responsibilities

Setup Synapse Linked Services and MS Synapse workspaces including Spark Clusters using Terraform
Spark version upgrades
Using Kafka, Oracle, etc.
Resource Management, Memory usage, optimizing CPU and resource usage for Spark Application
Launching and running executors on worker nodes
Monitor executor performance and scalability
Detecting handle worker node failures
Reallocating resources and restarting spark jobs
Monitoring the Spark cluster health, resource utilization, and job execution metrics
Monitoring by using Grafana and Prometheus
Performance tuning by fine-tuning Spark parameters
Working with Developers and helping to for improvement of code
IT Team Management

Our Requirements

Min. 4 years of experience with Apache Spark: Advanced knowledge of Spark architecture, including components (RDDs, DataFrames, Datasets)
Very good knowledge of Spark SQL
Proficiency in setting up, configuring, and maintaining Spark clusters
Understanding of YARN
Ability to optimize Spark jobs, memory usage, and parallelism
Familiarity with monitoring tools (preferred: Grafana and Prometheus)
Hadoop: HDFS, Hive, and other related technologies
Proficiency in Python, pyspark and spark SQL
Knowledge of securing Spark clusters and data.
Ability to troubleshoot
Experience in managing an IT team
Good communication with developers, data engineers, and stakeholders.
Experience in cross-functional teams

Benefits

Internal Mobility Program – enjoy many opportunities for career growth, job rotations, diversity of projects & technologies
Referral Program – enjoy cooperation with your colleagues and get a bonus!
5/10 Years NATEK Club – we offer long-term cooperation and celebrate each fifth-year cooperation anniversary with gifts
NATEK CSR Events & team buildings – enjoy our values: accountability, partnership and expertise and #workITwithus
Cafeteria
Medical Healthcare
Multisport Club Card
Technical and Personal training
Language courses

Additionally only for CoW: