Role overview

Qualifications

5-10+ years of experience as a PySpark/SQL Engineer
Prior experience in building threat detection or log analytics pipelines using PySpark, SQL, and Databricks
Hands-on experience with Terraform for deploying data infrastructure
Proficient in PySpark for large-scale data processing and transformation

Responsibilities

Design and build threat detection pipelines using PySpark, SQL, and Databricks
Support the migration of detection rules and content from a legacy platform to Databricks
Create and maintain PySpark log pipelines and associated rule configuration files
Write unit tests to ensure pipeline accuracy and stability

Key facts

Remote from: California (USA)
Full time
Senior (5-10 years)
English

Hard skills

Databricks SQL (Programming Language)

About the company

MDA Edge

Our objective is to establish an exceptional ecosystem by connecting individuals, technology, and prospects through the application of human intelligence.MDA Edge is a process-oriented company and our expertise lies in providing comprehensive Workforce Solutions, specifically focusing on Contingent Staffing, Bulk/Project Staffing, RPO/KPO/BPO, and Direct Hire services. We cater to a wide range of industries, including Infrastructure Consulting, Engineering Consulting, IT Consulting, Healthcare, Life Sciences, Pharmaceutical Consulting, Consumer Goods, Education, Transportation & Logistics, Media & Entertainment, Telecom, BFSI, Manufacturing, Utilities & Energies, and Corporate Recruitment. Moreover, we have developed a dedicated focus on Government Consulting.Our company prides itself on a meticulous approach, ensuring that we meet your specific needs. We have established a robust network of highly skilled professionals who are readily available to fulfill your requirements. With a strong commitment to excellence, we consistently deliver exceptional results for clients in India, Denmark, France, Germany, Ireland, Japan, Spain, Thailand, the USA, Canada and Mexico, with ongoing expansion efforts to serve more new regions in the near future.We consistently uphold the highest standards of quality by providing resources, time, and materials to design, implement, and support efficient operations for organizations. Our track record of measurable accomplishments demonstrates our commitment to cultivating a balanced work and societal ecosystem.Our continuous growth, successful customer engagements, and strong customer retention exemplify our achievements. Furthermore, our passion lies in streamlining complex business processes through the application of suitable technology, which has been integral to our success.We extend an invitation to join our dynamic workplace, offering rapid growth opportunities, excellent employee benefits, and a positive work-life balance.

Company details

Company typeScaleup

Company size201 - 500

Links

Website LinkedIn See all jobs

Your match analysis

See how your profile stacks up against this role.

We compared the job requirements to your profile to show where you're strong and where you fall short.

Job description

Job Summary: We are seeking a skilled PySpark / SQL Engineer to support our Threat Detection team in building and migrating security analytics pipelines using Databricks. This role will focus on a platform migration project, moving detection rules and associated content from a legacy system into Databricks' native detection framework. You will be responsible for creating equivalent PySpark log pipelines, rule configuration files, unit tests, and data validation checks, and deploying these pipelines using Terraform. A strong background in data engineering, particularly with large-scale log analytics, is essential.
Requirements:

5-10+ years of experience as a PySpark/SQL Engineer, with a strong focus on data engineering and analytics.
Prior experience in building threat detection or log analytics pipelines using PySpark, SQL, and Databricks.
Hands-on experience with Terraform for deploying data infrastructure.
Proficient in PySpark for large-scale data processing and transformation.
Familiarity with cloud platforms such as AWS or Azure is preferred.
Strong analytical skills and attention to detail when working with complex datasets.
Proven ability to work effectively in collaborative, cross-functional teams.
Excellent verbal and written communication skills in English.

Responsibilities:

Design and build threat detection pipelines using PySpark, SQL, and Databricks.
Support the migration of detection rules and content from a legacy platform to Databricks.
Create and maintain PySpark log pipelines and associated rule configuration files.
Write unit tests to ensure pipeline accuracy and stability.
Perform data validation checks to ensure data integrity.
Deploy pipelines and infrastructure using Terraform.
Optimize existing data workflows and queries for performance and scalability.
Collaborate with cross-functional teams to understand data requirements and ensure alignment with detection objectives.