Role overview

Qualifications

5-10plus years of experience as a PySpark / SQL Engineer
Prior experience in building threat detection or log analytics pipelines using PySpark, SQL, and Databricks
Hands-on experience with Terraform for deploying data infrastructure
Proficient in PySpark for large-scale data processing and transformation

Responsibilities

Design and build threat detection pipelines using PySpark, SQL, and Databricks
Support the migration of detection rules and content from a legacy platform to Databricks
Create and maintain PySpark log pipelines and associated rule configuration files
Write unit tests to ensure pipeline accuracy and stability

Key facts

Remote from: California (USA)
Fixed term
Senior (5-10 years)
Data Engineer
English

Hard skills

SQL (Programming Language)

About the company

Veracity Software Inc

The definition of Veracity is accuracy and habitual truthfulness. This truly defines our brand in simplistic terms. At Veracity, we believe in doing everything in the ‘Right Way’, which means you have to challenge your own innovations and solutions all the time. Our strength lies in our employees, and with the right blend of business and technical expertise, we set the bar high for Veracity. From inception, Veracity focused on building cloud based SaaS solutions while using the latest and most advanced technology stack. We have delivered cloud based customized solutions and services to various clients across both North America and India. We have a global headquarter in Woodbridge, NJ while our offshore office is in Pune, India. We believe in bringing Global talent together, which helps us in delivering cost effective, yet highly scalable and efficient solutions. With the vision of trying to be one of the trusted IT partners as well as one of the most respected solution providers with an onsite/offshore model, Veracity is equipped with the right set of highly skilled individuals who have expertise in delivering the most effective business services and solutions for our clients. Veracity’s mission is to become the Trusted IT Partner for various businesses across globe.

Company details

Company size51 - 200

Links

Website LinkedIn See all jobs

Your match analysis

See how your profile stacks up against this role.

We compared the job requirements to your profile to show where you're strong and where you fall short.

Job description

Pyspark / SQL Engineer
1 Contractor Remote position
Los Angeles, CA (Remote)
3 Months

Hours per week: 40
Monday to Friday, 9am to 5pm

Skills:
Pyspark
SQL
databricks
Threat Detection
Terraform
Log Analytics for Security

We are seeking a skilled PySpark / SQL Engineer to support our Threat Detection team in building and migrating security analytics pipelines using Databricks. This role will focus on a platform migration project, moving detection rules and associated content from a legacy system into Databricks' native detection framework. You will be responsible for creating equivalent PySpark log pipelines, rule configuration files, unit tests, and data validation checks, and deploying these pipelines using Terraform. A strong background in data engineering, particularly with large-scale log analytics, is essential.

Requirements
5-10+ years of experience as a PySpark / SQL Engineer, with a strong focus on data engineering and analytics.
Prior experience in building threat detection or log analytics pipelines using PySpark, SQL, and Databricks.
Hands-on experience with Terraform for deploying data infrastructure.
Proficient in PySpark for large-scale data processing and transformation.
Familiarity with cloud platforms such as AWS or Azure is preferred.
Strong analytical skills and attention to detail when working with complex datasets.
Proven ability to work effectively in collaborative, cross-functional teams.
Excellent verbal and written communication skills in English.

Responsibilities
Design and build threat detection pipelines using PySpark, SQL, and Databricks.
Support the migration of detection rules and content from a legacy platform to Databricks.
Create and maintain PySpark log pipelines and associated rule configuration files.
Write unit tests to ensure pipeline accuracy and stability.
Perform data validation checks to ensure data integrity.
Deploy pipelines and infrastructure using Terraform.
Optimize existing data workflows and queries for performance and scalability.
Collaborate with cross-functional teams to understand data requirements and ensure alignment with detection objectives.