Key Facts

Remote From:

United States

Category: Data Engineer

Full time

Senior (5-10 years)

English

Hard Skills

Databricks Apache Spark Data Structures Data Lineage Continuous Delivery Databricks Data Warehouse Architectures Identity And Access Management Data Architecture Relational Databases +14 more

Other Skills

•
Scheduling
•
Communication
•
Negotiation
•
Time Management
•
Teamwork
•
Strong Work Ethic
•
Analytical Thinking
•
Presentations
•
Problem Solving

Roles & Responsibilities

Bachelor’s degree required
5+ years of experience in Data Engineering with a focus on data warehousing, ETL/ELT development, and distributed systems
3+ years of hands-on experience developing enterprise solutions on the Databricks platform
Deep understanding of Delta Lake architecture and best practices for maintenance

Requirements:

Design and implement scalable data pipelines using Delta Lake and manage enterprise-wide data access, security, and lineage using Unity Catalog
Optimize large-scale Spark jobs (PySpark/SQL) and cluster configurations (Photon) to meet stringent SLA and cost performance targets across all workflows
Build resilient data scheduling via Databricks Workflows (Jobs) and establish automated CI/CD pipelines for reliable code promotion across Dev, Staging, and Prod workspaces
Migrate data and models from relational databases to Databricks

Job description

Description

Position at Healthmap Solutions

Company Background

Healthmap Solutions is the future of specialty health management that focuses on progressive diseases, with a particular expertise in kidney health populations. Healthmap Solutions uses clinical big data resources and high-powered analytics to power complex specialty health management programs. Healthmap Solutions is a diverse, growing company committed to our clients and our employees. We are champions for better health, for those who need us most.

Position Summary
The Senior Data Engineer drives the design, development, and operational excellence of our data platform built on Databricks and the Delta Lakehouse Architecture. This role requires deep expertise in scalable ETL/ELT, Spark optimization, and modern data governance using Unity Catalog. The Senior Data Engineer serves as a technical leader, establishing best practices for code quality, performance, and automated deployments across all environments.

Responsibilities

Design and implement scalable data pipelines using Delta Lake and manage enterprise-wide data access, security, and lineage using Unity Catalog
Optimize large-scale Spark jobs (PySpark/SQL) and cluster configurations (Photon) to meet stringent SLA and cost performance targets across all workflows
Build resilient data scheduling via Databricks Workflows (Jobs) and establish automated CI/CD pipelines for reliable code promotion across Dev, Staging, and Prod workspaces
Migrate data and models from relational databases to Databricks
Ensure best practices for development using industry standard development patterns
Monitor data pipelines performance
Partner with Data Management and Full Stack development engineers to operationalize models with current applications and processes
Support existing data pipelines to ensure business continuity
Stay updated with the latest trends and technologies in data engineering and cloud computing
Perform other related duties as assigned

Requirements

Bachelor’s degree is required
5+ years of experience in Data Engineering, with a significant focus on data warehousing, ETL/ELT development, and distributed systems
3+ years of hands-on experience developing enterprise solutions on the Databricks platform
Expertise in PySpark and high-performance SQL
Deep understanding practical knowledge of Delta Lake architecture and optimal maintenance best practices
Experience with cloud platforms (AWS preferred) and integrating Databricks with native cloud services (S3, Secret Manager, IAM)
Solid experience implementing CI/CD for Databricks notebooks and associated libraries
Healthcare experience is preferred

Skills

Strong development experience with: Python, Spark, or similar technologies
Excellent verbal, communication, negotiation, and presentation skills
Strong analytical and problem-solving skills
Ability to work autonomously while collaborating across teams to deliver timely projects
Ability to explain complex concepts in simple terms
Dedicated, hardworking employee who achieves maximum efficiency and productivity
Strong knowledge of domain-based design, data modeling and data structures
Strong knowledge of best practice in data management

No Travel

#LI-Remote

Americans with Disability Specifications

The physical demands described here are representative of those that must be met by an employee to successfully perform the essential functions of this job. Reasonable accommodations may be made to enable individuals with disabilities to perform the essential functions.

As an Equal Opportunity Employer, we will not discriminate against any job candidate or employee due to age, race, religion, ethnicity, national origin, gender, gender identity/expression, sexual orientation, disability, familial status, veteran status, marital status, parental status, or pregnancy. In our innovative and inclusive workplace, we prohibit discrimination and harassment of any kind.