Senior Databricks Data Engineer - Remote

Job Title: Senior Databricks Data Engineer - Remote

Job Description:

We are seeking an experienced Senior Databricks Data Engineer with strong expertise in designing, developing, and optimizing large-scale data engineering solutions on the Databricks platform. The ideal candidate will have extensive hands-on experience in building enterprise-grade data pipelines, processing high-volume datasets, and delivering scalable analytics solutions using PySpark, Spark SQL, and AWS cloud technologies.

Key Responsibilities:

• Design, develop, and maintain scalable data engineering solutions using Databricks, PySpark, Spark SQL, and Apache Spark. Build and optimize end-to-end ETL/ELT pipelines integrating Databricks with AWS S3, Snowflake, Airflow, and AWS Glue.
• Process and transform large-scale structured and semi-structured datasets to support enterprise reporting, analytics, and regulatory requirements.
• Develop reusable frameworks and components for: Data ingestion Data cleansing and transformation Data validation and reconciliation Data enrichment Business rule implementation Implement robust data quality checks, monitoring, and governance controls to ensure data accuracy and reliability.
• Perform performance tuning and optimization of Spark applications and Databricks workloads to improve scalability and processing efficiency.
• Collaborate with business stakeholders, architects, and analytics teams to understand data requirements and deliver scalable solutions.
• Troubleshoot and resolve production data pipeline issues while ensuring high availability and reliability.
• Support advanced analytics, customer segmentation, campaign analytics, AML/regulatory reporting, and data validation initiatives.

Required Skills:

• Technical Skills 7+ years of hands-on experience with Databricks and modern data engineering platforms.
• Strong expertise in: PySpark Spark SQL Apache Spark Databricks Lakehouse Architecture.
• Experience building and managing ETL/ELT pipelines on cloud platforms.
• Strong experience with: AWS S3 Snowflake Apache Airflow AWS Glue Knowledge of data modeling, data warehousing, and big data processing techniques.
• Experience working with structured and semi-structured data formats such as JSON, Parquet, Avro, and CSV.
• Hands-on experience implementing data quality frameworks and monitoring solutions. Strong SQL and data analysis skills. Preferred Skills Experience with regulatory reporting, AML, compliance, or financial data domains. Exposure to CI/CD and DevOps practices for data engineering.
• Experience with Delta Lake, Databricks Workflows, and Unity Catalog.
• Knowledge of Agile/Scrum delivery methodologies.

Key Experience Areas Candidates should have demonstrated experience in one or more of the following:

Customer Segmentation Analytics Campaign Analytics AML and Regulatory Reporting Enterprise Data Platforms Production Data Pipeline Support Data Validation and Reconciliation Large-Scale Data Migration and Modernization Programs