Match score not available

Lead Databricks Engineer

Remote:

Full Remote

Contract:

Full time

Experience:

Senior (5-10 years)

Work from:

Missouri (USA), United States

Offer summary

Qualifications:

Strong experience in Pyspark and Apache Spark, Experience in data governance solutions.

Key responsabilities:

Build orchestration layer in Databricks/ADF
Develop CICD for Databricks in Azure Devops
Implement security layer in Delta Lake
Design and implement complex data solutions

Diverse Lynx Large https://www.diverselynx.com/

1001 - 5000 Employees

See more Diverse Lynx offers

Job description

Your missions

Role – Lead Databricks Engineer
Location: Remote
Yrs. of experience: 10+ Yrs.
Mode of employment: Full-Time

Job Description:
Candidates are required to have these mandatory skills:
• Strong hands on in Pyspark and Apache Spark.
• Experience in Native Spark Migration to Databricks.
• Experience in Hadoop Migration to Databricks.
• Experience in Building Data Governance Solutions like Unity Catalog, StarBust ...etc
• Build Very Strong Orchestration Layer in Databricks/ADF.... Workflows.
• Build CICD for Databricks in Azure Devops.
• Process near Real time Data thru Auto Loader, DLT Pipelines.
• Implement Security Layer in Delta Lake.
• Implement Massive Parallel Processing Layers in Spark SQL and PySpark.
• Implement Cost effective Infrastructure in Databricks.
• Experience In extracting logic and from on prem layers like SSIS, Stored procedures, Informatica, Vertica, Apache Hudi, Filesystems. etc into Pyspark.
• Ability to Build Solutions as Cloud Agnostic.
• Proven experience in designing and implementing complex data solutions aligned with business objectives.
• Expertise in data modelling, integration, security, and governance
• Hands-on experience with guiding the virtual data model definition, defining Data Virtualization architecture and deployment with focus on Azure, Databricks, PySpark technologies.
• Prior experience with establishing best practices for business optimizations.
• Experience with relational and non-relational data stores (Hadoop, SQL, Mongo DB), ETL or ELT tools (SSIS, Informatica, Matillion, DBT), DevOps, Data Lake and Data Fabric concepts
• In-depth experience with data governance, data integration and related technologies.
• Proficiency in a variety of database technologies, both relational and non-relational.
• Knowledge of cloud-based data solutions (e.g., AWS, Azure).
• Excellent collaboration and communication skills

Diverse Lynx LLC is an Equal Employment Opportunity employer. All qualified applicants will receive due consideration for employment without any discrimination. All applicants will be evaluated solely on the basis of their ability, competence and their proven capability to perform the functions outlined in the corresponding role. We promote and support a diverse workforce across all levels in the company.