Offer summary

Qualifications:

Extensive experience with the Databricks platform and managing clusters., Proficiency in Spark for batch and streaming data processing., Strong knowledge of cloud platforms like AWS, Azure, or Google Cloud., Experience in designing data pipelines and ETL workflows..

Key responsabilities:

Lead the design and optimization of data pipelines using Databricks and Apache Spark.

Implement data engineering best practices in cloud environments for scalability and reliability.

Develop and maintain ETL pipelines for large datasets from various sources.

Collaborate with cross-functional teams to align data solutions with business requirements.

Job description

Databricks Expert
Remote Position
12+Month Contract

Client is seeking an experienced Databricks Expert to join the dynamic team. As a key member of our Data Engineering team, you will leverage Databricks' capabilities to design, implement, and optimize big data solutions. The ideal candidate will have deep expertise in Databricks, Apache Spark, cloud platforms (AWS, Azure, GCP), and modern data architecture practices.

Key Responsibilities:

Lead the design, development, and optimization of data pipelines using Databricks, Apache Spark, and other big data tools.
Implement data engineering best practices in cloud environments (Azure Databricks, AWS, or Google Cloud), ensuring scalability, reliability, and cost-efficiency.
Develop and maintain robust ETL pipelines to move, clean, transform, and analyze large datasets from various sources.
Work with cross-functional teams (data scientists, analysts, business intelligence teams) to ensure that data solutions align with business requirements and performance goals.
Optimize Spark jobs for performance improvements, leveraging best practices for partitioning, caching, and other techniques to handle large datasets efficiently.
Implement data governance, security protocols, and ensure compliance with relevant data privacy regulations.
Identify and resolve issues related to performance, scalability, and data integrity within Databricks environments.
Create clear and concise documentation for processes, solutions, and code. Communicate results to stakeholders and leadership.

Skills and Qualifications:

Extensive experience with Databricks platform, including managing clusters, running Spark jobs, and using Databricks notebooks.
Proficiency in working with Spark (especially in a distributed environment) for both batch and streaming data processing.
Strong knowledge of cloud-based platforms such as AWS, Azure, or Google Cloud with experience in Databricks integration.
Solid experience in designing and implementing data pipelines, ETL workflows, and managing large-scale datasets.
Proficient in languages like Python, Scala, or SQL for data processing and scripting.
Hands-on experience with other big data tools (e.g., Hadoop, Kafka, Delta Lake) and frameworks.
Understanding of modern data storage architectures, data lakes, and data warehouses.
Proven track record of optimizing distributed systems and databases for performance.
Strong analytical and troubleshooting skills to resolve complex data issues.
Excellent written and verbal communication skills, with the ability to present technical solutions to non-technical stakeholders.

Metasys Technologies is an equal opportunity employer. All applicants will be considered for employment without attention to race, color, religion, sex, sexual orientation, gender identify, national origin, veteran or disability status.

Required profile