Data Engineer - Databricks

Role: Data Engineer - Databricks
Position Type: Full-Time Contract (40hrs/week)
Contract Duration: 6 month+
Work Schedule: 8 hours/day (Mon-Fri)
Location: 100% Remote - Candidates can work from anywhere in LATAM Countries

Position Summary:

The Data Engineer will play a critical role in designing, building, and managing the data infrastructure that powers the Digital department's Analytics functions. This position is responsible for developing and maintaining robust, scalable, and high-quality data pipelines and solutions.

We are seeking a professional with deep, hands-on experience in data engineering. The ideal candidate will excel at building and optimizing data solutions within the Databricks ecosystem and bridging technical solutions with business objectives. This role involves close collaboration with cross-functional teams to translate diverse requirements into actionable data pipelines. Situated in Miami, Florida, and reporting to the Senior Manager of Digital Data Solutions, this position offers a unique opportunity to make a substantial impact on Client's digital journey.

Essential Duties and Responsibilities:

• Data Pipeline Development: Design, build, and maintain scalable, reliable ETL/ELT pipelines using Databricks, Spark, SQL, and Python to process large volumes of data.
• Data Architecture & Modeling: Implement and optimize data models within the Databricks Lakehouse Platform, focusing on creating and managing Delta Tables for performance and reliability.
• Technical Expertise: Serve as a subject matter expert on Databricks, Spark, and Delta Lake. Provide technical guidance on data engineering best practices, including data validation, performance testing, and production readiness.
• Process and Workflow Optimization: Support the standardization and automation of data workflows, including requirements gathering, architecture planning, development, quality control, and production deployment.
• Data Quality and Governance: Implement robust data quality checks and help ensure data governance standards are consistently met throughout the data lifecycle.
• Collaboration: Work closely with data scientists, analysts, and business stakeholders to understand their data needs and deliver high-quality, actionable data solutions.
• Documentation and Knowledge Management: Develop and maintain comprehensive documentation for data pipelines, table structures, dependencies, and business logic to ensure clarity and accessibility.
• Continuous Improvement and Innovation: Stay current with industry trends and proactively identify and implement enhancements to data processes, tools, and workflows.
• Cost Management and Optimization: Monitor and optimize data pipelines and cloud resource usage to ensure cost-effectiveness, implementing strategies to reduce compute and storage costs.
• CI/CD and Automation: Develop and maintain CI/CD (Continuous Integration/Continuous Deployment) pipelines for data workflows to improve deployment frequency and reliability.
• Security and Compliance: Partner with security teams to implement data encryption, access controls, and data masking techniques to ensure compliance with data privacy regulations.

Preferred Qualifications:

• Bachelor's degree in computer science, Data Science, or a related field preferred, or equivalent work experience. MBA or MS a bonus.
• Minimum of 3-5 years of experience in data solution design, development, and implementation.
• Demonstrated expertise in big data technologies, including Azure Data Factory, Databricks, AWS, as well as BI Systems and Adobe Analytics or other log-level data.
• Demonstrated ability in advanced performance tuning of Spark jobs and Databricks clusters.
• Proficiency in SQL, Python, and Agile (Scrum) methodologies.
• Excellent leadership, project management, and communication skills.
• Experience mentoring junior data engineers on technical concepts and best practices.
• Experience leading technical projects and supervising a team of data developers.
• Hands-on experience with data integration, segmentation, and campaign performance analysis.
• Familiarity with Agile collaboration tools like JIRA.
• Strong organizational and multitasking abilities.
• Proven ability to lead technical discussions and provide strategic direction.
• Ability to translate business needs into technical solutions.
• Demonstrated commitment to staying updated with industry trends and adopting new technologies or methodologies.