Lead Data Engineer at Vigil

SUMMARY:

As a Data Engineer, you will be responsible for designing, developing, maintaining, and optimising a data pipeline infrastructure using a proprietary data platform , which is based on Databricks. You will collaborate with cross-functional teams to design and implement scalable data solutions, ensuring efficient data ingestion, transformation, storage, and analysis.

WHAT WILL YOU BE DOING:

• Team Management: Accountable for leading a compact team of data engineers, ensuring strong technical guidance, efficient delivery, clear goal-setting, and ongoing mentoring. Also contributes to performance reviews and actively supports the recruitment process.

• Data Product Design & Development: Oversees the end-to-end delivery of scalable data products, infrastructure, and pipelines, primarily using Databricks. This includes implementing robust ingestion, processing, and storage solutions, as well as experimenting with innovative methods early in development cycles.

• Domain Knowledge: Serves as a key subject matter expert within the broader company environment, offering insight into the data foundations of the team’s products and promoting continuous learning through active involvement in internal guilds.

• Business Engagement: Works closely with commercial teams and data product stakeholders to clarify needs around data delivery and ensure visual outputs meet business expectations.

• Data Transformation & ELT: Designs and refines ELT workflows to ensure smooth, reliable data movement into the platform. Applies data validation, enrichment, and hygiene techniques to uphold quality standards for analytical outputs.

• Data Modelling & Architecture: Aligns with architectural frameworks and best practices, working with analysts and scientists to define requirements and build optimised models and storage strategies that support efficient querying and analysis.

• Prototyping Visual Solutions: Provides support to product teams in refining and building interactive dashboards and other visual presentation tools as part of broader data delivery.

• Performance Tuning & Optimisation: Continuously reviews and enhances pipeline performance, resolving bottlenecks, addressing integration complexities, and improving data quality through proactive monitoring.

• Data Quality: Defines clear expectations for data accuracy across deliverables and works alongside QA to implement automated quality checks, surfacing results through observability tools.

• Data Governance & Security: Ensures adherence to the company's data governance, privacy, and compliance frameworks, including applying appropriate controls for access, encryption, and retention, and following architectural oversight processes.

• Support & Monitoring: Ensures observability standards are met across all data products, working with operations and platform teams to establish appropriate metrics and monitoring dashboards for system health and stability.

• Collaboration & Documentation: Engages proactively with other technical teams—analysts, software developers, and data scientists—to deliver fit-for-purpose data solutions. Maintains thorough documentation and translates complex technical concepts into accessible formats for varied audiences, including presenting to wider internal groups.

• Continuous Improvement: Keeps pace with the evolving data landscape, identifying opportunities to modernise tooling, automate workflows, and adopt innovative practices. Collaborates with the team to support feature evolution and remains an active contributor to internal knowledge-sharing communities.

• Ways of Working: Leads engineering engagement in delivery activities by refining work items, distributing tasks within the team, defining and owning user stories, providing updates during stand-ups, and managing delivery expectations. Also supports improvements to CI/CD and related processes.

WHAT WE ARE LOOKING FOR:

• 2 years’ experience managing a compact team of data engineers, providing both technical direction and day-to-day guidance
• 4 years working hands-on in data engineering roles, delivering robust and scalable solutions
• 2 years designing and implementing data solutions using platforms such as Databricks, Hadoop (Cloudera/HortonWorks), and Apache Spark
• Proven strength in logical thinking, problem resolution, and detailed technical analysis
• Confident communicator and effective team collaborator, with a strong ability to work constructively across multidisciplinary teams
• Skilled in Python (especially PySpark), Scala, and SQL
• Experienced in designing and implementing scalable data pipelines for high-volume environments
• Solid understanding of ELT/ETL practices and data integration strategies
• Capable of writing robust production code with automated testing
• Familiar with CI/CD tools such as GitHub Actions and Jenkins for deploying code
• Hands-on experience with distributed data processing using Apache Spark
• Proficient with cloud services including AWS, Azure, or Google Cloud and tools like S3, Glue, Lambda, Redshift, and BigQuery
• Good knowledge of data modelling, relational databases, and SQL performance optimisation
• Strong analytical and problem-solving abilities, with attention to troubleshooting details
• Effective communicator with experience working in collaborative, cross-functional teams
• Basic understanding of machine learning concepts such as classification, regression, A/B testing, and experimental design

AWESOME BUT NOT REQUIRED:

• Understanding of the UK media landscape, including over-the-top (OTT) and traditional broadcast advertising
• Familiarity with concepts and trends in digital advertising and marketing analytics
• Experience applying statistical approaches such as regression and classification, as well as designing and analysing A/B and other controlled experiments
• Awareness of modern data architecture methodologies including Data Mesh, enterprise-level data frameworks, and business intelligence architecture
• Solid grasp of data protection practices, regulatory compliance, and governance principles
• Exposure to a variety of data management domains such as metadata handling, data quality enforcement, master data systems, and governance tooling
• Hands-on experience with data visualisation platforms like Tableau, Looker, AWS QuickSight, and ThoughtSpot for building interactive reporting solutions

WHAT’S IN IT FOR YOU?

• Be part of our collegial environment where responsibility and authority are shared equally amongst colleagues and help create our company culture
• A culture in which we don’t criticise failure but ensure we learn from our mistakes
• An Agile environment where your ideas are welcome
• The possibility to grow and experience different projects
• Ongoing Training & Mentoring
• The possibility to travel

- ATTENTION! THIS POSITION IS FOR PORTUGAL OR BRAZIL BASED ONLY