Senior Data Engineer (AWS to GCP) at Vigil

SUMMARY:

As a Data Engineer, you will join a team responsible for building and evolving commercial data products within a large-scale media and streaming environment.

This team is currently in the final stages of migrating key commercial data products from AWS to Google Cloud Platform (GCP). Following the migration, the focus will shift towards optimisation, enhancements, data quality improvements, and the delivery of new data products that support business decision-making across the organisation.

This is an excellent opportunity for a hands-on Data Engineer who enjoys working with modern cloud-native technologies, large-scale datasets, and end-to-end ownership of data solutions.

WHAT WILL YOU BE DOING:

• Design, build, and maintain scalable data pipelines and data products on GCP.
• Support migrating existing AWS-based workloads to GCP environments.
• Develop and optimise ETL/ELT workflows using Python, SQL, and PySpark.
• Build and maintain transformations using BigQuery and Dataproc.
• Create and manage orchestration workflows using Composer (Airflow).
• Implement and improve data quality validation frameworks.
• Work closely with analysts, product stakeholders, and other engineering teams to understand business requirements and deliver reliable data solutions.
• Participate in code reviews, technical discussions, and continuous improvement initiatives.
• Contribute to platform reliability, monitoring, and operational excellence.
• Maintain clear technical documentation and knowledge sharing across the team.

WHAT WE ARE LOOKING FOR:

• Strong commercial experience as a Data Engineer working with cloud-based data platforms.
• Strong hands-on experience with Google Cloud Platform (GCP).
• Experience building data solutions using BigQuery.
• Experience developing and maintaining Airflow pipelines (Composer preferred).
• Strong Python and SQL skills.
• Experience with PySpark and large-scale data processing.
• Understanding of data modelling and modern ETL/ELT practices.
• Experience working with Git-based development workflows.
• Strong communication skills and ability to collaborate with both technical and non-technical stakeholders.
• Experience working within Agile delivery environments.

AWESOME BUT NOT REQUIRED:

• Experience with Dataproc.
• Experience with Dataplex.
• Experience implementing data quality frameworks such as Great Expectations.
• Previous experience with AWS services such as S3, MWAA (Airflow), or Databricks.
• Experience supporting cloud migration projects.
• Experience working within media, streaming, or large-scale consumer platforms.

WHAT’S IN IT FOR YOU?

• Be part of our collegial environment where responsibility and authority are shared equally amongst colleagues and help create our company culture
• A culture in which we don’t criticise failure but ensure we learn from our mistakes
• An Agile environment where your ideas are welcome
• The possibility to grow and experience different projects
• Ongoing Training & Mentoring
• The possibility of travelling

- ATTENTION! THIS POSITION IS FOR PORTUGAL OR BRAZIL BASED ONLY