Role Overview:
The Senior Data Integration Engineer is responsible for integrating data from various sources, managing ETL/ELT pipelines on Databricks, and ensuring data pipeline scalability and compliance. Key skills include Databricks, Spark, Python, SQL, AWS, cloud computing, and big data technologies. The role involves collaboration with multiple departments and requires expertise in data integration, pipeline management, and automated monitoring strategies. This role requires a deep understanding of:
- Data Lake ingestion and integration processes and best practices
- ETL/ELT design
- CI/CD
- System integration tools
- Data pipeline management
This is based in Canada as a remote position. We will only consider candidates currently in Canada and are not offering relocation assistance at this time.
About the role:
- Integrate data from a variety of source systems and knowledge to tailor integration approaches on a per-system basis.
- Liaise with Technical Data Steward and understanding business requirements for ingestion and integration.
- Document ingestion and integration flows and pipelines.
- Use Airflow to schedule and automate ingestion jobs.
- Manage metadata and master data in technical data catalog.
- Manage, maintain, and oversee ETL/ELT pipelines on the Databricks platform
- Ensure ELT/ETL design meets required security and compliance guidelines, and ensure PII management, flagging, and risk assessment during ingestion and integration.
- Conduct pipeline monitoring.
- Ensure adherence to SDLC best practices.
About You:
You possess experience in data engineering, ingestion pipelining, and ETL/ELT with hands-on experience in:
- Databricks: Expertise in managing and maintaining ETL/ELT pipelines on the Databricks platform.
- Spark: Proficiency in using Apache Spark for big data processing and analytics.
- Python: Strong programming skills in Python for data manipulation, analysis, and automation.
- SQL: Advanced knowledge of SQL for querying.
- AWS: Experience with AWS services for data integration and management.
- Cloud: Familiarity with cloud computing concepts and services.
- Big Data: Knowledge of big data technologies and frameworks.
- Data Integration: Experience in integrating data from various source systems and tailoring integration approaches.
- ETL/ELT Design: Ability to design and optimize ETL/ELT pipelines for scalability and speed.
- CI/CD: Familiarity with continuous integration and continuous deployment practices.
- System Integration Tools: Knowledge of tools and techniques for system integration.
- Data Pipeline Management: Expertise in managing data pipelines and ensuring they meet security and compliance guidelines.
- Metadata and Master Data Management: Skills in managing metadata and master data in the technical data catalog.
#LI-Remote
Company Overview
McAfee is a leader in personal security for consumers. Focused on protecting people, not just devices, McAfee consumer solutions adapt to users’ needs in an always online world, empowering them to live securely through integrated, intuitive solutions that protects their families and communities with the right security at the right moment.
Company Benefits and Perks:
We work hard to embrace diversity and inclusion and encourage everyone at McAfee to bring their authentic selves to work every day. We offer a variety of social programs, flexible work hours and family-friendly benefits to all of our employees.
- Bonus Program
- Pension and Retirement Plans
- Medical, Dental and Vision Coverage
- Paid Time Off
- Paid Parental Leave
- Support for Community Involvement
We're serious about our commitment to diversity which is why McAfee prohibits discrimination based on race, color, religion, gender, national origin, age, disability, veteran status, marital status, pregnancy, gender expression or identity, sexual orientation or any other legally protected status.
