Description:
Develop new data pipelines using Databricks notebooks and Azure Data Factory to ingest and process data efficiently, ensuring reliability and scalability.
Utilize Databricks and Delta tables to optimize the performance of both new and existing data processing jobs, aiming to reduce operational costs and improve efficiency.
Maintain the data platform focusing on process monitoring, troubleshooting, and data readiness, ensuring high-quality data for regular reporting and system optimization.
Work with other data engineers to design and implement enhancements to the overall data platform, improving functionality and performance.
Effectively collaborate with operations, product management, and other departments to gather requirements, troubleshoot issues, and design system enhancements within an Agile SCRUM framework.
Participate in on-call support, addressing and resolving production issues as they arise, and coordinate with stakeholders to ensure continuous system operation.
Ensure a smooth transition of developed data pipelines to the L2 Support team for post-production maintenance, reducing the need for escalations.
Work independently on end-to-end implementation of data projects, from development through to deployment, within the Agile Scrum framework, demonstrating self-reliance and initiative.
Use DataFrame or PySpark operations to extract data from Azure Delta Lake, creating reports that support business decisions and meet client needs.
Actively engage in release activities, coordinating with cloud engineering teams for necessary infrastructure requirements.
Efficiently onboard new team members to the data cloud platform, organizing and granting access to ensure they can fully utilize the data platform for their work.
Strategically manage and integrate third-party data sources to complement and enhance our proprietary POS data, maximizing data value and insights.
Actively explore and evaluate new technologies or features through proof of concept (POC) and proof of value (POV) projects, driving innovation and technological advancement.
Build or improve data pipelines focusing on compliance, ensuring adherence to GDPR, CCPA, and other relevant regulations, and safeguarding data privacy and security.
Requirements:
Bachelor’s degree in Computer Science, Data Science, Engineering, or a related field; or equivalent experience.
5+ years of experience in data engineering, with specific expertise in big data platforms and technologies.
Proficiency in Apache Scala or Python programming languages.
Experience with Azure Cloud services, Azure Databricks, Delta Lake, and Azure Data Factory.
Experience with Database Systems such as Databricks SQL or Snowflake
Experience with Linux/Unix Systems and Scripting
Experience with Version Control Systems, such as Azure DevOps.
Knowledge of Spark and Apache open-source technologies like Confluent Kafka.
Willingness to learn and engage in on-premises data center technology, such as Yellowbrick, Informatica ETL, and Autosys, as required.
Capable of working in a high-performance, agile scrum environment, actively contributing to various projects, and participating in development sprints.
Strong ability to collaborate across departments and communicate technical concepts effectively.
Excellent communication skills in English are essential for discussing technical requirements, presenting design and feature demos in sprints, and effectively conveying developed solutions to stakeholders during scrum activities.
Able to establish and manage system performance metrics, focusing on rapid issue detection and effective problem resolution.
Job Responsibilities:
Develop new data pipelines using Databricks notebooks and Azure Data Factory to ingest and process data efficiently, ensuring reliability and scalability.
Utilize Databricks and Delta tables to optimize the performance of both new and existing data processing jobs, aiming to reduce operational costs and improve efficiency.
Maintain the data platform focusing on process monitoring, troubleshooting, and data readiness, ensuring high-quality data for regular reporting and system optimization.
Work with other data engineers to design and implement enhancements to the overall data platform, improving functionality and performance.
Effectively collaborate with operations, product management, and other departments to gather requirements, troubleshoot issues, and design system enhancements within an Agile SCRUM framework.
Participate in on-call support, addressing and resolving production issues as they arise, and coordinate with stakeholders to ensure continuous system operation.
Ensure a smooth transition of developed data pipelines to the L2 Support team for post-production maintenance, reducing the need for escalations.
Work independently on end-to-end implementation of data projects, from development through to deployment, within the Agile Scrum framework, demonstrating self-reliance and initiative.
Use DataFrame or PySpark operations to extract data from Azure Delta Lake, creating reports that support business decisions and meet client needs.
Actively engage in release activities, coordinating with cloud engineering teams for necessary infrastructure requirements.
Efficiently onboard new team members to the data cloud platform, organizing and granting access to ensure they can fully utilize the data platform for their work.
Strategically manage and integrate third-party data sources to complement and enhance our proprietary POS data, maximizing data value and insights.
Actively explore and evaluate new technologies or features through proof of concept (POC) and proof of value (POV) projects, driving innovation and technological advancement.
Build or improve data pipelines focusing on compliance, ensuring adherence to GDPR, CCPA, and other relevant regulations, and safeguarding data privacy and security.
What We Offer
Exciting Projects: We focus on industries like High-Tech, communication, media, healthcare, retail and telecom. Our customer list is full of fantastic global brands and leaders who love what we build for them.
Collaborative Environment: You Can expand your skills by collaborating with a diverse team of highly talented people in an open, laidback environment — or even abroad in one of our global centers or client facilities!
Work-Life Balance: GlobalLogic prioritizes work-life balance, which is why we offer flexible work schedules, opportunities to work from home, and paid time off and holidays.
Professional Development: Our dedicated Learning & Development team regularly organizes Communication skills training(GL Vantage, Toast Master),Stress Management program, professional certifications, and technical and soft skill trainings.
Excellent Benefits: We provide our employees with competitive salaries, family medical insurance, Group Term Life Insurance, Group Personal Accident Insurance , NPS(National Pension Scheme ), Periodic health awareness program, extended maternity leave, annual performance bonuses, and referral bonuses.
Fun Perks: We want you to love where you work, which is why we host sports events, cultural activities, offer food on subsidies rates, Corporate parties. Our vibrant offices also include dedicated GL Zones, rooftop decks and GL Club where you can drink coffee or tea with your colleagues over a game of table and offer discounts for popular stores and restaurants!
About GlobalLogic GlobalLogic is a leader in digital engineering. We help brands across the globe design and build innovative products, platforms, and digital experiences for the modern world. By integrating experience design, complex engineering, and data expertise—we help our clients imagine what’s possible, and accelerate their transition into tomorrow’s digital businesses. Headquartered in Silicon Valley, GlobalLogic operates design studios and engineering centers around the world, extending our deep expertise to customers in the automotive, communications, financial services, healthcare and life sciences, manufacturing, media and entertainment, semiconductor, and technology industries. GlobalLogic is a Hitachi Group Company operating under Hitachi, Ltd. (TSE: 6501) which contributes to a sustainable society with a higher quality of life by driving innovation through data and technology as the Social Innovation Business.
