Azure Data Engineer

Description

About Our Team

Our core values of People Matter, Integrity, and a Commitment to Excellence drive all that we do. By joining us, you’ll become a part of a fun and diverse team of talented and creative consultants who share the goal of using the latest technology to solve business challenges. We provide our clients with a dynamic mix of services and deliver focused solutions like no one else.

We're seeking talented and bright team players who are passionate about technology and want to work in a fast-paced, dynamic, and ego-free culture while applying a creative approach to problem-solving. Team members who like to grow their skill sets while solving challenging, real world business problems thrive.

About Our Role:

We're looking for an experienced Windows Azure Data Engineer to join our fast-growing team. In this role, you'll leverage your expertise in Azure PaaS services to design, develop, and maintain data pipelines, ETL processes, and cloud-based data solutions. You'll play a key role in transforming conditioned and refactored data into standardized models for enterprise-wide use. Additionally, you'll collaborate with enterprise users to develop advanced data intake, conditioning, and curation techniques—delivering actionable customer experience insights to VA customers and external partners with similar data needs.

This role also offers the opportunity to learn and grow into more advanced Dynamics roles, expanding your expertise in Microsoft’s cloud ecosystem while contributing to impactful, enterprise-wide data initiatives.

Key Responsibilities:

As part of an Agile team, our Azure Data Engineer will:

• Design and implement scalable, secure, and efficient data pipelines using Azure Data Factory (ADF).
• Develop and optimize ETL processes to ingest, transform, and store data in Azure Data Lake, SQL, and Synapse Analytics.
• Automate data ingestion from structured and unstructured sources like APIs, databases, and IoT streams.
• Build and maintain data lakes, warehouses, and lakehouses using Azure Data Lake, Synapse, and Databricks.
• Optimize storage and retrieval of data using Parquet, Delta Lake, and columnar storage formats.
• Ensure high availability, scalability, and performance tuning for large-scale datasets.
• Implement batch and real-time data streaming solutions using Azure Stream Analytics, Event Hub, and Kafka.
• Work with big data processing frameworks such as Apache Spark (Databricks) and HDInsight.
• Integrate data from various sources like SQL Server, CosmosDB, SAP, and third-party APIs.
• Implement data security best practices, including RBAC, Azure AD, Managed Identities, and Key Vault.
• Work with Purview, Data Catalog, and metadata management tools to enforce data governance.
• Monitor and troubleshoot data pipeline performance using Azure Monitor, Log Analytics, and Application Insights.
• Optimize query performance in Synapse Analytics, SQL, and CosmosDB.
• Automate data quality checks, anomaly detection, and alerting mechanisms.
• Work closely with BI Developers and Cloud Engineers to support analytics and AI/ML use cases.
• Participate in Agile/Scrum ceremonies, design reviews, and technical discussions.
• Document data flows, best practices, and architecture decisions.
• Design, develop, and optimize Power BI dashboards and reports for business intelligence and analytics.
• Create interactive visualizations using advanced DAX, Power Query (M), and custom visuals.
• Develop paginated reports using Power BI Report Builder
• Ensure compliance with cloud security best practices and industry standards (ISO 27001, GDPR, HIPAA).
• Automate deployments with Azure DevOps, GitHub Actions, and CI/CD pipelines.
• Work closely with Cloud Engineers and DevOps teams to optimize deployment strategies.
• Collaborate with software engineers, architects, and DevOps teams to build robust applications.
• Participate in code reviews, sprint planning, and Agile/Scrum ceremonies.
• Document technical solutions and best practices.

Minimum Qualifications

• Bachelor’s degree in computer science or a related technical field.
• Minimum of five (5) years of practical experience in cloud application development with Microsoft Azure.
• Expertise in Azure Data Factory (ADF), Synapse Analytics, Data Lake, Databricks, DAX, Power Query (M), and data modeling.
• Strong SQL skills with experience in T-SQL, query optimization, and database performance tuning.
• Experience in big data processing using Apache Spark, Python, or Scala.
• Experience integrating Power BI with Azure SQL, Data Lake, and Synapse Analytics.
• Understanding of ETL processes, data warehousing, and BI best practices.
• Strong knowledge of Row-Level Security (RLS) and Power BI service administration.
• Hands-on experience with real-time data streaming (Kafka, Event Hub, Stream Analytics).
• Familiarity with DevOps for data pipelines, CI/CD, and Infrastructure as Code (Terraform, Bicep).
• Knowledge of Azure AD authentication, security best practices, and cloud monitoring.
• Ability to obtain and maintain a Public Trust or Suitability/Fitness determination based on client requirements
• Knowledge of Agile, Scrum, and SAFe methodologies.
• Careful, reliable, and detail-oriented when performing high impact administration tasks

Preferred:

• Microsoft Certified: Azure Data Engineer Associate (DP-203)
• Microsoft Certified: Power BI Data Analyst Associate (PL-300)
• Experience with CosmosDB, Delta Lake, and NoSQL solutions.
• Knowledge of Machine Learning and AI integration with data pipelines.
• Familiarity with Power BI, Looker, or Tableau for data visualization.
• Knowledge of Azure architecture such as Azure services and portal, Azure Government Cloud, SQL Administrator, SQL reporting, APIs, authorization methods for applications and APIs, migrating SQL schemas and data between environments
• Experience in embedded analytics and Power BI API integration.