Key Facts

Remote From:

California (USA)

Category: Data Engineer

Fixed term

Senior (5-10 years)

English

Hard Skills

Snowflake (Data Warehouse) Observability SQL (Programming Language) Splunk Data Modeling Mathematical Optimization StreamSets Infrastructure as Code (IaC) Data Governance Reliability Engineering

Roles & Responsibilities

Strong hands-on experience with Snowflake, especially Data Metric Functions (DMFs)
Advanced experience with dbt (metadata, testing, orchestration)
Proven experience integrating observability tools like Splunk
Experience with incident management platforms (OpsGenie preferred)

Requirements:

Standardize and automate dbt metadata capture across all model runs.
Implement and manage observability rules across key dimensions such as validity and freshness.
Design and implement targeted Snowflake DMFs to optimize performance and reduce alert fatigue.
Build a Data Reliability Executive Dashboard to provide overall data health visibility.

SnapCode Inc

About SnapCode Inc

Introduction Snapcode Inc., a US-based IT staffing firm, provides specialized technology professionals to businesses seeking their expertise. The company’s mission is to connect talented individuals in the IT sector with organizations that are striving to achieve their business objectives. Core Services Snapcode Inc provides a range of staffing solutions tailored to the diverse needs of its clients: • Contract Staffing: Temporary placements to fill short-term roles for project-specific needs. • Permanent Staffing: Recruitment of candidates for full-time positions within organizations. • Managed Services: Comprehensive IT solutions, including team management and project execution. • Executive Search: Services to identify and place top-level IT executives in leadership roles. Industries Served Snapcode Inc specializes in staffing for multiple industries: • Information Technology • Financial Services • Healthcare • Retail and E-commerce • Telecommunications Commitment to Excellence Snapcode Inc. emphasizes a customer-centric approach, ensuring that both its clients and candidates derive personalized benefits from its solutions. The company employs stringent screening procedures and maintains a comprehensive network of highly skilled professionals, enabling it to meet the demands of rapidly changing technological landscapes. Conclusion Snapcode Inc. stands out in the competitive US IT staffing market due to its commitment to quality and its capacity to adapt to industry-specific challenges. By aligning expertise and innovation, the firm provides staffing solutions that enable organizations to flourish in the digital era.

Founded: 2018

Company size: 11 - 50

Website LinkedIn See all jobs →

Job description

Client: Workiva

Job Title: Senior Data Engineer – Data Observability (Snowflake, dbt, DMF)
Location: LATAM
Duration: 6months
Role Overview
We are looking for a highly skilled Senior Data Engineer with strong experience in Data Observability to help operationalize and scale a robust data reliability framework. This role will focus on implementing end-to-end observability across dbt, Snowflake Data Metric Functions (DMFs), Splunk, and OpsGenie, ensuring proactive detection and resolution of data quality issues across critical data assets.
The mission is to establish a self-healing, highly visible data reliability layer that eliminates silent data failures and enables faster incident response.

Key Responsibilities
1. Unified Metadata Collection & Persistence

Standardize and automate dbt metadata capture across all model runs.
Build and maintain a hardened dbt-to-Snowflake logging pipeline to persist run_results.json and manifest.json into an Observability schema.
Implement automated cleanup and retention policies to manage storage efficiently.
Apply data observability rules at scale by pushing dbt checks into Snowflake DMFs.

2. Data Quality & Observability Framework

Implement and manage observability rules across key dimensions:
- Validity (data types, formats)
- Freshness (timeliness, latency)
- Volume (record count reconciliation)
- Schema & Values (structural and value changes)
- Distribution (anomaly detection, trend deviations)
Ensure data quality monitoring for high-priority and Tier-1 tables.

3. DMF Thresholding & Performance Optimization

Design and implement targeted Snowflake DMFs instead of blanket monitoring.
Define dynamic thresholds (e.g., standard deviation-based) to reduce alert fatigue.
Analyze and optimize DMF credit consumption, keeping monitoring costs within 5–10% of total compute.

4. Observability & Alerting (Splunk Integration)

Build a single pane of glass for data observability using Splunk.
Create high-performance alerts correlating dbt job failures with DMF violations.
Ensure alerts include contextual payloads such as:
- Failing dbt model or code link
- Table owner
- Downstream BI impact

5. Incident Management & SLA Enforcement (OpsGenie)

Integrate Splunk alerts with OpsGenie for actionable incident management.
Configure:
- Priority-based alert routing (Warning vs Critical)
- Auto-resolution of alerts when issues self-heal
- SLA tracking for MTTD (Mean Time to Detect) and MTTR (Mean Time to Resolve)

6. Reporting & Executive Visibility

Build a Data Reliability Executive Dashboard (Splunk or Snowflake/Streamlit) to provide:
- Overall Data Health Score
- Volume and Freshness trends
- Top offending models/tables
- Month-over-month MTTD and MTTR improvements

Operational Standards & Documentation

Create detailed Runbooks / SOPs for on-call engineers.
Implement Monitoring-as-Code using version control (Terraform, dbt project files).
Maintain a weekly Observability Health Dashboard to identify noisy or unstable models.

Required Skills & Experience

Strong hands-on experience with Snowflake, especially Data Metric Functions (DMFs)
Advanced experience with dbt (metadata, testing, orchestration)
Proven experience integrating observability tools like Splunk
Experience with incident management platforms (OpsGenie preferred)
Strong SQL and data modeling skills
Experience building scalable, production-grade data pipelines
Familiarity with cost optimization and performance tuning in Snowflake