Principal Databricks Software Developer

PowerPlan is a profitable market leader with over 30 years of experience in asset-intensive financial systems and managing trillions of dollars of clients’ assets. PowerPlan is in a growth phase investing into next-generation solutions. We’re hiring a Principal Databricks Platform Engineer to take ownership of the health, reliability, performance, governance, and cost effectiveness of our Databricks platform.

This is not a role for someone who only advises from the sidelines. You’ll be hands-on with notebooks, pipelines, jobs, clusters, Spark configurations, Delta Lake patterns, and Unity Catalog. You’ll be the person teams turn to when something technically works but is slower than it should be, more expensive than it needs to be, harder to operate than it should be, or fragile enough to become tomorrow’s production issue.

• Take ownership of the highest-risk Databricks workloads and platform patterns, from configuration review through remediation and production behavior.
• Review existing clusters, jobs, notebooks, pipelines, and workspace settings to identify cost waste, performance bottlenecks, reliability risks, security gaps, and governance drift.
• Set technical direction for Databricks usage across teams, including compute policies, Spark configuration standards, notebook design expectations, pipeline patterns, orchestration practices, and Unity Catalog governance.
• Raise the bar for Databricks code and architecture reviews by identifying anti-patterns such as oversized clusters, unnecessary shuffles, driver-side collections, unbounded full scans, brittle job dependencies, missing retry logic, poor Delta maintenance, and hardcoded environment assumptions.
• Partner with engineering teams to turn findings into practical improvements: better defaults, reusable templates, reference architectures, automation, monitoring, and clear standards that teams can actually follow.
• Serve as an escalation point for complex Databricks failures, performance regressions, pipeline instability, data quality issues, and cost anomalies.
• Translate technical risk into business impact so stakeholders understand why a slow job, missing lineage, weak guardrail, or inefficient cluster design matters.
• Make engineers around you better through pairing, reviews, design conversations, internal workshops, and examples that demonstrate what excellent Databricks engineering looks like.

What success looks like

One month in: you’ve reviewed the most important Databricks assets, shipped or guided meaningful production improvements, built a real mental model of the platform, and can name the biggest architectural, operational, and cost risks with evidence.

Three months in: you own the design and improvement plan for at least one critical Databricks capability or platform standard, and your recommendations are showing up in how teams build, review, schedule, and operate Databricks workloads.

Six months in: teams seek your input before building high-risk workloads, not after they fail. Platform standards are clearer, recurring anti-patterns are declining, performance and reliability trends have improved, and Databricks cost decisions are being made with discipline instead of guesswork.

• 8+ years in data engineering, platform engineering, or related software engineering roles, with at least 3+ years of deep hands-on Databricks experience.
• Production experience with Spark internals and performance tuning, including shuffle behavior, query planning, adaptive query execution, memory pressure, skew, spill, partitioning, and broadcast strategy.
• Strong working knowledge of Databricks compute options, cluster policies, job clusters, all-purpose clusters, instance pools, autoscaling, serverless eligibility, Photon, and workload right-sizing.
• Real experience reviewing and improving notebooks, jobs, and pipelines that other people built, including the ability to quickly identify the few design choices causing most of the runtime, cost, or reliability risk.
• Production experience with Delta Lake, Delta maintenance strategies, Structured Streaming, Auto Loader, materialized views or streaming tables, and data quality expectations.
• Strong understanding of Unity Catalog, workspace governance, lineage, permission models, catalog/schema organization, table documentation, and secure handling of sensitive data.
• Experience designing reliable job orchestration: task decomposition, dependency management, retries, repairs, timeouts, alerting, scheduling, max concurrent runs, and recoverability.
• A track record of identifying systemic reliability, performance, governance, or cost issues and driving practical remediation across teams.
• Strong communication skills, including the judgment to explain technical findings in a way that helps engineering leaders, product owners, and stakeholders make better decisions.
• A STEM degree, or equivalent depth earned through substantial production experience.

Also valuable

• Experience with Azure-based data platforms and enterprise cloud operations.
• Experience partnering with FinOps or platform teams to reduce cloud spend without sacrificing reliability or service-level expectations.
• Experience building reusable Databricks templates, platform automation, internal standards, or reference architectures.
• Familiarity with data privacy, security, auditability, and regulated business environments.
• Opinions, backed by evidence, about when Spark or Databricks is the right tool — and when it is not.
• Experience helping teams adopt AI-assisted engineering practices responsibly, especially for code review, test generation, documentation, and operational analysis.

Why this role

You’ll shape how PowerPlan uses Databricks for critical data workloads, and your judgment will directly influence reliability, performance, governance, and cost. This is a senior, hands-on role where the work is visible, the problems are meaningful, and the best answer is not always obvious.

The right person will look at something that “works” and see what is fragile, wasteful, insecure, or difficult to operate before it becomes an incident. You’ll set the standard for Databricks technical excellence, and the improvements you drive will make the platform stronger for every team that depends on it.

PowerPlan is an EOE

Applicant and Candidate Privacy Notice

Please note that this is a hybrid role that involves a combination of onsite work from our corporate office as well as work from home. While we strive to accommodate flexible working arrangements when sensible, there will be times when onsite work is required. This could include scheduled office days, team meetings, client meetings, or special events.