Overview
We are hiring for a Data Engineering Chapter Lead / Principal Data Engineer. Headquartered in Los Angeles, California, Right Balance provides top-tier technology talent for innovative companies in the US. We’re in the top 50 companies to watch in LA.
Client Engagement Details
The Client Our client is a modern platform that provides Payfac as a service with a full suite of payments and risk management services built for vertical SaaS companies. With them, SaaS providers can embed payments and financial services in their native experience and add a new revenue stream in a few weeks. They have a strong product-market fit and a clear moat that has resulted in an exponential growth of 100% YoY and the trend will continue. They are a part of a larger company, who invented payment facilitation, enabling technology companies to provide integrated payment solutions, significantly expanding access to modern payment methods to SMBs, and rejuvenating end-user experience. About the team:
We are a passionate, global team of payments and software experts who provide vertical software companies with an all-in-one platform and a white-glove approach, to capitalize on the opportunities within embedded payments for growth, innovation, and transformation. Our clients are leading vertical software providers that want to make their products stickier and grow revenue by offering payment solutions to their end customers.
About the role: This role will be responsible for leading the design and architecture of our data platform to serve the financial reporting and reconciliation, forecasting, and data delivery needs of our internal as well as external customers. The person will work on transforming our current data architecture into highly scalable/resilient architecture while supporting high product velocity and customer asks. This role will review and oversee data engineering across multiple teams and will be responsible for encouraging standards and improving best practices related to data pipelines. The platform is complex with multiple functionalities and presents a great opportunity to learn about embedded payments and payfac-as-a-service capabilities.
Responsibilities
- Technical Leadership
- Provide expertise and guidance in big data technologies.
- Extend and optimize our data lake: improve query performance and integrate more data sources.
- Build ETL pipelines for our various data-driven services for our clients.
- Build ETL pipelines and data materializations for internal data-driven reporting for historical analysis, risk assessment, projections, financial forecasting, etc.
- Participate in the development and communication of data strategy and roadmaps across the technology organization to support product portfolio and business strategy.
- Define high-level migration plans to address the gaps between the current and future state.
- Lead the analysis of the technology environment to detect critical deficiencies and recommend improvement solutions.
- Promote the reuse of data assets, including the management of the data catalog for reference.
- Ensure that our data lake architecture and data catalog/dictionary are fully documented.
- Help build user guides for writing efficient queries to utilize the data lake maximally.
- Day-to-day governance for giving access to different parts of the data lake to different users after assessing their needs in coordination with infosec.
- Ensure that all ETL code and configuration are properly maintained in the version control.
- Drive digital innovation by leveraging innovative new technologies and approaches to renovate, extend, and transform the existing core data assets, including SQL-based, NoSQL-based, and Cloud-based data platforms.
- Identify and improve parts of the platform to make it more robust and scalable.
- Advise the leadership on key technology and product roadmap considerations.
- Align our technical decisions with broad strategic initiatives, while also advocating for needs specific to emerging new businesses.
Team Leadership
- Dotted line responsibility for a distributed data engineering team (both full-time and contractors) while remaining hands-on.
- Collaborate with and support the business intelligence team.
- Collaborate with and support the data science team.
- Mentor data engineers and software engineers in data principles, patterns, processes, and practices.
Project Management
- Prioritize projects based on customer and internal needs in consultation with product management, engineering, operations, and finance departments.
- Support teams as they deliver.
- Support the needs of business intelligence, financial reporting, and data science teams.
Data Infrastructure
- Build, implement and support data infrastructure extending the existing one.
- Maintain different data materializations to speed up reporting needs.
- Ensure reliable operations of various CDC and ETL pipelines.
- Define, track, and guarantee Service Level Objectives.
- Manage large and complex data sets that meet functional and non-functional business requirements.
- Watch over Data Lake's operating expenses and strive to reduce them.
- Design and oversee implementation of operational infrastructure related to the data pipelines
What’s in it for you
- Learn and evolve your skills using the latest and greatest technology tools in a rapidly growing company.
- Learn from the best people around you. We constantly challenge the status quo and invent new ways of building a great product.
- 100% remote. Work anywhere, whether it is remotely in the comfort of your home, in a shared co-working space, in an RV on the beach, or while being a nomad in another country.
- Work on challenging problems, innovate, and positively impact many people's lives while having fun doing it.
Required Qualifications
- Advanced speaking and writing English. Able to have a real-time conversation.
- 10+ years of full-time hands-on Data Engineering experience.
- 5+ years of full-time hands-on Tech Lead experience.
- 5+ years of full-time hands-on AWS experience.
- 3+ years of full-time hands-on Python experience.
- 3+ years of full-time hands-on SQL experience.
Technical Skills
- Programming Languages: Python, Java, Scala, C#, SQL
- Storage: HDFS, AWS S3
- Data Platforms: DataBricks, SnowFlake, RedShift
- Data Lakes: Apache Hudi, Delta Lake
- Relational Databases: Postgres, MySQL
- NoSQL Databases: DynamoDB, MongoDB
- Distributed Data Processing Engines: Apache Spark, Apache Flink, AWS EMR
- Query Engines: Apache Presto, AWS Athena
- Change Data Capture Solutions: Debezium, Qlik, Fivetran
- Business Intelligence: QuickSight, Tableau, DBT, Power BI
- Streaming: Kafka, RabbitMQ, ActiveMQ
- Data Governance: AWS LakeFormation
- Data Science and ML: NumPy, SciPy, Pandas, TensorFlow, PyTorch, SageMaker
- Storage Formats: Arrow, Parquet, AVRO
- Serverless Computing: AWS Lambda, Step Functions
- Other AWS Services: IAM, RAM, EC2
- Data Modeling & Architecture Patterns: Dimensional Modeling, Star Schema, Data Vault, Lake House Architecture
Soft Skills
- Communication: Effectively communicate with both technical and non-technical stakeholders. Clear explanations and cross-department collaboration
- Collaboration: Working closely with the mission-oriented product development teams where individual data engineers may be embedded.
- Problem-solving: Having an eye for detail and the ability to solve complex problems efficiently.
- Adaptability: Being adaptable to learn and integrate new tools and technologies as the tech landscape keeps evolving.
- Critical Thinking: Analyzing situations, making informed decisions, and anticipating challenges.
- Risk Management: Understanding risk factors related to data quality, security, and compliance.
- Negotiation: Ability to advocate for data engineering needs and influence decision-making
Management Skills
- Leadership: Guiding and motivating the team, setting goals, and ensuring alignment with organizational objectives.
- Project Management: Managing projects, timelines, and resources effectively.
- Strategic Thinking: Aligning data engineering efforts with broader business strategies.
- Cost Management: Managing data engineering infrastructure costs efficiently.
- Change Leadership: Guiding teams through organizational changes and technological shifts.
Experience
- Successful and applicable hands-on experience in data engineering including end-to-end design and implementation.
- Proven track record of managing or leading data engineering teams in designing and implementing efficient data models for large-scale datasets.
- Good understanding of network and data security architecture
- Good experience in data lake architecture, governance, and implementation
Nice to haves
- BS degree in engineering – computer science or related field
- Fintech or payments/banking product development experience
- Experience in working in a start-up environment
Frequently Asked Questions
What are your typical clients?
The majority of our clients are venture-backed startups at the growth stage. Usually, at this stage, the company already achieved a product-market fit and is looking to expand rapidly. That’s where we bring the best engineering practices, strong architecture, the latest technologies, and consistent processes to help companies scale.
What is the length of your engagements?
Like many of our engagements, this is a long-term opportunity that is expected to last for multiple years. It allows you to evolve your career with the client company taking on more responsibilities.
What’s your company size?
The Right Balance team is 60+ engineers going to 100+ by the end of the year. The current client size team is 165+ people. The timing is great to be a part of a rapidly growing team making meaningful contributions.
What are your core values?
Client First: we only win when our clients win. We treat client challenges as our own.
Ownership: we embrace responsibility, taking on challenges, getting them to completion, and enjoying getting things done.
Quality: we’re passionate about achieving quality outcomes by applying meticulous attention to detail.
