Experience with Databricks batch processing/data lake design
3 plus years hands-on expertise in Databricks to design, build, and optimize high throughput batch data pipelines
Bachelors
Requirements:
Use technical knowledge to solve business problems
Serve as a subject matter expert and represent department on complex assignments
Evaluate technologies' effectiveness through requirements gathering, testing, and make recommendations for improvements
Collaborate with others to develop new data pipelines and migrate existing data platform solutions to the cloud
Job description
Pay Rate $60.00 - $70.00
Location: St.Louis,MO - REMOTE Position
Duration: 6 months
Senior Data Engineer, SME
Visa Sponsorship not required - Potential Contract To Hire
Competencies: 10+ years experience
Digital : Microsoft Azure
Digital : Databricks
Roles & Responsibilities:
As a Senior Engineer, you will be responsible for using your technical knowledge to solve business problems. We are looking for a talented individual that can serve as a subject matter expert in their area of focus and represent their department on complex assignments. You will be responsible for evaluating elements of technologies effectiveness through requirements gathering, testing, research and investigation and make recommendations for improvements that result in increased quality and effectiveness. You will be required to listen to and evaluate customer needs to determine and provide high quality solutions that align with customer expectations.
In this role on the Cloud Data Engineering team, you will be a part of a team responsible for migrating existing data platform solutions to the cloud. You will also collaborate with others to develop new data pipelines. As a data engineer in the Cloud Data Engineering team, you will need to work in a data first mindset. You will use Microsoft Azure Cloud PaaS technologies to engineer our data solutions in a way that allows for us to optimize our information, make better decisions, and meet our customer’s needs. If you have a passion for engineering, working with massive amounts of data and empowering smart business decisions, this is the role for you.
Technical/Functional Skills
• Experience with Databricks batch processing/data lake design experience
• 3+ yrs hands on expertise in Databricks to design, build, and optimize high throughput batch data pipelines
• 3+ yrs hands-on expertise in Databricks lakehouse architectures on cloud platforms (Azure/AWS)
• Experience with Microsoft Azure platform technologies like Databricks applications, Event Hub, Data Factory, Scala, Synapse Analytics, Delta Lake, Cosmos DB, or DevOps
• Must have good troubleshooting and problem-solving skills, including debugging multi-threaded applications
• Experience with configuration management and build automation capabilities such as Git/Jenkins
• Must have good oral and written communication skills to effectively communicate with various IT teams and business representatives
• Must be able to both collaborate in a team-oriented environment and work independently with direction
Preferred:
• Experience with Big Data solutions and no SQL DBMSs
• Experience with Azure, AWS, or other Public Cloud environments
• Prior experience leading large-scale projects
• Knowledge and working experience with Agile methodologies
Experience Required 8-12 Years
Generic Managerial Skills
• Must have good oral and written communication skills to effectively communicate with various IT teams and business representatives
• Must be able to both collaborate in a team-oriented environment and work independently with direction
Diverse Lynx LLC is an Equal Employment Opportunity employer. All qualified applicants will receive due consideration for employment without any discrimination. All applicants will be evaluated solely on the basis of their ability, competence and their proven capability to perform the functions outlined in the corresponding role. We promote and support a diverse workforce across all levels in the company.
