Match score not available

Unlock Premium Benefits Today!
🚀 Elevate your career with priority access.
Stand out to top recruiters with Jobgether Priority Access:
  • ✅ Top Match: We help you identify job offers where you're a perfect fit.
  • ✅ Exclusive Referral Service: Connect directly with recruiters as a Jobgether Certified Premium User.
  • ✅ Personalized Feedback: Receive expert guidance on applications and have your profile manually reviewed.

Start Your Free Trial Now!

Start my free trial

Web Scraping Data Engineering Manager

72% Flex
Remote: 
Full Remote
Work from: 
United States

Offer summary

Qualifications:

Bachelor’s or higher in Computer science, engineering, analytics, or information technology., 10+ years data engineering experience, with 2+ in web scraping and 3+ managing a team., High proficiency in Python, experience with Celery, RabbitMQ, Django., Deep knowledge of Scrapy, Beautiful Soup, Selenium or similar framework., Familiarity with HTML, CSS, JavaScript; AWS, Azure, or GCP certification desirable..

Key responsabilities:

  • Develop advanced web scraping tools for efficient data extraction.
  • Implement robust monitoring to ensure accuracy and early issue detection.
  • Conduct quality testing and maintain high data standards.
  • Lead distributed data engineers, mentorship, issue resolution.
  • Align scraping activities with business objectives, assure timely project execution.
COMPLY logo
COMPLY Financial Services SME https://www.comply.com/
201 - 500 Employees
About COMPLY
At COMPLY, we pride ourselves on being the champion for compliance professionals. Merging technology, consulting and education, we help clients navigate the ever-changing regulatory environment. Our portfolio of firms includes ComplySci, RIA in a Box, National Regulatory Services (NRS) and illumis, whose more than 7,000 clients include some of the world’s largest financial institutions. Clients throughout our portfolio of firms enjoy access to our full suite of industry-leading governance, risk and compliance (GRC) consulting, technology, managed services, analytics and outsourcing solutions. Learn more at comply.com.
See more COMPLY offers

Job description

Logo Jobgether

Your missions

As the Data Engineering Manager for web scraping at COMPLY, you will play a pivotal role in the design and implementation of advanced web scraping tools and services. You will lead the development of workflows that enhance our data extraction capabilities, ensuring high accuracy and quality through diligent monitoring and testing. Under your guidance, our team will efficiently scale operations to meet increasing data demands. Your leadership will extend to mentoring in-house staff and managing offshore contractors, ensuring all activities align with COMPLY’s strategic goals.

Reporting directly to the Head of Data & Analytics, you will spearhead efforts to refactor our web scraping frameworks, significantly improving the speed, accuracy, and durability of our data processing. Your insights and deep technical expertise will be key in sustaining COMPLY’s competitive advantage and establishing new benchmarks in data excellence.

Job Responsibilities:
  • Robust System Design: Develop and refine advanced web scraping tools and services to efficiently extract and ingest data from diverse web sources.
  • Proactive Monitoring: Implement comprehensive monitoring systems to track the performance and health of web scraping operations, enabling early detection of issues before they affect data quality.
  •  Quality Assurance: Conduct rigorous testing partnering with QA to ensure the accuracy and reliability of data collected through web scraping. Implement quality control measures to maintain high standards.
  • Leadership: Lead a team of geographically distributed data engineers, providing mentorship and direction to enhance skills and foster a collaborative, innovative work environment.
  • Issue Resolution: Quickly identify and resolve technical issues and discrepancies in the data collection process, ensuring minimal disruption to operations.
  • Continuous Improvement and Innovation: Continuously evaluate and improve scraping processes and methodologies to increase efficiency and adapt to changing technological landscapes.
  • Strategic Planning: Collaborate with cross-functional teams to align web scraping activities with overall business objectives and strategic goals.
  • Project Execution: Ensure timely execution of project deliverables by overseeing project schedules and resource allocation.
  • Continuous Learning: Proactively stay current with advancements and trends in web scraping technologies to keep our practices in line with top industry standards.

    • Qualifications:
  • Education: Bachelor’s or higher in Computer science, engineering, analytics, or information technology or similar.
  • Experience: 10+ years in data engineering with 2+ years focused on web scraping. 3+ years managing a team.
  • Technical Expertise: High Proficiency in Python a must! Experience with Celery, RabbitMQ, and Django to manage existing web scraping processes. Understanding of Elasticsearch indices, complex queries, and search optimization
  • Web Scraping Framework: Deep knowledge in Scrapy, Beautiful Soup, Selenium or similar framework
  • Leadership: 3+ Experience in senior or lead roles, demonstrating growth and increased responsibility in data handling.
  • Web Technologies: Thorough understanding of HTML, CSS, and JavaScript to navigate complex websites.

    • Nice to Have:
  • Industry Knowledge: Experience in financial services, political contributions or other relevant industries
  • Certifications: AWS (preferred), Azure, or GCP. Data specific certification a plus.
  • Other Technologies: Kubernetes, Redis, Jenkins, Postgres, ETL, dbt, NoSQL, SQL
    • COMPLY is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, disability, sex, sexual orientation, gender identity, or national origin. Nothing in this job posting should be construed as an offer or guarantee of employment.

    Required profile

    Experience

    Industry :
    Financial Services
    Spoken language(s):
    English
    Check out the description to know which languages are mandatory.

    Hard Skills

    SeleniumWeb ScrapingData EngineeringPythonRabbitMQDjangoElasticsearchHTMLCSSJavaScriptAWS ECSAzureGoogle Cloud Platform (GCP)KubernetesRedisJenkinsPostgresETL DevelopmentdbtNoSQLSQL

    Soft Skills

    • Leadership
    • Team Collaboration
    Are you interested?

    Go Premium: Access the World's Largest Selection of Remote Jobs!

    • Largest Inventory: Dive into the world's largest remote job inventory. More than half of these opportunities can't be found on standard platforms.
    • Personalized Matches: Our AI-driven algorithms ensure you find job listings perfectly matched to your skills and preferences.
    • Application fast-lane: Discover positions where you rank in the TOP 5% of applicants, and get personally introduced to recruiters with Jobgether.
    • Try out our Premium Benefits with a 7-Day FREE TRIAL.
      No obligations. Cancel anytime.
    Upgrade to Premium

    Find more Data Engineer jobs

    SEE MORE JOBS
    Sign in or sign up with Google or LinkedIn