We are seeking a specialist who lives in the Network tab of DevTools. You will be responsible for designing, building, and maintaining the infrastructure that powers our data ingestion, ensuring our data pipeline delivers high success rates at scale.
What you will do:
Reverse Engineering: Dissect complex websites and mobile APIs (Android/iOS) to find hidden endpoints and efficiently extract data without rendering.
Infrastructure & Scale: Build and maintain scalable scrapers using Python that handle millions of requests daily.
Bypass Protections: Develop strategies to overcome anti-bot measures, including TLS fingerprinting, CAPTCHA solving, and header optimization.
Proxy Management: Orchestrate smart proxy rotation strategies.
Data Quality: Ensure the integrity and structure of harvested data, implementing automated validation.
Tooling: Maintain our internal scraping framework.
What you will do:
Reverse Engineering: Dissect complex websites and mobile APIs (Android/iOS) to find hidden endpoints and efficiently extract data without rendering.
Infrastructure & Scale: Build and maintain scalable scrapers using Python that handle millions of requests daily.
Bypass Protections: Develop strategies to overcome anti-bot measures, including TLS fingerprinting, CAPTCHA solving, and header optimization.
Proxy Management: Orchestrate smart proxy rotation strategies.
Data Quality: Ensure the integrity and structure of harvested data, implementing automated validation.
Tooling: Maintain our internal scraping framework.
Requirements:
Python Expertise: Strong proficiency in Python.
Network Fundamentals: Mastery of HTTP/S, TLS, cookies, and headers. You know exactly what happens in a TLS handshake and how to mimic it.
Database Knowledge: Experience with data storage (PostgreSQL, MongoDB, Redis) and message queues (Kafka, RabbitMQ, SQS).
Python Expertise: Strong proficiency in Python.
Network Fundamentals: Mastery of HTTP/S, TLS, cookies, and headers. You know exactly what happens in a TLS handshake and how to mimic it.
Database Knowledge: Experience with data storage (PostgreSQL, MongoDB, Redis) and message queues (Kafka, RabbitMQ, SQS).
This position is open to all candidates.
