Software Engineer, Web Crawling

About this job

As a Web Crawler engineer, you'd be responsible for crawling the entire web. Basically build Google-scale crawling! Who You Are You have extensive experience building and scaling web crawlers, or would be excited to ramp up very quickly You have experience with some high performance language (C++, Rust, etc.) You are familiar with TypeScript, Playwright, modern web design, CDP (Chrome DevTools Protocol) You’re comfortable optimizing a system to an exceptional degree You care about the problem of finding high quality knowledge and recognize how important this is for the world What You Could Do Build a distributed crawler that can handle 100M+ pages per day Optimize crawl politeness and rate limiting across thousands of domains Design systems to detect and handle dynamic content, JavaScript…

This is a short summary.

Want to know if you're a fit? Check your CV against this role — free, in 30 seconds.

Most-requested skills in Switzerland

Based on 130 Switzerland vacancies that list requirements, these are the skills employers ask for most often.

• Python36% of postings
• Java33% of postings
• CI/CD33% of postings
• SQL30% of postings
• Agile/Scrum27% of postings
• Git24% of postings
• React22% of postings
• Kubernetes20% of postings

Similar jobs

• Frontend Developer Intern · ParetoLabs
• Software Engineering Intern for CAD/CAE/Tooling · Sika
• Lead Software Engineer · Zurich Soft Robotics GmbH
• Staff Software Engineer - Robotics Platform · Gravis Robotics
• Software Engineer - Expenses Team · BLP Digital AG
• ESG Digitalization Intern – Software & Data Engineering · Sika