№ 1010 · Series B startup, San Francisco · Remote · San Francisco, USA
Web Crawler Engineer at a San Francisco Series B startup building an AI/LLM search engine — remote or relocation to the US
The Web Crawler Engineer will build a distributed crawler processing 100M+ pages a day for a next-generation search engine.
About the company
We are hiring for an ambitious Series B startup from San Francisco
Our client is building a next-generation search engine for AI/LLM, regularly runs batch jobs on clusters of >1000 machines and develops production services for processing terabytes of text.
- Fully in-house development — from the search engine to the vector DB
- A specialized architecture for generating structured answers for LLMs
- The system is designed for AI-first interactions, not for traditional "human browsing"
What matters
- We need a person who has already built or optimized a web crawler at search engine scale (Yandex, Google, Bing, etc.)
- Has experience with any high-performance language (Rust/C++, etc.)
- Fluent English is important
- Experience with big data and ML/LLM, as well as experience in early-stage/hyper-growth startups, will be a plus
What you will do
- Building a distributed crawler to process 100M+ pages/day
- Complex anti-bot systems and bypassing protection
- Dynamic content and JS rendering
- Developing algorithms for intelligent crawl scheduling for a distributed crawler
Stack
Rust, AWS, Kubernetes, Python(ML)
Terms
- Work format: remote OR relocation to San Francisco — your choice
- $150k-300k + equity (negotiable)