Build systems for extracting, connecting, retrieving, and reasoning over knowledge from the web to enable AI agents to answer questions with precision. Requires experience with complex distributed backend systems and petabyte-scale data pipelines.
Builds and scales Google-scale web crawlers to handle 100M+ pages daily, optimizing for politeness, dynamic content, anti-bot measures, and prioritization. Requires experience with high-performance languages like Rust/C++ and tools like TypeScript/Playwright.
Builds and optimizes backend systems for massive-scale web crawling, data processing pipelines handling millions of documents daily, and high-performance vector databases in Rust for AI search applications. Requires expertise in high throughput, low latency systems.
Build systems for extracting, connecting, retrieving, and reasoning over knowledge from the web to enable AI agents to answer questions with precision. Requires experience with complex distributed backend systems and petabyte-scale data pipelines.
Builds and scales Google-scale web crawlers to handle 100M+ pages daily, optimizing for politeness, dynamic content, anti-bot measures, and prioritization. Requires experience with high-performance languages like Rust/C++ and tools like TypeScript/Playwright.
Builds and optimizes backend systems for massive-scale web crawling, data processing pipelines handling millions of documents daily, and high-performance vector databases in Rust for AI search applications. Requires expertise in high throughput, low latency systems.