Crawlers
Monday, March 30, 2026 5:30:00 PM
Apple Web Crawler
How Applebot and Apple crawlers shape visibility across Apple’s ecosystem
This blog explains how Apple’s web crawlers, including Applebot and Applebot-Extended, work across its ecosystem to surface content in Siri, Spotlight, and Safari. It outlines the different crawlers, their purposes—such as search indexing, feed updates, and AI training—and how enabling them can improve content visibility, traffic, and control over how content is used, particularly in generative AI.
Tuesday, March 3, 2026 5:30:00 PM
Perplexity's robots and crawlers
Research into Perplexity's approach to scraping web content
Find out how we classify Perplexity's crawler scrape web content.
Friday, February 13, 2026 5:30:00 PM
Meta's robots and crawlers
Research into Meta's approach to scraping web content
Find out how we classify Meta's crawlers and AI content scrapers.