DuckDuckGo is widely regarded as a privacy-focused search engine that actively protects users’ personal information by blocking third-party trackers and minimizing data collection. It operates its own crawlers to access web content, improve search results and offer a secure search experience.
DuckDuckGo’s crawlers follow the standard robots.txt directives. Any changes made to these directives may take up to 72 hours to be reflected in their system.
The search engine also provides documentation about its web crawlers, including User-Agent identifiers, their intended purposes and usages. This helps in understanding how DuckDuckGo interacts with web content and the way the content may be used.
Exploring crawlers’ usage
DuckDuckBot
DuckDuckBot is DuckDuckGo’s primary web crawler for crawling web pages in its search results. It is used for discovering web content that gets included in the results. Hence, it has been identified that this crawler is used only for “Search” purposes.
Blocking DuckDuckBot will prevent a website from appearing in their search results, reducing its visibility on the platform.
The full User-Agent is:
DuckDuckBot/1.1; (+http://duckduckgo.com/duckduckbot.html)
This crawler is not associated with AI and therefore, the IsArtificialIntelligence property is set to “False”.
DuckAssistBot
DuckAssistBot is another web crawler used by DuckDuckGo to support and provide AI-assisted features by generating answers.
Blocking DuckAssistBot will prevent web content from being accessed for the use in AI features and citations of the answers.
This crawler is not used to train AI models. Hence, it has been identified that this crawler is used only for “Input” purposes.
The full User-Agent is:
DuckAssistBot/1.2; (+http://duckduckgo.com/duckassistbot.html)
This crawler is associated with AI and therefore, the IsArtificialIntelligence property is set to “True”.
Robots.txt Generator
Use 51Degrees to work with crawler usages rather than tracking individual crawlers and AIs. Check out the free Robots.txt Generator today.
Try our Robots.txt GeneratorAI Treatment
Want to know more about how to handle AIs including options that go beyond robots.txt? Check out our guide to the good, the bad, and the ugly.
AI Solutions