Robots.txt

So if you really want to know about crawlers, bots, and AIs, then you need three new properties from 51Degrees. We've got crawler usages, which tells you what the crawler intends to use the content for, whether that's AI, training, RAG, or search, among others. You've then got crawler URL, which tells you the information the crawler provides about their crawler and what they use the data for to support the prior property. And then also we've got the product tokens, which comply with the IETF RFC on robots.txt. So go and check out the 51Degrees website, its developers, and then robots.txt, and get started today.

All 51Degrees videos