AI and Robots.txt

So what you can do as a publisher for these good crawlers is rather than trying to deal with each crawler individually, there's gazillions of them, is you can say which uses do you want and which uses do you not want? And then you can put that into your robots.txt file. What 51Degrees, do is if you go to the website, 51degrees.com, then the developer's menu, and then you'll see robots.txt. You can go in there, select the usages that you're happy with, you can see all the different usages and the table on the right-hand side, put in your email address and then we'll email you whenever crawlers change with an updated robots.txt that reflects those rather high-level preferences.

All 51Degrees videos