Details

We'll use your email to send you a robots.txt file that implements the usage choices you select now.

We'll also email you an updated robots.txt whenever we spot new crawlers or usage changes that impact your choices.

We might also send you information related to our complimentary services.

You can unsubscribe at any time via a link in the email.

Please read our Privacy policy before submitting.

Handling Conflicts

Sometimes your choices can't be addressed via robots.txt alone. For example a single crawler that uses content for both general search and AI training. The robots.txt we generate will allow such crawlers and include a warning in the comments indicating the conflict and where possible a URL for you to contact the crawler operator to request they restrict their use of your content via contractual agreement.

Where we observe such conflicts from many users we may attempt to contact the operator to request they provide a method of resolving the conflict via robots.txt.

Usage Choices