Publishers have asked Common Crawl to stop collecting their material for datasets that may be used by downstream researchers and AI developers. Common Crawl responded by publishing an opt-out registry and reiterating that site owners can use standard crawl controls for its crawler, CCBot.
The issue is often described as one switch for “AI crawlers.” In practice, each crawler, dataset and search product can have different identifiers, policies and consequences.
What Common Crawl is
Common Crawl is a non-commercial foundation that maintains an open repository of web crawl data. It says it crawls billions of pages each month and makes the datasets available for research and other downstream use.
Because the dataset can be reused by organisations Common Crawl does not control, a publisher's decision is not only about server traffic. It can also concern licensing, archiving and future data use.
What the opt-out registry does
Common Crawl says it created a public registry of legal opt-out requests so downstream users can see which content owners have asked for exclusion. It states that the registry will be updated as new requests arrive and provides a contact route for legal requests.
The registry is a transparency mechanism. It should not be assumed to erase content from every historical crawl or from copies already obtained by third parties.
How robots.txt fits in
Common Crawl encourages publishers to use robots.txt to control CCBot. A targeted rule can address that crawler without blocking Googlebot:
User-agent: CCBot
Disallow: /
This is an example, not a recommendation for every site. Before changing a live robots file, verify Common Crawl's current crawler documentation and test that search crawlers remain allowed.
Robots.txt is an access instruction for compliant crawlers. It is not authentication, copyright licensing, or a deletion mechanism. Google similarly cautions that respectable crawlers may follow robots.txt while others may ignore it; private material should be protected through real access control.
Do not confuse CCBot with Googlebot
Blocking CCBot does not inherently remove a site from Google Search. Blocking Googlebot can prevent Google from crawling content and can damage search visibility.
Keep the purposes separate:
| Goal | Typical control | Important limitation |
|---|---|---|
| Stop a named crawler requesting pages | User-agent rule in robots.txt | Depends on crawler compliance |
| Keep a public page out of Google Search | noindex while Googlebot can crawl it |
Does not make the page private |
| Keep confidential content inaccessible | Authentication or network access control | May remove public discovery entirely |
| Seek exclusion from Common Crawl datasets | Common Crawl's published request process | Historical and downstream copies require separate analysis |
Adding noindex to robots.txt is not supported by Google. Blocking Googlebot also prevents it from seeing a page-level noindex directive, which can create an unintended result.
A publisher decision framework
1. Inventory crawler access
Review server logs and the current robots file. Verify crawler identities where the provider publishes a method; user-agent strings can be spoofed.
2. Define the content policy
Separate open marketing pages, licensed editorial content, customer-only material and confidential information. These categories should not rely on one blanket rule.
3. Assess business trade-offs
Search discovery, research reuse, AI training, citation and server cost are different outcomes. Decide which are acceptable for each content group.
4. Implement narrow controls
Target the documented crawler token. Avoid broad User-agent: * changes unless the business truly intends to block all compliant crawlers.
5. Monitor and record
After a change, inspect logs and Search Console for unintended effects. Save the date, owner and reason for each rule so future teams do not remove or broaden it without context.
What Symaxx website owners should do
For most service-business marketing pages, search discoverability remains valuable. A blanket block is unlikely to solve a specific licensing concern and could create collateral damage if implemented incorrectly.
Publishers with original paid or licensed material should get legal advice on the rights and jurisdictions involved, then implement crawler-specific technical controls as one part of that policy. Do not publish confidential information and assume robots.txt will protect it.
Sources
- Common Crawl: Foundation Opt-Out Registry
- Common Crawl: CCBot
- Google Search Central: Introduction to robots.txt
- Google Search Central: Block search indexing with noindex