Repository navigation
Open the site to crawlers; reserve only the gated full chapters - #67
Merged
Merged
Conversation
Crawlers can only reach the public surface (pages, free previews, stories, /agents); the full chapters sit behind the reader gate and a crawler is redirected to it. So drop the per-crawler blocks, set Content-Signal to ai-train=yes, disallow /read/*/full, and make the TDM file reserve only those full chapters. Terms, /license, /agents and the manifest say the same: surface material is open to crawlers including training, as an additional permission beyond the licence; commercial training on the full text still needs agreement. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Author's direction: let crawlers in, since they can only reach surface material (the public pages, the free previews, the stories and /agents) and the full chapters sit behind the reader gate (a crawler is redirected to it; checked with a GPTBot user agent).
Changes
Content-Signal: search=yes, ai-input=yes, ai-train=yes; addsDisallow: /read/*/fullso crawlers do not waste requests on the gate./read/*/full; every other path istdm-reservation: 0; the per-page meta matches (1 only on full-chapter pages).Worth knowing: the full text is also public in the open repository, so the gate does not stop a determined scraper; this states the terms rather than locking anything. A lawyer should read the AI wording together with the voices question.
Tests 51 pass; verify 59 invariants hold.
🤖 Generated with Claude Code