Fix agent-friendly documentation issues#159
Merged
JakeSCahill merged 15 commits intomainfrom Mar 5, 2026
Merged
Conversation
✅ Deploy Preview for redpanda-documentation ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
- Fix broken links (API URLs, removed dead Twitter/X link, re-add sitemap.xml) - Add markdown access documentation (indexify convention, content negotiation) - Add comprehensive AI agents and Agentic Data Plane documentation section - Convert llms.txt to llms.adoc page for dynamic URL generation - Configure convert-llms-to-txt extension for environment-aware URLs - Production: https://docs.redpanda.com - PR previews: https://deploy-preview-NNN--redpanda-documentation.netlify.app - Local: http://localhost:8000 - Generate llms-full.txt with all latest version markdown (12MB, 1583 pages) Requires docs-extensions-and-macros#173 to be merged and published to npm before this PR can be merged. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
a9b17ed to
4776081
Compare
- Add afdocs as dev dependency (^0.3.4) - Add npm script to run afdocs checks (excludes HTML-specific checks) - Create GitHub Actions workflow to run on PR deploys - Exclude HTML-specific checks: page-size-html, content-start-position - Include all other checks: llms.txt validation, markdown support, URL stability, caching, and authentication checks The workflow waits for Netlify deploy preview and runs checks against the deployed site to ensure agent-friendly documentation standards. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Updated all links to use .md extensions for direct markdown access: - Top-level components: /current/ → /current.md - All other pages: /path/page → /path/page/index.md This ensures AI agents get direct markdown links and makes afdocs checks pass: - llms-txt-links-markdown - markdown-url-support - content-negotiation All links now point to markdown files via Netlify redirects and the indexify convention. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Add commit_ref parameter to wait-for-netlify action to ensure it waits for the deploy of the specific commit that triggered the workflow - Add continue-on-error: true to afdocs step so workflow doesn't fail CI when some checks don't pass (informational only) - Improve PR comment to show pass/fail status and commit SHA - Remove redundant ternary for PR check in comment This fixes the timing issue where the action would run against an old deploy instead of waiting for the new one to finish building. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This version includes the fix for URL encoding in llms.txt where em dashes were breaking Netlify deploy preview URLs. Fixes: - Convert em dashes back to double hyphens in URLs - Resolves afdocs check failures for links-resolve, markdown-url-support, and content-negotiation
This version includes the fix for zero-width spaces and other invisible Unicode characters in llms.txt URLs. Fixes: - Strip zero-width space (U+200B) - Strip zero-width non-joiner (U+200C) - Strip zero-width joiner (U+200D) - Strip zero-width no-break space (U+FEFF) This should resolve the remaining broken links in afdocs checks.
- Fix console config URL: /current/console/config/index.md → /current/console/config.md - Remove /index.md from API doc URLs (HTML-only OpenAPI specs, no markdown) - Fix redpanda-labs shortcut: /redpanda-labs.md → /redpanda-labs/home/index.md - Fix version shortcut: /24.1.md → /24.1/home/index.md This resolves the 7 broken links identified by afdocs checks.
- Fix console config path: /current/console/config.md → /current/manage/console/index.md (path changed) - Remove Tutorials & use cases section (redpanda-labs doesn't have markdown version) All remaining URLs now resolve correctly with markdown support.
API docs do support markdown format at /api/doc/{name}.md URLs.
Added back:
- /api/doc/admin.md
- /api/doc/schema-registry.md
- /api/doc/http-proxy.md
- /api/doc/cloud-controlplane.md
API docs .md files were being returned with text/plain content-type from Bump.sh. Updated edge function to detect .md requests and set correct text/markdown; charset=utf-8 content-type. This fixes the content-negotiation warning in afdocs checks.
Feediver1
reviewed
Mar 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes agent-friendly documentation issues identified by the afdocs checker and adds comprehensive AI agent documentation.
https://deploy-preview-159--redpanda-documentation.netlify.app/llms-full.txt
https://deploy-preview-159--redpanda-documentation.netlify.app/llms.txt
Problem
The afdocs checker identified several issues:
docs.redpanda.comURLs prevented testing on deploy previewsSolution
1. Fixed Broken Links
proxy-api-docs.jsedge function2. Added Markdown Access Documentation
index.md)Accept: text/markdownheaders3. Added AI Agent & Agentic Data Plane Documentation
4. Implemented Dynamic llms.txt Generation
Key Innovation: Converted
llms.txttollms.adocpage with environment-aware URL generation via newconvert-llms-to-txtAntora extension.How it works:
llms.adocas a regular Antora page with{site-url}attribute placeholdersconvert-to-markdownextension converts HTML page to markdownconvert-llms-to-txtextension:PREVIEWenv var andplaybook.site.urlpage.markdownContentsllms.txtat site rootllms-full.txtwith all markdown from latest versions (~12MB)Environment-aware URLs:
https://docs.redpanda.comhttps://deploy-preview-NNN--redpanda-documentation.netlify.apphttp://localhost:8000This enables CI/CD testing with afdocs on deploy previews before merging to production.
Changes
Modified Files
home/modules/ROOT/pages/llms.adoc- New page (converted from llms.txt)antora-playbook.yml- Added convert-llms-to-txt extensionlocal-antora-playbook.yml- Added convert-llms-to-txt extension (local path)preview-antora-playbook.yml- Added convert-llms-to-txt extensionRemoved Files
home/modules/ROOT/attachments/llms.txt- Replaced by llms.adoc pageTesting
Local testing with netlify dev:
Link resolution:
afdocs results:
Related