8000 [AI, RAG, Docs] generate and expose llms.txt and llms-full.txt files on the docs website · Issue #60434 · angular/angular · GitHub
[go: up one dir, main page]

Skip to content

[AI, RAG, Docs] generate and expose llms.txt and llms-full.txt files on the docs website #60434

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
pawel-twardziak opened this issue Mar 18, 2025 · 18 comments
Assignees
Labels
area: docs-infra Angular.dev application and infrastructure
Milestone

Comments

@pawel-twardziak
Copy link
pawel-twardziak commented Mar 18, 2025

Which @angular/* package(s) are relevant/related to the feature request?

No response

Description

AI models are typically not up-to-date with the latest information, as they are periodically trained on snapshots of data.

As an Angular developer who uses AI tools for programming, I want the AI to be aware of the latest Angular releases so that I can generate modern code that leverages the newest features available in Angular.

Concept: https://llmstxt.org/

Exapmle: https://docs.stripe.com/llms.txt

This feature would greatly improve DX for Angular developers.

Proposed solution

Generate and expose the llms.txt and llms-full.txt files on the documentation website so that developers can use them as context for AI tool prompts.

Alternatives considered

Provide simple crawler-readable markdown (.md) files for each topic within the documentation.

@pawel-twardziak
Copy link
Author

I'm very interested in this feature and would love to contribute if you need any help! 😺

@psmyrdek
Copy link

Keeping my fingers crossed for this one - great idea!

@JeanMeche
Copy link
Member

This is an interesting idea.

Which usecases would be addressed by that providing that file ?
For the tools that are already able to crawl our docs, have you noticed some limitations ?

@JeanMeche JeanMeche added the area: docs-infra Angular.dev application and infrastructure label Mar 18, 2025
@ngbot ngbot bot added this to the needsTriage milestone Mar 18, 2025
@pawel-twardziak
Copy link
Author

@JeanMeche LLMs files are becoming more and more the standard for RAG.

I've tried some online crawlers and the results is, to say the least, not satisfactory IMHO.
E.g. https://www.firecrawl.dev/.

Resources:

  1. https://thedaviddias.medium.com/getting-started-with-llms-txt-226df8012257
  2. https://thedaviddias.medium.com/the-llms-txt-guide-you-need-for-your-next-project-f9f55a237bae
  3. https://llmstxthub.com/
  4. https://directory.llmstxt.cloud/
  5. https://llmstxt.site/

@pawel-twardziak
Copy link
Author
pawel-twardziak commented Mar 27, 2025

Hi @JeanMeche Any conclusions on this matter? 😺

@pawel-twardziak
Copy link
Author

yet another reference :) geromegrignon/angular-caniuse#16

@pawel-twardziak
Copy link
Author

https://llmstxt.org/

@geromegrignon
Copy link
Contributor

@pawel-twardziak
Copy link
Author

Thank you @geromegrignon - that's a great article!

8000

@geromegrignon
Copy link
Contributor

Another possible way: https://repomix.com/

@pawel-twardziak
Copy link
Author

Nice @geromegrignon thanks. That is kinda similar to https://gitingest.com/. But Repomix seems to be AI dedicated

@pawel-twardziak
Copy link
Author

But both of them are still not the llm files for Angular docs website

@AndrewKushnir AndrewKushnir self-assigned this Apr 18, 2025
@markgoho
Copy link
Contributor

fyi: https://github.com/google/A2A/blob/main/llms.txt

used with https://gitmcp.io/ it allows models to grok the codebase much more quickly and allow answers about API usage, best practices, source code to come from a tool, rather than the model's base knowledge or limited understanding of only how you might have used the framework in your codebase

@pawel-twardziak
Copy link
Author

@markgoho thanks! https://gitmcp.io/ seems really cool!

@JeanMeche
Copy link
Member

Hi there, I'm back on the topic.

Tbh, I'm having a hard time to find credible sources on the value of llms.txt files.
Beside hype driven blog post, I couldn't find constructive arguments for them.

Cursor for example, suggest to directly crawl the doc pages : https://docs.cursor.com/context/@-symbols/@-docs
Maybe for MCPs but again, couldn't find incredible recommendations on for those.

@manudss
Copy link
manudss commented May 16, 2025

@JeanMeche Cursor as file llm.txt: https://docs.cursor.com/llms-full.txt

You can find the list of llm.txt files on this website: https://llmstxt.site/

The advantage of having this, especially the full version, which contains all the documentation, is that we could give it to a llm, as a source, so that it gives us answers based on the Angular documentation.

Many llm chat applications now offer the possibility of providing a list of files to answer based only on these documents. To be sure to have reliable documentation.

This is the case with mistral AI, or Google notebook LM. I would like to be able to include all the Angular documentation in these tools, but I don't have a complete file (text or PDF) of the Angular documentation.

@AndrewKushnir
Copy link
Contributor

Hello everyone! Quick update: we've added a couple of files this week that should help with providing context to LLMs (thanks @JeanMeche!):

We'll be adding links to those files to the necessary angular.dev page(s) shorty.

As the next step, we plan to look into having more focused files (e.g. a file that collects the content about the template expression syntax, a file with signal APIs, etc), but we don't have any ETAs at this moment.

I will close this ticket as complete, please create new tickets if you have feedback or have proposals on how we can improve the content of those files.

Thank you.

@pawel-twardziak
Copy link
Author

@AndrewKushnir thank you! I was waiting for that! <3

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area: docs-infra Angular.dev application and infrastructure
Projects
None yet
Development

No branches or pull requests

7 participants
0