Generate llms.txt
Enter a website URL to automatically generate a spec-compliant llms.txt file
How it works
1.Crawl
Discovers pages via sitemap, homepage links, and BFS link-following
2.Classify
Categorizes pages into sections like Docs, API, Blog, and Examples
3.Enhance
Uses AI to generate summaries and descriptions for each page
4.Generate
Assembles a spec-compliant llms.txt with structured sections