If you’re dealing with AI crawler chaos hitting your servers, then LLMs.txt might be your new best friend.
Here’s the deal
Jeremy Howard proposed this simple file that lives at /llms.txt in your root directory. Think of it as a friendly guide for AI models, not a bouncer like robots.txt. It uses basic Markdown to point crawlers toward your money pages.
The Reality Check
If your shared hosting is getting hammered (and Cloudflare says AI traffic jumped 305% for GPTBot), then you’re probably feeling the pain. Imagine if it hit 30TB of bot bandwidth in a month – that’ll crush any stack.
What llms.txt does well
It gives AI tools a clean map. It points to the pages you want surfaced in AI answers. It reduces guesswork when a model tries to assemble context about your services. It is easy to update.
Where llms.txt falls short
It does not block crawlers. It does not throttle traffic. Some bots will ignore it. You still need controls in robots.txt, your CDN, and your server.
Quick Implementation
Drop this in your root as llms.txt
# HypeX Sri Lanka
HypeX Digital delivers web design, SEO, social media, and paid ads with a focus on measurable return.
## Key service pages
- https://hypesrilanka.com/web-design-sri-lanka/
- https://hypesrilanka.com/services/social-media-management/
- https://hypesrilanka.com/services/online-advertising/
[... your key URLs]For WordPress Developers
If you’re running WordPress, then Yoast SEO handles this automatically now. Enable it in Site Features, switch to manual selection, and add your URLs. Done.
If you prefer manual control, then just FTP the file to your WordPress root alongside wp-config.php.
The Defense Stack
If LLMs.txt is your guidance layer, then you still need the hard blocks:
Robots.txt
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /Nginx blocks
if ($http_user_agent ~* "GPTBot|ClaudeBot|PerplexityBot") {
return 403;
}Cloudflare rules: Set up rate limiting or blocks based on user agent strings.
The Monitoring Game
If you’re seeing traffic spikes, then track user agents in your logs. Watch for 5xx errors during bot bursts. Review weekly and adjust your blocks accordingly.
Before you add it, question the idea.
Assumptions to test
LLMs follow llms.txt. Some do, some do not. Adoption is growing in SEO tools and site builders. It is still early. Treat it as useful guidance, not a hard rule.
LLMs.txt will cut server load. It helps focus crawls, but it does not stop bad actors or heavy crawlers on its own. You also need robots.txt rules, rate limits, and firewall rules.
All AI crawlers respect rules. Recent tests show some actors ignore robots.txt and hide behind fake user agents. Plan for that risk.
Bottom Line
If good bots follow your LLMs.txt guidance, then they’ll pull better context about your services. If bad bots ignore everything, then your firewall rules catch them. It’s defense in depth, not a magic bullet.
Keep the file under 100KB, update it monthly, and monitor your server response times. If you’re still getting crushed after implementing all layers, then it’s time for better hosting or a CDN upgrade.
