monorail
/
Archipelago
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
Archipelago/WebHostLib/static/robots_file.txt

21 lines
275 B
Plaintext
Raw Normal View History

WebHost: Add `robots.txt` to WebHost (#3157) * Add a `robots.txt` file to prevent crawlers from scraping the site * Added `ASSET_RIGHTS` entry to config.yaml to control whether `/robots.txt` is served or not * Always import robots.py, determine config in route function * Finish writing a comment * Remove unnecessary redundant import and config
2024-04-21 00:58:56 +00:00
User-agent: Googlebot
Disallow: /
User-agent: APIs-Google
Disallow: /
User-agent: AdsBot-Google-Mobile
Disallow: /
User-agent: AdsBot-Google-Mobile
Disallow: /
User-agent: Mediapartners-Google
Disallow: /
User-agent: Google-Safety
Disallow: /
User-agent: *
Disallow: /