Commit Graph

1 Commits

Author SHA1 Message Date
Chris Wilson ad4451276d
WebHost: Add `robots.txt` to WebHost (#3157)
* Add a `robots.txt` file to prevent crawlers from scraping the site

* Added `ASSET_RIGHTS` entry to config.yaml to control whether `/robots.txt` is served or not

* Always import robots.py, determine config in route function

* Finish writing a comment

* Remove unnecessary redundant import and config
2024-04-20 20:58:56 -04:00