mirrors/bookwyrm
1
0
Fork 1
mirror of https://github.com/bookwyrm-social/bookwyrm.git synced 2025-06-05 15:28:52 +00:00
Code Issues Projects Releases Wiki Activity

Merge pull request #2419 from bookwyrm-social/robots

Browse source 
Expand robots.txt
This commit is contained in:
Mouse Reeve 2022-11-19 10:24:55 -08:00 committed by GitHub
commit 2adcbdea2f
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
1 changed files with 8 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

8
bookwyrm/templates/robots.txt
View file

@ -73,6 +73,14 @@ User-agent: PetalBot
Disallow: / Disallow: /
User-agent: DataForSeoBot
Disallow: /
User-agent: YisouSpider
Disallow: /
User-agent: * User-agent: *
Crawl-delay: 10
Disallow: /static/js/ Disallow: /static/js/
Disallow: /static/css/ Disallow: /static/css/