mirrors/takahe
1
1
Fork 0
mirror of https://github.com/jointakahe/takahe.git synced 2024-11-22 07:10:59 +00:00
Code Issues Projects Releases Packages Wiki Activity

Instruct robots to ignore proxy (#413)

Browse source
This commit is contained in:
Corry Haines 2023-01-14 10:15:18 -08:00 committed by GitHub
parent 2675ed69ee
commit ade954c2cd
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
1 changed files with 3 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

3
static/robots.txt
View file

@ -3,3 +3,6 @@ User-Agent: *
# Don't allow any bot to crawl tags. # Don't allow any bot to crawl tags.
Disallow: /tags/ Disallow: /tags/
Disallow: /tags/* Disallow: /tags/*
# Don't allow bots to crawl through the proxy
Disallow: /proxy/*