Commit graph

3 commits

Author SHA1 Message Date
Zed
138826fb4f Fix Twitterbot rule in robots.txt 2022-06-04 17:55:35 +02:00
Zed
dfb2519870 Explicitly allow Twitterbot to generate previews 2022-06-04 17:52:47 +02:00
minus
c543a1df8c
Block search engines via robots.txt (#631)
Prevents instances from being rate limited due to being senselessly
crawled by search engines. Since there is no reason to index Nitter
instances, simply block all robots. Notably, this does *not* affect link
previews (e.g. in various chat software).
2022-06-04 17:48:25 +02:00