From c543a1df8cf438d04ed47936069639c0543fbece Mon Sep 17 00:00:00 2001
From: minus <github@mnus.de>
Date: Sat, 4 Jun 2022 15:48:25 +0000
Subject: [PATCH] Block search engines via robots.txt (#631)

Prevents instances from being rate limited due to being senselessly
crawled by search engines. Since there is no reason to index Nitter
instances, simply block all robots. Notably, this does *not* affect link
previews (e.g. in various chat software).
---
 public/robots.txt | 2 ++
 1 file changed, 2 insertions(+)
 create mode 100644 public/robots.txt

diff --git a/public/robots.txt b/public/robots.txt
new file mode 100644
index 0000000..1f53798
--- /dev/null
+++ b/public/robots.txt
@@ -0,0 +1,2 @@
+User-agent: *
+Disallow: /