update robots.txt to prevent spidering

seanlinsley · seanlinsley · commit 07996ab14e44 · 2013-12-03T14:23:26.000-06:00
this is a precaution, as I'm not sure if the spam links are being
scraped by spiders
diff --git a/public/robots.txt b/public/robots.txt
@@ -1,5 +1,5 @@
 # See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file
 #
 # To ban all spiders from the entire site uncomment the next two lines:
-# User-Agent: *
-# Disallow: /
+User-Agent: *
+Disallow: /

Original file line number	Diff line number	Diff line change
`@@ -1,5 +1,5 @@`
`1`	`1`	`# See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file`
`2`	`2`	`#`
`3`	`3`	`# To ban all spiders from the entire site uncomment the next two lines:`
`4`		`-# User-Agent: *`
`5`		`-# Disallow: /`
	`4`	`+User-Agent: *`
	`5`	`+Disallow: /`