https://bascht.com/tags/robotstxt/Recent content in robotstxt on bascht.comHugo -- gohugo.iode-deThu, 20 Jun 2013 00:00:00 +0000https://bascht.com/tech/2013/06/20/emulate-a-robots.txt-file-with-a-simple-nginx-directive/Thu, 20 Jun 2013 00:00:00 +0000https://bascht.com/tech/2013/06/20/emulate-a-robots.txt-file-with-a-simple-nginx-directive/If you use nginx as a reverse proxy, you might want to emulate a simple robots.txt file just to be sure that Googlebot doesn’t traverse into each and every location. So why bother serving those bytes from a file when you can answer requests directly from nginx: location /robots.txt { return 200 "User-agent: *\nDisallow: /"; }