Mercurial > piecrust2
view docs/assets/robots.txt @ 483:64e1cd71b30b
serve: Try to serve taxonomy pages after all normal pages have failed.
author | Ludovic Chabant <ludovic@chabant.com> |
---|---|
date | Tue, 21 Jul 2015 22:42:47 -0700 |
parents | 26e59f837558 |
children |
line wrap: on
line source
# www.robotstxt.org/ # Allow crawling of all content User-agent: * Disallow: