view docs/assets/robots.txt @ 826:7f235e65ef5d

Allow PageSource to provide a custom assetor
author Ben Artin <ben@artins.org>
date Sun, 01 Jan 2017 18:02:13 -0500
parents 26e59f837558
children
line wrap: on
line source

# www.robotstxt.org/

# Allow crawling of all content
User-agent: *
Disallow: