view docs/assets/robots.txt @ 1169:978ed6deea91

tests: Add ability to test different expected outputs based on Python version. Similar to the previous change, there was also a change in what gets escaped by `urllib.parse.quote` (as in: the tilde sign ('~') stopped being escaped). So now we need to be able to test different outputs in the tests, yay again.
author Ludovic Chabant <ludovic@chabant.com>
date Fri, 04 Oct 2019 11:15:11 -0700
parents 26e59f837558
children
line wrap: on
line source

# www.robotstxt.org/

# Allow crawling of all content
User-agent: *
Disallow: