Mercurial > piecrust2
annotate docs/assets/robots.txt @ 461:b015e38d4ee1
internal: Handle data serialization more under the hood.
Implement a new queue class that handles pickling of objects, add option
(currently disabled) to also do zlib compression before sending bytes.
author | Ludovic Chabant <ludovic@chabant.com> |
---|---|
date | Sat, 11 Jul 2015 17:51:56 -0700 |
parents | 26e59f837558 |
children |
rev | line source |
---|---|
243
26e59f837558
docs: Add embryo of a documentation website.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
1 # www.robotstxt.org/ |
26e59f837558
docs: Add embryo of a documentation website.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
2 |
26e59f837558
docs: Add embryo of a documentation website.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
3 # Allow crawling of all content |
26e59f837558
docs: Add embryo of a documentation website.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
4 User-agent: * |
26e59f837558
docs: Add embryo of a documentation website.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
5 Disallow: |
26e59f837558
docs: Add embryo of a documentation website.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
6 |