Mercurial > piecrust2
annotate docs/assets/robots.txt @ 666:81d9c3a3a0b5
internal: Get rid of the whole "sub cache" business.
* Compute cache keys up front, so the cache directory is only chosen once.
* Buffer up config variants to apply before loading the config. Makes it
possible to cache variant-resulting configs, too.
* Make a factory class to reuse the logic that creates the `PieCrust` object
correctly for multi-process workers and such.
* Add a test.
author | Ludovic Chabant <ludovic@chabant.com> |
---|---|
date | Thu, 03 Mar 2016 08:22:41 -0800 |
parents | 26e59f837558 |
children |
rev | line source |
---|---|
243
26e59f837558
docs: Add embryo of a documentation website.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
1 # www.robotstxt.org/ |
26e59f837558
docs: Add embryo of a documentation website.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
2 |
26e59f837558
docs: Add embryo of a documentation website.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
3 # Allow crawling of all content |
26e59f837558
docs: Add embryo of a documentation website.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
4 User-agent: * |
26e59f837558
docs: Add embryo of a documentation website.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
5 Disallow: |
26e59f837558
docs: Add embryo of a documentation website.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
6 |