Mercurial > piecrust2
annotate docs/assets/robots.txt @ 1058:f6b975db2545
data: Make `family` properties return lists instead of generators.
data: Add ability to return pages and groups.
author | Ludovic Chabant <ludovic@chabant.com> |
---|---|
date | Tue, 13 Feb 2018 11:10:55 -0800 |
parents | 26e59f837558 |
children |
rev | line source |
---|---|
243
26e59f837558
docs: Add embryo of a documentation website.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
1 # www.robotstxt.org/ |
26e59f837558
docs: Add embryo of a documentation website.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
2 |
26e59f837558
docs: Add embryo of a documentation website.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
3 # Allow crawling of all content |
26e59f837558
docs: Add embryo of a documentation website.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
4 User-agent: * |
26e59f837558
docs: Add embryo of a documentation website.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
5 Disallow: |
26e59f837558
docs: Add embryo of a documentation website.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff
changeset
|
6 |