Mercurial > wikked

annotate docs/assets/robots.txt @ 476:71114096433c

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
core: Add support for Markdown extensions, add header anchor extension. - New configuration option to specify Markdown extensions. - Enable some extensions by default. - Add CSS to make tables pretty. - Add extension to generate anchors next to each HTML heading. - Provide CSS to show those anchors on mouse-hover.
author Ludovic Chabant <ludovic@chabant.com>
date Thu, 11 Oct 2018 23:24:06 -0700
parents 3a61f45702cb
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
382
3a61f45702cb docs: Add documentation site.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
1 # www.robotstxt.org/
3a61f45702cb docs: Add documentation site.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
2
3a61f45702cb docs: Add documentation site.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
3 # Allow crawling of all content
3a61f45702cb docs: Add documentation site.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
4 User-agent: *
3a61f45702cb docs: Add documentation site.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
5 Disallow:
3a61f45702cb docs: Add documentation site.
Ludovic Chabant <ludovic@chabant.com>
parents:
diff changeset
6