www/apache
1
0
Fork 0
mirror of https://github.com/apache/httpd.git synced 2026-01-06 09:01:14 +03:00
Code Activity

PR:

Browse Source 
tsk tsk, randy.  Can't find this on covalent.net either.


git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@80015 13f79535-47bb-0310-9956-ffa450edef68
This commit is contained in:
brian
1998-01-26 07:12:58 +00:00
parent feec2b0631
commit bbc3f6d277
1 changed files with 8 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
docs/manual/misc/howto.html
View File

@@ -130,7 +130,14 @@ is then used by a <em>search engine</em> to help locate information.</P>
<P><code>robots.txt</code> provides a means to request that robots limit their
activities at the site, or more often than not, to leave the site alone.</P>
<P>When the first robots were developed, they had a bad reputation for sending hundreds/thousands of requests to each site, often resulting in the site being overloaded. Things have improved dramatically since then, thanks to <A HREF="http://info.webcrawler.com/mak/projects/robots/guidelines.html"> Guidelines for Robot Writers</A>, but even so, some robots may <A HREF="http://www.zyzzyva.com/robots/alert/">exhibit unfriendly behavior</A> which the webmaster isn't willing to tolerate, and will want to stop.</P>
<P>When the first robots were developed, they had a bad reputation for
sending hundreds/thousands of requests to each site, often resulting
in the site being overloaded. Things have improved dramatically since
then, thanks to <A
HREF="http://info.webcrawler.com/mak/projects/robots/guidelines.html">
Guidelines for Robot Writers</A>, but even so, some robots may exhibit
unfriendly behavior which the webmaster isn't willing to tolerate, and
will want to stop.</P>
<P>Another reason some webmasters want to block access to robots, is to
stop them indexing dynamic information. Many search engines will use the