Home »
FUDforum »
How To »
robots.txt for path-info
() 1 Vote
Re: robots.txt for path-info [message #31535 is a reply to message #26997] |
Thu, 04 May 2006 14:27 |
matthieu_phpmv
Messages: 44 Registered: November 2004
Karma:
|
Member |
|
|
Here's mine on phpmyvisites.net/robots.txt
Quote: |
User-agent: *
Disallow: /forums/pdf.php
Disallow: /forums/index.php/m
Disallow: /forums/index.php/sp/
Disallow: /forums/index.php/ef/
Disallow: /forums/index.php/mv/
Disallow: /forums/index.php/r/
Disallow: /forums/index.php/pmm/
Disallow: /forums/index.php/rm/
Disallow: /forums/index.php/sel/
Disallow: /forums/index.php/pv/
Disallow: /forums/index.php/ma/
Disallow: /forums/index.php/u/
Disallow: /forums/index.php/s/
Disallow: /forums/index.php/h/
Disallow: /forums/index.php/i/
Disallow: /forums/index.php/l/
Disallow: /forums/rdf.php
|
With this file bots will ONLY spider the messages, it is what I want. (no profile, no pdf messages, etc. only real and uniques message)
[Updated on: Fri, 05 May 2006 12:09] Report message to a moderator
|
|
|
Goto Forum:
Current Time: Sun Nov 10 11:54:19 GMT 2024
Total time taken to generate the page: 0.04367 seconds