FUDforum
Fast Uncompromising Discussions. FUDforum will get your users talking.

Home » FUDforum » How To » robots.txt for path-info  () 1 Vote
Show: Today's Messages :: Polls :: Message Navigator
Return to the default flat view Create a new topic Submit Reply
Re: robots.txt for path-info [message #31535 is a reply to message #26997] Thu, 04 May 2006 14:27 Go to previous message
matthieu_phpmv is currently offline  matthieu_phpmv   France
Messages: 44
Registered: November 2004
Karma:
Member
Here's mine on phpmyvisites.net/robots.txt
Quote:


User-agent: *
Disallow: /forums/pdf.php
Disallow: /forums/index.php/m
Disallow: /forums/index.php/sp/
Disallow: /forums/index.php/ef/
Disallow: /forums/index.php/mv/
Disallow: /forums/index.php/r/
Disallow: /forums/index.php/pmm/
Disallow: /forums/index.php/rm/
Disallow: /forums/index.php/sel/
Disallow: /forums/index.php/pv/
Disallow: /forums/index.php/ma/
Disallow: /forums/index.php/u/
Disallow: /forums/index.php/s/
Disallow: /forums/index.php/h/
Disallow: /forums/index.php/i/
Disallow: /forums/index.php/l/
Disallow: /forums/rdf.php


With this file bots will ONLY spider the messages, it is what I want. (no profile, no pdf messages, etc. only real and uniques message)

[Updated on: Fri, 05 May 2006 12:09]

Report message to a moderator

[Message index]
 
Read Message
Read Message
Previous Topic: Change Forum display style
Next Topic: Understanding templates and themes better
Goto Forum:
  

-=] Back to Top [=-
[ Syndicate this forum (XML) ] [ RSS ]

Current Time: Tue Nov 26 06:35:44 GMT 2024

Total time taken to generate the page: 0.03996 seconds