| Detecting search engine bots [message #12094] |
Mon, 28 July 2003 16:38  |
jrobbio
Messages: 32
Registered: June 2003
Karma: 0
|
Member |
|
|
search engine bots can be a real problem in that they appear as many users at the same time. What I suggest is to start associating the ip's that search engines like google, FAST, inktomi etc come from so that we can see where they appear. It would be cool to have some sort of tracker in the admin section so we can see which ones have been picked up and those that have been missed.
Rob
|
|
|
|
| Re: Detecting search engine bots [message #12095 is a reply to message #12094] |
Mon, 28 July 2003 16:55  |
Ilia
Messages: 13241
Registered: January 2002
Karma: 0
|
Senior Member
Administrator
Core Developer |
|
|
Associating IPs is a bad practice because often, different people using the same proxy will appear as the same user. As for search engines there are sites that have ip ranges of the various spiders used by search engines and similar. If you don't want them to visit your site simply use the FUDforum's IP filter to deny them access.
FUDforum Core Developer
|
|
|
|
|
|
| Re: Detecting search engine bots [message #12099 is a reply to message #12096] |
Mon, 28 July 2003 17:12 |
Ilia
Messages: 13241
Registered: January 2002
Karma: 0
|
Senior Member
Administrator
Core Developer |
|
|
IPB, what's that?
FUDforum's sessions are nonvolatile and do not prevent search engine parsing. You can also make forum work without using URL sessions. Plus I really see no harm as having a search engine appear as multiple anonymous users during. To spend time looking ips to group search engines would be too performance inhibitive and not bare any worth while advantages.
FUDforum Core Developer
|
|
|
|
|
|
| Re: Detecting search engine bots [message #12103 is a reply to message #12102] |
Mon, 28 July 2003 17:56 |
Ilia
Messages: 13241
Registered: January 2002
Karma: 0
|
Senior Member
Administrator
Core Developer |
|
|
Like I said it is a minor annoyance if even that, but the cost of a solution which would affect every request is just too much.
FUDforum Core Developer
|
|
|
|
|
|
|
|
|
|
| Re: Detecting search engine bots [message #12109 is a reply to message #12108] |
Mon, 28 July 2003 19:55 |
Ilia
Messages: 13241
Registered: January 2002
Karma: 0
|
Senior Member
Administrator
Core Developer |
|
|
Actually even though your pages maybe indexed more often then not they are not included in search results or found near the very bottom of the search results. This is because search engine generally tend to ignore URLs with parameters, since those could be used to easily be used as a search engine trap.
FUDforum Core Developer
|
|
|
|
| Re: Detecting search engine bots [message #12121 is a reply to message #12094] |
Tue, 29 July 2003 02:34 |
jrobbio
Messages: 32
Registered: June 2003
Karma: 0
|
Member |
|
|
Which is why I wanted to use the path_info setting as on with session ID's as off. Since the path_info was buggy on my particular site I am looking forward to 2.51 so I can have it fully functioning.
As for forums at the bottom of the SERPS, you should tell that to my friend who has his Vbulletin forum at the top for loads of things because of the way he dealt with the search engine bots on the forum.
Rob
|
|
|
|
Calendar
Search
Help
Members
Register
Login
Home
.
]