| recognise google (/other) rather than having zillions of anon users [message #26846] |
Wed, 17 August 2005 09:06  |
djzort
Messages: 30
Registered: May 2005
Karma: 0
|
Member |
|
|
ive noticed that in the 'logged in users' invision (and possibly others) have your normal list of clickable users and an unclickable Google.com.
would be nice as currently when google and others hit you have 13+ anonymous users (if your forums arent closed to google with robots.txt)
perhaps a handy robots.txt editor would also be a feature for admins unawair of it.
|
|
|
|
|
|
|
|
|
|
| Re: recognise google (/other) rather than having zillions of anon users [message #27045 is a reply to message #27043] |
Thu, 25 August 2005 05:08  |
|
Anonymous
|
|
|
|
is it just me or is *every* feature request just rejected with some smart mouth comment?
have a robots.txt editor. it would be neat to have a 'property' of topics and forums to allow robots or not. and then fudforum would just generate the robots.txt. viola.
also
detect google. so in the 'currently online users' have
<link>user</link>,<link>user</link>,<link>user</link>,google bot, <link>user</link>,<link>user</link>
that or how about we have a pi computation page. just have php compute pi to infinity and spew it out. or perhaps a fast fourier transforms page. or maybe a page that differentiates delta? just in case little easy things arent worthy features?
|
|
|
|
| Re: recognise google (/other) rather than having zillions of anon users [message #27056 is a reply to message #27045] |
Thu, 25 August 2005 13:42 |
Ilia
Messages: 13241
Registered: January 2002
Karma: 0
|
Senior Member
Administrator
Core Developer |
|
|
Good features are added, inappropriate ones are not, that's all there is to it. Detecting search engine requires extra code for EVERY SINGLE request made to the forum, that is highly ineffecient. Especially if the only result of this is that on member list you see 1 vs 50 for Google's spider.
FUDforum Core Developer
|
|
|
|
| Re: recognise google (/other) rather than having zillions of anon users [message #27060 is a reply to message #27056] |
Thu, 25 August 2005 13:53 |
kenjb
Messages: 67
Registered: September 2004
Karma: 0
|
Member |
|
|
I'm not looking for a feature, I'm just asking if anyone else is getting this type of inappropriate attention from Googlebot.
As of this morning, just in overnights time the bandwidth consumed by Googlebot went from 5.79 GB to 6.45 GB.
So, my original question although maybe I should have started my own thread elsewhere, and I apologize for not doing that, is, Anyone else seeing Googlebot eat up that much bandwidth while parsing FUD forums?
Just wanted to add this:
There are 1 members(s), 0 invisible members and 2240 guest(s) visiting this board.
kenjb
[Updated on: Thu, 25 August 2005 13:56] Report message to a moderator |
|
|
|
|
|
| Re: recognise google (/other) rather than having zillions of anon users [message #27063 is a reply to message #27062] |
Thu, 25 August 2005 14:03 |
kenjb
Messages: 67
Registered: September 2004
Karma: 0
|
Member |
|
|
Yep, I'm sure. Here is one line out of my log:
66.249.65.146 - - [22/Aug/2005:13:31:16 -0500] "GET
/forums/index.php?t=index&cat=2&c=8:1_6:1_7:1_5:1_4:1_3:1_2:1&rid=0
HTTP/1.1" 200 18389 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
and that IP comes back as Network Owner: Google Inc.
If it keeps up, to my dismay I'll have to block her. That kind of defeats the purpose of being on the web then.
EDIT:
made it wrap to prevent scrolling to the right
kenjb
[Updated on: Sun, 04 September 2005 19:08] Report message to a moderator |
|
|
|
|
|
|
|
|
|
|
|
|
|
| Re: recognise google (/other) rather than having zillions of anon users [message #27096 is a reply to message #27056] |
Fri, 26 August 2005 01:04 |
|
Anonymous
|
|
|
|
| Ilia wrote on Thu, 25 August 2005 09:42 |
Good features are added, inappropriate ones are not, that's all there is to it. Detecting search engine requires extra code for EVERY SINGLE request made to the forum, that is highly ineffecient. Especially if the only result of this is that on member list you see 1 vs 50 for Google's spider.
|
whats wrong with an interface to robots.txt then?
that requires NO extra code for EVERY SINGLE request.
just have an option per forum and then per topic
something like 'allow search engine probes [x]'
robots.txt wipes out everything under it so if you turned it off on a forum you couldnt re-enable it on a per topic basis. but you could enable for a forum and then disable per topic.
sounds simple to me.
|
|
|
|
|
|
|
|
| Re: recognise google (/other) rather than having zillions of anon users [message #27300 is a reply to message #27296] |
Sat, 03 September 2005 19:16 |
Ilia
Messages: 13241
Registered: January 2002
Karma: 0
|
Senior Member
Administrator
Core Developer |
|
|
Looks like it is a beta portion of Google's functionality and you still need to submit it manually. It really won't address anon user "problem" (which I personally don't consider to be a problem), resulting of Google search bots not propogating cookies.
FUDforum Core Developer
|
|
|
|
| Re: recognise google (/other) rather than having zillions of anon users [message #27306 is a reply to message #27300] |
Sat, 03 September 2005 20:11 |
srchild
Messages: 88
Registered: December 2003
Location: UK
Karma: 1
|
Member |
|
|
| Ilia wrote on Sat, 03 September 2005 20:16 |
you still need to submit it manually.
|
You can set that up as a cron job.
| Quote: |
It really won't address anon user "problem" (which I personally don't consider to be a problem), resulting of Google search bots not propogating cookies.
|
No, I was thinking more of those who report a heavy use of bandwidth as googlebot repeatedly crawls their forums.
Just an idea. I have recently installed it (generated automatically, resubmitted by a cron job) on a TYPO3 based website that I run, and it seemed like an interesting idea though I have no stats to prove its benefit.
Simon Child
|
|
|
|
|
|
|
|
Calendar
Search
Help
Members
Register
Login
Home
or whatever you prefer to call it.
]