FUDforum: How To » Spiders, Bots and Crawlers

Home » FUDforum » How To » Spiders, Bots and Crawlers - Problem

Show: Today's Messages :: Polls :: Message Navigator

Spiders, Bots and Crawlers - Problem [message #169333]

Thu, 26 September 2013 10:54

danliker

Messages: 57
Registered: January 2004
Location: Schweiz

Karma: 0

Member

I have added Some Crawlers under "Spiders, Bots and Crawlers" also i have created
a group for them to only have rights to read / show. Then i selected auto-login.

But it looks like they get not logged in, are there some more options that needs to be changed to get it work?

Report message to a moderator

Re: Spiders, Bots and Crawlers - Problem [message #169334 is a reply to message #169333]

Thu, 26 September 2013 19:47

naudefj

Messages: 3771
Registered: December 2004

Karma: 28

Senior Member
Administrator
Core Developer

The feature is working as you can see on this forum.
The user agent string must match within the first 32 characters (will be 64 chars in 3.0.5).

Report message to a moderator

Aw: Spiders, Bots and Crawlers - Problem [message #169336 is a reply to message #169333]

Fri, 27 September 2013 08:30

danliker

Messages: 57
Registered: January 2004
Location: Schweiz

Karma: 0

Member

ok, i need to wait for 3.0.5...

Report message to a moderator

Aw: Spiders, Bots and Crawlers - Problem [message #169337 is a reply to message #169336]

Fri, 27 September 2013 09:03

danliker

Messages: 57
Registered: January 2004
Location: Schweiz

Karma: 0

Member

sorry but the only user agent string that is too long is the Google Bot, any hints how to troubleshoot my problem with other Bots that are correct in the DB?

[Updated on: Fri, 27 September 2013 09:04]

Report message to a moderator

Re: Aw: Spiders, Bots and Crawlers - Problem [message #169338 is a reply to message #169337]

Fri, 27 September 2013 16:13

naudefj

Messages: 3771
Registered: December 2004

Karma: 28

Senior Member
Administrator
Core Developer

Case should match as well. Besides that, I cannot think of anything else. To continue, maybe post your useragent mask as well as the "User-Agent" header of a bot that's not working.

Report message to a moderator

Re : Re: Aw: Spiders, Bots and Crawlers - Problem [message #169339 is a reply to message #169338]

Fri, 27 September 2013 17:07

TheDude
Messages: 120
Registered: November 2012
Location: France

Karma: 2

Senior Member

Stupid question perhaps, but what's the point of this, adding bots... ?

http://www.bartopmania.com

Report message to a moderator

Re: Re : Re: Aw: Spiders, Bots and Crawlers - Problem [message #169340 is a reply to message #169339]

Fri, 27 September 2013 17:55

naudefj

Messages: 3771
Registered: December 2004

Karma: 28

Senior Member
Administrator
Core Developer

You can block the bad bots and keep track of the good bots visiting your forum.

EDIT: Something else: you can present a bot with a certain view of the site. For example, ensure Google index the French version, but Yahoo only see the German side. Silly example, but you get the idea.

Report message to a moderator

Aw: Spiders, Bots and Crawlers - Problem [message #172197 is a reply to message #169333]

Mon, 30 September 2013 10:24

danliker

Messages: 57
Registered: January 2004
Location: Schweiz

Karma: 0

Member

ok, maybe i have some other problem.
my goal was to get correct amount of online users, as this is completely wrong in our forum.

as you can see on screen2 there is the same ip two times and this is the baiduspider ....

123.125.71.45 - - [30/Sep/2013:09:53:45 +0200] "GET /index.php/m/1223/efe26d61487b9e6e894945accc51efd1/ HTTP/1.1" 200 11899 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
123.125.71.45 - - [30/Sep/2013:10:07:57 +0200] "GET /index.php/m/1204/3bfc5723f883614c32e31a9ecd4aef58/ HTTP/1.1" 200 8065 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
123.125.71.45 - - [30/Sep/2013:10:43:38 +0200] "GET /index.php/e/Baiduspider/ HTTP/1.1" 200 1813 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
123.125.71.45 - - [30/Sep/2013:11:34:55 +0200] "GET /index.php/e/Baiduspider/ HTTP/1.1" 200 1812 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
123.125.71.45 - - [30/Sep/2013:11:41:58 +0200] "GET /index.php/m/1103/0f13c41485b761ea963f4e1bc61dc6a0/ HTTP/1.1" 200 7042 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
123.125.71.45 - - [30/Sep/2013:12:19:49 +0200] "GET /index.php/f/81/867941afb0ef4243549bfcba62da83d6/ HTTP/1.1" 302 505 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
123.125.71.45 - - [30/Sep/2013:12:19:59 +0200] "GET /index.php/m/1229/f46027036e10bdc9581b0bebf17c2411/ HTTP/1.1" 200 5775 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"

Attachment: screen1.jpg
(Size: 142.52KB, Downloaded 917 times)
Attachment: screen2.jpg
(Size: 94.83KB, Downloaded 907 times)

Report message to a moderator

Aw: Spiders, Bots and Crawlers - Problem [message #172198 is a reply to message #172197]

Mon, 30 September 2013 10:26

danliker

Messages: 57
Registered: January 2004
Location: Schweiz

Karma: 0

Member

also 220.181.108.163 is Baiduspider:

220.181.108.163 - - [30/Sep/2013:10:22:15 +0200] "GET /index.php/m/1195/15a8285e22b72251933bdb05f0e95e28/ HTTP/1.1" 200 7191 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
220.181.108.163 - - [30/Sep/2013:10:29:34 +0200] "GET /index.php/m/1189/f6f756c96cbe97b8bab3ba1749e8fd2c/ HTTP/1.1" 200 7235 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
220.181.108.163 - - [30/Sep/2013:10:29:40 +0200] "GET /index.php/m/1189/e320b79752a4308f46d2e856150bd843/ HTTP/1.1" 200 7235 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
220.181.108.163 - - [30/Sep/2013:11:06:21 +0200] "GET /index.php/e/Baiduspider/ HTTP/1.1" 200 1813 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
220.181.108.163 - - [30/Sep/2013:11:20:22 +0200] "GET /index.php/m/1139/3706e6fa5c19ee48313ba4f31b5fc044/ HTTP/1.1" 302 523 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
220.181.108.163 - - [30/Sep/2013:12:19:44 +0200] "GET /index.php/f/106/ed1e5b660769bbe16cdf924b4de0b9fa/ HTTP/1.1" 302 505 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
220.181.108.163 - - [30/Sep/2013:12:24:27 +0200] "GET /index.php/e/Baiduspider/ HTTP/1.1" 200 1813 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"

Report message to a moderator