| Re: nntp import of large groups creates troubles [message #20375 is a reply to message #20202] |
Mon, 11 October 2004 12:21   |
devnull
Messages: 16
Registered: September 2004
Karma:
|
Junior Member |
|
|
For a quick solution, I just changed the script so that it writes the last article-id after every retrieved article. Time is something I don't have much right now, so I chose a quick and dirty solution.
Most groups were imported fine that way, and after several runs, up to date.
A few seemed to be stuck; although there were still a lot of articles to import, the script only increased the count by 1. I decided to tcpdump a single run to search for the possible cause.
I've isolated one case where the counter only increases by 1 and checked the article-id file over several runs. The ID does not change, although according to the forum view, one message was imported. The only change I made to the nntp.inc file was to add the following at line 452: $this->set_end($i);
Now, when looking into the dump, I saw that the article with the ID in the file was retrieved. Then the script requested the next ID, and this article was very long (10406 lines). Here the script stopped and the ID in the file has not changed. However, the first message is added to the database; I forgot to check that for some time and now have a lot of dupes (reminds me, is there a dupe search?  )
I assume it's a problem with the size of the messages.
Attached is the conversation for two messages. The second message is requested, but not added. Then the script ends.
-
Attachment: posts.txt
(Size: 407.06KB, Downloaded 1531 times)
|
|
|
|
Calendar
Search
Help
Members
Register
Login
Home
]