| Rebuild search index part by part [message #30461] |
Mon, 27 February 2006 10:51  |
|
|
Hello!
I had a big database crash, now forum is working, but ... i do not have search index (fud26_search, fud26_index)...
Now I should rebuild search indexes, but I cannot.
I cannot do it because about 20 seconds after start to rebuild all is stopped (Throbber in my Firefox stops to animate). Probably because of any anti DoS or something on server.
In theory, I can do it in my home computer, but the database created at home is incompatible with mySQL on server, I cannot upload some tables, some has broken national characters.
What i'm looking for?
The rebuild script takes about 400% of power of my computer, probably same on server, and probably it is because the process is killed by the server. Any chances for the rebuild index script which will work:
- much, much longer
- can make the work in parts (no one more than 4 minutes)
- not use so much CPU power at once
This way I (and every other with similar problemm) can rebuild search index part by part on temporary disabled forum.
Is/will/may be it possible???
Any other ideas what can I do to rebuild index?
|
|
|
|
|
|
| Re: Rebuild search index part by part [message #30471 is a reply to message #30464] |
Mon, 27 February 2006 14:34  |
|
|
| Ilia wrote on Mon, 27 February 2006 15:15 |
You can run the search rebuild via command line, by following instructions at the top of the indexdb.php script.
|
Yes i know that, but i don't have shell access on server where forum is installed. So only modification of script, other script, but sill started "via apache" is possibe here.
BTW ... I'm looking into indexdb.php (FUDforum 2.7.4) ... as i understand, after clicking yes (sending form) the first what should i see is:
line 45: echo '<br>Disabling the forum for the duration of maintenance run<br>';
and before this line there is nothing what can take much time. Am I wrong? Because... on the server where i have the problemm the page is starting to load, but it is nor refreshing. It is still same page with same form, and Firefox Throbber stops after few seconds.
On my home computer where reindexing is working, it looks like something is working a while (few minutes), and than in few seconds i'm getting new page with botg disabling forum, blah, blah, blah, reenabling forum. Is it normal? It is like it should work? Maby there is a little bug in reindexing script.
I'm looking inside the file, but in fact i don't understand how it is working. What can I comment to make it working faster, what script can be extracted from it to make reIndex forum piece after piece. What I mean to make e.g.
manualReindexStart.php - blocking forum, dropping tables
manualReindex1.php - creating index for first forum
manualReindex2.php - creating index for 2nd forum
...
manualReindexn.php - creating index for last forum
manualReindexEnd.php - Enabling the forum.
Maby it is possible to make it like this, so i can start via apache every piece one after one. The manualReindex1-n.php is one file with some parameter changes to parse next file.
I'm not sure if you understand my idea, but it is only solution for my problemm I can imagine for now.
|
|
|
|
| Re: Rebuild search index part by part [message #30479 is a reply to message #30471] |
Mon, 27 February 2006 19:22 |
Ilia
Messages: 13241
Registered: January 2002
Karma: 0
|
Senior Member
Administrator
Core Developer |
|
|
The only way you can make it work in your situation is by making it index few messages at a time and then reload the page. Eventually it will index all of the forum's messages.
FUDforum Core Developer
|
|
|
|
| Re: Rebuild search index part by part [message #30484 is a reply to message #30479] |
Mon, 27 February 2006 21:27 |
|
|
| Ilia wrote on Mon, 27 February 2006 20:22 |
The only way you can make it work in your situation is by making it index few messages at a time and then reload the page. Eventually it will index all of the forum's messages.
|
OK but ... how?
By running new/special script which will stop every few messages?
Or just by using Admin Panel in way:
1. Admin panel
2. Rebuild search index
3. Yes (rebuild)
4. Throbber stopped
5. Return to forum
6. Goto 1.
???
If it is the second one how can I be sure that it is working?
|
|
|
|
| Re: Rebuild search index part by part [message #30501 is a reply to message #30484] |
Tue, 28 February 2006 14:29 |
Ilia
Messages: 13241
Registered: January 2002
Karma: 0
|
Senior Member
Administrator
Core Developer |
|
|
The idea is that you use LIMIT clause to fetch only a few messages each time, lets say 100 for example sakes. When you processed those 100 messages you would redirect the browsers to the same page, this time starting from message 101 and processing the next 100 messages. This process will be then repeated until there are no more messages.
FUDforum Core Developer
|
|
|
|
| Re: Rebuild search index part by part [message #30514 is a reply to message #30461] |
Tue, 28 February 2006 20:46 |
|
|
Do you mean to set
in adm/indexdb.php
To low value, e.g. 100 seconds and than start to rebuild index via admin panel?
Or maby:
you mean to change the line:
74: q('INSERT INTO '.$tbl.'search_cache (srch_query, query_type, expiry, msg_id, n_match) VALUES(\'\', -'.$r[0].', 0,0,0)');
75: index_text($subj, read_msg_body($r[3], $r[2], $r[4]), $r[0]);
to:
74: q('INSERT INTO '.$tbl.'search_cache (srch_query, query_type, expiry, msg_id, n_match) VALUES(\'\', -'.$r[0].', 0,0,0) LIMIT 100');
I'm not sure where it (LIMIT 100) should be placed ?
maby:
66: $c = q('SELECT id, subject, length, foff, file_id FROM '.$tbl.'msg WHERE '.($sid ? ' id>'.$sid.' AND ' : '').' apr=1 ORDER BY subject');
like:
66: $c = q('SELECT id, subject, length, foff, file_id FROM '.$tbl.'msg WHERE '.($sid ? ' id>'.$sid.' AND ' : '').' apr=1 ORDER BY subject' LIMIT 100);
please tell me if you are talking just about change like this in the file and than just rebuild index. I don't understand this script, and I'm affraid to make something wrong. I takes me 3 days (and 2 nights) to make a forum usable, don't want to kill it by wrong modification of indexdb.php, but also i want to get back search funcionality.
[Updated on: Wed, 01 March 2006 16:38] Report message to a moderator |
|
|
|
|
|
|
|
| Re: Rebuild search index part by part [message #30548 is a reply to message #30531] |
Thu, 02 March 2006 14:14 |
Ilia
Messages: 13241
Registered: January 2002
Karma: 0
|
Senior Member
Administrator
Core Developer |
|
|
Well, the query needs to be modified to start working from an offset and fetch X records (lets say 100 each time). At the end it would redirect to itself via header("Location: ..."); and via a GET parameter specify a new offset to start from.
FUDforum Core Developer
|
|
|
|
| Re: Rebuild search index part by part [message #30613 is a reply to message #30461] |
Sun, 05 March 2006 03:33 |
|
|
OK, i thing now i understand... i'll try to write it.
But can you please tell me. The posts are ordered by subject, is it important? Because to make a piece of work, and back to it l8r it is better to order messages by id.
And if it is important how can I back to work?
|
|
|
|
| Re: Rebuild search index part by part [message #30614 is a reply to message #30613] |
Sun, 05 March 2006 16:27 |
Ilia
Messages: 13241
Registered: January 2002
Karma: 0
|
Senior Member
Administrator
Core Developer |
|
|
It does not matter what ORDER BY statement you use. Normally the forum orders messages by id for the purpose of re-indexing, but you can easily change the ordering to "by-subject", there is no harm in doing that.
FUDforum Core Developer
|
|
|
|
|
|
| Re: Rebuild search index part by part [message #30647 is a reply to message #30461] |
Mon, 06 March 2006 23:50 |
|
|
Ok ... maby i did ... it seems to be wotking on test base, but before i start it on my real forum, please can you please look on and tell me what do you think?
I'm attaching indexdb.php which is changed to have possibility to start indexdb-poor.php
and indexdb-poor.php (in next post) which is the indexing db script made in a way you talked about (if not i didn't understand you  )
-
Attachment: indexdb.php
(Size: 3.41KB, Downloaded 623 times)
[Updated on: Mon, 06 March 2006 23:57] Report message to a moderator |
|
|
|
|
|
|
|
| Re: Rebuild search index part by part [message #30659 is a reply to message #30461] |
Tue, 07 March 2006 16:30 |
|
|
And any ideas, why db_unlock() is not working? If i uncomment it I get:
(/home/wariat/www/forum/ZorannorumNB/include/theme/default/db.inc:49
/home/wariat/www/forum/indexdb-poor.php:62
) internal: internal
Query: DB_UNLOCK: no previous lock established
_GET: array ( 'S' => 'ca17...', 'SQ' => 'b0c...', 'Indexed' => '50', )
Server Version: 4.1.14-log
[Referring Page] http://www.wariat.org.pl/Zorannorum/adm/indexdb.php?S=ca17...&SQ=b0c....
it says that no lock established, but i did it in first Pass, so what is more correct?
- to lock it and than unlock on every pass of script?
- don't lock anywhere?
If it will work correct on my slow server, any chanses to this feature (or somenthing like this) to official release?
|
|
|
|
| Re: Rebuild search index part by part [message #30674 is a reply to message #30659] |
Wed, 08 March 2006 13:59 |
Ilia
Messages: 13241
Registered: January 2002
Karma: 0
|
Senior Member
Administrator
Core Developer |
|
|
it is not working because the lock it was supposed to be unlocking was not created. The error message is pretty clear in that regard.
FUDforum Core Developer
|
|
|
|
| Re: Rebuild search index part by part [message #30795 is a reply to message #30461] |
Mon, 13 March 2006 15:07 |
|
|
But you didn't tell me what is better in your opinion:
| Quote: |
it says that no lock established, but i did it in first Pass, so what is more correct?
- to lock it and than unlock on every pass of script?
- don't lock anywhere?
|
and:
| Quote: |
If it will work correct on my slow server, any chanses to this feature (or something like this) to official release?
|
|
|
|
|
| Re: Rebuild search index part by part [message #30799 is a reply to message #30795] |
Tue, 14 March 2006 14:26 |
Ilia
Messages: 13241
Registered: January 2002
Karma: 0
|
Senior Member
Administrator
Core Developer |
|
|
The lock cannot stay active between passes, you need to create it and release it on every pass.
It should work on your server, if it works I'll onsider making something like this part of the main release.
FUDforum Core Developer
|
|
|
|
Calendar
Search
Help
Members
Register
Login
Home
Normally it is ordered by subject, and I will order it by Id 
]