Re: Migration Problem [message #187582 is a reply to message #187581] |
Fri, 14 September 2018 20:30 |
art
Messages: 35 Registered: June 2009
Karma:
|
Member |
|
|
I reloaded the "sandbox" with the v.2.8.0 flat files and the fud28 MySQL database. I've updated the GLOBALS.php file to point to the sandbox database. So now it's back running the old code.
I decided to start off deleting the index, title_index, search and search_cache tables. Might as well, 'cause it'll probably make the SQL conversion part of the upgrades go faster. The index tables were all partials anyway; I am planning to re-index as soon as the system is up-to-snuff.
By the way, the "throttled" reindexing code I use (on FUDforum v.2.8.0) is shown below. I'll probably do something like this on FUDforum 3.0.9.
<?php
/**
* copyright : (C) 2001-2009 Advanced Internet Designs Inc.
* email : forum(at)prohost(dot)org
* $Id: reindex.php,v 1.36 2009/02/16 05:37:11 frank Exp $
*
* This program is free software; you can redistribute it and/or modify it
* under the terms of the GNU General Public License as published by the
* Free Software Foundation; version 2 of the License.
**/
// this is the same file as indexdb.php, except CLI code is included
// Modified to allow throttling so the website can remain running.
// A config file is used to control reindexing speed. It is named
// reindex.ini, which has a single number in it - the delay after
// each insert into fud28_search_cache. The delay is expressed
// in seconds and can be less than 1, e.g. 0.25 or 0.5.
$default_throttle = 500000;
$configfile = "reindex.throttle";
@set_time_limit(0);
@ini_set("memory_limit", "100M");
require('./GLOBALS.php');
fud_use('adm_cli.inc', 1);
cli_execute(1);
fud_use('adm.inc', true);
fud_use('glob.inc', true);
fud_use('isearch.inc');
fud_use('fileio.inc');
fud_use('rev_fmt.inc');
if ($FUD_OPT_1 & 1) {
echo "\nDisabling the forum for the duration of maintenance run\n";
maintenance_status('Undergoing maintenance, please come back later.', 1);
}
echo "\nPlease wait while index is being rebuilt.\nThis may take a while depending on the size of your forum.";
ob_flush(); flush();
$tbl =& $DBHOST_TBL_PREFIX;
if (defined('forum_debug')) {
list($locale, $GLOBALS['usr']->lang) = db_saq("SELECT locale, lang FROM {$tbl}themes WHERE theme_opt & (1|2) LIMIT 1");
$GLOBALS['good_locale'] = setlocale(LC_ALL, $locale);
}
db_lock($tbl.'msg_store WRITE, '.$tbl.'search_cache WRITE, '.$tbl.'search WRITE, '.$tbl.'index WRITE, '.$tbl.'title_index WRITE, '.$tbl.'msg WRITE');
q('DELETE FROM '.$tbl.'search');
q('DELETE FROM '.$tbl.'index');
q('DELETE FROM '.$tbl.'title_index');
if (!($sid = q_singleval("SELECT MIN(query_type) FROM ".$tbl."search_cache WHERE srch_query='' AND query_type<0"))) {
q('DELETE FROM '.$tbl.'search_cache');
}
$i = 0;
$c = q('SELECT id, subject, length, foff, file_id FROM '.$tbl.'msg WHERE '.($sid ? ' id>'.$sid.' AND ' : '').' apr=1 ORDER BY subject');
$qty = mysql_num_rows($c);
echo "\n\nRe-indexing " . $qty . " messages.";
$old_subject = '';
while ($r = db_rowarr($c)) {
if ($old_subject != $r[1]) {
$subj = $old_subject = $r[1];
} else {
$subj = '';
}
q('INSERT INTO '.$tbl.'search_cache (srch_query, query_type, expiry, msg_id, n_match) VALUES(\'\', -'.$r[0].', 0,0,0)');
index_text($subj, read_msg_body($r[3], $r[2], $r[4]), $r[0]);
$i++;
db_unlock();
echo "\nProcessed " . $i;
// Throttling mechanism
$fud_reindex_delay = 0;
if (file_exists($configfile)) {
$config = fopen($configfile, "r");
$fud_reindex_delay = trim(fgets($config));
fclose($config);
}
if ($fud_reindex_delay > 0) {
$throttle = $fud_reindex_delay * 1000000;
} else {
$throttle = $default_throttle;
}
usleep($throttle); // Delay
db_lock($tbl.'msg_store WRITE, '.$tbl.'search_cache WRITE, '.$tbl.'search WRITE, '.$tbl.'index WRITE, '.$tbl.'title_index WRITE, '.$tbl.'msg WRITE');
}
unset($c);
q('DELETE FROM '.$tbl.'search_cache');
db_unlock();
echo "\n\nDone! All messages indexed\n";
if ($FUD_OPT_1 & 1) {
echo "\nRe-enabling the forum.\n";
maintenance_status($GLOBALS['DISABLED_REASON'], 0);
} else {
echo "\nYour forum is currently disabled, to re-enable it go to the Global Settings Manager and re-enable it.\n";
}
?>
The original code locked the database, parsed ALL the messages and inserted all the indexed records and then unlocked the database. This method works fine on small forums but takes too long on larger forums. It also steadily increases memory usage so the whole system begins to crawl after indexing a few thousand messages.
At first, I broke the "while" loop into sections of 1000 records, locking at first, doing 1000 messages and then unlocking. But this still had problems so I decreased the number of records processed in each group to 100 and then even to 10. That worked pretty well. Then I realized I could even lock/unlock for each individual message, and put a variable delay in between each one. Empirically, I found that a 1/2 second delay "throttle" pretty much ran as well as having no delay at all on forums larger than about 10,000 messages, and this mechanism allows the system to be usable to the "outside world." I can re-index a production site this way, and it only seems slightly slower. Essentially, the re-indexing function is kind of like a person is creating a post every half-second. And if you want to increase or decrease the cycle time just put a value in the "reindex.throttle" file.
[Updated on: Fri, 14 September 2018 20:33] Report message to a moderator
|
|
|