Migration Problem [message #187568] |
Mon, 10 September 2018 17:13 |
art
Messages: 35 Registered: June 2009
Karma: 0
|
Member |
|
|
Hello World!
I have been running FUDforum 2.8.0 for about 10 years. I migrated from WWWBoard and imported thousands of messages. Right now, my site has about 85,000 messages. FUDforum has always worked very well for me.
However, about two weeks ago, I hit a wall. The company that hosts my VPS sold to another company and they did a migration, but they didn't change the DNS nameserver entries, so I was still connected to the old box. Seemed like such a smooth migration, I was impressed! Then they finally pulled the plug and when I discovered this and had them point the nameservers properly, now my site was several months old. So out came the backups. Thank God I always grabbed a nightly backup.
I loaded FUDforum from backup and was mostly successful. As you might imagine, PHP and MySQL are both newer. By itself, that's a good thing. But I've had to work through a bunch of problems.
There are three problems I'm stumped on. The last one would likely fix the other two.
1. Some of my troubleshooting wiped out the fud28_index table so I had to reindex. I am able to run the script from CLI, and I added logic that throttles it to do locks/unlocks every so often. I've seen that logic described here and I use it. So I can reindex just fine. But it seems to create a lot of duplicates in the fud28_search and fud28_index tables. The index table gets huge. It's way bigger than what it was before the server change. Makes me wonder if there wasn't supposed to be "update existing or insert" logic that doesn't work because of the different MySQL version. But I'm guessing because I haven't been able to figure that out. I just know my index table grows too large.
2. Whenever a new post is made, anyone that's logged in clicks on the unread thread more than once will see duplicate entries in the thread list. I can see that the fud28_read table gets new records with the same thread_id and user_id rather than updating the existing entry with the new msg_id. If the user clicks the same message multiple times, then the read table will have multiple records with the same user_id, thread_id and msg_id. This is another reason I think maybe there's some logic that was intended to "update existing or insert new."
3. So all this makes the obvious answer be to upgrade FUDforum. But I've tried to upgrade to v.3.0.0 and wiped myself out good. Took me a day to recover from that. It wrote all the new files, and then started the SQL changes and then died. I didn't see any backout instructions so I did it manually. Kind of hard because after a restore, you also have to identify and remove any new 3.0 files in the src and thm directories or you can't rebuild themes.
What I think I really need to do is to create a sandbox installation somewhere to do a test-run of the upgrade(s) and any troubleshooting I've had to do. I've been messing around with the production box and that's always risky and painful. After the ISP clobbered me, it felt like nothing could possibly get worse, but I've clobbbered myself a few times. So I can definitely make things worse if I'm not careful.
Does anyone on this site do work for hire? I'd be willing to pay someone for assistance.
[Updated on: Mon, 10 September 2018 17:39] Report message to a moderator
|
|
|
|
|
Re: Migration Problem [message #187572 is a reply to message #187570] |
Tue, 11 September 2018 22:37 |
art
Messages: 35 Registered: June 2009
Karma: 0
|
Member |
|
|
Today I created a new FUDforum 3.0.0 installation.
It's on the same VPS "cloud" server as the one I'm having trouble with.
But it's installed on a different domain. I've put it in the usual "forum" directory.
And it's using the forum.db.php database, straight out of the box.
So my next step is to create a MySQL database and get it to use that. Then I'll see if I can bring in the v.2.8.0 data from the live forum. Not sure if I should try to build tables from a backup, or if I should run a conversion. Seems like the v.3.0.0 upgrade code modified the v.2.8.0 tables.
[Updated on: Tue, 11 September 2018 22:39] Report message to a moderator
|
|
|
|
Re: Migration Problem [message #187576 is a reply to message #187575] |
Thu, 13 September 2018 21:18 |
art
Messages: 35 Registered: June 2009
Karma: 0
|
Member |
|
|
I've created an exact copy of the limping v.2.8.0 forum on another domain. I've copied the MySQL database and I've copied all the FUDforum flat files.
My thought is to debug the v.3.0.0 upgrade script until it flies.
What got me stuck earlier was this error:
ERROR: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '60's' LIMIT 1' at line 1
QUERY: DELETE FROM fud28_search WHERE word=''60's' LIMIT 1
I imagine fud28_search probably has apostrophes in it. So I'm going to modify this section of the upgrade.php script:
while ($r = db_rowarr($c)) {
$con = '';
foreach ($f as $k => $v) {
$con .= "{$v}='{$r[$k]}' AND ";
}
$con = substr($con, 0, -4);
q("DELETE FROM {$tbl} WHERE {$con} LIMIT ".($r[$n] - 1));
}
I have backups of both the database and the flat files, so I can reload and re-run the script, fixing things as I find them.
[Updated on: Thu, 13 September 2018 21:19] Report message to a moderator
|
|
|
|
|
|
|
|
Re: Migration Problem [message #187582 is a reply to message #187581] |
Fri, 14 September 2018 20:30 |
art
Messages: 35 Registered: June 2009
Karma: 0
|
Member |
|
|
I reloaded the "sandbox" with the v.2.8.0 flat files and the fud28 MySQL database. I've updated the GLOBALS.php file to point to the sandbox database. So now it's back running the old code.
I decided to start off deleting the index, title_index, search and search_cache tables. Might as well, 'cause it'll probably make the SQL conversion part of the upgrades go faster. The index tables were all partials anyway; I am planning to re-index as soon as the system is up-to-snuff.
By the way, the "throttled" reindexing code I use (on FUDforum v.2.8.0) is shown below. I'll probably do something like this on FUDforum 3.0.9.
<?php
/**
* copyright : (C) 2001-2009 Advanced Internet Designs Inc.
* email : forum(at)prohost(dot)org
* $Id: reindex.php,v 1.36 2009/02/16 05:37:11 frank Exp $
*
* This program is free software; you can redistribute it and/or modify it
* under the terms of the GNU General Public License as published by the
* Free Software Foundation; version 2 of the License.
**/
// this is the same file as indexdb.php, except CLI code is included
// Modified to allow throttling so the website can remain running.
// A config file is used to control reindexing speed. It is named
// reindex.ini, which has a single number in it - the delay after
// each insert into fud28_search_cache. The delay is expressed
// in seconds and can be less than 1, e.g. 0.25 or 0.5.
$default_throttle = 500000;
$configfile = "reindex.throttle";
@set_time_limit(0);
@ini_set("memory_limit", "100M");
require('./GLOBALS.php');
fud_use('adm_cli.inc', 1);
cli_execute(1);
fud_use('adm.inc', true);
fud_use('glob.inc', true);
fud_use('isearch.inc');
fud_use('fileio.inc');
fud_use('rev_fmt.inc');
if ($FUD_OPT_1 & 1) {
echo "\nDisabling the forum for the duration of maintenance run\n";
maintenance_status('Undergoing maintenance, please come back later.', 1);
}
echo "\nPlease wait while index is being rebuilt.\nThis may take a while depending on the size of your forum.";
ob_flush(); flush();
$tbl =& $DBHOST_TBL_PREFIX;
if (defined('forum_debug')) {
list($locale, $GLOBALS['usr']->lang) = db_saq("SELECT locale, lang FROM {$tbl}themes WHERE theme_opt & (1|2) LIMIT 1");
$GLOBALS['good_locale'] = setlocale(LC_ALL, $locale);
}
db_lock($tbl.'msg_store WRITE, '.$tbl.'search_cache WRITE, '.$tbl.'search WRITE, '.$tbl.'index WRITE, '.$tbl.'title_index WRITE, '.$tbl.'msg WRITE');
q('DELETE FROM '.$tbl.'search');
q('DELETE FROM '.$tbl.'index');
q('DELETE FROM '.$tbl.'title_index');
if (!($sid = q_singleval("SELECT MIN(query_type) FROM ".$tbl."search_cache WHERE srch_query='' AND query_type<0"))) {
q('DELETE FROM '.$tbl.'search_cache');
}
$i = 0;
$c = q('SELECT id, subject, length, foff, file_id FROM '.$tbl.'msg WHERE '.($sid ? ' id>'.$sid.' AND ' : '').' apr=1 ORDER BY subject');
$qty = mysql_num_rows($c);
echo "\n\nRe-indexing " . $qty . " messages.";
$old_subject = '';
while ($r = db_rowarr($c)) {
if ($old_subject != $r[1]) {
$subj = $old_subject = $r[1];
} else {
$subj = '';
}
q('INSERT INTO '.$tbl.'search_cache (srch_query, query_type, expiry, msg_id, n_match) VALUES(\'\', -'.$r[0].', 0,0,0)');
index_text($subj, read_msg_body($r[3], $r[2], $r[4]), $r[0]);
$i++;
db_unlock();
echo "\nProcessed " . $i;
// Throttling mechanism
$fud_reindex_delay = 0;
if (file_exists($configfile)) {
$config = fopen($configfile, "r");
$fud_reindex_delay = trim(fgets($config));
fclose($config);
}
if ($fud_reindex_delay > 0) {
$throttle = $fud_reindex_delay * 1000000;
} else {
$throttle = $default_throttle;
}
usleep($throttle); // Delay
db_lock($tbl.'msg_store WRITE, '.$tbl.'search_cache WRITE, '.$tbl.'search WRITE, '.$tbl.'index WRITE, '.$tbl.'title_index WRITE, '.$tbl.'msg WRITE');
}
unset($c);
q('DELETE FROM '.$tbl.'search_cache');
db_unlock();
echo "\n\nDone! All messages indexed\n";
if ($FUD_OPT_1 & 1) {
echo "\nRe-enabling the forum.\n";
maintenance_status($GLOBALS['DISABLED_REASON'], 0);
} else {
echo "\nYour forum is currently disabled, to re-enable it go to the Global Settings Manager and re-enable it.\n";
}
?>
The original code locked the database, parsed ALL the messages and inserted all the indexed records and then unlocked the database. This method works fine on small forums but takes too long on larger forums. It also steadily increases memory usage so the whole system begins to crawl after indexing a few thousand messages.
At first, I broke the "while" loop into sections of 1000 records, locking at first, doing 1000 messages and then unlocking. But this still had problems so I decreased the number of records processed in each group to 100 and then even to 10. That worked pretty well. Then I realized I could even lock/unlock for each individual message, and put a variable delay in between each one. Empirically, I found that a 1/2 second delay "throttle" pretty much ran as well as having no delay at all on forums larger than about 10,000 messages, and this mechanism allows the system to be usable to the "outside world." I can re-index a production site this way, and it only seems slightly slower. Essentially, the re-indexing function is kind of like a person is creating a post every half-second. And if you want to increase or decrease the cycle time just put a value in the "reindex.throttle" file.
[Updated on: Fri, 14 September 2018 20:33] Report message to a moderator
|
|
|
Re: Migration Problem [message #187583 is a reply to message #187582] |
Fri, 14 September 2018 22:49 |
art
Messages: 35 Registered: June 2009
Karma: 0
|
Member |
|
|
This time, I got past the point where the v.3.0.9 upgrade script was creating the composite primary key on fud28_index(word_id,msg_id). So I think my assumption that fud28_index had duplicates was probably the sticking point there. That plus the fact that I've seen them.
But this time the 3.0.9 upgrade script broke on line 1036:
Fatal error: Call to undefined function q_limit() in ./forum/upgrade.php on line 1036
I've seen that code in the FUDforum cross-reference so I'm adding the q_limit() function (shown below) at the top of the 3.0.9 upgrade script. I expect the function was introduced in an earlier version of the code between 3.0.0 and 3.0.9, and I've skipped it. I'll take the chance that's all I'm missing, and I'm adding the function to the upgrade script.
function q_limit($query, $limit, $off=0)
{
if ($off) {
return $query .' LIMIT '. $off .','. $limit;
} else {
return $query .' LIMIT '. $limit;
}
}
So I'm back to the starting point:
1. Drop all tables in the migration-sandbox database.
2. Import the archived SQL dump of the v.2.8 database.
3. Delete all the forum files and subdirectories.
4. Install the v.2.8 flat files from the backup tarball.
5. Copy the modified GLOBALS.php file in place (with db/user for the sandbox database).
6. Delete the fud28_index, fud28_title_index, fud28_search and fud28_search_cache tables.
7. Run the consistency check.
8. Run the v.3.0.0 upgrade script.
9. Chmod 0755 ./forum/adm and run the consistency check.
10. Run the modified v.3.0.9 upgrade script (with q_limit function added).
11. If it works, run the consistency check.
12. Create a v.3.0.9 theme with the same look-and-feel as my v.2.8.0 forum.
We'll see how it goes...
[Updated on: Fri, 14 September 2018 23:21] Report message to a moderator
|
|
|
Re: Migration Problem [message #187584 is a reply to message #187583] |
Fri, 14 September 2018 23:45 |
|
GaryM
Messages: 118 Registered: May 2013 Location: Seattle
Karma: 3
|
Senior Member |
|
|
Wow Art! Good luck in getting to 3.0.9. You've leaped a 100 hurtles to get to base 3.x. Myself I'm so glad to take the next release when it comes along. I would have given up at this long ago if I was at 2.x.
Frank you may want to sticky this thread so others migrating from the 2.x base see what work is in store for them before posting.
Good luck Art and hopefully we'll see you on the other side.
Gary
http://vintagegruen.org
[Updated on: Fri, 14 September 2018 23:45] Report message to a moderator
|
|
|
Re: Migration Problem [message #187585 is a reply to message #187583] |
Fri, 14 September 2018 23:47 |
art
Messages: 35 Registered: June 2009
Karma: 0
|
Member |
|
|
Thanks for rooting for me, Gary!
That procedure got me all the way through the v.3.0.9 upgrade script except that it couldn't run the consistency checker at the end. That didn't surprise me because the v.3.0.0 upgrade script couldn't do it either. But every time I've run the v.3.0.0 upgrade script, I've simply turned on the least-significant (0x0001) bit of FUD_OPT_1, set the ./forum/adm directory permissions to 0755 and then run the consistency check manually. That has worked every time.
As usual after an upgrade, I found the least-significant bit of FUD_OPT_1 off so I turned it on. That removes the internet-facing "doing maintenance" message and displays the threads. So after doing the upgrade, I looked at the internet-facing view of the forum and saw all the posts, but it was missing the categories. This is a symptom I regularly see after table manipulation, and running the consistency checker always seems to fix that.
But even after setting the ./forum/adm directory permissions to 755, I still couldn't login as "admin." So I ran the consistency checker from the command line. This fixed the categories problem, as I often see. The consistency checker isn't seeing anything wrong with the database. All the categories now display properly in the main thread view for an anonymous user. I just cannot login.
I think I'm going to call it an evening and look at this again tomorrow. I'm not going to backout the v.3.0.9 installation, because it may just need some massaging to repair.
[Updated on: Fri, 14 September 2018 23:49] Report message to a moderator
|
|
|
Re: Migration Problem [message #187586 is a reply to message #187585] |
Sat, 15 September 2018 16:53 |
art
Messages: 35 Registered: June 2009
Karma: 0
|
Member |
|
|
It appears the main problem with my v.3.0.9 system was PHP version was 5.6. I can choose from multiple PHP options, so I set it to v.7.1. Then FUDforum ran clean for the most part. The one weird thing I've found is a blank screen is always shown after login. The URL is ./forum/index.php?t=login and the screen is blank. After that, if you manually go to the ./forum/index.php URL, it shows the forum and you're logged in. Same thing happens after logging out.
[Updated on: Sat, 15 September 2018 16:54] Report message to a moderator
|
|
|
|
|
|