FUDforum
Fast Uncompromising Discussions. FUDforum will get your users talking.

Home » FUDforum » FUDforum Installation Issues » Migration Problem
Show: Today's Messages :: Polls :: Message Navigator
Switch to threaded view of this topic Create a new topic Submit Reply
Migration Problem [message #187568] Mon, 10 September 2018 17:13 Go to next message
art is currently offline  art   
Messages: 35
Registered: June 2009
Karma: 0
Member

Hello World!

I have been running FUDforum 2.8.0 for about 10 years. I migrated from WWWBoard and imported thousands of messages. Right now, my site has about 85,000 messages. FUDforum has always worked very well for me.

However, about two weeks ago, I hit a wall. The company that hosts my VPS sold to another company and they did a migration, but they didn't change the DNS nameserver entries, so I was still connected to the old box. Seemed like such a smooth migration, I was impressed! Laughing Then they finally pulled the plug and when I discovered this and had them point the nameservers properly, now my site was several months old. So out came the backups. Thank God I always grabbed a nightly backup.

I loaded FUDforum from backup and was mostly successful. As you might imagine, PHP and MySQL are both newer. By itself, that's a good thing. But I've had to work through a bunch of problems.

There are three problems I'm stumped on. The last one would likely fix the other two.

1. Some of my troubleshooting wiped out the fud28_index table so I had to reindex. I am able to run the script from CLI, and I added logic that throttles it to do locks/unlocks every so often. I've seen that logic described here and I use it. So I can reindex just fine. But it seems to create a lot of duplicates in the fud28_search and fud28_index tables. The index table gets huge. It's way bigger than what it was before the server change. Makes me wonder if there wasn't supposed to be "update existing or insert" logic that doesn't work because of the different MySQL version. But I'm guessing because I haven't been able to figure that out. I just know my index table grows too large.

2. Whenever a new post is made, anyone that's logged in clicks on the unread thread more than once will see duplicate entries in the thread list. I can see that the fud28_read table gets new records with the same thread_id and user_id rather than updating the existing entry with the new msg_id. If the user clicks the same message multiple times, then the read table will have multiple records with the same user_id, thread_id and msg_id. This is another reason I think maybe there's some logic that was intended to "update existing or insert new."

3. So all this makes the obvious answer be to upgrade FUDforum. But I've tried to upgrade to v.3.0.0 and wiped myself out good. Took me a day to recover from that. It wrote all the new files, and then started the SQL changes and then died. I didn't see any backout instructions so I did it manually. Kind of hard because after a restore, you also have to identify and remove any new 3.0 files in the src and thm directories or you can't rebuild themes.

What I think I really need to do is to create a sandbox installation somewhere to do a test-run of the upgrade(s) and any troubleshooting I've had to do. I've been messing around with the production box and that's always risky and painful. After the ISP clobbered me, it felt like nothing could possibly get worse, but I've clobbbered myself a few times. So I can definitely make things worse if I'm not careful.

Does anyone on this site do work for hire? I'd be willing to pay someone for assistance.


[Updated on: Mon, 10 September 2018 17:39]

Report message to a moderator

Re: Migration Problem [message #187569 is a reply to message #187568] Mon, 10 September 2018 20:03 Go to previous messageGo to next message
naudefj is currently offline  naudefj   United States
Messages: 3771
Registered: December 2004
Karma: 28
Senior Member
Administrator
Core Developer
Duplicate entries means you have duplicate data in your tables. Try to drop the tables and remove all files/directories before importing. When done, remember to do a consistence check.

PS: I may be able to assist over the weekend. In the mean time, please keep us posted on your progress.
Re: Migration Problem [message #187570 is a reply to message #187569] Mon, 10 September 2018 20:59 Go to previous messageGo to next message
art is currently offline  art   
Messages: 35
Registered: June 2009
Karma: 0
Member

I regularly run the consistency check. Pretty much after everything I do to the tables.

Ever since I discovered the fud28_read duplicates, I've begun to manually delete them. In fact, I've even made a little PHP script that queries to find fud28_read table duplicates and deletes them. I run it in a cron job that fires each minute. That's one of the ugliest jury-rigs I've ever done, but it does keep the forum looking clean even though it's not. Laughing

As for the fud28_index size, I've done indexdb.php which drops the tables fud28_search, fud28_index and fud28_title_index before it starts gobbling them back up. I can see the table size go to zero, and I can see the disk space change using "df -k." I run it from the CLI using "nohup" and I monitor it with "tail -f" and I also watch "df -k" to see when the database is growing too large. I stop the reindexing process after it has done anywhere from 5000 to 15000 messages because space prohibits me from going any further. And then I do the consistency check. That lets users have access to the forum and it provides some of the data to FUDforum search engine.

I'm pretty much letting it limp along like that while I try to find the right solutions to the problems. Right now, I feel like I'm trying to solder SMD chips with the only tools in my bag being a sledgehammer and a blowtorch.


Re: Migration Problem [message #187572 is a reply to message #187570] Tue, 11 September 2018 22:37 Go to previous messageGo to next message
art is currently offline  art   
Messages: 35
Registered: June 2009
Karma: 0
Member

Today I created a new FUDforum 3.0.0 installation.

It's on the same VPS "cloud" server as the one I'm having trouble with.

But it's installed on a different domain. I've put it in the usual "forum" directory.

And it's using the forum.db.php database, straight out of the box.

So my next step is to create a MySQL database and get it to use that. Then I'll see if I can bring in the v.2.8.0 data from the live forum. Not sure if I should try to build tables from a backup, or if I should run a conversion. Seems like the v.3.0.0 upgrade code modified the v.2.8.0 tables.


[Updated on: Tue, 11 September 2018 22:39]

Report message to a moderator

Re: Migration Problem [message #187575 is a reply to message #187572] Thu, 13 September 2018 15:34 Go to previous messageGo to next message
art is currently offline  art   
Messages: 35
Registered: June 2009
Karma: 0
Member

I've created a new MySQL database which contains a copy of the FUDforum v.2.8.0 database I'm planning to migrate. So now I can attempt to convert that copy to FUDforum v.3.0.0, and then hopefully all the way up to the most recent stable release.

While I was waiting for the SQLdump, I copied the theme. Looks like I may need to convert that too.

Re: Migration Problem [message #187576 is a reply to message #187575] Thu, 13 September 2018 21:18 Go to previous messageGo to next message
art is currently offline  art   
Messages: 35
Registered: June 2009
Karma: 0
Member

I've created an exact copy of the limping v.2.8.0 forum on another domain. I've copied the MySQL database and I've copied all the FUDforum flat files.

My thought is to debug the v.3.0.0 upgrade script until it flies.

What got me stuck earlier was this error:

ERROR: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '60's' LIMIT 1' at line 1
QUERY: DELETE FROM fud28_search WHERE word=''60's' LIMIT 1

I imagine fud28_search probably has apostrophes in it. So I'm going to modify this section of the upgrade.php script:
	while ($r = db_rowarr($c)) {
				$con = '';
				foreach ($f as $k => $v) {
					$con .= "{$v}='{$r[$k]}' AND ";
				}
				$con = substr($con, 0, -4);
				q("DELETE FROM {$tbl} WHERE {$con} LIMIT ".($r[$n] - 1));
			}

I have backups of both the database and the flat files, so I can reload and re-run the script, fixing things as I find them.

[Updated on: Thu, 13 September 2018 21:19]

Report message to a moderator

Re: Migration Problem [message #187577 is a reply to message #187576] Thu, 13 September 2018 22:51 Go to previous messageGo to next message
art is currently offline  art   
Messages: 35
Registered: June 2009
Karma: 0
Member

So far, so good.

The FUDforum v.3.0.0 upgrade.php script has gotten through the tables. I'm getting ready to run the consistency checker.

Beginning SQL Upgrades.
Convert table fud28_index to UTF-8.
SQL Upgrades Complete.

Adding GLOBAL Variables.
Compiling theme default.
Please manually update custom theme xyzzy
Compiling theme xyzzy.
WARNING: {META-CONTENT-LANGUAGE} is depricated, please remove it from your templates.
WARNING: {META-CONTENT-LANGUAGE} is depricated, please remove it from your templates.
WARNING: {META-CONTENT-LANGUAGE} is depricated, please remove it from your templates.

Executing the Consistency Checker. 
If the popup with the consistency checker failed to appear you MUST >> click here << to complete the upgrade.

I changed upgrade.php, starting on line 442 as shown below:

			while ($r = db_rowarr($c)) {
				$con = '';
				foreach ($f as $k => $v) {
					$con .= "{$v}=\"{$r[$k]}\" AND ";
				}
				$con = substr($con, 0, -4);
				q("DELETE FROM {$tbl} WHERE {$con} LIMIT ".($r[$n] - 1));
			}


Re: Migration Problem [message #187578 is a reply to message #187577] Thu, 13 September 2018 22:57 Go to previous messageGo to next message
art is currently offline  art   
Messages: 35
Registered: June 2009
Karma: 0
Member

FUDforum v.3.0.0 is up, and I can see all my posts. However, the categories aren't there. Could be that'll work after I run the consistency check. I've seen it "lose" categories when attempting imports or conversions until the consistency check is run.

But all the admin pages show server error 500, so I can't run the consistency checker 'til I fix that. Probably just a permissions thing. I'll check it out later, probably tomorrow. At least I'm making progress.


Re: Migration Problem [message #187579 is a reply to message #187578] Fri, 14 September 2018 00:28 Go to previous messageGo to next message
art is currently offline  art   
Messages: 35
Registered: June 2009
Karma: 0
Member

It was permissions on the ./adm directory. I couldn't wait 'til tomorrow, 'cause I kind of knew that must be it and I just couldn't wait. So I ate dinner with the family and then checked perms. Sure enough, it was set to 777 so I set it to 755 and clicked on the admin panel. Worked. Then ran the consistency check. It worked too.

Sure enough, the categories are back. I'm up and running on 3.0.0, at least in the maintenance sandbox.

Tomorrow, I'll look at what I need to do to upgrade to the latest stable version.

Then I'll make a new theme that has the features I use.

Then I'll bring over the current data as it sits on the production server and do all this again, putting it into production.

Last thing - assuming the "read" table duplicates problem is fixed - is to re-index the whole for the search engine. Hopefully that process won't fill the disk. But I'll burn that bridge when I get there. Laughing

Re: Migration Problem [message #187580 is a reply to message #187579] Fri, 14 September 2018 15:42 Go to previous messageGo to next message
art is currently offline  art   
Messages: 35
Registered: June 2009
Karma: 0
Member

I've begun the v.3.0.9 upgrade script on my sandbox forum. Noticed it say, "Create composite primary key on fud28_index(word_id,msg_id)." So I'm hoping that upgraded logic will reduce the duplicates in the index table.

Fingers crossed. After this step, my sandbox forum will be at the latest stable release. Then I can create a theme that looks just like my old v.2.8.0 theme.

If all goes well, I can start over with a fresh copy of the production data. Do a sqldump of the production database and make a tarball of the forum. Bring it over to the sandbox area and upgrade to v.3.0.0 with the upgrade script modified as described above. Then upgrade that to v.3.0.9.

Then I can re-index the whole thing to create entries in the search tables. That will probably require a lot of disk space so I might delete the sandbox and re-index in production using a throtlled script that I can set the speed of.


Re: Migration Problem [message #187581 is a reply to message #187580] Fri, 14 September 2018 19:40 Go to previous messageGo to next message
art is currently offline  art   
Messages: 35
Registered: June 2009
Karma: 0
Member

The update hung at the point where it was creating the composite primary key on fud28_index(word_id,msg_id). Could be that was from the duplicates. I'll start over and when I get to the point where I'm updating from v.3.0.0 to v.3.0.9, I'll delete all the records in the index table first.


Re: Migration Problem [message #187582 is a reply to message #187581] Fri, 14 September 2018 20:30 Go to previous messageGo to next message
art is currently offline  art   
Messages: 35
Registered: June 2009
Karma: 0
Member

I reloaded the "sandbox" with the v.2.8.0 flat files and the fud28 MySQL database. I've updated the GLOBALS.php file to point to the sandbox database. So now it's back running the old code.

I decided to start off deleting the index, title_index, search and search_cache tables. Might as well, 'cause it'll probably make the SQL conversion part of the upgrades go faster. The index tables were all partials anyway; I am planning to re-index as soon as the system is up-to-snuff.

By the way, the "throttled" reindexing code I use (on FUDforum v.2.8.0) is shown below. I'll probably do something like this on FUDforum 3.0.9.

<?php
/**
* copyright            : (C) 2001-2009 Advanced Internet Designs Inc.
* email                : forum(at)prohost(dot)org
* $Id: reindex.php,v 1.36 2009/02/16 05:37:11 frank Exp $
*
* This program is free software; you can redistribute it and/or modify it
* under the terms of the GNU General Public License as published by the
* Free Software Foundation; version 2 of the License.
**/

// this is the same file as indexdb.php, except CLI code is included

// Modified to allow throttling so the website can remain running.
// A config file is used to control reindexing speed.  It is named
// reindex.ini, which has a single number in it - the delay after
// each insert into fud28_search_cache.  The delay is expressed
// in seconds and can be less than 1, e.g. 0.25 or 0.5.

	$default_throttle   =  500000;
	$configfile         = "reindex.throttle";

	@set_time_limit(0);
	@ini_set("memory_limit", "100M");

	require('./GLOBALS.php');
	fud_use('adm_cli.inc', 1);
	cli_execute(1);

	fud_use('adm.inc', true);
	fud_use('glob.inc', true);
	fud_use('isearch.inc');
	fud_use('fileio.inc');
	fud_use('rev_fmt.inc');

	if ($FUD_OPT_1 & 1) {
		echo "\nDisabling the forum for the duration of maintenance run\n";
		maintenance_status('Undergoing maintenance, please come back later.', 1);
	}

	echo "\nPlease wait while index is being rebuilt.\nThis may take a while depending on the size of your forum.";
	ob_flush(); flush();

	$tbl =& $DBHOST_TBL_PREFIX;

	if (defined('forum_debug')) {
		list($locale, $GLOBALS['usr']->lang) = db_saq("SELECT locale, lang FROM {$tbl}themes WHERE theme_opt & (1|2) LIMIT 1");
		$GLOBALS['good_locale'] = setlocale(LC_ALL, $locale);
	}

	db_lock($tbl.'msg_store WRITE, '.$tbl.'search_cache WRITE, '.$tbl.'search WRITE, '.$tbl.'index WRITE, '.$tbl.'title_index WRITE, '.$tbl.'msg WRITE');
	q('DELETE FROM '.$tbl.'search');
	q('DELETE FROM '.$tbl.'index');
	q('DELETE FROM '.$tbl.'title_index');
	if (!($sid = q_singleval("SELECT MIN(query_type) FROM ".$tbl."search_cache WHERE srch_query='' AND query_type<0"))) {
		q('DELETE FROM '.$tbl.'search_cache');
	}

	$i = 0;
	$c = q('SELECT id, subject, length, foff, file_id FROM '.$tbl.'msg WHERE '.($sid ? ' id>'.$sid.' AND ' : '').' apr=1 ORDER BY subject');
	$qty = mysql_num_rows($c);
	echo "\n\nRe-indexing " . $qty . " messages.";

	$old_subject = '';
	while ($r = db_rowarr($c)) {
		if ($old_subject != $r[1]) {
			$subj = $old_subject = $r[1];
		} else {
			$subj = '';
		}
		q('INSERT INTO '.$tbl.'search_cache (srch_query, query_type, expiry, msg_id, n_match) VALUES(\'\', -'.$r[0].', 0,0,0)');
		index_text($subj, read_msg_body($r[3], $r[2], $r[4]), $r[0]);
		$i++;
		db_unlock();
		echo "\nProcessed " . $i;
		// Throttling mechanism
		$fud_reindex_delay = 0;
		if (file_exists($configfile)) {
			$config = fopen($configfile, "r");
			$fud_reindex_delay = trim(fgets($config));
			fclose($config);
		}
		if ($fud_reindex_delay > 0) {
			$throttle = $fud_reindex_delay * 1000000;
		} else {
			$throttle = $default_throttle;
		}
		usleep($throttle);  // Delay
		db_lock($tbl.'msg_store WRITE, '.$tbl.'search_cache WRITE, '.$tbl.'search WRITE, '.$tbl.'index WRITE, '.$tbl.'title_index WRITE, '.$tbl.'msg WRITE');
	}
	unset($c);
	q('DELETE FROM '.$tbl.'search_cache');
	db_unlock();

	echo "\n\nDone!  All messages indexed\n";

	if ($FUD_OPT_1 & 1) {
		echo "\nRe-enabling the forum.\n";
		maintenance_status($GLOBALS['DISABLED_REASON'], 0);
	} else {
		echo "\nYour forum is currently disabled, to re-enable it go to the Global Settings Manager and re-enable it.\n";
	}
?>

The original code locked the database, parsed ALL the messages and inserted all the indexed records and then unlocked the database. This method works fine on small forums but takes too long on larger forums. It also steadily increases memory usage so the whole system begins to crawl after indexing a few thousand messages.

At first, I broke the "while" loop into sections of 1000 records, locking at first, doing 1000 messages and then unlocking. But this still had problems so I decreased the number of records processed in each group to 100 and then even to 10. That worked pretty well. Then I realized I could even lock/unlock for each individual message, and put a variable delay in between each one. Empirically, I found that a 1/2 second delay "throttle" pretty much ran as well as having no delay at all on forums larger than about 10,000 messages, and this mechanism allows the system to be usable to the "outside world." I can re-index a production site this way, and it only seems slightly slower. Essentially, the re-indexing function is kind of like a person is creating a post every half-second. And if you want to increase or decrease the cycle time just put a value in the "reindex.throttle" file.


[Updated on: Fri, 14 September 2018 20:33]

Report message to a moderator

Re: Migration Problem [message #187583 is a reply to message #187582] Fri, 14 September 2018 22:49 Go to previous messageGo to next message
art is currently offline  art   
Messages: 35
Registered: June 2009
Karma: 0
Member

This time, I got past the point where the v.3.0.9 upgrade script was creating the composite primary key on fud28_index(word_id,msg_id). So I think my assumption that fud28_index had duplicates was probably the sticking point there. That plus the fact that I've seen them.

But this time the 3.0.9 upgrade script broke on line 1036:

Fatal error: Call to undefined function q_limit() in ./forum/upgrade.php on line 1036

I've seen that code in the FUDforum cross-reference so I'm adding the q_limit() function (shown below) at the top of the 3.0.9 upgrade script. I expect the function was introduced in an earlier version of the code between 3.0.0 and 3.0.9, and I've skipped it. I'll take the chance that's all I'm missing, and I'm adding the function to the upgrade script.

function q_limit($query, $limit, $off=0)
{
	if ($off) {
		return $query .' LIMIT '. $off .','. $limit;
	} else {
		return $query .' LIMIT '. $limit;
	}
}

So I'm back to the starting point:

1. Drop all tables in the migration-sandbox database.
2. Import the archived SQL dump of the v.2.8 database.
3. Delete all the forum files and subdirectories.
4. Install the v.2.8 flat files from the backup tarball.
5. Copy the modified GLOBALS.php file in place (with db/user for the sandbox database).
6. Delete the fud28_index, fud28_title_index, fud28_search and fud28_search_cache tables.
7. Run the consistency check.
8. Run the v.3.0.0 upgrade script.
9. Chmod 0755 ./forum/adm and run the consistency check.
10. Run the modified v.3.0.9 upgrade script (with q_limit function added).
11. If it works, run the consistency check.
12. Create a v.3.0.9 theme with the same look-and-feel as my v.2.8.0 forum.

We'll see how it goes...

[Updated on: Fri, 14 September 2018 23:21]

Report message to a moderator

Re: Migration Problem [message #187584 is a reply to message #187583] Fri, 14 September 2018 23:45 Go to previous messageGo to next message
GaryM is currently offline  GaryM   United States
Messages: 118
Registered: May 2013
Location: Seattle
Karma: 3
Senior Member
Wow Art! Good luck in getting to 3.0.9. You've leaped a 100 hurtles to get to base 3.x. Myself I'm so glad to take the next release when it comes along. I would have given up at this long ago if I was at 2.x.

Frank you may want to sticky this thread so others migrating from the 2.x base see what work is in store for them before posting.

Good luck Art and hopefully we'll see you on the other side.

Gary


[Updated on: Fri, 14 September 2018 23:45]

Report message to a moderator

Re: Migration Problem [message #187585 is a reply to message #187583] Fri, 14 September 2018 23:47 Go to previous messageGo to next message
art is currently offline  art   
Messages: 35
Registered: June 2009
Karma: 0
Member

Thanks for rooting for me, Gary!

That procedure got me all the way through the v.3.0.9 upgrade script except that it couldn't run the consistency checker at the end. That didn't surprise me because the v.3.0.0 upgrade script couldn't do it either. But every time I've run the v.3.0.0 upgrade script, I've simply turned on the least-significant (0x0001) bit of FUD_OPT_1, set the ./forum/adm directory permissions to 0755 and then run the consistency check manually. That has worked every time.

As usual after an upgrade, I found the least-significant bit of FUD_OPT_1 off so I turned it on. That removes the internet-facing "doing maintenance" message and displays the threads. So after doing the upgrade, I looked at the internet-facing view of the forum and saw all the posts, but it was missing the categories. This is a symptom I regularly see after table manipulation, and running the consistency checker always seems to fix that.

But even after setting the ./forum/adm directory permissions to 755, I still couldn't login as "admin." So I ran the consistency checker from the command line. This fixed the categories problem, as I often see. The consistency checker isn't seeing anything wrong with the database. All the categories now display properly in the main thread view for an anonymous user. I just cannot login.

I think I'm going to call it an evening and look at this again tomorrow. I'm not going to backout the v.3.0.9 installation, because it may just need some massaging to repair.


[Updated on: Fri, 14 September 2018 23:49]

Report message to a moderator

Re: Migration Problem [message #187586 is a reply to message #187585] Sat, 15 September 2018 16:53 Go to previous messageGo to next message
art is currently offline  art   
Messages: 35
Registered: June 2009
Karma: 0
Member

It appears the main problem with my v.3.0.9 system was PHP version was 5.6. I can choose from multiple PHP options, so I set it to v.7.1. Then FUDforum ran clean for the most part. The one weird thing I've found is a blank screen is always shown after login. The URL is ./forum/index.php?t=login and the screen is blank. After that, if you manually go to the ./forum/index.php URL, it shows the forum and you're logged in. Same thing happens after logging out.


[Updated on: Sat, 15 September 2018 16:54]

Report message to a moderator

Re: Migration Problem [message #187587 is a reply to message #187586] Sat, 15 September 2018 17:34 Go to previous messageGo to next message
naudefj is currently offline  naudefj   United States
Messages: 3771
Registered: December 2004
Karma: 28
Senior Member
Administrator
Core Developer
A blank screen is probably an indication of a PHP error. Check you web server's error log for details.
Re: Migration Problem [message #187588 is a reply to message #187587] Sun, 16 September 2018 00:14 Go to previous messageGo to next message
art is currently offline  art   
Messages: 35
Registered: June 2009
Karma: 0
Member

Thanks, I was pretty busy today and couldn't work on it until just now. But that's the first thing I did when I sat down with it this evening.

It was giving this error:

[16-Sep-2018 00:00:25 UTC] PHP Warning: Cannot modify header information - headers already sent by (output started at ./forum/include/GLOBALS.php:1) in ./forum/index.php on line 1104

So I looked at GLOBALS.php. What I found was whitespace in front of </php. Removed that and voilĂ ! Everything worked.

Not sure what caused that but I know it's working now. So I think I'll create a theme for v.3.0.9 that has the same look-and-feel as my old v.2.8.0 theme. Then I'll grab the latest copies of my flat files and database from the production FUDforum server. Getting close to putting this to bed.

Thanks for all you do guys!


Re: Migration Problem [message #187591 is a reply to message #187588] Mon, 17 September 2018 23:32 Go to previous message
GaryM is currently offline  GaryM   United States
Messages: 118
Registered: May 2013
Location: Seattle
Karma: 3
Senior Member
Glad to see you made it to the latest and greatest Art! Like you I was having issues with the older PHP version. I think I was also on 5.6 or at least the 5 branch. Once on version 7 I was golden again.

We just have to continue keeping up with the Joneses. Wink

Gary


  Switch to threaded view of this topic Create a new topic Submit Reply
Previous Topic: upgrade from 3.07 to 3.09 fails
Next Topic: Failing to change forum to RTL
Goto Forum:
  

-=] Back to Top [=-
[ Syndicate this forum (XML) ] [ RSS ]

Current Time: Wed Nov 27 04:19:36 GMT 2024

Total time taken to generate the page: 0.02666 seconds