Data dump hangs [message #34111] |
Fri, 06 October 2006 05:09 |
mdbrown
Messages: 18 Registered: December 2005
Karma: 0
|
Junior Member |
|
|
On my FUDforum 2.7.3 installation a forum datadump apparently hangs while processing the index table. I've let it cook all night, but in the morning it's still stuck on the fud26_index. I'm getting a little nervous about not having a current backup, so I hope someone has a solution. Here's the datadump screen display for your perusal:
Compressing forum datafiles
Processing directory: /home/rvconnection/FUDforum
Processing directory: /home/rvconnection/FUDforum/thm
Processing directory: /home/rvconnection/FUDforum/scripts
Processing directory: /home/rvconnection/FUDforum/include
Processing directory: /home/rvconnection/FUDforum/messages
Processing directory: /home/rvconnection/FUDforum/files
Processing directory: /home/rvconnection/FUDforum/sql
Processing directory: /home/rvconnection/FUDforum/src
Processing directory: /home/rvconnection/FUDforum/errors
Processing directory: /home/rvconnection/FUDforum/cache
Processing directory: /home/rvconnection/FUDforum/template
Processing directory: /home/rvconnection/FUDforum/thm/default
Processing directory: /home/rvconnection/FUDforum/thm/path_info
Processing directory: /home/rvconnection/FUDforum/sql/mysql
Processing directory: /home/rvconnection/FUDforum/sql/pdo
Processing directory: /home/rvconnection/FUDforum/sql/pgsql
Processing directory: /home/rvconnection/FUDforum/errors/.nntp
Processing directory: /home/rvconnection/FUDforum/errors/.mlist
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n
Processing directory: /home/rvconnection/FUDforum/thm/default/tmpl
Processing directory: /home/rvconnection/FUDforum/thm/path_info/tmpl
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/chinese_big5
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/spanish
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/english
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/bulgarian
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/finnish
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/russian-1251
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/italian
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/portuguese_br
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/german
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/swedish
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/norwegian
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/turkish
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/czech
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/chinese
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/dutch
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/russian
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/latvian
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/korean
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/romanian
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/slovak
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/french
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/lithuanian
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/portuguese
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/japanese
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/polish
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/indonesian
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/chinese_big5/img
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/chinese_big5/help
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/spanish/img
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/english/img
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/english/help
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/bulgarian/img
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/bulgarian/help
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/finnish/img
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/russian-1251/img
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/russian-1251/help
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/italian/img
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/italian/help
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/portuguese_br/img
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/german/img
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/german/help
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/swedish/img
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/norwegian/img
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/turkish/img
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/turkish/help
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/czech/img
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/chinese/img
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/chinese/help
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/dutch/img
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/dutch/help
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/russian/img
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/russian/help
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/latvian/img
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/romanian/img
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/slovak/img
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/french/img
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/french/help
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/lithuanian/img
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/portuguese/img
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/polish/img
Processing directory: /home/rvconnection/FUDforum/thm/default/i18n/polish/help
Processing directory: /home/rvconnection/public_html/images
Processing directory: /home/rvconnection/public_html/images/smiley_icons
Processing directory: /home/rvconnection/public_html/images/message_icons
Processing directory: /home/rvconnection/public_html/images/mime
Processing directory: /home/rvconnection/public_html/images/avatars
Processing directory: /home/rvconnection/public_html/images/forum_icons
Processing directory: /home/rvconnection/public_html/images/custom_avatars
Processing directory: /home/rvconnection/public_html/adm
Processing table: fud26_action_log (1212) .... DONE
Processing table: fud26_ann_forums (0) .... DONE
Processing table: fud26_announce (1) .... DONE
Processing table: fud26_attach (1035) .... DONE
Processing table: fud26_avatar (0) .... DONE
Processing table: fud26_blocked_logins (0) .... DONE
Processing table: fud26_buddy (76) .... DONE
Processing table: fud26_cat (4) .... DONE
Processing table: fud26_custom_tags (1) .... DONE
Processing table: fud26_email_block (0) .... DONE
Processing table: fud26_ext_block (0) .... DONE
Processing table: fud26_fc_view (11) .... DONE
Processing table: fud26_forum (11) .... DONE
Processing table: fud26_forum_notify (5) .... DONE
Processing table: fud26_forum_read (330) .... DONE
Processing table: fud26_group_cache (22) .... DONE
Processing table: fud26_group_members (22) .... DONE
Processing table: fud26_group_resources (13) .... DONE
Processing table: fud26_groups (13) .... DONE
Processing table: fud26_index (793793) ....
|
|
|
Re: Data dump hangs [message #34119 is a reply to message #34111] |
Fri, 06 October 2006 15:05 |
Ilia
Messages: 13241 Registered: January 2002
Karma: 0
|
Senior Member Administrator Core Developer |
|
|
It is a very large table as you can tell almost 800,000 records, so it will take some time to dump to disk. I suspect your browser has simply timed out before the request had a chance to complete. I'd recommend running the script via command line for a large forum.
FUDforum Core Developer
|
|
|
|
Re: Data dump hangs [message #34132 is a reply to message #34130] |
Sat, 07 October 2006 00:10 |
Ilia
Messages: 13241 Registered: January 2002
Karma: 0
|
Senior Member Administrator Core Developer |
|
|
Fraid not, it just a big write of lots of data.
FUDforum Core Developer
|
|
|
|
Re: Data dump hangs [message #34141 is a reply to message #34139] |
Sun, 08 October 2006 05:56 |
mdbrown
Messages: 18 Registered: December 2005
Karma: 0
|
Junior Member |
|
|
mdbrown wrote on Sat, 07 October 2006 17:29 | Which command-line script are you referring to, specifically? I don't see any likely candidates in the /FUDforum/scripts directory.
|
Nevermind, I had overlooked adm/admdump.php.
Actually, I ended up making a copy named "clidump.php" and then uncommented the "fud_use('adm_cli.inc', 1)" and "cli_execute('')" lines in it.
Executing the script from the command line, the datadump finished in (ha!) record time -- and the key word is "finished." Furthermore, it passed gzip's integrity check. I'm going to look at the code to see how tough it would be to suppress all the unneeded HTML code that's generated by the script.
But right now I'm in the process of importing the datadump into a new FUDforum installation. It's been chugging away for about twenty minutes, which makes me nervous about what will happen to the import process if my browser times out. I examined admimport.php to see if it could also be executed from the command line, but guess not. Anyway, I'm keeping my fingers crossed that the import finishes without incident.
BTW, have I mentioned how much I love FUDforum? If not, let me correct the oversight here and now.
|
|
|
|
Re: Data dump hangs [message #34151 is a reply to message #34111] |
Mon, 09 October 2006 00:20 |
mdbrown
Messages: 18 Registered: December 2005
Karma: 0
|
Junior Member |
|
|
To heck with using the browser interface to import. The command line is the way to go. I copied admimport.php to cliimport.php and modified it with the --
define('recovery_mode', 1);
$_POST['path'] = '/path/to/dumpfile';
-- method, and fired if off in background mode. Now we're getting somewhere! So far, it has processed 350,000 queries in about an hour and a quarter, so it's not quite half-way finished yet, but it's processing to beat the band. I'm guessing it will take about three hours or so to import the entire 800,000 record dump.
At some point, if no one else has done so, I'll modify both clidump.php and cliimport.php to output non-HTML text to the terminal.
For large, time-consuming operations, running scripts via the command line can't be beat.
|
|
|
Re: Data import blows up [message #34152 is a reply to message #34111] |
Mon, 09 October 2006 01:09 |
mdbrown
Messages: 18 Registered: December 2005
Karma: 0
|
Junior Member |
|
|
D'OH again!
php ./cliimport.php &
Processed 10000 queries<br>Processed 20000 queries<br>Processed 30000 queries<br>Processed 40000 queries<br>Processed 50000 queries<br>Processed 60000 queries<br>Processed 70000 queries<br>Processed 80000 queries<br>Processed 90000 queries<br>Processed 100000 queries<br>Processed 110000 queries<br>Processed 120000 queries<br>Processed 130000 queries<br>Processed 140000 queries<br>Processed 150000 queries<br>Processed 160000 queries<br>Processed 170000 queries<br>Processed 180000 queries<br>Processed 190000 queries<br>Processed 200000 queries<br>Processed 210000 queries<br>Processed 220000 queries<br>Processed 230000 queries<br>Processed 240000 queries<br>Processed 250000 queries<br>Processed 260000 queries<br>Processed 270000 queries<br>Processed 280000 queries<br>Processed 290000 queries<br>Processed 300000 queries<br>Processed 310000 queries<br>Processed 320000 queries<br>Processed 330000 queries<br>Processed 340000 queries<br>Processed 350000 queries<br>Processed 360000 queries<br>Processed 370000 queries<br>Processed 380000 queries<br>Processed 390000 queries<br>Processed 400000 queries<br>Processed 410000 queries<br>Processed 420000 queries<br>Processed 430000 queries<br>Processed 440000 queries<br>Processed 450000 queries<br>Processed 460000 queries<br>Processed 470000 queries<br>Processed 480000 queries<br>Processed 490000 queries<br>
Fatal error: Maximum execution time of 6000 seconds exceeded in /www/assimil8ed/adm/cliimport.php on line 198
[1] Exit 255 php ./cliimport.php
Now what? I'm about 310,000 queries short. If no one has a solution, I guess I'm screwed, blued, and tatooed.
[Updated on: Mon, 09 October 2006 01:29] Report message to a moderator
|
|
|
Re: Data import blows up [message #34153 is a reply to message #34111] |
Mon, 09 October 2006 01:28 |
mdbrown
Messages: 18 Registered: December 2005
Karma: 0
|
Junior Member |
|
|
I increased max_execution_time to 28800 seconds in php.ini. Hope that will solve the problem. But now I have to hack cliimport.php to force it to resume the import at 490000 so as to not duplicate the effort. I guess I'm only screwed and blued, but not tatooed just yet.
[Updated on: Mon, 09 October 2006 01:29] Report message to a moderator
|
|
|
|
Re: Data import blows up [message #34157 is a reply to message #34155] |
Mon, 09 October 2006 13:49 |
Ilia
Messages: 13241 Registered: January 2002
Karma: 0
|
Senior Member Administrator Core Developer |
|
|
Well, increasing the execution time in the script via set_time_limit(0) will probably ensure a complete run. If it still does not work you can forgo the forum's built-in data dump and simply use mysql dump for database info and tar for forum's files.
FUDforum Core Developer
|
|
|
|
Re: Data import blows up [message #34176 is a reply to message #34169] |
Tue, 10 October 2006 13:27 |
Ilia
Messages: 13241 Registered: January 2002
Karma: 0
|
Senior Member Administrator Core Developer |
|
|
The warnings you are seeing can be ignored, they are caused by the fact that you are bypassing authentication system and running the script on the command line.
No progress indicator from import? You may want to add flush() calls to force PHP to dump output to screen right away and not buffer it in memory.
FUDforum Core Developer
|
|
|