FUDforum
Fast Uncompromising Discussions. FUDforum will get your users talking.

Home » FUDforum » FUDforum Suggestions » MAILMAN: using maillist.php as PRIVATE_EXTERNAL_ARCHIVER
Show: Today's Messages :: Polls :: Message Navigator
Switch to threaded view of this topic Create a new topic Submit Reply
MAILMAN: using maillist.php as PRIVATE_EXTERNAL_ARCHIVER [message #10868] Thu, 12 June 2003 16:32 Go to next message
Tobias Eigen is currently offline  Tobias Eigen   United States
Messages: 85
Registered: June 2003
Location: Seattle, WA USA
Karma: 0
Member
Hi all,

I have been gratefully exploring FUD - it is a beautiful program. Many thanks for developing it and sharing it with the open source community. It will certainly help my organization to help African civil society organizations to make better use of mailing lists for networking and information sharing.

I have a suggestion for a feature that might make alot of mailman administrators happy. If it were possible to specify the list name instead of the forum topic number when running maillist.php, then we would be able to set up FUD as the Mailman archiver. This would be very slick indeed.

Mailman generates mbox archives as well as web-based archives, either with PIPERMAIL (the built-in archiver) or MHONARC. Below is the command I have set in ~mailman/Mailman/mm_cfg.py to use MHONARC as the private external archiver:

PRIVATE_EXTERNAL_ARCHIVER = '/usr/bin/mhonarc -add -outdir
/usr/local/mailman/archives/private/%(listname)s -rcfile
/usr/local/mailman/archives/private/mhonarc.rc -htmlext html -idxfname index.html -title
%(listname)s -reverse -spammode >> /usr/local/mailman/logs/mhonarc_private'


I'd propose to change this to something like this:

PRIVATE_EXTERNAL_ARCHIVER = '/path/to/maillist.php %(listname)s

In python, %(listname)s is replaced with the actual list name, ie. the part of the address before the @ symbol (ie. test-list for test-list(at)lists(dot)kabissa(dot)org). With this setup, I don't need to do anything special to configure mailman archives.. it simply works by default. No aliases to set up, etc.

This setup is also completely secure - nothing gets archived without going through mailman first. This means that mailman can bounce spam and moderators can reject unwanted posts so they are not archived. It's also not possible to send email directly to the archive - which the current FUD solutions all seem to make possible (ie. mail to foo-archive(at)foo(dot)org is posted to the forum automatically - not necessarily a good thing).

I don't know how FUD runs internally, but if we could specify the mailman listname in the mailing list admin interface and use it when running maillist.php, then mailman/fud integration would be pretty much seamless. An alternative would be to strip out the first part of the list's email address to get the list name.

Another angle altogether might be to emulate a slick little script I use on my qmail system called qmail-to-mailman.py. Found in the mailman /contrib directory, this script can be set up to be run via a .qmail file in a dedicated domain mailbox. It takes the message, strips the @foo.com from it to get the listname, then sends the message to mailman for handling on that list. If the list doesn't exist, it returns a helpful bounce message. Works completely flawlessly.

I'd be happy to help out in any way I can - I'm not a programmer but am happy to test out and troubleshoot.

Cheers,

Tobias





Kabissa - Space for change in Africa
Re: MAILMAN: using maillist.php as PRIVATE_EXTERNAL_ARCHIVER [message #10880 is a reply to message #10868] Thu, 12 June 2003 17:05 Go to previous messageGo to next message
Ilia is currently offline  Ilia   Canada
Messages: 13241
Registered: January 2002
Karma: 0
Senior Member
Administrator
Core Developer
Cannot be done without adding mailman specific hacks and I refuse to do that. The only possiblity is to allow maillist.php to accept the list's e-mail address identifier rather then the list id. That can be done with fairly light alterations to the current code.

FUDforum Core Developer
Re: MAILMAN: using maillist.php as PRIVATE_EXTERNAL_ARCHIVER [message #10882 is a reply to message #10880] Thu, 12 June 2003 17:13 Go to previous messageGo to next message
Tobias Eigen is currently offline  Tobias Eigen   United States
Messages: 85
Registered: June 2003
Location: Seattle, WA USA
Karma: 0
Member
Hmm.. makes sense. The email address won't work for us, though. What about using the forum name itself? I would be happy if I could call the forum test-list and use/path/to/mailllist.php test-list in my PRIVATE_EXTERNAL_ARCHIVER.

Thanks for the quick response. Smile

Cheers,

Tobias


Kabissa - Space for change in Africa
Re: MAILMAN: using maillist.php as PRIVATE_EXTERNAL_ARCHIVER [message #10884 is a reply to message #10882] Thu, 12 June 2003 17:19 Go to previous messageGo to next message
Ilia is currently offline  Ilia   Canada
Messages: 13241
Registered: January 2002
Karma: 0
Senior Member
Administrator
Core Developer
The forum name my contain special characters that would require all sort's of encoding to be safely passed via the shell. So for security reasons it is probably not a good idea to use.

That said, you could easily implement some go between function that you could add to maillist.php that would translate any input into the understood rule id.

For example:
<?php
$mailman_id
= array('LIST1' => 1, 'LIST2' => 2);

if (isset(
$mailman_id[$_SERVER['argv'][1]])) {
$_SERVER['argv'][1] = $mailman_id[$_SERVER['argv'][1]];
} else {
$_SERVER['argv'][1] = NULL;
}
?>


FUDforum Core Developer
Re: MAILMAN: using maillist.php as PRIVATE_EXTERNAL_ARCHIVER [message #10886 is a reply to message #10882] Thu, 12 June 2003 17:21 Go to previous messageGo to next message
Tobias Eigen is currently offline  Tobias Eigen   United States
Messages: 85
Registered: June 2003
Location: Seattle, WA USA
Karma: 0
Member
You know - I just had a thought.. your suggestion might work if the line reads like this:

PRIVATE_EXTERNAL_ARCHIVER = '/path/to/maillist.php %(listname)s(at)lists(dot)kabissa(dot)org'

I'm not quite sure what happens to messages to lists that are set up to 'prefer' a domain name other than the default lists.kabissa.org, but I think it might just work.

How do we proceed? I'd love to try this out.

Cheers,

Tobias


Kabissa - Space for change in Africa
Re: MAILMAN: using maillist.php as PRIVATE_EXTERNAL_ARCHIVER [message #10888 is a reply to message #10886] Thu, 12 June 2003 17:29 Go to previous messageGo to next message
Ilia is currently offline  Ilia   Canada
Messages: 13241
Registered: January 2002
Karma: 0
Senior Member
Administrator
Core Developer
Basically the e-mail address provided needs to be indetical the one specified as the 'Mailing List Email' on the mailling list manager.

To try it do the following changes:

Change:
if ($_SERVER['argc'] < 2 || (!($fid = (int)$_SERVER['argv'][1]))) {

To:

if ($_SERVER['argc'] < 2) {

Change:

$mlist = db_sab('SELECT * FROM '.sql_p.'mlist WHERE id='.$fid);


To:

$mlist = db_sab('SELECT * FROM '.sql_p.'mlist WHERE name=\''.addslashes($_SERVER['argv'][1]).'\'');


FUDforum Core Developer
icon14.gif  Re: MAILMAN: using maillist.php as PRIVATE_EXTERNAL_ARCHIVER [message #10890 is a reply to message #10888] Thu, 12 June 2003 17:36 Go to previous messageGo to next message
Tobias Eigen is currently offline  Tobias Eigen   United States
Messages: 85
Registered: June 2003
Location: Seattle, WA USA
Karma: 0
Member
Brilliant!

This worked. Many thanks. Will you incorporate this into the next version, or will I need to do this hack again in the future?

Cheers,

Tobias


Kabissa - Space for change in Africa
Re: MAILMAN: using maillist.php as PRIVATE_EXTERNAL_ARCHIVER [message #10892 is a reply to message #10890] Thu, 12 June 2003 17:37 Go to previous messageGo to next message
Ilia is currently offline  Ilia   Canada
Messages: 13241
Registered: January 2002
Karma: 0
Senior Member
Administrator
Core Developer
The exact patch will not be used, but I will use something similar to allow specification of the list e-mail address and the 'key' in the next release.

FUDforum Core Developer
Re: MAILMAN: using maillist.php as PRIVATE_EXTERNAL_ARCHIVER [message #10893 is a reply to message #10892] Thu, 12 June 2003 17:39 Go to previous messageGo to next message
Tobias Eigen is currently offline  Tobias Eigen   United States
Messages: 85
Registered: June 2003
Location: Seattle, WA USA
Karma: 0
Member
Sweet - thanks much.

Kabissa - Space for change in Africa
Re: MAILMAN: using maillist.php as PRIVATE_EXTERNAL_ARCHIVER [message #10979 is a reply to message #10868] Sun, 15 June 2003 18:51 Go to previous messageGo to next message
Anonymous   United States
Hi,

Just about to start testing maillist.php as external archiver for mailman.. and had a question for you. What happens when a message can't be imported for whatever reason? Ie. the forum was not properly identified on the command-line? Is there a log file that is added to, and is there a directory where "bad" messages get saved? If not, these features might be a good idea to add for error-tracking.

Cheers,

Tobias

(I can edit this anonymous post - why is that?)

[Updated on: Sun, 15 June 2003 22:17]

Report message to a moderator

Re: MAILMAN: using maillist.php as PRIVATE_EXTERNAL_ARCHIVER [message #10981 is a reply to message #10979] Sun, 15 June 2003 22:15 Go to previous messageGo to next message
Tobias Eigen is currently offline  Tobias Eigen   United States
Messages: 85
Registered: June 2003
Location: Seattle, WA USA
Karma: 0
Member
btw - this anonymous person is me. Smile I forgot to log in.

(I was struck that before I logged in, I could edit my last post - after logging in I could not.. ?)


Kabissa - Space for change in Africa
Re: MAILMAN: using maillist.php as PRIVATE_EXTERNAL_ARCHIVER [message #10986 is a reply to message #10981] Mon, 16 June 2003 01:35 Go to previous messageGo to next message
Ilia is currently offline  Ilia   Canada
Messages: 13241
Registered: January 2002
Karma: 0
Senior Member
Administrator
Core Developer
Yes, there is a log file where the error messages go to, it can be found inside theerrors/.mlist directory of the forum.

FUDforum Core Developer
Re: MAILMAN: using maillist.php as PRIVATE_EXTERNAL_ARCHIVER [message #11077 is a reply to message #10868] Tue, 17 June 2003 16:43 Go to previous messageGo to next message
Tobias Eigen is currently offline  Tobias Eigen   United States
Messages: 85
Registered: June 2003
Location: Seattle, WA USA
Karma: 0
Member
Hi,

I'm having a little trouble getting maillist.php working as PRIVATE_EXTERNAL_ARCHIVER in Mailman. I know this is a little specific, but I think other Mailman administrators would also appreciate having this figured out.

I'm getting the error message below when the list does not correspond with an existing forum -

<b>UNABLE TO WRITE TO SQL LOG FILE</b><br>
(maillist.php) 1064: You have an error in your SQL syntax near '' at line 1<br />
Query: SELECT id, tag_style, message_threshold, (max_attach_size * 1024) AS max_attach_size, max_file_attachments FROM fud25_forum WHERE id=<br />
Server Version: 3.23.54<br />


Can I have these messages dumped into a forum for me to review and/or move to their proper place, then set up properly?

I'm also trying to set it up so that both mhonarc and maillist.php work simultaneously, while I'm setting up FUD forum. This is not working for me. Here's the line I'm using in ~Mailman/mm_cfg.py:

PRIVATE_EXTERNAL_ARCHIVER = '(/usr/bin/mhonarc -add -outdir
/usr/local/mailman/archives/private/%(listname)s -rcfile
/usr/local/mailman/archives/private/mhonarc.rc -htmlext html -idxfname index.html -title
%(listname)s -reverse -spammode ;
/home/httpd/vhosts/kabissa.org/httpdocs/team/fud-data/scripts/maillist.php
%(listname)s(at)lists(dot)kabissa(dot)org)'


Maillist.php generates a line in the errors/.mlist/error_log file and creates a zero-length filename in the same directory like this:

ERROR :: Tue, 17 Jun 2003 11:29:45 -0500 :: no name or email for :: /home/httpd/vhosts/kabissa.org/httpdocs/team/fud-data/errors/.mlist/1055867 385_d41d8cd98f00b204e9800998ecf8427e

I presume this means it's not getting the message piped to it - so I'm clearly doing something wrong.

Any thoughts?

Cheers,

Tobias


Kabissa - Space for change in Africa
Re: MAILMAN: using maillist.php as PRIVATE_EXTERNAL_ARCHIVER [message #11079 is a reply to message #11077] Tue, 17 June 2003 17:03 Go to previous messageGo to next message
Ilia is currently offline  Ilia   Canada
Messages: 13241
Registered: January 2002
Karma: 0
Senior Member
Administrator
Core Developer
The second error message indicates that the script failed to locate the header which would tell it who the message came from (From: header).

The 1st error message sounds like the list identifier that is passed is not correct and it failes to retrieve data from MySQL about this list rule.


FUDforum Core Developer
Re: MAILMAN: using maillist.php as PRIVATE_EXTERNAL_ARCHIVER [message #11090 is a reply to message #10868] Tue, 17 June 2003 18:04 Go to previous messageGo to next message
Tobias Eigen is currently offline  Tobias Eigen   United States
Messages: 85
Registered: June 2003
Location: Seattle, WA USA
Karma: 0
Member
OK - I think I've come that far myself. I need to figure out how to fix them. Let me know if you have any suggestions.

Cheers,

Tobias


Kabissa - Space for change in Africa
Re: MAILMAN: using maillist.php as PRIVATE_EXTERNAL_ARCHIVER [message #11091 is a reply to message #11090] Tue, 17 June 2003 18:05 Go to previous messageGo to next message
Ilia is currently offline  Ilia   Canada
Messages: 13241
Registered: January 2002
Karma: 0
Senior Member
Administrator
Core Developer
No idea Wink I am not sure how you are executing the script so I don't know.

Ideally you'd put some debug code inside maillist.php to 1st of all ensure that the variable you pass it to does match one of the forum rules & then make sure that the message (body + headers) gets piped correctly.


FUDforum Core Developer
Re: MAILMAN: using maillist.php as PRIVATE_EXTERNAL_ARCHIVER [message #11092 is a reply to message #11091] Tue, 17 June 2003 18:18 Go to previous messageGo to next message
Tobias Eigen is currently offline  Tobias Eigen   United States
Messages: 85
Registered: June 2003
Location: Seattle, WA USA
Karma: 0
Member
Cool I'll keep plowing away at it.

Basically Mailman is piping the message to maillist.php in the same way that a .qmail file would. Mailman provides the list name via a variable, which I use to generate the listname(at)lists(dot)kabissa(dot)org on the command line. This works fine for lists configured in FUD, and when it's the only program configured as the external archiver.

If the listname(at)lists(dot)kabissa(dot)org rule isn't specified in FUD someplace, then it chokes on the message and saves it in the .mlist error directory. My preference would be for it to be posted to a default holding forum where I can then deal with it by deleting it or creating the appropriate forum and moving the message there.

I've had no luck getting Mailman to pipe the message to two external archivers at the same time - this is a disappointment since it hampers my testing and transition efforts. I must be missing something about how shell commands work. There must be a way to it, ie.

cat blah.txt | (program1 >> log1 ; program 2 >> log2)


Cheers,

Tobias


Kabissa - Space for change in Africa

[Updated on: Tue, 17 June 2003 18:18]

Report message to a moderator

Re: MAILMAN: using maillist.php as PRIVATE_EXTERNAL_ARCHIVER [message #11098 is a reply to message #11092] Tue, 17 June 2003 19:04 Go to previous messageGo to next message
Ilia is currently offline  Ilia   Canada
Messages: 13241
Registered: January 2002
Karma: 0
Senior Member
Administrator
Core Developer
You can easily hackup the default forum by doing:

<?php
if (isset($_SERVER['argv'][1])) {
$_SERVER['argv'][1] = default_rule_id;
$_SERVER['argc']++;
}
?>


FUDforum Core Developer
Re: MAILMAN: using maillist.php as PRIVATE_EXTERNAL_ARCHIVER [message #11104 is a reply to message #11092] Tue, 17 June 2003 19:30 Go to previous messageGo to next message
Tobias Eigen is currently offline  Tobias Eigen   United States
Messages: 85
Registered: June 2003
Location: Seattle, WA USA
Karma: 0
Member
Sweet - a kind soul from the mailman-users list provided me with the answer to one of my MAILMAN/FUD questions:

http://www.mail-archive.com/mailman-users%40python.org/msg17520.html

The answer is to do it like this:

PRIVATE_EXTERNAL_ARCHIVER = 'cat > /tmp/mailman_pvt_tmp$$; archiver-1-command 
%(listname)s-stuff < /tmp/teststuff$$; archiver-2-command 
%(listname)s-stuff < /tmp/teststuff$$; rm /tmp/teststuff$$'


This isn't elegant for the long term, but at least now I can go ahead and continue testing FUD and migrating my lists into it. The only problem that remains is that mail to unknown forums is being treated as an error instead of being imported into a holding forum where I can deal with it manually.

This should be possible via maillist.php - can you possibly give me some guidance?

Cheers,

Tobias


Kabissa - Space for change in Africa
Re: MAILMAN: using maillist.php as PRIVATE_EXTERNAL_ARCHIVER [message #11105 is a reply to message #11104] Tue, 17 June 2003 20:14 Go to previous messageGo to next message
Ilia is currently offline  Ilia   Canada
Messages: 13241
Registered: January 2002
Karma: 0
Senior Member
Administrator
Core Developer
In RC6 you can provide the maillist.php script with an e-mail address of the list rather then id.

so

cat message.txt | php maillist.php mylist(at)email(dot)com will work


FUDforum Core Developer
Re: MAILMAN: using maillist.php as PRIVATE_EXTERNAL_ARCHIVER [message #11107 is a reply to message #11098] Tue, 17 June 2003 20:35 Go to previous messageGo to next message
Tobias Eigen is currently offline  Tobias Eigen   United States
Messages: 85
Registered: June 2003
Location: Seattle, WA USA
Karma: 0
Member
Hi Protoss,

Thanks for the very proactive help today - I really appreciate it. I'm psyched regarding the ability to use the email address instead of the list id. Many thanks.

Regarding the default forum - above you wrote:

Quote:

You can easily hackup the default forum by doing:
<?php
if (isset($_SERVER['argv'][1])) {
$_SERVER['argv'][1] = default_rule_id;
$_SERVER['argc']++;
}
?> 



Where can I include this? Sorry for being thick. Where do I specify the default forum id/name?

Cheers,

Tobias


Kabissa - Space for change in Africa
Re: MAILMAN: using maillist.php as PRIVATE_EXTERNAL_ARCHIVER [message #11108 is a reply to message #11107] Tue, 17 June 2003 20:40 Go to previous messageGo to next message
Ilia is currently offline  Ilia   Canada
Messages: 13241
Registered: January 2002
Karma: 0
Senior Member
Administrator
Core Developer
The code should go above this line:
<?php
if ($_SERVER['argc'] < 2) {
?>


FUDforum Core Developer
Re: MAILMAN: using maillist.php as PRIVATE_EXTERNAL_ARCHIVER [message #11109 is a reply to message #11108] Tue, 17 June 2003 20:57 Go to previous messageGo to next message
Tobias Eigen is currently offline  Tobias Eigen   United States
Messages: 85
Registered: June 2003
Location: Seattle, WA USA
Karma: 0
Member
<sigh> this isn't working for me - remember I had commented out the line just above the one you're talking about - perhaps that's causing me problems:

//      if ($_SERVER['argc'] < 2 || (!($fid = (int)$_SERVER['argv'][1]))) {

        if (isset($_SERVER['argv'][1])) {
        $_SERVER['argv'][1] = 4;
        $_SERVER['argc']++;
        }


Anyhoo - no tragedy for the moment, I'm pleased with my successes today and will head home. I will take the time tomorrow to create a forum for each list hosted on this server.

Before I do that I will reinstall FUD with default values again - is there a new version I should get?

Thanks again,

Cheers,

Tobias


Kabissa - Space for change in Africa
Re: MAILMAN: using maillist.php as PRIVATE_EXTERNAL_ARCHIVER [message #11110 is a reply to message #11109] Tue, 17 June 2003 21:24 Go to previous messageGo to next message
Ilia is currently offline  Ilia   Canada
Messages: 13241
Registered: January 2002
Karma: 0
Senior Member
Administrator
Core Developer
You do not have to re-install just upgrade to RC6 (which is the lastest 2.5.0 release at the moment).

FUDforum Core Developer
Re: MAILMAN: using maillist.php as PRIVATE_EXTERNAL_ARCHIVER [message #11115 is a reply to message #10868] Wed, 18 June 2003 00:09 Go to previous messageGo to next message
Tobias Eigen is currently offline  Tobias Eigen   United States
Messages: 85
Registered: June 2003
Location: Seattle, WA USA
Karma: 0
Member
OK - glad to know it's just a matter of doing an upgrade.

I do want to wipe it clean - ie. the user database and forums - before starting again. I've got alot of test messages and users in there.

Cheers,

Tobias


Kabissa - Space for change in Africa
Re: MAILMAN: using maillist.php as PRIVATE_EXTERNAL_ARCHIVER [message #11204 is a reply to message #11115] Sat, 21 June 2003 19:21 Go to previous messageGo to next message
Tobias Eigen is currently offline  Tobias Eigen   United States
Messages: 85
Registered: June 2003
Location: Seattle, WA USA
Karma: 0
Member
I just tried out the default forum hack and it is now sending everything into there - I just want the messages to lists that are not defined to go in there.

Here again is the hack:

Quote:

if (isset($_SERVER['argv'][1])) {
$_SERVER['argv'][1] = 3;
$_SERVER['argc']++;
}


3 is my default forum. I put it just before the following line like you said:

Quote:

if ($_SERVER['argc'] < 2) {


Getting closer and closer! Smile Would appreciate any assistance!

Cheers,

Tobias


Kabissa - Space for change in Africa
Re: MAILMAN: using maillist.php as PRIVATE_EXTERNAL_ARCHIVER [message #11205 is a reply to message #11204] Sat, 21 June 2003 20:14 Go to previous message
Ilia is currently offline  Ilia   Canada
Messages: 13241
Registered: January 2002
Karma: 0
Senior Member
Administrator
Core Developer
In that case change:
<?php
if (isset($_SERVER['argv'][1])) {
?>


to
<?php
if (!isset($_SERVER['argv'][1])) {
?>


Now only time maillist.php is called without a list id, will it try sending the message to the default forum.


FUDforum Core Developer
  Switch to threaded view of this topic Create a new topic Submit Reply
Previous Topic: Subscription feature improvements
Next Topic: SEARCH: search for posts by username
Goto Forum:
  

-=] Back to Top [=-
[ Syndicate this forum (XML) ] [ RSS ]

Current Time: Thu Dec 12 14:16:05 GMT 2024

Total time taken to generate the page: 0.02619 seconds