FUDforum: How To » Mailing list integration question

Home » FUDforum » How To » Mailing list integration question

Show: Today's Messages :: Polls :: Message Navigator

Mailing list integration question [message #158742]

Tue, 17 March 2009 12:51

Jim O

Messages: 4
Registered: March 2009

Karma: 0

Junior Member

I am in the process of setting up a new forum part of which I want to integrate with a mailing list.

My question regards the headers sent out by FUDforum. Specifically will they include the "In-Reply-To:" and/or "References:" headers so that they are properly sorted by mailing list archives and MUA's?

Jim

Report message to a moderator

Re: Mailing list integration question [message #158743 is a reply to message #158742]

Tue, 17 March 2009 14:10

naudefj

Messages: 3772
Registered: December 2004

Karma: 28

Senior Member
Administrator
Core Developer

Hi Jim,

I haven't tried it myself. However, from "mlist_post.inc" it looks like it would use an "In-Reply-To":

if ($reply_to) {
        $header .= "In-Reply-To: <".$reply_to.">\n";
}

Best regards.

Frank

Report message to a moderator

Re: Mailing list integration question [message #158777 is a reply to message #158743]

Fri, 20 March 2009 01:01

Jim O

Messages: 4
Registered: March 2009

Karma: 0

Junior Member

Thanks Frank.

In doing some testing I found that all messages in a thread seem to default to "in reply to" the original entry, no the most recent one. That could affect threading on list archives and in MUA's.

Also, how hard would it be to add an extra field for references headers and include them? This is a request of list members.

Jim

Report message to a moderator

Re: Mailing list integration question [message #158779 is a reply to message #158777]

Fri, 20 March 2009 15:39

naudefj

Messages: 3772
Registered: December 2004

Karma: 28

Senior Member
Administrator
Core Developer

I shouldn't be too difficult. However, I don't have time to do it right now (unless you can somehow motivate me). If you are willing to take it on, I can guide you through the high-level steps.

Best regards.

Frank

Report message to a moderator

Re: Mailing list integration question [message #167484 is a reply to message #158779]

Sun, 01 July 2012 03:57

cpreston

Messages: 160
Registered: July 2012
Location: Oceanside

Karma: 6

Senior Member

Like to bump this thread. I also would like for the mailing list integration piece to include these headers. Naudefj, I'm prepared to motivate you. Send me a PM on what would do that. Wink

Wink

There's also one other feature I'd like to add to the mailing list module, and I'd be glad to pay for the dev time. On my current phpbb/mail2forum setup, I have moderation turned on. But I've changed the default behavior so that once you've been successfully moderated, you're no longer moderated. Basically I read the person's first post(s). If they're not spam, then I approve their posts. Once I approve their post, their user id is put in a safe_users table, and they're never moderated again. Possible?

Report message to a moderator

Re: Mailing list integration question [message #168785 is a reply to message #167484]

Thu, 25 July 2013 04:41

cpreston

Messages: 160
Registered: July 2012
Location: Oceanside

Karma: 6

Senior Member

Frank,

I really need this now and am prepared to pay for it. Please send me a PM or email to curtis - at - backupcentral.com. TIA

Report message to a moderator

Re: Mailing list integration question [message #168815 is a reply to message #168785]

Sat, 27 July 2013 00:40

cpreston

Messages: 160
Registered: July 2012
Location: Oceanside

Karma: 6

Senior Member

Frank,

I know you have a real job. Wink

Wink

So unless I hear a "yo!" real soon, I'm going to get my developers started on this. Here's what I've come up with for a plan. It would be great if you could take a look and see what you think.

I plan to edit the core FUDforum code so that it includes "In-Reply-To" and "References" headers when it sends a reply to an email.

As I understand it, to do this, I need to do the following:
1. Get the Message-ID of each message
- Grab the Message-ID from any incoming emails
- Create and/or get the Message-ID for any outgoing emails

2. Store the Message-ID of each post, associated with the post.

3. Insert "In-Reply-To" and "References" headers into any outgoing emails:

(In the following, "Previous-Message-ID" is the Message-ID of message currently being replied TO, AKA the Message-ID of the previous post in the thread -- not the message-ID of the outbound message.)

<code>In-Reply-To: <Previous-Message-ID>
References: <First-message-ID-of-current-thread> <next-message-ID-in-thread> <next-message-ID-in-thread> <etc> <Previous-message-ID></code>

(If there is only one previous message in thread, there is only one message-ID in the References line. If there are more than ten previous message IDs, you are supposed to drop the second one until you have ten or fewer. Always start with the first-message-ID and end with the Previous-Message-ID.)

I will submit diffs from the core code. It seems like adding a field to the posts table would be the most likely place to store the Message-ID for each post, but that does mean adding a field to a core table. What do you think about that, Frank?

Just in case I read it wrong, the details of how to do In-reply-to and References headers is based on the info in this web page:
http://cr.yp.to/immhf/thread.html

Report message to a moderator

Re: Mailing list integration question [message #168823 is a reply to message #168815]

Sat, 27 July 2013 13:50

naudefj

Messages: 3772
Registered: December 2004

Karma: 28

Senior Member
Administrator
Core Developer

Some comments:

FUDforum doesn't generate Message-ID's, but will grab and store it if the message is returned to it from the list (fud30_msg.mlist_msg_id column). So, 1 & 2 are already implemented. The obvious problem with this approach is that users can reply to a message on the forum before its Message-ID is captured. Current solution is to sync frequently (every few seconds).

Function mail_list_post() in mlist_post.inc will add 'In-Reply-To:' headers to mails. Next step would be to get someone to provide code to add the 'References:' header as well.

Best regards.
Frank

Report message to a moderator

Re: Mailing list integration question [message #168828 is a reply to message #168823]

Sun, 28 July 2013 03:38

cpreston

Messages: 160
Registered: July 2012
Location: Oceanside

Karma: 6

Senior Member

You're using PHPMailer, right? It is generating the Message-ID, but it's making a random one because you're not specifying it as an environment variable MessageID. That is according to this:
http://www.tig12.net/downloads/apidocs/wp/wp-includes/PHPMailer.class.html# det_fields_MessageID

What message-ID are you grabbing if you're getting it from the one the list returns? Are you grabbing the message ID of the incoming email or the Message-ID of the email you sent that it is replying to, which would be the one you should specify in the "In-Reply-To" field. Whichever one you're grabbing, we end up with not enough info for a complete "References" tag. If you're grabbing the Message-ID of incoming message that is a reply to a forum post, you have the latest message-ID but not the first one, which is what you're supposed to put in the References tag. If you're getting the previous one, then you don't have the current one....

It seems more proper to generate the Message-ID for any outgoing messages, and then pass them on to PHPMailer (pretty easy to generate one http://www.jwz.org/doc/mid.html), then store it with the message in the mlist_msg_id column. Then you do what you're already doing and grab any Message-IDs of any incoming messages and store them in the same column. Then we've got Message-IDs for every message.

You're already doing the right thing with mail_list_post with the "In-Reply-To:" tag. Since we would have Message-IDs for all messages, then it's just a matter of a database query to get the first, last, and other Message-IDs so we can generate the References header.

What do you think?

Report message to a moderator

Re: Mailing list integration question [message #168830 is a reply to message #168828]

Sun, 28 July 2013 06:38

cpreston

Messages: 160
Registered: July 2012
Location: Oceanside

Karma: 6

Senior Member

I've been poking around in the code, and I have a few questions.

Lines 45-49 of mlist_post.inc seems to be creating our own Message ID, updating mlist_msg_id w/the value, and making it part of the email before it is sent. So isn't the code already doing what I'm suggesting regarding Message IDs?

If my understanding is correct, all messages that have been sent out via email should have a messageid in their database entry. That means all I really need to do to accomplish what I want is the proper database query and formatting of a References header. It would seem that all this could be put right after line 41 in moist_post.inc.

This one could be my misunderstanding of PHP, but doesn't line 12 of moist_post.inc set the reply_to to null no matter what is passed to the function? Lines 40-42 would then not do anything because reply-to will always be null? I see similar behavior in the send_email function in remail.php, where $header is set to '' no matter what headers are sent to it. This would suggest that even if the Reply-To header was set, it wouldn't be used.

it appears that the send_email function (with the class fur_smtp) in remail.php sends the actual email, so you're not using PHPMailer. is that correct?

Report message to a moderator

Re: Mailing list integration question [message #168839 is a reply to message #168830]

Sun, 28 July 2013 16:20

naudefj

Messages: 3772
Registered: December 2004

Karma: 28

Senior Member
Administrator
Core Developer

cpreston wrote on Sun, 28 July 2013 08:38

Lines 45-49 of mlist_post.inc seems to be creating our own Message ID, updating mlist_msg_id w/the value, and making it part of the email before it is sent. So isn't the code already doing what I'm suggesting regarding Message IDs?

Yes, you're right! Just remember that the mailing list may assign a different Message ID, and we will grab and store the list's ID as soon as we spot the message on the list.

cpreston wrote on Sun, 28 July 2013 08:38

This one could be my misunderstanding of PHP, but doesn't line 12 of moist_post.inc set the reply_to to null no matter what is passed to the function? Lines 40-42 would then not do anything because reply-to will always be null?

It sets the default value to NULL. So, if no value is supplied, we will start with nothing.

cpreston wrote on Sun, 28 July 2013 08:38

...it appears that the send_email function (with the class fur_smtp) in remail.php sends the actual email, so you're not using PHPMailer. is that correct?

Correct, we're most definitely not using PHPMailer. Should we?

Report message to a moderator

Re: Mailing list integration question [message #168841 is a reply to message #168839]

Sun, 28 July 2013 16:57

cpreston

Messages: 160
Registered: July 2012
Location: Oceanside

Karma: 6

Senior Member

Quote:

Yes, you're right! Just remember that the mailing list may assign a different Message ID, and we will grab and store the list's ID as soon as we spot the message on the list.

That's not how Message-ID is SUPPOSED to work. Each message should have a message-id and should not be changed. Some mailing lists (like Mailman) will give an email a message-id if it doesn't have one, because it should have one for proper threading. I've verified that mailman and listserv will only generate a message-id if the incoming message doesn't have one, or if IT is the one sending the message (e.g. a reminder or something).

Quote:

we're most definitely not using PHPMailer. Should we?

Everybody else sending email in PHP sure does. Not sure how big of a deal it is to change that, though, or what benefit you would get by changing it.

[Updated on: Sun, 28 July 2013 19:37]

Report message to a moderator

Re: Mailing list integration question [message #168843 is a reply to message #168841]

Sun, 28 July 2013 20:32

cpreston

Messages: 160
Registered: July 2012
Location: Oceanside

Karma: 6

Senior Member

OK, I tested the functionality we're talking about. I viewed the Message-IDs and the In-Reply-To references in Outlook by looking at the email headers. I viewed what FUDforum was placing in the database by monitoring the mlist_msg_id column for the associated messages.

I verified:
1. FUDforum creates a Message-ID for original forum messages that are sent to a mailing list and it stores that info in the mlist_msg_id column
2. Neither FUDforum nor Mailman changes a Message-ID of an incoming message if it has one (which it should if it was sent by any decent MTA), but Mailman will give it one if it doesn't have one
3. FUDforum does properly grab the Message-ID of the incoming message and stores that in the mlist_msg_id column
4. Most importantly, FUDforum uses the values in that column to properly create the In-Reply-To header. (I did several posts back and forth from email and the forum, and each time FUDforum assigned the previous Message-ID as the In-Reply-To header.

So really all that's left is to create a database query that would get all the Message-IDs for a given thread, then create a References line based on that. Awesome!

Report message to a moderator

Re: Mailing list integration question [message #168844 is a reply to message #168843]

Sun, 28 July 2013 21:08

cpreston

Messages: 160
Registered: July 2012
Location: Oceanside

Karma: 6

Senior Member

I'm guessing about line 48 in mlist_post.inc we do something like this. I'm reaching here since I barely understand the syntax, but I thought I'd give it a shot. It can probably done with one query and a join, right? I don't get joins. Wink

Wink

I stole the db_saq string from another .inc file. I'm guessing it's what i need here. Wink

Wink

#Get the thread_id this message is in
db_saq('SELECT m.thread_id FROM $GLOBALS['DBHOST_TBL_PREFIX'] .'msg WHERE m.msg_id='. $msg_id .' AND m.subject='$subject');

#Get all the msg_ids from this thread, ordered by time.
db_saq('SELECT m.msg_id, m.post_stamp FROM $GLOBALS['DBHOST_TBL_PREFIX'] .'msg WHERE m.thread_id='. $thread_id .' ORDER BY m.post_stamp');

This should go into an array that we can then grab them from. Does that sound about right?

Report message to a moderator

Re: Mailing list integration question [message #168868 is a reply to message #168844]

Wed, 31 July 2013 17:18

cpreston

Messages: 160
Registered: July 2012
Location: Oceanside

Karma: 6

Senior Member

I have completed the code to make this happen, but am going to post it in the hacks forum. Feel free to put it in 3.0.5! Once we figured out how to do it, it didn't take that many lines of code.

Report message to a moderator

Previous Topic:	Not getting any emails from FUDforum
Next Topic:	Post shows up above Sticky and Announcement Threads

Goto Forum:

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

]

Current Time: Wed Dec 18 01:49:55 GMT 2024

Total time taken to generate the page: 0.06710 seconds