email2trac ver 0.60 - login name not set and email bounces

["Pusateri, Matthew" <Matthew.Pusateri@…>]

Hi,

I'm using the latest version of email2trac 0.60 released 1/19/2009 on
trac 0.11 with ubuntu 8.10, and sendmail.

I'm seeing two issues and do not know if they are a bug or not. I can
post to the trac-user mailing list if need be and or open bugs if
necessary.

1. I have ignore_trac_user_settings set to 0 (zero) in my
email2trac.conf file(Defaults section), but the reporter never gets
set
to the login name, it will only set it to an email address

2. We send email to our exchange server which forwards to our ticket
system, the exchange server changes the mail header so the from email
address looks like firstname.lastname@servername.domain.com instead of
firstname.lastname@domain.com   The ticket gets created fine, and
users
get notified of the new ticket, but we also get a new ticket created
from a email bounc as first@servername.domainname.com being
undeliverable.  Not sure where it is stripping the lastname from?



Any ideas on these?


Thanks,

Matt Pusateri
also on trac IRC as Jeeplover or on trac-users as flatfender@gmail.com

untitled-part.html​

[Pusateri, Matthew]

Added by email2trac

[Pusateri, Matthew]

This message has 1 attachment(s)

[matthew.pusateri@…]

update, I captured what is being sent to the email2trac script by exchange, and it's not rewriting the headers. I think the problem is that the reporter address is set to "Pusateri, Matthew" <Matthew.Pusateri@…>, and when our mail gateway mis-interprets that address or something I'm not sure. Maybe it's a configuration issue. as a side note this happened in 0.30 as well.

[bas]

When you want to set tickets to the login name, the user must set its email address in the preferences tab. Then email2trac will find the address and translate it to the login name.

[Pusateri, Matthew]

The user has already set their email in the preferences tab. Verified
by
looking in the sqlitedb.

-----Original Message-----
From: email2trac [mailto:email2trac@sara.nl]
Sent: Wednesday, January 21, 2009 4:57 PM
Cc: basv@sara.nl
Subject: Re: [email2trac] #118: email2trac ver 0.60 - login name not
set
and email bounces

#118: email2trac ver 0.60 - login name not set and email bounces
---------------------------------------------------------------+------
--
---------------------------------------------------------------+----
Reporter:  "Pusateri, Matthew" <Matthew.Pusateri@tekelec.com>  |
Owner:  bas
    Type:  setup                                               |
Status:  assigned
Priority:  major                                               |
Component:  email2trac
 Version:  0.6                                                 |
Resolution:
Keywords:                                                      |
---------------------------------------------------------------+------
--
---------------------------------------------------------------+----
Changes (by bas):

  * status:  new => assigned
  * type:  defect => setup


Comment:

 When you want to set tickets to the login name, the user must set its
email address in the preferences tab. Then email2trac will find the
address and translate it to the login name.

--
Ticket URL:
<https://subtrac.sara.nl/oss/email2trac/ticket/118#comment:3>
email2trac <https://subtrac.sara.nl/oss/email2trac>
Documentation server for email2trac

[matthew.pusateri@…]

okay here is more info:

1. User email is set upon login via script that get's email address from AD
2. if login name is used as the reporter via web ui, user get's all the expected emails
3. if user email address format: firstname.lastname@… is used via web ui, user get's all expected emails.
4. if user email address format "FirstName? LastName? <firstname.lastname@…> is used via the web ui, user get's all expected emails.
5. if user sends email and it get's processed via email2trac, then login name or sid does not get set to the reporter even if the user has a valid email address in session_attribute table. But user get's notified of the new ticket, and we get a bounce notification that creates a new ticket with the bounce. I'll talk with IT about what our email gateway may be doing as the email address that is getting mangled as listed above contains a servername in the email address not just the domain name

So it seems that login name lookup is not being set and possibly the format of the email address is different from how trac sets it. Note if a user enters a ticket via the web ui, no bounce message is generated.

[matthew.pusateri@…]

Replying to matthew.pusateri@tekelec.com:

okay here is more info:

1. User email is set upon login via script that get's email address from AD
2. if login name is used as the reporter via web ui, user get's all the expected emails
3. if user email address format: firstname.lastname@… is used via web ui, user get's all expected emails.
4. if user email address format "FirstName? LastName? <firstname.lastname@…> is used via the web ui, user get's all expected emails.
5. if user sends email and it get's processed via email2trac, then login name or sid does not get set to the reporter even if the user has a valid email address in session_attribute table. But user get's notified of the new ticket, and we get a bounce notification that creates a new ticket with the bounce. I'll talk with IT about what our email gateway may be doing as the email address that is getting mangled as listed above contains a servername in the email address not just the domain name

So it seems that login name lookup is not being set and possibly the format of the email address is different from how trac sets it. Note if a user enters a ticket via the web ui, no bounce message is generated.

Looking in the email logs, it seems we're sending the ticket notification method to two recipients, the first recipient get's set to firstname@… and the second to firstname.lastname@… as seen in this snipit of the mail log `Jan 22 12:24:21 nsglab7001204 sm-mta[15080]: n0MHOJ6d015078: to=<Matthew@…>,<Matthew.Pusateri@…>, delay=00:00:01, xdelay=00:00:01, mailer=relay, pri=151695, rel ay=mailhost [198.89.40.60], dsn=2.0.0, stat=Sent (n0MHOKV11579 Message accepted for delivery`

From tracing through the script it seems that self.author gets set to lastname, firstname and the ticket fields get updated. Self.Author is the only place I see the lastname, firstname being set, now, what is causing it to generate two recipients, I'm not sure.

Also it seems odd the in set_reply_fields definition that the "if self.notification:" statement line is indented under "if self.REPLY_ALL:" statement but I would think that is just me not understanding enough python yet.

[matthew.pusateri@…]

So I eliminated the email being sent to multiple addresses problem by enabling REPLY_ALL to true in email2track.conf, but I really don't understand why that worked.

I also figured out why the email to username/sid lookup was failing. In our environment the email gets auto populated from Active Directory and is stored as all lower case in session_attribute, but the emails that arrive from Exchange list the email_from as "Firstname.Lastname@" so the lookup failed since the lookup is case sensitive. I could see that failing if the email address was not auto populated since you can't be sure the user will fill in their preferences the exact same way the email client/server will send the email_from address

[bas]

Thanks for the info _{can also attach to this ticket an email example message. If you set debug it will save the original email in /tmp or /var/tmp}

[bas]

If i read the info correctly we have to case insensitive match, can you test this little patch without the replay_all option set:

• get_sender_info

        # Is this a registered user, use email address as search key:
        # result: 
        #   u : login name
        #   n : Name that the user has set in the settings tab
        #   e : email address that the user has set in the settings tab 
        #
        tmp_email_addrs = self.email_addr.lower()
        users = [ (u,n,e) for (u, n, e) in self.env.get_known_users(self.db)
                if e == tmp_email_addr ]

thanks

[matthew.pusateri@…]

email message used for testing, captured from tmp debug

[matthew.pusateri@…]

well I actually just did this this which worked, but would obviously fail if the user entered everything in all caps. I see 3 methods which are probably most common, all lowercase, all upper case, or mixed case. Maybe what should be done is say convert e to lower and convert email_addr to lower and then do the compare so that you would always be comparing the same case.

 users = [ (u,n,e) for (u, n, e) in self.env.get_known_users(self.db)
                                if e == self.email_addr.lower() ]

Attached is the email I've been using for testing.

I see this as two problems still. Problem 1. case insensitive match. Problme 2. Why were two recipients being added to the notification email when reply_all was false? Can you explain how reply_all is supposed to work as right now it seems reversed. I'm also not understanding this section of code. Why are you doing addr.remove if reply_all is true and notification is on?

  def set_reply_fields(self, ticket, message):
                """
                Set all the right fields for a new ticket
                """
                ticket['reporter'] = self.email_from

                # Put all CC-addresses in ticket CC field
                #
                if self.REPLY_ALL:
                        #tos = message.get_all('to', [])
                        ccs = message.get_all('cc', [])

                        addrs = email.Utils.getaddresses(ccs)
                        if not addrs:
                                return

                        # Remove reporter email address if notification is
                        # on
                        #
                        if self.notification:
                                try:
                                        addrs.remove((self.author, self.email_addr))
                                        print "set_reply_fields_author %s" %(self.author)
                                except ValueError, detail:
                                        pass

                        for name,mail in addrs:
                                try:
                                        mail_list = '%s, %s' %(mail_list, mail)
                                        print "mail_list: %s" %(mail_list)
                                except UnboundLocalError, detail:
                                        mail_list = mail
                                        print "mail: %s" %(mail)

                        if mail_list:
                                ticket['cc'] = self.email_to_unicode(mail_list)

[bas]

The first is no problem, we just translate all the mail address to lower cases and if this matches then a login name will be filled in. So you do not have to worry about upper/lower and mixed.

Second the reply_all function is use to capture all the CC-addresses. If the reporter is also in CC-list we removed it, else the user will get two emails. I do not how the replay_all setting solves your problem, because you do not have any cc address. So it will skip the rest of the function:

if not addr:
 return

So it does nothing!

In the whole email2trac just before notify i get the right email address

notify:  "Pusateri, Matthew" <Matthew.Pusateri@tekelec.com>

So or its a bug in the notify function of trac? Which version do you use and which plugins are enabled?

[matthew.pusateri@…]

okay thanks for clarifying the set_reply fields function, and Thanks for your responsiveness as well.

We're using trac 0.11, mostly b/c my boss won't use a package that isn't in a debian/ubuntu repository.

Plugins are: Batch Modify 0.20, BlackMagic? Ticket Tweaks 0.1, NavAdd? 0.1, Notebox 0.1.dev, Private wiki's 1.0.0, Ticket box 0.11, Ticket Validator 0.1, Trac Add Headers 0.1, Trac Date Field 1.0.1, Trac Nav 3.92, Trac News Flash 1.0,1, Trac Private tickets 2.0.2, Trac User Manager 0.3, Trac wysiwyg 0.2

[matthew.pusateri@…]

okay so we did some more testing and the reply_all switch has no effect, unfortunately we were testing with two different accounts and that made it look like it had and effect when reply_all was true.

So the question still becomes why are we getting two recipients?

[bas]

oke that is right the replay_all option does nothing/ I still think it is a bug in the trac notification function that can not handle this email address fomrat or one of your plugins is causing this problem. Can you test this patch, it will only use the email address and not the name and email adress:

set_reply_all(...)

 ticket['reporter'] = self.email_addr

instead of:

ticket['reporter'] = self.email_from
}}

[matthew.pusateri@…]

Replying to bas:

oke that is right the replay_all option does nothing/ I still think it is a bug in the trac notification function that can not handle this email address fomrat or one of your plugins is causing this problem. Can you test this patch, it will only use the email address and not the name and email adress:

set_reply_all(...)

 ticket['reporter'] = self.email_addr

instead of:

ticket['reporter'] = self.email_from
}}

That patch had no effect, but I think I've finally tracked down how to duplicate this problem and under what circumstances it's an issue.

Exchange by default and I will confirm sets the From: line in the headers to

"Lastname, Firstname" <Firstname.Lastname@domain.com"

What ends up happening is in get_sender_info self.author gets set to "Lastname, Firstname" self.email_addr get set to <Firstname.Lastname@…>. Also self.author gets inserted into ticket_changes as the author and if "always_notify_updater = true" is set in the [Notification] section of trac.ini, then a second recipient gets added to the notification email. The first recipient is the reporter, the second recipient is the author in ticket_changes. Now trac core has an issue with having "Lastname, Firstname" as the author and incorrectly parses it(not sure how yet) and strips off the "Lastname," part so you end up getting a firstname only inserted into the email, with the default domain appended, you end up getting a recipient of "firstname@…" which in my case was/is causing the bounce.

So I guess I need to file a bug with trac, that is notify_updater is true that they should parse the ticket_changes author differently.

But from the email2trac side, I don't see why self.author really should end up anything other than either the known username/sid, or a correct email address such as what email_from_addr gets set to. At least in my opinion the person who sent the email is the reporter if the ticket is new and the author if the ticket isn't.

As a side note, it looks like if you enter a ticket from the web UI, the author in ticket_changes tables gets set to the logged in persons name field of session_attribute table, but I haven't really looked into that. Which depending on the value could also cause weird recipient names that are not valid email addresses.

Thanks,

Matt P.

[bas]

self.author is always set to:

Pusateri, Matthew

maybe we must set this to email_address and everything would work as expected. So can you test this in get_sender_info:

self.author, self.email_addr  = email.Utils.parseaddr(self.email_from)
self.author = self.email_addr

[matthew.pusateri@…]

Replying to bas:

That seems to work, I wonder if we should also set self.author to a sid like what is done with self.email_from when we match on email to sid lookup? To be it would make sense to be consistent so you don't have reporter set to the sid and author set to the email address.

Something like:

  users = [ (u,n,e) for (u, n, e) in self.env.get_known_users(self.db)
                                if e == self.email_addr.lower() ]
                    if len(users) == 1:
                       self.email_from = users[0][0]
                       self.author = self.email_from

[bas]

Thanks this is already in the trunk version of email2trac, see the roadmap

[Pusateri, Matthew]

Thanks for all your help and your quick responses.

-----Original Message-----
From: email2trac [mailto:email2trac@sara.nl]
Sent: Tuesday, January 27, 2009 11:13 AM
Cc: basv@sara.nl
Subject: Re: [email2trac] #118: email2trac ver 0.60 - login name not
set
and email bounces

#118: email2trac ver 0.60 - login name not set and email bounces
-----------------------------------------------------------------+----
--
-----------------------------------------------------------------+----
  Reporter:  "Pusateri, Matthew" <Matthew.Pusateri@tekelec.com>  |
Owner:  bas
      Type:  defect                                              |
Status:  assigned
  Priority:  major                                               |
Milestone:  0.70
 Component:  email2trac                                          |
Version:  trunk
Resolution:                                                      |
Keywords:
-----------------------------------------------------------------+----
--
-----------------------------------------------------------------+----

Comment(by bas):

 Thanks this is already in the trunk version of email2trac, see the
roadmap

--
Ticket URL:
<https://subtrac.sara.nl/oss/email2trac/ticket/118#comment:20>
email2trac <https://subtrac.sara.nl/oss/email2trac>
Documentation server for email2trac

[bas]

fixed in version 0.7, close ticket so we van release a new version