Modify

Opened 9 years ago

Closed 9 years ago

Last modified 9 years ago

#267 closed defect (fixed)

Deadlock with "UnicodeEncodeError: 'ascii' codec can't encode character u'\xfc' in position ##: ordinal not in range(128)"

Reported by: hju@… Owned by: bas
Priority: critical Milestone: New email2trac release 2.4.0
Component: email2trac Version: 2.4.0
Keywords: Cc:

Description

This error is back! It was reported and solved in Ticket #184 for attachments.

Now its back for the summary... as I think. There are no attachments at the mail. (disregarding the html-content and a .gif)

The summary of the mail is called:

Spamfilter auf Kunden-Emails prüfen WG: Bewerbungen Ausbildungsplatz

The error is from the trac core (a known Problem:

2011-08-17 16:04:06 msg 122/122 (27140 bytes) msgid 1220533924/INBOX/1903 from <####@jochenkuhl.de>
2011-08-17 16:06:06 Delivery error (command run_email2trac 26957 error (0, Traceback (most recent call last):
  File "/usr/lib/python2.5/logging/handlers.py", line 667, in emit
    self.socket.send(msg)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xfc' in position 63: ordinal not in range(128)))

The error interrupts email2trac. The mail is converted to a ticket and created but is never marked as processed. It`s a deadlock! You receive a Ticket every time the cron-job reads the inbox until you manually delete the Mail!

I'm using the trunk-version 556

Attachments (0)

Change History (24)

comment:1 Changed 9 years ago by hju@…

Additional info: I can reproduce the deadlock with the summary!

In debug-mode, these are the last lines:

Aug 17 23:02:46 ks35958 email2trac ticket: function notify()
Aug 17 23:02:46 ks35958 email2trac ticket: #011 Notify reporter set
Aug 17 23:02:46 ks35958 email2trac ticket: #011 Using Email2TracNotification function AlwaysNotifyReporter
Aug 17 23:02:46 ks35958 email2trac ticket: function email_header_acl: notify_reporter_black_list

looks like there is the call causing the problem

comment:2 Changed 9 years ago by hju@…

sorry, this led in the wrong direction. The log is identical if there is no problem with the mail.

The mails to inform reporter and owner were sent in both cases.

the real problem seems to occure later.

I posted the wrong lines of the logfile... here are the correct lines.

2011-08-17 16:06:06 Delivery error (command run_email2trac 26957 error (0, Traceback (most recent call last):
  File "/usr/lib/python2.5/logging/handlers.py", line 667, in emit
    self.socket.send(msg)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xfc' in position 63: ordinal not in range(128)))
2011-08-17 16:06:06 msg 122/122 (27140 bytes) msgid 1220533924/INBOX/1903 from <####@jochenkuhl.de>

I give up for now.

comment:3 Changed 9 years ago by anonymous

Last info: it´s no new problem... I get find same issue with email2trac 1.4.6.

Aug 18 00:28:15 ks35958 email2trac: Traceback (most recent call last):
Aug 18 00:28:15 ks35958 email2trac:   File "/usr/local/bin/email2trac", line 2184, in <module>#012    tktparser.parse(sys.stdin)
Aug 18 00:28:15 ks35958 email2trac:   File "/usr/local/bin/email2trac", line 1475, in parse#012    print "TD:", subject
Aug 18 00:28:15 ks35958 email2trac: UnicodeEncodeError: 'ascii' codec can't encode character u'\xfc' in position 42: ordinal not in range(128)

comment:4 Changed 9 years ago by bas

  • Status changed from new to assigned
  • Version changed from 2.1.0 to 2.4.0

Do you use a patched email2trac version?

Aug 18 00:28:15 ks35958 email2trac:   File "/usr/local/bin/email2trac", line 1475, in parse#012    print "TD:", subject

I can not find this line. I you can not print a subject line with unicode chars with this statement. I just tested the trunk version with your subject line:

Spamfilter auf Kunden-Emails prüfen WG: Bewerbungen Ausbildungsplatz

and did not encounter any problems.

comment:5 Changed 9 years ago by bas

  • Milestone set to New email2trac release 2.4.0

comment:6 Changed 9 years ago by hju@…

yes, the 1.4.6 is a patched version. sorry for confusion with the line numbers

main

		if int(settings['debug']) > 0:
			print 'Loading environment', settings['project']

		env = Environment(settings['project'], create=0)

		tktparser = TicketEmailParser(env, settings, float(version))
2184		tktparser.parse(sys.stdin)

def parse

		if not m['Subject']:
			subject  = 'No Subject'
		else:
			subject  = self.email_to_unicode(m['Subject'])

		if self.DEBUG:
1475			 print "TD:", subject

Strange that you did not encounter any problem with the trunk version... I attach the conf-settings.

[DEFAULT]
project: /var/trac/ticket
debug: 0
umask: 022
spam_level: 5
reply_all : 0
mailto_link: 0
umask: 022
email_header: 0
trac_version: 0.11
enable_syslog : 1
alternate_notify_template :
alternate_notify_template_update :
drop_spam : 0
verbatim_format: 0
strip_signature: 1
email_quote: >
strip_quotes: 1
ignore_trac_user_settings: 0
ticket_update = 1
black_list: MAILER-DAEMON@,noreply@jochenkuhl.de
notify_reporter_black_list: support@kunde.de
drop_alternative_html_version: 0
python_egg_cache: /tmp
parentdir: /var/trac/Projekte
notify_reporter: 1

[ticket]
project: /var/trac/Projekte/ticket
alternate_notify_template: jku_notify_new.txt
alternate_notify_template_update: jku_notify_new.txt.update
workflow_waiting external: reaktivate

comment:7 Changed 9 years ago by bas

Oke i am using 2.4.0 pre-release and the statements are like this:

        if not m['Subject']:
            subject  = 'No Subject'
        else:
            subject  = self.email_to_unicode(m['Subject'])

        self.logger.info('subject: %s' %subject)

So the subject is unicoded ;-) and in your version not when debug is set.

comment:8 Changed 9 years ago by bas

sorry. The response is to quick!. I see that subject is also unicoded in your version.

comment:9 Changed 9 years ago by anonymous

Yes! I use the [556] Revision which ist not to far away from you. Sorry, but good guess.

And: the error occurs also with debug: 0

comment:10 Changed 9 years ago by bas

Huh, you also get an error when debug is set to 0! This must be another error. I just tested with your subject line and did not encounter any problems.

comment:11 Changed 9 years ago by anonymous

Thats strange. The initial error-post is from our getmail-log. debug: 0

Do you have an email-adress, where I can send the email I've tested with to? (Don't want to use email2trac at sara.nl for this)

comment:12 Changed 9 years ago by hju@…

Does your test-environment use python2.5? Our trac is 11.6 with python2.5 While the error is located in python2.5/logging/handlers.py, perhaps the problem is fixed in a later python-release?

comment:13 Changed 9 years ago by bas

Oke you can set t to basv@… and i am using 0.12 and python2.6. So it can be a python 2.5 problem. So the error is n the logging environment. Can you replace the self.logger.debug by:

  • self.print_unicode(s<strng>)

See if this wlll catch the error.

comment:14 Changed 9 years ago by anonymous

I have tracked down the error to this line in def parse

        self.logger.info('subject: %s' %subject)

seems that the prior email_to_unicode(mSubject?) does not work as expected.

comment:15 Changed 9 years ago by anonymous

Thanks. So you say the error is in email_to_unicode() function? or was the error in your patched email2trac version?

comment:16 Changed 9 years ago by anonymous

it seems so that email_to_unicode() is part of the problem (or not solving the Problem). There is no patch in this part of email2trac.

As I understand, the subject as result from email_to_unicode() is logged as an info. This log-attemt causese the "UnicodeEncodeError?" in handler.py, but execution of email2trac is continued.

Don't know why there is no problem in your installation. Perhaps it has something to do with the python version. As a quick fix I've disabled this line of code.

While there are no other error messages from email_to_unicode(), the code seems to take one of the following choices

        for text,format in results:
            if format:
                try:
                    temp = unicode(text, format)
                ...        
            else:
                temp = string.strip(text)
                temp = unicode(text, 'iso-8859-15')

both should work...

comment:17 Changed 9 years ago by bas

Just be clear. Do you run patched version of email2trac of a vanilla version?

Can you test the following code. replace:

  • self.logger.info("subject: %s' %subject)

with:

  • self.print_unicode("subject: %s' %subject)

And add some print statements to email_to_unicode:

  • after the for loop: print repr(text), format
  • and before s is returned, print repr(s)
Last edited 9 years ago by bas (previous) (diff)

comment:18 Changed 9 years ago by hju@…

Hi Bas,

the replace does not change anything. Same error in handler.py

And to be clear ;-): The problem occures with vanilla Version of email2trac [556].

comment:19 Changed 9 years ago by bas

That is odd because print_unicode should catch this error:

    def print_unicode(self,s):
        """
        This function prints unicode strings if possible else it will quote it
        """
        try:
            self.logger.debug(s)
        except UnicodeEncodeError, detail:
            self.logger.debug(util.text.unicode_quote(s))

Thanks for the clarification! :-D

comment:20 Changed 9 years ago by hju@…

I've inserted a quick fix to eliminate the responsible character.

        if not m['Subject']:
            subject  = 'No Subject'
        else:
            subject  = self.email_to_unicode(m['Subject'])

        subject = subject.replace(u'\xfc', 'ue')
        subject = subject.replace(u'\xdc', 'Ue')
        self.logger.info('subject: %s' %subject)

It helps to keep logging.info going but slightly changes the summary. Otherwise I can only delete this logging line...

comment:21 Changed 9 years ago by bas

you can better use repr(subject), else we have to catch all the execeptions. It must be a python 2.5 bug?

comment:22 Changed 9 years ago by hju@…

  • Priority changed from blocker to critical

I'll leave it as it is for now. Thems to be a well known bug in Python, which is still existing in python3.#

If you are looking at this ticket or search with UnicodeEncodeError '\xfc' at http://bugs.python.org you'll find several issues with this error in lots of python modules...

comment:23 Changed 9 years ago by bas

  • Resolution set to fixed
  • Status changed from assigned to closed

(In [563]) fixed unicode error when printing subject line. Python logging module problem, closes #267

comment:24 Changed 9 years ago by bas

(In [564]) Only use repr() in subject logging, still convert subject to unicode, see #267

Add Comment

Modify Ticket

Change Properties
Action
as closed The owner will remain bas.
The resolution will be deleted. Next status will be 'reopened'.
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.