Fix for problem with long attachment's filenames on native language

[anonymous]

Sympthoms: when user attached into email for trac file with long name (more then 25+ characters) or file with filename on native laguage (like russian) - email2trac can't store that attachment from message and do nothing. No notify about fail for user, or create new ticket or modify exist ticket WITHOUT attachment from message.

Solution: Required external python module trans. URL: ​http://pypi.python.org/pypi/trans "At the moment, Czech, Greek, Latvian, Polish, Turkish, Russian, Ukrainian alphabets are supported (it covers 99% of needs)."

Path file in attach. Also included some fix for reopen ticket via email.

[slevin@…]

patch for problem with attachment on native language (like Russian)

[bas]

Thanks for the patch. Can attach a diff -u so i can apply it easily. I see that trans is not a standard pythin module. So i will make an option for it! We also have really long filenames but never experienced any problems with it. Is it only a problem with the languages you mentioned?

There is no error message in a log (syslog, trac.log).

[slevin@…]

I can talk only about problem with my native language - Russian,a after call util.text.unicode function we're get name where each russian character equal FOR (4) unicode characters, example: "Имя" ("Name" on eng.) have 3 characters in it, after util.text.unicdoe I reveived the name with 12 chacters in it! For my organisation offten the situations when user put into message file with name on russian nearly 40-50 characters lenght, after unicoding I received teh filename lenght nearly 160-300 characters, o'course without path and name extention! And the system can't store that file. Grep from error log (by procmail) I'm attached to that message, also attached patch file in universal format.

[slevin@…]

grep from error log (problem with convert russian filename

[slevin@…]

patch in universal format

[bas]

Stevin the fix for re-open tickets via email is for workflow option. In the past we has another option for email2trac.conf:

workflow: <something>

So when i ticket was closed and a update was received with workflow set. It would trigger this workflow. It is still here for backward compatibility. I will slightly modify the patch. It is a bug, we have set workflow_reopen in email2trac.conf. So didn't catch this one.

I am thinking about a good option for using the trans package as it is not a standard python module.

[bas]

Does trac not provide some function for this? Translate filenames.

[anonymous]

Last new about trans module, atm my user testing patch and I have not good news, we're have problem with attachment only with english letters in filename. I think about fix it.

[bas]

Wahat worries me is that the output you send about this problem:

Subject: =?KOI8-R?Q?=EE=CF=D7=CF=C5_=FA=C5=D2=CB=C1=CC=CF_=C4=CC=D1_?=
4	  Folder: /usr/bin/run_email2trac --project=admin                        359820
5	Traceback (most recent call last):
6	  File "/usr/lib64/python2.5/logging/handlers.py", line 667, in emit
7	    self.socket.send(msg)
8	UnicodeEncodeError: 'ascii' codec can't encode characters in position 46-52: ordinal not in range(128)
9	Traceback (most recent call last

I have just patches email_to_unicode, maybe there is a charset missing?

[bas]

fx = open(part_file, 'wb')
201	IOError: [Errno 36] File name too long: '/tmp/%D0%94%D0%BE%D0%B3%D0%BE%D0%B2%D0%BE%D1%80%20%D0%BE%20%D0%BF%D1%80%D0%B5%D0%B4%D0%BE%D1%81%D1%82%D0%B0%D0%B2%D0%BB%D0%B5%D0%BD%D0%B8%D0%B8%20%D1%83%D1%81%D0%BB%D1%83%D0%B3%20%D1%82%D0%B5%D0%BB%D0%B5%D0%BA%D0%BE%D0%BC%D0%BC%D1%83%D0%BD%D0%B8%D0%BA%D0%B0%D1%86%D0%B8%D0%B9.doc'

We can catch this error and just generate attachments names that are not longer then a configurable lenght

[bas]

(In [526]) Fix re-opening for closed tickets for trac 0.12 and higher. Only triggered when no workflow was defined in email2trac.conf, see #247

[bas]

(In [527]) workflow must be set to None, see #247

[anonymous]

Bas, I'm really glad to help you, but I don't know python language :) only python handbook on my table :)

[bas]

No problem. I was just express my thoughts. Maybe we can save the attachemnts with random names and use the filename as link. It has some impact on the code,

[anonymous]

U mean cutting attachment from message into file with numeric_name.real_ext and into tickets/blog/etc body we're put real name of attachment and do hyperlink to real file in file system... I guess that can be solve problem with name forever. Good idea.

[bas]

That is the general idea, just use number for example and make the link the real name

[bas]

I just looked at the source of trac and what i try to accomplish is not supported :-(. Must search for another solution.

[bas]

Can you test the trunk, with debug set to 0 and see if trac will complain or fail in the attachement functions.

I forgot to ask which OS you use? and maybe i can obtain the raw email message to test some things?

[anonymous]

a long silence from my side again :( Atm our trac worked on the server with OS gentoo linux. U can take the raw email from attach to this ticket.

About test the trunk: ok - tomorrow I'll tets the trunk on mirror server of our trac in production.

[slevin@…]

raw email with attach with a long filename on russian language.

[slevin@…]

Glad to see U again, Bas! Today I'm testing pre 2.4.0 version email2trac from svn. Short report:

1. Reopened ticket worked fine.
2. Still have problem with attachments filename on native language (like russian)

# cat maillog:

From slevin@adriver.ru  Wed Aug 10 16:50:10 2011
 Subject: Re: [admin-tst] #3555: Test em2tr 2.4.0
  Folder: /usr/bin/run_email2trac --project=admin-tst                     29883
Traceback (most recent call last):
  File "/usr/bin/email2trac", line 2558, in <module>
    tktparser.save_email_for_debug(m, settings.project_name, True)
  File "/usr/bin/email2trac", line 518, in save_email_for_debug
    self.debug_attachments(message_parts)
  File "/usr/bin/email2trac", line 476, in debug_attachments
    fx = open(part_file, 'wb')
IOError: [Errno 36] File name too long: '/tmp/%D0%9F%D1%80%D0%BE%D0%B2%D0%B5%D1%80%D0%BA%D0%B0%20%D1%80%D1%83%D1%81%D1%81%D0%BA%D0%B8%D1%85%20%D0%B8%D0%BC%D0%B5%D0%BD%20%D1%84%D0%B0%D0%B9%D0%BB%D0%BE%D0%B2%20%D0%BD%D0%BE%D0%BC%D0%B5%D1%80%20%D0%B4%D0%B2%D0%B0%D0%A7%D1%82%D0%BE%D0%B1%D1%8B%20%D0%B5%D1%89%D0%B5%20%D1%82%D0%B0%D0%BA%D0%BE%D0%B5%20%D1%82%D1%83%D1%82%20%D0%BD%D0%B0%D0%BF%D0%B8%D1%81%D0%B0%D1%82%D1%8C%20%D0%BF%D0%BE-%D1%80%D1%83%D1%81%D1%81%D0%BA%D0%B8-%D1%82%D0%BE%20%D1%8B%20324354%20nginx.doc.att_email2trac'

And we're can see result problem, what about I told early. email2trac convert filename from UTF-8 charset to URL-encode, as result we're get filename lenght 480 characters from source filename lenght 182 characters.

echo %D0%9F%D1%80%D0%BE%D0%B2%D0%B5%D1%80%D0%BA%D0%B0%20%D1%80%D1%83%D1%81%D1%81%D0%BA%D0%B8%D1%85%20%D0%B8%D0%BC%D0%B5%D0%BD%20%D1%84%D0%B0%D0%B9%D0%BB%D0%BE%D0%B2%20%D0%BD%D0%BE%D0%BC%D0%B5%D1%80%20%D0%B4%D0%B2%D0%B0%D0%A7%D1%82%D0%BE%D0%B1%D1%8B%20%D0%B5%D1%89%D0%B5%20%D1%82%D0%B0%D0%BA%D0%BE%D0%B5%20%D1%82%D1%83%D1%82%20%D0%BD%D0%B0%D0%BF%D0%B8%D1%81%D0%B0%D1%82%D1%8C%20%D0%BF%D0%BE-%D1%80%D1%83%D1%81%D1%81%D0%BA%D0%B8-%D1%82%D0%BE%20%D1%8B%20324354%20nginx.doc.att_email2trac |wc -c
480
echo Проверка русских имен файлов номер дваЧтобы еще такое тут написать по-русски-то ы 324354 nginx.doc.att_email2trac|wc -c
182

It is a problem. A big problem. Example of file I'm attached to this ticket, o'couse and raw mail too. I hope U can fix it and anyway I'm ready to test fix asap.

--
WBR,
Sergey V.Levin

[slevin@…]

raw email with attach with a long filename on russian language.

[slevin@…]

BODY of raw email with attach with a long filename on russian language.

[slevin@…]

raw email with attach with a long filename on russian language. In /tmp I received both copies of RAW mail on each email with attachment

[slevin@…]

raw email with attach with a long filename on russian language. In /tmp I received both copies of RAW mail on each email with attachment

[slevin@…]

So as I see in received mail from yours trac

#244: Re: [admin-tst] #3555: Test em2tr 2.4.0
-------------------------------+-------------------
 Reporter:  slevin@…           |      Type:  defect
   Status:  new                |  Priority:  major
Component:  testje             |
-------------------------------+-------------------
[skiped]

 [attachment:"Проверка русских имен файлов номер дваЧтобы еще такое тут
 написать по-русски-то ы 324354 nginx.doc"]

you start works with fix and I guess - done the fix... right?

[bas]

(In [548]) email2trac can now handle large filenames, trac report an error, see #247

[bas]

I see that my report is vanished. It is trac error. You get the same error if you try to add the attachment via the trac web interface:

Trac detected an internal error:
OSError: [Errno 36] File name too long: '/data/trac/osd/project/test_bas/attachments/ticket/244/%D0%9F%D1%80%D0%BE%D0%B2%D0%B5%D1%80%D0%BA%D0%B0%20%D1%80%D1%83%D1%81%D1%81%D0%BA%D0%B8%D1%85%20%D0%B8%D0%BC%D0%B5%D0%BD%20%D1%84%D0%B0%D0%B9%D0%BB%D0%BE%D0%B2%20%D0%BD%D0%BE%D0%BC%D0%B5%D1%80%20%D0%B4%D0%B2%D0%B0%D0%A7%D1%82%D0%BE%D0%B1%D1%8B%20%D0%B5%D1%89%D0%B5%20%D1%82%D0%B0%D0%BA%D0%BE%D0%B5%20%D1%82%D1%83%D1%82%20%D0%BD%D0%B0%D0%BF%D0%B8%D1%81%D0%B0%D1%82%D1%8C%20%D0%BF%D0%BE-%D1%80%D1%83%D1%81%D1%81%D0%BA%D0%B8-%D1%82%D0%BE%20%D1%8B%20324354%20nginx.doc.att_email2trac'

So you have to report a bug at Trac.

[anonymous]

Ahhh! Yea, today I allready make other bugreport to trac :) Time to do it again :) Thank you, Bas! In this ticket I'll put link to the ticket in trac bugtracker.

[bas]

(In [549]) added a check_filename_length function. This function will check if the filename is too long for the OS. Is so we try to truncate it, see #247

and removed some obsolete code

[bas]

can you test the trunk release?

[bas]

(In [550]) fixed an error in check_filename_length must return 'None' if filename is empty. see #247

[bas]

For reference trac core ticket:

• ​http://trac.edgewall.org/ticket/10313

[slevin@…]

Replying to bas:

can you test the trunk release?

Can U give me link on trunk release? I'm sleeping nearly 2,5 hours today and a my mind thinking very slowly :(

[bas]

maybe test an another day ;-). It is on the main wiki page:

• svn co ​https://subtrac.sara.nl/svn/email2trac/svn/trunk

[slevin@…]

I forgot show it previously:

svn co  https://subtrac.sara.nl/svn/email2trac/svn/trunk
svn: OPTIONS of 'https://subtrac.sara.nl/svn/email2trac/svn/trunk': 200 OK (https://subtrac.sara.nl)

svn export https://subtrac.sara.nl/svn/email2trac/svn/trunk
svn: OPTIONS of 'https://subtrac.sara.nl/svn/email2trac/svn/trunk': 200 OK (https://subtrac.sara.nl)

early I sync with trunk with:

svn export https://subtrac.sara.nl/oss/email2trac/svn/trunk

hmmm, what the mistake?

[bas]

Was an error in the documentation is now fixed:

• svn export ​https://subtrac.sara.nl/oss/email2trac/svn/trunk

[slevin@…]

Test last trunk revision (553)

[slevin@…]

In log - 1st test on last trunk release, 2nd - test on prvisous trunk release.

[bas]

(In [555]) rewrote save_email_for_debug. So no unicode problems and all files have the same prefix, see #247

[slevin@…]

log with errors when parsing filename on russian language again.

[slevin@…]

Sorry Bas, I'm attached the file log of email2trac (version from svn - pobably v.2.4.0). We're get more errors when try to use it on production trac.

[bas]

see ticket #271