Modify

Opened 9 years ago

Closed 9 years ago

Last modified 9 years ago

#247 closed enhancement (fixed)

Fix for problem with long attachment's filenames on native language

Reported by: anonymous Owned by: bas
Priority: major Milestone: New email2trac release 2.4.0
Component: email2trac Version: 2.0.2
Keywords: Cc:

Description

Sympthoms: when user attached into email for trac file with long name (more then 25+ characters) or file with filename on native laguage (like russian) - email2trac can't store that attachment from message and do nothing. No notify about fail for user, or create new ticket or modify exist ticket WITHOUT attachment from message.

Solution: Required external python module trans. URL: http://pypi.python.org/pypi/trans "At the moment, Czech, Greek, Latvian, Polish, Turkish, Russian, Ukrainian alphabets are supported (it covers 99% of needs)."

Path file in attach. Also included some fix for reopen ticket via email.

Attachments (10)

2longfilename.patch (1.5 KB) - added by slevin@… 9 years ago.
patch for problem with attachment on native language (like Russian)
lubov.mail (11.6 KB) - added by slevin@… 9 years ago.
grep from error log (problem with convert russian filename
2longfilename.2.patch (3.8 KB) - added by slevin@… 9 years ago.
patch in universal format
rawemail.email2trac (27.8 KB) - added by slevin@… 9 years ago.
raw email with attach with a long filename on russian language.
tmpNsEc4m.admin-tst.email2trac (79.2 KB) - added by slevin@… 9 years ago.
raw email with attach with a long filename on russian language.
tmpdtxgQI.body_email2trac (541 bytes) - added by slevin@… 9 years ago.
BODY of raw email with attach with a long filename on russian language.
tmphKvdSv.body_email2trac (541 bytes) - added by slevin@… 9 years ago.
raw email with attach with a long filename on russian language. In /tmp I received both copies of RAW mail on each email with attachment
tmpjsyhG1.admin-tst.email2trac (79.2 KB) - added by slevin@… 9 years ago.
raw email with attach with a long filename on russian language. In /tmp I received both copies of RAW mail on each email with attachment
email2trac.log (13.5 KB) - added by slevin@… 9 years ago.
Test last trunk revision (553)
email2trac.log.gz (60.1 KB) - added by slevin@… 9 years ago.
log with errors when parsing filename on russian language again.

Download all attachments as: .zip

Change History (45)

Changed 9 years ago by slevin@…

patch for problem with attachment on native language (like Russian)

comment:1 Changed 9 years ago by bas

Thanks for the patch. Can attach a diff -u so i can apply it easily. I see that trans is not a standard pythin module. So i will make an option for it! We also have really long filenames but never experienced any problems with it. Is it only a problem with the languages you mentioned?

There is no error message in a log (syslog, trac.log).

comment:2 Changed 9 years ago by slevin@…

I can talk only about problem with my native language - Russian,a after call util.text.unicode function we're get name where each russian character equal FOR (4) unicode characters, example: "Имя" ("Name" on eng.) have 3 characters in it, after util.text.unicdoe I reveived the name with 12 chacters in it! For my organisation offten the situations when user put into message file with name on russian nearly 40-50 characters lenght, after unicoding I received teh filename lenght nearly 160-300 characters, o'course without path and name extention! And the system can't store that file. Grep from error log (by procmail) I'm attached to that message, also attached patch file in universal format.

Changed 9 years ago by slevin@…

grep from error log (problem with convert russian filename

Changed 9 years ago by slevin@…

patch in universal format

comment:3 Changed 9 years ago by bas

Stevin the fix for re-open tickets via email is for workflow option. In the past we has another option for email2trac.conf:

workflow: <something>

So when i ticket was closed and a update was received with workflow set. It would trigger this workflow. It is still here for backward compatibility. I will slightly modify the patch. It is a bug, we have set workflow_reopen in email2trac.conf. So didn't catch this one.

I am thinking about a good option for using the trans package as it is not a standard python module.

comment:4 Changed 9 years ago by bas

Does trac not provide some function for this? Translate filenames.

comment:5 Changed 9 years ago by anonymous

Last new about trans module, atm my user testing patch and I have not good news, we're have problem with attachment only with english letters in filename. I think about fix it.

comment:6 Changed 9 years ago by bas

Wahat worries me is that the output you send about this problem:

Subject: =?KOI8-R?Q?=EE=CF=D7=CF=C5_=FA=C5=D2=CB=C1=CC=CF_=C4=CC=D1_?=
4	  Folder: /usr/bin/run_email2trac --project=admin                        359820
5	Traceback (most recent call last):
6	  File "/usr/lib64/python2.5/logging/handlers.py", line 667, in emit
7	    self.socket.send(msg)
8	UnicodeEncodeError: 'ascii' codec can't encode characters in position 46-52: ordinal not in range(128)
9	Traceback (most recent call last

I have just patches email_to_unicode, maybe there is a charset missing?

comment:7 Changed 9 years ago by bas

fx = open(part_file, 'wb')
201	IOError: [Errno 36] File name too long: '/tmp/%D0%94%D0%BE%D0%B3%D0%BE%D0%B2%D0%BE%D1%80%20%D0%BE%20%D0%BF%D1%80%D0%B5%D0%B4%D0%BE%D1%81%D1%82%D0%B0%D0%B2%D0%BB%D0%B5%D0%BD%D0%B8%D0%B8%20%D1%83%D1%81%D0%BB%D1%83%D0%B3%20%D1%82%D0%B5%D0%BB%D0%B5%D0%BA%D0%BE%D0%BC%D0%BC%D1%83%D0%BD%D0%B8%D0%BA%D0%B0%D1%86%D0%B8%D0%B9.doc'

We can catch this error and just generate attachments names that are not longer then a configurable lenght

comment:8 Changed 9 years ago by bas

(In [526]) Fix re-opening for closed tickets for trac 0.12 and higher. Only triggered when no workflow was defined in email2trac.conf, see #247

comment:9 Changed 9 years ago by bas

(In [527]) workflow must be set to None, see #247

comment:10 Changed 9 years ago by anonymous

Bas, I'm really glad to help you, but I don't know python language :) only python handbook on my table :)

comment:11 Changed 9 years ago by bas

No problem. I was just express my thoughts. Maybe we can save the attachemnts with random names and use the filename as link. It has some impact on the code,

comment:12 Changed 9 years ago by anonymous

  • Owner changed from bas to anonymous

U mean cutting attachment from message into file with numeric_name.real_ext and into tickets/blog/etc body we're put real name of attachment and do hyperlink to real file in file system... I guess that can be solve problem with name forever. Good idea.

comment:13 Changed 9 years ago by bas

  • Owner changed from anonymous to bas
  • Status changed from new to assigned

That is the general idea, just use number for example and make the link the real name

comment:14 Changed 9 years ago by bas

I just looked at the source of trac and what i try to accomplish is not supported :-(. Must search for another solution.

comment:15 Changed 9 years ago by bas

Can you test the trunk, with debug set to 0 and see if trac will complain or fail in the attachement functions.

I forgot to ask which OS you use? and maybe i can obtain the raw email message to test some things?

Last edited 9 years ago by bas (previous) (diff)

comment:16 Changed 9 years ago by anonymous

a long silence from my side again :( Atm our trac worked on the server with OS gentoo linux. U can take the raw email from attach to this ticket.

About test the trunk: ok - tomorrow I'll tets the trunk on mirror server of our trac in production.

Changed 9 years ago by slevin@…

raw email with attach with a long filename on russian language.

comment:17 Changed 9 years ago by slevin@…

Glad to see U again, Bas! Today I'm testing pre 2.4.0 version email2trac from svn. Short report:

  1. Reopened ticket worked fine.
  2. Still have problem with attachments filename on native language (like russian)

# cat maillog:

From slevin@adriver.ru  Wed Aug 10 16:50:10 2011
 Subject: Re: [admin-tst] #3555: Test em2tr 2.4.0
  Folder: /usr/bin/run_email2trac --project=admin-tst                     29883
Traceback (most recent call last):
  File "/usr/bin/email2trac", line 2558, in <module>
    tktparser.save_email_for_debug(m, settings.project_name, True)
  File "/usr/bin/email2trac", line 518, in save_email_for_debug
    self.debug_attachments(message_parts)
  File "/usr/bin/email2trac", line 476, in debug_attachments
    fx = open(part_file, 'wb')
IOError: [Errno 36] File name too long: '/tmp/%D0%9F%D1%80%D0%BE%D0%B2%D0%B5%D1%80%D0%BA%D0%B0%20%D1%80%D1%83%D1%81%D1%81%D0%BA%D0%B8%D1%85%20%D0%B8%D0%BC%D0%B5%D0%BD%20%D1%84%D0%B0%D0%B9%D0%BB%D0%BE%D0%B2%20%D0%BD%D0%BE%D0%BC%D0%B5%D1%80%20%D0%B4%D0%B2%D0%B0%D0%A7%D1%82%D0%BE%D0%B1%D1%8B%20%D0%B5%D1%89%D0%B5%20%D1%82%D0%B0%D0%BA%D0%BE%D0%B5%20%D1%82%D1%83%D1%82%20%D0%BD%D0%B0%D0%BF%D0%B8%D1%81%D0%B0%D1%82%D1%8C%20%D0%BF%D0%BE-%D1%80%D1%83%D1%81%D1%81%D0%BA%D0%B8-%D1%82%D0%BE%20%D1%8B%20324354%20nginx.doc.att_email2trac'

And we're can see result problem, what about I told early. email2trac convert filename from UTF-8 charset to URL-encode, as result we're get filename lenght 480 characters from source filename lenght 182 characters.

echo %D0%9F%D1%80%D0%BE%D0%B2%D0%B5%D1%80%D0%BA%D0%B0%20%D1%80%D1%83%D1%81%D1%81%D0%BA%D0%B8%D1%85%20%D0%B8%D0%BC%D0%B5%D0%BD%20%D1%84%D0%B0%D0%B9%D0%BB%D0%BE%D0%B2%20%D0%BD%D0%BE%D0%BC%D0%B5%D1%80%20%D0%B4%D0%B2%D0%B0%D0%A7%D1%82%D0%BE%D0%B1%D1%8B%20%D0%B5%D1%89%D0%B5%20%D1%82%D0%B0%D0%BA%D0%BE%D0%B5%20%D1%82%D1%83%D1%82%20%D0%BD%D0%B0%D0%BF%D0%B8%D1%81%D0%B0%D1%82%D1%8C%20%D0%BF%D0%BE-%D1%80%D1%83%D1%81%D1%81%D0%BA%D0%B8-%D1%82%D0%BE%20%D1%8B%20324354%20nginx.doc.att_email2trac |wc -c
480
echo Проверка русских имен файлов номер дваЧтобы еще такое тут написать по-русски-то ы 324354 nginx.doc.att_email2trac|wc -c
182

It is a problem. A big problem. Example of file I'm attached to this ticket, o'couse and raw mail too. I hope U can fix it and anyway I'm ready to test fix asap.

--
WBR,
Sergey V.Levin

Changed 9 years ago by slevin@…

raw email with attach with a long filename on russian language.

Changed 9 years ago by slevin@…

BODY of raw email with attach with a long filename on russian language.

Changed 9 years ago by slevin@…

raw email with attach with a long filename on russian language. In /tmp I received both copies of RAW mail on each email with attachment

Changed 9 years ago by slevin@…

raw email with attach with a long filename on russian language. In /tmp I received both copies of RAW mail on each email with attachment

comment:18 Changed 9 years ago by slevin@…

So as I see in received mail from yours trac

#244: Re: [admin-tst] #3555: Test em2tr 2.4.0
-------------------------------+-------------------
 Reporter:  slevin@…           |      Type:  defect
   Status:  new                |  Priority:  major
Component:  testje             |
-------------------------------+-------------------
[skiped]

 [attachment:"Проверка русских имен файлов номер дваЧтобы еще такое тут
 написать по-русски-то ы 324354 nginx.doc"]

you start works with fix and I guess - done the fix... right?

comment:19 Changed 9 years ago by bas

(In [548]) email2trac can now handle large filenames, trac report an error, see #247

comment:20 Changed 9 years ago by bas

I see that my report is vanished. It is trac error. You get the same error if you try to add the attachment via the trac web interface:

Trac detected an internal error:
OSError: [Errno 36] File name too long: '/data/trac/osd/project/test_bas/attachments/ticket/244/%D0%9F%D1%80%D0%BE%D0%B2%D0%B5%D1%80%D0%BA%D0%B0%20%D1%80%D1%83%D1%81%D1%81%D0%BA%D0%B8%D1%85%20%D0%B8%D0%BC%D0%B5%D0%BD%20%D1%84%D0%B0%D0%B9%D0%BB%D0%BE%D0%B2%20%D0%BD%D0%BE%D0%BC%D0%B5%D1%80%20%D0%B4%D0%B2%D0%B0%D0%A7%D1%82%D0%BE%D0%B1%D1%8B%20%D0%B5%D1%89%D0%B5%20%D1%82%D0%B0%D0%BA%D0%BE%D0%B5%20%D1%82%D1%83%D1%82%20%D0%BD%D0%B0%D0%BF%D0%B8%D1%81%D0%B0%D1%82%D1%8C%20%D0%BF%D0%BE-%D1%80%D1%83%D1%81%D1%81%D0%BA%D0%B8-%D1%82%D0%BE%20%D1%8B%20324354%20nginx.doc.att_email2trac'

So you have to report a bug at Trac.

comment:21 Changed 9 years ago by anonymous

Ahhh! Yea, today I allready make other bugreport to trac :) Time to do it again :) Thank you, Bas! In this ticket I'll put link to the ticket in trac bugtracker.

comment:22 Changed 9 years ago by bas

(In [549]) added a check_filename_length function. This function will check if the filename is too long for the OS. Is so we try to truncate it, see #247

and removed some obsolete code

comment:23 follow-up: Changed 9 years ago by bas

can you test the trunk release?

comment:24 Changed 9 years ago by bas

(In [550]) fixed an error in check_filename_length must return 'None' if filename is empty. see #247

comment:25 Changed 9 years ago by bas

For reference trac core ticket:

comment:26 in reply to: ↑ 23 Changed 9 years ago by slevin@…

Replying to bas:

can you test the trunk release?

Can U give me link on trunk release? I'm sleeping nearly 2,5 hours today and a my mind thinking very slowly :(

comment:27 Changed 9 years ago by bas

maybe test an another day ;-). It is on the main wiki page:

comment:28 Changed 9 years ago by bas

  • Milestone set to New email2trac release 2.4.0

comment:29 Changed 9 years ago by slevin@…

I forgot show it previously:

svn co  https://subtrac.sara.nl/svn/email2trac/svn/trunk
svn: OPTIONS of 'https://subtrac.sara.nl/svn/email2trac/svn/trunk': 200 OK (https://subtrac.sara.nl)

svn export https://subtrac.sara.nl/svn/email2trac/svn/trunk
svn: OPTIONS of 'https://subtrac.sara.nl/svn/email2trac/svn/trunk': 200 OK (https://subtrac.sara.nl)

early I sync with trunk with:

svn export https://subtrac.sara.nl/oss/email2trac/svn/trunk

hmmm, what the mistake?

comment:30 Changed 9 years ago by bas

Was an error in the documentation is now fixed:

Changed 9 years ago by slevin@…

Test last trunk revision (553)

comment:31 Changed 9 years ago by slevin@…

In log - 1st test on last trunk release, 2nd - test on prvisous trunk release.

comment:32 Changed 9 years ago by bas

(In [555]) rewrote save_email_for_debug. So no unicode problems and all files have the same prefix, see #247

comment:33 Changed 9 years ago by bas

  • Resolution set to fixed
  • Status changed from assigned to closed

Changed 9 years ago by slevin@…

log with errors when parsing filename on russian language again.

comment:34 Changed 9 years ago by slevin@…

Sorry Bas, I'm attached the file log of email2trac (version from svn - pobably v.2.4.0). We're get more errors when try to use it on production trac.

comment:35 Changed 9 years ago by bas

see ticket #271

Add Comment

Modify Ticket

Change Properties
Action
as closed The owner will remain bas.
The resolution will be deleted. Next status will be 'reopened'.
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.