MIME parsing in Mozilla

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

MIME parsing in Mozilla

Jim-267
As promised (I think I promised this anyway), here's my take on where we
should be headed with MIME parsing in Mozilla (read:
Thunderbird/Seamonkey). Currently, we have some obvious problems related
to bug 602718[1], since it regresses our ability to view attachments in
certain kinds of malformed messages.

Before I get too deep into this, I want to say that I absolutely think
we need to plan out where we're going with this, since bug 602718 was
meant to fix fallout from bug 351224[2], and we've got bug 674473[3] to
fix fallout from 602718. We really need to figure out what the most
appropriate behavior is before we continue further.

In the short term, we may want to consider backing out bug 351224 and
friends, since detaching multipart/related parts is much less important
than being able to see parts from (admittedly malformed) messages. While
a part of me wants to say that Thunderbird is behaving correctly here,
and that we shouldn't have to deal with malformed messages, the reality
is that they exist and there's not a whole lot that users can do about
it. We should try to follow the old adage: "be lenient in what you
accept and strict in what you produce." I'm not dead-set on this course,
but I do think it's worth considering.

In the medium term, we should strive to be fully standards-compliant
with messages by default, with options to tweak behavior according to
user preferences or to deal with malformed messages. Specifically, this
means that by default we should:

* Show "Content-Disposition: inline" parts inline only
* Show "Content-Disposition: attachment" parts in the attachment pane
   only
* Ignore attachment-like parts in other containers, e.g.
   * multipart/alternative parts we don't handle (in a properly formed
     message, these parts should only have presentational differences)
   * multipart/related parts that aren't referenced in the message

For the last case, we could probably be smart enough to throw up a
message along the lines of "This message is malformed and some parts of
it may be hidden. Please contact the sender to let them know. In the
meantime, would you like to show hidden attachments?" Saying yes would
flip a pref that treats every conceivable part as an attachment, i.e.
Content-Disposition: inline parts, all multipart/related parts,
multipart/alternative parts we don't render (or maybe just those we
*can't* render).

We should also have an option for how to handle Content-Disposition. We
already have this, but it's currently a boolean, and I think it should
be a 3-state: 1) use defaults from the message, 2) show all inline, 3)
show none inline. In the case of (3), we may or may not want to list
Content-Disposition: attachment parts in the attachment list. I think we
probably should, since that makes it easier to save those parts.

We could also add the ability to open/detach/delete parts that are
displayed inline. This wouldn't help with dealing with large batches of
inline parts, but it would let people do everything with inline parts
that they can do with attachment parts.

Farther in the future, we should look into supporting other parts of the
RFCs, e.g. message/partial. We should also ensure that the messages
produced by Mozilla are well-formed, including letting people change the
various properties of parts, e.g. setting Content-Disposition. This
would help us to be good neighbors and to let users at least attempt to
indicate how they want their message to be rendered.

I'm sure there are other libmime gurus who have some input on this, and
who probably have a more thorough understanding of the relevant RFCs, so
I'm interested to hear what others think of this plan.

- Jim

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=602718
[2] https://bugzilla.mozilla.org/show_bug.cgi?id=351224
[3] https://bugzilla.mozilla.org/show_bug.cgi?id=674473
_______________________________________________
dev-apps-thunderbird mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-apps-thunderbird
Reply | Threaded
Open this post in threaded view
|

Re: MIME parsing in Mozilla

Blake Winton
On 11-09-07 0:40 , Jim wrote:
> Before I get too deep into this, I want to say that I absolutely think
> we need to plan out where we're going with this, since bug 602718 was
> meant to fix fallout from bug 351224[2], and we've got bug 674473[3] to
> fix fallout from 602718. We really need to figure out what the most
> appropriate behavior is before we continue further.

I completely agree.

> In the short term, we may want to consider backing out bug 351224 and
> friends, since detaching multipart/related parts is much less important
> than being able to see parts from (admittedly malformed) messages. While
> a part of me wants to say that Thunderbird is behaving correctly here,
> and that we shouldn't have to deal with malformed messages, the reality
> is that they exist and there's not a whole lot that users can do about
> it. We should try to follow the old adage: "be lenient in what you
> accept and strict in what you produce." I'm not dead-set on this course,
> but I do think it's worth considering.

I think, just as Firefox has to deal with malformed pages, we need to
deal with malformed email messages as best we can.  I also don't think
we have the market penetration to force Outlook/Exchange to conform to
the standards, so attempting to pressure them by not showing their
malformed messages would only hurt us.

> Specifically, this means that by default we should:
>
> * Show "Content-Disposition: inline" parts inline only
> * Show "Content-Disposition: attachment" parts in the attachment pane
> only

I'm with you so far.

> * Ignore attachment-like parts in other containers, e.g.
> * multipart/alternative parts we don't handle (in a properly formed
> message, these parts should only have presentational differences)
> * multipart/related parts that aren't referenced in the message

I think this is a bad idea, for the reasons given above.  Even the
preference/popup wouldn't be enough to mitigate it, I'm afraid.

> I'm sure there are other libmime gurus who have some input on this, and
> who probably have a more thorough understanding of the relevant RFCs, so
> I'm interested to hear what others think of this plan.

I'm not exactly a libmime guru, but those are my thoughts anyways.  ;)

Later,
Blake.

_______________________________________________
dev-apps-thunderbird mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-apps-thunderbird
Reply | Threaded
Open this post in threaded view
|

Re: MIME parsing in Mozilla

Andrew Sutherland-3
In reply to this post by Jim-267
On 09/06/2011 09:40 PM, Jim wrote:
> For the last case, we could probably be smart enough to throw up a
> message along the lines of "This message is malformed and some parts of
> it may be hidden. Please contact the sender to let them know. In the
> meantime, would you like to show hidden attachments?" Saying yes would
> flip a pref that treats every conceivable part as an attachment, i.e.
> Content-Disposition: inline parts, all multipart/related parts,
> multipart/alternative parts we don't render (or maybe just those we
> *can't* render).

I dub this the "moral superiority box".

I don't see the benefit of this so-called moral superiority box.  Why
wouldn't I want to see attachments?  Maybe if we spun it more as a box
that just says "You can safely feel superior to this sender" as a
branding thing and then had it auto-hide, or the button said "yes, I
have felt superior than this person, don't show this again for them."

The only reason I could think of is that it might make some things
easier for spammers if it allowed them to bypass our phishing/spam defenses.

In general, I am on board with the "we should show everything that's in
there unless we have a reason to believe there's a good reason to ignore
it."  In ambiguous cases, it seems reasonable to just throw it in the
list of attachments.

Andrew
_______________________________________________
dev-apps-thunderbird mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-apps-thunderbird
Reply | Threaded
Open this post in threaded view
|

Re: MIME parsing in Mozilla

JoeS-3
In reply to this post by Jim-267
On 9/7/2011 12:40 AM, Jim wrote:
> In the short term, we may want to consider backing out bug 351224 and
> friends, since detaching multipart/related parts is much less important
> than being able to see parts from (admittedly malformed) messages. While
> a part of me wants to say that Thunderbird is behaving correctly here,
> and that we shouldn't have to deal with malformed messages, the reality
> is that they exist and there's not a whole lot that users can do about
> it. We should try to follow the old adage: "be lenient in what you
> accept and strict in what you produce." I'm not dead-set on this course,
> but I do think it's worth considering.

This would be a very large back-step IMO.
It would rely on most users knowing the old work-around of using
View>>Plaintext only to see the attachments in the attachment pane. And
I'd bet that is a very low % of the user base.

The current situation is that we are now able to see attachments that
were previously hidden (I do recall a bug where an airline ticket
confirmation was not even viewable/detectable)
Let's capitalize on that and move forward.
So, for bug 674473, just make it safe by dis-allowing detach/delete all
and start thinking about actual UI to enable the hidden pref in bug
602718 Just put it in the display options.

Automatically offering the pref with a notification bar would be ideal,
but I fear that might be too difficult to do in the short term.

Bottomline is: If there is info in that email intended for the recipient
and we don't have the ability to show it...that's a blackmark against TB
and a very real usability problem for some.

> We could also add the ability to open/detach/delete parts that are
> displayed inline. This wouldn't help with dealing with large batches of
> inline parts, but it would let people do everything with inline parts
> that they can do with attachment parts.

It also wouldn't help in the case of non-displayable types like Pdf's


> [1] https://bugzilla.mozilla.org/show_bug.cgi?id=602718
> [2] https://bugzilla.mozilla.org/show_bug.cgi?id=351224
> [3] https://bugzilla.mozilla.org/show_bug.cgi?id=674473


--
JoeS Using TB 9.0.."I am endeavoring to construct a pneumonic memory
circuit using stone knives and bear skins."(Spock)
<http://kb.mozillazine.org/Thunderbird_6.0,_etc.>
Daily Build Thread RSS feed http://th3oretiker.de/stuff/thunderbirdfeed.xml
_______________________________________________
dev-apps-thunderbird mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-apps-thunderbird
Reply | Threaded
Open this post in threaded view
|

Re: MIME parsing in Mozilla

Jonathan Kamens
In reply to this post by Jim-267
Jim <[hidden email]> writes:
>Before I get too deep into this, I want to say that I absolutely think
>we need to plan out where we're going with this, since bug 602718 was
>meant to fix fallout from bug 351224[2], and we've got bug 674473[3] to
>fix fallout from 602718. We really need to figure out what the most
>appropriate behavior is before we continue further.

Yes, but figuring out the appropriate behavior isn't all that
needs to be done here.

I'll say publicly here what I've already written in various
private email threads: the libmime code is old, convoluted,
insufficiently documented, confusing, jumbled, and anywhere
from painfully difficult to impossible to maintain.

Unless we take drastic steps, the odds are that we will
continue to break things as we try to fix things inside this
code. And by "drastic steps," I mean replacing all the libmime
code with something newer and better, preferably something
which is (a) object-oriented rather than stream-oriented and
(b) maintained by somebody else. There are a number of
open-source C++ MIME parsers available nowadays. I understand
that when libmime was originally written, most (perhaps all)
of those parsers didn't exist, but now they do, and we should
take advantage of them rather than reinventing the wheel.

With the proper choice of a third-party MIME parser to
integrate into Thunderbird, it will be much easier to maintain
and enhance MIME-related functionality in Thunderbird in the
future.

>In the short term, we may want to consider backing out bug 351224 and
>friends, since detaching multipart/related parts is much less important
>than being able to see parts from (admittedly malformed) messages.

If you do this, you're just choosing which constituency to
upset.

Yes, there are a lot of people upset about not being able to
see some attachments.

However, if you look at the bug which prompted me to dive into
this mess in the first place, i.e., the bug about being unable
to delete attachments from certain multipart/related messages,
you will see that there were also a whole lot of people upset
about that bug as well.

Your claim that one is "much less important than" the other
is not at all obvious to me.

Furthermore, with the Show All Body Parts functionality, we
have a solution to the hidden attachments problem. Not an
ideal solution, mind you, but a solution. Whereas if you back
out the changes which now allow attachments in
multipart/related messages to be removed, then we will *not*
have a solution to that problem.

In my opinion, I don't think we should back out the changes
(and yes, I obviously am biased, since I wrote them). Rather,
I think we should make the View | Message Body As | All Body
Parts command visible to everyone rather than only visible if
you install an add-on to enable a hidden preference. I don't
agree with the UI decision to hide this functionality.

>In the medium term, we should strive to be fully standards-compliant
>with messages by default, with options to tweak behavior according to
>user preferences or to deal with malformed messages.

I agree with others who have expressed disagreement with this
approach. Thunderbird should simply do the best it can to
automatically and without complaint display as much of the
message as it can in the way the user is most likely to
expect. Whether or not the message is standards-compliant is
irrelevant to most users. No, I don't like that the clients
generating malformed MIME have to be accommodated, but that's
life.
_______________________________________________
dev-apps-thunderbird mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-apps-thunderbird
Reply | Threaded
Open this post in threaded view
|

Re: MIME parsing in Mozilla

JoeS-3
In reply to this post by JoeS-3
On 9/7/2011 8:11 PM, JoeS wrote:
> just make it safe by dis-allowing detach/delete all and start thinking
> about actual UI to enable the hidden pref in bug 602718 Just put it in
> the display options.

Wrong bug reference here.
This should be:
https://bugzilla.mozilla.org/show_bug.cgi?id=685072


--
JoeS Using TB 9.0.."I am endeavoring to construct a pneumonic memory
circuit using stone knives and bear skins."(Spock)
<http://kb.mozillazine.org/Thunderbird_6.0,_etc.>
Daily Build Thread RSS feed http://th3oretiker.de/stuff/thunderbirdfeed.xml
_______________________________________________
dev-apps-thunderbird mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-apps-thunderbird
Reply | Threaded
Open this post in threaded view
|

Re: MIME parsing in Mozilla

Jim-267
In reply to this post by JoeS-3
On 09/07/2011 07:11 PM, JoeS wrote:
> The current situation is that we are now able to see attachments that
> were previously hidden (I do recall a bug where an airline ticket
> confirmation was not even viewable/detectable)
> Let's capitalize on that and move forward.

Actually, the current situation is quite the opposite: "attachments"*
that were previously visible are now hidden, namely those in
multipart/related containers that aren't referred to in the main part.
Bear in mind that in Thunderbird 5-7, there's *no* way to get at this
information, since the "Show All Body Parts" option isn't in there. Even
in Thunderbird 8+, that option is locked behind a hidden pref, and then
you have to go and enable it in the menu.

To be safe, I loaded up 3.1.14 and tried this: "attachments" contained
in multipart/related objects are shown in the attachment pane. On trunk,
they are not, unless you turn on "Show All Body Parts". This is true
regardless of whether the message is HTML or text, and "Show as Plain
Text" does nothing to the attachments on either version.

I'm not arguing that we back the code out permanently, but that we back
it out until we get it working right. Then we can ship it.

> So, for bug 674473, just make it safe by dis-allowing detach/delete all
> and start thinking about actual UI to enable the hidden pref in bug
> 602718 Just put it in the display options.

That's bug 685072, not bug 674473 (you mentioned this in a later
message). The latter is about restoring the behavior in 3.1, which we
should do one way or another. Until then, we shouldn't ship code that
regresses functionality for an easily-diagnosable error like this. I'd
be less concerned about this if it weren't for the fact that Outlook
apparently produces these messages in question, and like it or not, we
need to deal with their brokenness.

> Automatically offering the pref with a notification bar would be ideal,
> but I fear that might be too difficult to do in the short term.

Unfortunately, offering the "Show All Body Parts" pref probably won't
fix things, since that pref breaks rendering of HTML bodies with inline
images. Ideally, we'd have an option that people can set permanently to
change what goes into the attachment pane: not only for
currently-invisible parts, but also to show inline parts in the
attachment pane for batch operation (e.g. saving all images referenced
in an HTML part).

* I hesitate to call anything an "attachment" that's not a subpart of a
   multipart/mixed with Content-Disposition: attachment.

- Jim
_______________________________________________
dev-apps-thunderbird mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-apps-thunderbird
Reply | Threaded
Open this post in threaded view
|

Re: MIME parsing in Mozilla

Jim-267
In reply to this post by Andrew Sutherland-3
On 09/07/2011 12:24 PM, Andrew Sutherland wrote:

> On 09/06/2011 09:40 PM, Jim wrote:
>> For the last case, we could probably be smart enough to throw up a
>> message along the lines of "This message is malformed and some parts of
>> it may be hidden. Please contact the sender to let them know. In the
>> meantime, would you like to show hidden attachments?" Saying yes would
>> flip a pref that treats every conceivable part as an attachment, i.e.
>> Content-Disposition: inline parts, all multipart/related parts,
>> multipart/alternative parts we don't render (or maybe just those we
>> *can't* render).
>
> I dub this the "moral superiority box".

I wouldn't argue with that naming. :) (Granted, I was also being a bit
flippant in my wording, since this wouldn't be a problem if people just
followed the RFCs.)

> I don't see the benefit of this so-called moral superiority box. Why
> wouldn't I want to see attachments?

For multipart/related subparts that aren't referenced anywhere, but
whose primary subpart is being displayed, there's probably no reason you
would want to hide them. However, for other cases, it's not as simple:
should we show multipart/alternative parts that we can't render? For
instance:

multipart/alternative
   text/plain
   text/html
   text/calendar

(Assuming we don't have Lightning installed), if a message with this
MIME structure is well-formed, we shouldn't need to bother with the
text/calendar part at all; the text/plain and text/html parts should
contain all the necessary info. I think the most correct/least obnoxious
way to handle this is to hide the text/calendar part by default.
However, if the message is malformed and those parts *don't* contain the
necessary info, we probably want some way of getting at the
text/calendar part.

Likewise, what should we do if we have multipart/related parts in a
not-actively-rendered part (e.g. images associated with an RTF part)?
Those parts might be useful, especially if they're largish image files,
but we have no way of knowing whether or not they're actually useful.

Admittedly, I'm trying to go with the simplest method here: follow the
RFCs to the letter by default, and provide knobs to go into "quirks
mode" for malformed messages. The "show hidden attachments" knob would
be serving double duty: 1) for people dealing with malformed messages as
described above, and 2) for people who want inline parts to show up in
the attachment pane, e.g. to allow saving all parts of a message with a
single click.

I'm open to the idea that there are some types of malformed messages
that are so common, we should just handle them silently. I'm not sure
exactly what those cases are, though I think multipart/related subparts
with no references to them are a pretty good starting point. Maybe there
are others.

- Jim
_______________________________________________
dev-apps-thunderbird mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-apps-thunderbird
Reply | Threaded
Open this post in threaded view
|

Re: MIME parsing in Mozilla

Jim-267
In reply to this post by Blake Winton
On 09/07/2011 10:04 AM, Blake Winton wrote:
>> * Show "Content-Disposition: inline" parts inline only
>> * Show "Content-Disposition: attachment" parts in the attachment pane
>> only
>
> I'm with you so far.

As an aside, I think this means we'd be able to hide mailing list sigs
from the attachment pane automatically (granted, Jonathan Protzenko
patched this in trunk).

>> * Ignore attachment-like parts in other containers, e.g.
>> * multipart/alternative parts we don't handle (in a properly formed
>> message, these parts should only have presentational differences)
>> * multipart/related parts that aren't referenced in the message
>
> I think this is a bad idea, for the reasons given above. Even the
> preference/popup wouldn't be enough to mitigate it, I'm afraid.

Bad for both cases, or just for the last one? Should we treat
multipart/alternative parts that we don't understand as attachments?

- Jim
_______________________________________________
dev-apps-thunderbird mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-apps-thunderbird
Reply | Threaded
Open this post in threaded view
|

Re: MIME parsing in Mozilla

Jim-267
In reply to this post by Jonathan Kamens
On 09/07/2011 08:00 PM, Jonathan Kamens wrote:

> Jim<[hidden email]>  writes:
>> Before I get too deep into this, I want to say that I absolutely think
>> we need to plan out where we're going with this, since bug 602718 was
>> meant to fix fallout from bug 351224[2], and we've got bug 674473[3] to
>> fix fallout from 602718. We really need to figure out what the most
>> appropriate behavior is before we continue further.
>
> Yes, but figuring out the appropriate behavior isn't all that
> needs to be done here.
>
> I'll say publicly here what I've already written in various
> private email threads: the libmime code is old, convoluted,
> insufficiently documented, confusing, jumbled, and anywhere
> from painfully difficult to impossible to maintain.

We've already been discussing this on mdat:
<http://groups.google.com/group/mozilla.dev.apps.thunderbird/browse_thread/thread/cf8c349907d129d1>.
Last I'd heard, Joshua Cranmer had a prototype parser written in
Javascript working. I'm not sure if third-party libs would be
appropriate here, but you're welcome to discuss this in that thread, or
on IRC.

As someone who works on libmime occasionally, I'm sympathetic to
people's desire to destroy it, but we shouldn't let libmime deteriorate
further while we wait for a different MIME parser.

>> In the short term, we may want to consider backing out bug 351224 and
>> friends, since detaching multipart/related parts is much less important
>> than being able to see parts from (admittedly malformed) messages.
>
> If you do this, you're just choosing which constituency to
> upset.
>
> Yes, there are a lot of people upset about not being able to
> see some attachments.
>
> However, if you look at the bug which prompted me to dive into
> this mess in the first place, i.e., the bug about being unable
> to delete attachments from certain multipart/related messages,
> you will see that there were also a whole lot of people upset
> about that bug as well.
>
> Your claim that one is "much less important than" the other
> is not at all obvious to me.

If we look at the totally-unscientific measure of which bugs have more
activity, then seeing the currently-hidden attachments is clearly a more
common use case: in 4 years, 3 people commented wanting the ability to
detach multipart/related subparts. In contrast, bug 602718 has 5 dupes
(they should probably be duped to bug 674473, but oh well) and at least
10 people posting about it (I lost count at this point) in various bugs
in the span of about a year.

> Furthermore, with the Show All Body Parts functionality, we
> have a solution to the hidden attachments problem. Not an
> ideal solution, mind you, but a solution. Whereas if you back
> out the changes which now allow attachments in
> multipart/related messages to be removed, then we will *not*
> have a solution to that problem.

This functionality isn't available in Thunderbird 7, so there's no
solution to this problem in that version. At the very least, this should
be backed out of 7. I'm ok with leaving the code in 9, and maybe in 8 if
we can get fixes in time, but not for 7.

> In my opinion, I don't think we should back out the changes
> (and yes, I obviously am biased, since I wrote them). Rather,
> I think we should make the View | Message Body As | All Body
> Parts command visible to everyone rather than only visible if
> you install an add-on to enable a hidden preference. I don't
> agree with the UI decision to hide this functionality.

If the option didn't negatively affect rendering of the body, I think
this would be ok. However, since it does, it's not something most people
would be able to enable for all messages*. I think that's what we want,
since people who need this option will probably need it for many
messages (e.g. messages from the same sender).

>> In the medium term, we should strive to be fully standards-compliant
>> with messages by default, with options to tweak behavior according to
>> user preferences or to deal with malformed messages.
>
> I agree with others who have expressed disagreement with this
> approach. Thunderbird should simply do the best it can to
> automatically and without complaint display as much of the
> message as it can in the way the user is most likely to
> expect. Whether or not the message is standards-compliant is
> irrelevant to most users. No, I don't like that the clients
> generating malformed MIME have to be accommodated, but that's
> life.

This is fine, provided people give compelling reasons why some malformed
messages should be handled in a certain way. We obviously need to do
*something* for malformed multipart/related objects, but for other types
of hidden content, it's less clear (e.g. multipart/alternative objects
with subparts we can't handle).

As I mentioned in an earlier reply, I'm not dead-set on being myopically
standards-compliant, but following the RFCs is the simplest course of
action, so I think it makes sense as a starting point. We can certainly
adjust the behavior to be more lenient, provided we aren't hurting
well-formed messages.

Basically, I'm ok with treating unreferenced multipart/related subparts
as attachments, but I want to make sure everyone agrees on what exactly
the behavior should be.

- Jim

* See WADA's comment here for more details:
   <https://bugzilla.mozilla.org/show_bug.cgi?id=674473#c19>
_______________________________________________
dev-apps-thunderbird mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-apps-thunderbird
Reply | Threaded
Open this post in threaded view
|

Re: MIME parsing in Mozilla

Jonathan Kamens
Jim <[hidden email]> writes:
>If we look at the totally-unscientific measure of which bugs have more
>activity, then seeing the currently-hidden attachments is clearly a more
>common use case: in 4 years, 3 people commented wanting the ability to
>detach multipart/related subparts. In contrast, bug 602718 has 5 dupes
>(they should probably be duped to bug 674473, but oh well) and at least
>10 people posting about it (I lost count at this point) in various bugs
>in the span of about a year.

I'm not sure what bug you're looking at. I see 15 users on
the CC list and 7 votes for bug 351224 about deleting
multipart/related attachments. Bug 417646, which was
eventually marked a dup of 351224, has 17 users on the CC
list and 8 votes (no, I didn't look at how much overlap there
was).

>This functionality isn't available in Thunderbird 7, so there's no
>solution to this problem in that version. At the very least, this should
>be backed out of 7. I'm ok with leaving the code in 9, and maybe in 8 if
>we can get fixes in time, but not for 7.

We could just as easily backport the All Body Parts
functionality to Thunderbird 7 as back out the fix for bug
351224. In fact, backporting the All Body Parts changes would
probably be easier.

>If the option didn't negatively affect rendering of the body, I think
>this would be ok. However, since it does, it's not something most people
>would be able to enable for all messages*. I think that's what we want,
>since people who need this option will probably need it for many
>messages (e.g. messages from the same sender).

I don't agree.

I think while it's not ideal, it's perfectly reasonable for
users to switch to View | Message Body As | All Body Parts
temporarily when they notice that there's an attachment
they're supposed to be seeing but don't.

Certainly, I think asking users to do this temporarily until
we improve libmime is better than going back to the days when
it was impossible to detach or delete attachments in certain
MIME messages.
_______________________________________________
dev-apps-thunderbird mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-apps-thunderbird
Reply | Threaded
Open this post in threaded view
|

Re: MIME parsing in Mozilla

Joshua Cranmer-2
In reply to this post by Jonathan Kamens
On 9/7/2011 8:00 PM, Jonathan Kamens wrote:

> Unless we take drastic steps, the odds are that we will continue to
> break things as we try to fix things inside this code. And by "drastic
> steps," I mean replacing all the libmime code with something newer and
> better, preferably something which is (a) object-oriented rather than
> stream-oriented and (b) maintained by somebody else. There are a
> number of open-source C++ MIME parsers available nowadays. I
> understand that when libmime was originally written, most (perhaps
> all) of those parsers didn't exist, but now they do, and we should
> take advantage of them rather than reinventing the wheel. With the
> proper choice of a third-party MIME parser to integrate into
> Thunderbird, it will be much easier to maintain and enhance
> MIME-related functionality in Thunderbird in the future.

I have not looked into any of the existing parsers, but I have doubts
that too many of the existing parsers can be placed into Thunderbird
without considerable work. The minimum criteria (as I understand it)
that we would need to be able to replace libmime:
1. It must be able to support all of our supported platforms (Win32,
Linux x86 and x86_64, and Mac OS X).
2. It must be license-compatible with Thunderbird. To my understanding,
only MPL/<something LGPL compatible, maybe Apache might work here> or
BSD would fit the bill.
3. It must be able to interface with our network code and our crypto
library.
4. It must support IMAP parts-on-demand capabilities.
5. It must properly support l10n and i18n concerns where important,
including, notably, RFC 2231 and RFC 2047 support (and probably RFC 5355).
Other important, but not necessarily deal-breakers:
6. It should not bring in Yet Another Base Library; we already build
chromium, nspr, and xpcom as underlying cross-platform capability
layers; I don't want to shoehorn yet another.
7. Support for uuencode, yenc, TNEF, and PGP should either be builtin or
buildable without restructuring the backend.
8. I'd rather not see a library which requires the entire message just
to parse the header. Actually, for some parts of our codebase, I would
really like to feed in *just* the headers.
9. The HTML output (both from the text/html and from text-to-HTML
conversion) should mesh nicely with our Gecko rendering engine--I'm
concerned about the impact of things like the cid: on output.
10. The output should be easily reflectable into JS. Face it: many MIME
consumers want to be in JS.
11. The code should be at least as stable as libmime is (in terms of
crashes), and the upstream maintainers should be willing to accept
patches that we propose to fix actual problems.

MIME is a monstrous, ugly mess of standards; combined with a
surprisingly widespread tendency to violate the MIME standards
(including, in a way, MIME itself [1]), this means that building one is
quite difficult. My experience with a side project was that I found
Python's standard-library implementation of MIME insufficient to handle
real-world data after fewer than 72 hours of data collection [2],
finding three distinct cases that caused the parsing to fail hard (i.e.,
throw an exception); I am unaware of the number of files it parsed
silently incorrectly. I doubt that a MIME parser that is not already
used as the core MIME engine in an email client is capable of satisfying
all of the requirements that we need it to.

Even though I am working on a brand new MIME parser written from
scratch, I have often found myself going back to read what the current
MIME parser does (mostly in error-recovery scenarios). We actually have
a good MIME parser deep in our code; it is just difficult to use or
modify. I furthermore doubt that I would trust my new MIME parser to be
sufficient to replace the current libmime anytime this year.

[1] MIME specifically states that container types must not be base64
encoded. S/MIME's message/encrypted is effectively a MIME container,
but, since it is binary text, it is in base64.
[2] For a moderate stress test for a MIME parser, pipe the output of
HEAD'ing every single message on any Usenet server into it. See how long
before it produces a bad parse. And this doesn't even cover the whole
spectrum of abuse, only that which gets outputted by NNTP clients (which
mostly tend to avoid MIME).
_______________________________________________
dev-apps-thunderbird mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-apps-thunderbird
Reply | Threaded
Open this post in threaded view
|

Re: MIME parsing in Mozilla

Arivald
In reply to this post by Jim-267
W dniu 2011-09-08 03:35, Jim pisze:

> On 09/07/2011 12:24 PM, Andrew Sutherland wrote:
>> On 09/06/2011 09:40 PM, Jim wrote:
>>> For the last case, we could probably be smart enough to throw up a
>>> message along the lines of "This message is malformed and some parts of
>>> it may be hidden. Please contact the sender to let them know. In the
>>> meantime, would you like to show hidden attachments?" Saying yes would
>>> flip a pref that treats every conceivable part as an attachment, i.e.
>>> Content-Disposition: inline parts, all multipart/related parts,
>>> multipart/alternative parts we don't render (or maybe just those we
>>> *can't* render).
>>
>> I dub this the "moral superiority box".
>
> I wouldn't argue with that naming. :) (Granted, I was also being a bit
> flippant in my wording, since this wouldn't be a problem if people just
> followed the RFCs.)
>
>> I don't see the benefit of this so-called moral superiority box. Why
>> wouldn't I want to see attachments?
>
> For multipart/related subparts that aren't referenced anywhere, but
> whose primary subpart is being displayed, there's probably no reason you
> would want to hide them. However, for other cases, it's not as simple:
> should we show multipart/alternative parts that we can't render? For
> instance:
>
> multipart/alternative
> text/plain
> text/html
> text/calendar
>
> (Assuming we don't have Lightning installed), if a message with this
> MIME structure is well-formed, we shouldn't need to bother with the
> text/calendar part at all; the text/plain and text/html parts should
> contain all the necessary info. I think the most correct/least obnoxious
> way to handle this is to hide the text/calendar part by default.
> However, if the message is malformed and those parts *don't* contain the
> necessary info, we probably want some way of getting at the
> text/calendar part.
>

For multipart/alternative, maybe we should give user choice which
version he want to see? Come kind of contol in message header (tabs?
menu-button?), which will list all alternatives, and allow to select
alternative version. Plus preferences to select default MIME part to
view (like: i prefer HTML, then RTF, then plain).



> Likewise, what should we do if we have multipart/related parts in a
> not-actively-rendered part (e.g. images associated with an RTF part)?
> Those parts might be useful, especially if they're largish image files,
> but we have no way of knowing whether or not they're actually useful.
>

In case of not referenced multipart/related parts, i prefer to show them
like attachments, but separately. Make distinct that this parts are not
pure attachments, they are just parts of content.

--
Arivald
_______________________________________________
dev-apps-thunderbird mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-apps-thunderbird
Reply | Threaded
Open this post in threaded view
|

Re: MIME parsing in Mozilla

günter
In reply to this post by Jim-267
Am 08.09.11 03:35, schrieb Jim:

> For multipart/related subparts that aren't referenced anywhere, but
> whose primary subpart is being displayed, there's probably no reason you
> would want to hide them. However, for other cases, it's not as simple:
> should we show multipart/alternative parts that we can't render? For
> instance:
>
> multipart/alternative
>    text/plain
>    text/html
>    text/calendar
>
> (Assuming we don't have Lightning installed), if a message with this
> MIME structure is well-formed, we shouldn't need to bother with the
> text/calendar part at all; the text/plain and text/html parts should
> contain all the necessary info. I think the most correct/least obnoxious
> way to handle this is to hide the text/calendar part by default.
> However, if the message is malformed and those parts *don't* contain the
> necessary info, we probably want some way of getting at the
> text/calendar part.

Not sure how you are going to handle multipart/alternative (eg.
text/calendar). You shouldn't assume a specific app has to be installed
to display/handle the attachment.
F.e. Lightning isn't the only one to deal with text/calendar.
There are others out there (like ReminderFox) which needs to handle
those attachments as well. So a mechanic to hide that type pending on a
no installed app is a bad decision.
Maybe I got you wrong?
_______________________________________________
dev-apps-thunderbird mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-apps-thunderbird