The purpose of binary components

classic Classic list List threaded Threaded
84 messages Options
12345
Reply | Threaded
Open this post in threaded view
|

Re: The purpose of binary components

PhillipJones
Kent James wrote:

> On 1/17/2012 1:24 PM, Zack Weinberg wrote:
>> To inform the choice between D and E, could authors of applications
>> outside m-c please chime in and tell us what they still need binary
>> components for? It seems *highly* desirable to me to get to a point
>> where Thunderbird, Seamonkey, etc don't need any binary components
>> beyond what's already in libxul.
>
> I've been a proponent of adding capability to Thunderbird to allow
> adding new account types beyond the current IMAP, POP3, News, and RSS
> types. The addon TweeQuilla (written in JavaScript) is a demonstration
> of adding a Twitter account type, and uses a binary addon "New Account
> Types" as a glue layer to the base Thunderbird code.
>
> The fundamental mailnews design model for accounts essentially has a
> base set of C++ classes (example: nsMsgDBFolder.cpp) that are then
> extended with C++ supertypes to implement account-specific functionality
> on top of XPCOM interfaces. So, for example, nsIMsgDBFolder has specific
> methods that need to be implemented for each account type(for IMAP in
> nsImapMailFolder.cpp), as well as has account-specific extensions to the
> XPCOM interface (for example nsIImapIncomingServer for the server).
>
> So why do I need binary XPCOM extensions? Because the entire mailnews
> architecture is based on a C++ inheritance model which does not map well
> onto javascript. Although the "New Account Types" addon showed that, in
> fact, it is possible to overcome those issues in javascript, everyone
> seemed to hate the complexity and ugliness required to do so, so
> attempts to propose adding some of that to core got quickly mired in
> criticism. While it would be possible to rewrite all of mailnews I
> suppose to be more javascript-friendly, realistically that is not likely
> to happen. I also know from bitter experience that even fairly small bug
> fixes in mailnews often lead to YEARS of chasing regressions through fix
> after fix, so do not kid yourself that any of this would be easy.
>
> As for the compatibility issue, for New Account Types I currently ship
> DLLs supporting versions 8, 9, and 10 in the same addon. In the review
> queue (for three weeks) is the next version that will support 9, 10, and
> 11. With this overall scheme, someone can switch seamlessly between the
> last-release, current release, and beta versions seamlessly. I as the
> developer run aurora. This really works just fine. I just wish that
> Lightning would go to similar scheme!
>
> So in case it is not clear, I'm quite opposed to any attempts to prevent
> remove binary compatibility for extensions for Thunderbird.
>
> rkent

Best thing is stop messing with the extension developers and pulling the
rug out from under them at every new update.  Lock it down to one system
and leave it alone. soon Mozilla will po the extension developers and
there will be no extension for Mozilla Products.  Mozilla and extension
should sit down together agree to a system. and the stick to that system.

--
Phillip M. Jones, C.E.T.        "If it's Fixed, Don't Break it"
http://www.phillipmjones.net        mailto:[hidden email]
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: The purpose of binary components

Mook-6
In reply to this post by Zack Weinberg-2
On 1/17/2012 1:24 PM, Zack Weinberg wrote:
> On 2012-01-17 7:30 AM, Benjamin Smedberg wrote:

> To inform the choice between D and E, could authors of applications
> outside m-c please chime in and tell us what they still need binary
> components for? It seems *highly* desirable to me to get to a point
> where Thunderbird, Seamonkey, etc don't need any binary components
> beyond what's already in libxul.

For things I've worked on: (Please remember these are personal opinions
and don't represent the projects involved.)

Songbird (application shipping its private XULRunner):
  - gstreamer integration - media playback, transcoding.  You really
don't want to be doing playback on the UI thread.  Chances of landing
this on m-c are low; the existing gstreamer media backend bug (for
Linux) is in limbo (and is unrelated anyway), and this was used for
Linux/Windows/Mac anyway.
  - Windows media integration (playback of wma, DRMed tracks, etc.) -
even less likely to land on m-c; basically, in a hypothetical world
where XPCOM is gone, js-ctypes is the only option.  I believe MS-COM is
involved on background threads.
  - Media tagging - gstreamer, wm, plus taglib.  Possibly js-ctypes,
assuming that works on worker threads.  (I've been told it does.)
  - USB interaction (as mass-storage devices).  Possibly whatever B2G
work, assuming it works cross-platform (Linux/Windows/Mac).
  - USB/MTP interaction.  This basically boils down to off-main-thread
MS-COM calls, I think.  If js-ctypes started supported that it might
work?  Not sure, but it might need to be called on threads it didn't
create too...
  - Some database stuff (its own copy of sqlite and weird caching and...
stuff.)

Overall: if binary XPCOM is dead, it's more likely to just stop
upgrading Gecko than it is to do the work to go to a new version.
Luckily, it seems like this discussion is, so far, limited to
extensions-for-Firefox and not non-Firefox-applications

Komodo (non-XR application, XUL based):
  - The main thing here is PyXPCOM.  Komodo ships a private copy of
Python (yes!) and hg.m.o-hosted-but-not-m-c code implements a gateway
for that to talk XPCOM, both on the main thread and background threads.
  This is basically unfeasible for js-ctypes.
  - (This also uses a NPAPI plugin extensively; wrapped via a JS
component right now through npruntime.  This will not change due to this
discussion, but might due to the aforementioned block post about killing
NPAPI.)
  - That's about it here - all the fun interaction goes via Python here.
  - Note that we don't build anything special as part of libxul, here,
either.

And for ancient history:
  - Minimizetotray was a binary-XPCOM-component extension.  (Roughly,
Gecko 1.7 to 1.9.0.) No longer maintained, but because of the trilicense
there's a few forks floating about on AMO.  Showed up on the top binary
components list Jorge linked to, even.  If I were to start from scratch
and XPCOM is dead, js-ctypes is my best guess - assuming that can
implement a WndProc.
  - From this experience, I'd say hoping binary XPCOM bits can land on
m-c is overly optimistic.  This code did end up getting reviewed and
landing... on svn.mozilla.org/projects/webrunner/.  Since Firefox itself
didn't need this code, people didn't want to land it in CVS (hey, it's
ancient history).  This matches what happened to hg.m.o/ipccode - sure
it landed, but not in a repo that actually gets shipped.

Other stuff:
  - My experience with js-ctypes has been... poor.  The syntax is
confusing (looks nothing like idiomatic JS or C++), no useful samples
(the unit tests are opaque, and no users in tree I know of), and
debugging is impossible.  It's a black box, much worse than XPCOM
components where C++ compilers usually spit out useful information, you
can go read the headers, and you can find useful debuggers.

--
Mook
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: The purpose of binary components

Nicholas Nethercote
In reply to this post by Asa Dotzler
On Tue, Jan 17, 2012 at 8:28 AM, Asa Dotzler <[hidden email]> wrote:
>
> Preliminary numbers say that binary add-on installs outnumber JS add-on
> installs by about two to one. There are more than 30 add-ons with binary
> components that each have more than a million Firefox users today. There are
> only 13 JS add-ons that have more than a million Firefox users today.

How many of those 30 add-ons with binary components are things like
.Net Assistant and Java Console that users don't know/care about?  And
how many of them are anti-virus type things?

FF8 made the 3rd party add-on situation better, I'm wondering how much
effect it had in practice.

Nick
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: The purpose of binary components

Blair McBride (Unfocused)
In reply to this post by Benjamin Smedberg
On 18/01/2012 6:22 a.m., Benjamin Smedberg wrote:
> I'm looking at
> https://addons.mozilla.org/en-US/firefox/compatibility/11.0?appver=1-11.0&type=binary

Note that AMO's definition of "binary component" is NOT "binary XPCOM
component" - it's *any* binary code (including .sh files, oddly enough).
Bug 718694 will add support for specifically detecting binary XPCOM
components.

Back to the original topic... there are still numerous XPCOM
interfaces/functions that are marked [noscript], so there are still
things that can only be done via binary XPCOM. There aren't nearly as
many as there once was, and it's a fixable problem, but there is work to
be done to make them JS-friendly or provide alternatives.

- Blair

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: The purpose of binary components

Henri Sivonen
In reply to this post by Benjamin Smedberg
On Tue, Jan 17, 2012 at 6:31 PM, Benjamin Smedberg
<[hidden email]> wrote:

>>> D. Stop loading binary components from extensions. Continue exporting the
>>> XPCOM functions so that application authors can continue to use them.
>>> Users
>>> might still be exposed to Oracle-type issues.
>>
>> Who are "application authors" in this case? Authors of XULRunner apps?
>> Don't authors of XULRunner apps pretty much end up compiling XULRunner
>> themselves and, therefore, could bake their code into a static binary
>> of the type mentioned in option E?
>
> Do you have data on this? I don't think that most app authors compile their
> own XULRunner. In some cases they just try to use firefox -app.

I don't have proper data. Just the observation that big apps like
BlueGriffon, Thunderbird and Songbird seem to compile Gecko themselves
and, therefore, could add whatever C++ they'd like at Gecko compile
time. Maybe calling these "XULRunner apps" is technically wrong.

If there is a long tail of XULRunner apps that use Mozilla-supplied
XULRunner binaries, one option would be to lock down Firefox against
dynamically-added vtable-dependent code while leaving XULRunner open
to vtable-dependent extensions.

--
Henri Sivonen
[hidden email]
http://hsivonen.iki.fi/
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: The purpose of binary components

Mike Hommey
In reply to this post by Justin Dolske-2
On Tue, Jan 17, 2012 at 06:56:19PM -0800, Justin Dolske wrote:
> Has anyone done some kind of survey to see what the common use-cases
> are for binary addons? Would it be interesting to look at ways to
> expose non-XPCOM binary hooks, with greater stability levels?
> [Essentially Jetpack for native code, if you will. :)]

I think that's the core of the problem: we don't know what APIs they
need or use. Boris has been saying that stabilizing e.g. nsINode would
stall progress on new web features. But how many binary xpcom components
need to fiddle with nsINode?
We certainly don't need to stabilize all our APIs for binary xpcom
components to be sustainable.

Mike
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: The purpose of binary components

Mike Hommey
In reply to this post by Zack Weinberg-2
On Tue, Jan 17, 2012 at 01:24:59PM -0800, Zack Weinberg wrote:

> On 2012-01-17 7:30 AM, Benjamin Smedberg wrote:
>
> >A. Change nothing, leave the status quo as-is. Extensions which use
> >binary components and are not recompiled every 6 weeks will "break", but
> >they at least have the opportunity to do cool stuff . Users will
> >continue to be vulnerable to Oracle-type crashes.
> >B. Make no technical changes, but disallow binary components in release
> >extensions on AMO. We may allow binary components in beta or
> >experimental addons, with the understanding that these would need to be
> >updated for each release.
> >C. (copied from Asa's post) Only change interfaces every 3rd release (or
> >something like that). This would mean that extensions which use C++
> >components would need to compile less frequently. Users would still be
> >exposed to Oracle-type issues, but less frequently.
> >D. Stop loading binary components from extensions. Continue exporting
> >the XPCOM functions so that application authors can continue to use
> >them. Users might still be exposed to Oracle-type issues.
> >E. Stop loading binary components completely: compile Firefox as a
> >static binary and stop exporting the XPCOM functions completely.
>
> I think option B is an excellent one for the near term, but that in
> the long run we want option D or E, and that concurrent with the
> change to AMO, we should announce that binary components are
> deprecated and will stop working at a stated time in the future (one
> or two years seems reasonable to me).
>
> To inform the choice between D and E, could authors of applications
> outside m-c please chime in and tell us what they still need binary
> components for?  It seems *highly* desirable to me to get to a point
> where Thunderbird, Seamonkey, etc don't need any binary components
> beyond what's already in libxul.

Do you suggest that things like the ldap client implementation would
need to be in m-c's libxul?

Mike
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: The purpose of binary components

Boris Zbarsky
In reply to this post by Justin Dolske-2
On 1/18/12 3:06 AM, Mike Hommey wrote:
> I think that's the core of the problem: we don't know what APIs they
> need or use. Boris has been saying that stabilizing e.g. nsINode would
> stall progress on new web features. But how many binary xpcom components
> need to fiddle with nsINode?

So...  I probably have a biased sample, but about 2/3 of the ones I've
ended up having to deal with (because they were causing breakage of
various sorts) were touching things like nsINode/nsIContent/nsIDocument.

-Boris
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: The purpose of binary components

Mike Hommey
On Wed, Jan 18, 2012 at 03:22:54AM -0500, Boris Zbarsky wrote:

> On 1/18/12 3:06 AM, Mike Hommey wrote:
> >I think that's the core of the problem: we don't know what APIs they
> >need or use. Boris has been saying that stabilizing e.g. nsINode would
> >stall progress on new web features. But how many binary xpcom components
> >need to fiddle with nsINode?
>
> So...  I probably have a biased sample, but about 2/3 of the ones
> I've ended up having to deal with (because they were causing
> breakage of various sorts) were touching things like
> nsINode/nsIContent/nsIDocument.

You probably have a biased sample, but then, that raises the corollary
question: what do they need to touch them for? More specifically, why do
they need to do it from C++ instead of javascript?

Mike
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: The purpose of binary components

Boris Zbarsky
In reply to this post by Boris Zbarsky
On 1/18/12 3:39 AM, Mike Hommey wrote:

> On Wed, Jan 18, 2012 at 03:22:54AM -0500, Boris Zbarsky wrote:
>> On 1/18/12 3:06 AM, Mike Hommey wrote:
>>> I think that's the core of the problem: we don't know what APIs they
>>> need or use. Boris has been saying that stabilizing e.g. nsINode would
>>> stall progress on new web features. But how many binary xpcom components
>>> need to fiddle with nsINode?
>>
>> So...  I probably have a biased sample, but about 2/3 of the ones
>> I've ended up having to deal with (because they were causing
>> breakage of various sorts) were touching things like
>> nsINode/nsIContent/nsIDocument.
>
> You probably have a biased sample, but then, that raises the corollary
> question: what do they need to touch them for? More specifically, why do
> they need to do it from C++ instead of javascript?

For the most part, for absolutely no good reason I could see.

There was one sorta-exception: an addon that switched from DOM mutation
observers to nsIMutationObserver to not cause a large performance drag.
  But imo that was the wrong fix for their problem; the right fix would
have been removing the mutation observer when it got triggered, then
readding after they had rescanned the entire page off a timeout (which
they continued to do).  I sort of failed to convince them of that.

-Boris
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: The purpose of binary components

Gervase Markham
In reply to this post by Philip Chee
On 18/01/12 04:24, Philip Chee wrote:
> Well here's the problem. Enigmail tried multiple times to get their IPC
> code into core but was rejected each time. Eventually they were
> grudgingly allowed to put their code into a separate repository
> somewhere in hg.mozilla.org where nobody can find it.

I remember that; it was a long time ago. Without wanting to revisit the
rights or wrongs of the decision back then, perhaps it's worth making a
few brief enquiries of the relevant module owners as to whether the
answer would still be the same?

Gerv


_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: The purpose of binary components

Pavol Mišík
In reply to this post by Joshua Cranmer-2
On 18. 1. 2012 0:39, Joshua Cranmer wrote:

> On 1/17/2012 3:50 PM, Zack Weinberg wrote:
>> I know that *presently* there's a lot of C++ code in the comm-central
>> version of libxul that isn't in the m-c version, but I don't know what
>> it's all for, and I could imagine a great deal of it being either
>> reimplementable in JS without much trouble, or reasonable to add to
>> the m-c codebase.
>
> As a prelude, I should point out that the mailnews code is often old and
> crufty enough to make reimplementation a major task, fraught with
> complications, even without the burden of translating to a more limiting
> API boundary. I can explain the major problems of rewriting most/all of
> comm-central in JS:
>
> 1. Charsets. This means a lot of strings will not pass through xpconnect
> happily; additionally, a fair amount of the m-c charset/MIME code we
> need to use is either not available via XPIDL or it is marked as
> [noscript].
> 2. Threads. Some portions of our codebase--particularly IMAP and
> import--need to access things from different threads. It is absolutely
> impossible to use a JS-implemented XPCOM component from another thread,
> which means that it is almost inevitable that the core objects would
> have to be implemented in C++. Additionally, IMAP relies on using socket
> communications on a different thread, which would mean the same
> limitation would still apply even if the core objects were implemented
> without using XPCOM at all [which is highly unlikely].
> 3. RDF. It is impossible to implement RDF in JavaScript (and I have
> tried before). While we do have an ongoing project to remove RDF from
> mailnews, it has effectively stalled (the main pusher of the effort
> appears to have retired from coding Thunderbird), and the last features
> are where code is most likely to produce pernicious regressions.
> 4. System integration. The WABI address books, at the very least, uses a
> MS COM implementation, which makes access via jsctypes all but
> impossible. Now, this could probably be implemented via a jsctypes
> thunk, but I'm just pointing out that it is almost certainly infeasible
> to have all of comm-central be in JS.
> 5. I/O. Some of the I/O layers we use are again unusable from JS (e.g.,
> nsIAuthModule). Admittedly, this is mostly fallout from 1 and 2, but I
> still want to point out that it is a limit on what is possible.
> 6. All-or-nothing migration. Much of the current codebase is effectively
> impossible to partially migrate to JS, due to several factors. There are
> probably some individual files in base/src that could be independently
> migrated, but the MIME, filter, composition, import, and database
> libraries probably all have to be migrated as full libraries. Address
> book might be partially migratible with some work, but the rest of
> mailnews would pretty much have to swap all at the same time, or at
> least in two units (the protocol communication and the actual base
> object instances), due to liberal use of inheritance.
>
> In short: it is, IMHO, highly unlikely that comm-central could be
> rewritten in JS without several man-months of concentrated effort and
> significant expansion of available APIs (I, for one, would love to see
> some sort of "binary string" feature in xpconnect/JS/ishy-things, as
> that pretty much eliminates my major concern for rewriting libmime in
> JS). Unless, of course, you want to add all/most of comm-central back
> into mozilla-central :-).

Company I work for has written extension for Thunderbird to provide
tight integration of our product. Joshua mentioned there are many
limitations, so we did it in xpcom. We've successfully supported Tb2-Tb5
within *one* dll.

We created thin layer that wrapped all interfaces we need. At startup,
our extension verified if it is compatible with all necessary
interfaces.  We could get necessary information from nsIInterfaceInfo.
We relied that xpcom\reflect doesn’t change interfaces. We did this
verification for two reasons to get necessary information to dynamically
build wrappers and to protect impatient users that tried to change
em:maxVersion in install.rdf

In Tb7/FF7 was changed nsIInterfaceInfo. In order to avoid potential
users problems in future with our integration we stopped supporting
newer versions because there is no guarantee that xpcom\reflect
interfaces will not change again. :-(
We also support other products e.g Microsoft. So our users have option
to use them.

If there will be frozen interface or this interface will be changed more
carefully(adding new methods at the end of interface - not in the
middle), we could get information about interfaces (xpcom\reflect) in
FF/TB/gecko safely we could reconsider our decision.
Probably there are also other companies that do their extension this way.

pm-
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: The purpose of binary components

R Kent James
In reply to this post by Gervase Markham
On 1/18/2012 4:48 AM, Gervase Markham wrote:

> On 18/01/12 04:24, Philip Chee wrote:
>> Well here's the problem. Enigmail tried multiple times to get their IPC
>> code into core but was rejected each time. Eventually they were
>> grudgingly allowed to put their code into a separate repository
>> somewhere in hg.mozilla.org where nobody can find it.
>
> I remember that; it was a long time ago. Without wanting to revisit the
> rights or wrongs of the decision back then, perhaps it's worth making a
> few brief enquiries of the relevant module owners as to whether the
> answer would still be the same?
>
> Gerv
>
>

I think that for the purposes of this discussion, the point here is that
it is not a trivial task to get a major binary interface added to the
core code, even for an extension like Enigmail that is viewed as
critical to the Thunderbird ecosystem. And that is the past. If you
remove binary support, then the future Enigmails can never even happen,
particularly since in their early experimental phase there may not be
the recognized need yet from core developers.

It sounds great to say that we will fix the core code to remove these
deficiencies - but we are also talking about core code here that still
uses RSS and Mork. The resources are just not there for the rewrite that
you need to remove all possible requirements for binary addons.

rkent
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: The purpose of binary components

Benjamin Smedberg
In reply to this post by Philip Chee
On 1/17/2012 11:24 PM, Philip Chee wrote:
> Well here's the problem. Enigmail tried multiple times to get their IPC
> code into core but was rejected each time. Eventually they were
> grudgingly allowed to put their code into a separate repository
> somewhere in hg.mozilla.org where nobody can find it.
Patrick and I mutually decided to make the code separate "for now",
since the Mozilla core code didn't need it. We can revisit that decision
as necessary.

--BDS

_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: The purpose of binary components

Robert Kaiser
In reply to this post by Asa Dotzler
Asa Dotzler schrieb:
> If we do B, we do nothing about most add-ons with XPCOM components. Most
> are hosted outside of AMO already. If we do B, the rest will probably
> just move outside of AMO where we have even less visibility.

I fear that as well. Would give us no win but a lot more pain. :(

> If we do D and E, we have to be prepared for the authors of all the
> "security"-related add-ons that use XPCOM components to message to our
> users that they will be unsafe and should not upgrade to newer versions
> of Firefox that block them. We've already seen some of that from AV
> vendors in the past.

Actually, a number of AV vendors (if not all of them) use Windows
facilities to hook binaries into our processes and don't use XPCOM at
all, from what I can tell. This makes things just crash because there
are no XPCOM version checks at all. Oh beauty.

I fear a lot that people now using XPCOM might then wander off to using
those hooking facilities as well, which makes breakage and pain even
more likely ([startup] crashes are worse than incompatibility warnings)
and also makes them drop any support for non-Windows if they previously
might have had that. Not sure that's what we want to propagate.

Robert Kaiser
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: The purpose of binary components

Simon Paquet-2
In reply to this post by Benjamin Smedberg
Benjamin Smedberg wrote on 17. Jan 2012:

> [...]
>
> There are few possible alternatives for what we could do to make
> using binary components easier or harder:
>
> [...]
>
> B. Make no technical changes, but disallow binary components in
> release extensions on AMO. We may allow binary components in beta
> or experimental addons, with the understanding that these would
> need to be updated for each release.
>
> [...]
>
> == bsmedberg's Opinion ==
>
> I see binary components as a useful tool in very specific
> circumstances. Primarily, they are a good way to prototype and
> experiment with new features which require mucking about with
> Mozilla internals. But I tend to think that we should discourage
> their use in any production environment, including perhaps
> disallowing them on AMO. I tend to think we should consider
> option "B".

Just as a FYI:
Lightning, Thunderbird most popular addon (nearly 20% of all TB users
use Lightning) with over 1.4m daily users on weekdays, uses binary
components extensively.

So going with option "B" would hurt the Thunderbird ecosystem a lot.

Lightning uses binary components to embed an external library
(libical) that is used to work with .ics files, which allow Lightning
users to use non-local calendars hosted on web servers, ftp servers,
CalDAV servers, Google Calendar, etc.

It may be that libical is used for other places as well. I'll wait
for the lead developer, Philipp Kewisch (Fallen on IRC) to chime in
here.

From what I know Philipp is already working on moving the Lightning
codebase to js-ctypes. The main problem that he grapples with is
AFAIK the memory management, particularly how to free() ctypes
components attached to js objects (there are no destructors!).

--
Simon
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: The purpose of binary components

Steve Wendt
In reply to this post by Philip Chee
On 1/17/2012 8:24 PM, Philip Chee wrote:

> Well here's the problem. Enigmail tried multiple times to get their IPC
> code into core but was rejected each time. Eventually they were
> grudgingly allowed to put their code into a separate repository
> somewhere in hg.mozilla.org where nobody can find it.

Yes, too often there's the attitude that "Firefox doesn't need that."
The same attitude has been there somewhat even for Firefox, when it is
for non-tier 1 platforms.  The argument tends to be "we don't want our
code polluted with stuff we don't care about."  I can certainly
understand that position, but things like this are the bad side of it.
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: The purpose of binary components

Wes Garland
In reply to this post by Simon Paquet-2
>
> The main problem that he grapples with is
> AFAIK the memory management, particularly how to free() ctypes
> components attached to js objects (there are no destructors!).
>

I wrote an FFI for GPSEE (a FOSS SpiderMonkey embedding) before js-ctypes
was around and bumped into this same issue.  We solved it by creating a
sort of C function closure by adding a boxing object to the return value of
FFI calls and using it to hook the finalization stage of the garbage
collector.   You can do this for pretty much any function where you can
compute the cleanup before it needs to happen -- the classic examples being
malloc/free - so, we can write real-world code like this:

function mmap(filename)
{
  var fd = _open(filename, ffi.std.O_RDONLY, parseInt('0666'));
  if (fd == -1)
    throw(new Error(perror("Cannot open file " + filename)));

  try
  {
    var sb = new ffi.MutableStruct("struct stat");
    if (_fstat(fd, sb) != 0)
      throw(new Error(perror("Cannot stat file " + filename)));

    var mem =* _mmap.call*(null, sb.st_size, ffi.std.PROT_READ,
ffi.std.MAP_PRIVATE, fd, 0);
    if (mem == ffi.Memory(-1))
      throw(new Error(perror("Cannot mmap file " + filename)));

    mem.length = sb.st_size;
*    mem.finalizeWith(_munmap, mem, sb.st_size);*
    return mem;
  }
  finally
  {
    _close(fd);
  }
}

So, what happens in this case is that when the return value (mem) is
collected, munmap is invoked on the pointer and mapping size boxed by the
return value of _mmap.call().  The "C function closure" is created during
the finalizeWith() method, because it's too late to do it when the garbage
collector is running -- the private slot of the mem object will be
annotated with something like 'call function 0x1234 with argv [0xc0ffee,
800]'.

This technique could probably easily be adapted for use by js-ctypes if the
demand were there -- although it should be understood that this technique
also represents a pretty serious footgun.

Wes

--
Wesley W. Garland
Director, Product Development
PageMail, Inc.
+1 613 542 2787 x 102
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: The purpose of binary components

Bobby Holley-2
We've considered putting this kind of thing into js-ctypes. Unfortunately,
since the GC doesn't guarantee any kind of finalization ordering, the
prospect of bugs resulting from destructors being called in unexpected
orders seemed quite high. If anyone has any clever ideas of how to do this
more effectively please let me know.

bholley

On Wed, Jan 18, 2012 at 11:32 AM, Wes Garland <[hidden email]> wrote:

> >
> > The main problem that he grapples with is
> > AFAIK the memory management, particularly how to free() ctypes
> > components attached to js objects (there are no destructors!).
> >
>
> I wrote an FFI for GPSEE (a FOSS SpiderMonkey embedding) before js-ctypes
> was around and bumped into this same issue.  We solved it by creating a
> sort of C function closure by adding a boxing object to the return value of
> FFI calls and using it to hook the finalization stage of the garbage
> collector.   You can do this for pretty much any function where you can
> compute the cleanup before it needs to happen -- the classic examples being
> malloc/free - so, we can write real-world code like this:
>
> function mmap(filename)
> {
>  var fd = _open(filename, ffi.std.O_RDONLY, parseInt('0666'));
>  if (fd == -1)
>    throw(new Error(perror("Cannot open file " + filename)));
>
>  try
>  {
>    var sb = new ffi.MutableStruct("struct stat");
>    if (_fstat(fd, sb) != 0)
>      throw(new Error(perror("Cannot stat file " + filename)));
>
>    var mem =* _mmap.call*(null, sb.st_size, ffi.std.PROT_READ,
> ffi.std.MAP_PRIVATE, fd, 0);
>    if (mem == ffi.Memory(-1))
>      throw(new Error(perror("Cannot mmap file " + filename)));
>
>    mem.length = sb.st_size;
> *    mem.finalizeWith(_munmap, mem, sb.st_size);*
>    return mem;
>  }
>  finally
>  {
>    _close(fd);
>  }
> }
>
> So, what happens in this case is that when the return value (mem) is
> collected, munmap is invoked on the pointer and mapping size boxed by the
> return value of _mmap.call().  The "C function closure" is created during
> the finalizeWith() method, because it's too late to do it when the garbage
> collector is running -- the private slot of the mem object will be
> annotated with something like 'call function 0x1234 with argv [0xc0ffee,
> 800]'.
>
> This technique could probably easily be adapted for use by js-ctypes if the
> demand were there -- although it should be understood that this technique
> also represents a pretty serious footgun.
>
> Wes
>
> --
> Wesley W. Garland
> Director, Product Development
> PageMail, Inc.
> +1 613 542 2787 x 102
> _______________________________________________
> dev-planning mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-planning
>
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
Reply | Threaded
Open this post in threaded view
|

Re: The purpose of binary components

Wes Garland
On 18 January 2012 14:49, Bobby Holley <[hidden email]> wrote:

> We've considered putting this kind of thing into js-ctypes. Unfortunately,
> since the GC doesn't guarantee any kind of finalization ordering, the
> prospect of bugs resulting from destructors being called in unexpected
> orders seemed quite high. If anyone has any clever ideas of how to do this
> more effectively please let me know.
>

AFAICT, there's not a whole lot of "more cleverly" you can do.  You need to
select basic interfaces like fopen/fclose, open/close, mmap/munmap,
malloc/free, nothing which interacts with other objects, nothing which can
call back into JS.

Here is the key to making this technique workable: the FFI marshalling must
happen when the finalizer is declared, not later.  This means that all
arguments are immediately turned into their C equivalents, ffi_prep_cif()
is invoked, etc, while the arguments are still guaranteed to be reachable.

You cannot hold on to references to your C data in places that are not
visible to the JS GC, but that goes for any embedding scenario where the JS
GC is allowed to manage non-JS storage, not just this scenario.

Wes

--
Wesley W. Garland
Director, Product Development
PageMail, Inc.
+1 613 542 2787 x 102
_______________________________________________
dev-planning mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-planning
12345