SpiderMonkey: Issues for embedders

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

SpiderMonkey: Issues for embedders

Yves
Dear SpiderMonkey developers

I'm Yves, programmer from Wildfire Games, a group of volunteer developers
who are working on the open source game 0 A.D. Many of you have already
met me on IRC or we worked together on bugs.

We are using SpiderMonkey as our Javascript engine for gameplay, UI, AI,
random maps and other scripts in our game.

In the past months I've been working on upgrading from v1.8.5 to a more
recent version of SpiderMonkey. This has emerged into a major task of
hundreds of man-hours. In addition, some severe issues concerning the
future perspective of SpiderMonkey as our scripting engine have been
revealed.

I'm opening this ticket to ask for some input on these issues to see if
we can find a solution for the future together.

Our main problem with Spidermonkey is the lack of API stability. Adding
some flags or additional arguments here and there aren't a problem, but
changes in fundamental constraints are the issue we are facing.

A few examples:
---------------
As you can see in our SpiderMonkey upgrade ticket (http://
trac.wildfiregames.com/ticket/1886), most of changes are related to the
change in the relation between compartments and global objects.

GUI pages need their own global scope and in the past we have used one
compartment with multiple global objects. This allowed sharing of data
between GUI pages and gave them their own global object too.

Now we have a 1 to 1 relation between compartments and global objects and
we can't do that anymore.

The same problem applies to AI players. Each one needs its own scope but
they still need some shared data. Before the upgrade we could use one
compartment and multiple globals and the sharing was possible without
much of a hassle.

Another problem is multi-threading. SpiderMonkey dropped support for
multi-threading in one Runtime. This also has a lot of impact on how data
can be shared efficiently between threads or between compartments within
the same thread. One thing is getting it to work "somehow" but if
performance is a concern (it is), it gets even more difficult. Cloning
and wrapping both have their downsides performance-wise and they even
change the behaviour of our scripts sometimes. When we pass data as a
clone, this creates new objects while wrapping uses the existing
references. Wrapping and cloning also has impact on JIT compiling and JIT
performance. Currently it looks like Ion Monkey works poorly with
wrappers and with clones and instead of bringing a performance advantage,
the upgrade makes it even worse.

At the moment there is work being done on eliminating the "Contexts" or
at least changing what they are completely and moving a lot to the
runtime level. In addition there are a lot of discussions about exact
stack rooting and if this will suddenly become the only valid rooting
approach, we have another huge task.

It looks like this kind of "design breaking changes" are not going to
become less in the near future.

A future-proof design:
----------------------
For issues like the AI interface we need a stable and future-proof
design. With our limited resources it's not acceptable for us to spend
months of work each time a constraint in SpiderMonkey changes to redesign
something that complex as an AI interface and its multi-threading
implementation every 6/12 months.

We need to know how multi-threading is going to work in the next years.
We also need to know how data can be shared between threads, compartments
and globals and what impact it has on performance and we need to be sure
this doesn't change completely with the next SpiderMonkey version.

Our requirements and "Spidermonkey as a general purpose Javascript Engine"
---------------------------------------------------------------------------
These are basically our requirements for a scripting engine. I know that
your are well aware of other projects using SpiderMonkey (https://
developer.mozilla.org/en-US/docs/SpiderMonkey/FOSS) and also of the
benefits of getting code tested in different environments and the
feedback coming back from open source developers. Our work on the upgrade
for 0 A.D. for example revealed some SpiderMonkey issues such as the ones
here (https://bugzilla.mozilla.org/show_bug.cgi?id=897962) and helped
make the engine better.

We have noticed the increasing efforts of Mozilla to provide bundled
libraries for ESR releases and appreciate this. However, if encouraging
the use of SpiderMonkey as a scripting API in the open source community
and other projects than Firefox is the strategy, that alone is not
sufficient.

If Mozilla doesn't focus more on a relatively stable API and predictable
development, SpiderMonkey will not become an attractive general purpose
Javascript engine and will instead be mostly limited to the use in
Firefox.

Our alternatives:
-----------------
At the current point it's very difficult for us to switch to another
scripting engine or even another scripting language. Also I personally
enjoy the open development and appreciated all the help we got in the
past which is certainly not something we can take for granted. This would
really be the last way out if all other approaches fail.

But still, we have to do something to solve this problem and we will
spare no efforts if there's a sufficiently promising solution in sight.

I hope we can find a solution for this problem. Do you have any inputs
how we could make it easier to update to future versions of Spidermonkey
and about Mozilla's strategy for Spidermonkey as a general purpose
library? Would it be possible for you to somehow reduce the amount and
frequency of such severe API changes?

If you have specific inputs on our problem with the data sharing and
threading design, you can also contact me on IRC, send a mail or join the
discussion on our forums (http://www.wildfiregames.com/forum/index.php?
showtopic=17289).

Kind regards and thanks in advance for all input,
Yves Gwerder
_______________________________________________
dev-tech-js-engine mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-tech-js-engine
Reply | Threaded
Open this post in threaded view
|

Re: SpiderMonkey: Issues for embedders

Claudiu Saftoiu
Disclaimer: I'm not affiliated with Spidermonkey in any way.

Why do you have to upgrade your version of spidermonkey? This is why I personally never update any library I'm using unless it's absolutely necessary. If starting a new project then you can start with whatever the latest version of spidermonkey is at that time but once begun, why the need to change it?

Regards,
Claudiu



> On Dec 7, 2013, at 5:23 PM, Yves <[hidden email]> wrote:
>
> Dear SpiderMonkey developers
>
> I'm Yves, programmer from Wildfire Games, a group of volunteer developers
> who are working on the open source game 0 A.D. Many of you have already
> met me on IRC or we worked together on bugs.
>
> We are using SpiderMonkey as our Javascript engine for gameplay, UI, AI,
> random maps and other scripts in our game.
>
> In the past months I've been working on upgrading from v1.8.5 to a more
> recent version of SpiderMonkey. This has emerged into a major task of
> hundreds of man-hours. In addition, some severe issues concerning the
> future perspective of SpiderMonkey as our scripting engine have been
> revealed.
>
> I'm opening this ticket to ask for some input on these issues to see if
> we can find a solution for the future together.
>
> Our main problem with Spidermonkey is the lack of API stability. Adding
> some flags or additional arguments here and there aren't a problem, but
> changes in fundamental constraints are the issue we are facing.
>
> A few examples:
> ---------------
> As you can see in our SpiderMonkey upgrade ticket (http://
> trac.wildfiregames.com/ticket/1886), most of changes are related to the
> change in the relation between compartments and global objects.
>
> GUI pages need their own global scope and in the past we have used one
> compartment with multiple global objects. This allowed sharing of data
> between GUI pages and gave them their own global object too.
>
> Now we have a 1 to 1 relation between compartments and global objects and
> we can't do that anymore.
>
> The same problem applies to AI players. Each one needs its own scope but
> they still need some shared data. Before the upgrade we could use one
> compartment and multiple globals and the sharing was possible without
> much of a hassle.
>
> Another problem is multi-threading. SpiderMonkey dropped support for
> multi-threading in one Runtime. This also has a lot of impact on how data
> can be shared efficiently between threads or between compartments within
> the same thread. One thing is getting it to work "somehow" but if
> performance is a concern (it is), it gets even more difficult. Cloning
> and wrapping both have their downsides performance-wise and they even
> change the behaviour of our scripts sometimes. When we pass data as a
> clone, this creates new objects while wrapping uses the existing
> references. Wrapping and cloning also has impact on JIT compiling and JIT
> performance. Currently it looks like Ion Monkey works poorly with
> wrappers and with clones and instead of bringing a performance advantage,
> the upgrade makes it even worse.
>
> At the moment there is work being done on eliminating the "Contexts" or
> at least changing what they are completely and moving a lot to the
> runtime level. In addition there are a lot of discussions about exact
> stack rooting and if this will suddenly become the only valid rooting
> approach, we have another huge task.
>
> It looks like this kind of "design breaking changes" are not going to
> become less in the near future.
>
> A future-proof design:
> ----------------------
> For issues like the AI interface we need a stable and future-proof
> design. With our limited resources it's not acceptable for us to spend
> months of work each time a constraint in SpiderMonkey changes to redesign
> something that complex as an AI interface and its multi-threading
> implementation every 6/12 months.
>
> We need to know how multi-threading is going to work in the next years.
> We also need to know how data can be shared between threads, compartments
> and globals and what impact it has on performance and we need to be sure
> this doesn't change completely with the next SpiderMonkey version.
>
> Our requirements and "Spidermonkey as a general purpose Javascript Engine"
> ---------------------------------------------------------------------------
> These are basically our requirements for a scripting engine. I know that
> your are well aware of other projects using SpiderMonkey (https://
> developer.mozilla.org/en-US/docs/SpiderMonkey/FOSS) and also of the
> benefits of getting code tested in different environments and the
> feedback coming back from open source developers. Our work on the upgrade
> for 0 A.D. for example revealed some SpiderMonkey issues such as the ones
> here (https://bugzilla.mozilla.org/show_bug.cgi?id=897962) and helped
> make the engine better.
>
> We have noticed the increasing efforts of Mozilla to provide bundled
> libraries for ESR releases and appreciate this. However, if encouraging
> the use of SpiderMonkey as a scripting API in the open source community
> and other projects than Firefox is the strategy, that alone is not
> sufficient.
>
> If Mozilla doesn't focus more on a relatively stable API and predictable
> development, SpiderMonkey will not become an attractive general purpose
> Javascript engine and will instead be mostly limited to the use in
> Firefox.
>
> Our alternatives:
> -----------------
> At the current point it's very difficult for us to switch to another
> scripting engine or even another scripting language. Also I personally
> enjoy the open development and appreciated all the help we got in the
> past which is certainly not something we can take for granted. This would
> really be the last way out if all other approaches fail.
>
> But still, we have to do something to solve this problem and we will
> spare no efforts if there's a sufficiently promising solution in sight.
>
> I hope we can find a solution for this problem. Do you have any inputs
> how we could make it easier to update to future versions of Spidermonkey
> and about Mozilla's strategy for Spidermonkey as a general purpose
> library? Would it be possible for you to somehow reduce the amount and
> frequency of such severe API changes?
>
> If you have specific inputs on our problem with the data sharing and
> threading design, you can also contact me on IRC, send a mail or join the
> discussion on our forums (http://www.wildfiregames.com/forum/index.php?
> showtopic=17289).
>
> Kind regards and thanks in advance for all input,
> Yves Gwerder
> _______________________________________________
> dev-tech-js-engine mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-tech-js-engine
_______________________________________________
dev-tech-js-engine mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-tech-js-engine
Reply | Threaded
Open this post in threaded view
|

Re: SpiderMonkey: Issues for embedders

Boris Zbarsky
In reply to this post by Yves
On 12/7/13 5:23 PM, Yves wrote:
> Our main problem with Spidermonkey is the lack of API stability.

I can see that, for sure, over the last several years.  Unfortunately,
the API evolved a huge amount because the requirements changed
radically.  :(

> Now we have a 1 to 1 relation between compartments and global objects and
> we can't do that anymore.

Out of curiosity, what sort of data are you moving back and forth?  For
strings, you can put it all into a single zone.  For generic object
graphs, you have a problem...  What other sorts of things do you need
sharing for?

There has been some recent action towards making cross-compartment
proxies faster (e.g. Eric added some ICs for proxies in general).  But
they're obviously still much slower than normal object accesses.  I
_think_ we can make cross-compartment typed array access pretty fast
too; not sure how much that helps you.

I assume needing multiple globals here is a hard requirement?

> Another problem is multi-threading. SpiderMonkey dropped support for
> multi-threading in one Runtime.

Note that this never really worked right...

> In addition there are a lot of discussions about exact
> stack rooting and if this will suddenly become the only valid rooting
> approach, we have another huge task.

It will certainly be the only valid rooting approach when generational
GC is enabled.  I don't know whether we plan to support a mode when it's
not enabled, but I think you should assume not.

> It looks like this kind of "design breaking changes" are not going to
> become less in the near future.

I think they actually should.  We've totally changed how we do GC and
security.  There are few other cross-cutting things like that to change...

I can't speak to most of your questions about future plans, though I
hope the folks more involved than I with SpiderMonkey planning can.  But
I'm _very_ interested in helping solve your performance issues.

-Boris
_______________________________________________
dev-tech-js-engine mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-tech-js-engine
Reply | Threaded
Open this post in threaded view
|

Re: SpiderMonkey: Issues for embedders

Yves
In reply to this post by Yves
Am Sat, 07 Dec 2013 17:44:02 -0500 schrieb Claudiu Saftoiu:

> Disclaimer: I'm not affiliated with Spidermonkey in any way.
>
> Why do you have to upgrade your version of spidermonkey? This is why I
> personally never update any library I'm using unless it's absolutely
> necessary. If starting a new project then you can start with whatever
> the latest version of spidermonkey is at that time but once begun, why
> the need to change it?
>
> Regards,
> Claudiu
>

There are a lot of reasons to upgrade the library.
- Because there are bugs in the 1.8.5 version that will never get fixed
in this version
- Because there are security leaks in the old version that will never get
fixed in this version
- Because Linux distributions will not package the old version forever
- Because we need new features from the newer versions
- Because it's much less likely to get help with your embedding problems
if you use a completely outdated version
- Because the Spidermonkey development doesn't benefit from us if we
don't provide the feedback from using (testing) the current version
- etc...

Regards,
Yves
_______________________________________________
dev-tech-js-engine mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-tech-js-engine
Reply | Threaded
Open this post in threaded view
|

Re: SpiderMonkey: Issues for embedders

Yves
In reply to this post by Boris Zbarsky
I'll try to answer the questions a bit better than I could yesterday on
IRC.

First, I would also like to ask if it would be possible to have a central
Wiki page or something informing about what parts of the engine will
change soon or are currently being changed. Something like release notes,
but looking into the future instead of the past.
It doesn't even have to be very detailed, maybe something like this(just
examples, I don't know if they are true):

Multi-threading: Subject to change completely
Garbage collection: Dropping support for conservative stack rooting
in ESR XY
JS Debugger API: The API is relatively stable. No significant
changes are planned.

Currently embedders are a bit like a flag in the wind without much
possibilites to predict on which parts of the API they can rely and which
parts they should better avoid using or be prepared if they change.


> Out of curiosity, what sort of data are you moving back and forth?  For
> strings, you can put it all into a single zone.  For generic object
> graphs, you have a problem...

We have the shared AI scripts that update common data between AI players.
It's data such as terrain passability maps, territory maps and entity
(units, buildings etc.) data. It's a lot of data and I guess the
performance for accessing it can make a big difference.
Basically we update this data once per simulation turn (around 5 times
per second) in the shared AI script and then access it from each AI
player script through wrappers.

>
> I assume needing multiple globals here is a hard requirement?
>

I will check if it's possible to move this all into one global without
rewriting the whole AI. I didn't try this before because I thought it's
not compatible with multi-threading. Now my impression is that we aren't
going to have a multi-threaded AI in the next year anyway because
SpiderMonkey doesn't really support it yet.
Any details about how the future implementation of multi-threading in
SpiderMonkey will work would help. I don't have a good feeling when we
have to work on an AI interface that's supposed to support multi-
threading in the future but we don't know what requirements and
constraints will have to be fulfilled to implement threading with
SpiderMonkey.
I've found this article today but I haven had the time to read it though
completely yet.
http://billmccloskey.wordpress.com/2013/12/05/multiprocess-firefox/

Maybe putting all the AI players in one compartment isn't too bad.
Improving AI performance is better than distributing the load across
processor cores. I think this design should still allow us to move the AI
as a whole to a separate thread, but probably won't allow moving each AI
player to its own thread.


>> It looks like this kind of "design breaking changes" are not going to
>> become less in the near future.
>
> I think they actually should.  We've totally changed how we do GC and
> security.  There are few other cross-cutting things like that to
> change...
Well, we have the multi-threading, the exact stack rooting, changes
related to contexts...
That should be enough to keep us busy.


> I'm _very_ interested in helping solve your performance issues.
Thanks, I'll keep you updated about my tests for using one global for the
whole AI and will probably ask some questions on IRC.

_______________________________________________
dev-tech-js-engine mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-tech-js-engine
Reply | Threaded
Open this post in threaded view
|

Re: SpiderMonkey: Issues for embedders

Claudiu Saftoiu
In reply to this post by Yves


> On Dec 7, 2013, at 7:25 PM, Yves <[hidden email]> wrote:
>
> Am Sat, 07 Dec 2013 17:44:02 -0500 schrieb Claudiu Saftoiu:
>
>> Disclaimer: I'm not affiliated with Spidermonkey in any way.
>>
>> Why do you have to upgrade your version of spidermonkey? This is why I
>> personally never update any library I'm using unless it's absolutely
>> necessary. If starting a new project then you can start with whatever
>> the latest version of spidermonkey is at that time but once begun, why
>> the need to change it?
>>
>> Regards,
>> Claudiu
>
> There are a lot of reasons to upgrade the library.
> - Because there are bugs in the 1.8.5 version that will never get fixed
> in this version
True. What I do in that case is fork the library and use my fork. Easier to fix one but than spend hundreds of man hours upgrading .. besides new version will probably have other bugs anyway.
> - Because there are security leaks in the old version that will never get
> fixed in this version
Same as above
> - Because Linux distributions will not package the old version forever
True. You don't ship the library with the rest of the program?
> - Because we need new features from the newer versions
Ah that's actually a good reason. Then question is whether it's worth trade off of upgrading vs working around it. Conversely it also sounds like you need old features from the older versions.
> - Because it's much less likely to get help with your embedding problems
> if you use a completely outdated version
That is true
> - Because the Spidermonkey development doesn't benefit from us if we
> don't provide the feedback from using (testing) the current version
Also true though this is more an altruistic motive than a practical one, but it's one I share
> - etc...

Regards
- Claudiu
>
> Regards,
> Yves
> _______________________________________________
> dev-tech-js-engine mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-tech-js-engine
_______________________________________________
dev-tech-js-engine mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-tech-js-engine
Reply | Threaded
Open this post in threaded view
|

Re: SpiderMonkey: Issues for embedders

Anthony Catel-4
In reply to this post by Yves
Hey all,

I understand the fact that mozilla have to move fast on spidermonkey
since javascript speed is the "sinews of war" of vendors since a few
years now.

I personally have setup something like a "continuous integration
branch" just for porting new changes on a daily basis and merge to my
main branch when the engine reach a "big goal".

The big problem IMHO is the communication.

One thing that could be less frustrating would be to get into the habit
of talking on this list instead of IRC so that everybody can benefit
from the various exchanges about JSAPI and have access to previous
discussions.

Anthony Catel

Le 2013-12-08 14:59, Yves a écrit :

> I'll try to answer the questions a bit better than I could yesterday
> on
> IRC.
>
> First, I would also like to ask if it would be possible to have a
> central
> Wiki page or something informing about what parts of the engine will
> change soon or are currently being changed. Something like release
> notes,
> but looking into the future instead of the past.
> It doesn't even have to be very detailed, maybe something like
> this(just
> examples, I don't know if they are true):
>
> Multi-threading: Subject to change completely
> Garbage collection: Dropping support for conservative stack rooting
> in ESR XY
> JS Debugger API: The API is relatively stable. No significant
> changes are planned.
>
> Currently embedders are a bit like a flag in the wind without much
> possibilites to predict on which parts of the API they can rely and
> which
> parts they should better avoid using or be prepared if they change.
>
>
>> Out of curiosity, what sort of data are you moving back and forth?  
>> For
>> strings, you can put it all into a single zone.  For generic object
>> graphs, you have a problem...
>
> We have the shared AI scripts that update common data between AI
> players.
> It's data such as terrain passability maps, territory maps and entity
> (units, buildings etc.) data. It's a lot of data and I guess the
> performance for accessing it can make a big difference.
> Basically we update this data once per simulation turn (around 5
> times
> per second) in the shared AI script and then access it from each AI
> player script through wrappers.
>
>>
>> I assume needing multiple globals here is a hard requirement?
>>
>
> I will check if it's possible to move this all into one global
> without
> rewriting the whole AI. I didn't try this before because I thought
> it's
> not compatible with multi-threading. Now my impression is that we
> aren't
> going to have a multi-threaded AI in the next year anyway because
> SpiderMonkey doesn't really support it yet.
> Any details about how the future implementation of multi-threading in
> SpiderMonkey will work would help. I don't have a good feeling when
> we
> have to work on an AI interface that's supposed to support multi-
> threading in the future but we don't know what requirements and
> constraints will have to be fulfilled to implement threading with
> SpiderMonkey.
> I've found this article today but I haven had the time to read it
> though
> completely yet.
> http://billmccloskey.wordpress.com/2013/12/05/multiprocess-firefox/
>
> Maybe putting all the AI players in one compartment isn't too bad.
> Improving AI performance is better than distributing the load across
> processor cores. I think this design should still allow us to move
> the AI
> as a whole to a separate thread, but probably won't allow moving each
> AI
> player to its own thread.
>
>
>>> It looks like this kind of "design breaking changes" are not going
>>> to
>>> become less in the near future.
>>
>> I think they actually should.  We've totally changed how we do GC
>> and
>> security.  There are few other cross-cutting things like that to
>> change...
> Well, we have the multi-threading, the exact stack rooting, changes
> related to contexts...
> That should be enough to keep us busy.
>
>
>> I'm _very_ interested in helping solve your performance issues.
> Thanks, I'll keep you updated about my tests for using one global for
> the
> whole AI and will probably ask some questions on IRC.
>
> _______________________________________________
> dev-tech-js-engine mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-tech-js-engine

_______________________________________________
dev-tech-js-engine mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-tech-js-engine
Reply | Threaded
Open this post in threaded view
|

Re: SpiderMonkey: Issues for embedders

Boris Zbarsky
In reply to this post by Yves
On 12/8/13 8:59 AM, Yves wrote:
> First, I would also like to ask if it would be possible to have a central
> Wiki page or something informing about what parts of the engine will
> change soon or are currently being changed.

This would be awesome, yes....

> We have the shared AI scripts that update common data between AI players.
> It's data such as terrain passability maps, territory maps and entity
> (units, buildings etc.) data. It's a lot of data and I guess the
> performance for accessing it can make a big difference.

Right, but my question was what sort of data it is.  For example, is a
terrain passability map something like an array of objects with some
numeric-valued properties on them, or is it more like a single array of
numbers or something else?

For some kinds of objects (typed arrays, maybe the new binary data
stuff, basically anything where we know at compile time the access will
return a number and know how to get it from memory) I suspect we can
make cross-compartment wrappers pretty fast. But getting objects from a
cross-compartment wrapper needs to cross-compartment wrap them,
unfortunately, and anything that has a nontrivial getter requires
entering the target compartment, so is hard to make fast.

It sounds like for your use case a setup where you write the data on one
thread, read from multiple threads, and explicitly synchronize would be
ideal in some ways, right?  I'm not sure whether PJS will let you get there.

> I will check if it's possible to move this all into one global without
> rewriting the whole AI. I didn't try this before because I thought it's
> not compatible with multi-threading.

It's not.  But neither is having them be in a single compartment, of
course, even in the pre-compartment-per-global days.

> http://billmccloskey.wordpress.com/2013/12/05/multiprocess-firefox/

That's pretty unrelated to what you're doing here, I think.  It also
doesn't have _fast_ cross-process communication (certainly I would
expect it to be slower than a cross-compartment wrapper).

-Boris
_______________________________________________
dev-tech-js-engine mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-tech-js-engine
Reply | Threaded
Open this post in threaded view
|

Re: SpiderMonkey: Issues for embedders

Yves
>> We have the shared AI scripts that update common data between AI
>> players.
>> It's data such as terrain passability maps, territory maps and entity
>> (units, buildings etc.) data. It's a lot of data and I guess the
>> performance for accessing it can make a big difference.
>
> Right, but my question was what sort of data it is.  

There's some data like arrays of integers but this kind of data will
likely become less in the future because terrain analysis and pathfinding
should only be done in JS until a new pathfinder with a JS interface is
ready.
Most of the data probably consists of entities and entity metadata which
is quite generic object data.


> It sounds like for your use case a setup where you write the data on one
> thread, read from multiple threads, and explicitly synchronize would be
> ideal in some ways, right?  I'm not sure whether PJS will let you get
> there.

I've started designing an UML diagram(well more or less UML) showing how
the AI interface design could work in the future and partially how it
already does.
I should have more time this Thursday to continue the work and this
discussion. :)

http://www.wildfiregames.com/forum/index.php?showtopic=17935

_______________________________________________
dev-tech-js-engine mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-tech-js-engine
Reply | Threaded
Open this post in threaded view
|

Re: SpiderMonkey: Issues for embedders

Yves
In reply to this post by Boris Zbarsky
I got distracted a bit from the discussion because I tried your
suggestion about moving the AI to one global object using the modules
pattern. I think it worked quite well:
http://trac.wildfiregames.com/ticket/2322

It looks like the main cause for the performance problems is really the
wrapping. I still need some time until I can test these changes with the
new SpiderMonkey version.
Here are some measurements:
http://www.wildfiregames.com/forum/index.php?
showtopic=17289&page=6#entry281010

Thanks again for the tip! :)

Any feedback on this or the multi-threading approach?
I think both approaches shouldn't be too prone to API changes.

Regards,
Yves
_______________________________________________
dev-tech-js-engine mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-tech-js-engine
Reply | Threaded
Open this post in threaded view
|

Re: SpiderMonkey: Issues for embedders

Yves
In reply to this post by Yves
> I personally have setup something like a "continuous integration branch"
> just for porting new changes on a daily basis and merge to my main
> branch when the engine reach a "big goal".
>

I think this is a very good approach and I will try that too.
When you integrate changes more often, you can always check the recent IRC
logs or commits for background information. This way it should be much
easier to find information than if you integrate the changes once in a
year.
The only problem could be bugs. I had this a few times when updating to
the development version directly. I always wasted some time until I
figured out if it's a bug or just something that changed and needs
adoptions in our code.

That still doesn't allow planning for the future, but at least it helps
keeping track of what happens at the present.
_______________________________________________
dev-tech-js-engine mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-tech-js-engine
Reply | Threaded
Open this post in threaded view
|

Re: SpiderMonkey: Issues for embedders

Boris Zbarsky
In reply to this post by Yves
On 12/28/13 12:56 PM, Yves wrote:
> Any feedback on this or the multi-threading approach?

I'm really hoping someone who actually know something about the JS
engine will answer this....

-Boris

_______________________________________________
dev-tech-js-engine mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-tech-js-engine
Reply | Threaded
Open this post in threaded view
|

Re: SpiderMonkey: Issues for embedders

Jeff Walden-2
In reply to this post by Yves
On 12/08/2013 07:59 AM, Yves wrote:

> First, I would also like to ask if it would be possible to have a central
> Wiki page or something informing about what parts of the engine will
> change soon or are currently being changed. Something like release notes,
> but looking into the future instead of the past.
> It doesn't even have to be very detailed, maybe something like this(just
> examples, I don't know if they are true):
>
> Multi-threading: Subject to change completely
> Garbage collection: Dropping support for conservative stack rooting
> in ESR XY
> JS Debugger API: The API is relatively stable. No significant
> changes are planned.
>
> Currently embedders are a bit like a flag in the wind without much
> possibilites to predict on which parts of the API they can rely and which
> parts they should better avoid using or be prepared if they change.

I've tried to do that sometimes, for things where I've had clear notions of what would happen.  See for example the "Upcoming JSAPI changes embedders should start considering" thread where I mentioned JS::CallArgs and jsval being replaced with JS::Value.  When there's an implemented, usable start at a transition path, it's definitely worth promoting as such.

Oftentimes, however, I haven't had much idea what I could reasonably say, that would be of any practical use to embedders.  It's one thing to have an idea that something will change.  But that doesn't mean I know exactly how it will change, nor does it mean that the two approaches can coexist.  And just saying "this will change", with no reasonable suggestion of what an embedder can do to prepare, well, I dunno how useful it is to say "this is coming, no we don't know when, no there's nothing you can do about it now".

Still, we can probably start up some sort of page for this, even if it's not clear how helpful what it'd say, would actually be.

> Any details about how the future implementation of multi-threading in
> SpiderMonkey will work would help. I don't have a good feeling when we
> have to work on an AI interface that's supposed to support multi-
> threading in the future but we don't know what requirements and
> constraints will have to be fulfilled to implement threading with
> SpiderMonkey.

It's never going to be possible to perform fully general operations on one JS object, from multiple threads.  Objects are tied to the JSRuntime where they were created.  JSRuntime is tied to a single thread.  There's various experimentation to allow certain algorithms to operate upon objects and other data in a JSRuntime, from multiple helper threads.  But those helper threads will be used, or unused, according to SpiderMonkey's determination.  The operations that can be run in parallel across helper threads, won't be the full set.  It's not clear what set of operations will be split this way.

At the moment, the parallel space includes several experiments: Rivertrail, PJS (or is that Rivertrail?), and nascent work on exposing SIMD operations.  You can add to that a potential experiment to evaluate adding ArrayBuffer instances backed by shared memory, that could be shared among multiple JSRuntimes/threads through the structured cloning operations in js/StructuredClone.h, with operations to act upon their data.  It's really too early to say what parallel will look like for the long term in JS the language.  And for that reason, it's really too early to say what parallel will look like for SpiderMonkey.  This stuff's really tricky to get right.  If we move too quickly here, we could easily saddle ourselves with something we'd regret for a very long time.  Generally, the best bet is to do as much work in separate JSRuntimes as possible, and to pass the minimum amount of data between threads backed by separate JSRuntimes.

Another possibility might be to make the clone algorithm JSClass-pluggable, so that you could do your own sort of shared-memory-passing tricks to pass larger amounts of data between JSRuntimes.  You'd have to implement it, but that might be a reasonable technique.

Jeff
_______________________________________________
dev-tech-js-engine mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-tech-js-engine
Reply | Threaded
Open this post in threaded view
|

Re: SpiderMonkey: Issues for embedders

Jeff Walden-2
On 1/7/14 10:33 AM, Jeff Walden wrote:
> Still, we can probably start up some sort of page for this, even if it's not clear how helpful what it'd say, would actually be.

Okay, I finally got around to starting up this page.  I filled it in
with various things I want to see done, at some point or another.  No
timelines for anything, no guarantees anything will happen exactly as
described, etc.  Just ideas, aspirations, etc. that we try to keep in
mind as we write patches and come up with designs for fixes to other
things.  Also some minor wish-fulfillment things that aren't deserving
specifically of being full-fledged goals, etc.

https://wiki.mozilla.org/JavaScript:SpiderMonkey:LongTermPlans

There's I'm sure a whole lot more where this stuff came from, that I
just haven't thought to put on there.  (Also lots that others haven't
yet put on there.)  So it's nothing at all like complete roadmap plans.
  :-)  But as far as vague ideas/inklings go, as were originally
requested, this may fit the bill a bit.

Jeff
_______________________________________________
dev-tech-js-engine mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-tech-js-engine
Reply | Threaded
Open this post in threaded view
|

Re: SpiderMonkey: Issues for embedders

Yves
In reply to this post by Yves
Am Mon, 03 Feb 2014 18:12:50 -0800 schrieb Jeff Walden:

> On 1/7/14 10:33 AM, Jeff Walden wrote:
>> Still, we can probably start up some sort of page for this, even if
>> it's not clear how helpful what it'd say, would actually be.
>
> Okay, I finally got around to starting up this page.
>

> Jeff

Thanks very much, there's a lot of new information there! I think it will
help us focusing on the right tasks and planning our upgrades.

Do you think it should be merged with this page?
https://developer.mozilla.org/en-US/docs/SpiderMonkey/Future_directions

One specific question I have is how long conservative stack scanning will
still be supported. Is there anything definitive you can say about that?
Will it still be supported in the next ESR release?

I've started tracking our transition, but I'm still working on other
issues with the v24 upgrade and this is a big task that will most likely
not be completed before ESR31.
http://trac.wildfiregames.com/ticket/2415


Yves
_______________________________________________
dev-tech-js-engine mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-tech-js-engine
Reply | Threaded
Open this post in threaded view
|

Re: SpiderMonkey: Issues for embedders

Terrence Cole-3
We will keep the conservative scanner at least through ESR31.

On 2/11/14 11:47 AM, Yves wrote:

> Am Mon, 03 Feb 2014 18:12:50 -0800 schrieb Jeff Walden:
>
>> On 1/7/14 10:33 AM, Jeff Walden wrote:
>>> Still, we can probably start up some sort of page for this, even if
>>> it's not clear how helpful what it'd say, would actually be.
>> Okay, I finally got around to starting up this page.
>>
>> Jeff
> Thanks very much, there's a lot of new information there! I think it will
> help us focusing on the right tasks and planning our upgrades.
>
> Do you think it should be merged with this page?
> https://developer.mozilla.org/en-US/docs/SpiderMonkey/Future_directions
>
> One specific question I have is how long conservative stack scanning will
> still be supported. Is there anything definitive you can say about that?
> Will it still be supported in the next ESR release?
>
> I've started tracking our transition, but I'm still working on other
> issues with the v24 upgrade and this is a big task that will most likely
> not be completed before ESR31.
> http://trac.wildfiregames.com/ticket/2415
>
>
> Yves
> _______________________________________________
> dev-tech-js-engine mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-tech-js-engine

_______________________________________________
dev-tech-js-engine mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-tech-js-engine