Big spike in Windows jit-test failure

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Big spike in Windows jit-test failure

Ryan VanderMeulen
Recently, we've seen a very big spike in Windows jit-test failures. From
what I can tell, it started around Thursday last week. Can we please get
someone on the JS team to look into this soon?

JS intermittents filed in the last week:
https://bugzilla.mozilla.org/buglist.cgi?keywords=intermittent-failure%2C%20&keywords_type=allwords&list_id=9412412&chfieldto=Now&query_format=advanced&chfield=[Bug%20creation]&chfieldfrom=-168h&component=JavaScript&component=JavaScript%20Engine&component=JavaScript%20Engine%3A%20JIT&component=JavaScript%3A%20GC&component=JavaScript%3A%20Internationalization%20API&component=JavaScript%3A%20Standard%20Library

Thanks!

-Ryan
_______________________________________________
dev-tech-js-engine mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-tech-js-engine
Reply | Threaded
Open this post in threaded view
|

Re: Big spike in Windows jit-test failure

Jan de Mooij-3
(CC'ing the internals list)

I looked at some of the reports. Most of them are timeouts. Many with
--no-baseline --no-ion JIT flags, so it's likely unrelated to the
JITs.

I glanced at the commit log before this started, here are some bugs
that seem related at a quick glance:

Bug 941805 - Make the pool of JS workers be per process rather than
per runtime (maybe there's a race or deadlock somewhere?)
Bug 965745 - Always patch loop backedges in Ion if interrupt flag is
set (a bug there could cause issues like this but since it also
happens without the JITs it's unlikely).

Brian/Bill, could this be bug 941805? Maybe something in NSPR?

Thanks,
Jan

On Tue, Feb 11, 2014 at 1:50 PM, Ryan VanderMeulen <[hidden email]> wrote:

> Recently, we've seen a very big spike in Windows jit-test failures. From
> what I can tell, it started around Thursday last week. Can we please get
> someone on the JS team to look into this soon?
>
> JS intermittents filed in the last week:
> https://bugzilla.mozilla.org/buglist.cgi?keywords=intermittent-failure%2C%20&keywords_type=allwords&list_id=9412412&chfieldto=Now&query_format=advanced&chfield=[Bug%20creation]&chfieldfrom=-168h&component=JavaScript&component=JavaScript%20Engine&component=JavaScript%20Engine%3A%20JIT&component=JavaScript%3A%20GC&component=JavaScript%3A%20Internationalization%20API&component=JavaScript%3A%20Standard%20Library
>
> Thanks!
>
> -Ryan
> _______________________________________________
> dev-tech-js-engine mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-tech-js-engine
_______________________________________________
dev-tech-js-engine mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-tech-js-engine
Reply | Threaded
Open this post in threaded view
|

Re: Big spike in Windows jit-test failure

Jan de Mooij-3
I downloaded debug and opt JS shell builds from tbpl and ran
jit_test.py --tbpl --repeat 1000000 but 400,000 tests in I'm not
seeing any failures... This is on Windows 7 though and the failures
are on XP. Maybe somebody could do the same on XP?

Does this also happen with the new "Jit" jobs we run on Windows 7 and 8 on Try?

Jan

On Wed, Feb 12, 2014 at 4:58 PM, Jan de Mooij <[hidden email]> wrote:

> (CC'ing the internals list)
>
> I looked at some of the reports. Most of them are timeouts. Many with
> --no-baseline --no-ion JIT flags, so it's likely unrelated to the
> JITs.
>
> I glanced at the commit log before this started, here are some bugs
> that seem related at a quick glance:
>
> Bug 941805 - Make the pool of JS workers be per process rather than
> per runtime (maybe there's a race or deadlock somewhere?)
> Bug 965745 - Always patch loop backedges in Ion if interrupt flag is
> set (a bug there could cause issues like this but since it also
> happens without the JITs it's unlikely).
>
> Brian/Bill, could this be bug 941805? Maybe something in NSPR?
>
> Thanks,
> Jan
>
> On Tue, Feb 11, 2014 at 1:50 PM, Ryan VanderMeulen <[hidden email]> wrote:
>> Recently, we've seen a very big spike in Windows jit-test failures. From
>> what I can tell, it started around Thursday last week. Can we please get
>> someone on the JS team to look into this soon?
>>
>> JS intermittents filed in the last week:
>> https://bugzilla.mozilla.org/buglist.cgi?keywords=intermittent-failure%2C%20&keywords_type=allwords&list_id=9412412&chfieldto=Now&query_format=advanced&chfield=[Bug%20creation]&chfieldfrom=-168h&component=JavaScript&component=JavaScript%20Engine&component=JavaScript%20Engine%3A%20JIT&component=JavaScript%3A%20GC&component=JavaScript%3A%20Internationalization%20API&component=JavaScript%3A%20Standard%20Library
>>
>> Thanks!
>>
>> -Ryan
>> _______________________________________________
>> dev-tech-js-engine mailing list
>> [hidden email]
>> https://lists.mozilla.org/listinfo/dev-tech-js-engine
_______________________________________________
dev-tech-js-engine mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-tech-js-engine
Reply | Threaded
Open this post in threaded view
|

Re: Big spike in Windows jit-test failure

Ryan VanderMeulen
In reply to this post by Jan de Mooij-3
On 2/12/2014 2:49 PM, Jan de Mooij wrote:
> I downloaded debug and opt JS shell builds from tbpl and ran
> jit_test.py --tbpl --repeat 1000000 but 400,000 tests in I'm not
> seeing any failures... This is on Windows 7 though and the failures
> are on XP. Maybe somebody could do the same on XP?
>
> Does this also happen with the new "Jit" jobs we run on Windows 7 and 8 on Try?
>
> Jan

Note that it's not really "XP" per se, it's Windows Server 2003, which
is what our build slaves are.

I don't know about the Jit runs on Try as they're currently perma-fail
for other reasons.

-Ryan
_______________________________________________
dev-tech-js-engine mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-tech-js-engine
Reply | Threaded
Open this post in threaded view
|

Re: Big spike in Windows jit-test failure

Ryan VanderMeulen
On 2/12/2014 2:58 PM, Ryan VanderMeulen wrote:
> Note that it's not really "XP" per se, it's Windows Server 2003, which
> is what our build slaves are.
>
> I don't know about the Jit runs on Try as they're currently perma-fail
> for other reasons.
>
> -Ryan

I pinged philor about it and he says that we are hitting them on Try as
well.
https://tbpl.mozilla.org/php/getParsedLog.php?id=34571544&tree=Try
_______________________________________________
dev-tech-js-engine mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-tech-js-engine