C-C TB Try server issue: Windows build fails most of the time for me.

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

C-C TB Try server issue: Windows build fails most of the time for me.

ISHIKAWA,chiaki
Hi,

For the last few weeks,
I get win32 build/configure error on the Tryserver when I submit a job
that includes the win32 debug build.

--- begin quote: excerpt of the error ---
mozmake.exe[1]: *** No rule to make target
'c:/builds/moz2_slave/tb-try-c-cen-w32-d-00000000000/build/mozilla/aclocal.m4',
needed by
'c:/builds/moz2_slave/tb-try-c-cen-w32-d-00000000000/build/configure'. Stop.
mozmake.exe: ***
[c:/builds/moz2_slave/tb-try-c-cen-w32-d-00000000000/build/objdir-tb/Makefile]
Error
--- end quote

What went wrong?

For example, the try job:
https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=7983cac21e61

Can anyone think of what I am doing wrong.
The used patches are in the above URL.
(notably I used client-py.patch to apply patches to ./mozilla
subdirectory,  and win32 hack to remove
ALWAYS_RUN_CLIENT_PY=1, etc.)

I don't know any other requirement specific to windows build.

Linux build works OK.

MacOS X 10.x occasionally barfs due to some esoteric system-dependent
issue or configuration issue, but more or less it is OK since I can
guess why the build fails and remedy the situation.

It is windows build that has failed me.

A few times, the build worked for no apparent reason, well, at least not
clear why windows build worked on those particular submissions. That
confuses me to no end.

TIA
_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: C-C TB Try server issue: Windows build fails most of the time for me.

Mike Hommey
On Wed, Jul 08, 2015 at 04:03:20AM +0900, ISHIKAWA, Chiaki wrote:

> Hi,
>
> For the last few weeks,
> I get win32 build/configure error on the Tryserver when I submit a job
> that includes the win32 debug build.
>
> --- begin quote: excerpt of the error ---
> mozmake.exe[1]: *** No rule to make target
> 'c:/builds/moz2_slave/tb-try-c-cen-w32-d-00000000000/build/mozilla/aclocal.m4',
> needed by
> 'c:/builds/moz2_slave/tb-try-c-cen-w32-d-00000000000/build/configure'. Stop.
> mozmake.exe: ***
> [c:/builds/moz2_slave/tb-try-c-cen-w32-d-00000000000/build/objdir-tb/Makefile]
> Error
> --- end quote
>
> What went wrong?
>
> For example, the try job:
> https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=7983cac21e61
>
> Can anyone think of what I am doing wrong.
> The used patches are in the above URL.
> (notably I used client-py.patch to apply patches to ./mozilla
> subdirectory,  and win32 hack to remove
> ALWAYS_RUN_CLIENT_PY=1, etc.)
>
> I don't know any other requirement specific to windows build.
>
> Linux build works OK.
>
> MacOS X 10.x occasionally barfs due to some esoteric system-dependent
> issue or configuration issue, but more or less it is OK since I can
> guess why the build fails and remedy the situation.
>
> It is windows build that has failed me.
>
> A few times, the build worked for no apparent reason, well, at least not
> clear why windows build worked on those particular submissions. That
> confuses me to no end.

Looks like your build didn't clone mozilla-central for whatever reason.

Mike
_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: C-C TB Try server issue: Windows build fails most of the time for me.

ISHIKAWA,chiaki
In reply to this post by ISHIKAWA,chiaki
On 2015/07/08 6:32, Mike Hommey wrote:
 > On Wed, Jul 08, 2015 at 04:03:20AM +0900, ISHIKAWA, Chiaki wrote:>>
Hi,>>>> For the last few weeks,>> I get win32 build/configure error on
the Tryserver when I submit a job>> that includes the win32 debug
build.>>>> --- begin quote: excerpt of the error --->> mozmake.exe[1]:
*** No rule to make target>>
'c:/builds/moz2_slave/tb-try-c-cen-w32-d-00000000000/build/mozilla/aclocal.m4',>>
needed by>>
'c:/builds/moz2_slave/tb-try-c-cen-w32-d-00000000000/build/configure'.
Stop.>> mozmake.exe: ***>>
[c:/builds/moz2_slave/tb-try-c-cen-w32-d-00000000000/build/objdir-tb/Makefile]>>
Error>> --- end quote>>>> What went wrong?>>>> For example, the try
job:>>
https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=7983cac21e61>>>>
Can anyone think of what I am doing wrong.>> The used patches are in the
above URL.>> (notably I used client-py.patch to apply patches to
./mozilla>> subdirectory,  and win32 hack to remove>>
ALWAYS_RUN_CLIENT_PY=1, etc.)>>>> I don't know any other requirement
specific to windows build.>>>> Linux build works OK.>>>> MacOS X 10.x
occasionally barfs due to some esoteric system-dependent>> issue or
configuration issue, but more or less it is OK since I can>> guess why
the build fails and remedy the situation.>>>> It is windows build that
has failed me.>>>> A few times, the build worked for no apparent reason,
well, at least not>> clear why windows build worked on those particular
submissions. That>> confuses me to no end.> > Looks like your build
didn't clone mozilla-central for whatever reason.> > Mike>
You mean that "for windows build", my build didn't clone mozilla-central (?)

[linux and macos X build works, so they seem to clone mozilla-central
correctly. Correct? Or did my job fail to clone it? ]

Then I wonder if the unwanted behavior can be
related to the following issue

Bug 1154222 - comm-central tryserver does not clean mozilla patches if
m-c tip has not changed

mentioned in the "Known issues" section of
https://wiki.mozilla.org/ReleaseEngineering/ThunderbirdTryServer

But it may not explain the random nature of the failure and the above
bug seems to apply linux and macos X build as well.
(Oh, is my trying to mix windows build with linux and macos X builds
triggering something on the tryserver?)

Anyway, I will submit the job again to see if anything changes...

TIA

_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: C-C TB Try server issue: Windows build fails most of the time for me.

ISHIKAWA,chiaki
In reply to this post by ISHIKAWA,chiaki
(Sorry for top-posting).

Oops, sorry the "Bug 1143570 - Copy/Paste into plain text editor deletes
newlines from quoted text" seems to hit this.
I didn't realize the loss of newlines while I was editing the e-mail.

Yes, I use TB for all my e-mail needs these days in a cross-platform manner
(mostly linux and windows).

Now I realize if I ask for windows 64-bit debug build on try server,
I simply get windows 32-bit build (and testing, etc. will be done using
win32 binary),
and so I can focus why win32 build fails most of the time for my job submission.

- Plan 1. I will compare how other people's win32 build is handled (already
I notice there are jobs that do not seem to have win32 specific tryserver
job submission patches. I have no idea why they seem to work.), and

- PLan 2. I decide to check M-C changes by submitting firefox tryserver job,
assuming
     windows build for ff tryserver works for me. (I think the chance is
higher. C-C TB build
     on tryserver, especially windows, is getting very complex.)

Plan 2 is not ideal for my purpose of reporting and fixing bug in win32
version of C-C TB, but
getting the patch accepted into M-C trunk is more important (then the source
change will be effective on C-C TB side eventually) and
I can usually test the patch locally very well for linux and usually the
bug/fix is platform-agnostics and this should go a long way for fixing
noticed issues in TB.

TIA

CI

On 2015年07月08日 11:37, ISHIKAWA, Chiaki wrote:

> On 2015/07/08 6:32, Mike Hommey wrote:
>> On Wed, Jul 08, 2015 at 04:03:20AM +0900, ISHIKAWA, Chiaki wrote:>> Hi,>>>> For the last few weeks,>> I get win32 build/configure error on the
> Tryserver when I submit a job>> that includes the win32 debug build.>>>> ---
> begin quote: excerpt of the error --->> mozmake.exe[1]: *** No rule to make
> target>>
> 'c:/builds/moz2_slave/tb-try-c-cen-w32-d-00000000000/build/mozilla/aclocal.m4',>>
> needed by>>
> 'c:/builds/moz2_slave/tb-try-c-cen-w32-d-00000000000/build/configure'.
> Stop.>> mozmake.exe: ***>>
> [c:/builds/moz2_slave/tb-try-c-cen-w32-d-00000000000/build/objdir-tb/Makefile]>>
> Error>> --- end quote>>>> What went wrong?>>>> For example, the try job:>>
> https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=7983cac21e61>>>>
> Can anyone think of what I am doing wrong.>> The used patches are in the
> above URL.>> (notably I used client-py.patch to apply patches to ./mozilla>>
> subdirectory,  and win32 hack to remove>> ALWAYS_RUN_CLIENT_PY=1, etc.)>>>>
> I don't know any other requirement specific to windows build.>>>> Linux
> build works OK.>>>> MacOS X 10.x occasionally barfs due to some esoteric
> system-dependent>> issue or configuration issue, but more or less it is OK
> since I can>> guess why the build fails and remedy the situation.>>>> It is
> windows build that has failed me.>>>> A few times, the build worked for no
> apparent reason, well, at least not>> clear why windows build worked on
> those particular submissions. That>> confuses me to no end.> > Looks like
> your build didn't clone mozilla-central for whatever reason.> > Mike>
> You mean that "for windows build", my build didn't clone mozilla-central (?)
>
> [linux and macos X build works, so they seem to clone mozilla-central
> correctly. Correct? Or did my job fail to clone it? ]
>
> Then I wonder if the unwanted behavior can be
> related to the following issue
>
> Bug 1154222 - comm-central tryserver does not clean mozilla patches if m-c
> tip has not changed
>
> mentioned in the "Known issues" section of
> https://wiki.mozilla.org/ReleaseEngineering/ThunderbirdTryServer
>
> But it may not explain the random nature of the failure and the above bug
> seems to apply linux and macos X build as well.
> (Oh, is my trying to mix windows build with linux and macos X builds
> triggering something on the tryserver?)
>
> Anyway, I will submit the job again to see if anything changes...
>
> TIA
>
_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: C-C TB Try server issue: Windows build fails most of the time for me.

Edmund Wong-2
In reply to this post by ISHIKAWA,chiaki
Mike Hommey wrote:

> On Wed, Jul 08, 2015 at 04:03:20AM +0900, ISHIKAWA, Chiaki wrote:
>> Hi,
>>
>> For the last few weeks,
>> I get win32 build/configure error on the Tryserver when I submit a job
>> that includes the win32 debug build.
>>
>> --- begin quote: excerpt of the error ---
>> mozmake.exe[1]: *** No rule to make target
>> 'c:/builds/moz2_slave/tb-try-c-cen-w32-d-00000000000/build/mozilla/aclocal.m4',
>> needed by
>> 'c:/builds/moz2_slave/tb-try-c-cen-w32-d-00000000000/build/configure'. Stop.
>> mozmake.exe: ***
>> [c:/builds/moz2_slave/tb-try-c-cen-w32-d-00000000000/build/objdir-tb/Makefile]
>> Error
>> --- end quote
>>
>> What went wrong?
>>
>> For example, the try job:
>> https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=7983cac21e61

The problem from what I see is that you removed the client.py running
via your mozconfig changes:

  # Run client.py
     1.11  mk_add_options CLIENT_PY_ARGS="$([ -f
$topsrcdir/build/client.py-args ] && cat $topsrcdir/build/client.py-args)"
     1.12 -mk_add_options ALWAYS_RUN_CLIENT_PY=1

TB's buildbot releng (from what I see) requires that to be run otherwise
it doesn't get the m-c code.


Edmund

_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: C-C TB Try server issue: Windows build fails most of the time for me.

ISHIKAWA,chiaki
On 2015年07月09日 13:16, Edmund Wong wrote:

> Mike Hommey wrote:
>> On Wed, Jul 08, 2015 at 04:03:20AM +0900, ISHIKAWA, Chiaki wrote:
>>> Hi,
>>>
>>> For the last few weeks,
>>> I get win32 build/configure error on the Tryserver when I submit a job
>>> that includes the win32 debug build.
>>>
>>> --- begin quote: excerpt of the error ---
>>> mozmake.exe[1]: *** No rule to make target
>>> 'c:/builds/moz2_slave/tb-try-c-cen-w32-d-00000000000/build/mozilla/aclocal.m4',
>>>
>>> needed by
>>> 'c:/builds/moz2_slave/tb-try-c-cen-w32-d-00000000000/build/configure'. Stop.
>>> mozmake.exe: ***
>>> [c:/builds/moz2_slave/tb-try-c-cen-w32-d-00000000000/build/objdir-tb/Makefile]
>>>
>>> Error
>>> --- end quote
>>>
>>> What went wrong?
>>>
>>> For example, the try job:
>>> https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=7983cac21e61
>>>
>
> The problem from what I see is that you removed the client.py running
> via your mozconfig changes:
>
>  # Run client.py
>     1.11  mk_add_options CLIENT_PY_ARGS="$([ -f
> $topsrcdir/build/client.py-args ] && cat $topsrcdir/build/client.py-args)"
>     1.12 -mk_add_options ALWAYS_RUN_CLIENT_PY=1
>
> TB's buildbot releng (from what I see) requires that to be run otherwise
> it doesn't get the m-c code.
>
>
> Edmund

I see, thank you for noticing the problem.

Short Answer: I think I don't need this removal any more.

Long answer:

I wondered why I did this and vaguely recall I had some Q&A sessions on the
issue.

Searching via google, I realized that, back in 2013, I was advised to delete
"ALWAYS_RUN_CLIENT_PY" from win32 client.py.
Reason back then was that the tryserver tried to apply patches twice and
failed for me.

http://comments.gmane.org/gmane.comp.mozilla.devel.platform/2920

Also from the search, I noticed in
https://wiki.mozilla.org/Thunderbird/StatusMeetings/2011-06-21

--- begin quote
Infrastructure Update

    bug 656049 c-c:0f79491664e9 client.mk can now invoke client.py for
comm-central builds via .mozconfig

 mk_add_options CLIENT_PY_ARGS="--verbose --skip-venkman"
 mk_add_options ALWAYS_RUN_CLIENT_PY=1

This can be directly invoked as make checkout, or implicitely whenever using
make some-target when ALWAYS_RUN_CLIENT_PY is set.

Coming to Try Server soon...
--- end quote

Maybe either I misread the tips, and was too eager to remove this
"ALWAYS_RUN_CLIENT_PY" in not-to-be-touched files [I thought it was likely,
but looking at the patch I am not so sure now.]
or maybe with the changes in between 2013 and 2015, this removal
"ALWAYS_RUN_CLIENT_PY" may not be necessary any more. (I think this is the
case now.)

All I can say is that there seemed to be an interaction between
--apply-patches and
ALWAYS_RUN_CLIENT_PY in unwanted manner back in 2013.

As I wrote the post-postscript re my observation testing framework failure,
I realize that
the recent post to explain the sorting order issue of handling M-C patches
by Joshua Cranmer
solves this question for me. His script no longer removes the
"ALAYWAS_RUN_CLIENT_PY" line!
I should have read the script carefully. There was "#" before the sed
command to do exactly that! But back then, I was more eager to figure out
the filename sorting issue.)

Thank you again for spotting this problem. I could not figure this out myself.

CI

PS: In the meantime, I tried FF build with the M-C patches I wanted to test,
and
it compiled, *BUT* the test seems broken.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=d16bb4c8debf

I mean the testing framework hiccuped.
It said "success" for win32  xpcshell tests (and later for win8 test, too),
and I could not believe it since xpcshell-tests on C-C TB side is full of
warnings and intermittent errors.

When I checked the error log, I saw very short log.
I noticed the testing frameworks barfed and died prematurely without running
any tests, but it was reported as "success".

Ah, testing is fun :-(

At least, I could verify that the my updated patches compile under windows :-).

PPS:

win32 test barfed due to WindowsError: [Error 6] The handle is invalid.
win8 version seems to have failed due to " 23:43:00 INFO - WindowsError:
[Error 5] Access is denied".

I wonder if the test frame work's access issue is related to "There is a bug
in that, under certain circumstances, it could try to apply patches due to
working directory reuse on build slaves." mentioned in the post to my
question last month by Joshua Cranmer.

https://groups.google.com/forum/#!topic/mozilla.dev.builds/pRr8gw0Ued0

Hist post is in the latter half of the post, and while I was re-reading his
post, I realize
there is an answer to "ALAYWAS_RUN_CLIENT_PY" (!).
Between 2013-2015, there was a change to make the deletion of the macro
unnecessary !






_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: C-C TB Try server issue: Windows build fails most of the time for me.

ISHIKAWA,chiaki
On 2015/07/09 16:50, ishikawa wrote:

> On 2015年07月09日 13:16, Edmund Wong wrote:
>> Mike Hommey wrote:
>>> On Wed, Jul 08, 2015 at 04:03:20AM +0900, ISHIKAWA, Chiaki wrote:
>>>> Hi,
>>>>
>>>> For the last few weeks,
>>>> I get win32 build/configure error on the Tryserver when I submit a job
>>>> that includes the win32 debug build.
>>>>
>>>> --- begin quote: excerpt of the error ---
>>>> mozmake.exe[1]: *** No rule to make target
>>>> 'c:/builds/moz2_slave/tb-try-c-cen-w32-d-00000000000/build/mozilla/aclocal.m4',
>>>>
>>>> needed by
>>>> 'c:/builds/moz2_slave/tb-try-c-cen-w32-d-00000000000/build/configure'. Stop.
>>>> mozmake.exe: ***
>>>> [c:/builds/moz2_slave/tb-try-c-cen-w32-d-00000000000/build/objdir-tb/Makefile]
>>>>
>>>> Error
>>>> --- end quote
>>>>
>>>> What went wrong?
>>>>
>>>> For example, the try job:
>>>> https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=7983cac21e61
>>>>
>>
>> The problem from what I see is that you removed the client.py running
>> via your mozconfig changes:
>>
>>   # Run client.py
>>      1.11  mk_add_options CLIENT_PY_ARGS="$([ -f
>> $topsrcdir/build/client.py-args ] && cat $topsrcdir/build/client.py-args)"
>>      1.12 -mk_add_options ALWAYS_RUN_CLIENT_PY=1
>>
>> TB's buildbot releng (from what I see) requires that to be run otherwise
>> it doesn't get the m-c code.
>>
>>
>> Edmund
>
> I see, thank you for noticing the problem.
>
> Short Answer: I think I don't need this removal any more.
>
> Long answer:
>
> I wondered why I did this and vaguely recall I had some Q&A sessions on the
> issue.
>
> Searching via google, I realized that, back in 2013, I was advised to delete
> "ALWAYS_RUN_CLIENT_PY" from win32 client.py.
> Reason back then was that the tryserver tried to apply patches twice and
> failed for me.
>
> http://comments.gmane.org/gmane.comp.mozilla.devel.platform/2920
>
> Also from the search, I noticed in
> https://wiki.mozilla.org/Thunderbird/StatusMeetings/2011-06-21
>
> --- begin quote
> Infrastructure Update
>
>      bug 656049 c-c:0f79491664e9 client.mk can now invoke client.py for
> comm-central builds via .mozconfig
>
>   mk_add_options CLIENT_PY_ARGS="--verbose --skip-venkman"
>   mk_add_options ALWAYS_RUN_CLIENT_PY=1
>
> This can be directly invoked as make checkout, or implicitely whenever using
> make some-target when ALWAYS_RUN_CLIENT_PY is set.
>
> Coming to Try Server soon...
> --- end quote
>
> Maybe either I misread the tips, and was too eager to remove this
> "ALWAYS_RUN_CLIENT_PY" in not-to-be-touched files [I thought it was likely,
> but looking at the patch I am not so sure now.]
> or maybe with the changes in between 2013 and 2015, this removal
> "ALWAYS_RUN_CLIENT_PY" may not be necessary any more. (I think this is the
> case now.)
>
> All I can say is that there seemed to be an interaction between
> --apply-patches and
> ALWAYS_RUN_CLIENT_PY in unwanted manner back in 2013.
>
> As I wrote the post-postscript re my observation testing framework failure,
> I realize that
> the recent post to explain the sorting order issue of handling M-C patches
> by Joshua Cranmer
> solves this question for me. His script no longer removes the
> "ALAYWAS_RUN_CLIENT_PY" line!
> I should have read the script carefully. There was "#" before the sed
> command to do exactly that! But back then, I was more eager to figure out
> the filename sorting issue.)
>
> Thank you again for spotting this problem. I could not figure this out myself.
>
> CI
>
> PS: In the meantime, I tried FF build with the M-C patches I wanted to test,
> and
> it compiled, *BUT* the test seems broken.
>
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=d16bb4c8debf
>
> I mean the testing framework hiccuped.
> It said "success" for win32  xpcshell tests (and later for win8 test, too),
> and I could not believe it since xpcshell-tests on C-C TB side is full of
> warnings and intermittent errors.
>
> When I checked the error log, I saw very short log.
> I noticed the testing frameworks barfed and died prematurely without running
> any tests, but it was reported as "success".
>
> Ah, testing is fun :-(
>
> At least, I could verify that the my updated patches compile under windows :-).
>
> PPS:
>
> win32 test barfed due to WindowsError: [Error 6] The handle is invalid.
> win8 version seems to have failed due to " 23:43:00 INFO - WindowsError:
> [Error 5] Access is denied".
>
> I wonder if the test frame work's access issue is related to "There is a bug
> in that, under certain circumstances, it could try to apply patches due to
> working directory reuse on build slaves." mentioned in the post to my
> question last month by Joshua Cranmer.
>
> https://groups.google.com/forum/#!topic/mozilla.dev.builds/pRr8gw0Ued0
>
> Hist post is in the latter half of the post, and while I was re-reading his
> post, I realize
> there is an answer to "ALAYWAS_RUN_CLIENT_PY" (!).
> Between 2013-2015, there was a change to make the deletion of the macro
> unnecessary !
>

By not removing "ALWAYS_RUN_CLIENT_PY", my submission finally succeeded
in getting windows version compiled (!)
Come to think of it, the first few times windows compilation succeeded
was when I failed to apply this unnecessary patch :-(
I tweaked my local patch preparation script to make sure this
extra patch is not included, etc.

So it is confirmed that ALWAYS_RUN_CLIENT_PY should not be removed any
more (unlike the time when I had an issue back in 2013.)

I would like to thank everyone who took the time to figure out my problem!

CI


_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds