localized builds, status and questions

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

localized builds, status and questions

Axel Hecht
Hi,

as some of you probably noticed, I've been hacking around our localized
builds a bit over the past few weeks.

Context:

For https://bugzilla.mozilla.org/show_bug.cgi?id=1362617, I wonder where
to put the generation of res/multilocale.json in a regular build.
And I hope that someone volunteers to tackle l10n repacks for local
artifact builds.

Status:

For desktop, we're now at the point where all that we do outside of
browser/locales is jar packaging, and installer packaging. For android,
we just need to move the search stuff from mobile/locales to
mobile/android/locales.

We're also only running the jar packaging once, and we're only running
repack-l10n.py once.

I also think we can drop the -j1 from the entry points.

My hope here is that now that we're only using recursive make in a
defined scope, and that parallelism isn't a problem anymore, we should
have a better chance to get other backends to support repacks? Notably
the artifact builds one?

The flow of an l10n repack is also constructive now. It's preparing the
localization, possibly getting a repo, and running compare-locales for
l10n-merge. Then it's doing all the jar stuff, and then the app-specific
funkyness like search.
Then it packages the stuff as applicable for the project. One desktop,
that's langpack, package, and possibly full mar. Then it creates a
windows installer if on windows. On mobile, it's just the apk (and full
mar?).

All of this is done by explicit steps in the recipe, and I got rid of
dependencies as much as possible. The dependencies turned out to be
primary pain point, as they weren't dependencies to begin with.

Questions:

Where to put multilocale.json, in en-US builds, and in multi-locale
builds, and in single-locale repacks?

Do we need the XPI_ROOT_APPID='$(XPI_ROOT_APPID)' jazz? That makes the
recipes way more complex than I think they should be. Some conditional
hacks that we have depending on this could be replaced with
IS_LANGUAGE_REPACK ?

What do we really need to do to support l10n repacks on artifact builds?
The build backend code is above me, sadly.

Which builds should run l10n-check in automation? I see that many things
disable that, and I'd like to have an entry point to run l10n-related
tests. We'll need that. Maybe it's time to have a dedicated TC task for
them?

Axel

_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: localized builds, status and questions

Gregory Szorc-3
On Mon, Aug 28, 2017 at 9:16 AM, Axel Hecht <[hidden email]> wrote:
Hi,

as some of you probably noticed, I've been hacking around our localized builds a bit over the past few weeks.

Context:

For https://bugzilla.mozilla.org/show_bug.cgi?id=1362617, I wonder where to put the generation of res/multilocale.json in a regular build.
And I hope that someone volunteers to tackle l10n repacks for local artifact builds.

Status:

For desktop, we're now at the point where all that we do outside of browser/locales is jar packaging, and installer packaging. For android, we just need to move the search stuff from mobile/locales to mobile/android/locales.

We're also only running the jar packaging once, and we're only running repack-l10n.py once.

I also think we can drop the -j1 from the entry points.

My hope here is that now that we're only using recursive make in a defined scope, and that parallelism isn't a problem anymore, we should have a better chance to get other backends to support repacks? Notably the artifact builds one?

The flow of an l10n repack is also constructive now. It's preparing the localization, possibly getting a repo, and running compare-locales for l10n-merge. Then it's doing all the jar stuff, and then the app-specific funkyness like search.
Then it packages the stuff as applicable for the project. One desktop, that's langpack, package, and possibly full mar. Then it creates a windows installer if on windows. On mobile, it's just the apk (and full mar?).

All of this is done by explicit steps in the recipe, and I got rid of dependencies as much as possible. The dependencies turned out to be primary pain point, as they weren't dependencies to begin with.

Questions:

Where to put multilocale.json, in en-US builds, and in multi-locale builds, and in single-locale repacks?

Do we need the XPI_ROOT_APPID='$(XPI_ROOT_APPID)' jazz? That makes the recipes way more complex than I think they should be. Some conditional hacks that we have depending on this could be replaced with IS_LANGUAGE_REPACK ?

XPI_ROOT_APPID feels like something that should be defined once in the entrypoint and shouldn't need to be passed around everywhere. But I have no clue what all is keyed off of it.
 

What do we really need to do to support l10n repacks on artifact builds? The build backend code is above me, sadly.

Conceptually, an artifact build is a glorified cache. At the top of the build we extract files from a previous build into their normal locations in the objdir. And we disable all the build rules that would produce those files.

If l10n repacks were implemented as a post-build action that didn't interfere with existing files (e.g. didn't delete binaries or other files from the objdir it didn't itself produce), I would expect l10n repacks to be compatible with artifact builds. The work to actually make them compatible is likely a bunch of configure muckery and ensuring the artifact build backend has enough context to let l10n repacks work. I wouldn't be surprised if we were disabling things required by l10n repacks.
 

Which builds should run l10n-check in automation? I see that many things disable that, and I'd like to have an entry point to run l10n-related tests. We'll need that. Maybe it's time to have a dedicated TC task for them?

As many of them as possible. Running l10n-check on any push where we do a build seems like a reasonable target. I'm not sure if there are perf/capacity concerns with that though.

Making a dedicated TC task for l10n-check seems reasonable. We try to have more, smaller tasks than fewer, larger tasks. But, there are compelling reasons to shoehorn into an existing task (namely scaling concerns due to large setup/teardown costs or redundancy).

_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: localized builds, status and questions

Mike Hommey
On Mon, Aug 28, 2017 at 06:10:26PM -0700, Gregory Szorc wrote:

> On Mon, Aug 28, 2017 at 9:16 AM, Axel Hecht <[hidden email]> wrote:
>
> > Hi,
> >
> > as some of you probably noticed, I've been hacking around our localized
> > builds a bit over the past few weeks.
> >
> > Context:
> >
> > For https://bugzilla.mozilla.org/show_bug.cgi?id=1362617, I wonder where
> > to put the generation of res/multilocale.json in a regular build.
> > And I hope that someone volunteers to tackle l10n repacks for local
> > artifact builds.
> >
> > Status:
> >
> > For desktop, we're now at the point where all that we do outside of
> > browser/locales is jar packaging, and installer packaging. For android, we
> > just need to move the search stuff from mobile/locales to
> > mobile/android/locales.
> >
> > We're also only running the jar packaging once, and we're only running
> > repack-l10n.py once.
> >
> > I also think we can drop the -j1 from the entry points.
> >
> > My hope here is that now that we're only using recursive make in a defined
> > scope, and that parallelism isn't a problem anymore, we should have a
> > better chance to get other backends to support repacks? Notably the
> > artifact builds one?
> >
> > The flow of an l10n repack is also constructive now. It's preparing the
> > localization, possibly getting a repo, and running compare-locales for
> > l10n-merge. Then it's doing all the jar stuff, and then the app-specific
> > funkyness like search.
> > Then it packages the stuff as applicable for the project. One desktop,
> > that's langpack, package, and possibly full mar. Then it creates a windows
> > installer if on windows. On mobile, it's just the apk (and full mar?).
> >
> > All of this is done by explicit steps in the recipe, and I got rid of
> > dependencies as much as possible. The dependencies turned out to be primary
> > pain point, as they weren't dependencies to begin with.
> >
> > Questions:
> >
> > Where to put multilocale.json, in en-US builds, and in multi-locale
> > builds, and in single-locale repacks?
> >
> > Do we need the XPI_ROOT_APPID='$(XPI_ROOT_APPID)' jazz? That makes the
> > recipes way more complex than I think they should be. Some conditional
> > hacks that we have depending on this could be replaced with
> > IS_LANGUAGE_REPACK ?
> >
>
> XPI_ROOT_APPID feels like something that should be defined once in the
> entrypoint and shouldn't need to be passed around everywhere. But I have no
> clue what all is keyed off of it.
>
>
> >
> > What do we really need to do to support l10n repacks on artifact builds?
> > The build backend code is above me, sadly.
> >
>
> Conceptually, an artifact build is a glorified cache. At the top of the
> build we extract files from a previous build into their normal locations in
> the objdir. And we disable all the build rules that would produce those
> files.
>
> If l10n repacks were implemented as a post-build action that didn't
> interfere with existing files (e.g. didn't delete binaries or other files
> from the objdir it didn't itself produce), I would expect l10n repacks to
> be compatible with artifact builds. The work to actually make them
> compatible is likely a bunch of configure muckery and ensuring the artifact
> build backend has enough context to let l10n repacks work. I wouldn't be
> surprised if we were disabling things required by l10n repacks.

I, for one, think l10n repacks should be deprecated in favor of artifact
builds. Because, really, when you take a step back, l10n repacks are
just a special kind of artifact builds where the strings data is taken
from a different place.

Mike
_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: localized builds, status and questions

Axel Hecht
In reply to this post by Gregory Szorc-3
Am 29.08.17 um 06:16 schrieb Mike Hommey:

> On Mon, Aug 28, 2017 at 06:10:26PM -0700, Gregory Szorc wrote:
>> On Mon, Aug 28, 2017 at 9:16 AM, Axel Hecht <[hidden email]> wrote:
>>
>>> Hi,
>>>
>>> as some of you probably noticed, I've been hacking around our localized
>>> builds a bit over the past few weeks.
>>>
>>> Context:
>>>
>>> For https://bugzilla.mozilla.org/show_bug.cgi?id=1362617, I wonder where
>>> to put the generation of res/multilocale.json in a regular build.
>>> And I hope that someone volunteers to tackle l10n repacks for local
>>> artifact builds.
>>>
>>> Status:
>>>
>>> For desktop, we're now at the point where all that we do outside of
>>> browser/locales is jar packaging, and installer packaging. For android, we
>>> just need to move the search stuff from mobile/locales to
>>> mobile/android/locales.
>>>
>>> We're also only running the jar packaging once, and we're only running
>>> repack-l10n.py once.
>>>
>>> I also think we can drop the -j1 from the entry points.
>>>
>>> My hope here is that now that we're only using recursive make in a defined
>>> scope, and that parallelism isn't a problem anymore, we should have a
>>> better chance to get other backends to support repacks? Notably the
>>> artifact builds one?
>>>
>>> The flow of an l10n repack is also constructive now. It's preparing the
>>> localization, possibly getting a repo, and running compare-locales for
>>> l10n-merge. Then it's doing all the jar stuff, and then the app-specific
>>> funkyness like search.
>>> Then it packages the stuff as applicable for the project. One desktop,
>>> that's langpack, package, and possibly full mar. Then it creates a windows
>>> installer if on windows. On mobile, it's just the apk (and full mar?).
>>>
>>> All of this is done by explicit steps in the recipe, and I got rid of
>>> dependencies as much as possible. The dependencies turned out to be primary
>>> pain point, as they weren't dependencies to begin with.
>>>
>>> Questions:
>>>
>>> Where to put multilocale.json, in en-US builds, and in multi-locale
>>> builds, and in single-locale repacks?
>>>
>>> Do we need the XPI_ROOT_APPID='$(XPI_ROOT_APPID)' jazz? That makes the
>>> recipes way more complex than I think they should be. Some conditional
>>> hacks that we have depending on this could be replaced with
>>> IS_LANGUAGE_REPACK ?
>>>
>>
>> XPI_ROOT_APPID feels like something that should be defined once in the
>> entrypoint and shouldn't need to be passed around everywhere. But I have no
>> clue what all is keyed off of it.
>>
>>
>>>
>>> What do we really need to do to support l10n repacks on artifact builds?
>>> The build backend code is above me, sadly.
>>>
>>
>> Conceptually, an artifact build is a glorified cache. At the top of the
>> build we extract files from a previous build into their normal locations in
>> the objdir. And we disable all the build rules that would produce those
>> files.
>>
>> If l10n repacks were implemented as a post-build action that didn't
>> interfere with existing files (e.g. didn't delete binaries or other files
>> from the objdir it didn't itself produce), I would expect l10n repacks to
>> be compatible with artifact builds. The work to actually make them
>> compatible is likely a bunch of configure muckery and ensuring the artifact
>> build backend has enough context to let l10n repacks work. I wouldn't be
>> surprised if we were disabling things required by l10n repacks.
>
> I, for one, think l10n repacks should be deprecated in favor of artifact
> builds. Because, really, when you take a step back, l10n repacks are
> just a special kind of artifact builds where the strings data is taken
> from a different place.

When we talk about --disable-compile-environment vs
--enable-artifact-builds, I agree that the differences should be minimal.

When we talk about how to get the binaries to repack, I guess that's
mostly a question of the chain of trust on production rebuilds, I hope
that Callek has opinions on that when he gets back.

Axel
_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: localized builds, status and questions

Axel Hecht
In reply to this post by Axel Hecht
Am 29.08.17 um 03:10 schrieb Gregory Szorc:

> On Mon, Aug 28, 2017 at 9:16 AM, Axel Hecht <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     Hi,
>
>     as some of you probably noticed, I've been hacking around our
>     localized builds a bit over the past few weeks.
>
>     Context:
>
>     For https://bugzilla.mozilla.org/show_bug.cgi?id=1362617
>     <https://bugzilla.mozilla.org/show_bug.cgi?id=1362617>, I wonder
>     where to put the generation of res/multilocale.json in a regular build.
>     And I hope that someone volunteers to tackle l10n repacks for local
>     artifact builds.
>
>     Status:
>
>     For desktop, we're now at the point where all that we do outside of
>     browser/locales is jar packaging, and installer packaging. For
>     android, we just need to move the search stuff from mobile/locales
>     to mobile/android/locales.
>
>     We're also only running the jar packaging once, and we're only
>     running repack-l10n.py once.
>
>     I also think we can drop the -j1 from the entry points.
>
>     My hope here is that now that we're only using recursive make in a
>     defined scope, and that parallelism isn't a problem anymore, we
>     should have a better chance to get other backends to support
>     repacks? Notably the artifact builds one?
>
>     The flow of an l10n repack is also constructive now. It's preparing
>     the localization, possibly getting a repo, and running
>     compare-locales for l10n-merge. Then it's doing all the jar stuff,
>     and then the app-specific funkyness like search.
>     Then it packages the stuff as applicable for the project. One
>     desktop, that's langpack, package, and possibly full mar. Then it
>     creates a windows installer if on windows. On mobile, it's just the
>     apk (and full mar?).
>
>     All of this is done by explicit steps in the recipe, and I got rid
>     of dependencies as much as possible. The dependencies turned out to
>     be primary pain point, as they weren't dependencies to begin with.
>
>     Questions:
>
>     Where to put multilocale.json, in en-US builds, and in multi-locale
>     builds, and in single-locale repacks?
>
>     Do we need the XPI_ROOT_APPID='$(XPI_ROOT_APPID)' jazz? That makes
>     the recipes way more complex than I think they should be. Some
>     conditional hacks that we have depending on this could be replaced
>     with IS_LANGUAGE_REPACK ?
>
>
> XPI_ROOT_APPID feels like something that should be defined once in the
> entrypoint and shouldn't need to be passed around everywhere. But I have
> no clue what all is keyed off of it.

https://dxr.mozilla.org/mozilla-central/source/config/rules.mk#1258 
seems to be where they're used. Which ends up in
https://dxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/jar.py#283,
and then I'm lost ;-)

>     What do we really need to do to support l10n repacks on artifact
>     builds? The build backend code is above me, sadly.
>
>
> Conceptually, an artifact build is a glorified cache. At the top of the
> build we extract files from a previous build into their normal locations
> in the objdir. And we disable all the build rules that would produce
> those files.
 >
> If l10n repacks were implemented as a post-build action that didn't
> interfere with existing files (e.g. didn't delete binaries or other
> files from the objdir it didn't itself produce), I would expect l10n
> repacks to be compatible with artifact builds. The work to actually make
> them compatible is likely a bunch of configure muckery and ensuring the
> artifact build backend has enough context to let l10n repacks work. I
> wouldn't be surprised if we were disabling things required by l10n repacks.


The first trigger for failing l10n repacks is that

   make -C toolkit/locales chrome

doesn't actually do anything. At least that's as far as my research goes.

>
>     Which builds should run l10n-check in automation? I see that many
>     things disable that, and I'd like to have an entry point to run
>     l10n-related tests. We'll need that. Maybe it's time to have a
>     dedicated TC task for them?
>
>
> As many of them as possible. Running l10n-check on any push where we do
> a build seems like a reasonable target. I'm not sure if there are
> perf/capacity concerns with that though.
>
> Making a dedicated TC task for l10n-check seems reasonable. We try to
> have more, smaller tasks than fewer, larger tasks. But, there are
> compelling reasons to shoehorn into an existing task (namely scaling
> concerns due to large setup/teardown costs or redundancy).

Filed https://bugzilla.mozilla.org/show_bug.cgi?id=1394733 to get it on
more generally.

Axel
_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: localized builds, status and questions

Axel Hecht
In reply to this post by Gregory Szorc-3
Am 29.08.17 um 06:16 schrieb Mike Hommey:

> On Mon, Aug 28, 2017 at 06:10:26PM -0700, Gregory Szorc wrote:
>> On Mon, Aug 28, 2017 at 9:16 AM, Axel Hecht <[hidden email]> wrote:
>>
>>> Hi,
>>>
>>> as some of you probably noticed, I've been hacking around our localized
>>> builds a bit over the past few weeks.
>>>
>>> Context:
>>>
>>> For https://bugzilla.mozilla.org/show_bug.cgi?id=1362617, I wonder where
>>> to put the generation of res/multilocale.json in a regular build.
>>> And I hope that someone volunteers to tackle l10n repacks for local
>>> artifact builds.
>>>
>>> Status:
>>>
>>> For desktop, we're now at the point where all that we do outside of
>>> browser/locales is jar packaging, and installer packaging. For android, we
>>> just need to move the search stuff from mobile/locales to
>>> mobile/android/locales.
>>>
>>> We're also only running the jar packaging once, and we're only running
>>> repack-l10n.py once.
>>>
>>> I also think we can drop the -j1 from the entry points.
>>>
>>> My hope here is that now that we're only using recursive make in a defined
>>> scope, and that parallelism isn't a problem anymore, we should have a
>>> better chance to get other backends to support repacks? Notably the
>>> artifact builds one?
>>>
>>> The flow of an l10n repack is also constructive now. It's preparing the
>>> localization, possibly getting a repo, and running compare-locales for
>>> l10n-merge. Then it's doing all the jar stuff, and then the app-specific
>>> funkyness like search.
>>> Then it packages the stuff as applicable for the project. One desktop,
>>> that's langpack, package, and possibly full mar. Then it creates a windows
>>> installer if on windows. On mobile, it's just the apk (and full mar?).
>>>
>>> All of this is done by explicit steps in the recipe, and I got rid of
>>> dependencies as much as possible. The dependencies turned out to be primary
>>> pain point, as they weren't dependencies to begin with.
>>>
>>> Questions:
>>>
>>> Where to put multilocale.json, in en-US builds, and in multi-locale
>>> builds, and in single-locale repacks?
>>>
>>> Do we need the XPI_ROOT_APPID='$(XPI_ROOT_APPID)' jazz? That makes the
>>> recipes way more complex than I think they should be. Some conditional
>>> hacks that we have depending on this could be replaced with
>>> IS_LANGUAGE_REPACK ?
>>>
>>
>> XPI_ROOT_APPID feels like something that should be defined once in the
>> entrypoint and shouldn't need to be passed around everywhere. But I have no
>> clue what all is keyed off of it.
>>
>>
>>>
>>> What do we really need to do to support l10n repacks on artifact builds?
>>> The build backend code is above me, sadly.
>>>
>>
>> Conceptually, an artifact build is a glorified cache. At the top of the
>> build we extract files from a previous build into their normal locations in
>> the objdir. And we disable all the build rules that would produce those
>> files.
>>
>> If l10n repacks were implemented as a post-build action that didn't
>> interfere with existing files (e.g. didn't delete binaries or other files
>> from the objdir it didn't itself produce), I would expect l10n repacks to
>> be compatible with artifact builds. The work to actually make them
>> compatible is likely a bunch of configure muckery and ensuring the artifact
>> build backend has enough context to let l10n repacks work. I wouldn't be
>> surprised if we were disabling things required by l10n repacks.
>
> I, for one, think l10n repacks should be deprecated in favor of artifact
> builds. Because, really, when you take a step back, l10n repacks are
> just a special kind of artifact builds where the strings data is taken
> from a different place.
>

In the light of https://bugzilla.mozilla.org/show_bug.cgi?id=1352595#c38 
(brotli and artifact builds), I actually strongly disagree with that
sentiment.

Our localized builds should have all the optimizations we have for en-US
release builds, and I read your comment in the bug that that's not the
case for artifact builds.

Axel
_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: localized builds, status and questions

Axel Hecht
In reply to this post by Axel Hecht
Am 28.08.17 um 18:16 schrieb Axel Hecht:
>
> Questions:
>
> Where to put multilocale.json, in en-US builds, and in multi-locale
> builds, and in single-locale repacks?

This question seems to have drowned in the conversation, but we'll need
to come to a constructive solution for this.

Axel
_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: localized builds, status and questions

Mike Hommey
In reply to this post by Axel Hecht
On Thu, Sep 07, 2017 at 12:19:14AM +0200, Axel Hecht wrote:

> Am 29.08.17 um 06:16 schrieb Mike Hommey:
> > On Mon, Aug 28, 2017 at 06:10:26PM -0700, Gregory Szorc wrote:
> > > On Mon, Aug 28, 2017 at 9:16 AM, Axel Hecht <[hidden email]> wrote:
> > >
> > > > Hi,
> > > >
> > > > as some of you probably noticed, I've been hacking around our localized
> > > > builds a bit over the past few weeks.
> > > >
> > > > Context:
> > > >
> > > > For https://bugzilla.mozilla.org/show_bug.cgi?id=1362617, I wonder where
> > > > to put the generation of res/multilocale.json in a regular build.
> > > > And I hope that someone volunteers to tackle l10n repacks for local
> > > > artifact builds.
> > > >
> > > > Status:
> > > >
> > > > For desktop, we're now at the point where all that we do outside of
> > > > browser/locales is jar packaging, and installer packaging. For android, we
> > > > just need to move the search stuff from mobile/locales to
> > > > mobile/android/locales.
> > > >
> > > > We're also only running the jar packaging once, and we're only running
> > > > repack-l10n.py once.
> > > >
> > > > I also think we can drop the -j1 from the entry points.
> > > >
> > > > My hope here is that now that we're only using recursive make in a defined
> > > > scope, and that parallelism isn't a problem anymore, we should have a
> > > > better chance to get other backends to support repacks? Notably the
> > > > artifact builds one?
> > > >
> > > > The flow of an l10n repack is also constructive now. It's preparing the
> > > > localization, possibly getting a repo, and running compare-locales for
> > > > l10n-merge. Then it's doing all the jar stuff, and then the app-specific
> > > > funkyness like search.
> > > > Then it packages the stuff as applicable for the project. One desktop,
> > > > that's langpack, package, and possibly full mar. Then it creates a windows
> > > > installer if on windows. On mobile, it's just the apk (and full mar?).
> > > >
> > > > All of this is done by explicit steps in the recipe, and I got rid of
> > > > dependencies as much as possible. The dependencies turned out to be primary
> > > > pain point, as they weren't dependencies to begin with.
> > > >
> > > > Questions:
> > > >
> > > > Where to put multilocale.json, in en-US builds, and in multi-locale
> > > > builds, and in single-locale repacks?
> > > >
> > > > Do we need the XPI_ROOT_APPID='$(XPI_ROOT_APPID)' jazz? That makes the
> > > > recipes way more complex than I think they should be. Some conditional
> > > > hacks that we have depending on this could be replaced with
> > > > IS_LANGUAGE_REPACK ?
> > > >
> > >
> > > XPI_ROOT_APPID feels like something that should be defined once in the
> > > entrypoint and shouldn't need to be passed around everywhere. But I have no
> > > clue what all is keyed off of it.
> > >
> > >
> > > >
> > > > What do we really need to do to support l10n repacks on artifact builds?
> > > > The build backend code is above me, sadly.
> > > >
> > >
> > > Conceptually, an artifact build is a glorified cache. At the top of the
> > > build we extract files from a previous build into their normal locations in
> > > the objdir. And we disable all the build rules that would produce those
> > > files.
> > >
> > > If l10n repacks were implemented as a post-build action that didn't
> > > interfere with existing files (e.g. didn't delete binaries or other files
> > > from the objdir it didn't itself produce), I would expect l10n repacks to
> > > be compatible with artifact builds. The work to actually make them
> > > compatible is likely a bunch of configure muckery and ensuring the artifact
> > > build backend has enough context to let l10n repacks work. I wouldn't be
> > > surprised if we were disabling things required by l10n repacks.
> >
> > I, for one, think l10n repacks should be deprecated in favor of artifact
> > builds. Because, really, when you take a step back, l10n repacks are
> > just a special kind of artifact builds where the strings data is taken
> > from a different place.
> >
>
> In the light of https://bugzilla.mozilla.org/show_bug.cgi?id=1352595#c38
> (brotli and artifact builds), I actually strongly disagree with that
> sentiment.
>
> Our localized builds should have all the optimizations we have for en-US
> release builds, and I read your comment in the bug that that's not the case
> for artifact builds.

That's not the case for artifact builds *today*. Because the brotli
support was made partial on purpose because, what's the point of
implementing the whole thing if we're not going to do it in the end?

Mike
_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: localized builds, status and questions

Axel Hecht
In reply to this post by Axel Hecht
Am 07.09.17 um 00:45 schrieb Mike Hommey:

> On Thu, Sep 07, 2017 at 12:19:14AM +0200, Axel Hecht wrote:
>> Am 29.08.17 um 06:16 schrieb Mike Hommey:
>>> On Mon, Aug 28, 2017 at 06:10:26PM -0700, Gregory Szorc wrote:
>>>> On Mon, Aug 28, 2017 at 9:16 AM, Axel Hecht <[hidden email]> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> as some of you probably noticed, I've been hacking around our localized
>>>>> builds a bit over the past few weeks.
>>>>>
>>>>> Context:
>>>>>
>>>>> For https://bugzilla.mozilla.org/show_bug.cgi?id=1362617, I wonder where
>>>>> to put the generation of res/multilocale.json in a regular build.
>>>>> And I hope that someone volunteers to tackle l10n repacks for local
>>>>> artifact builds.
>>>>>
>>>>> Status:
>>>>>
>>>>> For desktop, we're now at the point where all that we do outside of
>>>>> browser/locales is jar packaging, and installer packaging. For android, we
>>>>> just need to move the search stuff from mobile/locales to
>>>>> mobile/android/locales.
>>>>>
>>>>> We're also only running the jar packaging once, and we're only running
>>>>> repack-l10n.py once.
>>>>>
>>>>> I also think we can drop the -j1 from the entry points.
>>>>>
>>>>> My hope here is that now that we're only using recursive make in a defined
>>>>> scope, and that parallelism isn't a problem anymore, we should have a
>>>>> better chance to get other backends to support repacks? Notably the
>>>>> artifact builds one?
>>>>>
>>>>> The flow of an l10n repack is also constructive now. It's preparing the
>>>>> localization, possibly getting a repo, and running compare-locales for
>>>>> l10n-merge. Then it's doing all the jar stuff, and then the app-specific
>>>>> funkyness like search.
>>>>> Then it packages the stuff as applicable for the project. One desktop,
>>>>> that's langpack, package, and possibly full mar. Then it creates a windows
>>>>> installer if on windows. On mobile, it's just the apk (and full mar?).
>>>>>
>>>>> All of this is done by explicit steps in the recipe, and I got rid of
>>>>> dependencies as much as possible. The dependencies turned out to be primary
>>>>> pain point, as they weren't dependencies to begin with.
>>>>>
>>>>> Questions:
>>>>>
>>>>> Where to put multilocale.json, in en-US builds, and in multi-locale
>>>>> builds, and in single-locale repacks?
>>>>>
>>>>> Do we need the XPI_ROOT_APPID='$(XPI_ROOT_APPID)' jazz? That makes the
>>>>> recipes way more complex than I think they should be. Some conditional
>>>>> hacks that we have depending on this could be replaced with
>>>>> IS_LANGUAGE_REPACK ?
>>>>>
>>>>
>>>> XPI_ROOT_APPID feels like something that should be defined once in the
>>>> entrypoint and shouldn't need to be passed around everywhere. But I have no
>>>> clue what all is keyed off of it.
>>>>
>>>>
>>>>>
>>>>> What do we really need to do to support l10n repacks on artifact builds?
>>>>> The build backend code is above me, sadly.
>>>>>
>>>>
>>>> Conceptually, an artifact build is a glorified cache. At the top of the
>>>> build we extract files from a previous build into their normal locations in
>>>> the objdir. And we disable all the build rules that would produce those
>>>> files.
>>>>
>>>> If l10n repacks were implemented as a post-build action that didn't
>>>> interfere with existing files (e.g. didn't delete binaries or other files
>>>> from the objdir it didn't itself produce), I would expect l10n repacks to
>>>> be compatible with artifact builds. The work to actually make them
>>>> compatible is likely a bunch of configure muckery and ensuring the artifact
>>>> build backend has enough context to let l10n repacks work. I wouldn't be
>>>> surprised if we were disabling things required by l10n repacks.
>>>
>>> I, for one, think l10n repacks should be deprecated in favor of artifact
>>> builds. Because, really, when you take a step back, l10n repacks are
>>> just a special kind of artifact builds where the strings data is taken
>>> from a different place.
>>>
>>
>> In the light of https://bugzilla.mozilla.org/show_bug.cgi?id=1352595#c38
>> (brotli and artifact builds), I actually strongly disagree with that
>> sentiment.
>>
>> Our localized builds should have all the optimizations we have for en-US
>> release builds, and I read your comment in the bug that that's not the case
>> for artifact builds.
>
> That's not the case for artifact builds *today*. Because the brotli
> support was made partial on purpose because, what's the point of
> implementing the whole thing if we're not going to do it in the end?

Yet, the purpose of artifact builds and l10n repacks are rather
different. Artifact builds (I guess) should make building Firefox
locally easy for developers. L10n repacks minimize the risk of shipping
builds in 100 languages on 5 operating systems, and minimizing the time
it takes.

As much as I like synergies, trying to shoehorn one into the other
doesn't sound right to me.

All of that is said with minimal experience on artifact builds how they
work today, and no idea on how they're expected to work in the long run.

Axel
_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: localized builds, status and questions

Dustin Mitchell
Since artifact builds came up, I want to make a distinction:

As Axel said, artifact bulids are something that run on laptops and
help us get local test results faster, making a tighter
edit-build-test loop.  For automation, we have an optimization process
that can do the same thing, but in a more reliable, inspectable, and
deterministic way, by building an explicit graph of task dependencies.
Among other advantages, this allows us to do end-to-end validation
("chain of trust") that all inputs to a Firefox build were
appropriate.

I want to avoid the dynamically searching for source material during
automation tasks.  That dependency information should be determined in
advance in the decision task.  This is not entirely the case today,
but I consider the exceptions to be bugs and would prefer not to
introduce more.

The other bits of artifact builds, around repacking binaries for
example, are perfectly suitable for automation.  If it makes sense for
the same code to repack omni.ja as to repack strings, then my
distinction doesn't apply :)

Dustin

2017-09-07 8:05 GMT-04:00 Axel Hecht <[hidden email]>:

> Am 07.09.17 um 00:45 schrieb Mike Hommey:
>>
>> On Thu, Sep 07, 2017 at 12:19:14AM +0200, Axel Hecht wrote:
>>>
>>> Am 29.08.17 um 06:16 schrieb Mike Hommey:
>>>>
>>>> On Mon, Aug 28, 2017 at 06:10:26PM -0700, Gregory Szorc wrote:
>>>>>
>>>>> On Mon, Aug 28, 2017 at 9:16 AM, Axel Hecht <[hidden email]> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> as some of you probably noticed, I've been hacking around our
>>>>>> localized
>>>>>> builds a bit over the past few weeks.
>>>>>>
>>>>>> Context:
>>>>>>
>>>>>> For https://bugzilla.mozilla.org/show_bug.cgi?id=1362617, I wonder
>>>>>> where
>>>>>> to put the generation of res/multilocale.json in a regular build.
>>>>>> And I hope that someone volunteers to tackle l10n repacks for local
>>>>>> artifact builds.
>>>>>>
>>>>>> Status:
>>>>>>
>>>>>> For desktop, we're now at the point where all that we do outside of
>>>>>> browser/locales is jar packaging, and installer packaging. For
>>>>>> android, we
>>>>>> just need to move the search stuff from mobile/locales to
>>>>>> mobile/android/locales.
>>>>>>
>>>>>> We're also only running the jar packaging once, and we're only running
>>>>>> repack-l10n.py once.
>>>>>>
>>>>>> I also think we can drop the -j1 from the entry points.
>>>>>>
>>>>>> My hope here is that now that we're only using recursive make in a
>>>>>> defined
>>>>>> scope, and that parallelism isn't a problem anymore, we should have a
>>>>>> better chance to get other backends to support repacks? Notably the
>>>>>> artifact builds one?
>>>>>>
>>>>>> The flow of an l10n repack is also constructive now. It's preparing
>>>>>> the
>>>>>> localization, possibly getting a repo, and running compare-locales for
>>>>>> l10n-merge. Then it's doing all the jar stuff, and then the
>>>>>> app-specific
>>>>>> funkyness like search.
>>>>>> Then it packages the stuff as applicable for the project. One desktop,
>>>>>> that's langpack, package, and possibly full mar. Then it creates a
>>>>>> windows
>>>>>> installer if on windows. On mobile, it's just the apk (and full mar?).
>>>>>>
>>>>>> All of this is done by explicit steps in the recipe, and I got rid of
>>>>>> dependencies as much as possible. The dependencies turned out to be
>>>>>> primary
>>>>>> pain point, as they weren't dependencies to begin with.
>>>>>>
>>>>>> Questions:
>>>>>>
>>>>>> Where to put multilocale.json, in en-US builds, and in multi-locale
>>>>>> builds, and in single-locale repacks?
>>>>>>
>>>>>> Do we need the XPI_ROOT_APPID='$(XPI_ROOT_APPID)' jazz? That makes the
>>>>>> recipes way more complex than I think they should be. Some conditional
>>>>>> hacks that we have depending on this could be replaced with
>>>>>> IS_LANGUAGE_REPACK ?
>>>>>>
>>>>>
>>>>> XPI_ROOT_APPID feels like something that should be defined once in the
>>>>> entrypoint and shouldn't need to be passed around everywhere. But I
>>>>> have no
>>>>> clue what all is keyed off of it.
>>>>>
>>>>>
>>>>>>
>>>>>> What do we really need to do to support l10n repacks on artifact
>>>>>> builds?
>>>>>> The build backend code is above me, sadly.
>>>>>>
>>>>>
>>>>> Conceptually, an artifact build is a glorified cache. At the top of the
>>>>> build we extract files from a previous build into their normal
>>>>> locations in
>>>>> the objdir. And we disable all the build rules that would produce those
>>>>> files.
>>>>>
>>>>> If l10n repacks were implemented as a post-build action that didn't
>>>>> interfere with existing files (e.g. didn't delete binaries or other
>>>>> files
>>>>> from the objdir it didn't itself produce), I would expect l10n repacks
>>>>> to
>>>>> be compatible with artifact builds. The work to actually make them
>>>>> compatible is likely a bunch of configure muckery and ensuring the
>>>>> artifact
>>>>> build backend has enough context to let l10n repacks work. I wouldn't
>>>>> be
>>>>> surprised if we were disabling things required by l10n repacks.
>>>>
>>>>
>>>> I, for one, think l10n repacks should be deprecated in favor of artifact
>>>> builds. Because, really, when you take a step back, l10n repacks are
>>>> just a special kind of artifact builds where the strings data is taken
>>>> from a different place.
>>>>
>>>
>>> In the light of https://bugzilla.mozilla.org/show_bug.cgi?id=1352595#c38
>>> (brotli and artifact builds), I actually strongly disagree with that
>>> sentiment.
>>>
>>> Our localized builds should have all the optimizations we have for en-US
>>> release builds, and I read your comment in the bug that that's not the
>>> case
>>> for artifact builds.
>>
>>
>> That's not the case for artifact builds *today*. Because the brotli
>> support was made partial on purpose because, what's the point of
>> implementing the whole thing if we're not going to do it in the end?
>
>
> Yet, the purpose of artifact builds and l10n repacks are rather different.
> Artifact builds (I guess) should make building Firefox locally easy for
> developers. L10n repacks minimize the risk of shipping builds in 100
> languages on 5 operating systems, and minimizing the time it takes.
>
> As much as I like synergies, trying to shoehorn one into the other doesn't
> sound right to me.
>
> All of that is said with minimal experience on artifact builds how they work
> today, and no idea on how they're expected to work in the long run.
>
>
> Axel
> _______________________________________________
> dev-builds mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-builds
_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: localized builds, status and questions

Gregory Szorc-3
In reply to this post by Axel Hecht
On Thu, Sep 7, 2017 at 5:05 AM, Axel Hecht <[hidden email]> wrote:
Am 07.09.17 um 00:45 schrieb Mike Hommey:
On Thu, Sep 07, 2017 at 12:19:14AM +0200, Axel Hecht wrote:
Am 29.08.17 um 06:16 schrieb Mike Hommey:
On Mon, Aug 28, 2017 at 06:10:26PM -0700, Gregory Szorc wrote:
On Mon, Aug 28, 2017 at 9:16 AM, Axel Hecht <[hidden email]> wrote:

Hi,

as some of you probably noticed, I've been hacking around our localized
builds a bit over the past few weeks.

Context:

For https://bugzilla.mozilla.org/show_bug.cgi?id=1362617, I wonder where
to put the generation of res/multilocale.json in a regular build.
And I hope that someone volunteers to tackle l10n repacks for local
artifact builds.

Status:

For desktop, we're now at the point where all that we do outside of
browser/locales is jar packaging, and installer packaging. For android, we
just need to move the search stuff from mobile/locales to
mobile/android/locales.

We're also only running the jar packaging once, and we're only running
repack-l10n.py once.

I also think we can drop the -j1 from the entry points.

My hope here is that now that we're only using recursive make in a defined
scope, and that parallelism isn't a problem anymore, we should have a
better chance to get other backends to support repacks? Notably the
artifact builds one?

The flow of an l10n repack is also constructive now. It's preparing the
localization, possibly getting a repo, and running compare-locales for
l10n-merge. Then it's doing all the jar stuff, and then the app-specific
funkyness like search.
Then it packages the stuff as applicable for the project. One desktop,
that's langpack, package, and possibly full mar. Then it creates a windows
installer if on windows. On mobile, it's just the apk (and full mar?).

All of this is done by explicit steps in the recipe, and I got rid of
dependencies as much as possible. The dependencies turned out to be primary
pain point, as they weren't dependencies to begin with.

Questions:

Where to put multilocale.json, in en-US builds, and in multi-locale
builds, and in single-locale repacks?

Do we need the XPI_ROOT_APPID='$(XPI_ROOT_APPID)' jazz? That makes the
recipes way more complex than I think they should be. Some conditional
hacks that we have depending on this could be replaced with
IS_LANGUAGE_REPACK ?


XPI_ROOT_APPID feels like something that should be defined once in the
entrypoint and shouldn't need to be passed around everywhere. But I have no
clue what all is keyed off of it.



What do we really need to do to support l10n repacks on artifact builds?
The build backend code is above me, sadly.


Conceptually, an artifact build is a glorified cache. At the top of the
build we extract files from a previous build into their normal locations in
the objdir. And we disable all the build rules that would produce those
files.

If l10n repacks were implemented as a post-build action that didn't
interfere with existing files (e.g. didn't delete binaries or other files
from the objdir it didn't itself produce), I would expect l10n repacks to
be compatible with artifact builds. The work to actually make them
compatible is likely a bunch of configure muckery and ensuring the artifact
build backend has enough context to let l10n repacks work. I wouldn't be
surprised if we were disabling things required by l10n repacks.

I, for one, think l10n repacks should be deprecated in favor of artifact
builds. Because, really, when you take a step back, l10n repacks are
just a special kind of artifact builds where the strings data is taken
from a different place.


In the light of https://bugzilla.mozilla.org/show_bug.cgi?id=1352595#c38
(brotli and artifact builds), I actually strongly disagree with that
sentiment.

Our localized builds should have all the optimizations we have for en-US
release builds, and I read your comment in the bug that that's not the case
for artifact builds.

That's not the case for artifact builds *today*. Because the brotli
support was made partial on purpose because, what's the point of
implementing the whole thing if we're not going to do it in the end?

Yet, the purpose of artifact builds and l10n repacks are rather different. Artifact builds (I guess) should make building Firefox locally easy for developers. L10n repacks minimize the risk of shipping builds in 100 languages on 5 operating systems, and minimizing the time it takes.

As much as I like synergies, trying to shoehorn one into the other doesn't sound right to me.

All of that is said with minimal experience on artifact builds how they work today, and no idea on how they're expected to work in the long run.

Speaking as a build system maintainer, something that quacks like artifact builds is the best solution for localized builds for many reasons. Does it have the exact mechanics of artifact builds today, no. But the principal is the same: obtain a pre-built default/en-US build and morph it into a localized build. We can and should make this simple to do via `mach build`. But under the hood it will likely create a default en-US build and then do post-build processing to produce the appropriate localized build. Having hooks in the core of the build system to support various locales is a lot of complexity. It is much simpler to implement and maintain localized builds if we think of them as a stage in the pipeline performed after the main build.

_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds