Tracking Python 3 for Firefox tooling

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Tracking Python 3 for Firefox tooling

Gregory Szorc-3
In the past few weeks, a few independent projects have inquired about using Python 3 for Firefox build/development tooling.

tl;dr we now have bug 1388447 (aliased as "buildpython3") tracking everything Python 3 related to Firefox development. Please use it.

My thoughts on Python 3 and Firefox follow.

Python 2 will stop being supported in 2020 (exact date TBD). In theory, there will be a 2.7.X release in 2020 and no more official Python 2 releases will be made. It is therefore prudent for any actively-developed Python code to transition away from Python 2 so it remains on a supported programming language. Python 2 won't disappear over night and is likely to be a fixture of operating system installs for years to come. But as time goes by, it will be more and more annoying having to support legacy Python 2 code. Any developer who spends a lot of time writing Python will likely scoff at the thought of maintaining Python 2 code after they are accustomed to Python 3. Python 2 code will tend to rot from neglect. And, running Python 2 code will likely expose you to unpatched security vulnerabilities. It's best to be off of Python 2 by its end of life in 2020.

That being said, you don't need to transition to Python 3 and nobody is forcing you to transition to Python 3. While there are gains from transitioning, it is a lot of work. In many cases, porting is of the same scope as a rewrite and therefore many of the reasons why rewrites are a bad idea apply to porting to Python 3. So it would be prudent to conduct a cost-benefit analysis on porting, paying special attention to what other work is being neglected because a port is being done.

IMO Python 3 is a superior programming language to Python 2. The run-time performance of Python 3 is substantially better than Python 2 as of the 3.6 release (although start-up time is still an issue). There are compelling Python 3 features that make it attractive. These include support for type hinting, which combined with a type aware editor like PyCharm (which Mozilla employees can get a license for), should facilitate easier hacking on large code bases like the Firefox build system. Asyncio allows much higher concurrency for I/O bound tasks (which there are several of in the build system and tools). And there are a ton of standard library improvements that reduce code complexity, attain perf wins, lower development overhead, etc. In aggregate, these make large, actively developed, performance sensitive code the most lucrative targets to port.

We should be encouraging the use of Python 3 for Firefox tooling. I dare say that new Python code should strive to be Python 3 only if possible, as new Python 2 in 2017 is a form of technical debt. At the very least, I encourage all newly-written Python code to be as Python 3 compatible as possible to minimize future porting work.

Requiring Python 3 to build Firefox incurs a non-trivial amount of work, especially to downstream packagers. Bug 636155 tracks *requiring* Python 3 to build Firefox. A summary of my thoughts are captured in comment #9. That milestone likely won't happen until 2018Q2 or later.

However, we don't need to require Python 3 to build Firefox in order to use Python 3 in Firefox tooling. Non-essential development activities can adopt Python 3 sooner. Examples of non-essential development activities include mach commands not used by the build system, such as `mach watch`. Strictly speaking, I even consider testing harnesses non-essential, as they aren't required to *build*. (Although I'm not sure how much testing downstream packagers do and we need to be cognizant about introducing a Python 3 requirement on them.)

Bug 1385381 recently landed to detect Python 3 in the build system. The PYTHON3 subst variable, if defined, will point to a Python 3.5+ executable. The MozbuildObject base class now exposes a `python3` attribute. It returns a 2-tuple of (path, version). So, from mach command classes inheriting from MachCommandBase and various build system contexts, you can now do something like:

    python3 = self.python3[0]
    if not python3:
        raise Exception('Python 3 not found')
    subprocess.check_call([python3, ...])

Mach itself is still Python 2 and likely will be for some time. There have been some discussions around making it work with Python 3. We may even facilitate light magic to allow mach @Command functions to transparently be invoked as Python 3. But for now, you'll have to invoke a separate Python 3 process from mach commands to use Python 3.

There is a group interested in porting various Mozilla packages (like mozbase) so they are Python 3 compatible. Porting things like mozprocess will be a prerequisite for making large amounts of the Python in the Firefox repo Python 3 compatible.

Managing a flag day transition to Python 3 across the entire Firefox code base is not feasible: it is just too much work. The conversion to Python 3 will be gradual and take years. Expect Python 2 and 3 to run side-by-side for years, well past 2020. Components will adopt different schedules and priorities for the transition. Early adopters of Python 3 will likely be hindered by various package dependencies not yet working on Python 3. For this reason, early Python 3 efforts should be focused on shared packages like mozbase and new code that doesn't need too much of the in-tree Python (which is likely unported).

Because Python 2 and 3 will run side-by-side for years, shared packages should strive to run on both Python 2 and 3. This is achievable in many cases. Packages like six do a great job of wallpapering over differences. In cases where the same code can't run on both 2 and 3 simultaneously, we can look at using source rewriting. I have some experience with this and you can reach out if you have any questions.

Again, nobody is forcing you to use or port to Python 3. But Python 3 is the future and anyone writing or maintaining a significant amount of Python should be cognizant of the technical debt being incurred by continuing to write Python 2 in 2017 and beyond. We should start thinking "Python 3 first" and start planning for a world that is "Python 3 everywhere." For most of you, it should be business as usual for the next several months. But once more Python 3 "infrastructure" is in place in Firefox tooling, expect the calls for Python 3 to grow louder. There's no stopping the Python 3 train. So please be aware that it's coming.

_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: Tracking Python 3 for Firefox tooling

Dustin Mitchell
This feels like a project that could benefit from a number of people coordinating a slow hacking-away at it, with the leaf nodes of that dependency tree being good bite-sized projects suitable for hacking on in spare cycles or for casual contributors.  In my experience with other projects, one of the important pieces to making forward progress on Py3 migrations is to use automated testing to prevent regressions. That requires chasing specific milestones like "all print statements removed and lint configured to reject them".

Are there plans for some kind of coordination?  I'd like to be involved hacking on those leaf nodes, but I don't have the breadth of understanding to figure out where to start.

Dustin

2017-08-16 12:54 GMT-04:00 Gregory Szorc <[hidden email]>:
In the past few weeks, a few independent projects have inquired about using Python 3 for Firefox build/development tooling.

tl;dr we now have bug 1388447 (aliased as "buildpython3") tracking everything Python 3 related to Firefox development. Please use it.

My thoughts on Python 3 and Firefox follow.

Python 2 will stop being supported in 2020 (exact date TBD). In theory, there will be a 2.7.X release in 2020 and no more official Python 2 releases will be made. It is therefore prudent for any actively-developed Python code to transition away from Python 2 so it remains on a supported programming language. Python 2 won't disappear over night and is likely to be a fixture of operating system installs for years to come. But as time goes by, it will be more and more annoying having to support legacy Python 2 code. Any developer who spends a lot of time writing Python will likely scoff at the thought of maintaining Python 2 code after they are accustomed to Python 3. Python 2 code will tend to rot from neglect. And, running Python 2 code will likely expose you to unpatched security vulnerabilities. It's best to be off of Python 2 by its end of life in 2020.

That being said, you don't need to transition to Python 3 and nobody is forcing you to transition to Python 3. While there are gains from transitioning, it is a lot of work. In many cases, porting is of the same scope as a rewrite and therefore many of the reasons why rewrites are a bad idea apply to porting to Python 3. So it would be prudent to conduct a cost-benefit analysis on porting, paying special attention to what other work is being neglected because a port is being done.

IMO Python 3 is a superior programming language to Python 2. The run-time performance of Python 3 is substantially better than Python 2 as of the 3.6 release (although start-up time is still an issue). There are compelling Python 3 features that make it attractive. These include support for type hinting, which combined with a type aware editor like PyCharm (which Mozilla employees can get a license for), should facilitate easier hacking on large code bases like the Firefox build system. Asyncio allows much higher concurrency for I/O bound tasks (which there are several of in the build system and tools). And there are a ton of standard library improvements that reduce code complexity, attain perf wins, lower development overhead, etc. In aggregate, these make large, actively developed, performance sensitive code the most lucrative targets to port.

We should be encouraging the use of Python 3 for Firefox tooling. I dare say that new Python code should strive to be Python 3 only if possible, as new Python 2 in 2017 is a form of technical debt. At the very least, I encourage all newly-written Python code to be as Python 3 compatible as possible to minimize future porting work.

Requiring Python 3 to build Firefox incurs a non-trivial amount of work, especially to downstream packagers. Bug 636155 tracks *requiring* Python 3 to build Firefox. A summary of my thoughts are captured in comment #9. That milestone likely won't happen until 2018Q2 or later.

However, we don't need to require Python 3 to build Firefox in order to use Python 3 in Firefox tooling. Non-essential development activities can adopt Python 3 sooner. Examples of non-essential development activities include mach commands not used by the build system, such as `mach watch`. Strictly speaking, I even consider testing harnesses non-essential, as they aren't required to *build*. (Although I'm not sure how much testing downstream packagers do and we need to be cognizant about introducing a Python 3 requirement on them.)

Bug 1385381 recently landed to detect Python 3 in the build system. The PYTHON3 subst variable, if defined, will point to a Python 3.5+ executable. The MozbuildObject base class now exposes a `python3` attribute. It returns a 2-tuple of (path, version). So, from mach command classes inheriting from MachCommandBase and various build system contexts, you can now do something like:

    python3 = self.python3[0]
    if not python3:
        raise Exception('Python 3 not found')
    subprocess.check_call([python3, ...])

Mach itself is still Python 2 and likely will be for some time. There have been some discussions around making it work with Python 3. We may even facilitate light magic to allow mach @Command functions to transparently be invoked as Python 3. But for now, you'll have to invoke a separate Python 3 process from mach commands to use Python 3.

There is a group interested in porting various Mozilla packages (like mozbase) so they are Python 3 compatible. Porting things like mozprocess will be a prerequisite for making large amounts of the Python in the Firefox repo Python 3 compatible.

Managing a flag day transition to Python 3 across the entire Firefox code base is not feasible: it is just too much work. The conversion to Python 3 will be gradual and take years. Expect Python 2 and 3 to run side-by-side for years, well past 2020. Components will adopt different schedules and priorities for the transition. Early adopters of Python 3 will likely be hindered by various package dependencies not yet working on Python 3. For this reason, early Python 3 efforts should be focused on shared packages like mozbase and new code that doesn't need too much of the in-tree Python (which is likely unported).

Because Python 2 and 3 will run side-by-side for years, shared packages should strive to run on both Python 2 and 3. This is achievable in many cases. Packages like six do a great job of wallpapering over differences. In cases where the same code can't run on both 2 and 3 simultaneously, we can look at using source rewriting. I have some experience with this and you can reach out if you have any questions.

Again, nobody is forcing you to use or port to Python 3. But Python 3 is the future and anyone writing or maintaining a significant amount of Python should be cognizant of the technical debt being incurred by continuing to write Python 2 in 2017 and beyond. We should start thinking "Python 3 first" and start planning for a world that is "Python 3 everywhere." For most of you, it should be business as usual for the next several months. But once more Python 3 "infrastructure" is in place in Firefox tooling, expect the calls for Python 3 to grow louder. There's no stopping the Python 3 train. So please be aware that it's coming.

_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds



_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: Tracking Python 3 for Firefox tooling

Alex Gaynor-2
A great fire milestone is likely getting python3 -mcompileall python/ clean. It looks like there's a couple dozen syntax errors that I bet could be knocked out in an hour.

Alex

On Wed, Aug 16, 2017 at 1:15 PM, Dustin Mitchell <[hidden email]> wrote:
This feels like a project that could benefit from a number of people
coordinating a slow hacking-away at it, with the leaf nodes of that
dependency tree being good bite-sized projects suitable for hacking on in
spare cycles or for casual contributors.  In my experience with other
projects, one of the important pieces to making forward progress on Py3
migrations is to use automated testing to prevent regressions. That
requires chasing specific milestones like "all print statements removed and
lint configured to reject them".

Are there plans for some kind of coordination?  I'd like to be involved
hacking on those leaf nodes, but I don't have the breadth of understanding
to figure out where to start.

Dustin

2017-08-16 12:54 GMT-04:00 Gregory Szorc <[hidden email]>:

> In the past few weeks, a few independent projects have inquired about
> using Python 3 for Firefox build/development tooling.
>
> tl;dr we now have bug 1388447 (aliased as "buildpython3") tracking
> everything Python 3 related to Firefox development. Please use it.
>
> My thoughts on Python 3 and Firefox follow.
>
> Python 2 will stop being supported in 2020 (exact date TBD). In theory,
> there will be a 2.7.X release in 2020 and no more official Python 2
> releases will be made. It is therefore prudent for any actively-developed
> Python code to transition away from Python 2 so it remains on a supported
> programming language. Python 2 won't disappear over night and is likely to
> be a fixture of operating system installs for years to come. But as time
> goes by, it will be more and more annoying having to support legacy Python
> 2 code. Any developer who spends a lot of time writing Python will likely
> scoff at the thought of maintaining Python 2 code after they are accustomed
> to Python 3. Python 2 code will tend to rot from neglect. And, running
> Python 2 code will likely expose you to unpatched security vulnerabilities.
> It's best to be off of Python 2 by its end of life in 2020.
>
> That being said, you don't need to transition to Python 3 and nobody is
> forcing you to transition to Python 3. While there are gains from
> transitioning, it is a lot of work. In many cases, porting is of the same
> scope as a rewrite and therefore many of the reasons why rewrites are a bad
> idea apply to porting to Python 3. So it would be prudent to conduct a
> cost-benefit analysis on porting, paying special attention to what other
> work is being neglected because a port is being done.
>
> IMO Python 3 is a superior programming language to Python 2. The run-time
> performance of Python 3 is substantially better than Python 2 as of the 3.6
> release (although start-up time is still an issue). There are compelling
> Python 3 features that make it attractive. These include support for type
> hinting, which combined with a type aware editor like PyCharm (which
> Mozilla employees can get a license for), should facilitate easier hacking
> on large code bases like the Firefox build system. Asyncio allows much
> higher concurrency for I/O bound tasks (which there are several of in the
> build system and tools). And there are a ton of standard library
> improvements that reduce code complexity, attain perf wins, lower
> development overhead, etc. In aggregate, these make large, actively
> developed, performance sensitive code the most lucrative targets to port.
>
> We should be encouraging the use of Python 3 for Firefox tooling. I dare
> say that new Python code should strive to be Python 3 only if possible, as
> new Python 2 in 2017 is a form of technical debt. At the very least, I
> encourage all newly-written Python code to be as Python 3 compatible as
> possible to minimize future porting work.
>
> Requiring Python 3 to build Firefox incurs a non-trivial amount of work,
> especially to downstream packagers. Bug 636155 tracks *requiring* Python 3
> to build Firefox. A summary of my thoughts are captured in comment #9. That
> milestone likely won't happen until 2018Q2 or later.
>
> However, we don't need to require Python 3 to build Firefox in order to
> use Python 3 in Firefox tooling. Non-essential development activities can
> adopt Python 3 sooner. Examples of non-essential development activities
> include mach commands not used by the build system, such as `mach watch`.
> Strictly speaking, I even consider testing harnesses non-essential, as they
> aren't required to *build*. (Although I'm not sure how much testing
> downstream packagers do and we need to be cognizant about introducing a
> Python 3 requirement on them.)
>
> Bug 1385381 recently landed to detect Python 3 in the build system. The
> PYTHON3 subst variable, if defined, will point to a Python 3.5+ executable.
> The MozbuildObject base class now exposes a `python3` attribute. It returns
> a 2-tuple of (path, version). So, from mach command classes inheriting from
> MachCommandBase and various build system contexts, you can now do something
> like:
>
>     python3 = self.python3[0]
>     if not python3:
>         raise Exception('Python 3 not found')
>     subprocess.check_call([python3, ...])
>
> Mach itself is still Python 2 and likely will be for some time. There have
> been some discussions around making it work with Python 3. We may even
> facilitate light magic to allow mach @Command functions to transparently be
> invoked as Python 3. But for now, you'll have to invoke a separate Python 3
> process from mach commands to use Python 3.
>
> There is a group interested in porting various Mozilla packages (like
> mozbase) so they are Python 3 compatible. Porting things like mozprocess
> will be a prerequisite for making large amounts of the Python in the
> Firefox repo Python 3 compatible.
>
> Managing a flag day transition to Python 3 across the entire Firefox code
> base is not feasible: it is just too much work. The conversion to Python 3
> will be gradual and take years. Expect Python 2 and 3 to run side-by-side
> for years, well past 2020. Components will adopt different schedules and
> priorities for the transition. Early adopters of Python 3 will likely be
> hindered by various package dependencies not yet working on Python 3. For
> this reason, early Python 3 efforts should be focused on shared packages
> like mozbase and new code that doesn't need too much of the in-tree Python
> (which is likely unported).
>
> Because Python 2 and 3 will run side-by-side for years, shared packages
> should strive to run on both Python 2 and 3. This is achievable in many
> cases. Packages like six do a great job of wallpapering over differences.
> In cases where the same code can't run on both 2 and 3 simultaneously, we
> can look at using source rewriting. I have some experience with this and
> you can reach out if you have any questions.
>
> Again, nobody is forcing you to use or port to Python 3. But Python 3 is
> the future and anyone writing or maintaining a significant amount of Python
> should be cognizant of the technical debt being incurred by continuing to
> write Python 2 in 2017 and beyond. We should start thinking "Python 3
> first" and start planning for a world that is "Python 3 everywhere." For
> most of you, it should be business as usual for the next several months.
> But once more Python 3 "infrastructure" is in place in Firefox tooling,
> expect the calls for Python 3 to grow louder. There's no stopping the
> Python 3 train. So please be aware that it's coming.
>
> _______________________________________________
> dev-builds mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-builds
>
>
_______________________________________________
tools mailing list
[hidden email]
https://lists.mozilla.org/listinfo/tools


_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: Tracking Python 3 for Firefox tooling

Gregory Szorc-3
In reply to this post by Dustin Mitchell
On Wed, Aug 16, 2017 at 10:15 AM, Dustin Mitchell <[hidden email]> wrote:
This feels like a project that could benefit from a number of people coordinating a slow hacking-away at it, with the leaf nodes of that dependency tree being good bite-sized projects suitable for hacking on in spare cycles or for casual contributors.  In my experience with other projects, one of the important pieces to making forward progress on Py3 migrations is to use automated testing to prevent regressions. That requires chasing specific milestones like "all print statements removed and lint configured to reject them".

Are there plans for some kind of coordination?  I'd like to be involved hacking on those leaf nodes, but I don't have the breadth of understanding to figure out where to start.

We had a meeting a few weeks ago and we definitely want to establish some kind of "lint" test that ensures basic Python 3 conformance once a package/module has achieved it. Bug 1388013 and associated bugs are starting to lay the groundwork for running Python unit tests with Python 3. Once we're doing that, it shouldn't be too much effort to write a test for Python 3 conformance.

FWIW, the general process Mercurial took for Python 3 compatibility looked like:

1) Make all .py files tokenize/ast compile on Python 3
2) Make all .py files import cleanly on Python 3
3) Make all .py files run properly on Python 3

I think that worked out pretty well and recommend repeating the pattern at Mozilla. Having a Python module import cleanly unblocks usage of just a single function in that module from Python 3, which for many consumers is all you need. In terms of tests, we'll want to establish "gates" where once a module passes a tokenize/ast or import threshold, it stays that way.
 
2017-08-16 12:54 GMT-04:00 Gregory Szorc <[hidden email]>:
In the past few weeks, a few independent projects have inquired about using Python 3 for Firefox build/development tooling.

tl;dr we now have bug 1388447 (aliased as "buildpython3") tracking everything Python 3 related to Firefox development. Please use it.

My thoughts on Python 3 and Firefox follow.

Python 2 will stop being supported in 2020 (exact date TBD). In theory, there will be a 2.7.X release in 2020 and no more official Python 2 releases will be made. It is therefore prudent for any actively-developed Python code to transition away from Python 2 so it remains on a supported programming language. Python 2 won't disappear over night and is likely to be a fixture of operating system installs for years to come. But as time goes by, it will be more and more annoying having to support legacy Python 2 code. Any developer who spends a lot of time writing Python will likely scoff at the thought of maintaining Python 2 code after they are accustomed to Python 3. Python 2 code will tend to rot from neglect. And, running Python 2 code will likely expose you to unpatched security vulnerabilities. It's best to be off of Python 2 by its end of life in 2020.

That being said, you don't need to transition to Python 3 and nobody is forcing you to transition to Python 3. While there are gains from transitioning, it is a lot of work. In many cases, porting is of the same scope as a rewrite and therefore many of the reasons why rewrites are a bad idea apply to porting to Python 3. So it would be prudent to conduct a cost-benefit analysis on porting, paying special attention to what other work is being neglected because a port is being done.

IMO Python 3 is a superior programming language to Python 2. The run-time performance of Python 3 is substantially better than Python 2 as of the 3.6 release (although start-up time is still an issue). There are compelling Python 3 features that make it attractive. These include support for type hinting, which combined with a type aware editor like PyCharm (which Mozilla employees can get a license for), should facilitate easier hacking on large code bases like the Firefox build system. Asyncio allows much higher concurrency for I/O bound tasks (which there are several of in the build system and tools). And there are a ton of standard library improvements that reduce code complexity, attain perf wins, lower development overhead, etc. In aggregate, these make large, actively developed, performance sensitive code the most lucrative targets to port.

We should be encouraging the use of Python 3 for Firefox tooling. I dare say that new Python code should strive to be Python 3 only if possible, as new Python 2 in 2017 is a form of technical debt. At the very least, I encourage all newly-written Python code to be as Python 3 compatible as possible to minimize future porting work.

Requiring Python 3 to build Firefox incurs a non-trivial amount of work, especially to downstream packagers. Bug 636155 tracks *requiring* Python 3 to build Firefox. A summary of my thoughts are captured in comment #9. That milestone likely won't happen until 2018Q2 or later.

However, we don't need to require Python 3 to build Firefox in order to use Python 3 in Firefox tooling. Non-essential development activities can adopt Python 3 sooner. Examples of non-essential development activities include mach commands not used by the build system, such as `mach watch`. Strictly speaking, I even consider testing harnesses non-essential, as they aren't required to *build*. (Although I'm not sure how much testing downstream packagers do and we need to be cognizant about introducing a Python 3 requirement on them.)

Bug 1385381 recently landed to detect Python 3 in the build system. The PYTHON3 subst variable, if defined, will point to a Python 3.5+ executable. The MozbuildObject base class now exposes a `python3` attribute. It returns a 2-tuple of (path, version). So, from mach command classes inheriting from MachCommandBase and various build system contexts, you can now do something like:

    python3 = self.python3[0]
    if not python3:
        raise Exception('Python 3 not found')
    subprocess.check_call([python3, ...])

Mach itself is still Python 2 and likely will be for some time. There have been some discussions around making it work with Python 3. We may even facilitate light magic to allow mach @Command functions to transparently be invoked as Python 3. But for now, you'll have to invoke a separate Python 3 process from mach commands to use Python 3.

There is a group interested in porting various Mozilla packages (like mozbase) so they are Python 3 compatible. Porting things like mozprocess will be a prerequisite for making large amounts of the Python in the Firefox repo Python 3 compatible.

Managing a flag day transition to Python 3 across the entire Firefox code base is not feasible: it is just too much work. The conversion to Python 3 will be gradual and take years. Expect Python 2 and 3 to run side-by-side for years, well past 2020. Components will adopt different schedules and priorities for the transition. Early adopters of Python 3 will likely be hindered by various package dependencies not yet working on Python 3. For this reason, early Python 3 efforts should be focused on shared packages like mozbase and new code that doesn't need too much of the in-tree Python (which is likely unported).

Because Python 2 and 3 will run side-by-side for years, shared packages should strive to run on both Python 2 and 3. This is achievable in many cases. Packages like six do a great job of wallpapering over differences. In cases where the same code can't run on both 2 and 3 simultaneously, we can look at using source rewriting. I have some experience with this and you can reach out if you have any questions.

Again, nobody is forcing you to use or port to Python 3. But Python 3 is the future and anyone writing or maintaining a significant amount of Python should be cognizant of the technical debt being incurred by continuing to write Python 2 in 2017 and beyond. We should start thinking "Python 3 first" and start planning for a world that is "Python 3 everywhere." For most of you, it should be business as usual for the next several months. But once more Python 3 "infrastructure" is in place in Firefox tooling, expect the calls for Python 3 to grow louder. There's no stopping the Python 3 train. So please be aware that it's coming.

_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds




_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: Tracking Python 3 for Firefox tooling

Andrew Halberstadt
On Wed, Aug 16, 2017 at 1:27 PM Gregory Szorc <[hidden email]> wrote:
FWIW, the general process Mercurial took for Python 3 compatibility looked
like:

1) Make all .py files tokenize/ast compile on Python 3
2) Make all .py files import cleanly on Python 3
3) Make all .py files run properly on Python 3

There's been some progress on the step 1) front. There will soon be two linters
running under the 'py-compat' task called py2 and py3 (mach lint -l py2 -l py3).
The py3 linter makes sure all files it checks can be parsed with python3. The py2
linter makes sure all files it checks can be parsed with python2 and additionally
enforces all files have:
from __future__ import print_function, absolute_import

in them. As of now, almost all python files are being excluded from these linters,
so please take a moment to enable these checks on the files you are responsible
for. To do this, remove your directory from the exclude lists under:
tools/lint/py2.yml
tools/lint/py3.yml

This also makes great [good-first-bug] material.
 

I think that worked out pretty well and recommend repeating the pattern at
Mozilla. Having a Python module import cleanly unblocks usage of just a
single function in that module from Python 3, which for many consumers is
all you need. In terms of tests, we'll want to establish "gates" where once
a module passes a tokenize/ast or import threshold, it stays that way.


> 2017-08-16 12:54 GMT-04:00 Gregory Szorc <[hidden email]>:
>
>> In the past few weeks, a few independent projects have inquired about
>> using Python 3 for Firefox build/development tooling.
>>
>> tl;dr we now have bug 1388447 (aliased as "buildpython3") tracking
>> everything Python 3 related to Firefox development. Please use it.
>>
>> My thoughts on Python 3 and Firefox follow.
>>
>> Python 2 will stop being supported in 2020 (exact date TBD). In theory,
>> there will be a 2.7.X release in 2020 and no more official Python 2
>> releases will be made. It is therefore prudent for any actively-developed
>> Python code to transition away from Python 2 so it remains on a supported
>> programming language. Python 2 won't disappear over night and is likely to
>> be a fixture of operating system installs for years to come. But as time
>> goes by, it will be more and more annoying having to support legacy Python
>> 2 code. Any developer who spends a lot of time writing Python will likely
>> scoff at the thought of maintaining Python 2 code after they are accustomed
>> to Python 3. Python 2 code will tend to rot from neglect. And, running
>> Python 2 code will likely expose you to unpatched security vulnerabilities.
>> It's best to be off of Python 2 by its end of life in 2020.
>>
>> That being said, you don't need to transition to Python 3 and nobody is
>> forcing you to transition to Python 3. While there are gains from
>> transitioning, it is a lot of work. In many cases, porting is of the same
>> scope as a rewrite and therefore many of the reasons why rewrites are a bad
>> idea apply to porting to Python 3. So it would be prudent to conduct a
>> cost-benefit analysis on porting, paying special attention to what other
>> work is being neglected because a port is being done.
>>
>> IMO Python 3 is a superior programming language to Python 2. The run-time
>> performance of Python 3 is substantially better than Python 2 as of the 3.6
>> release (although start-up time is still an issue). There are compelling
>> Python 3 features that make it attractive. These include support for type
>> hinting, which combined with a type aware editor like PyCharm (which
>> Mozilla employees can get a license for), should facilitate easier hacking
>> on large code bases like the Firefox build system. Asyncio allows much
>> higher concurrency for I/O bound tasks (which there are several of in the
>> build system and tools). And there are a ton of standard library
>> improvements that reduce code complexity, attain perf wins, lower
>> development overhead, etc. In aggregate, these make large, actively
>> developed, performance sensitive code the most lucrative targets to port.
>>
>> We should be encouraging the use of Python 3 for Firefox tooling. I dare
>> say that new Python code should strive to be Python 3 only if possible, as
>> new Python 2 in 2017 is a form of technical debt. At the very least, I
>> encourage all newly-written Python code to be as Python 3 compatible as
>> possible to minimize future porting work.
>>
>> Requiring Python 3 to build Firefox incurs a non-trivial amount of work,
>> especially to downstream packagers. Bug 636155 tracks *requiring* Python 3
>> to build Firefox. A summary of my thoughts are captured in comment #9. That
>> milestone likely won't happen until 2018Q2 or later.
>>
>> However, we don't need to require Python 3 to build Firefox in order to
>> use Python 3 in Firefox tooling. Non-essential development activities can
>> adopt Python 3 sooner. Examples of non-essential development activities
>> include mach commands not used by the build system, such as `mach watch`.
>> Strictly speaking, I even consider testing harnesses non-essential, as they
>> aren't required to *build*. (Although I'm not sure how much testing
>> downstream packagers do and we need to be cognizant about introducing a
>> Python 3 requirement on them.)
>>
>> Bug 1385381 recently landed to detect Python 3 in the build system. The
>> PYTHON3 subst variable, if defined, will point to a Python 3.5+ executable.
>> The MozbuildObject base class now exposes a `python3` attribute. It returns
>> a 2-tuple of (path, version). So, from mach command classes inheriting from
>> MachCommandBase and various build system contexts, you can now do something
>> like:
>>
>>     python3 = self.python3[0]
>>     if not python3:
>>         raise Exception('Python 3 not found')
>>     subprocess.check_call([python3, ...])
>>
>> Mach itself is still Python 2 and likely will be for some time. There
>> have been some discussions around making it work with Python 3. We may even
>> facilitate light magic to allow mach @Command functions to transparently be
>> invoked as Python 3. But for now, you'll have to invoke a separate Python 3
>> process from mach commands to use Python 3.
>>
>> There is a group interested in porting various Mozilla packages (like
>> mozbase) so they are Python 3 compatible. Porting things like mozprocess
>> will be a prerequisite for making large amounts of the Python in the
>> Firefox repo Python 3 compatible.
>>
>> Managing a flag day transition to Python 3 across the entire Firefox code
>> base is not feasible: it is just too much work. The conversion to Python 3
>> will be gradual and take years. Expect Python 2 and 3 to run side-by-side
>> for years, well past 2020. Components will adopt different schedules and
>> priorities for the transition. Early adopters of Python 3 will likely be
>> hindered by various package dependencies not yet working on Python 3. For
>> this reason, early Python 3 efforts should be focused on shared packages
>> like mozbase and new code that doesn't need too much of the in-tree Python
>> (which is likely unported).
>>
>> Because Python 2 and 3 will run side-by-side for years, shared packages
>> should strive to run on both Python 2 and 3. This is achievable in many
>> cases. Packages like six do a great job of wallpapering over differences.
>> In cases where the same code can't run on both 2 and 3 simultaneously, we
>> can look at using source rewriting. I have some experience with this and
>> you can reach out if you have any questions.
>>
>> Again, nobody is forcing you to use or port to Python 3. But Python 3 is
>> the future and anyone writing or maintaining a significant amount of Python
>> should be cognizant of the technical debt being incurred by continuing to
>> write Python 2 in 2017 and beyond. We should start thinking "Python 3
>> first" and start planning for a world that is "Python 3 everywhere." For
>> most of you, it should be business as usual for the next several months.
>> But once more Python 3 "infrastructure" is in place in Firefox tooling,
>> expect the calls for Python 3 to grow louder. There's no stopping the
>> Python 3 train. So please be aware that it's coming.
>>
>> _______________________________________________
>> dev-builds mailing list
>> [hidden email]
>> https://lists.mozilla.org/listinfo/dev-builds
>>
>>
>
_______________________________________________
tools mailing list
[hidden email]
https://lists.mozilla.org/listinfo/tools

_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds