PSA: |mach python-test| can now run with python 3

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

PSA: |mach python-test| can now run with python 3

Andrew Halberstadt-2
Dave Hunt has recently landed a lot of great work to get
some of our python unittests running with Python 3. You
can see his blog post on it here:

The march to Python 3 has been long and slow, but this is a
major milestone along the way. It gives us more confidence
that porting a module won't be immediately regressed again
with the next change.  Thanks Dave!

I want to emphasize that the time is ripe to start porting
any python modules you maintain to support Python 3
(in addition to 2). Here are some things you can do:

1) Enable the py2/py3 and flake8 linters on your module. You
can do this by adjusting the include/exclude paths at the
following locations:

Then run |mach lint <path/to/module>| and fix any errors that
show up (use --fix to automatically fix whatever it can).

2) Run your |mach python-test| tests under Python 3 in CI. The
first step is to remove the 'skip-if' annotation from the relevant
manifest:

Next, run |mach python-test <path/to/module> --python 3| and
fix any test failures that show up.

Finally, adjust the task (if applicable) to run with both python 2
and 3 by modifying this line in the appropriate task:

3) Write all new python code to be both Python 2 and Python 3
compatible. The 'six' library should be available in all contexts
of mozilla-central (both locally and in CI). If it isn't, please file
a bug.

Here is a handy cheat sheet you can reference to solve
incompatibilities:

4) File bugs! Even if you don't have time to do the conversion
now, get bugs on file and make them block the 'buildpython3'
meta bug:

Having bugs like these on file can also turn into great first
projects for a contributor.

There is also infrastructure work we could implement to make
the transition easier. For example, we should run the 'flake8'
linter under python 3 as well as 2. We should also make
|mach python-test| default to running tests with both.

We'll only ever get to full python 3 compatibility if everyone
buys in. While the end goal is still a long long way away, that
shouldn't stop us from starting to make some progress now.

Cheers,
Andrew



_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: PSA: |mach python-test| can now run with python 3

David Burns-9
This is really exciting news!

David

On 3 July 2018 at 16:48, Andrew Halberstadt <[hidden email]> wrote:
Dave Hunt has recently landed a lot of great work to get
some of our python unittests running with Python 3. You
can see his blog post on it here:

The march to Python 3 has been long and slow, but this is a
major milestone along the way. It gives us more confidence
that porting a module won't be immediately regressed again
with the next change.  Thanks Dave!

I want to emphasize that the time is ripe to start porting
any python modules you maintain to support Python 3
(in addition to 2). Here are some things you can do:

1) Enable the py2/py3 and flake8 linters on your module. You
can do this by adjusting the include/exclude paths at the
following locations:

Then run |mach lint <path/to/module>| and fix any errors that
show up (use --fix to automatically fix whatever it can).

2) Run your |mach python-test| tests under Python 3 in CI. The
first step is to remove the 'skip-if' annotation from the relevant
manifest:

Next, run |mach python-test <path/to/module> --python 3| and
fix any test failures that show up.

Finally, adjust the task (if applicable) to run with both python 2
and 3 by modifying this line in the appropriate task:

3) Write all new python code to be both Python 2 and Python 3
compatible. The 'six' library should be available in all contexts
of mozilla-central (both locally and in CI). If it isn't, please file
a bug.

Here is a handy cheat sheet you can reference to solve
incompatibilities:

4) File bugs! Even if you don't have time to do the conversion
now, get bugs on file and make them block the 'buildpython3'
meta bug:

Having bugs like these on file can also turn into great first
projects for a contributor.

There is also infrastructure work we could implement to make
the transition easier. For example, we should run the 'flake8'
linter under python 3 as well as 2. We should also make
|mach python-test| default to running tests with both.

We'll only ever get to full python 3 compatibility if everyone
buys in. While the end goal is still a long long way away, that
shouldn't stop us from starting to make some progress now.

Cheers,
Andrew



_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds



_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: PSA: |mach python-test| can now run with python 3

Axel Hecht
In reply to this post by Andrew Halberstadt-2
Nice.

I'm just devoting my Friday productivity to fixing bustages in python
3.7 compared to 3.6. Which affect code I copied from mozpack.path ;-) [0]

Which leads me to: We should run tests in automation for at least 3.7
and 3.6. Not sure what still ships 3.5, if at all?

Testing some version of 3.x sadly isn't covering all our bases.

Axel

[0] https://bugzilla.mozilla.org/show_bug.cgi?id=1473847 for my copy

Am 03.07.18 um 17:48 schrieb Andrew Halberstadt:

> Dave Hunt has recently landed a lot of great work to get
> some of our python unittests running with Python 3. You
> can see his blog post on it here:
> http://davehunt.co.uk/2018/06/29/python-unit-tests-now-running-with-python-3-at-mozilla.html
>
> The march to Python 3 has been long and slow, but this is a
> major milestone along the way. It gives us more confidence
> that porting a module won't be immediately regressed again
> with the next change.  Thanks Dave!
>
> I want to emphasize that the time is ripe to start porting
> any python modules you maintain to support Python 3
> (in addition to 2). Here are some things you can do:
>
> 1) Enable the py2/py3 and flake8 linters on your module. You
> can do this by adjusting the include/exclude paths at the
> following locations:
> https://searchfox.org/mozilla-central/source/tools/lint/py2.yml
> https://searchfox.org/mozilla-central/source/tools/lint/py3.yml
> https://searchfox.org/mozilla-central/source/tools/lint/flake8.yml
>
> Then run |mach lint <path/to/module>| and fix any errors that
> show up (use --fix to automatically fix whatever it can).
>
> 2) Run your |mach python-test| tests under Python 3 in CI. The
> first step is to remove the 'skip-if' annotation from the relevant
> manifest:
> https://searchfox.org/mozilla-central/search?q=skip-if.*python+%3F%3D%3D+%3F3&case=true&regexp=true&path=ini
>
> Next, run |mach python-test <path/to/module> --python 3| and
> fix any test failures that show up.
>
> Finally, adjust the task (if applicable) to run with both python 2
> and 3 by modifying this line in the appropriate task:
> https://searchfox.org/mozilla-central/source/taskcluster/ci/source-test/python.yml#94
>
> 3) Write all new python code to be both Python 2 and Python 3
> compatible. The 'six' library should be available in all contexts
> of mozilla-central (both locally and in CI). If it isn't, please file
> a bug.
>
> Here is a handy cheat sheet you can reference to solve
> incompatibilities:
> http://python-future.org/compatible_idioms.html
>
> 4) File bugs! Even if you don't have time to do the conversion
> now, get bugs on file and make them block the 'buildpython3'
> meta bug:
> https://bugzilla.mozilla.org/show_bug.cgi?id=1388447
>
> Having bugs like these on file can also turn into great first
> projects for a contributor.
>
> There is also infrastructure work we could implement to make
> the transition easier. For example, we should run the 'flake8'
> linter under python 3 as well as 2. We should also make
> |mach python-test| default to running tests with both.
>
> We'll only ever get to full python 3 compatibility if everyone
> buys in. While the end goal is still a long long way away, that
> shouldn't stop us from starting to make some progress now.
>
> Cheers,
> Andrew
>
>

_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: PSA: |mach python-test| can now run with python 3

Andrew Halberstadt-2
That bug makes me sad.

Right now the tests just use whatever version of python 3 is installed
on the system. Expanding them to run on multiple versions of python 3
would likely mean doing our own bootstrapping with pyenv or the like.

Tbh, I think our efforts would be better spent getting things working on
python 3 *at all* before worrying too much about backwards
incompatible changes between minor releases. The case you found
shouldn't really be happening, hopefully there aren't many other cases
like it.

Andrew

p.s AIUI the build system is still going to require 3.5, at least to start. So
we likely won't be able to use 3.6+ features in many of these modules
under test anyway.

On Fri, Jul 6, 2018 at 6:15 AM Axel Hecht <[hidden email]> wrote:
Nice.

I'm just devoting my Friday productivity to fixing bustages in python
3.7 compared to 3.6. Which affect code I copied from mozpack.path ;-) [0]

Which leads me to: We should run tests in automation for at least 3.7
and 3.6. Not sure what still ships 3.5, if at all?

Testing some version of 3.x sadly isn't covering all our bases.

Axel

[0] https://bugzilla.mozilla.org/show_bug.cgi?id=1473847 for my copy

Am 03.07.18 um 17:48 schrieb Andrew Halberstadt:
> Dave Hunt has recently landed a lot of great work to get
> some of our python unittests running with Python 3. You
> can see his blog post on it here:
> http://davehunt.co.uk/2018/06/29/python-unit-tests-now-running-with-python-3-at-mozilla.html
>
> The march to Python 3 has been long and slow, but this is a
> major milestone along the way. It gives us more confidence
> that porting a module won't be immediately regressed again
> with the next change.  Thanks Dave!
>
> I want to emphasize that the time is ripe to start porting
> any python modules you maintain to support Python 3
> (in addition to 2). Here are some things you can do:
>
> 1) Enable the py2/py3 and flake8 linters on your module. You
> can do this by adjusting the include/exclude paths at the
> following locations:
> https://searchfox.org/mozilla-central/source/tools/lint/py2.yml
> https://searchfox.org/mozilla-central/source/tools/lint/py3.yml
> https://searchfox.org/mozilla-central/source/tools/lint/flake8.yml
>
> Then run |mach lint <path/to/module>| and fix any errors that
> show up (use --fix to automatically fix whatever it can).
>
> 2) Run your |mach python-test| tests under Python 3 in CI. The
> first step is to remove the 'skip-if' annotation from the relevant
> manifest:
> https://searchfox.org/mozilla-central/search?q=skip-if.*python+%3F%3D%3D+%3F3&case=true&regexp=true&path=ini
>
> Next, run |mach python-test <path/to/module> --python 3| and
> fix any test failures that show up.
>
> Finally, adjust the task (if applicable) to run with both python 2
> and 3 by modifying this line in the appropriate task:
> https://searchfox.org/mozilla-central/source/taskcluster/ci/source-test/python.yml#94
>
> 3) Write all new python code to be both Python 2 and Python 3
> compatible. The 'six' library should be available in all contexts
> of mozilla-central (both locally and in CI). If it isn't, please file
> a bug.
>
> Here is a handy cheat sheet you can reference to solve
> incompatibilities:
> http://python-future.org/compatible_idioms.html
>
> 4) File bugs! Even if you don't have time to do the conversion
> now, get bugs on file and make them block the 'buildpython3'
> meta bug:
> https://bugzilla.mozilla.org/show_bug.cgi?id=1388447
>
> Having bugs like these on file can also turn into great first
> projects for a contributor.
>
> There is also infrastructure work we could implement to make
> the transition easier. For example, we should run the 'flake8'
> linter under python 3 as well as 2. We should also make
> |mach python-test| default to running tests with both.
>
> We'll only ever get to full python 3 compatibility if everyone
> buys in. While the end goal is still a long long way away, that
> shouldn't stop us from starting to make some progress now.
>
> Cheers,
> Andrew
>
>

_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds

_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: PSA: |mach python-test| can now run with python 3

Dustin Mitchell
I agree -- we faced similar issues in the days when py2.5 - 2.7 were
still available.

I think that if we're clear about the Python version we test on in CI,
and willing to make the effort to fix bugs when we change that (e.g.,
to 3.8), that's sufficient.

Dustin
_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: PSA: |mach python-test| can now run with python 3

Axel Hecht
In reply to this post by Andrew Halberstadt-2
Am 06.07.18 um 16:04 schrieb Dustin Mitchell:
> I agree -- we faced similar issues in the days when py2.5 - 2.7 were
> still available.
>
> I think that if we're clear about the Python version we test on in CI,
> and willing to make the effort to fix bugs when we change that (e.g.,
> to 3.8), that's sufficient.
>
> Dustin
>

So, if you thought that story was sad ...

brew update
brew upgrade something-else

something-else depended on python, brew updated python from 3.6.5 to
3.7.0. I did intend to not update python deliberately, but no luck.

/usr/local/Cellar/python/3.6.5/bin/python can't be used for virtualenvs,
'cause sys.executable points to /usr/local/opt/python/bin/python3.6,
which doesn't exist anymore.

... so on mac/homebrew, there's very little power over which python you
end up using.

I do think we need to run tests on at least 2.7.latest and
3.[5-7].latest. For now, the 3.x ones can probably be tier 2-3, but we
need visibility into them.

Axel
_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: PSA: |mach python-test| can now run with python 3

Andrew Halberstadt-2
I was about to double-down on my earlier assertion that it wasn't
worth it at this point in time (while agreeing that we would do it in
the ideal case).

But it looks like Pipenv automatically installs a version of python
via pyenv already if you specify it with --python. E.g:
pipenv --python 3.7 install <package>

First downloads and compiles python 3.7 if it doesn't exist on the
system. Now this isn't a penalty we would want to incur on every
task, but the path forward is maybe a bit easier than I thought.

I still don't think anyone is going to have the time to work on it right
now, but feel free to get a bug on file in Testing :: Python Test (and
make it block `buildpython3`).

On Fri, Jul 6, 2018 at 10:50 AM Axel Hecht <[hidden email]> wrote:
Am 06.07.18 um 16:04 schrieb Dustin Mitchell:
> I agree -- we faced similar issues in the days when py2.5 - 2.7 were
> still available.
>
> I think that if we're clear about the Python version we test on in CI,
> and willing to make the effort to fix bugs when we change that (e.g.,
> to 3.8), that's sufficient.
>
> Dustin
>

So, if you thought that story was sad ...

brew update
brew upgrade something-else

something-else depended on python, brew updated python from 3.6.5 to
3.7.0. I did intend to not update python deliberately, but no luck.

/usr/local/Cellar/python/3.6.5/bin/python can't be used for virtualenvs,
'cause sys.executable points to /usr/local/opt/python/bin/python3.6,
which doesn't exist anymore.

... so on mac/homebrew, there's very little power over which python you
end up using.

I do think we need to run tests on at least 2.7.latest and
3.[5-7].latest. For now, the 3.x ones can probably be tier 2-3, but we
need visibility into them.

Axel
_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds

_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds
Reply | Threaded
Open this post in threaded view
|

Re: PSA: |mach python-test| can now run with python 3

Gregory Szorc-3
In reply to this post by Axel Hecht
On Fri, Jul 6, 2018 at 3:12 AM, Axel Hecht <[hidden email]> wrote:
Nice.

I'm just devoting my Friday productivity to fixing bustages in python 3.7 compared to 3.6. Which affect code I copied from mozpack.path ;-) [0]

Which leads me to: We should run tests in automation for at least 3.7 and 3.6. Not sure what still ships 3.5, if at all?

Testing some version of 3.x sadly isn't covering all our bases.

I was initially hoping to go with 3.6+. I can't remember why exactly, but I was talked into 3.5+. I may have been optimizing to avoid causing too much disruption / difficulty: 3.5+ is definitely less controversial to require than 3.6+. And where my mind was a few months ago, I wanted to push us over the Python 3 hill any way possible with as little disruption / effort as possible, even if we landed on an older Python 3. My perception is the transition to require 3.5+ went pretty well and wasn't overly disruptive. So I think we made the right call there.

That being said, I did push for non-Docker environments in CI to install 3.6. I believe we now have 3.6.3+ installed in the Windows and macOS environments. But the Linux build environment is 3.5. I'm not sure what the Linux test environment is.

I'd entertain bumping the build requirement to 3.6. I'm all for getting all the goodies in 3.6 and not having to work around known bugs / deficiencies in 3.5. If someone can provide basic research about the state of 3.5 vs 3.6 and convince me 3.6 is relatively safe, let's do it. But be advised that we're planning on requiring Node.js soon and others are moving things to Clang. I'm wary about introducing too much developer disruption. Python 3.6 might be a difficult pill to swallow at the moment and we may want to put it on hold in favor of transitions that are more important (Node.js unlocks more than 3.5 -> 3.6 and Clang is hugely impactful).

Am 03.07.18 um 17:48 schrieb Andrew Halberstadt:

Dave Hunt has recently landed a lot of great work to get
some of our python unittests running with Python 3. You
can see his blog post on it here:
http://davehunt.co.uk/2018/06/29/python-unit-tests-now-running-with-python-3-at-mozilla.html

The march to Python 3 has been long and slow, but this is a
major milestone along the way. It gives us more confidence
that porting a module won't be immediately regressed again
with the next change.  Thanks Dave!

I want to emphasize that the time is ripe to start porting
any python modules you maintain to support Python 3
(in addition to 2). Here are some things you can do:

1) Enable the py2/py3 and flake8 linters on your module. You
can do this by adjusting the include/exclude paths at the
following locations:
https://searchfox.org/mozilla-central/source/tools/lint/py2.yml
https://searchfox.org/mozilla-central/source/tools/lint/py3.yml
https://searchfox.org/mozilla-central/source/tools/lint/flake8.yml

Then run |mach lint <path/to/module>| and fix any errors that
show up (use --fix to automatically fix whatever it can).

2) Run your |mach python-test| tests under Python 3 in CI. The
first step is to remove the 'skip-if' annotation from the relevant
manifest:
https://searchfox.org/mozilla-central/search?q=skip-if.*python+%3F%3D%3D+%3F3&case=true&regexp=true&path=ini

Next, run |mach python-test <path/to/module> --python 3| and
fix any test failures that show up.

Finally, adjust the task (if applicable) to run with both python 2
and 3 by modifying this line in the appropriate task:
https://searchfox.org/mozilla-central/source/taskcluster/ci/source-test/python.yml#94

3) Write all new python code to be both Python 2 and Python 3
compatible. The 'six' library should be available in all contexts
of mozilla-central (both locally and in CI). If it isn't, please file
a bug.

Here is a handy cheat sheet you can reference to solve
incompatibilities:
http://python-future.org/compatible_idioms.html

4) File bugs! Even if you don't have time to do the conversion
now, get bugs on file and make them block the 'buildpython3'
meta bug:
https://bugzilla.mozilla.org/show_bug.cgi?id=1388447

Having bugs like these on file can also turn into great first
projects for a contributor.

There is also infrastructure work we could implement to make
the transition easier. For example, we should run the 'flake8'
linter under python 3 as well as 2. We should also make
|mach python-test| default to running tests with both.

We'll only ever get to full python 3 compatibility if everyone
buys in. While the end goal is still a long long way away, that
shouldn't stop us from starting to make some progress now.

Cheers,
Andrew



_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds


_______________________________________________
dev-builds mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-builds