At ITDev, we like Python. It's a massively popular language with libraries that makes it incredibly versatile which we can use for all kinds of tasks, from automation scripts to web apps.
However, as of 1st January 2020, Python 2.7 is no longer maintained and has been replaced by the non-backwards-compatible Python 3. Most major Python libraries have also announced that they will no longer be supporting Python 2 (refer to this list here), which makes Python 3 the only option for security updates and up-to-date libraries.
Therefore, any and all Python 2 scripts and apps need updating. Luckily, Python 3 was released back in 2008, which means there are plenty of tools and documentation to assist us with the upgrade.
In this blog, we will be using the example of an internal Python web app which uses a Flask front-end paired with a Django back-end. This web app is a GUI that gives our developers access to, and the ability to modify the VLANs that their test hardware is connected to. It is standard legacy code; it was written a long time ago and except for a few minor additions and modifications, hasn't been touched since.
Preparation
A few preparation steps should be taken before considering an upgrade to Python 3:
- Verify code test coverage - the larger the codebase, the more important this step is as having a high coverage is essential for testing that the upgrade works.
- Verify the source code is running on Python 2.7 - most upgrade tools and documentation assume Python 2.7, and it is simpler to go 2.6 -> 2.7 -> 3.x than attempt 2.6 -> 3.x directly.
- Verify that all dependencies support Python 3.x - this may include modifying code to support updated dependencies or finding alternative libraries that do support Python 3.x. If possible, it is preferable to upgrade to dependency versions that support both Python 2.7 and Python 3.
It is also important to decide whether a project needs to support just Python 3.x or both Python 2.7 and 3.x. Larger projects receiving rolling upgrades or that require legacy support for Python 2 may need to support both versions. It is possible to support both Python 2.7 and 3.x with the same source code; however if you do, we strongly suggest using a Continuous Integration (CI) solution to ensure you maintain compatibility with both. ITDev has developed 'Accelerate-CI', a service for acclerating the deployment of Continuous Integration environments. Using 'Accelerate-CI' it was easy to tailor an environment for our testing strategy. In this instance we were able to run a number of automated tasks including pre-commit checks on review requests, executing both Python 2 and Python 3 tests on the same source code.
Migrating our web app
Our web app example is relatively simple and and is only running on a single server so we decided to migrate it entirely to Python 3.x. The approach we used is as follows:
In order to estimate the level of difficulty with upgrading the web app, we cloned our initial Python 2.7 source code into a clean test environment. We then ran tests to make sure the documentation was up-to-date, all dependencies correct and the application was working as expected.
Once the application was confirmed as working, we ran the ‘2to3.py’ script (which is included in Python installs).
By default, running the ‘2to3.py’ script on a Python 2.7 source file prints a ‘diff’ comparing the existing Python 2.7 code with its Python 3 transformation, showing the necessary modifications. The ‘2to3.py’ script can also write these code changes right back to the source code by running ‘2to3.py’ with the ‘-w’ flag. This will overwrite your files, so manually verify the changes before applying them, as the changes may have unintended repercussions on the code.
Examples of modifications for backwards compatibility
When updating your code, there are a few common translation patterns that you may encounter:
Importing from the __future__
To support the relevant Python 3.x syntax in Python2.7 we can import from the ‘__future__’ module.
For example, to support the new Python 3.x print syntax:
# Python 3.x;
>>> print ("Hello ", "World!")
Hello World!
# Python 2.x
>>> print ("Hello ", "World!")
("Hello ", "World!")
>>> from __future__ import print_function
>>> print ("Hello", "World!")
Hello World!
Exceptional dependencies
For dependencies that only support either Python 2.7 or Python 3, you may need separate imports:
try:
from [Python 3.x Library] import abc
except ImportError:
from [Python 2.7 Library] import abc
Handling incompatible changes:
There are some changes in Python 3.x that are not covered by ‘2to3.py’. For example, all strings in Python 3.x are Unicode by default, meaning there is a new distinction between strings and bytes. This may cause complications for example if a package relies on strings or bytes specifically. In these situations, duplicate code is unavoidable. The best solution here is to use an ‘if/else’ clause:
if sys.version_info.major > 2:
[Python 3.x code]
else:
[Python 2.7 code]
Finalising
Finally, when you have completed all your changes it is once again time to run your testing using your automation framework (in our case Accelerate-CI) . Most errors will be missing ‘pip3’ packages, or incompatible syntax that was not translated by the 2to3.py tool. For syntax errors you can review and manually make changes as in the examples above.
In our web app example, once all the changes were made, we ran the Python 2.7 tests on the updated source code in our CI system. When the tests completed successfully, we re-ran the testing using Python 3! It is important not to miss out this testing step in your project.
All dependencies that are Python-specific had to be installed for Python 3.x. For many Python 2.7 libraries that were installed with the ‘pip’ command, all we had to do was reinstall them with the ‘pip3’ command instead.
For your migration project, other dependencies and tools may need to be researched to find the best practice for updating them to Python 3. For our web app example, we had to configure Apache to use Python 3.x by installing ‘libapache2-mod-wsgi-py3’.
You should also verify and update as necessary any shebangs (the '#!' line used to indicate the interpreter to be used) at the top of the file and all docstrings, as most tools will not update these.
Summary
To summarise, Python 2.7 is no longer maintained so it is important to update your legacy Python applications to Python 3.x, particularly those that are security-related, as soon as you can.
I’ve mentioned some of the key steps for a successful migration:
-
Verify your code test coverage.
-
Verify your code is running on Python 2.7, or convert any pre-Python 2.7 code to Python 2.7, before migrating it all to Python 3.x.
-
Verify all dependencies support Python 3.x.
-
Decide whether both Python 2.7 and Python 3.x versions need to be supported.
-
Carry out appropriate testing of a copy of your code in a clean test environment, e.g. use ITDev’s Accelerate-CI.
-
Run the ‘
2to3.py’ script. -
Update all the dependencies.
-
Update top-of-file and docstrings and anything else the tools don’t automatically convert.
-
Carry out appropriate final testing of your newly updated Python 3 code.
How ITDev Can Help
We wish you luck with your upgrades, however if you need assistance migrating your projects to Python 3.x, there is wealth of experience to draw on at ITDev, so don’t hesitate to get in touch with us. Email us or call us on +44 (0)23 8098 8890.
Main image: www.commons.wikimedia.org
