Stepping up to Python 3, without falling down

Call: +44 (0)23 8098 8890
E-mail:

Posted 7th January 2020, By Lucas N
Python 3

At ITDev, we like Python. It's a massively popular language with libraries that makes it incredibly versatile which we can use for all kinds of tasks, from automation scripts to web apps. 

However, as of 1st January 2020, Python 2.7 is no longer maintained and has been replaced by the non-backwards-compatible Python 3. Most major Python libraries have also announced that they will no longer be supporting Python 2 (refer to this list here), which makes Python 3 the only option for security updates and up-to-date libraries. 

Therefore, any and all Python 2 scripts and apps need updating. Luckily, Python 3 was released back in 2008, which means there are plenty of tools and documentation to assist us with the upgrade. 

In this blog, we will be using the example of an internal Python web app which uses a Flask front-end paired with a Django back-end. This web app is a GUI that gives our developers access to, and the ability to modify the VLANs that their test hardware is connected to. It is standard legacy code; it was written a long time ago and except for a few minor additions and modifications, hasn't been touched since. 

Preparation 

A few preparation steps should be taken before considering an upgrade to Python 3: 

  • Verify code test coverage - the larger the codebase, the more important this step is as having a high coverage is essential for testing that the upgrade works. 
  • Verify the source code is running on Python 2.7 - most upgrade tools and documentation assume Python 2.7, and it is simpler to go 2.6 -> 2.7 -> 3.x than attempt 2.6 -> 3.x directly. 
  • Verify that all dependencies support Python 3.x - this may include modifying code to support updated dependencies or finding alternative libraries that do support Python 3.x. If possible, it is preferable to upgrade to dependency versions that support both Python 2.7 and Python 3. 

It is also important to decide whether a project needs to support just Python 3.x or both Python 2.7 and 3.x. Larger projects receiving rolling upgrades or that require legacy support for Python 2 may need to support both versions. It is possible to support both Python 2.7 and 3.x with the same source code; however if you do, we strongly suggest using a Continuous Integration (CI) solution to ensure you maintain compatibility with both. ITDev has developed 'Accelerate-CI', a service for acclerating the deployment of Continuous Integration environments. Using 'Accelerate-CI' it was easy to tailor an environment for our testing strategy. In this instance we were able to run a number of automated tasks including pre-commit checks on review requests, executing both Python 2 and Python 3 tests on the same source code. 

Migrating our web app 

Our web app example is relatively simple and and is only running on a single server so we decided to migrate it entirely to Python 3.x. The approach we used is as follows: 

In order to estimate the level of difficulty with upgrading the web app, we cloned our initial Python 2.7 source code into a clean test environment. We then ran tests to make sure the documentation was up-to-date, all dependencies correct and the application was working as expected.  

Once the application was confirmed as working, we ran the ‘2to3.py’ script (which is included in Python installs).  

By default, running the ‘2to3.py’ script on a Python 2.7 source file prints a ‘diff’ comparing the existing Python 2.7 code with its Python 3 transformation, showing the necessary modifications. The ‘2to3.py’ script can also write these code changes right back to the source code by running ‘2to3.py’ with the ‘-w’ flag. This will overwrite your files, so manually verify the changes before applying them, as the changes may have unintended repercussions on the code. 

Examples of modifications for backwards compatibility 

When updating your code, there are a few common translation patterns that you may encounter:

Importing from the __future__ 

To support the relevant Python 3.x syntax in Python2.7 we can import from the ‘__future__’ module.  
For example, to support the new Python 3.x print syntax:  

# Python 3.x;
>>> print ("Hello ", "World!") 
Hello World!

# Python 2.x
>>> print ("Hello ", "World!")
("Hello ", "World!")
>>> from __future__ import print_function
>>> print ("Hello", "World!")
Hello World!

Exceptional dependencies  

For dependencies that only support either Python 2.7 or Python 3, you may need separate imports: 

try:
    from [Python 3.x Library] import abc
except ImportError:
    from [Python 2.7 Library] import abc

Handling incompatible changes:

There are some changes in Python 3.x that are not covered by ‘2to3.py’. For example, all strings in Python 3.x are Unicode by default, meaning there is a new distinction between strings and bytes. This may cause complications for example if a package relies on strings or bytes specifically. In these situations, duplicate code is unavoidable. The best solution here is to use an ‘if/else’ clause: 

if sys.version_info.major > 2:
    [Python 3.x code]
else:
    [Python 2.7 code]

Finalising 

Finally, when you have completed all your changes it is once again time to run your testing using your automation framework (in our case Accelerate-CI) . Most errors will be missing ‘pip3’ packages, or incompatible syntax that was not translated by the 2to3.py tool. For syntax errors you can review and manually make changes as in the examples above.

In our web app example, once all the changes were made, we ran the Python 2.7 tests on the updated source code in our CI system. When the tests completed successfully, we re-ran the testing using Python 3! It is important not to miss out this testing step in your project.

All dependencies that are Python-specific had to be installed for Python 3.x. For many Python 2.7 libraries that were installed with the ‘pip’ command, all we had to do was reinstall them with the ‘pip3’ command instead. 

For your migration project, other dependencies and tools may need to be researched to find the best practice for updating them to Python 3. For our web app example, we had to configure Apache to use Python 3.x by installing ‘libapache2-mod-wsgi-py3’. 

You should also verify and update as necessary any shebangs (the '#!' line used to indicate the interpreter to be used) at the top of the file and all docstrings, as most tools will not update these. 

Summary 

To summarise, Python 2.7 is no longer maintained so it is important to update your legacy Python applications to Python 3.x, particularly those that are security-related, as soon as you can.  

I’ve mentioned some of the key steps for a successful migration: 

  • Verify your code test coverage.

  • Verify your code is running on Python 2.7, or convert any pre-Python 2.7 code to Python 2.7, before migrating it all to Python 3.x. 

  • Verify all dependencies support Python 3.x.

  • Decide whether both Python 2.7 and Python 3.x versions need to be supported.

  • Carry out appropriate testing of a copy of your code in a clean test environment, e.g. use ITDev’s Accelerate-CI.

  • Run the ‘2to3.py’ script.

  • Update all the dependencies.

  • Update top-of-file and docstrings and anything else the tools don’t automatically convert.

  • Carry out appropriate final testing of your newly updated Python 3 code.

How ITDev Can Help

We wish you luck with your upgrades, however if you need assistance migrating your projects to Python 3.x, there is wealth of experience to draw on at ITDev, so don’t hesitate to get in touch with us. Email us or call us on +44 (0)23 8098 8890.

Main image: www.commons.wikimedia.org

 

 

 

 

Lucas

Graduate Software Engineer

BSc - Computer Science, University of Southampton

I started at ITDev as a Graduate Software Engineer in Autumn 2018, having recently graduated from the University of Southampton. During my degree I worked on a multitude of small projects in different programming languages, including Python, Java and C++.

The varied projects together with working alongside the enthusiastic and knowledgeable team has helped develop my skills as a software engineer.

In my spare time I enjoy playing board games, riding motorbikes and practicing Kung Fu.

View all posts by Lucas    |    View all ITDev team members

IET Enterprise Partners logo

Latest Blog Posts

Vivado plus Git icons
Posted 6th May 2020, By Tom J
Maintaining a neat and tidy desk is something many of us are familiar with. Doing the same for your FPGA repository can be slightly more challenging. Tom ...more
Linux penguin
Posted 1st May 2020, By Quentin D
Are you thinking about contributing to the Linux kernel? Let's have a look at how our internal 'Linux Maintainers Group' does it with Quentin's detailed guide ...more
Raspberry Pi image
Posted 9th April 2020, By Kris D
So you’ve got a project that needs both low and high-level interfacing. Perhaps it requires the use of multiple sensors or inputs to determine an output or ...more
Posted 18th March 2020, By Simon H
We recently used the Raspberry Pi as the basis of an Industrial IoT proof of concept. Here Simon reviews the benefits of this approach and why it was suitable ...more

Latest News

Spring 2020
Posted 17th April 2020
Following the beautiful weather over the Easter weekend, we are proud to be able to provide an update on our current activities, showing business is continuing ...more
Business as usual
Posted 20th March 2020
With a strong commitment to keeping our staff and clients safe during this period, we would like to reassure all our business contacts that we remain fully ...more
South Coast Tech Awards Trophy
Posted 6th December 2019
ITDev was excited to attend the first South Coast Technology Awards on Thursday 5th December at the Ageas Bowl. A much needed celebratory awards for a sector ...more
South Coast Tech Awards Finalist nomination image
Posted 31st October 2019
We're proud to announce that ITDev has been shorlisted as a Finalist in the Best 'Tech Employer' category in the South Coast Tech Awards.