Porting applications that happen to be written in Python

The section for applications is for software where the user doesn’t care in which programming language it is written. For example, it doesn’t matter if it is written in Python 2 or 3 because the application should run the same.

Applications have to have an executable, which you can check by running dnf repoquery -l your-package-name | grep /usr/bin/.

If your package is not being imported by third-party projects (e.g. import some_module in a Python file), it is most likely an application.

Try running dnf repoquery --whatrequires your-package-name to see a list of packages that depend on yours. If there are none, your package is likely an application, because there would be little reason to package a module that nobody uses. However, if there are some packages that depend on yours, we cannot be sure if it’s an application-only package or not, as some of these packages might be depending on your application itself, instead of importing modules from your package.

If this doesn’t fit your package, look into other sections.

Porting the specfile to Python 3

Applications behave the same when run under Python 2 and Python 3, therefore all you need to do is change the spec file to use Python 3 instead of Python 2.

In essence, porting of an application RPM mostly consists of going through the spec file and adding number 3 or substituting number 3 for number 2 where appropriate. Occasionally also substituting old macros for new ones that are more versatile.

So let’s take an example spec file and port it to illustrate the process. We start with a spec file for an application that is being run with Python 2:

%global srcname example

Name:           %{srcname}
Version:        1.2.3
Release:        1%{?dist}
Summary:        An example Python application

License:        MIT
URL:            http://pypi.python.org/pypi/%{srcname}
Source0:        https://files.pythonhosted.org/packages/source/e/%{srcname}/%{srcname}-%{version}.tar.gz

BuildArch:      noarch
BuildRequires:  python-devel

%description
A Python application which provides a convenient example.


%prep
%autosetup -n %{srcname}-%{version}


%build
%{__python} setup.py build


%install
%{__python} setup.py install --skip-build --root $RPM_BUILD_ROOT


%check
%{__python} setup.py test


%files
%license COPYING
%doc README
%{python_sitelib}/*
%{_bindir}/sample-exec


%changelog
...

Modifications

First it is recommended to update the software you are packaging to its newest upstream version. If it already is at the latest version, increment the release number. Don’t forget to add a %changelog entry as well.

Note

In this document you will encounter lot of RPM macros. You can look up many of the Python macros in the Python packaging guidelines (click the Expand button on the right side).

BuildRequires and Requires

Change BuildRequires from python-devel to python3-devel and adjust all other Requires and BuildRequires entries to point only to python3 packages.

It is very important that you don’t use any Python 2 dependencies as that would make your package depend both on Python version 2 and version 3, which would render your porting efforts useless.

%build

In the build section you can find a variety of macros, for example %py_build and its newer version %py2_build. You can freely substitute these by the new Python 3 macro %py3_build.

Occasionally you will find a custom build command prefixed by the %{__python} or %{__python2} macros, or in some cases just prefixed by the Python interpreter invoked without a macro at all, e.g.:

%{__python} setup.py build
    or
python setup.py build

In these cases first try substituting the whole build command by the new smart macro %py3_build which should in many cases correctly figure out what ought to be done automatically. Otherwise change the invocation of the Python interpreter to the %{__python3} macro.

In rare cases, you might encounter some non-Python build script such as a Makefile. In these instances you have to adjust the script on your own, consult the documentation for the specific build method.

%install and %check

The %install section will oftentimes contain the %py_install and %py2_install macros; you can replace these with the new Python 3 macro %py3_install.

Furthermore, as in the preceding %build section, you will frequently find custom scripts or commands prefixed by either the %{__python} or %{__python2} macros or simply preceded by an invocation of the Python interpreter without the use of macros at all.

In the install section, try substituting it with the new %py3_install macro, which should figure out what to do automatically. If that doesn’t work, or if you’re porting the %check section, just make sure that any custom scripts or commands are invoked by the new %{__python3} macro.

Again as in the %build section, in the rare cases where you encounter a non-Python install script such as a Makefile, consult documentation for the specific install method and make adjustments on your own.

Lastly, in the %check section you can also encounter a custom Python command that runs the tests, such as nosetests or py.test. In that case find out what is the name of the executable for Python 3 and use it instead of the Python 2 version.

%check
py.test-3
or
nosetests-%{python3_version}

As you can see on the example of the nosetests, not all packages follow the proper naming conventions for executables. To list what executables a package contains, you can use:

$ dnf repoquery -l python3-nose | grep /usr/bin/
/usr/bin/nosetests-3.4

%files

In the files section you can regularly find the following macros: %{python2_sitelib}, %{python2_sitearch}, %{python2_version}, %{python2_version_nodots} or their unversioned alternatives. Substitute these with their counterparts for Python 3, e.g. %{python3_sitelib}.

The files section may also contain the versioned executable, usually %{_bindir}/sample-exec-2.7 in which case it should be substituted by %{_bindir}/sample-exec-%{python3_version}.

Are shebangs dragging you down (to Python 2)?

A shebang is an indicator on the first line of an executable script that indicates in what interpreter is the script supposed to be launched, examples include python, bash and perl. When software gets ported to Python 3, the lone shebang often remains forgotten and keeps pointing to Python 2. In most cases this is handled automatically: the setup.py script (usually run through the %py3_build and %py3_install RPM macros) adjusts shebangs for you. However, sometimes it’s up to you to handle the situation.

RPM has very good capabilities of automatically finding dependencies, and one of the ways it accomplishes that is by looking at the shebangs of all the files in the package. Therefore it is important to check if the shebangs are not dragging in a runtime dependency on Python 2.

As the porting of the spec file is nearly finished, build it and then run the following analysis on the resulting Python 3 RPM file:

$ rpm -qp --requires path/to/an.rpm | grep -E '/usr/bin/(python|env)'

This will list all the Python executables your RPM package depends on as well as the /usr/bin/env executable which usually invokes python. The use of env is dangerous: applications should be using the safe system version of Python and not trust whatever version env might try to substitute. If you find that an RPM package for Python 3 depends on Python 2 or /usr/bin/env you need to fix it.

Fixing shebangs

First find out what shebangs are used in your package by unpacking the sources for the project, cd-ing into the unpacked directory and trying the following command(s):

$ # Searches for all shebangs among the sources
$ grep -r '^#!/' .

$ # Searches only Python shebangs
$ grep -rE '^#!/usr/bin/(python|env python)' .

You will usually find one of these two shebangs:

#!/usr/bin/python
#!/usr/bin/env python

It is advisable to change both of these to #!/usr/bin/python3. /usr/bin/env can be useful for scripts, but applications should link to the system version of Python outright.

To change the shebangs in the files you can use one (or a combination) of the following commands, which you should place at the end of the %prep section. They will change the shebangs to point to the Python 3 interpreter stored in the ${__python3} macro.

$ # Change shebang in individual files
$ sed -i '1s=^#!/usr/bin/\(python\|env python\)[0-9.]*=#!%{__python3}=' path/to/file1 file2 file3 ...

$ # Change shebang in all relevant files in this directory and all subdirectories
$ # See `man find` for how the `-exec command {} +` syntax works
$ find -type f -exec sed -i '1s=^#!/usr/bin/\(python\|env python\)[23]\?=#!%{__python3}=' {} +

$ # Change shebang in all relevant executable files in this directory and all subdirectories
$ find -type f -executable -exec sed -i '1s=^#!/usr/bin/\(python\|env python\)[23]\?=#!%{__python3}=' {} +

You don’t have to worry about accidentally corrupting other files as these scripts will only change a file if the beginning of its first line exactly matches one of the two aforementioned shebangs.

Ported RPM spec file

Here you can peruse the entire ported spec file:

%global srcname example

Name:           %{srcname}
Version:        1.2.3
Release:        2%{?dist}
Summary:        An example Python application

License:        MIT
URL:            http://pypi.python.org/pypi/%{srcname}
Source0:        https://files.pythonhosted.org/packages/source/e/%{srcname}/%{srcname}-%{version}.tar.gz

BuildArch:      noarch
BuildRequires:  python3-devel

%description
A Python application which provides a convenient example.


%prep
%autosetup -n %{srcname}-%{version}


%build
%py3_build


%install
%py3_install


%check
%{__python3} setup.py test


%files
%license COPYING
%doc README
%{python3_sitelib}/*
%{_bindir}/sample-exec


%changelog
...

Diff of the changes

And here you can see the diff of the original and the ported spec files to fully observe all the changes that were made:

--- specs/application.spec.orig
+++ specs/application.spec
@@ -2,7 +2,7 @@
 
 Name:           %{srcname}
 Version:        1.2.3
-Release:        1%{?dist}
+Release:        2%{?dist}
 Summary:        An example Python application
 
 License:        MIT
@@ -10,7 +10,7 @@
 Source0:        https://files.pythonhosted.org/packages/source/e/%{srcname}/%{srcname}-%{version}.tar.gz
 
 BuildArch:      noarch
-BuildRequires:  python-devel
+BuildRequires:  python3-devel
 
 %description
 A Python application which provides a convenient example.
@@ -21,21 +21,21 @@
 
 
 %build
-%{__python} setup.py build
+%py3_build
 
 
 %install
-%{__python} setup.py install --skip-build --root $RPM_BUILD_ROOT
+%py3_install
 
 
 %check
-%{__python} setup.py test
+%{__python3} setup.py test
 
 
 %files
 %license COPYING
 %doc README
-%{python_sitelib}/*
+%{python3_sitelib}/*
 %{_bindir}/sample-exec