Porting Python modules¶

This section is for Python packages that do not have any executables. Usually, these are normal Python modules that are being imported by third-party projects.

If this doesn’t fit your package, look into other sections.

Table of Contents

Porting the specfile to Python 3
Modifications
Ported RPM spec file
Diff of the changes

Porting the specfile to Python 3 ¶

Because the software you’re packaging is going to be imported by third-party projects, it is crucial to think about what Python versions your package will support.

If you switch your package to use only Python 3, suddenly projects running on Python 2 will no longer be able to import your modules. And of course, if you continue using Python 2 only, new Python 3 projects won’t get to use your software either.

For these reasons, it is highly advised to split your package into two subpackages, one for each major Python version.

Let’s take an example spec file and port it to illustrate the process. We start with a spec file for Python module packaged for Python 2.

%global srcname example

Name:           python-%{srcname}
Version:        1.2.3
Release:        1%{?dist}
Summary:        An example Python module

License:        MIT
URL:            https://pypi.org/project/%{srcname}
Source0:        https://files.pythonhosted.org/packages/source/e/%{srcname}/%{srcname}-%{version}.tar.gz

BuildArch:      noarch
BuildRequires:  python-devel
Requires:       python-some-module
Requires:       python2-other-module

%description
A Python module which provides a convenient example.


%prep
%autosetup -n %{srcname}-%{version}


%build
%{__python} setup.py build


%install
%{__python} setup.py install --skip-build --root $RPM_BUILD_ROOT


%check
%{__python} setup.py test


%files
%license COPYING
%doc README
%{python_sitelib}/*


%changelog
...

Modifications ¶

First it is recommended to update the software you are packaging to its newest upstream version. If it already is at the latest version, increment the release number. Don’t forget to add a %changelog entry as well.

Note

In this document you will encounter lot of RPM macros. You can look up many of the Python macros in the Python packaging guidelines (click the Expand button on the right side).

Creating subpackages ¶

Each subpackage you create will need to have its own name, summary and description. If you haven’t already, it is thus advised to declare macros for common values at the top of the specfile:

%global srcname example

Now we can use these to create the subpackages. The following should be placed beneath the %description section of the base package:

%package -n python2-%{srcname}
Summary:  %{summary}
Requires: python-some-module
Requires: python2-other-module
%{?python_provide:%python_provide python2-%{srcname}}

%description -n python2-%{srcname}
A Python tool which provides a convenient example.


%package -n python3-%{srcname}
Summary:  %{summary}
Requires: python3-some-module
Requires: python3-other-module
%{?python_provide:%python_provide python3-%{srcname}}

%description -n python3-%{srcname}
A Python tool which provides a convenient example.

First, using the %package macro you start defining a new subpackage, specifying its full name as python2-%{srcname}, which in this case will become python2-example. Next we provide the summary which we defined earlier.

BuildRequires: tags from the original spec file will remain where they are—declared in the definition of the base package at the top of the spec file. However, the runtime requirements—the ones listed using the Requires: tag—will be different for the two subpackages, so they have to be moved here to the definition of each subpackage.

While you can cut and paste all the Requires: tags directly from the base package to the python2- subpackage, remember that for the python3- subpackage you need to find Python 3 versions of each of the runtime dependencies.

Note

You can see that the naming of Python 2 packages isn’t uniform: some follow the current convention of using the python2- prefix, older ones use only the python- prefix, and the oldest might be without a prefix at all.

In many cases the Python 2 package can be found under both the python2- and python- prefixes, one of them being virtually provided by the Provides: tag. Whenever possible, use the version with the python2- prefix.

%python_provide ¶

Now that we’re splitting the package python-example into python2-example and python3-example, we need to define what will happen when the user tries to install the unversioned name python-example.

At the time of this writing, the packaging guidelines say that the default version should be the one for Python 2. However, it is expected to change to Python 3 some time in the future. To avoid having to adjust all Python packages in Fedora when that time comes, the %python_provide macro was devised:

%{?python_provide:%python_provide python2-%{srcname}}
and
%{?python_provide:%python_provide python3-%{srcname}}

This is a line you should include in each of your subpackages and it works thus: First the part ?python_provide: checks whether the macro exists and if not, the entire line is ignored. After that we actually use the %python_provide macro and give it one argument—the name of the given subpackage.

The macro will then check whether this Python version is default or not—if not, the line is again ignored. However, if indeed this is the currently default Python version, the macro is replaced with a virtual provides tag: Provides: python-%{srcname}. This will tell the packaging system (dnf, yum, …) to install this subpackage when user searches for python-example.

%description ¶

Each subpackage also needs to contain its own description. However, unlike the Summary: and Requires: tags, which are automatically applied to the subpackage declared above them, the %description macro needs to be told to which subpackage it belongs. You can do that by appending the same name as you did with the %package macro itself.

%description -n python3-%{srcname}
A Python tool which provides a convenient example.

BuildRequires and Requires ¶

Now that you’re building subpackages for both Python 2 and Python 3, you need to adjust the BuildRequires: by adding Python 3 versions of all the existing build dependencies. Starting with python-devel: Use its new version-specific name python2-devel and add it’s Python 3 equivalent python3-devel.

As described above, Requires: tags are a bit more complicated. You should move the current set of Requires: underneath the definition of the Python 2 subpackage, and for the Python 3 subpackage, you need to find Python 3 alternatives for all the current Python 2 runtime requirements that are specified with the Requires: tags.

%build ¶

Currently your package is building the software for Python 2, what we need to do is also add building for Python 3. While we’re modifying the spec file, however, it’s a good idea to also update it to new standards—in this case a new macro.

In the ideal case, you’ll find the build done with either the %py2_build macro or its older version %py_build, which you then should exchange for the former. In either case, you can just add the macro %py3_build afterwards, and this part is done. Note that to use these macros, you need to have python2-devel and/or python3-devel listed among BuildRequires, but most Python packages already do.

%build
%py2_build
%py3_build

In many cases, however, you will find a custom build command prefixed by the %{__python} or %{__python2} macros, or in some cases just prefixed by the python interpreter invoked without a macro at all, e.g.:

%{__python} custombuild.py --many-flags
    or
python custombuild.py --many-flags

In these cases first try substituting the whole build command by the new pair of smart macros %py2_build and %py3_build, which should in many cases correctly figure out what ought to be done automatically. Otherwise, duplicate the entire command and change the invocation of the python interpreter to the %{__python2} macro in one of them and to the %{__python3} in the other.

%build
%{__python2} custombuild.py --many-flags
%{__python3} custombuild.py --many-flags

Rarely, you might encounter some non-Python build script such as a Makefile. In these instances you have to adjust the script on your own, consult the documentation for the specific build method.

%install ¶

First, in the same manner as in the preceding %build section, it is advisable to upgrade the current Python 2 install command to use the new %py2_install macro, however, if that doesn’t work for you, you can stick with the current install command, just make sure it’s invoked by the %{__python2} macro.

After that, add the corresponding Python 3 install command, which will be either the custom command prefixed by %{__python3} or the new %py3_install macro.

%install
%py2_install
%py3_install

Again as in the %build section, in the rare cases where you encounter a non-Python install script such as a Makefile, consult documentation for the specific install method and make adjustments on your own.

%check ¶

Unlike in previous sections, there’s no special macro for the %check section, and so here if the original spec file uses any sort of a python script for testing, just make sure that the tests are invoked once using the %{__python2} macro and a second time using the %{__python3} macro.

%check
%{__python2} setup.py test
%{__python3} setup.py test

Chances are that you will encounter a custom Python command that runs the tests, such as nosetests or py.test. In that case find out what is the name of the executable for Python 3 and run it after the Python 2 command.

If the command for Python 2 can be invoked explicitly for Python 2, e.g. as py.test-2 instead of just py.test, use it. Note that to use py.test commands, you need to have python2-pytest and/or python3-pytest listed among BuildRequires.

%check
py.test-2
py.test-3

or

nosetests-%{python2_version}
nosetests-%{python3_version}

As you can see on the example of the nosetests, not all packages follow the proper naming conventions for executables. To list what executables a package contains, you can use:

$ dnf repoquery -l python3-nose | grep /usr/bin/
/usr/bin/nosetests-3.4

%files ¶

The presence or absence of a %files section is the deciding factor in whether a given package or subpackage gets built or not. Therefore, to assure that our base package doesn’t get built (as all the content has been moved to the two subpackages), make sure there is no %files section without a subpackage name.

You can reuse the current %files section for the Python 2 submodule by giving it the appropriate package name. You can keep it almost the same as before, just make sure that, where appropriate, it uses the new macros %{python2_sitelib}, %{python2_sitearch}, %{python2_version} or perhaps %{python2_version_nodots}.

%files -n python2-%{srcname}
%license COPYING
%doc README
%{python2_sitelib}/*

Accordingly we’ll also add a %files section for the Python 3 subpackage. You can copy the previous files section, but make sure you change all the Python 2 macros into Python 3 versions.

%files -n python3-%{srcname}
%license COPYING
%doc README
%{python3_sitelib}/*

Ported RPM spec file ¶

Here you can peruse the entire ported spec file:

%global srcname example

Name:           python-%{srcname}
Version:        1.2.3
Release:        2%{?dist}
Summary:        An example Python module

License:        MIT
URL:            https://pypi.org/project/%{srcname}
Source0:        https://files.pythonhosted.org/packages/source/e/%{srcname}/%{srcname}-%{version}.tar.gz

BuildArch:      noarch
BuildRequires:  python2-devel
BuildRequires:  python3-devel

%description
A Python module which provides a convenient example.


%package -n python2-%{srcname}
Summary:        %{summary}
Requires:       python2-some-module
Requires:       python2-other-module
%{?python_provide:%python_provide python2-%{srcname}}

%description -n python2-%{srcname}
A Python module which provides a convenient example.


%package -n python3-%{srcname}
Summary:        %{summary}
Requires:       python3-some-module
Requires:       python3-other-module
%{?python_provide:%python_provide python3-%{srcname}}

%description -n python3-%{srcname}
A Python module which provides a convenient example.


%prep
%autosetup -n %{srcname}-%{version}


%build
%py2_build
%py3_build


%install
%py2_install
%py3_install


%check
%{__python2} setup.py test
%{__python3} setup.py test


# Note that there is no %%files section for the unversioned Python package
# if we are building for several Python runtimes
%files -n python2-%{srcname}
%license COPYING
%doc README
%{python2_sitelib}/*

%files -n python3-%{srcname}
%license COPYING
%doc README
%{python3_sitelib}/*


%changelog
...

Diff of the changes ¶

And here you can see the diff of the original and the ported spec files to fully observe all the changes that were made:

--- /home/docs/checkouts/readthedocs.org/user_builds/python-rpm-porting/checkouts/latest/specs/module.spec.orig
+++ /home/docs/checkouts/readthedocs.org/user_builds/python-rpm-porting/checkouts/latest/specs/module.spec
@@ -2,7 +2,7 @@
 
 Name:           python-%{srcname}
 Version:        1.2.3
-Release:        1%{?dist}
+Release:        2%{?dist}
 Summary:        An example Python module
 
 License:        MIT
@@ -10,11 +10,30 @@
 Source0:        https://files.pythonhosted.org/packages/source/e/%{srcname}/%{srcname}-%{version}.tar.gz
 
 BuildArch:      noarch
-BuildRequires:  python-devel
-Requires:       python-some-module
-Requires:       python2-other-module
+BuildRequires:  python2-devel
+BuildRequires:  python3-devel
 
 %description
+A Python module which provides a convenient example.
+
+
+%package -n python2-%{srcname}
+Summary:        %{summary}
+Requires:       python2-some-module
+Requires:       python2-other-module
+%{?python_provide:%python_provide python2-%{srcname}}
+
+%description -n python2-%{srcname}
+A Python module which provides a convenient example.
+
+
+%package -n python3-%{srcname}
+Summary:        %{summary}
+Requires:       python3-some-module
+Requires:       python3-other-module
+%{?python_provide:%python_provide python3-%{srcname}}
+
+%description -n python3-%{srcname}
 A Python module which provides a convenient example.
 
 
@@ -23,21 +42,31 @@
 
 
 %build
-%{__python} setup.py build
+%py2_build
+%py3_build
 
 
 %install
-%{__python} setup.py install --skip-build --root $RPM_BUILD_ROOT
+%py2_install
+%py3_install
 
 
 %check
-%{__python} setup.py test
+%{__python2} setup.py test
+%{__python3} setup.py test
 
 
-%files
+# Note that there is no %%files section for the unversioned Python package
+# if we are building for several Python runtimes
+%files -n python2-%{srcname}
 %license COPYING
 %doc README
-%{python_sitelib}/*
+%{python2_sitelib}/*
+
+%files -n python3-%{srcname}
+%license COPYING
+%doc README
+%{python3_sitelib}/*
 
 
 %changelog