Building RPMs for fun and profit

Specfile overview

The packaging format used by all RHEL based system is RPM. This consists of a cpio archive, a header on this archive with metadata and optionally a gpg signature for integrity verification.

The contents of the RPM and the metadata is determined by a spec file which has various directives to provide the values for the data in the header and specifically how to build resulting package.

Source packages (aka SRPMs) have the extension src.rpm whereas built packages will be >arch<.rpm where arch could be noarch, i686, x86_64 or a number of other alternatives.

The bare minimum required in a spec file is:

Name:	minimal	
Version:	1.0
Release:	1%{?dist}
Summary:	minimal spec
License: foo
%description
bar

Of course though this will pass a rpmbuild -ba minimal.spec it wouldn't result in much worth while installing!

To delve into the spec format in detail it's worth reading, though parts are quite out of date now, the original Max RPM guide as well as the more up to date Fedora wiki article on the subject.

This article won't go into detail on how RPM and spec work together as the wiki is the best place for updated information but it will touch on certain points on the path to the clean build system.

Fedora guidelines

For many the Fedora guidelines seem onerous and the question is often raised about how much attention should be paid to them when constructing internal/personal only packages.

Certain areas of the guidelines such as naming conventions, bundling, licensing and changelog may indeed be relatively irrelevant for these but by staying within them for the larger part there are several benefits in doing so:

  • Building for newer releases less painful
  • Standard locations means less local knowledge for new admins
  • If it's potentially submittable to Fedora less additional work to bring it in line
  • The macros in the guidelines have made the spec cleaner and easier to follow

Making use of rpmdevtools

One of the quality of life packages to make life easier is rpmdevtools.

Making use of this it's simple enough to create an RPM development environment via rpmdev-setuptree. In addition there are support commands to quickly bump the release number, get checksums of all files installed, get diffs of packages and more.

Making good use of RPM macros

Those new to packaging often question why so many macros are used in many spec files. What's wrong with just listing %config /etc/foo directly?

One recent use in the Fedora world was the UsrMove change which would have been almost impossible with spec files listing /bin/foo but with the macro in use of %{_bindir}/foo a rebuild makes this transparent.

There are macros for standardised configure, make and make install along with equivalents for other languages, such as py_build and py_install. These keep the options used to build applications consistent and cuts down on a lot of the repetitive boilerplate that would make reading teh spec file harder.

As a comparison for the average C/C++ application that has a tarball, couple of patches and then needs to be built and installed compare these two specs:

The first with no macro use and everything expanded:

Name:    myspecialsnowflake
Version:  1.0
Release:  1.fc23
License:  MIT

Source0: myspecialsnowflake.tar.gz
Patch1:  myspecialsnowflake-0001-fixsomethingbroken.patch
Patch2:  myspecialsnowflake-0002-customise to distro.patch

Summary: An example fake application

%description
This myspecialsnowflake is a fake application designed to give an example of spec usage.

%prep
tar xf %{source0}
cd myspecialsnowflake
patch -p1 < %{patch1}
patch -p1 < %{patch2}

%build
./configure --build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu \
	--program-prefix= \
	--disable-dependency-tracking \
	--prefix=/usr \
	--exec-prefix=/usr \
	--bindir=/usr/bin \
	--sbindir=/usr/sbin \
	--sysconfdir=/etc \
	--datadir=/usr/share \
	--includedir=/usr/include \
	--libdir=/usr/lib64 \
	--libexecdir=/usr/libexec \
	--localstatedir=/var \
	--sharedstatedir=/var/lib \
	--mandir=/usr/share/man \
	--infodir=/usr/share/info
make

%install
make install DESTDIR=${RPM_BUILD_ROOT}-myspecialsnowflake-1.0-1.fc23.x86_64

%post
if [ $1 -eq 1 ] ; then 
        # Initial installation 
        systemctl preset  myspecialsnowflake.service >/dev/null 2>&1 || : 
fi 

%preun
if [ $1 -eq 0 ] ; then 
        # Package removal, not upgrade 
        systemctl --no-reload disable myspecialsnowflake.service > /dev/null 2>&1 || : 
        systemctl stop myspecialsnowflake.service > /dev/null 2>&1 || : 
fi 

%postun
if [ $1 -ge 1 ] ; then 
        # Package upgrade, not uninstall 
        systemctl try-restart myspecialsnowflake.service >/dev/null 2>&1 || : 
fi 

%files
/usr/sbin/myspecialsnowflake
/usr/lib/systemd/system/myspecialsnowflake.service
%dir %config(noreplace) /etc/myspecialsnowflake
%doc /usr/share/doc/myspecialsnowflake/README
%doc /usr/share/man/man8/myspecialsnowflake.8*
%license /usr/share/licenses/myspecialshowflake/COPYING

And now the version making use of macros instead:

Name:    myspecialsnowflake
Version:  1.0
Release:  1.fc23
License:  MIT

Source0: %{name}.tar.gz
Patch1:  %{name}-0001-fixsomethingbroken.patch
Patch2:  %{name}-0002-customise to distro.patch

Summary: An example fake application

%description
This myspecialsnowflake is a fake application designed to give an example of spec usage.

%prep
%{autosetup} -n %{name}

%build
%{configure}
make

%install
%{make_install}

%post
%systemd_post %{name}.service

%preun
%systemd_preun %{name}.service

%postun
%systemd_postun_with_restart %{name}.service 

%files
%{_sbindir}/%{name}
%{_unitdir}/%{name}.service
%dir %config(noreplace) /etc/%{name}
%doc README
%doc %{_mandir}/man8/%{name}.8*
%license COPYING

Hopefully in terms of maintenance and reuse it's easier to see which ends up being simpler to deal with.

To see a list of all macros your system knows about use the command rpm --showrc and to find out how a particular macro gets expanded use rpm --eval "%{macroname}".

If building for multiple releases or distributions bear in mind that these can vary between them so it's important to check on the target system. In addition certain *-devel packages may supplement the macros with new capabilities - for example python-devel providing the py_build and py_install.

To get a list of all packages that provide rpm macros using yum or dnf a simple query can be run: dnf provides '/usr/lib/rpm/macros.d*'.

Creating the package with rpmbuild

The tool/command used to actually build the package is rpmbuild. This can be used to build an RPM for the system from an SRPM, or build an SRPM and/or RPM from the spec file and the sources required for it.

  • Building from existing SRPM
    rpmbuild --rebuild myspecialsnowflake.src.rpm
  • Building an SRPM from sources and spec file:
    rpmbuild -bs SPECS/myspecialsnowflake.spec
  • Building an RPM from sources and spec file:
    rpmbuild -bb SPECS/myspecialsnowflake.spec
  • Building both an SRPM and RPM from sources and spec file:
    rpmbuild -bs SPECS/myspecialsnowflake.spec

Using the --rebuild option with an SRPM allows skipping installing the SRPM and building directly from sources and spec, most often though the latter is used to allow modifying the spec is probably why the package is being built in the first place and mock (coming up soon) is better to build with than using rpmbuild --rebuild.

When the binary build is carried out rpmbuild will check the BuildRequires fields of the spec file and ensure that something is installed that fulfills these requirements. It's important to note that Requires is not used at this point and rpmbuild doesn't install the build requirements - only checks they are fulfilled.

The build requirements can be installed directly via dnf builddep myspecialsnowflake.(srpm|spec). It can use either an SRPM or a spec file to interrogate for requirements. To keep the system clean of packages that were only installed last to meet these requirements dnf history can be interrogated to clean up: dnf history undo last or better still mock can be used (coming up really, really soon).

When specifying Requires and BuildRequires it's usually better to use any virtual provides that achieves the goal rather than specific package names as this makes it easier to maintain if the package providing a feature changes name, for example BuildRequires: pkgconfig(glib-2.0).

Why use mock?

There are several downsides to just using rpmbuild -bb foo.spec on any given system to build binary packages for installing.

  • Building packages for other than the build host architecture becomes problematic
  • The build host can easily end up with lots of extra -devel packages that were only needed to build a certain package but not required otherwise
  • Without a minimal environment extra dependencies can end up in the package (via the automatic Requires detection) than actually needed.
  • Missing BuildRequires can happen if the requirement is already installed on the build host causing the build to pass. This can then cause future build failures if the package is then built in a system missing this as it's not mentioned in the spec.
  • Needing to build a workflow with dnf builddep to get BuildRequires or lots of manual intervention required to get a build.

Mock removes all these issues by creating a chroot and installing into it a bare minimal environment. Then it installs into it anything required to fulfill the BuildRequires of the SRPM. After this it builds the SRPM in the minimal chroot environment. That way it's ensured the build is reproducible on any given system with all requirements to build the application clearly defined.

Configuration of mock

Mock uses configuration files located in /etc/mock - one of which is default.cfg and is usually a symlink to one of these files. Unless specified otherwise mock uses default.cfg to carry out the build.

It is important to keep in mind that mock does not use the host system's yum/dnf configuration so files in /etc/yum.repos.d are ignored. If any additional repositories are desired to be included in a mock based build (such as rpmfusion, IUS, NuxRepo, or a local/internal repo) it's best to create a new mock configuration (based on an existing one) including the additional repos required. The default symlink can be changed to point to this to avoid specifying the custom configuration each time.

It does carry out the installs from the point of view of the host system via --installroot so a file based local repo should still work, though if integrating into a larger build system or working with others an http based repo would be better to use.

Making use of mock to carry out a build

Mock can be used to create an SRPM from a spec file plus sources, but to carry out a build it needs to be passed an SRPM, not just a spec, so it can be common to call rpmbuild -bs just to get an SRPM that can then be passed to mock. This then does not require any BuildRequires to be installed as building an SRPM just creates the archive with any sources required and the spec to go with it.

It is incorrect and possibly dangerous to the system depending on the package to invoke mock as the root user. Either create a specific build user to switch to first (ensuring it is in the mock group) or add mock to the groups of your own user usermod -a -G mock `whoami`.

To carry out the build of myspecialsnowflake.src.rpm using the default mock configuration just pass it as an argument direct to mock:

mock myspecialsnowflake.src.rpm

If a specific configuration is required (for one including an internal/local repo, to build for CentOS on Fedora or to build for a different version of the distribution) the option -r is used to specify the configuration - note the .cfg extension is skipped.

mock -r epel-7-x86_64 myspecialsnowflake.src.rpm

The result of the mock build will be located in /var/lib/mock/ in a folder appropriate to the build target. The logs will be in a results directory under that and those are the first point of call for a failed build to check for things like missing dependencies. The results directory will have the RPMs resulting from the build on a success as well.

The resulting binary RPMs can then be placed in a local/internal repo for other systems to be able to install, depending on how the sources are being managed it can be worthwhile archiving the SRPM as well to easily reproduce in future.

This is worthwhile doing even on local systems with a quick basic RPM rather than installing from source so that it's trivial to track where files have come from and to rebuild systems in the future in a clean and minimal way.

Creating a local yum repo for centralised installs of custom packages

Once there are one or more RPMs in a directory it's very simple to create a local repository to use with yum/dnf either for the local system or as a centralised point for multiple in an organisation.

The package required for this is creatively named createrepo.

It's common to set up a repository hierarchy in a way that supports multiple distributions and versions. This way the same (or at least minimally different) repo files can be used between CentOS and Fedora versions by making use of the yum variables $basearch and $releasever for instance. Creating a repo with Apache httpd can be as simple as:

dnf install httpd
systemctl enable httpd
systemctl start httpd
mkdir -p /var/www/html/repos/fedora/23/x86_64
<Copy a bunch of RPMs into the directory>
cd /var/www/html/repos/fedora/23/x86_64
createrepo .
cat > /etc/yum.repos.d/myrepo.repo << EOF
[myrepo]
name=My custom repo 
baseurl=http://myhost/fedora/$releasever/$basearch
gpgcheck=0
enabled=1
EOF

The createrepo command takes the directory to the RPMs as an argument, and is recursive so packages can be split if the number of them in a single directory causes issues.

This is a very basic repository set up. For more advanced situations solutions such as Spacewalk, Katello or Pulp may be used and then RPMs imported using their particular clients and APIs.

To sign or not to sign

The sample yum repo above used gpgcheck=0. There's arguments both ways as to whether to sign internal packages.

  • It adds overhead to the package creation process
  • If packages get auto-signed how much security does this really add?
  • Checksums for package integrity are already in the RPM and repodata
  • In a business where auditors are involved mandating GPG signing can help show compliance
  • It's relatively trivial to configure
  • It doesn't make things worse to have signing

In Fedora and EL7 the signing mechanism was moved to rpm-sign but on EL6 or older rpm itself has the --addsign (or --resign which does the same in rpm-sign) argument.

In order to carry out signing of packages a working GPG configuration is required to be in place and the ~/.rpmmacros file needs some configuring.

If %_gpg_path is not set it'll use the user's default one, typically ~/.gpg, to locate the key to use. In most situations I prefer to keep rpm signing activities away from general user activities so this example will assume that. Depending on the environment this may not be required.

Create the keyring and key to sign with

mkdir ~/.rpmgpg
gpg --no-default-keyring --homedir ~/.rpmgpg --gen-key

The real name and email will appear in the GPG key when it's checked later so have something useful in there. A password is mandatory to work with the rpm-sign tool.

gpg: WARNING: unsafe permissions on homedir `/home/james/.rpmgpg/'
gpg (GnuPG) 1.4.20; Copyright (C) 2015 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

gpg: keyring `/home/james/.rpmgpg//secring.gpg' created
gpg: keyring `/home/james/.rpmgpg//pubring.gpg' created
Please select what kind of key you want:
   (1) RSA and RSA (default)
   (2) DSA and Elgamal
   (3) DSA (sign only)
   (4) RSA (sign only)
Your selection? 
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048) 
Requested keysize is 2048 bits
Please specify how long the key should be valid.
         0 = key does not expire
        = key expires in n days
      w = key expires in n weeks
      m = key expires in n months
      y = key expires in n years
Key is valid for? (0) 
Key does not expire at all
Is this correct? (y/N) y

You need a user ID to identify your key; the software constructs the user ID
from the Real Name, Comment and Email Address in this form:
    "Heinrich Heine (Der Dichter) "

Real name: RPM Signer
Email address: rpmsigner@example.com
Comment: 
You selected this USER-ID:
    "RPM Signer "

Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? o
You need a Passphrase to protect your secret key.

We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
.+++++
.+++++
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
..+++++
....+++++
gpg: /home/james/.rpmgpg//trustdb.gpg: trustdb created
gpg: key 8418EA8C marked as ultimately trusted
public and secret key created and signed.

gpg: checking the trustdb
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
pub   2048R/8418EA8C 2016-01-26
      Key fingerprint = 1DA7 76D3 7855 55C3 4AA3  B832 B9BA DD25 8418 EA8C
uid                  RPM Signer 
sub   2048R/62E3F0EB 2016-01-26

Extract the public key that can then be imported to systems

gpg --no-default-keyring --homedir ~/.rpmgpg/ --export -a "RPM Signer" > RPM-GPG-rpmsigner

Configure rpm to actually use these keys.

cat >> ~/.rpmmacros <<EOF
%_gpg_path ~/.rpmgpg
%_gpg_name RPM Signer <rpmsigner@example.com>
EOF

Sign an RPM with the new key

rpmsign --addsign foo-1.0-1.x86_64.rpm

In an environment with gpg-agent configured it's possible for gpg to get the passphrase from there. However in an automated environment something like an expect script would be required to provide the credentials to rpmsign, if there isn't a staging area that has someone manually sign packages before they are pushed to a repository.

Creating a lightweight build system with Jenkins

For a full fledged build system complete with multiple architectures, chain builds and specific build roots it's possible to mimic Fedora and install koji. This is a fairly heavyweight thing to get in place however.

For a majority of use cases it's substantially simpler to have Jenkins or a similar lightweight orchestration tool that can poll (or be hooked by) a code repository.

After installing jenkins configure the user that jenkins will run builds as (either locally on the jenkins master server or on the slaves depending on how the jenkins setup is designed).

At a minimal have the user jobs are run as in the mock group with mock, rpm-build and rpm-sign installed.

In order to rely on packages built internally make sure mock is configured with the internal repo as well.

Although --resultdir and --rootdir options can be set making use of ${WORKSPACE} keeping the different environments segregated basedir cannot be set via an argument to mock. Also if per build config files are used with different basedir options set to enable concurrent builds since the host system yum/dnf command is used they lock each other out from simultaneous actions anyway, resulting in a lot of "waiting for yumcache lock" messages and the various mocks taking turns to carry out activities. As a result I'd only suggest using --resultdir to bring the build artifacts under ${WORKSPACE} and having just one executor for RPM building jobs on each slave.

Chain builds where it's known dependencies exist so that if a library is updated then all packages dependent on that get automatically built.

The basic build steps should look something like:

  1. Checkout sources and spec from SCM (git clone foo) ... If using jenkins make use of the appropriate SCM plugins to automate this. If using git keep in mind large file issues and utilise git annex or spectool -g with a full URL to %{source} to get tarballs.
  2. Build the srpm: rpmbuild --define "_topdir $(pwd)" -bs SPEC/myspecialsnowflake.spec
  3. Build the rpm with mock: mock --resultdir=${WORKSPACE}/rpmresult SRPMS/myspecialsnowflake.fc23.src.rpm
  4. Optionally sign the RPM with the preconfigured GPG key: ./signrpm.sh ${WORKSPACE}/rpmresult/myspecial*.rpm ... The script should use something like expect to script the passphrase. An alternative specifically for Jenkins use could be the RPM signing plugin.
  5. Push the rpm to the repo: scp ${WORKSPACE}/rpmresult/*.x86_64.rpm reposerver:/var/www/html/repos/fedora/23/x86_64/
  6. Create the repo metadata: ssh reposerver 'cd /var/www/html/repos/fedora/23/x86_64 && createrepo .'

This could of course be further improved by having jenkins automatically grab the rpm artifact and have it used by a job that specifically creates the repository metadata. This could then be used to prevent multiple systems trying to run createrepo at once. Or if using pulp or spacewalk their pulp-admin and rhnpush commands respectively could be used to upload the rpm through their API which entirely sidesteps the createrepo concurrency issue.

Add new comment