package management

(Written by Paul Cobbaut, https://github.com/paulcobbaut/, with contributions by: Alex M. Schapelle, https://github.com/zero-pytagoras/, Bert Van Vreckem https://github.com/bertvv/)

Most Linux distributions have a package management system with online repositories containing thousands of packages. This makes it very easy to install, update and remove applications, operating system components, documentation and much more.

In this module, we introduce some basic terminology and concepts. Details about managing packages on the major distributions, specifically the Debian and Enterprise Linux families, are covered in separate modules. In this module, we also discuss a two additional types of package management tools that are not distribution-specific.

Some programming languages have their own package managers (e.g. pip for Python).
In the 2010s, containerization became popular and started to be used as a solution for package management independent of the Linux distribution, e.g. snap and flatpak.

Finally, we show how you can install software outside of the package management system, starting from the source code.

package terminology

repository

A lot of software and documentation for your Linux distribution is available as packages in one or more centrally distributed repositories. The packages in such a repository are tested and very easy to install (or remove) with a graphical or command line installer.

.deb packages

Debian, Ubuntu, Mint and all derivatives of Debian and Ubuntu use .deb packages. To manage software on these systems, you can use apt or apt-get, both these tools are a front end for dpkg.

.rpm packages

Red Hat, Fedora, CentOS, OpenSUSE, Mandriva, Red Flag and others use .rpm packages. The tools to manage software packages on these systems are dnf and rpm.

dependency

Some packages need other packages to function. Tools like apt-get, apt and dnf will install all dependencies you need. When using dpkg or rpm, or when building from source, you will need to install dependencies yourself.

open source

These repositories contain a lot of independent open source software. Often the source code is customized to integrate better with your distribution. Most distributions also offer this modified source code as a package in one or more source repositories.

You are free to go to the project website itself (samba.org, apache.org, github.com ...) and download the vanilla (= without the custom distribution changes) source code.

GUI software management

End users have several graphical applications available via the desktop (look for add/remove software or something similar).

Below a screenshot of Ubuntu Software Center running on Ubuntu 12.04. Graphical tools are not discussed in this book.

pip, the Python package manager

Some programming languages, a.o. Python, have their own package management system that allows you to install applications and/or libraries. In the case of Python, the package manager is called pip. It is used to install Python packages from the Python Package Index (PyPI). In fact, there are multiple package managers for Python (a.o. easy_install, conda, etc.), but pip is the most widely used.

As a system administrator, or as an end user, this sometimes puts you in a difficult position. Some widely known and used Python libraries can be installed both through your distribution's package manager, and through pip. Which one to choose is not always clear. In general, it is best to use the distribution's package manager, as it will integrate the package into the system and will be updated when the system is updated. However, some packages are not available in the distribution's repositories, or the version you get with pip is more recent. In that case, you can use pip to install the package.

Another thing to note is that pip can be used as a normal user, or as root, and in each case it will install the package in a different location. When you install a package as a normal user, it will be installed in your home directory, and will only be available to you. When you install a package as root, it will be installed system-wide, and will be available to all users. However, if you install a package as root, you will get a warning message:

WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

A virtual environment is a way to create an isolated environment for a Python project, where you can install packages without affecting the system's Python installation. This is especially useful when you are developing Python applications, and you want to make sure that the libraries you use are the same as the ones used in production. Using and managing virtual environments is beyond the scope of this course, but you can find more information in the Python documentation.

As general guidelines, we suggest the following:

If the library or application is available in the distribution's repositories, use the distribution's package manager to install it.
Do not install Python libraries or applications system-wide as root using pip.
Normal users may use pip to install Python libraries or applications in their home directory.
Use virtual environments per project to manage Python libraries and applications.

installing pip

By default, pip is probably not installed on your system. You can install it using your distribution's package manager.

For example, on Red Hat-based systems, you can install it using dnf:

student@el ~$ sudo dnf install python3-pip

On Debian-based systems, you can install it using apt:

student@debian:~$ sudo apt install python3-pip

However, some Linux distributions like Linux Mint 22 enforce the guidelines mentioned above. When you try to install a Python package using pip, you will get an error message:

student@linuxmint:~$ sudo apt install python3-pip
student@linuxmint:~$ pip install pandas
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
    python3-xyz, where xyz is the package you are trying to
    install.

    If you wish to install a non-Debian-packaged Python package,
    create a virtual environment using python3 -m venv path/to/venv.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
    sure you have python3-full installed.

    If you wish to install a non-Debian packaged Python application,
    it may be easiest to use pipx install xyz, which will manage a
    virtual environment for you. Make sure you have pipx installed.

    See /usr/share/doc/python3.12/README.venv for more information.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.

In this case, you can't use pip at all to install Python packages, not even as a user. So, you should use the distribution's package manager to install Python packages system-wide and virtual environments to install Python packages on a project-by-project basis.

listing packages

You can list the packages installed with pip using the list command:

student@linux:~$ pip list
Package         Version
--------------- --------
dbus-python     1.2.18
distro          1.5.0
gpg             1.15.1
libcomps        0.1.18
nftables        0.1
pip             21.2.3
PyGObject       3.40.1
python-dateutil 2.8.1
PyYAML          5.4.1
rpm             4.16.1.3
selinux         3.5
sepolicy        3.5
setools         4.4.3
setuptools      53.0.0
six             1.15.0
systemd-python  234

searching for packages

Searching for packages can NOT be done on the command line. To search for packages, you can use the Python Package Index website instead. If you try pip search, you will get an error message:

student@linux:~$ pip search ansible
ERROR: XMLRPC request failed [code: -32500]
RuntimeError: PyPI no longer supports 'pip search' (or XML-RPC search). Please use https://pypi.org/search (via a browser) instead. See https://warehouse.pypa.io/api-reference/xml-rpc.html#deprecated-methods for more information.

installing packages

You can install a package using the install command:

student@linux:~$ pip install ansible

Just like apt and dnf, pip will install the package and its dependencies.

removing packages

Uninstalling a package is done with the uninstall command:

student@linux:~$ pip uninstall ansible

Unfortunately, dependencies are not removed when you uninstall a package with pip.

using a virtual environment

A virtual environment allows you to install Python packages in an isolated environment, so they don't interfere with the system's Python installation. You can create a virtual environment using the venv module:

student@linuxmint:~$ sudo apt install python3-venv
[...output omitted...]
student@linuxmint:~$ mkdir pyproject
student@linuxmint:~$ cd pyproject
student@linuxmint:~/pyproject$ python3 -m venv .venv

This will create a new directory called .venv in the current directory, which contains scripts and tools to set up the virtual environment. You can choose the name of the directory, but .venv is a common convention. By adding the dot at the beginning of the directory name, it is hidden from the output of the ls command.

student@linuxmint:~/pyproject$ ls .venv/
bin  include  lib  lib64  pyvenv.cfg
student@linuxmint:~/pyproject$ ls .venv/bin/
activate  activate.csh  activate.fish  Activate.ps1  f2py  numpy-config  pip  pip3  pip3.12  python  python3  python3.12

You can activate the virtual environment using the source command:

student@linuxmint:~/pyproject$ source .venv/bin/activate
(.venv) student@linuxmint:~/pyproject$

The source command will execute the script in the current environment (i.e. without creating a subshell). When the virtual environment is activated, the prompt will change to indicate that you are now working in the virtual environment. You can now use pip to install Python packages, and they will be installed in the virtual environment, not in the system.

(.venv) student@linuxmint:~/pyproject$ pip install pandas
Collecting pandas
  Downloading pandas-2.2.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (89 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 89.9/89.9 kB 2.1 MB/s eta 0:00:00
[...some output omitted...]
Installing collected packages: pytz, tzdata, six, numpy, python-dateutil, pandas
Successfully installed numpy-2.1.2 pandas-2.2.3 python-dateutil-2.9.0.post0 pytz-2024.2 six-1.16.0 tzdata-2024.2

Packages will be installed in the directory .venv/lib/python<version>/site-packages.

It is good practice to explicitly list the packages you have installed in the virtual environment in a requirements.txt file. This file can be used to recreate the virtual environment on another system. You can create the file using the freeze command, or edit it manually.

(.venv) student@linuxmint:~/pyproject$ pip freeze > requirements.txt
(.venv) student@linuxmint:~/pyproject$ cat requirements.txt 
numpy==2.1.2
pandas==2.2.3
python-dateutil==2.9.0.post0
pytz==2024.2
six==1.16.0
tzdata==2024.2

When you are done working in the virtual environment, you can deactivate it using the deactivate command:

(.venv) student@linuxmint:~/pyproject$ deactivate
student@linuxmint:~/pyproject$

If you keep the code in this directory in a Git repository, it's best to add the .venv directory to the .gitignore file, so it is not included in the repository.

container-based package managers

With the release of Docker, container-based virtualization has become very popular as a method of distributing and deploying applications on servers. One of the advantages of containers is that they offer a sandbox environment for applications, meaning the application and its dependencies are isolated from the rest of the system. This makes it possible to run applications with different dependencies on the same server, without the risk of conflicts. Containers are also very lightweight, they don't impose much overhead on the host system.

Now, there is no reason why containers can't be used to deploy applications on desktop systems as well. In fact, there are several container-based package managers that allow you to install and run applications in containers on your desktop. The advantage is that third party software vendors can distribute their applications independent of the Linux distribution, so they don't need to maintain different packages for (each family of) distribution(s). The disadvantage is that each application comes with their own dependencies, so you lose the advantage of sharing libraries between applications. Also, since the application is running in a container, it may not integrate well with the rest of the system, or may have only limited permissions to access files or other resources on your computer.

As with many Linux-based technologies, there are multiple tools to choose from. The most popular ones are Flatpak and Snap.

flatpak

Flatpak is a container-based package manager developed by an independent community of contributors, volunteers and supporting organizations. It is available for most Linux distributions and is supported by a large number of third party software vendors. Red Hat was one of the first to endorse Flatpak, and many others followed. Fedora Silverblue is a variant of Fedora that uses Flatpak as its primary package manager. Linux Mint also has Flatpak support enabled by default: in the Software Manager, some applications like Bitwarden, Slack, VS Code, etc. are available as Flatpaks.

If you want to use a container based package manager, Flatpak is probably the best choice for any Linux distribution other than Ubuntu.

In the following example, we'll install the open source password manager Bitwarden with Flatpak on a Linux Mint system. Remark that you don't need to be root to install Flatpak applications!

student@mint:~$ flatpak search Bitwarden
Name       Description                            Application ID         Version  Branch Remotes
Bitwarden  A secure and free password manager for com.bitwarden.desktop  2024.2.0 stable flathub
Goldwarden A Bitwarden compatible desktop client  com.quexten.Goldwarden 0.2.13   stable flathub
student@mint:~$ flatpak install Bitwarden
Looking for matches…
Found ref ‘app/com.bitwarden.desktop/x86_64/stable’ in remote ‘flathub’ (system).
Use this ref? [Y/n]: y
Required runtime for com.bitwarden.desktop/x86_64/stable (runtime/org.freedesktop.Platform/x86_64/23.08) found in remote flathub
Do you want to install it? [Y/n]: y

com.bitwarden.desktop permissions:
    ipc                    network                      wayland       x11       dri       file access [1]
    dbus access [2]        system dbus access [3]

    [1] xdg-download
    [2] com.canonical.AppMenu.Registrar, org.freedesktop.Notifications, org.freedesktop.secrets, org.kde.StatusNotifierWatcher
    [3] org.freedesktop.login1


        ID                                   Branch      Op Remote  Download
 1. [✓] com.bitwarden.desktop.Locale         stable      i  flathub 300.7 kB / 9.8 MB
 2. [✓] org.freedesktop.Platform.GL.default  23.08       i  flathub 162.0 MB / 162.3 MB
 3. [✓] org.freedesktop.Platform.GL.default  23.08-extra i  flathub  17.9 MB / 162.3 MB
 4. [✓] org.freedesktop.Platform.Locale      23.08       i  flathub  17.9 kB / 359.9 MB
 5. [✓] org.freedesktop.Platform             23.08       i  flathub 171.6 MB / 225.6 MB
 6. [✓] com.bitwarden.desktop                stable      i  flathub 132.5 MB / 133.4 MB

Installation complete.

To remove a Flatpak application, you can use the uninstall command:

student@mint:~$ flatpak uninstall Bitwarden
Found installed ref ‘app/com.bitwarden.desktop/x86_64/stable’ (system). Is this correct? [Y/n]: y


        ID                                   Branch         Op
 1. [-] com.bitwarden.desktop                stable         r
 2. [-] com.bitwarden.desktop.Locale         stable         r

Uninstall complete.

snap

Snap was developed by Canonical and is installed by default on Ubuntu. It is also available for other distributions (like the official Ubuntu derivatives, Solus and Zorin OS), but it is not as widely supported as Flatpak. Snap was also designed to work for cloud applications and Internet of Things devices.

In the following example, we'll install Grafana on an Ubuntu Server system.

student@ubuntu:~$ snap search grafana
Name          Version Publisher  Notes Summary
grafana       6.7.4   canonical✓ -     feature rich metrics dashboard and graph editor        
grafana-agent 0.35.4  0x12b      -     Telemetry Agent
[...]
student@ubuntu:~$ sudo snap install grafana
grafana 6.7.4 from Canonical✓ installed

To uninstall a Snap application, you can use the remove command:

student@ubuntu:~$ sudo snap remove grafana
grafana removed

downloading software outside the repository

These days, the case where you need software that is not available as a binary package has become exceedingly rare. However, if you want to install some experimental tool that hasn't been packaged yet, or you want to test the very latest experimental version of an application, you may have to download the source code and compile it yourself. Usually, the source code is available on the project's website or on a code hosting platform like GitHub, GitLab or Bitbucket. You then either download the source code as a tgz, .tar.gz, .tar.bz2, tar.xz file (also called a tarball) or you can clone the repository using git.

In the example below, we assume that you have downloaded the source code of an application written in C or C++, as is common for many Linux applications. Remark that in order to be able to compile the source code, you need to have the C compiler gcc and the build tool make installed on your system. You can install these using your distribution's package manager. Also, many applications depend on other libraries, which also have to be installed as source.

example: compiling zork

As an example, we will download the source code for Zork, an ancient text based adventure game, and compile it on a Fedora system. The source code is available on GitHub. Remark that you should check wheter git, gcc and make are installed beforehand. See the modules on RPM and DEB package management for more information.

[student@fedora ~]$ git clone https://github.com/devshane/zork.git
Cloning into 'zork'...
remote: Enumerating objects: 79, done.
remote: Total 79 (delta 0), reused 0 (delta 0), pack-reused 79
Receiving objects: 100% (79/79), 241.70 KiB | 2.14 MiB/s, done.
Resolving deltas: 100% (20/20), done.
[student@fedora ~]$ cd zork/
[student@fedora zork]$ ls
actors.c  demons.c  dmain.c  dso3.c  dso6.c  dtextc.dat  dverb2.c  history   Makefile  np2.c  nrooms.c  README.md   sobjs.c   vars.h
ballop.c  dgame.c   dso1.c   dso4.c  dso7.c  dungeon.6   funcs.h   lightp.c  nobjs.c   np3.c  objcts.c  readme.txt  supp.c    verbs.c
clockr.c  dinit.c   dso2.c   dso5.c  dsub.c  dverb1.c    gdt.c     local.c   np1.c     np.c   parse.h   rooms.c     sverbs.c  villns.c
[student@fedora ~]$ make
cc -g    -c -o actors.o actors.c
cc -g    -c -o ballop.o ballop.c
cc -g    -c -o clockr.o clockr.c
[...etc...]
cc -g  -o zork actors.o ballop.o clockr.o demons.o dgame.o dinit.o dmain.o dso1.o dso2.o dso3.o dso4.o dso5.o dso6.o dso7.o dsub.o dverb1.o dverb2.o gdt.o lightp.o local.o nobjs.o np.o np1.o np2.o np3.o nrooms.o objcts.o rooms.o sobjs.o supp.o sverbs.o verbs.o villns.o -ltermcap
/usr/bin/ld: cannot find -ltermcap: No such file or directory
collect2: error: ld returned 1 exit status
make: *** [Makefile:69: dungeon] Error 1

As you can see, the make command fails because it cannot find the termcap library. This is a library that is used to control the terminal, and it is not installed on our system. This is a common problem when you try to install packages from source. You need to install these dependencies yourself and these are not always easy to find. In this case, we can install the ncurses-devel library, which is a modern replacement for termcap. How did we now that? We used dnf provides to find library files that contain the string termcap (remark that the command took a long time to finish):

[student@fedora zork]$ dnf provides '*libtermcap.so*'
Last metadata expiration check: 1:56:05 ago on Mon 26 Feb 2024 05:46:43 PM UTC.
ncurses-devel-6.4-7.20230520.fc39.i686 : Development files for the ncurses library
Repo        : fedora
Matched from:
Other       : *libtermcap.so*
[student@fedora ~]$ sudo dnf install ncurses-devel
[...etc...]

Let's try to compile again:

[student@fedora zork]$ make
cc -g    -c -o actors.o actors.c
cc -g    -c -o ballop.o ballop.c
[...etc...]
cc -g    -c -o villns.o villns.c
cc -g  -o zork actors.o ballop.o clockr.o demons.o dgame.o dinit.o dmain.o dso1.o dso2.o dso3.o dso4.o dso5.o dso6.o dso7.o dsub.o dverb1.o dverb2.o gdt.o lightp.o local.o nobjs.o np.o np1.o np2.o np3.o nrooms.o objcts.o rooms.o sobjs.o supp.o sverbs.o verbs.o villns.o -ltermcap
[student@fedora zork]$

The command seems to have succeeded. The current directory now contains a new file called zork. This is the compiled application and it has execute permissions. You can run it by typing ./zork:

[student@fedora zork]$ ls -l zork
-rwxr-xr-x. 1 student student 400968 Feb 26 19:45 zork
[student@fedora zork]$ file zork
zork: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=3089e3cb1c1a7fc1cc1db41c3aa578c0b52f83f3, for GNU/Linux 3.2.0, with debug_info, not stripped
[student@fedora zork]$ ./zork
Welcome to Dungeon.                     This version created 11-MAR-91.
You are in an open field west of a big white house with a boarded
front door.
There is a small mailbox here.
>

In this case, installing the game is as simple as copying the zork file to a directory in your PATH, like /usr/local/bin or (for a computer game) /usr/local/games. However, most Makefiles provide a way to install the application in the system, usually by running make install. This will copy the executable, manual pages and other documentation to the correct location.

[student@fedora zork]$ sudo make install
mkdir -p /usr/games  /usr/share/man/man6
cp zork /usr/games
cp dtextc.dat /usr/games/lib
cp dungeon.6 /usr/share/man/man6/

Remark that the "official" location where manually installed applications belong in a Linux directory structure is /usr/local (for applications that follow the Filesystem Hierarchy Standard) or /opt (for applications that want to keep all files in a single directory).

installing from a tarball

Before unpacking a tarball, it's useful to check its contents:

student@linux:~$ tar tf $downloadedFile.tgz

The t option lists the content of the archive, f should be followed by the filename of the tarball. For .tgz, you may add option z and for .tar.bz2 option j. However, the tar command should recognize the compression method automatically.

Check whether the package archive unpacks in a subdirectory (which is the preferred case) or in the current directory and create a subdirectory yourself if necessary. After that, you can unpack the tarball:

student@linux:~$ tar xf $downloadedFile.tgz

Now, be sure to read the README file carefully! Normally the readme will explain what to do after download.

Usually the steps are always the same three:

running a script ./configure. It will gather information about your system that is needed to compile the software so that it can actually run on your system
executing the command make (which is the actual compiling)
finally, executing make install to copy the files to their proper location.

practice: package management

Look up where your Linux distribution has published their package repositories. What is the URL? Can you find and download individual packages from there?
Open the graphical package manager tool on your system. What kind of software is available? Can you find the software you need? Or games that you would like to play?
Why would it be a bad idea to install Python packages with pip as root? When would you install a package (e.g. pandas) with pip instead of the package manager (e.g. python3-pandas)?
Check whether Snap or Flatpak is installed or available on your system. How can you search online for applications in these repositories?

solution: package management

Look up where your Linux distribution has published their package repositories. What is the URL? Can you find and download individual packages from there?

For Ubuntu, you can start by going to http://archive.ubuntu.com/ubuntu/. Information about which packages are available can be found in the dists directory. For example, the release Noble Nombat (24.04 LTS) has several subdirectories called noble, noble-backports, noble-proposed, and noble-security. The pool directory contains the actual .deb files.

For Fedora, you can go to https://mirrormanager.fedoraproject.org and look for a mirror for the Fedora release you are using and the architecture of your machine (typically x86_64). An example of such a repository mirror is https://fedora.cu.be/linux/releases/ (in Belgium). You can then navigate to the subdirectory of your release (e.g. 40), and then to the Everything directory, x86_64, os, and Packages to find the RPM packages (sorted in subdirectories a to z according to the first letter of the package name).
Open the graphical package manager tool on your system. What kind of software is available? Can you find the software you need? Or games that you would like to play?

On Ubuntu, you can use the Software Center. On Fedora, you can use the Software application. Both tools provide a graphical interface to search for and install software, just like you would on the Appstore, Google Play store, Windows store, etc. In fact, graphical package managers on Linux predate these stores by many years.

You may not necessarily find commercial applications, but usually open source alternatives exist.

Installing games is also possible, but the selection is not as large as on Steam. However, you can install Steam on Linux and play many games that are available on that platform.
Why would it be a bad idea to install Python packages with pip as root? When would you install a package (e.g. pandas) with pip instead of the package mnanager (e.g. python3-pandas)?

Installing Python packages with pip as root could interfere with the Python packages installed with the system package manager. Generally, if you need Python packages to be available system-wide, use the system package manager.

If you only need the package for a specific user or a specific Python virtual environment, you can use pip. This is especially useful if the package is not available in the system package manager, or if you need a specific version of the package that is not available in the system package manager.
Check whether Snap or Flatpak is installed or available on your system. How can you search online for applications in these repositories?

On Ubuntu, Snap is installed by default. You can search for applications in the Snap store by going to https://snapcraft.io/store. You can also use the snap find command to search for applications from the command line.

On most other distributions, Flatpak is the default (if it is installed). You can search for applications in the Flathub repository by going to https://flathub.org/. You can also use the flatpak search command to search for applications from the command line.

Remark that installing Snap is usually also possible on distributions that default to Flatpak.