Building packages on a clean-room environment
This HOWTO tries to solve the common problems related to package building
- Prevent installing unwanted cruft on your system by doing the build process on a chroot (as a sandboxed, jailed) environment
- Making sure that the package works for other people too who are trying it
- Automatically downloaded dependencies make process smoother
- (Easier to automate larger scale building process and testing)
pbuilder is a tool which is
"used for creating and maintaining chroot environment and building Debian package in the chroot environment".
l'll be doing all the configuration locally only, by overriding the defaults. Another way is to make pbuilder settings to /etc/pbuilderrc
and create your hooks on /etc/pbuilder/hook.d accordingly
Installing
- Install necessary packages
Code:
sudo aptitude install pbuilder devscripts
- Create the pbuilder environment
Code:
sudo pbuilder create
Now we have ready chroot environment for package building using Ubuntu defaults from /etc/pbuilderrc
(You can test environment by doing sudo pbuilder login. Command exit will leave the environment)
- Override the default pbuilder settings
These are the file contents.
Code:
## Overrides /etc/pbuilderrc
# Default distribution
#DISTRIBUTION=feisty
COMPONENTS="main restricted universe multiverse"
# Repositories
MIRRORSITE=http://archive.ubuntu.com/ubuntu
#OTHERMIRROR="deb ${MIRRORSITE} ${DISTRIBUTION}-updates ${COMPONENTS}|deb ${MIRRORSITE} ${DISTRIBUTION}-security ${COMPONENTS}"
# For build results
BUILDRESULT=${HOME}/pbuilder/result
# Hooks for chroot environment
HOOKDIR=${HOME}/pbuilder/hook.d
# Mount directories inside chroot environment
BINDMOUNTS=${BUILDRESULT}
# Bash prompt inside pbuilder
export debian_chroot="pbuild$$"
# For D70results hook
export LOCALREPO=${BUILDRESULT}
# Always include source package
#DEBBUILDOPTS="-sa"
- Create a hook that allows to include local packages as dependencies. This feature is ofter necessary
when the package you're building depends on other package(s) which is not found on Ubuntu repositories.
Code:
mkdir -p ~/pbuilder/result
mkdir ~/pbuilder/hook.d
nano ~/pbuilder/hook.d/D70results
These are the file contents
Code:
#!/bin/sh
echo "Executing hook: $0"
cd ${LOCALREPO}
dpkg-scanpackages . /dev/null > Packages
echo "deb file:${LOCALREPO} ./" >> /etc/apt/sources.list
apt-get update
This code block creates a list of the packages in ${LOCALREPO} folder for APT to use as repository.
Hook is always executed inside the chroot environment before the build phase, and packages are available
for APT to use on dependency resolving.
- Make the hook file executable
Code:
chmod +x ~/pbuilder/hook.d/D70results
- Update the chroot environment, so that new modules (repositories) are included too.
(Also, If you later need to change your chroot environment, use this command)
Code:
sudo pbuilder update --override-config
Building packages
- Building the packages should be as easy as
Code:
mkdir -p ~/packages/mypackage
cd ~/packages/mypackage
apt-get source mypackage
Then either
Code:
sudo pbuilder build mypackage_x.x.x.dsc
or
Code:
cd mypackage-x.x.x
pdebuild --use-pdebuild-internal
- Succefully built packages are placed on ${BUILDRESULT} folder
- These packages are also used as dependencies, so if you get weird behaviour when building other packages,
clean the contents of this folder and try again
Updating the base build environment
- You should update the build environment when new updates are released for the base system
Code:
sudo pbuilder update --autocleanaptcache
A real life example: nautilus
Code:
# enable Ubuntu source repositories on /etc/apt/sources.list
mkdir -p ~/packages/nautilus
cd ~/packages/nautilus
apt-get source nautilus
sudo pbuilder build nautilus_2.14.1-0ubuntu9.dsc
#or alternatively do pdebuild --use-pdebuild-internal on nautilus folder
Notes
- Prefer pbuilder instead of pdebuild wrapper
- pbuilder uses .orig.tar.gz and .diff.gz to build the package.
- If you change anything inside the source folder, source package must be rebuilt for pbuilder (or use pdebuild instead).
- To rebuild source package
Code:
dpkg-source -b folder
- /usr/share/doc/pbuilder/examples contains some nice example hooks, especially C10shell which is invoked when something goes wrong on build process.
You can find the build target files in /tmp/buildd. - If you want to pass DEB_BUILD_OPTIONS to pbuilder and you're using sudo, you must prevent reset of certain environment variables. Using visudo, add definition
Code:
Defaults env_keep+="DEB_* BUILD* PATH"
- Default dependency resolution logic using apt-get is quite slow. To make it way more faster, change the resolver to gdebi. Add to your ~/.pbuilderrc
Code:
# use gdebi for dependency resolution
PBUILDERSATISFYDEPENDSCMD=/usr/lib/pbuilder/pbuilder-satisfydepends-gdebi
- pdebuild doesn't support hooks.
- pdebuild ignores BUILDRESULT variable set on /etc/pbuilderrc. You can specify it using --buildresult. see pdebuild man page for details.
- if you want to improve pbuilder's speed, check out package called cowdancer
- Useful way to clean out obsolete packages from pbuilder cache
Code:
sudo pbuilder update --autocleanaptcache
References
[edit]
- Debian version of pbuilder doesn't currently have support for COMPONENTS option
- added OTHERMIRROR to ~/.pbuilderrc. This allows to use additional Ubuntu repositories. Not enabled by default.
If you enable it, use pbuilder update --overrideconfig to make the changes apply - added PBUILDERSATISFYDEPENDSCMD to ~/.pbuilderrc. Makes dependency resolution very fast by using gdebi instead (In Ubuntu package only)
Re: Building packages on a clean-room environment
As second usecase example, we'll test how self-buit packages which are not in official repositories
work with this pbuilder setup.
Picture a scenario where you would want to build a package that's not on any Ubuntu repository,
but Debian upsteam has it available. You've cheked out the changelogs and verified that no major
dependencies have changed. If library is on main Ubuntu repository, think twice before actually upgrading it.
I won't be inserting any changelog info on the packages (or changing package naming), but you may do so if you want.
Use dch -iDdapper on the directory where sources were extracted (contains debian/ subdirectory).
Code:
cd mypackage-x.xx
dch -iDdapper
gmpc
- Add debian unstable to your /etc/apt/sources.list
Code:
deb-src http://ftp.uk.debian.org/debian unstable main contrib non-free
- Get the sources
Code:
mkdir -p ~/packages/gmpc
cd ~/packages/gmpc
sudo aptitude update
apt-get source gmpc
- Try to build it first time
Code:
sudo pbuilder build gmpc_0.13.0-2.dsc
When you fist time try to build this package, you'll notice that it fails miserably because some dependency is missing.
This is usually caused by missing module (respository) on pbuilderrc or like in this case, unmet version number although package is available.
Code:
-> Considering libmpd-dev (>= 0.12.0)
Tried versions: 0.01-3
-> Does not satisfy version, not trying
Poking the Ubuntu package info (or use apt tools for this) for libmpd you notice that it's really too old for satisfying
the build-time requirement. Debian upstream luckily provides updated version.
libmpd
- Get libmpd source from debian unstable
Code:
mkdir -p ~/packages/libmpd
cd ~/packages/libmpd
apt-get source libmpd
- Build libmpd
Code:
sudo pbuilder build libmpd_0.12.0-2.dsc
When build process is finished, new packages are available on /var/cache/pbuilder/result. Good thing is that
pbuilder chroot environment that was set previously, can see these packages too (using the provided hook).
gmpc (2nd try..)
- Back to building gmpc
Code:
cd ~/packages/gmpc
sudo pbuilder build gmpc_0.13.0-2.dsc
Install
- Install builded packages
Code:
cd /var/cache/pbuilder/result
sudo dpkg -i libmpd0_0.12.0-2_i386.deb gmpc_0.13.0-2_i386.deb
Re: Building packages on a clean-room environment
mplayer
Initial setup
- Disable any multiverse source repositories from /etc/apt/sources.list
- Add new source repository to /etc/apt/sources.list
Code:
deb-src http://www.debian-multimedia.org sid main
- Update package list
Code:
sudo aptitude update
ffmpeg
- Build ffmpeg first
Code:
mkdir -p ~/packages/ffmpeg
cd ~/packages/ffmpeg
apt-get source libavcodeccvs51-dev
- NOTE if you get error "can't find a register in class 'GENERAL_REGS' while reloading 'asm'" you must edit debian/rules and
change CFLAGS to appropriate values. For my system I had to use
Code:
CFLAGS = -Wall
LDCONFIG =
ifneq (,$(findstring noopt,$(DEB_BUILD_OPTIONS)))
CFLAGS += -O0 -fPIC -DPIC
else
CFLAGS += -pipe -fomit-frame-pointer
- If you edited anything inside the source folder, you must regenerate the diffs against the original package. (Argument is the source folder)
Code:
dpkg-source -b ffmpegcvs-20060625
- Build
Code:
sudo pbuilder build ffmpegcvs_20060625-0.0.dsc
libdivxdecore0
- Build libdivxdecore0
Code:
mkdir -p ~/packages/libdivxdecore0
cd ~/packages/libdivxdecore0
apt-get source libdivxdecore0
sudo pbuilder build divx4linux_5.0.1-1.dsc
libvstream
- Build libvstream
Code:
mkdir -p ~/packages/libvstream
cd ~/packages/libvstream
apt-get source libvstream-dev
sudo pbuilder build vstream-client_1.2-0.1.dsc
mplayer
- Get source first
Code:
mkdir -p ~/packages/mplayer
cd ~/packages/mplayer
apt-get source mplayer
- I had to change mplayer-skin (virtual) package dependency to mplayer-skins. To do this, modify debian/control.marillat file
on mplayer source folder and either remove mplayer-skin dependency from mplayer-custom Package or change it to mplayer-skins
Code:
Package: mplayer
Architecture: any
Depends: ${shlibs:Depends}, mplayer-skins
- Regenerate the diffs against the original package (if you changed dependecies)
Code:
dpkg-source -b mplayer-1.0-pre8
- Build
Code:
DEB_BUILD_OPTIONS=marillat sudo pbuilder build mplayer_1.0-pre8-0.0.dsc
Install
- Install mplayer and dependencies
Code:
sudo aptitude install libjack0.100.0-0 libdirectfb-0.9-22 libsvga1
cd /var/cache/pbuilder/result/
sudo dpkg -i mplayer_1.0-pre8-0.0_i386.deb libavcodeccvs51_20060625-0.0_i386.deb libdivxdecore0_5.0.1-1_i386.deb libavutilcvs49_20060625-0.0_i386.deb
Re: Building packages on a clean-room environment
pbuilder can also be used to test your build scripts or other stuff on a sandboxed environment.
Code:
cp -R ~/packages/mypackage /tmp
sudo pbuilder login --bindmount /tmp/mypackage
apt-get install nano
(start editing)
Now /tmp/mypackage is accessible/visible on chrooted environment (on / folder). Be aware that changes you make on --bindmounted
folders are persisted and because you're logged as root, you'll get some unwanted permission issues. It may be
better to use temporary folders.
Every package that you install, or any other change you make (excluding --bindmount folders) are not preserved.
So you can safely do almost anything you want.
If you login using --save-after-login, then all changes you make on the environment are persisted.
It's convinient way to install nano for example, which is not included by default.
Re: Building packages on a clean-room environmend
These guides are great mlind. Would it be possible to take the debian folder from say a older version of a program and use it for the latest cvs? How do you recommend going about compiling things without source debs?
Re: Building packages on a clean-room environmend
Quote:
Originally Posted by PsychoTrauma
These guides are great mlind. Would it be possible to take the debian folder from say a older version of a program and use it for the latest cvs? How do you recommend going about compiling things without source debs?
Yes it's possible and you should do it like that. Let's say you have source package
foo-1.0.3 that's already been been debianized (it contains the debian folder)
and you'd like use that with new version foo-1.0.4 that you've downloaded.
Code:
~/packages/foo$ ls
foo-1.0.3 foo_1.0.3-1.dsc foo-1.0.4.tar.gz
foo_1.0.3-1.diff.gz foo_1.0.3.orig.tar.gz
To apply foo-1.0.3 debian folder contents (it will actually patch using .diff.gz file)
Code:
cd foo-1.0.3
uupdate ../foo-1.0.4.tar.gz
Quote:
Originally Posted by PsychoTrauma
After building files with the method above how would you go about removing everything so your system is the same before you started?
What method do you mean? If you use pbuilder then your system stays the same.
Everything is installed on chrooted environment which is always untarred from
same archive for each pbuilder session.
Downloaded packages are stored on /var/cache/pbuilder/aptcache, which you
can clean using pbuilder clean.
Sometimes, if you ctrl-c the pbuilder process /proc is not correctly umounted from
chroot environment and as result, /var/cache/pbuilder/build/ may contain folders
left behind by the build process. To get rid of these, see /etc/mtab and umount
everything pbuilder related, then rm -rf build folder's contents.
If you wanted to remove everything that this guide made you to install, then
Code:
sudo aptitude purge pbuilder fakeroot devscripts
and make sure /var/cache/pbuilder doesn't exist.
Re: Building packages on a clean-room environment
Quote:
Originally Posted by PsychoTrauma
How do you recommend going about compiling things without source debs?
You should read http://www.debian.org/doc/maint-guide/
To debianize the source, use debhelper dh_make (see man debhelper). This will
create debian folder with initial scripts inside, maintainer guide explains briefly what
files are for. I also suggest to check out cdbs which is intended to do package maintaining easier.
You can test your packaging rules on pbuilder environment by doing
- pbuilder login --bindmounts yourtestfolder
- test by executing debian/rules build (test install and clean targets too)
- after executing install target, make sure debian/yourpackage/ contents look okay
Re: Building packages on a clean-room environment
This is really great mlind !
Now I don't need to install tons of -dev packages on my main system : D
Great How-To!
I wish there was a GUI to debianize source packages tho (yeah, I'm lazy) :P
Re: Building packages on a clean-room environment
Quote:
Originally Posted by
LKRaider
This is really great mlind !
Now I don't need to install tons of -dev packages on my main system : D
Great How-To!
I wish there was a GUI to debianize source packages tho (yeah, I'm lazy) :P
Thanks, that was why I wanted to look for alternative solutions, keep unwatend cruft out and be sure that package would actually work for others too.
I've modified proposed pbuilder configuration a bit. My BUILDRESULT, where builded packages appear is BUILDRESULT=/home/myusername/pbuilder/result, which makes package handling little easier.
You can set aliases to pbuilder commands to make invoking little non-verbosive, but working gui would be nice too.
Here's few tips too
- If you are testing packages using pbuilder login --bindmounts /path/to/mypackage, you can copy /usr/lib/pbuilder/pbuilder-satisfydepends to your hookdir and call it inside chroot environment to automatically fetch and install all dependencies.
- Hooks are found on /tmp/hooks/ inside chroot environment, and may be called explicitly too.
- You can run X-server applications "the easy, but not right" -way inside chroot, by bindmounting your /tmp folder to chroot environment. This contains necessary files for applications to use X. You also need to grant privileges for root user. This can be done by using xhost on normal environment.
(Remember to umount bindmounted /tmp before exiting the environment)
Re: Building packages on a clean-room environmend
I seem to be having trouble with the new changes. Pbuilder no longer sees the files I have compiled previously. A example is I compiled libx264-dev for the newer version on ffmpeg. I go to compile ffmpeg and it can't find the libx264-dev deb in ~/pbuilder/result.
I also should add this is on a fresh install so I never had the previous method set up yet.