The Goal.
I’m trying to set up a locally available Ubuntu mirror, from which I can install software on any computer that I manage, without having to download the software packages from the internet onto every computer on which I wish to install them. It is definitely not my intention to burn the downloaded files to DVD (or whichever other appropriate medium).
Preparatory Steps.
As an initial test run, I decide to select only a small sample from the (huge!) Ubuntu repositories. I prefer not to perform the “Big Download” until I can get my small sample to work. I decide to settle on the “restricted” section of the Ubuntu Jaunty Jackalope release—which is some 59 MiB in size, and, therefore, perfectly manageable for my test run.
First, I create a directory in which I can download the repository files:
Code:
mkdir ~/UbuntuRepos
Next, I install the “debmirror” package:
Code:
sudo apt-get install debmirror
As BobSongs documented in the first post of this thread, I can now issue the following command to download my selected sample from an online Ubuntu mirror:
Code:
debmirror --nosource --md5sums --passive --host=online-repository-host --root=ubuntu \
--method=http --progress --dist=jaunty --section=restricted --arch=i386 \
--ignore-release-gpg \
~/UbuntuRepos
Notes:
- In the command above, you will have to replace the “online-repository-host” with your selected Ubuntu mirror—e.g., with the generic “archive.ubuntu.com” or with your national mirror (which, in my case, would be “be.archive.ubuntu.com”).
- I use “--method=http” instead of ftp, since the latter apparently won’t work on my national Ubuntu mirror.
With the necessary infrastructure in place, I can add my local mirror to my “/etc/apt/sources.list” file, to make it known to the APT system. I add the following line to the top of the file:
Code:
deb file:///home/luvr/UbuntuRepos/ jaunty restricted
Note:
You will need
root privileges in order to edit the
“/etc/apt/sources.list” file. For example, if you want to use the
“gedit” text editor, you will have to run the following command:
Code:
sudo gedit /etc/apt/sources.list
Alternatively, you can start the editor from the GNOME desktop by pressing
<Alt>-<F2> and running the following command line:
Code:
gksudo gedit /etc/apt/sources.list
Make sure to add the line to the top of the file; if the system finds the same package in multiple places (e.g., once in your local repository, and once in a repository on the internet), it will load it from the first location in the sources.list file where it finds the package. In other words, if the internet repository is specified before the local repository in sources.list, then the internet repository will be preferred, and the local repository will not be used.
Then, I refresh my local package indexes:
Code:
sudo apt-get update
Experiment 1: Will APT Use my Local Mirror?
To see if the APT system will actually use the packages that I downloaded, I run the following command:
Code:
sudo apt-get install fglrx-amdcccle
This command produces the following output:
Code:
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
dkms fakeroot fglrx-kernel-source xorg-driver-fglrx
Suggested packages:
libamdxvba1
The following NEW packages will be installed:
dkms fakeroot fglrx-amdcccle fglrx-kernel-source xorg-driver-fglrx
0 upgraded, 5 newly installed, 0 to remove and 0 not upgraded.
Need to get 24.6MB of archives.
After this operation, 69.0MB of additional disk space will be used.
Do you want to continue [Y/n]?
Note:
Since I don’t actually want to install any packages at this time, I reply “N” to the question if I want to continue.
My observations:
- The output includes the message that “24.6MB of archives” will be downloaded from online repositories. In other words, my locally available repository is ignored.
This doesn’t really come as a surprise, since the local repository doesn’t have a digital signature—which was excluded from the download as a result of the “--ignore-release-gpg” option that I specified on the “debmirror” command. As a result, my local repository will not be trusted—i.e., the APT system will ignore it whenever it can, and prefer to download packages from an online, but trusted repository instead. - Two packages that are not available in my local repository, will be installed as well: “dkms” and “fakeroot.” To ensure that they won’t interfere with my further experiments, I decide to install them separately, using the following command:
Code:
sudo apt-get install dkms fakeroot
Experiment 2: Will “debmirror” Download the Digital Signature if I Ask it to?
Since I want my local mirror to become trusted, I want to ensure that it includes a digital signature—i.e., its “Release.gpg” file. Therefore, for my next experiment, I will omit the “--ignore-release-gpg” option from the “debmirror” command line:
Code:
debmirror --nosource --md5sums --passive --host=online-repository-host --root=ubuntu \
--method=http --progress --dist=jaunty --section=restricted --arch=i386 \
~/UbuntuRepos
This command produces the following output:
Code:
Mirroring to /home/luvr/UbuntuRepos from http://online-repository-host/ubuntu/
Arches: i386
Dists: jaunty
Sections: restricted
Passive mode on.
Checking md5sums.
Will clean up AFTER mirroring.
Pdiff mode: use.
Attempting to get lock, this might take 2 minutes before it fails.
Get Release files.
[0%] Getting: dists/jaunty/Release... ok
[0%] Getting: dists/jaunty/Release.gpg... ok
gpgv: keyblock resource `/home/luvr/.gnupg/trustedkeys.gpg': general error
gpgv: Signature made Wed 22 Apr 2009 11:35:26 PM CEST using DSA key ID 437D05B5
[GNUPG:] ERRSIG 40976EAF437D05B5 17 2 00 1240436126 9
[GNUPG:] NO_PUBKEY 40976EAF437D05B5
gpgv: Can't check signature: public key not found
Release signature does not verify.
Errors:
Release signature does not verify.
Failed to download some Release or Release.gpg files!
WARNING: releasing 1 pending lock...
Clearly, the “debmirror” command does now download the “Release.gpg” file, after which it invokes the “gpgv” utility (i.e., the “OpenPGP signature verification tool”) to verify the digital signature. The “gpgv” utility looks for the appropriate public key in a keyring file named “trustedkeys.gpg,” which it expects to find in my “.gnupg” directory. Since that keyring file does not exist, the program cannot verify the digital signature, and will return an error. When “debmirror” subsequently sees this error, it apparently deletes the “Release.gpg” file—which it assumes to be in error.
Note:
I could, at this point, decide to download the
“Release.gpg” file manually, using the following commands:
Code:
cd ~/UbuntuRepos/dists/jaunty
wget http://online-repository-host/ubuntu/dists/jaunty/Release.gpg
If I subsequently refresh my local indexes, and rerun the command to install one of the packages from my local mirror, then the system will tell me that it won’t need to download any packages. Indeed, if I execute the commands:
Code:
sudo apt-get update
sudo apt-get install fglrx-amdcccle
then the output includes the following line:
Code:
Need to get 0B/24.4MB of archives.
In other words, nothing will need to be downloaded from the online repositories, but the packages will be read from my local mirror instead.
I prefer an automatic solution, though—where
“debmirror” will download the
“Release.gpg” file for me, and be able to verify it.
Remedy: Getting “debmirror” to Download and Verify the Digital Signature.
As I mention above, the “debmirror” command attempts to verify the “Release.gpg” file against a public key that resides in the “~/.gnupg/trustedkeys.gpg” keyring file. The trick to getting “debmirror” to accept the digital signature, then, must be, to create this keyring file, and importing the appropriate public key into it.
Fortunately, the “debmirror” man page hints at a possible solution, in that it contains the following text snippet:
Code:
Debmirror uses gpgv to verify Release and Release.gpg using the
default keying ~/.gnupg/trustedkeys.gpg. [...]
To add the right key to this keyring you can import it from the
debian keyring (in case of the debian archive) using:
gpg --keyring /usr/share/keyrings/debian-archive-keyring.gpg --export \
| gpg --import
Note:
Incidentally, this command needs a little tweaking to make it work right—but more on that in a moment.
So, apparently, the “/usr/share/keyrings” directory is the place where the appropriate keyring should be found. Thus, I run the following command to list the files that are located there:
Code:
ls -l /usr/share/keyrings
This produces the following output:
Code:
total 16
-rw-r--r-- 1 root root 2198 2008-04-20 13:05 medibuntu-keyring.gpg
-rw-r--r-- 1 root root 6713 2008-03-04 05:20 ubuntu-archive-keyring.gpg
-rw------- 1 root root 0 2009-04-20 16:00 ubuntu-archive-removed-keys.gpg
-rw-r--r-- 1 root root 1227 2008-03-04 05:20 ubuntu-master-keyring.gpg
To find out which keys are available in, e.g., the “ubuntu-archive-keyring.gpg” file, I run the following command:
Code:
gpg --no-default-keyring --keyring /usr/share/keyrings/ubuntu-archive-keyring.gpg --list-keys
This command produces the following output:
Code:
/usr/share/keyrings/ubuntu-archive-keyring.gpg
----------------------------------------------
pub 1024D/437D05B5 2004-09-12
uid Ubuntu Archive Automatic Signing Key <ftpmaster@ubuntu.com>
sub 2048g/79164387 2004-09-12
pub 1024D/FBB75451 2004-12-30
uid Ubuntu CD Image Automatic Signing Key <cdimage@ubuntu.com>
Returning to the error messages that “debmirror” produced when I wanted it to include the “Release.gpg” file in its download, I notice the key ID that it was looking for:
Code:
gpgv: keyblock resource `/home/luvr/.gnupg/trustedkeys.gpg': general error
gpgv: Signature made Wed 22 Apr 2009 11:35:26 PM CEST using DSA key ID 437D05B5
[GNUPG:] ERRSIG 40976EAF437D05B5 17 2 00 1240436126 9
[GNUPG:] NO_PUBKEY 40976EAF437D05B5
gpgv: Can't check signature: public key not found
The required key is available in the “ubuntu-archive-keyring.gpg” file (it’s the one with user id “Ubuntu Archive Automatic Signing Key <ftpmaster@ubuntu.com>”). Therefore, this is the keyring from which I will build my “~/.gnupg/trustedkeys.gpg” keyring file.
Thus, I run the following (tweaked—see above) command to create the keyring:
Code:
gpg --no-default-keyring --keyring /usr/share/keyrings/ubuntu-archive-keyring.gpg --export \
| gpg --no-default-keyring --keyring trustedkeys.gpg --import
Note:
You can run the following command if you want to verify that the expected keys were loaded into the new keyring file:
Code:
gpg --no-default-keyring --keyring ~/.gnupg/trustedkeys.gpg --list-keys
You should see the same keys listed as for the
“/usr/share/keyrings/ubuntu-archive-keyring.gpg” file.
When I rerun the “debmirror” command to bring my local mirror up-to-date (including the “Release.gpg” file), the digital signature can now be correctly verified:
- Command:
Code:
debmirror --nosource --md5sums --passive --host=online-repository-host --root=ubuntu \
--method=http --progress --dist=jaunty --section=restricted --arch=i386 \
~/UbuntuRepos
- Output from the command:
Code:
Mirroring to /home/luvr/UbuntuRepos from http://online-repository-host/ubuntu/
Arches: i386
Dists: jaunty
Sections: restricted
Passive mode on.
Checking md5sums.
Will clean up AFTER mirroring.
Pdiff mode: use.
Attempting to get lock, this might take 2 minutes before it fails.
Get Release files.
[0%] Getting: dists/jaunty/Release... ok
[0%] Getting: dists/jaunty/Release.gpg... ok
Get Packages and Sources files and other miscellany.
[ 53%] Getting: dists/jaunty/restricted/binary-i386/Packages.bz2... ok
Parse Packages and Sources files and add to the file list everything therein.
Download all files that we need to get (1 MiB).
Downloaded 1 MiB in 31s at 2.63 kiB/s
Everything OK. Moving meta files.
Cleanup mirror.
All done.
Experiment 3: Will APT Use my Local Mirror Now?
Now that I believe that my local mirror is properly trusted, I refresh my local indexes again, and I retry the command to install the “fglrx-amdcccle” package:
Code:
sudo apt-get update
sudo apt-get install fglrx-amdcccle
This time, the software installation command produces the following output:
Code:
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
fglrx-kernel-source xorg-driver-fglrx
Suggested packages:
libamdxvba1
The following NEW packages will be installed:
fglrx-amdcccle fglrx-kernel-source xorg-driver-fglrx
0 upgraded, 3 newly installed, 0 to remove and 0 not upgraded.
Need to get 0B/24.4MB of archives.
After this operation, 68.2MB of additional disk space will be used.
Do you want to continue [Y/n]?
I observe that the packages will not be downloaded from the online repositories, but that they will be taken from my local mirror instead. In other words, I now have a trusted local Ubuntu mirror, from which I can install software packages.
Wrapping Up.
If you want to create a trusted local Ubuntu mirror, from which you can install software packages, then you will need to take the following steps:
- Create a directory in which you can download the repository files—e.g.:
Code:
mkdir ~/UbuntuRepos
- Install the “debmirror” package:
Code:
sudo apt-get install debmirror
- Create the “~/.gnupg/trustedkeys.gpg” keyring, to allow “debmirror” to verify the digital signature of the Ubuntu repository:
Code:
gpg --no-default-keyring --keyring /usr/share/keyrings/ubuntu-archive-keyring.gpg --export \
| gpg --no-default-keyring --keyring trustedkeys.gpg --import
- Run the “debmirror” command to download the repository:
Code:
debmirror --nosource --md5sums --passive --host=online-repository-host --root=ubuntu \
--method=http --progress --dist=distribution --section=section-list --arch=arch-list \
~/UbuntuRepos
Note:
Omit the “--nosource” option if you want to download not only the binary packages, but the sources as well.
- Add the following line to the top of the “/etc/apt/sources.list” file:
Code:
deb file:///home/username/UbuntuRepos/ distribution section-list
- Refresh your local indexes:
Code:
sudo apt-get update
Now you can install software packages from your locally available mirror.
Whenever you want to bring your local mirror up-to-date, simply rerun the “debmirror” command, and refresh your local indexes again.
ADDENDUM: How Do I Set Up a Local “Medibuntu” Mirror?
Important:
Before we go any further, let’s return to a critically important question that
BobSongs answered in his original post:
Q:
What if I wanted to combine or add in more repositories?
Something like Canonical Commercial? Or MediBuntu?
A:
Don't do it!
[...]
The quick way to add files from different servers (such as MediBuntu and Canonical) is to put the additional files in unique folders.
Examples:
~/MediBuntuRepos and/or
~/CanonicalRepos (your folder names may vary).
Thus, you will need to create a new directory—e.g.,
“~/MediBuntuRepos”—for your local Medibuntu mirror; do
not try to add the Medibuntu software packages to your existing
“~/UbuntuRepos” directory.
I repeat:
Do not try to add the Medibuntu software packages to your existing “~/UbuntuRepos” directory!
Please reread this statement until you can resist the temptation to try it anyway, or you will destroy your existing Multi-Gigabyte mirror!
Note:
I assume that you have already added the Medibuntu repository as a software source to your system. If you haven’t, then here’s a quick way to do it now:
- Add the following line to your “/etc/apt/sources.list” file:
Code:
deb http://packages.medibuntu.org/ jaunty free non-free
- Run the following command to refresh your local package indexes:
Code:
sudo apt-get update
An error message will be displayed, because the public key for the Medibuntu repository is not available. - Run the following command to install the public key for the Medibuntu repository:
Code:
sudo apt-get --allow-unauthenticated install medibuntu-keyring
- Rerun the command to refresh your local package indexes:
Code:
sudo apt-get update
The error message will no longer appear.
The public key for the Medibuntu repository should be present in your “/usr/share/keyrings” directory:
Code:
$ ls -l /usr/share/keyrings/medibuntu-keyring.gpg
-rw-r--r-- 1 root root 2198 2008-04-20 13:05 /usr/share/keyrings/medibuntu-keyring.gpg
If you attempt to list the public keys available in this file, however, then an error message will appear:
Code:
$ gpg --no-default-keyring --keyring /usr/share/keyrings/medibuntu-keyring.gpg --list-keys
gpg: [don't know]: invalid packet (ctb=2d)
gpg: keydb_search_first failed: invalid packet
In fact, the name of the file—“medibuntu-keyring.gpg”—is a bit of a misnomer, since the file isn’t really a keyring.
You can use the “file” command to try and identify the type of any file—e.g.:
Code:
$ file /usr/share/keyrings/ubuntu-archive-keyring.gpg
/usr/share/keyrings/ubuntu-archive-keyring.gpg: GPG key public ring
$ file /usr/share/keyrings/medibuntu-keyring.gpg
/usr/share/keyrings/medibuntu-keyring.gpg: PGP public key block
Apparently, the “medibuntu-keyring.gpg” file is a “PGP public key block”—as opposed to “ubuntu-archive-keyring.gpg,” which is a “GPG key public ring.”
Note:
The
“medibuntu-keyring.gpg” file is actually a plain text file (though its contents won’t make much sense).
You can open it with any text editor, or view its contents with the
“cat” command, or with the
“more” or
“less” pagers—e.g.:
Code:
cat /usr/share/keyrings/medibuntu-keyring.gpg
Since the “medibuntu-keyring.gpg” file is a public key block (instead of a ring), you can use the following command to import it into the “~/.gnupg/trustedkeys.gpg” keyring file (where “debmirror” expects to find it):
Code:
gpg --no-default-keyring --keyring trustedkeys.gpg \
--import < /usr/share/keyrings/medibuntu-keyring.gpg
Note:
Importing additional keys into an existing keyring file won’t remove any of the existing keys in the keyring; the new keys will simply be added to it.
You can subsequently download the Medibuntu software packages into your new local mirror as follows:
Code:
debmirror --nosource --md5sums --passive --host=packages.medibuntu.org --root=/ \
--method=http --progress --dist=jaunty --section=free,non-free --arch=i386 \
~/MediBuntuRepos
Note:
It is well worth repeating again (and again, and again, ...) that you should be extra careful to specify the correct directory—e.g., “~/MediBuntuRepos”—for your local Medibuntu mirror, or you risk damaging your existing repository mirror!
ADDENDUM: What if “debmirror” complains because it cannot find the “Packages.bz2” file?
See my post entitled “ERROR: "debmirror" fails if the online repository does not have a "Packages.bz2" file” (later in this thread).
ADDENDUM: Why does “debmirror” complain about “duplicate dists” on the “Canonical” repository?
See my post entitled “ERROR: "debmirror" cannot download the "Canonical" repository” (later in this thread).
ADDENDUM: How can I install the Flash Player Plugin from my local mirror, without internet access?
See my post entitled “Installing the Flash Player Plugin from a Local Mirror, without Internet Access.” (later in this thread).
ADDENDUM: How do I set up a Local Mirror with Source Packages only?
See my post entitled “How to Create a "Source-Only" Local Mirror.” (later in this thread).
ADDENDUM: Can I perform a partial download, and control exactly which files will get downloaded?
If you want to limit a “debmirror” run, then you can specify the “--max-batch=number” parameter, where “number” represents the maximum number of files that you wish to download. You cannot, however, instruct “debmirror” exactly which files you want to download, or specify a size limit for the download.
To obtain much more fine-grained control over the files that get selected for download, you will have to learn about some further utilities—as I describe in my post entitled “Having Fun with the "debmirror" Dry Run—and Learning Something New Along the Way.” (later in this thread).
CAUTION: Changing your “--host” Parameter May Confuse “debmirror”!
The Issue.
When I set up my local mirror, I initially downloaded from my national mirror,
“be.archive.ubuntu.com”; in other words, I specified my
“--host” parameter as follows:
Code:
--host=be.archive.ubuntu.com
I started with a small sample that consisted only of the
“restricted” section—i.e., with the following
“--section” parameter value:
Code:
--section=restricted
Once that worked, I added in the
“main” section:
Code:
--section=main,restricted
Unfortunately, my national mirror had rather inconsistent data transfer rates, and caused far too many timeouts, and consequently, the download took much longer than I found acceptable. Therefore, when I got ready to add the
“universe” section (which is even bigger than
“main”!), I decided to switch to the generic mirror,
“archive.ubuntu.com,” to see if that would work any better. My
“--host” and
“--section” parameter values, then, looked like this:
Code:
--host=archive.ubuntu.com --section=main,restricted,universe
When I fired up
“debmirror” again, everything seemed to go fine, so I left the computer alone to complete the download.
Sometime later, when I returned to the computer, I noticed that it had finished the download, but to my horror, it displayed a whole sequence of lines saying that it had
deleted packages from the “main” section! When I wanted to check what had happened to the
“main” section, … it had apparently vanished—
“debmirror” had effectively eradicated it!
I retried the
“debmirror” command, but that didn’t help any; I did see the following messages, which indicated that the
“Packages” files for the
“main” section were being downloaded alright:
Code:
Getting: dists/jaunty-backports/main/binary-i386/Packages.bz2... ok
Getting: dists/jaunty-proposed/main/binary-i386/Packages.bz2... ok
Getting: dists/jaunty-security/main/binary-i386/Packages.bz2... ok
Getting: dists/jaunty-updates/main/binary-i386/Packages.bz2... ok
Getting: dists/jaunty/main/binary-i386/Packages.bz2... ok
Even so,
“debmirror” refused to download any of the packages from the
“main” section, but chose to ignore them instead. My local mirror appeared to be badly damaged.
It won’t come as a surprise that I hadn’t made a backup yet—in keeping with the established traditions, I make backups only after the facts...
I dreaded the idea of having to start all over again, though—especially since I still had a perfectly good copy of the
“universe” section; having to redownload just the
“main” section was bad enough already!
The Work-Around.
In addition to the
“dists,” the
“pool,” and the
“project” directories, the
“debmirror” command will create a fourth, hidden directory in the location that you selected for your local mirror:
Code:
$ ls -lA ~/UbuntuRepos
drwxr-xr-x 7 luvr luvr 4096 2009-06-02 17:35 dists
drwxr-xr-x 6 luvr luvr 4096 2009-06-02 01:04 pool
drwxr-xr-x 3 luvr luvr 4096 2009-06-02 11:38 project
drwxr-xr-x 3 luvr luvr 4096 2009-06-02 11:30 .temp
Apparently,
“debmirror” will download the
“Release” and
“Packages” files (plus the
“Sources” files, if you omit the
“--nosource” command-line option) there first, to ensure that the copies in their final locations won’t get overwritten until the repository download completes successfully.
You can safely delete this
“.temp” directory before you run
“debmirror” again—if the directory doesn’t exist, then
“debmirror” will simply recreate it. In fact, deleting the
“.temp” directory was all that it took to repair my local mirror—though, of course, I still had to redownload the complete
“main” section.
Conclusion.
If your local mirror appears to malfunction, then it may be a good idea to delete its
“.temp” directory:
Code:
cd ~/UbuntuRepos
rm -R .temp
Note:
If you want to use the graphical desktop environment to delete the—hidden—“.temp” directory, then you will have to turn on the “Show Hidden Files” option from the Nautilus file browser “View” menu.
If you subsequently retry the
“debmirror” command, then it should work normally again.
Actually, deleting the
“.temp” directory is a pretty harmless operation. Whenever you take any action that you suspect may affect the proper operation of the
“debmirror” command, you may want to delete the
“.temp” directory as a preventative measure.
References.
Adv Reply
), I stopped after a few seconds.
Originally Posted by egalvan
Bookmarks