How To: Make Your Own Ubuntu Repository DVDs

[Black_Tanto]

Good info in the OP.


Can anybody tell me if its possible to download only the apps that are already installed on a particular distro including the apps that it came with.....AptonCD wont work for this because it only downloads apps as deb files for apps you downloaded yourself. I am trying to export the specific apps that one release has and simply ADD it to another.

Any help please?

I do not want to download the entire repo, just what came on the disk.

[luvr]

Can anybody tell me if its possible to download only the apps that are already installed on a particular distro including the apps that it came with

I can't give a complete answer to your question off-hand, but if you want to obtain a list of the packages that are currently installed on your system, then you can run the following command:

dpkg --get-selections

Then, for each of the packages that appear in the list, you can run the following:

sudo apt-get --download-only --reinstall install package-name

This will download the specified package(s) into the /var/cache/apt/archives directory.

[luvr]

The Issue.

You are trying to install the Adobe Flash Player Plugin software package (i.e., “flashplugin-installer”) on a computer that has only limited internet access. Because of the limited connectivity, you install from a local software repository mirror.

Since you are installing from a local mirror, you are hoping that the software will not need to be downloaded from the internet, but that it will be taken from your local mirror instead.

Your assumption initially appears to be confirmed when you run the following command to perform the installation:

sudo apt-get install flashplugin-installer

Indeed, the first few lines of output from this command look like this:

Reading package lists... Done
Building dependency tree
Reading state information... Done
Suggested packages:
  xulrunner-1.9 konqueror-nsplugins ttf-xfree86-nonfree xfs
The following packages will be upgraded:
  flashplugin-installer
1 upgraded, 0 newly installed, 0 to remove and 7 not upgraded.
Need to get 0B/19.9kB of archives.
After this operation, 0B of additional disk space will be used.

You notice, specifically, the line that reads “Need to get 0B/19.9kB of archives.”

From this output line, you draw the conclusion that, as expected, the software packages will not be downloaded from the internet, but that the files that you want to install, will be taken from your local repository instead.

However, as the operation continues, you discover that the installer still attempts to download a “.tar.gz” archive file; this download will, obviously, fail if the computer does not have an active internet connection—e.g.:

Setting up flashplugin-installer (10.1.85.3ubuntu0.10.04.1) ...
Downloading...
--2010-09-21 22:04:11--  http://archive.canonical.com/pool/partner/a/adobe-flashplugin/adobe-flashplugin_10.1.85.3.orig.tar.gz
Resolving archive.canonical.com... failed: Name or service not known.
wget: unable to resolve host address `archive.canonical.com'
download failed
The Flash plugin is NOT installed.

This document will explain how you can examine the “flashplugin-installer” package, and find a way to avoid having the package download the archive file during installation.

Note:

This document assumes, as an example, that the local Ubuntu software mirror is located in the following directory:

/LinuxDistros/Ubuntu-10.04/Repositories/Ubuntu

Likewise, the document refers to a local Canonical software mirror in the following directory:

/LinuxDistros/Ubuntu-10.04/Repositories/Canonical

If your local mirrors are located elsewhere, then, obviously, you will have to modify the paths to your mirrors accordingly.

Note:

This document is based on my earlier, similar, post on Installing the Flash Player Plugin from a customised local software repository, but it was updated to a more recent version of the Flash Player Plugin.

1. Getting a Copy of the “flashplugin-installer” Package.

Since you are installing from your local software repository mirror, the “flashplugin-installer” package should be available in your mirror. To find out where exactly it is located, you can run the following command:

apt-cache showpkg flashplugin-installer

The first few lines of output from this command look like this:

Package: flashplugin-installer
Versions:
10.1.85.3ubuntu0.10.04.1 (/var/lib/apt/lists/_LinuxDistros_Ubuntu-10.04_Repositories_Ubuntu_dists_lucid-security_multiverse_binary-amd64_Packages) (/var/lib/apt/lists/_LinuxDistros_Ubuntu-10.04_Repositories_Ubuntu_dists_lucid-updates_multiverse_binary-amd64_Packages) (/var/lib/dpkg/status)

This output identifies the directory that contains the package list (i.e., the “Packages” file) in which the “flashplugin-installer” package is described:

/LinuxDistros/Ubuntu-10.04/Repositories/Ubuntu/dists/lucid-security/multiverse/binary-amd64

If you list the contents of this directory, then you will see that, apparently, only the compressed copies of the package list, “Packages.bz2” and “Packages.gz,” are available. You can open either copy with the “Archive Manager,” and you will find that they contain a “Packages” file—which you can open in the text editor.

Note:

As an alternative, you can use the “less” page viewer to view either copy in a command-line shell window. Just move to the directory that contains the two copies, and then run the following command:

less Packages.gz

The “flashplugin-installer” package is documented in this file under the following heading:

Package: flashplugin-installer

One of the items in this section is the “Filename:” line, which looks like this:

Filename: pool/multiverse/f/flashplugin-nonfree/flashplugin-installer_10.1.85.3ubuntu0.10.04.1_amd64.deb

Thus, the package file name is “flashplugin-installer_10.1.85.3ubuntu0.10.04.1_amd64.deb,” and the file is located in the following directory:

/LinuxDistros/Ubuntu-10.04/Repositories/Ubuntu/pool/multiverse/f/binary-amd64/flashplugin-nonfree

Copy the file, “flashplugin-installer_10.1.85.3ubuntu0.10.04.1_amd64.deb,” from this directory to an easily accessible location—e.g., to your home directory.

2. Extracting the Control Information from the Package.

To take a look at what a software package actually does when you install it, you will have to extract its control information—which includes configuration settings and script files to guide the installation process (as well as the uninstallation process).

To extract the control information from a Debian package file, you can use the “dpkg-deb” command, and pass it the “--control” option. The following command, for instance, will extract the control information from your copy of the “flashplugin-installer” package file:

dpkg-deb --control flashplugin-installer_10.1.85.3ubuntu0.10.04.1_amd64.deb

This command will not produce any visible output, but it will create a “DEBIAN” directory, in which it copies the control information from the package file. You can run the following command to obtain a list of the extracted control files:

ls -l DEBIAN

The output will look something like this:

total 36
-rwxr-xr-x 1 luvr luvr 1457 2010-09-21 01:40 config
-rw-r--r-- 1 luvr luvr 1942 2010-09-21 01:40 control
-rw-r--r-- 1 luvr luvr  334 2010-09-21 01:40 md5sums
-rwxr-xr-x 1 luvr luvr 5126 2010-09-21 01:40 postinst
-rwxr-xr-x 1 luvr luvr  206 2010-09-21 01:40 postrm
-rwxr-xr-x 1 luvr luvr 2505 2010-09-21 01:40 prerm
-rw-r--r-- 1 luvr luvr 5438 2010-09-21 01:40 templates

Two of these files are of particular interest:

• config
  This is the script that will drive the configuration of the installation process.
  In the case of the “flashplugin-installer” package, for example, this script will decide if the supporting files will have to be downloaded, or have been made available locally.
• postinst
  This is the script that will take any post-installation steps required to make the installed software fully functional.

These two script files can help you determine which files you will have to make available locally, and where you have to copy them, in order to keep the installer from attempting to download any further files during the installation process.

3. Examining the “config” File.

If you open the “config” file in any text editor, then you will see the following code fragments near the top of the file:

FLASH_VERSION=10.1.85.3
FILENAME=adobe-flashplugin_${FLASH_VERSION}.orig.tar.gz
SHA256SUM_TGZ="cd58b8550aa6c2e99d2b392f854a985e2d73a38d47a0f6d7709b69e9b4dae3e3"

db_get flashplugin-installer/local
echo "$SHA256SUM_TGZ  $RET/$FILENAME" \
| sha256sum -c > /dev/null 2>&1 \
|| db_set flashplugin-installer/local /var/cache/flashplugin-installer

This code apparently looks for a file named “adobe-flashplugin_10.1.85.3.orig.tar.gz” in a directory named “/var/cache/flashplugin-installer”; if it finds the file there, then it will run the “sha256sum” utility to verify the integrity of the file.

Presumably, if the file is found, and if its checksum matches, then the Flash player plugin will be installed from this local copy; otherwise, the script will force a download from the internet during the installation process—though it is not yet clear from which online location.

4. Examining the “postinst” File.

Next, if you open the “postinst” file in any text editor, then you will find the following code near the top of the file:

FLASH_VERSION=10.1.85.3
FILENAME=adobe-flashplugin_${FLASH_VERSION}.orig.tar.gz
SHA256SUM_TGZ="cd58b8550aa6c2e99d2b392f854a985e2d73a38d47a0f6d7709b69e9b4dae3e3"
PARTNER_URL=http://archive.canonical.com/pool/partner/a/adobe-flashplugin/$FILENAME

This code appears to set up the URL from which the file, “adobe-flashplugin_10.1.85.3.orig.tar.gz,” will be downloaded (if no valid local copy of the file is available).

The file will be downloaded from the following online directory resource:

http://archive.canonical.com/pool/partner/a/adobe-flashplugin/

Incidentally, this is a location within the online Canonical software repository; if you created a local mirror of this repository, then your system will have downloaded this resource into a local directory—e.g.:

/LinuxDistros/Ubuntu-10.04/Repositories/Canonical/pool/partner/a/adobe-flashplugin

For easy access, you may want to temporarily copy the “adobe-flashplugin_10.1.85.3.orig.tar.gz” file from this location to, e.g., your home directory. If you want to do this from the command-line shell, then just move to the directory that contains the file, and run the following command:

cp adobe-flashplugin_10.1.85.3.orig.tar.gz ~

Note:

Alternatively, if you have not set up a local mirror of the Canonical software repository, you will temporarily need internet access to download the file, e.g., with the following command:

wget http://archive.canonical.com/pool/partner/a/adobe-flashplugin/adobe-flashplugin_10.1.85.3.orig.tar.gz

5. OPTIONAL: Verifying the Checksum.

If you wish, you can now verify the checksum of the “adobe-flashplugin_10.1.85.3.orig.tar.gz” file, with the following command:

sha256sum -b adobe-flashplugin_10.1.85.3.orig.tar.gz

The output from this command should look like this:

cd58b8550aa6c2e99d2b392f854a985e2d73a38d47a0f6d7709b69e9b4dae3e3 *adobe-flashplugin_10.1.85.3.orig.tar.gz

In particular, the checksum value (i.e., the 64-character string with which the line begins) should match the SHA256SUM_TGZ value that you encountered in both the “config” and “postinst” scripts.

Alternatively, you can first create a checksum verification file—i.e., a file with the following contents, all on one line:

• The expected checksum of the file that you want to verify—i.e., the SHA256SUM_TGZ value, which you can copy from the “config” and “postinst” scripts:
  

  cd58b8550aa6c2e99d2b392f854a985e2d73a38d47a0f6d7709b69e9b4dae3e3

• A blank space, followed by an asterisk:
  

   *

• The name of the file that you want to verify:
  

  adobe-flashplugin_10.1.85.3.orig.tar.gz

(Note that the resulting line is identical to the output from the “sha256sum” command, above.)
Save this file as, e.g., “SHA256,” and run the following command:

sha256sum -c SHA256

This time, the output should look like this:

adobe-flashplugin_10.1.85.3.orig.tar.gz: OK

6. Installing the Flash Player Plugin, without Internet Access.

You are now ready to install the Flash Player Plugin from your local software repository mirror, without requiring internet access during the process. Before you do so, you will have to create the “/var/cache/flashplugin-installer” directory, and copy the “adobe-flashplugin_10.1.85.3.orig.tar.gz” file into it. Just run the following commands:

sudo mkdir /var/cache/flashplugin-installer
sudo cp adobe-flashplugin_10.1.85.3.orig.tar.gz /var/cache/flashplugin-installer

You can subsequently install the plugin with, e.g., the following command:

sudo apt-get install flashplugin-installer

The following output line will, then, confirm that the plugin will be installed from the locally available copy of the “adobe-flashplugin_10.1.85.3.orig.tar.gz” file:

Installing from local file /var/cache/flashplugin-installer/adobe-flashplugin_10.1.85.3.orig.tar.gz

[klein de usa]

hi
this is just what i'v been looking for, but there is one problem for me. is it possible to start and stop the download ? i only have 625 mb a day i can download except between 2am and 7am if i could start downloading them at 2am then stop at 7am and then start them again at 2am the next night with a cron script?

[luvr]

is it possible to start and stop the download ?

You can interrupt the download at any time. Then, when you restart it, it will pick up where it left off.

i only have 625 mb a day i can download except between 2am and 7am if i could start downloading them at 2am then stop at 7am and then start them again at 2am the next night with a cron script?

There must, indeed, be a way to schedule the download to start at a particular time of day (e.g., 2 AM), and then to interrupt it some time later (e.g., 7 AM). A cron job sounds like a good idea to me, though I'm not sufficiently familiar with cron to give you any advice on it.

[emanamini]

Hey GOOD JOB MEN

[luvr]

The Problem.

Note:

This post applies to Ubuntu 10.04 (a.k.a. the “Lucid Lynx”) and Ubuntu 10.10 (a.k.a. the “Maverick Meerkat”)—which apparently share the same version of the “debmirror” utility.
The error also occurs with the “debmirror” version that, at the time of writing, is available with the Ubuntu 11.04 (a.k.a. the “Natty Narwhal”) pre-release.
Even though the “debmirror” version that comes with “natty” is different from the version distributed with “maverick” and “lucid,” the patch that I provide as an attachment to this post, will work on either version of the “debmirror” utility.

I’m trying to set up a local mirror for the Canonical “Partner” repository—similar to the local Ubuntu mirror that I documented in an earlier post in this thread.

First, I create a directory in which I will download the repository files:

mkdir ~/PartnerRepos

Next, I perform a “debmirror” test run (using the “--dry-run” option):

debmirror --dry-run --method=http --host=archive.canonical.com --root=ubuntu \ 
   --dist=lucid,lucid-backports,lucid-proposed,lucid-security,lucid-updates --section=partner \ 
   --arch=amd64,i386 --progress ~/PartnerRepos

However, instead of running to completion, “debmirror” reports an error after it downloads two of the “Release” files:

[0%] Getting: dists/lucid/Release... ok 
[0%] Getting: dists/lucid/Release.gpg... ok 
[0%] Getting: dists/lucid-backports/Release... ok 
[0%] Getting: dists/lucid-backports/Release.gpg... ok 
Duplicate dist lucid.

Apparently, the new “debmirror” version assumes that a repository will consist of only one dist by default. Thus, it will have to handle the “Ubuntu” repository—which includes multiple dists—in a special way. However, in doing so, it overlooks the Canonical Partner repository—which also has multiple dists, but gets treated as a “default,” single-dist repository instead.

Technical Note:

From looking at the “debmirror” code, I’m puzzled as to why it would want to restrict repositories to a single dist by default. In fact, what the “debmirror” command line calls “dists” should translate to “suites” internally. Conversely, what the code internally calls a “dist” corresponds to the code name of the distribution—e.g., “lucid,” “maverick,” “natty,” etc.
It is my impression that the code is confused about its own terminology—which shouldn’t really come as a surprise, because of the two different meanings of the term “dist.” However, I’m not sufficiently familiar with the internal details of the code to understand how to clear up the confusion.

The Fix.

Since “debmirror” handles the “Ubuntu” repository in a special way, it must contain code to test the repository name, to allow it to process the “Ubuntu” repository differently from any other (supposedly single-dist) repository.

To ensure that it will allow the “Canonical” repository to contain multiple dists, I developed a fairly simple patch that extends the test for the repository name, to cover not only the “Ubuntu” repository, but the “Canonical” repository as well. The patch can be applied as described next.

Applying the patch to the “debmirror” script.

I don’t particularly like applying a patch to a system file in-place; instead, I prefer to copy the affected file to, e.g., my home directory, and apply the patch to the copy. Therefore, the following procedure will apply my patch to a copy of the “debmirror” script in the home directory of the user that is currently logged in to the system:

• Execute the following commands to copy the “debmirror” script to your home directory:
  

  cd
  cp /usr/bin/debmirror .

• Download the “debmirror_origin_patch.zip” file attached to this post. Save it into your home directory (next to your “debmirror” copy), and run the following command to unzip it and create the “debmirror_origin_patch” file:
  

  unzip debmirror_origin_patch.zip

• Execute the following command to apply the patch (from the “debmirror_origin_patch” file) to your copy of the “debmirror” script:
  

  patch < debmirror_origin_patch

Assuming that the “patch” command reports no errors, your “debmirror” copy will now include my patch.

Trying out the patched “debmirror” script.

To test the patched “debmirror” script, run it from your home directory (i.e., prepend “~/” to the command name):

~/debmirror --dry-run --method=http --host=archive.canonical.com --root=ubuntu \
   --dist=lucid,lucid-backports,lucid-proposed,lucid-security,lucid-updates --section=partner \
   --arch=amd64,i386 --progress ~/PartnerRepos

This time, “debmirror” should not complain about “duplicate dist” values, but it should complete successfully instead.

See also: Bug Report.

I created a bug report about this issue—cfr. Bug #674153 in debmirror (Ubuntu): “debmirror cannot download Canonical repository, but reports "Duplicate dist" error instead”

[klein de usa]

hi
luvr you are right cron is the way to do it, starting it isn't the problem stopping it is,

starting debmirror:

create a bash script with only the big long debmirror command in it, then set up cron to start it at a set time.

stopping debmirror:

you have to creat a bash script that calls killprocess debmirror witch calls a bash script that find the pid # of debmirror and then kills it.

if anyone should need a complete detailed explanation , i will try my best to explain it. i put this on a headless michine , i can ssh and vnc into it , also have the michine set up to send cli mail with a cron job.

my hope it to get this machine to a better Internet connection lol

[luvr]

if anyone should need a complete detailed explanation , i will try my best to explain it.

I'm sure we would all appreciate it very much if you could document what you did in order to get it to work. Anything that may help others get the most out of their local repository mirrors, is definitely worth documenting here!

[Souravampire]

Thank you thank you and thank you for this beautiful post.
Cheers.Long live Bobsong and luvr.