HOWTO: Backup all installed programs/packages

[abhiroopb]

PROBLEM: Lets say you have set up your *buntu box. Now you want to backup all the installed deb files so that you can restore them quickly and efficiently.

UPDATE: Why would you want this?
When I was using windows I had a directory of EVERY Single program that I had downloaded. For one thing it would be easier to install everything as I would not have to go online and hunt for them. Further it would be useful where a computer does not have internet connection. In ubuntu "hunting" for programs is a rare occurrence thanks to the fantastic package managing system. However, I personally have about 20-30 programs that I have either compiled from source (using checkinstall, so that that a deb package is created and they are added to APT), or downloaded debs from obscure locations. Now each of these debs I will save in a directory so that in the future I do not have to go hunting for them. However, this command I have outlined backs up ALL packages, including the ones in the package manager. So, why would you want that?

Firstly, this is VERY useful if lets say you have setup a very basic installation with all updates, and all non-free video/audio/etc codecs. Further you have installed some basic useful software. Now lets say you want to install the SAME setup on your grandmothers computer, except she does not have internet connection, or at the time you go to set it up she does not have a net connection. Using this script you can have all your debs in one simple location, so you will not have to redownload everything.

Secondly, lets assume you work for a school, or a company, and you need to install the SAME ubuntu installation on 30 computers. Wouldn't it be easier to simply put all these debs in a central server and issue the dpkg -i *.deb command. This way you don't have to individually select the packages AND the packages don't have to download.

Thirdly, (and this is purely personal) I like to be able to have all my installed packages at hand. This command doesn't take much effort, and for me it only requires 1.4 gb of space, so for a bit of piece of mind I can easily have all my packages on hand.

There is no real reason to do this if you are already doing a full system backup (e.g. an image of your Ubuntu partition using partimage). This is just something I discovered and feel could be beneficial to other users.

These commands will do that for you.

Open a terminal and paste the following into it:

$ sudo apt-get install dpkg-repack fakeroot
$ mkdir ~/dpkg-repack; cd ~/dpkg-repack
$ fakeroot -u dpkg-repack `dpkg --get-selections | grep install | cut -f1`

(the last command will take some time)

Now if you scroll to your home folder, you should find a folder called "dpkg-repack" which should have all the deb files of all your installed packages.


RE-INSTALL
If you want to re-install the packages, navigate to the folder with the packages and input the following command in the terminal:
sudo dpkg -i *.deb

Thanks to https://answers.launchpad.net/aptoncd/+question/15592/+index (Rafael)

NB: I know AptonCD does this but what prompted me to use this method was that I had done the sudo apt-get clean command which had erased all the files in the /var/cache/apt directory, rendering AptonCD useless (as all it does is take the files from there and put it in the list). So I find this method is more efficient, and easier for me to control!

[Het Irv]

This is pretty cool, This will save some time when restoring my computer if I have screwed up the file system.

If you don't mind I am going to link this in the forums of the Backup Program that I use, QuickStart. http://quickstart.phpbb.net/index.php

Question: Do you have to save this in your home folder or can you change the path to an external drive?

[abhiroopb]

you should be able to do it anywhere you like. Haven't tested that though.

Just change: mkdir ~/dpkg-repack; cd ~/dpkg-repack

to wherever your external directory is (e.g. mkdir /media/external/dpkg-repack; cd /media/external/dpkg-repack)

I don't really want to create a username but would it be ok if I were to suggest requests for QuickStart? It seems like a great program!

[Samhain13]

How do I restore these back-ups (from say, /media/Storage/dpkg-repack) after I've reinstalled?

[abhiroopb]

Thanks I added it.

[mdpalow]

$ sudo apt-get install dpkg-repack fakeroot
$ mkdir ~/dpkg-repack; cd ~/dpkg-repack
$ fakeroot -u dpkg-repack `dpkg --get-selections | grep install | cut -f1`

dpkg -i *.deb



This is pretty cool.

If you don't mind I am going to link this in the forums of the Backup Program that I use, QuickStart. http://quickstart.phpbb.net/index.php



Hey Irv,

This is 'pretty cool' as you say.

Thanks for posting the link to here in the QuickStart forum. I'm always interested in back-up methods.

Do I understand this correctly in that this will create the .deb files again from the apps that are installed? Even if they were deleted using apt-get clean?

If this is something you want Irv, I don't see why we can't do it. If 'abhiroopb' would like to come over to our forum, we could talk with him some more and get his input for the addition.

Drop me a note in the QS forum Irv and let me know.

Thanks... mdpalow

[abhiroopb]

Yes this basically scans ALL installed deb packages (including those NOT installed through synaptic/apt) and downloads them to a local folder. Also I really liked the QuickStart script, although there are numerous additions/changes I would like to suggest. This is a bad time for me to work on anything though, if you would be willing to wait a couple of weeks I'd be glad to add my input.

[Promaster91]

This will only backup programs not personal data, right?

[abhiroopb]

This will only backup programs not personal data, right?

A debian file is much like an exe file in windows or a dmg file in OSX. So it is basically an installer. What this does is looks through ALL the installed programs in you're computer and downloads them to a local directory.

So to answer your question it does not touch any data of any kind. It simply downloads your installed programs.

[Bruce M.]

What is all this?:
dpkg-deb: ignoring 1 warnings about the control file(s)

warning, './dpkg-repack-20691/DEBIAN/control' contains user-defined field 'Original-Maintainer'


It's doing it's thing now.

Have a good day.
Bruce

EDIT: Once it's done can I just copy the directory to a DVD?

[abhiroopb]

It'll give you a massive directory of .deb files. Its completely upto you what you want to do with them. I have them backed up in an external hard drive. You could burn them to DVD if you want.

[Bruce M.]

It'll give you a massive directory of .deb files. Its completely upto you what you want to do with them. I have them backed up in an external hard drive. You could burn them to DVD if you want.

Well, my /var/cache/apt tells me 359+MB so if it's the same it could go to my second HD or even a CD.

Could I use this to "install" on another machine (ie: not mine)?

[abhiroopb]

the /var/cache/apt directory is debs from a while back (which haven't been erased and are still lying around even after uninstall) and some new installed packages won't be listed there.

Installing on a clean machine is ok. But I have NO idea what will happen if you try it on a machine that already has some programs installed. Apt is quite smart so it should be ok, be careful though.

[Het Irv]

To reinstall everything couldn't you 'cd' to the directory that ther are in and use sudo dpkg -i *.deb

If remembered the command correctly, that will install all deb's in the current folder, it will also handle any depencencies (install packages that need to be installed first, first).

[abhiroopb]

I have added that already (but I forgot to add the sudo)

[Het Irv]

I don't really want to create a username but would it be ok if I were to suggest requests for QuickStart? It seems like a great program!

I guess so, but It would be easier for you to join. If you are busy thats fine, We used to have a thread here in the Ubuntu Forums, but there was a little dust up, and the program was getting to big for one thread anyway. It's not the most active of forums but mdplaow is usually on two or three times a day.

[starcannon]

Have you tried apt on cd?

sudo apt-get install aptoncd

[abhiroopb]

I have already mentioned EXACTLY why aptoncd is not appropriate. Thanks for mentioning anyway.

[Sef]

Moved to Tips and Tricks.

[PinkFloyd102489]

Wouldnt it just be easier to do this?


To make the list:
sudo dpkg --get-selections > installed-software

To restore packages:
sudo dpkg --set-selections < installed-software


This gives you the list of packages without all of the deb files. Mind that you'll need an internet connection, but most computers do.



EDIT: installed-software is a file btw. It just acts as a list that you pipe back into the dpkg command so it will know what packages to download.

[abhiroopb]

yes i usually use that if as it takes a lot less space. The point of this is say for example you have to install the packages on multiple computers, using this command since they're downloaded you only have to run them.

[PinkFloyd102489]

yes i usually use that if as it takes a lot less space. The point of this is say for example you have to install the packages on multiple computers, using this command since they're downloaded you only have to run them.

Understandable. I just like my way better because I have a fast connection. :-)

[abhiroopb]

Thats the beauty of linux, a million ways to do the same thing :)!

[Bruce M.]

Wouldnt it just be easier to do this?


To make the list:
sudo dpkg --get-selections > installed-software

To restore packages:
sudo dpkg --set-selections < installed-software


Thanks for this. :)
I've made the list, nice.

yes i usually use that if as it takes a lot less space. The point of this is say for example you have to install the packages on multiple computers, using this command since they're downloaded you only have to run them.

This is, of course, the plus side of things as I have a slow internet connection (128kbs), so it did take a while to get them (633.2 MB)

Understandable. I just like my way better because I have a fast connection. :-)

Thats the beauty of linux, a million ways to do the same thing :)!

And combined, I now have a list I can use to look up a file, and the files on a CD.

Call that: 1,000,001 ways to do the same thing.

Thanks to both of you.
Have a nice day.
Bruce

[Gourgi]

Wouldnt it just be easier to do this?


To make the list:
sudo dpkg --get-selections > installed-software

To restore packages:
sudo dpkg --set-selections < installed-software


This gives you the list of packages without all of the deb files. Mind that you'll need an internet connection, but most computers do.



EDIT: installed-software is a file btw. It just acts as a list that you pipe back into the dpkg command so it will know what packages to download.
i liked your method and i gave it a try, but ..

i exported the list of files, i edited it and added 1 application that i don't have installed
2vcard install


and after that sudo dpkg --set-selections < installed-software it is supposed to have 2vcard installed :confused:
it does that automatically?? i tried sudo apt-get update && sudo apt-get upgrade but nothing happened
2vcard yet not installed

[abhiroopb]

You can't edit the file and add something like that. Just recreate the file, it takes a couple of seconds.

[abhiroopb]

IMPORTANT: I just discovered that using this command:

Code:

dpkg --get-selections > installed-software

And if you wanted to use the list to reinstall this software on a fresh ubuntu setup,

Code:

dpkg --set-selections < installed-software

followed by

Code:

dselect

DOES NOT install packages not in repos. So if you have used this to backup packages it will be no good installing say a .deb package you downloaded from getdeb. Keep this in mind. Mine however, should work fine.

[unutbu]

Thank you abhiroopb for the very informative tutorial!

I have one reservation about the method however, that I hope someone here will put to rest:

If you run

sudo dpkg -i *.deb

will it really install the debs in the correct order? I thought you would need a tool like apt-get to straighten out the dependency problems.




EDIT: Please ignore the rest of this message. I've tried it -- and it does not work. Setting up apt-get and sources.list properly is more complicated than what I showed below.




If that is true perhaps what we could do is

edit /etc/apt/sources.list by adding something like

deb file://home/user/dpkg-repack gutsy universe multiverse restricted main security gutsy-backports

(changing gutsy to hardy if that's the way you roll) then generate the list of packages with

% dpkg --get-selections | grep install | cut -f1 > installed-pkgs

(if you didn't have the list already) and install them with
% sudo apt-get install `cat installed-pkgs`

[abhiroopb]

I haven't tested it exactly, but from what I've heard dpkg is able to do it automatically installing first the packages that need to be installed. What exactly is the difference? I'm sorry I'm a bit confused with what your solution does?

[unutbu]

Here is a little experiment I ran to test if dpkg is smart enough to resolve dependencies on its own.

I've downloaded
warsow_0.42-1~getdeb1_i386.deb
warsow-data_0.42-1~getdeb1_all.deb

from http://www.getdeb.net/app.php?name=Warsow.
(This is a noble sacrifice in the name of science, mind you. I have no interest in warsow whatsoever except insofar as it shall help us explore the capabilities of dpkg. :rolleyes:)

% dpkg --info warsow_0.42-1~getdeb1_i386.deb
<snip>
Depends: libc6 (>= 2.6-1), libcurl3-gnutls (>= 7.16.2-1), libjpeg62, libkrb53 (>= 1.6.dfsg.1), libsdl1.2debian (>= 1.2.10-1),
libvorbisfile3 (>= 1.2.0), libx11-6, libxext6, libxinerama1, libxxf86dga1, libxxf86vm1, zlib1g (>= 1:1.2.3.3.dfsg-1), mesa-utils, xbase-clients,
warsow-data (= 0.42-1~getdeb1)
<snip>


Notice warsow-data is one of the packages upon which warsow depends.

So what if I say

sudo dpkg -i warsow_0.42-1~getdeb1_i386.deb?

Will it install warsow-data first, or will it fail, complaining that warsow-data is not installed?

% sudo dpkg -i warsow_0.42-1~getdeb1_i386.deb
Selecting previously deselected package warsow.
(Reading database ... 169934 files and directories currently installed.)
Unpacking warsow (from warsow_0.42-1~getdeb1_i386.deb) ...
dpkg: dependency problems prevent configuration of warsow:
warsow depends on warsow-data (= 0.42-1~getdeb1); however:
Package warsow-data is not installed.
dpkg: error processing warsow (--install):
dependency problems - leaving unconfigured
Errors were encountered while processing:
warsow

This shows that dpkg is not smart enough to resolve dependencies.

I then tried to set up sources.list so that apt-get could work with a local directory filled with debs.
The reason why I tried to use apt-get to install the packages instead of dpkg is because I know apt-get is smart enough to resolve dependencies for me.

I've never done that before and I've just found out its a bit more complicated than I made it out to be in my previous post.

You need the right directory structure and a Package.gz file, a proper sources.list. I don't know how to do it... yet. Maybe a friendly neighborhood apt-wizard will be kind enough to show us the way...

[unutbu]

APT-wizard Capink has provided us with the solution: http://ubuntuforums.org/showpost.php?p=3501424&postcount=3!

In the commands below, change engine to your username, and
change /home/engine/dpkg-repack to the directory where you keep your debs.

export private=/home/engine/dpkg-repack
echo $private /home/engine/dpkg-repack
apt-ftparchive packages ${private} > ${private}/Packages
sed --in-place 's_Filename: '${private}'/_Filename: _' ${private}/Packages
gzip -9c "${private}/Packages" > ${private}/Packages.gz
gksu gedit /etc/apt/sources.list

Add to the bottom:

deb file:///home/engine/dpkg-repack ./

sudo apt-get update

You will now be able to use your dpkg-repack directory just like any other official or unofficial repository. You'll be able to use Synaptic or apt-get, for example.

Instead of

sudo dpkg -i *.deb

you can now

sudo apt-get install `cat installed-pkgs`
to install all .debs from your local repository and apt-get will install them all in an order which handles dependencies correctly and automatically.

[abhiroopb]

Ok I think the mistake you made was this:

sudo dpkg -i warsow_0.42-1~getdeb1_i386.deb


No expert but this is basically telling dpkg to install warsow first. What I suggested was you do:

sudo dpkg -i *.deb


For whatever reason I think this scans the directory. Alphabetically warsow-data is lined up first so a better test may be needed. Now this isn't the best test, perhaps if you point me towards a prgoram that has a lot of smaller dependencies we can run a more thorough test.

Here is the output:

abhiroop@Vanimo:~/MyDownloads/New Folder$ sudo dpkg -i *.deb
Selecting previously deselected package warsow.
(Reading database ... 134783 files and directories currently installed.)
Unpacking warsow (from warsow_0.31.dfsg-6_i386.deb) ...
Selecting previously deselected package warsow-data.
Unpacking warsow-data (from warsow-data_0.31-1_all.deb) ...
Setting up warsow-data (0.31-1) ...
Setting up warsow (0.31.dfsg-6) ...


The key here is that you DON'T point it towards a certain file, but just tell it to do a general one.

Thanks for your tip though seems useful!

[unutbu]

I believe you are right! I just tried the following test:

graphviz depends on libgraphviz3.
So libgraphviz3 has to be installed first, but graphviz comes first alphabetically (unlike warsow and warsow-data, where I'm a bit unsure about the order in which "*.deb" is expanded).

sudo apt-get --download-only install graphviz libgraphviz3

This downloads graphviz_2.12-4ubuntu3_i386.deb and
libgraphviz3_2.12-4ubuntu3_i386.deb into /var/cache/apt/archives without installing them.

I then copied the debs into a test directory:
cp /var/cache/apt/archives/libgraphviz3_2.12-4ubuntu3_i386.deb /var/cache/apt/archives/graphviz_2.12-4ubuntu3_i386.deb .

sudo dpkg -i *.deb

Selecting previously deselected package graphviz.
(Reading database ... 170366 files and directories currently installed.)
Unpacking graphviz (from graphviz_2.12-4ubuntu3_i386.deb) ...
Selecting previously deselected package libgraphviz3.
Unpacking libgraphviz3 (from libgraphviz3_2.12-4ubuntu3_i386.deb) ...
Setting up libgraphviz3 (2.12-4ubuntu3) ...

Setting up graphviz (2.12-4ubuntu3) ...

Processing triggers for libc6 ...
ldconfig deferred processing now taking place

dpkg does seem to be able to sort out the dependency problem as long as all the required deb packages are listed at once (in the same command)!

Thanks for helping me learn this, abhiroopb.

[abhiroopb]

No problem. So it seems that all it takes is putting ALL debs in one directory and running dpkg -i *.deb. But you have to make sure that all dependencies are in the directory.

[blaz_boy]

ok, thats good but how can i repackage a single program with it's dependences ? it's really urgent for me as i sometime want to transfere a program form my labtop to my friend labtop and i have no way ? could any one tell me ?

[unutbu]

On the friend's computer run

sudo apt-get install --simulate LIST-OF-PACKAGES

You don't have to have an internet connection to run this.

Here is an example of the output using the package 'kate':

% sudo apt-get install --simulate kate
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
kdelibs-data kdelibs4c2a libarts1c2a libavahi-qt3-1 liblua50 liblualib50
Suggested packages:
kate-plugins khelpcenter konsole fam
Recommended packages:
kregexpeditor perl-suid libarts1-akode
The following NEW packages will be installed:
kate kdelibs-data kdelibs4c2a libarts1c2a libavahi-qt3-1 liblua50
liblualib50
0 upgraded, 7 newly installed, 0 to remove and 87 not upgraded.

Note this line:

The following NEW packages will be installed:
kate kdelibs-data kdelibs4c2a libarts1c2a libavahi-qt3-1 liblua50
liblualib50


This shows all the packages your friend would need to install kate.

Now go to your machine and type abhiroopb's commands (slightly modified):

sudo apt-get install dpkg-repack fakeroot
mkdir ~/dpkg-repack; cd ~/dpkg-repack
fakeroot -u dpkg-repack kate kdelibs-data kdelibs4c2a libarts1c2a libavahi-qt3-1 liblua50 liblualib50 # Just list all the packages you need

[adept_king]

is this just for .debs or is it for other installed things too?
does it also include things like drivers i installed like flash or restricted old drivers, etc? if not, is there a method to encompass both?

is there a way to make a list of the programs i installed? i dont know exactly where to look if i want a list of stuff like

envy
mysql
php
shorewall
wordpress
etc...

just want the programs i installed. a list of everything i added.

[Nausser]

I've used this tool several times in the past and has always worked flawlessly until Jaunty came along.

Anyone know why this may be?

Here is what the terminal spat back at me:

fakeroot -u dpkg-repack `dpkg --get-selections | grep install | cut
> -f1`
cut: you must specify a list of bytes, characters, or fields
Try `cut --help' for more information.
bash: -f1: command not found
Usage: dpkg-repack [options] packagename [packagename ..]
--root=dir Take package from filesystem rooted on <dir>.
--arch=arch Force the parch to be built for architecture <arch>.
--generate Generate build directory but do not build deb.
packagename The name of the package to attempt to repack.
renken@renken-desktop:~/dpkg-repack$ fakeroot -u dpkg-repack `dpkg --get-selections | grep install
>

I hope someone else has been brave enough to try Jaunty and came up with the same issue.

[abhiroopb]

are you putting cut -f1 on the same line?

it should be:


fakeroot -u dpkg-repack `dpkg --get-selections | grep install | cut -f1`


from what you posted it seems to be doing:

fakeroot -u dpkg-repack `dpkg --get-selections | grep install | cut
> -f1`

[bbowdry]

Can you add remarks to this file such as date and information on how back and/or restore?

[abhiroopb]

Can you add remarks to this file such as date and information on how back and/or restore?

Not sure how this would be possible. But I suppose you could create another list with all the packages (in one txt file) that has the information you require. When I remember how to do this I will let you know!

[hero1900]

thanks it was very helpful i was looking for it

[Cuba71]

I finished repackaging my Jaunty, a total of 1,425 pkgs in 942 MB of storage.
It took about 3 hours and I did get a couple of error messages indicating broken packages. I have to assume that my system was not utilizing these broken packages.

[unutbu]

Cuba71, if you have ever uninstalled a package then when you run

fakeroot -u dpkg-repack `dpkg --get-selections | grep install | cut -f1`

you will get an error because

dpkg --get-selections | grep install | cut -f1

will list "deinstalled" packages as well as "installed" packages.

A way to avoid this problem is to run

fakeroot -u dpkg-repack $(aptitude -F '%p' search '~i')

instead, because unlike the dpkg command

aptitude -F '%p' search '~i'

will list installed packages only.

[Cuba71]

unutbu:
I see where I made the mistake. I think I could've used:


fakeroot –u dpkg-repack `dpkg –get-selections | grep -v deinstall | cut -f1`


to eliminate the "deinstall" packages