Howto: defrag /var/lib/dpkg/info to speed up dpkg

[gmargo]

As promised, a safer version of the script to reorder the /var/lib/dpkg/info directory. If you experience problems with the new directory, just restore the saved backup directory.

#!/bin/sh

# Script to reorder the /var/lib/dpkg/info directory to speed up "dpkg -S".
#
# Original by Peter Cordes, from Ubuntu Forums thread
# http://ubuntuforums.org/showthread.php?p=8982470
#
# "Safer" modified version by gmargo 2009-03-17

# Original Code:
# cd
# strace -efile -o dpkg.tr dpkg -S /bin/ls
# cd /var/lib/dpkg
# mkdir info.new
# grep '^open' ~/dpkg.tr | sed -r '/dpkg\/info/sX.*"(.*)".*X\1Xp' -n | xargs sudo cp -a -t info.new 
# # cmd line length limits prevent info/*.  I could have used rsync -au info/ info.new
# sudo cp -iau info/[a-k]* info.new/
# sudo cp -iau info/[l]* info.new/
# sudo cp -iau info/[m-z]* info.new/
# diff -ur info info.new/
# sudo rm -rf info
# sudo mv info.new info
# 
# sync
# echo 3 | sudo tee /proc/sys/vm/drop_caches
# time dpkg -S /bin/ls


# Modified code:

ORIGINAL=/var/lib/dpkg/info
NEW=/var/lib/dpkg/info.new
BACKUP=/var/lib/dpkg/info.saved.$(date "+%Y%m%d_%H%M%S")

#------------------------------
# You must be root or use sudo.
#------------------------------
if [ `id -u` -ne 0 ] ; then
    echo "ERROR: You must be root for this to work!"
    exit 1
fi

if [ -e "$NEW" ]; then
    echo "Remove $NEW directory first."
    exit 1
fi
mkdir "$NEW"
rc=$? ; if [ $rc -ne 0 ] ; then echo "ERROR: mkdir $NEW failed rc=$rc" ; exit 2 ; fi
chmod 755 "$NEW"

echo "Optimize:"
strace -efile -o /tmp/dpkg.tr.$$ dpkg -S /bin/ls >/dev/null
grep '^open' /tmp/dpkg.tr.$$ | sed -r '/dpkg\/info/sX.*"(.*)".*X\1Xp' -n | xargs cp -p -t "$NEW"
find /var/lib/dpkg/info -type f -print | xargs cp -pu -t "$NEW"
rm -f /tmp/dpkg.tr.$$

#--------------------------------------------------
# Rename old info directory.
# DO NOT DELETE UNTIL YOU ARE CONVINCED dpkg WORKS.
#--------------------------------------------------
echo "Create backup:"
if [ -e "$BACKUP" ]; then
    echo "Backup directory $BACKUP already exists."
    exit 1
fi
mv "$ORIGINAL" "$BACKUP"
rc=$? ; if [ $rc -ne 0 ] ; then echo "ERROR: mv $ORIGINAL $BACKUP failed rc=$rc" ; exit 2 ; fi

mv "$NEW" "$ORIGINAL"
rc=$? ; if [ $rc -ne 0 ] ; then echo "ERROR: mv $NEW $ORIGINAL failed rc=$rc" ; exit 2 ; fi

echo "Test:"
sync
echo 3 > /proc/sys/vm/drop_caches
time dpkg -S /bin/ls

[mgol]

Thx, but...

• ...are the packages, whose deb i have to download these in the error msg of the dpkg?
• ...how to do it a lil more automatic? Downloading AND installing 3271 files would drive me mad.

That's why I keep a list of ALL packages I installed manually and when I experience a problem like above I can just reinstall my system and do a huge 'sudo apt-get install' - a few hours and my system's back. Home partition is intact so I even have my old settings unchanged.

It's the easiest way to keep things in order. (however, at least relatively fast internet connection is advisable)

[cthlhu1987]

[...]Me too, I would write some scripts to do it.

Could you give me some hints, how to do that (I know a lil about bash, but some hints would be very helpful)? Thx in advance

The alternative is to reinstall from scratch.

Between a rock and a hard place ;( You said, i have to rename the control-files,once i extracted them from the *.deb files (7z'd do, i reckon). Which names I'd have to give them?

Time to test the Lucid Beta due out tomorrow?

I dunno? Should I (stable enough so that my data won't be blown to Shajtanych's (Russian colloquial name for "Shaitan") little vacation location)?

[gmargo]

Could you give me some hints, how to do that (I know a lil about bash, but some hints would be very helpful)? Thx in advance

Do you really want to go this way? I can help, but it will take time and effort.

We just duplicate a basic procedure for each installed package.

• Identify package version (from /var/lib/dpkg/status file).
• Find .deb file for that version (check repositories: karmic-security, karmic-updates, karmic)
• Download .deb file.
• Extract control files from .deb file (dpkg-deb -e pkgname.deb)
• Rename control files (* => pkgname.*)

Simple, right? Just do that 3300 times. No problem.

[cthlhu1987]

Do you really want to go this way? I can help, but it will take time and effort.

We just duplicate a basic procedure for each installed package.

• Identify package version (from /var/lib/dpkg/status file).
• Find .deb file for that version (check repositories: karmic-security, karmic-updates, karmic)
• Download .deb file.
• Extract control files from .deb file (dpkg-deb -e pkgname.deb)
• Rename control files (* => pkgname.*)

Simple, right? Just do that 3300 times. No problem.

How to...

1. ...auto-check repos?
2. ...get the package with the right version?
3. ...get the url for a package?

Downloading, extracting and renaming souldn't be sooooooo hard and since a pc can't get bored, i'll leave this predestined-for-machines-task (like dish-wahsing or laundering) to the pc overnight.

[tgalati4]

The improved script worked fine in Jaunty. Cut my synaptic wait time from ~30 seconds to 4 seconds.

Nice contribution by gmargo!!

Other than a little formatting for the output, can you explain some of the outputs?

Optimize:
Create backup:
Test:
coreutils: /bin/ls
0.42user 0.31system 0:02.39elapsed 30%CPU (0avgtext+0avgdata 0maxresident)k
44408inputs+0outputs (8major+8977minor)pagefaults 0swaps

Would it be helpful to perform a dpkg search before optimization so we can compare the real time performance?

[gmargo]

How to...

1. ...auto-check repos?
2. ...get the package with the right version?
3. ...get the url for a package?

Downloading, extracting and renaming souldn't be sooooooo hard and since a pc can't get bored, i'll leave this predestined-for-machines-task (like dish-wahsing or laundering) to the pc overnight.

I'm working on a script... hopefully will post a first version sometime today....

[ATJ]

Works like a charm

Thanks a lot

[gmargo]

Here's an initial version of a script to recover the missing /var/lib/dpkg/info/* files. No system files are actually modified. You are left with a directory of files to move into the info directory. It's about 400 lines of perl. See attachment.

[tgalati4]

I modified the script slighty to provide a before and after comparison:

#!/bin/sh

# Script to reorder the /var/lib/dpkg/info directory to speed up "dpkg -S".
#
# Original by Peter Cordes, from Ubuntu Forums thread
# http://ubuntuforums.org/showthread.php?p=8982470
#
# "Safer" modified version by gmargo 2009-03-17

# Original Code:
# cd
# strace -efile -o dpkg.tr dpkg -S /bin/ls
# cd /var/lib/dpkg
# mkdir info.new
# grep '^open' ~/dpkg.tr | sed -r '/dpkg\/info/sX.*"(.*)".*X\1Xp' -n | xargs sudo cp -a -t info.new 
# # cmd line length limits prevent info/*.  I could have used rsync -au info/ info.new
# sudo cp -iau info/[a-k]* info.new/
# sudo cp -iau info/[l]* info.new/
# sudo cp -iau info/[m-z]* info.new/
# diff -ur info info.new/
# sudo rm -rf info
# sudo mv info.new info
# 
# sync
# echo 3 | sudo tee /proc/sys/vm/drop_caches
# time dpkg -S /bin/ls


# Modified code:

ORIGINAL=/var/lib/dpkg/info
NEW=/var/lib/dpkg/info.new
BACKUP=/var/lib/dpkg/info.saved.$(date "+%Y%m%d_%H%M%S")

#------------------------------
# You must be root or use sudo.
#------------------------------
if [ `id -u` -ne 0 ] ; then
    echo "ERROR: You must be root for this to work!"
    exit 1
fi

if [ -e "$NEW" ]; then
    echo "Remove $NEW directory first."
    exit 1
fi

echo "Time to perform search for package that provides ls, before optimization:"
sync
echo 3 > /proc/sys/vm/drop_caches
time dpkg -S /bin/ls

mkdir "$NEW"
rc=$? ; if [ $rc -ne 0 ] ; then echo "ERROR: mkdir $NEW failed rc=$rc" ; exit 2 ; fi
chmod 755 "$NEW"

echo "Optimize:"
strace -efile -o /tmp/dpkg.tr.$$ dpkg -S /bin/ls >/dev/null
grep '^open' /tmp/dpkg.tr.$$ | sed -r '/dpkg\/info/sX.*"(.*)".*X\1Xp' -n | xargs cp -p -t "$NEW"
find /var/lib/dpkg/info -type f -print | xargs cp -pu -t "$NEW"
rm -f /tmp/dpkg.tr.$$

#--------------------------------------------------
# Rename old info directory.
# DO NOT DELETE UNTIL YOU ARE CONVINCED dpkg WORKS.
#--------------------------------------------------
echo "Create backup:"
if [ -e "$BACKUP" ]; then
    echo "Backup directory $BACKUP already exists."
    exit 1
fi
mv "$ORIGINAL" "$BACKUP"
rc=$? ; if [ $rc -ne 0 ] ; then echo "ERROR: mv $ORIGINAL $BACKUP failed rc=$rc" ; exit 2 ; fi

mv "$NEW" "$ORIGINAL"
rc=$? ; if [ $rc -ne 0 ] ; then echo "ERROR: mv $NEW $ORIGINAL failed rc=$rc" ; exit 2 ; fi

echo "Dpkg search time after optimization:"
sync
echo 3 > /proc/sys/vm/drop_caches
time dpkg -S /bin/ls

Time to perform search for package that provides ls, before optimization:
coreutils: /bin/ls
0.21user 0.27system 0:08.86elapsed 5%CPU (0avgtext+0avgdata 0maxresident)k
35064inputs+0outputs (7major+12188minor)pagefaults 0swaps
Optimize:
Create backup:
Dpkg search time after optimization:
coreutils: /bin/ls
0.17user 0.23system 0:01.01elapsed 40%CPU (0avgtext+0avgdata 0maxresident)k
33000inputs+0outputs (7major+12187minor)pagefaults 0swaps

From 8 seconds to 1 second on a dual core, 3 GHz machine. You can see that CPU goes up in the second case, as there is less waiting for I/O.