Re: Howto: defrag /var/lib/dpkg/info to speed up dpkg
As promised, a safer version of the script to reorder the /var/lib/dpkg/info directory. If you experience problems with the new directory, just restore the saved backup directory.
Code:
#!/bin/sh
# Script to reorder the /var/lib/dpkg/info directory to speed up "dpkg -S".
#
# Original by Peter Cordes, from Ubuntu Forums thread
# http://ubuntuforums.org/showthread.php?p=8982470
#
# "Safer" modified version by gmargo 2009-03-17
# Original Code:
# cd
# strace -efile -o dpkg.tr dpkg -S /bin/ls
# cd /var/lib/dpkg
# mkdir info.new
# grep '^open' ~/dpkg.tr | sed -r '/dpkg\/info/sX.*"(.*)".*X\1Xp' -n | xargs sudo cp -a -t info.new
# # cmd line length limits prevent info/*. I could have used rsync -au info/ info.new
# sudo cp -iau info/[a-k]* info.new/
# sudo cp -iau info/[l]* info.new/
# sudo cp -iau info/[m-z]* info.new/
# diff -ur info info.new/
# sudo rm -rf info
# sudo mv info.new info
#
# sync
# echo 3 | sudo tee /proc/sys/vm/drop_caches
# time dpkg -S /bin/ls
# Modified code:
ORIGINAL=/var/lib/dpkg/info
NEW=/var/lib/dpkg/info.new
BACKUP=/var/lib/dpkg/info.saved.$(date "+%Y%m%d_%H%M%S")
#------------------------------
# You must be root or use sudo.
#------------------------------
if [ `id -u` -ne 0 ] ; then
echo "ERROR: You must be root for this to work!"
exit 1
fi
if [ -e "$NEW" ]; then
echo "Remove $NEW directory first."
exit 1
fi
mkdir "$NEW"
rc=$? ; if [ $rc -ne 0 ] ; then echo "ERROR: mkdir $NEW failed rc=$rc" ; exit 2 ; fi
chmod 755 "$NEW"
echo "Optimize:"
strace -efile -o /tmp/dpkg.tr.$$ dpkg -S /bin/ls >/dev/null
grep '^open' /tmp/dpkg.tr.$$ | sed -r '/dpkg\/info/sX.*"(.*)".*X\1Xp' -n | xargs cp -p -t "$NEW"
find /var/lib/dpkg/info -type f -print | xargs cp -pu -t "$NEW"
rm -f /tmp/dpkg.tr.$$
#--------------------------------------------------
# Rename old info directory.
# DO NOT DELETE UNTIL YOU ARE CONVINCED dpkg WORKS.
#--------------------------------------------------
echo "Create backup:"
if [ -e "$BACKUP" ]; then
echo "Backup directory $BACKUP already exists."
exit 1
fi
mv "$ORIGINAL" "$BACKUP"
rc=$? ; if [ $rc -ne 0 ] ; then echo "ERROR: mv $ORIGINAL $BACKUP failed rc=$rc" ; exit 2 ; fi
mv "$NEW" "$ORIGINAL"
rc=$? ; if [ $rc -ne 0 ] ; then echo "ERROR: mv $NEW $ORIGINAL failed rc=$rc" ; exit 2 ; fi
echo "Test:"
sync
echo 3 > /proc/sys/vm/drop_caches
time dpkg -S /bin/ls
Re: Howto: defrag /var/lib/dpkg/info to speed up dpkg
Quote:
Originally Posted by
cthlhu1987
Thx, but...
- ...are the packages, whose deb i have to download these in the error msg of the dpkg?
- ...how to do it a lil more automatic? Downloading AND installing 3271 files would drive me mad.
That's why I keep a list of ALL packages I installed manually and when I experience a problem like above I can just reinstall my system and do a huge 'sudo apt-get install' - a few hours and my system's back. Home partition is intact so I even have my old settings unchanged.
It's the easiest way to keep things in order. (however, at least relatively fast internet connection is advisable)
Re: Howto: defrag /var/lib/dpkg/info to speed up dpkg
Quote:
Originally Posted by
gmargo
[...]Me too, I would write some scripts to do it.
Could you give me some hints, how to do that (I know a lil about bash, but some hints would be very helpful)? Thx in advance ;)
Quote:
The alternative is to reinstall from scratch.
Between a rock and a hard place :( ;( You said, i have to rename the control-files,once i extracted them from the *.deb files (7z'd do, i reckon). Which names I'd have to give them?
Quote:
Time to test the Lucid Beta due out tomorrow?
I dunno? Should I (stable enough so that my data won't be blown to Shajtanych's (Russian colloquial name for "Shaitan") little vacation location)?
Re: Howto: defrag /var/lib/dpkg/info to speed up dpkg
Quote:
Originally Posted by
cthlhu1987
Could you give me some hints, how to do that (I know a lil about bash, but some hints would be very helpful)? Thx in advance ;)
Do you really want to go this way? I can help, but it will take time and effort.
We just duplicate a basic procedure for each installed package.
- Identify package version (from /var/lib/dpkg/status file).
- Find .deb file for that version (check repositories: karmic-security, karmic-updates, karmic)
- Download .deb file.
- Extract control files from .deb file (dpkg-deb -e pkgname.deb)
- Rename control files (* => pkgname.*)
Simple, right? Just do that 3300 times. No problem. :D
Re: Howto: defrag /var/lib/dpkg/info to speed up dpkg
Quote:
Originally Posted by
gmargo
Do you
really want to go this way? I can help, but it will take time and effort.
We just duplicate a basic procedure for each installed package.
- Identify package version (from /var/lib/dpkg/status file).
- Find .deb file for that version (check repositories: karmic-security, karmic-updates, karmic)
- Download .deb file.
- Extract control files from .deb file (dpkg-deb -e pkgname.deb)
- Rename control files (* => pkgname.*)
Simple, right? Just do that 3300 times. No problem. :D
How to...
- ...auto-check repos?
- ...get the package with the right version?
- ...get the url for a package?
Downloading, extracting and renaming souldn't be sooooooo hard and since a pc can't get bored, i'll leave this predestined-for-machines-task (like dish-wahsing or laundering) to the pc overnight.
Re: Howto: defrag /var/lib/dpkg/info to speed up dpkg
The improved script worked fine in Jaunty. Cut my synaptic wait time from ~30 seconds to 4 seconds.
Nice contribution by gmargo!!
Other than a little formatting for the output, can you explain some of the outputs?
Optimize:
Create backup:
Test:
coreutils: /bin/ls
0.42user 0.31system 0:02.39elapsed 30%CPU (0avgtext+0avgdata 0maxresident)k
44408inputs+0outputs (8major+8977minor)pagefaults 0swaps
Would it be helpful to perform a dpkg search before optimization so we can compare the real time performance?
Re: Howto: defrag /var/lib/dpkg/info to speed up dpkg
Quote:
Originally Posted by
cthlhu1987
How to...
- ...auto-check repos?
- ...get the package with the right version?
- ...get the url for a package?
Downloading, extracting and renaming souldn't be sooooooo hard and since a pc can't get bored, i'll leave this predestined-for-machines-task (like dish-wahsing or laundering) to the pc overnight.
I'm working on a script... hopefully will post a first version sometime today....
Re: Howto: defrag /var/lib/dpkg/info to speed up dpkg
Works like a charm :)
Thanks a lot :D
1 Attachment(s)
Re: Howto: defrag /var/lib/dpkg/info to speed up dpkg
Here's an initial version of a script to recover the missing /var/lib/dpkg/info/* files. No system files are actually modified. You are left with a directory of files to move into the info directory. It's about 400 lines of perl. See attachment.
Re: Howto: defrag /var/lib/dpkg/info to speed up dpkg
I modified the script slighty to provide a before and after comparison:
Code:
#!/bin/sh
# Script to reorder the /var/lib/dpkg/info directory to speed up "dpkg -S".
#
# Original by Peter Cordes, from Ubuntu Forums thread
# http://ubuntuforums.org/showthread.php?p=8982470
#
# "Safer" modified version by gmargo 2009-03-17
# Original Code:
# cd
# strace -efile -o dpkg.tr dpkg -S /bin/ls
# cd /var/lib/dpkg
# mkdir info.new
# grep '^open' ~/dpkg.tr | sed -r '/dpkg\/info/sX.*"(.*)".*X\1Xp' -n | xargs sudo cp -a -t info.new
# # cmd line length limits prevent info/*. I could have used rsync -au info/ info.new
# sudo cp -iau info/[a-k]* info.new/
# sudo cp -iau info/[l]* info.new/
# sudo cp -iau info/[m-z]* info.new/
# diff -ur info info.new/
# sudo rm -rf info
# sudo mv info.new info
#
# sync
# echo 3 | sudo tee /proc/sys/vm/drop_caches
# time dpkg -S /bin/ls
# Modified code:
ORIGINAL=/var/lib/dpkg/info
NEW=/var/lib/dpkg/info.new
BACKUP=/var/lib/dpkg/info.saved.$(date "+%Y%m%d_%H%M%S")
#------------------------------
# You must be root or use sudo.
#------------------------------
if [ `id -u` -ne 0 ] ; then
echo "ERROR: You must be root for this to work!"
exit 1
fi
if [ -e "$NEW" ]; then
echo "Remove $NEW directory first."
exit 1
fi
echo "Time to perform search for package that provides ls, before optimization:"
sync
echo 3 > /proc/sys/vm/drop_caches
time dpkg -S /bin/ls
mkdir "$NEW"
rc=$? ; if [ $rc -ne 0 ] ; then echo "ERROR: mkdir $NEW failed rc=$rc" ; exit 2 ; fi
chmod 755 "$NEW"
echo "Optimize:"
strace -efile -o /tmp/dpkg.tr.$$ dpkg -S /bin/ls >/dev/null
grep '^open' /tmp/dpkg.tr.$$ | sed -r '/dpkg\/info/sX.*"(.*)".*X\1Xp' -n | xargs cp -p -t "$NEW"
find /var/lib/dpkg/info -type f -print | xargs cp -pu -t "$NEW"
rm -f /tmp/dpkg.tr.$$
#--------------------------------------------------
# Rename old info directory.
# DO NOT DELETE UNTIL YOU ARE CONVINCED dpkg WORKS.
#--------------------------------------------------
echo "Create backup:"
if [ -e "$BACKUP" ]; then
echo "Backup directory $BACKUP already exists."
exit 1
fi
mv "$ORIGINAL" "$BACKUP"
rc=$? ; if [ $rc -ne 0 ] ; then echo "ERROR: mv $ORIGINAL $BACKUP failed rc=$rc" ; exit 2 ; fi
mv "$NEW" "$ORIGINAL"
rc=$? ; if [ $rc -ne 0 ] ; then echo "ERROR: mv $NEW $ORIGINAL failed rc=$rc" ; exit 2 ; fi
echo "Dpkg search time after optimization:"
sync
echo 3 > /proc/sys/vm/drop_caches
time dpkg -S /bin/ls
Time to perform search for package that provides ls, before optimization:
coreutils: /bin/ls
0.21user 0.27system 0:08.86elapsed 5%CPU (0avgtext+0avgdata 0maxresident)k
35064inputs+0outputs (7major+12188minor)pagefaults 0swaps
Optimize:
Create backup:
Dpkg search time after optimization:
coreutils: /bin/ls
0.17user 0.23system 0:01.01elapsed 40%CPU (0avgtext+0avgdata 0maxresident)k
33000inputs+0outputs (7major+12187minor)pagefaults 0swaps
From 8 seconds to 1 second on a dual core, 3 GHz machine. You can see that CPU goes up in the second case, as there is less waiting for I/O.