Page 2 of 3 FirstFirst 123 LastLast
Results 11 to 20 of 25

Thread: Howto: defrag /var/lib/dpkg/info to speed up dpkg

  1. #11
    Join Date
    Feb 2010
    Location
    Silicon Valley
    Beans
    1,898
    Distro
    Xubuntu 12.04 Precise Pangolin

    Re: Howto: defrag /var/lib/dpkg/info to speed up dpkg

    As promised, a safer version of the script to reorder the /var/lib/dpkg/info directory. If you experience problems with the new directory, just restore the saved backup directory.

    Code:
    #!/bin/sh
    
    # Script to reorder the /var/lib/dpkg/info directory to speed up "dpkg -S".
    #
    # Original by Peter Cordes, from Ubuntu Forums thread
    # http://ubuntuforums.org/showthread.php?p=8982470
    #
    # "Safer" modified version by gmargo 2009-03-17
    
    # Original Code:
    # cd
    # strace -efile -o dpkg.tr dpkg -S /bin/ls
    # cd /var/lib/dpkg
    # mkdir info.new
    # grep '^open' ~/dpkg.tr | sed -r '/dpkg\/info/sX.*"(.*)".*X\1Xp' -n | xargs sudo cp -a -t info.new 
    # # cmd line length limits prevent info/*.  I could have used rsync -au info/ info.new
    # sudo cp -iau info/[a-k]* info.new/
    # sudo cp -iau info/[l]* info.new/
    # sudo cp -iau info/[m-z]* info.new/
    # diff -ur info info.new/
    # sudo rm -rf info
    # sudo mv info.new info
    # 
    # sync
    # echo 3 | sudo tee /proc/sys/vm/drop_caches
    # time dpkg -S /bin/ls
    
    
    # Modified code:
    
    ORIGINAL=/var/lib/dpkg/info
    NEW=/var/lib/dpkg/info.new
    BACKUP=/var/lib/dpkg/info.saved.$(date "+%Y%m%d_%H%M%S")
    
    #------------------------------
    # You must be root or use sudo.
    #------------------------------
    if [ `id -u` -ne 0 ] ; then
        echo "ERROR: You must be root for this to work!"
        exit 1
    fi
    
    if [ -e "$NEW" ]; then
        echo "Remove $NEW directory first."
        exit 1
    fi
    mkdir "$NEW"
    rc=$? ; if [ $rc -ne 0 ] ; then echo "ERROR: mkdir $NEW failed rc=$rc" ; exit 2 ; fi
    chmod 755 "$NEW"
    
    echo "Optimize:"
    strace -efile -o /tmp/dpkg.tr.$$ dpkg -S /bin/ls >/dev/null
    grep '^open' /tmp/dpkg.tr.$$ | sed -r '/dpkg\/info/sX.*"(.*)".*X\1Xp' -n | xargs cp -p -t "$NEW"
    find /var/lib/dpkg/info -type f -print | xargs cp -pu -t "$NEW"
    rm -f /tmp/dpkg.tr.$$
    
    #--------------------------------------------------
    # Rename old info directory.
    # DO NOT DELETE UNTIL YOU ARE CONVINCED dpkg WORKS.
    #--------------------------------------------------
    echo "Create backup:"
    if [ -e "$BACKUP" ]; then
        echo "Backup directory $BACKUP already exists."
        exit 1
    fi
    mv "$ORIGINAL" "$BACKUP"
    rc=$? ; if [ $rc -ne 0 ] ; then echo "ERROR: mv $ORIGINAL $BACKUP failed rc=$rc" ; exit 2 ; fi
    
    mv "$NEW" "$ORIGINAL"
    rc=$? ; if [ $rc -ne 0 ] ; then echo "ERROR: mv $NEW $ORIGINAL failed rc=$rc" ; exit 2 ; fi
    
    echo "Test:"
    sync
    echo 3 > /proc/sys/vm/drop_caches
    time dpkg -S /bin/ls
    Last edited by gmargo; March 17th, 2010 at 07:32 PM.

  2. #12
    Join Date
    Jan 2008
    Location
    Poland/Warsaw
    Beans
    269
    Distro
    Ubuntu 11.04 Natty Narwhal

    Re: Howto: defrag /var/lib/dpkg/info to speed up dpkg

    Quote Originally Posted by cthlhu1987 View Post
    Thx, but...
    • ...are the packages, whose deb i have to download these in the error msg of the dpkg?
    • ...how to do it a lil more automatic? Downloading AND installing 3271 files would drive me mad.
    That's why I keep a list of ALL packages I installed manually and when I experience a problem like above I can just reinstall my system and do a huge 'sudo apt-get install' - a few hours and my system's back. Home partition is intact so I even have my old settings unchanged.

    It's the easiest way to keep things in order. (however, at least relatively fast internet connection is advisable)
    Michał Gołębiowski
    Dell Latitude E6500: P8600 | Intel GMA 4500MHD | 15,4'' 1440x900 LED (matte) | 4GiB DDR2 | 233 GiB HDD 7200 rpm (with Free Fall Sensor).
    Jabber ID: mgol /at/ jabster.pl - let me write in within my profile details, please!

  3. #13
    Join Date
    Nov 2008
    Location
    Old Europe
    Beans
    79
    Distro
    Lubuntu 13.04 Raring Ringtail

    Re: Howto: defrag /var/lib/dpkg/info to speed up dpkg

    Quote Originally Posted by gmargo View Post
    [...]Me too, I would write some scripts to do it.
    Could you give me some hints, how to do that (I know a lil about bash, but some hints would be very helpful)? Thx in advance
    The alternative is to reinstall from scratch.
    Between a rock and a hard place ;( You said, i have to rename the control-files,once i extracted them from the *.deb files (7z'd do, i reckon). Which names I'd have to give them?
    Time to test the Lucid Beta due out tomorrow?
    I dunno? Should I (stable enough so that my data won't be blown to Shajtanych's (Russian colloquial name for "Shaitan") little vacation location)?

  4. #14
    Join Date
    Feb 2010
    Location
    Silicon Valley
    Beans
    1,898
    Distro
    Xubuntu 12.04 Precise Pangolin

    Re: Howto: defrag /var/lib/dpkg/info to speed up dpkg

    Quote Originally Posted by cthlhu1987 View Post
    Could you give me some hints, how to do that (I know a lil about bash, but some hints would be very helpful)? Thx in advance
    Do you really want to go this way? I can help, but it will take time and effort.

    We just duplicate a basic procedure for each installed package.

    • Identify package version (from /var/lib/dpkg/status file).
    • Find .deb file for that version (check repositories: karmic-security, karmic-updates, karmic)
    • Download .deb file.
    • Extract control files from .deb file (dpkg-deb -e pkgname.deb)
    • Rename control files (* => pkgname.*)

    Simple, right? Just do that 3300 times. No problem.

  5. #15
    Join Date
    Nov 2008
    Location
    Old Europe
    Beans
    79
    Distro
    Lubuntu 13.04 Raring Ringtail

    Re: Howto: defrag /var/lib/dpkg/info to speed up dpkg

    Quote Originally Posted by gmargo View Post
    Do you really want to go this way? I can help, but it will take time and effort.

    We just duplicate a basic procedure for each installed package.

    • Identify package version (from /var/lib/dpkg/status file).
    • Find .deb file for that version (check repositories: karmic-security, karmic-updates, karmic)
    • Download .deb file.
    • Extract control files from .deb file (dpkg-deb -e pkgname.deb)
    • Rename control files (* => pkgname.*)

    Simple, right? Just do that 3300 times. No problem.
    How to...
    1. ...auto-check repos?
    2. ...get the package with the right version?
    3. ...get the url for a package?

    Downloading, extracting and renaming souldn't be sooooooo hard and since a pc can't get bored, i'll leave this predestined-for-machines-task (like dish-wahsing or laundering) to the pc overnight.

  6. #16
    Join Date
    Feb 2007
    Location
    West Hills CA
    Beans
    10,048
    Distro
    Ubuntu 14.04 Trusty Tahr

    Re: Howto: defrag /var/lib/dpkg/info to speed up dpkg

    The improved script worked fine in Jaunty. Cut my synaptic wait time from ~30 seconds to 4 seconds.

    Nice contribution by gmargo!!

    Other than a little formatting for the output, can you explain some of the outputs?

    Optimize:
    Create backup:
    Test:
    coreutils: /bin/ls
    0.42user 0.31system 0:02.39elapsed 30%CPU (0avgtext+0avgdata 0maxresident)k
    44408inputs+0outputs (8major+8977minor)pagefaults 0swaps

    Would it be helpful to perform a dpkg search before optimization so we can compare the real time performance?
    Last edited by tgalati4; March 17th, 2010 at 11:58 PM.
    -------------------------------------
    Oooh Shiny: PopularPages

    Unumquodque potest reparantur. Patientia sit virtus.

  7. #17
    Join Date
    Feb 2010
    Location
    Silicon Valley
    Beans
    1,898
    Distro
    Xubuntu 12.04 Precise Pangolin

    Re: Howto: defrag /var/lib/dpkg/info to speed up dpkg

    Quote Originally Posted by cthlhu1987 View Post
    How to...
    1. ...auto-check repos?
    2. ...get the package with the right version?
    3. ...get the url for a package?

    Downloading, extracting and renaming souldn't be sooooooo hard and since a pc can't get bored, i'll leave this predestined-for-machines-task (like dish-wahsing or laundering) to the pc overnight.
    I'm working on a script... hopefully will post a first version sometime today....

  8. #18
    Join Date
    Sep 2006
    Beans
    8

    Re: Howto: defrag /var/lib/dpkg/info to speed up dpkg

    Works like a charm

    Thanks a lot

  9. #19
    Join Date
    Feb 2010
    Location
    Silicon Valley
    Beans
    1,898
    Distro
    Xubuntu 12.04 Precise Pangolin

    Re: Howto: defrag /var/lib/dpkg/info to speed up dpkg

    Here's an initial version of a script to recover the missing /var/lib/dpkg/info/* files. No system files are actually modified. You are left with a directory of files to move into the info directory. It's about 400 lines of perl. See attachment.
    Attached Files Attached Files

  10. #20
    Join Date
    Feb 2007
    Location
    West Hills CA
    Beans
    10,048
    Distro
    Ubuntu 14.04 Trusty Tahr

    Re: Howto: defrag /var/lib/dpkg/info to speed up dpkg

    I modified the script slighty to provide a before and after comparison:

    Code:
    #!/bin/sh
    
    # Script to reorder the /var/lib/dpkg/info directory to speed up "dpkg -S".
    #
    # Original by Peter Cordes, from Ubuntu Forums thread
    # http://ubuntuforums.org/showthread.php?p=8982470
    #
    # "Safer" modified version by gmargo 2009-03-17
    
    # Original Code:
    # cd
    # strace -efile -o dpkg.tr dpkg -S /bin/ls
    # cd /var/lib/dpkg
    # mkdir info.new
    # grep '^open' ~/dpkg.tr | sed -r '/dpkg\/info/sX.*"(.*)".*X\1Xp' -n | xargs sudo cp -a -t info.new 
    # # cmd line length limits prevent info/*.  I could have used rsync -au info/ info.new
    # sudo cp -iau info/[a-k]* info.new/
    # sudo cp -iau info/[l]* info.new/
    # sudo cp -iau info/[m-z]* info.new/
    # diff -ur info info.new/
    # sudo rm -rf info
    # sudo mv info.new info
    # 
    # sync
    # echo 3 | sudo tee /proc/sys/vm/drop_caches
    # time dpkg -S /bin/ls
    
    
    # Modified code:
    
    ORIGINAL=/var/lib/dpkg/info
    NEW=/var/lib/dpkg/info.new
    BACKUP=/var/lib/dpkg/info.saved.$(date "+%Y%m%d_%H%M%S")
    
    #------------------------------
    # You must be root or use sudo.
    #------------------------------
    if [ `id -u` -ne 0 ] ; then
        echo "ERROR: You must be root for this to work!"
        exit 1
    fi
    
    if [ -e "$NEW" ]; then
        echo "Remove $NEW directory first."
        exit 1
    fi
    
    echo "Time to perform search for package that provides ls, before optimization:"
    sync
    echo 3 > /proc/sys/vm/drop_caches
    time dpkg -S /bin/ls
    
    mkdir "$NEW"
    rc=$? ; if [ $rc -ne 0 ] ; then echo "ERROR: mkdir $NEW failed rc=$rc" ; exit 2 ; fi
    chmod 755 "$NEW"
    
    echo "Optimize:"
    strace -efile -o /tmp/dpkg.tr.$$ dpkg -S /bin/ls >/dev/null
    grep '^open' /tmp/dpkg.tr.$$ | sed -r '/dpkg\/info/sX.*"(.*)".*X\1Xp' -n | xargs cp -p -t "$NEW"
    find /var/lib/dpkg/info -type f -print | xargs cp -pu -t "$NEW"
    rm -f /tmp/dpkg.tr.$$
    
    #--------------------------------------------------
    # Rename old info directory.
    # DO NOT DELETE UNTIL YOU ARE CONVINCED dpkg WORKS.
    #--------------------------------------------------
    echo "Create backup:"
    if [ -e "$BACKUP" ]; then
        echo "Backup directory $BACKUP already exists."
        exit 1
    fi
    mv "$ORIGINAL" "$BACKUP"
    rc=$? ; if [ $rc -ne 0 ] ; then echo "ERROR: mv $ORIGINAL $BACKUP failed rc=$rc" ; exit 2 ; fi
    
    mv "$NEW" "$ORIGINAL"
    rc=$? ; if [ $rc -ne 0 ] ; then echo "ERROR: mv $NEW $ORIGINAL failed rc=$rc" ; exit 2 ; fi
    
    echo "Dpkg search time after optimization:"
    sync
    echo 3 > /proc/sys/vm/drop_caches
    time dpkg -S /bin/ls
    Time to perform search for package that provides ls, before optimization:
    coreutils: /bin/ls
    0.21user 0.27system 0:08.86elapsed 5%CPU (0avgtext+0avgdata 0maxresident)k
    35064inputs+0outputs (7major+12188minor)pagefaults 0swaps
    Optimize:
    Create backup:
    Dpkg search time after optimization:
    coreutils: /bin/ls
    0.17user 0.23system 0:01.01elapsed 40%CPU (0avgtext+0avgdata 0maxresident)k
    33000inputs+0outputs (7major+12187minor)pagefaults 0swaps

    From 8 seconds to 1 second on a dual core, 3 GHz machine. You can see that CPU goes up in the second case, as there is less waiting for I/O.
    -------------------------------------
    Oooh Shiny: PopularPages

    Unumquodque potest reparantur. Patientia sit virtus.

Page 2 of 3 FirstFirst 123 LastLast

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •