Page 2 of 2 FirstFirst 12
Results 11 to 15 of 15

Thread: Backup for Snap configurations?

  1. #11
    Join Date
    Mar 2010
    Location
    Squidbilly-Land
    Beans
    Hidden!
    Distro
    Ubuntu

    Re: Backup for Snap configurations?

    So, with lots of assumptions, seems that snap save run on the lxd host system will create a full ZIP file of lxd settings, configs, DBs, all lxc containers and data for all users. I have ZFS setup for the storage pool used by lxd, but /var is an LVM LV.

    Code:
    $ sudo snap save lxd
    $ sudo ls -l /var/lib/snapd/snapshots/
    total 1120224
    -rw------- 1 root root 1147103316 Mar  2 08:31 2_lxd_5.11-ad0b61e_24483.zip
    That lxd zip "snapshot" is for ISO/rootfs images, too. Looks like everything under /var/snap/lxd/common/
    Code:
    # du -sh  /var/snap/lxd/* 
    4.0K    /var/snap/lxd/24323
    4.0K    /var/snap/lxd/24483
    1.2G    /var/snap/lxd/common
    0       /var/snap/lxd/current
    The resulting zip says 0% compression. Inside 2_lxd_5.11-ad0b61e_24483.zip, are:
    Code:
    $ tree 
    .
    ├── archive.tgz
    ├── meta.json
    ├── meta.sha3_384
    └── user
        ├── thefu.tgz
        └── root.tgz
    The contents of the archive.tgz includes all the configs and storage for each container. So, if you want to migrate everything from one LXD host to another, use that command to create a snapshot.

    I've read about an lxc export command which isn't supported on my 20.04 system. I have lxd 5.11 and lxc 4.0.12.
    Https://www.cyberciti.biz/faq/how-to...xd-containers/ has a script to loop over lxc instances and back them up in series.

    None of these methods are friendly towards versioned backups. They are nearly the same as creating an disk image that can be imported.
    As an example concerning the bloat involved,
    Code:
    $ du -h *
    4.0K    lxd.config.2023-03-02
    4.0K    lxd.instances.list.2023-03-02
    4.0K    lxd-version.2023-03-02
    3.8G    nextcloud-lxd-bkp-2023-03-02.tar.xz
    1.8G    pihole2-bkp-2023-03-02.tar.xz
    890M    spam2-bkp-2023-03-02.tar.xz
    For my normal backups, these are the sizes:
    Code:
    $ sudo rdiff-backup --list-increment-sizes  spam2
            Time                       Size        Cumulative size
    -----------------------------------------------------------------------------
    Thu Mar  2 00:03:16 2023         3.73 MB           3.73 MB   (current mirror)
    Wed Mar  1 08:38:30 2023         1.42 KB           3.74 MB
    
    $ sudo rdiff-backup --list-increment-sizes  spam1
            Time                       Size        Cumulative size
    -----------------------------------------------------------------------------
    Tue Feb 28 00:03:28 2023         6.34 MB           6.34 MB   (current mirror)
    Mon Feb 27 00:03:28 2023         1.07 KB           6.34 MB
    Sun Feb 26 00:03:37 2023         1.07 KB           6.34 MB
    Sat Feb 25 00:03:25 2023         1.29 KB           6.34 MB
    Fri Feb 24 00:03:29 2023         2.85 KB           6.35 MB
    ...
    Tue Jul  5 01:01:30 2022       947 bytes           6.80 MB
    Mon Jul  4 01:03:47 2022       947 bytes           6.80 MB
    Sun Jul  3 01:01:39 2022       997 bytes           6.80 MB
    So with daily backups going back to Last July, the versioned backups are less than 7MB compared to 1 "snapshot" of 900MB, which will be that size every day. I'd rather not waste 1GB/day to backup a single container when less than 10MB is sufficient for an entire year.

    Perhaps I'm not being fair. Let's look at a nextcloud backup.
    3.8G nextcloud-lxd-bkp-2023-03-02.tar.xz # for a single "snapshot"
    vs
    Code:
    $ sudo rdiff-backup --list-increment-sizes nextcloud
            Time                       Size        Cumulative size
    -----------------------------------------------------------------------------
    Tue Feb 28 00:05:59 2023         8.49 GB           8.49 GB   (current mirror)
    Mon Feb 27 00:06:00 2023         2.74 KB           8.49 GB
    Sun Feb 26 00:06:33 2023         12.0 MB           8.50 GB
    Sat Feb 25 00:05:50 2023         3.06 KB           8.50 GB
    Fri Feb 24 00:06:21 2023         3.34 KB           8.50 GB
    Thu Feb 23 00:06:41 2023         54.3 MB           8.55 GB
    ...
    Sat Dec  3 00:05:43 2022         63.4 MB           14.2 GB
    Fri Dec  2 10:41:49 2022         55.9 MB           14.2 GB
    Fri Dec  2 00:06:09 2022         6.42 MB           14.2 GB
    So, 90 days of versioned backups take 15G of storage or 1 "snapshot" is 3.8G? 3.8 x 90 = 342GB.

    Hopefully, I've made the point that images and tgz "snapshots" are nowhere near as efficient as versioned backups.

    BTW, the nextcloud-lxd system was installed a few days ago. Last night was the first "normal" backup ... but the versioning seems to be working:
    Code:
    $ sudo rdiff-backup --list-increment-sizes nextcloud-lxd
            Time                       Size        Cumulative size
    -----------------------------------------------------------------------------
    Thu Mar  2 00:06:30 2023          785 MB            785 MB   (current mirror)
    Wed Mar  1 09:23:16 2023          468 KB            785 MB
    Time will tell.

    I'm not using any LVM snapshots here. The container storage is ZFS
    Code:
    # df -Th
    Filesystem                                   Type           Size  Used Avail Use% Mounted on
    lxd/containers/nextcloud-lxd                 zfs             45G  4.2G   41G  10% /
    For nextcloud-lxd, the backup script "pulls" the backups, so it runs on a backup server system, connects to the remote nextcloud-lxd system over ssh (multiple times), does some pre-backup stuff, then finally uses rdiff-backup to pull the data to the backup server.
    Code:
    #!/bin/bash
    
    TGT="/d/D7/backups/$REMOTE"
    REMOTE="nextcloud-lxd"  # Remote system
    RUSER=back1134  # Remote system "root" equiv account
    DAYS="90D"      # Days of backups to retain
    NC_SNAP_BAK_TOP="/var/snap/nextcloud/common/backups"
    
    source /usr/local/sbin/pull-backup-UTILS.sh
    #   To use, just call "Pre-Backup-Info() function"
        Pre-Backup-Info &> /dev/null
    
    # ##################[ Nextcloud Backup Data+Apps ]##################
    echo "INFO: Export the nextcloud snap data, db, config.php and user data"
    
    ssh $RUSER@$REMOTE 'bash -s' <<END_NC_BACK
    # Step 1: remove the last backup directory
    /usr/bin/rm -rf "$NC_SNAP_BAK_TOP/current.bak"
    
    # Step 2: Export the nextcloud snap data, db, config.php and user data
    #         The redirection should retain stderr output, just not stdout
    #         50K lines of junk.
    /snap/bin/nextcloud.export > /dev/null
    
    # Step 3: Move the time-stampped backup directory to "current.bak" so 
    #         a real versioning backup tool can be efficient with files 
    #         that didn't change. Aka - rdiff-backup
    /usr/bin/mv  $NC_SNAP_BAK_TOP/20* "$NC_SNAP_BAK_TOP/current.bak"
    END_NC_BACK
    
    echo "INFO: TODO capture snap settings for NC and snap system"
    sudo ssh $RUSER@$REMOTE 'bash -s' <<END_LXD_BACK
    /usr/bin/snap get system -d > /root/backup/snap-system-settings.txt
    /usr/bin/curl -sS --unix-socket /run/snapd.socket \
                  Http://localhost/v2/system-info |\
                  /usr/bin/jq > /root/backup/snap-system-info.txt
    END_LXD_BACK
    echo "INFO: Running  rdiff-backup $DIRS $RUSER@$REMOTE::/  $TGT"
    /usr/bin/rdiff-backup  --exclude-sockets --exclude-device-files --exclude-fifos \
            --exclude '**/.cache/mozilla' \
            --exclude '**/.cache/chromium' \
            --include /root \
            --include /etc \
            --include /usr/local \
            --include $NC_SNAP_BAK_TOP/current.bak \
            --include /home \
            --exclude '**' $RUSER@$REMOTE::/  "$TGT"
    
    # Remove older backups .... 
    echo "INFO: Removing backups older than $DAYS"
    /usr/bin/rdiff-backup  --remove-older-than "$DAYS"  --force "$TGT"
    
        Post-Backup-Cleanup
    I left out lots of setup and helper functions that get sourced. Also, externally connected storage (lxd external mounts) isn't included. Those helper functions fill /root/backup/ with system data as text files to make restoring to a new system easier. Some of the data captured is just handy for problem solving which don't need file restores.

    The remote system doesn't have any pre-installed scripts. Those are all xferred over as part of the UTILS and the script above. I use heredocs for that. Wrapping your head around which system any specific code in the script is running inside can be challenging for less experienced people.

    The script will probably get tweaked a few more times. What I'll probably end up with is either a weekly or monthly "snapshot" tgz combined with the daily versioned backups trying to achieve the best solution. More complications.

  2. #12
    Join Date
    Jul 2013
    Location
    Wisconsin
    Beans
    4,967

    Re: Backup for Snap configurations?

    You seem to have identified a use case. You should tell the snapcraft team about it. Perhaps they will add a feature for versioned backups of settings.

  3. #13
    Join Date
    Aug 2016
    Location
    Wandering
    Beans
    Hidden!
    Distro
    Xubuntu Development Release

    Re: Backup for Snap configurations?

    Looks Great....very useful share.
    With realization of one's own potential and self-confidence in one's ability, one can build a better world.
    Dalai Lama>>
    Code Tags | System-info | Forum Guide lines | Arch Linux, Debian Unstable, FreeBSD

  4. #14
    Join Date
    Mar 2010
    Location
    Squidbilly-Land
    Beans
    Hidden!
    Distro
    Ubuntu

    Re: Backup for Snap configurations?

    Quote Originally Posted by ian-weisser View Post
    You seem to have identified a use case. You should tell the snapcraft team about it. Perhaps they will add a feature for versioned backups of settings.
    The last few years, I've been dissatisfied with my interactions with the snap/lxd teams. We are both stubborn. Really wish lxd wasn't tied to snaps. I understand that the next stable Debian release will have an lxd .deb package, not tied to snaps at all. https://wiki.debian.org/LXD Stripping out the snap dependency should make things less convoluted. At least that is my hope.

  5. #15
    Join Date
    Aug 2016
    Location
    Wandering
    Beans
    Hidden!
    Distro
    Xubuntu Development Release

    Re: Backup for Snap configurations?

    I can only add lxd on Debian is a breath of fresh air:
    Code:
    apt policy lxd
    lxd:
      Installed: 5.0.2-2+b2
      Candidate: 5.0.2-2+b2
      Version table:
     *** 5.0.2-2+b2 990
            990 http://deb.debian.org/debian sid/main amd64 Packages
            990 http://ftp.us.debian.org/debian sid/main amd64 Packages
            100 /var/lib/dpkg/status
    I'm just hinting here, but Debian Testing has been more trouble free than Ubuntu...in fact I've not had one thing break on my side in 6 months time.
    @TheFu, I'm just talking, I know better to suggest anything testing in your direction.
    But I've found a new home.
    With realization of one's own potential and self-confidence in one's ability, one can build a better world.
    Dalai Lama>>
    Code Tags | System-info | Forum Guide lines | Arch Linux, Debian Unstable, FreeBSD

Page 2 of 2 FirstFirst 12

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •