One pendrive for all PC (Intel/AMD) computers - single-boot dual-boot multi-boot

[sudodus]

Do you mean to transfer /home between an installed system and a persistent live system?

I have not addressed that task, but it should be possible, maybe even rather easy, to add it.

[C.S.Cameron]

It has worked the times I have tried it, once was from a Full install to a USB with persistent home-rw file.

[sudodus]

New version of mkusb-backup-n-restore-home

I found some bugs and some lack of information in the dialogues, so I made a new version of mkusb-backup-n-restore-home.

Please notice that the screenshots illustrate an upgrade from Lubuntu 16.04.1 LTS to Lubuntu 18.04 LTS.

I have tested upgrading using Ubuntu and Lubuntu iso files. It works to

• upgrade from 14.04.1 LTS to 16.04.1 LTS and directly to 18.04 LTS,
  
• downgrade from 18.04 LTS to 16.04.1 LTS and directly to 14.04.1 LTS,
  
• upgrade and downgrade from 16.04.1 LTS to 18.04 LTS and 14.04.1 LTS.
  
• Edit 1: test with Ubuntu 12.04 LTS,
  
  • mkusb has general problems with 12.04.1 LTS. It is not a hybrid iso file, but after treatment with isoybrid, cloning made a bootable USB drive. mkusb could not make persistence work because the overlay structure does not match what mkusb expects. 12.04 LTS has passed end of life, but I wanted to test anyway,
    
  • mkusb works with 12.04.5 LTS (with the trusty kernel). I used the 32-bit version, because I had that iso file already,
    
  • mkusb-backup-n-restore-home can backup /home from the 'casper-rw' partition of Ubuntu 12.04.5 LTS (32-bit) and restore it to the 'casper-rw' partition of Ubuntu 18.04 LTS, (64-bit),
    
  • Ubuntu 18.04 LTS works as it should with /home from 12.04.5 LTS.
• Edit 2: Improved finish of the shellscript
  
  • "Please wait for sync (flushing file system buffers to the device)"
• Edit 3:
  
  • Further improved finish of the shellscript
  • Merged into mkusb-dus version 12.3.2 - as options in the 'starter' menu. See screenshot #5.

---

The same links are upgraded with a new tarball and a corresponding md5sum file

phillw.net/isos/linux-tools/uefi-n-bios

$ md5sum -c mkusb-backup-n-restore-home.tar.gz.md5
mkusb-backup-n-restore-home.tar.gz: OK

Please test that it works for you too. Feedback is welcome

End of output to console during backup:

...
home/lubuntu/.config/user-dirs.locale
home/lubuntu/.bash_history
'/home' from 'casper-rw'  backed up  to the file
 /home/olle/mkusb-backup-home-lubuntu-16.04.tar.gz 
from the device
 name:  /dev/sdb 
 model: Extreme          
 size:  29,8G 
 OS:    Lubuntu 16.04.1 LTS amd64 
Please wait for sync (flushing file system buffers to the device)
 Done :-)

End of output to console during restore:

home/lubuntu/.config/user-dirs.locale
home/lubuntu/.bash_history
'/home' in 'casper-rw'  restored  from the file
 /home/olle/mkusb-backup-home-lubuntu-16.04.tar.gz 
to the device
 name:  /dev/sdc 
 model: Extreme          
 size:  14,9G 
 OS:    Lubuntu 18.04 LTS amd64
Please wait for sync (flushing file system buffers to the device)
 Done :-)

[sudodus]

I like mBNR.
Everything works for me backing up and restoring the home directory embedded in casper-rw.
How do I use mBRN to backup the desktop's home directory and copy it to casper-rw on a USB?
And maybe vice-versa.

Do you mean to transfer /home between an installed system and a persistent live system?

I have not addressed that task, but it should be possible, maybe even rather easy, to add it.

It has worked the times I have tried it, once was from a Full install to a USB with persistent home-rw file.

I have thought about transfer between a full install to a USB with persistent home-rw file and vice-versa.

- If the user IDs are different, you must either create a user ID that matches the subdirectory in /home or rename that subdirectory.

- The user's numeric ID is 999 in [persistent] live systems, but 1000, 1001, ... for user IDs in installed systems. I think this should also be fixed for the ownership and permissions to work correctly. You can also add a matching user.

- There is a corresponding issue between flavours of Ubuntu. The live user ID in standard Ubuntu is ubuntu, but lubuntu in Lubuntu. (The live user's numeric ID is 999 in both cases, so in this case it is enough to rename the subdirectory in /home.)

But these things are beyond what I think should be included in this shellscript tool. I suggest that advanced users (like you) can do it manually.

Please tell me if you have a different opinion, and in that case what you suggest

Finally, there is an issue about size. It should work well to use the /home directory from a persistent live system in an installed system, but after using an installed system for a while, the /home directory can be quite big, and may not fit in the casper-rw or home-rw partition of a persistent live system. Maybe you can exclude the personal files, when making the tarball.

[C.S.Cameron]

I recall that it only worked for me if user name and passwords were the same on desktop and on persistent flash drive.
I did not try to find a solution to user names being different.
I think it should be easy enough to make a backup of the desltop home folder manually and then use mBRN to extract home to the flash drive.

Well done!

[sudodus]

Using /home from an installed system in a persistent live system

I tested with Ubuntu 18.04 LTS.

1. In an installed system with a lot of data (iso files, other compressed image files and multimedia files) I checked the size with a dry run with rsync

sudo rsync -Havn --exclude={Downloads,Video} /home /target-directory

2. Then used the same exclude setting to create a tarball

sudo tar -cvz --exclude={Downloads,Video} -f /target-directory/backup-home-of-installed-system.tar.gz /home

An alternative is to exclude the file types, that are big and should not be backup to (and not transferred to the new system),

... --exclude=*.{iso,img*} ...

This way I could get a fair size of the tarball.

3. I created a persistent live system in a USB pendrive with mkusb.

4. I restored from the tarball to the persistent live system.

5. I booted into the persistent live system. It used the user ID 'ubuntu' (and not my name), and did not use the restored settings and files.

6. I moved /home/ubuntu to /home/ubuntu-orig

7. I moved /home/my-name to /home/ubuntu

8. I changed owner of this /home/ubuntu

sudo chown -R 999:root ubuntu

9. and rebooted.

---

Now the persistent live system can use the settings and files.

There is a glitch that I have not fixed. The system wants you to

Enter password to unlock your login keyring

and is not happy with the blank password (typical for a [persistent] live system), but it works to

click on the 'Cancel' button

in the log in dialogue window and arrive at the desktop. It works to enter the password of the original installed system too, but as a lazy user I will click on the Cancel button

---

Edit 1: The password from the installed system works, but the persistent live system can be logged in via the cancel button.

In the installed system, before creating the tarball, if you install the package gnome-system-tools, you get the program users-admin, that you can use to make the system skip the login window. This should be inherited to the persistent live system. I will check and confirm later ... and here is the result:

See this link: How to Enable Automatic Login in Ubuntu

For GNOME users, the process is pretty much the same, but it’s not worth the effort, as you are required to unlock your password keyring on startup rendering auto-login meaningless.

So this should work for Ubuntu community flavours, but it will not work with standard Ubuntu 18.04 LTS.

See also this link: How to enable Automatic Login on Ubuntu 18.04 Bionic Beaver Linux

which seems more helpful, but does not really help, because the problem is that the window asks you to unlock the keyring, as the previous link informed about.

- Click near the top right corner of the desktop
- Click on the 'tools' button
- Select 'Details' (in English at the bottom of the list)
. Select 'Users'
. if necessary: click on 'Unlock' at the top right corner of the window
. Click on the "button" ON/OFF to set Automatic Login (might already be selected)

Close and reboot.

---

Edit 2: I tested with Lubuntu 18.04 LTS.

The method described works without any glitch with Lubuntu 18.04 LTS. There is no complaint about the login keyring after /home is transferred from an installed system to a persistent live system.

[sudodus]

Update and upgrade of persistent live systems

The partition that stores persistent overlay data is easily damaged, particularly if you remove the pendrive before it is unmounted.

Update and upgrade

It is a good idea to use persistence to add functionality by installing packages and tweak the system to what you like.

Common experience

But it is bad idea to update/upgrade a persistent live system continuously. Then you will soon get a borked system according to the experience of several people.

Test with update and full-upgrade

I have tested (and am still testing) a persistent live Lubuntu 16.04.1 LTS system in a 60 GB SSD. Compared to a typical USB pendrive the SSD is fast, big and reliable. It was possible to update and upgrade it to be up to date in June 2018, which is almost 2 years after the iso file was released, and the system works after several reboots and installation of new program packages.

sudo apt update
sudo apt full-upgrade

'df -h' reports 1.9 GiB used space in the 'casper-rw' partition, and there are no problems so far. Maybe this system will fail soon, maybe it will continue working for a long time. Anyway, particularly if you have a rather small USB pendrive, you don't want to waste drive space on system upgrades. So it is a good idea upgrade by switching to a system made from an iso file, that is up to date.

There are reasons to use a current kernel and its hardware drivers in the kernel series or skip to the next kernel series of 16.04.2 - 16.04.5 and 18.04 LTS. A persistent live system will only use the kernel from the iso file, so start from a current iso file for this purpose too.

home directory survives

If/when the persistence no longer works you may need to start all over again by removing what has been stored. Often you can keep the [overlay copy of the] home directory (and delete all other subdirectories of your overlay system in the casper-rw partition). Delete while running it live-only! This will save many settings, tweaks and personal files, but you must reinstall the program packages, that have been added.

---

Edit: I should 'admit' that apt full-upgrade throws errors caused by cryptsetup (and affecting the installer ubiquity too), but these are not affecting the way I use the persistent live drive. For example,

- I can still install new program packages into the persistent live system,

- I tested today, and I can install (Lubuntu 16.04.1) from the persistent live system, so ubiquity is still working,

- It is always possible to select 'Try Lubuntu' or 'Install Lubuntu' when booting and run live-only, in order to install (with or without encrypted disk alias LVM with LUKS encryption).

$ sudo apt full-upgrade 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Calculating upgrade... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
3 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] y
Setting up cryptsetup (2:1.6.6-5ubuntu2.1) ...
update-initramfs is disabled since running on read-only media
update-rc.d: warning: start and stop actions are no longer supported; falling back to defaults
insserv: Service checkroot has to be enabled to start service cryptdisks-early
insserv: exiting now!
update-rc.d: error: insserv rejected the script header
dpkg: error processing package cryptsetup (--configure):
 subprocess installed post-installation script returned error exit status 1
dpkg: dependency problems prevent configuration of ubiquity:
 ubiquity depends on cryptsetup; however:
  Package cryptsetup is not configured yet.

dpkg: error processing package ubiquity (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of ubiquity-frontend-gtk:
 ubiquity-frontend-gtk depends on ubiquity (= 2.21.63.6); however:
  Package ubiquity is not configured yet.

dpkg: error processing package ubiquity-frontend-gtk (--configure):
 dependency problems - leaving unconfigured
No apport report written because the error message indicates its a followup error from a previous failure.
                          No apport report written because the error message indicates its a followup error from a previous failure.
                                                    Errors were encountered while processing:
 cryptsetup
 ubiquity
 ubiquity-frontend-gtk
E: Sub-process /usr/bin/dpkg returned an error code (1)
lubuntu@lubuntu:~$

[sudodus]

Replace the 'usbdata' label and the NTFS file system partition of a mkusb persistent drive with a 'home-rw' label and the UDF file system.

This is an alternative to mk-persistent-live_with_home-rw and mkusb-backup-n-restore-home which was created some months ago.

The new shellscript has a descriptive name, udf2home-rw.

Please download the attached gzip-compressed file, extract the file, and test it. Notice that it is an experimental shellscript.

Edit: A third version is uploaded (replacing the earlier versions).

$ gzip -d udf2home-rw.gz
$ md5sum udf2home-rw
470ad275030ec462015bd35eb32616f5  udf2home-rw

End of edit.

1. Create a persistent live drive with mkusb.

2. Run ./udf2home-rw locally.

$ ./udf2home-rw
udf2home-rw
live system or temporary superuser permissions
mpcrw=/tmp/tmp.71OiVQxST8
Device      Model            Size     Persistent live OS
/dev/sdc    Extreme          29,8G    Xubuntu Core 18.04 - amd64
cntcrw=1
srcdev=/dev/sdc5
 Device: /dev/sdc  OS: Xubuntu Core 18.04 - amd64 
bupfile=
umount: /dev/sdc1: inte monterad.
umount: /dev/sdc3: inte monterad.
umount: /dev/sdc4: inte monterad.
umount: /dev/sdc5: inte monterad.
1+0 poster in
1+0 poster ut
1048576 byte (1,0 MB, 1,0 MiB) kopierade, 0,0387125 s, 27,1 MB/s
filename=/dev/sdc1
label=home-rw
uuid=5b548178af6da87e
blocksize=512
blocks=30314496
udfrev=2.01
start=0, blocks=64, type=ERASE 
start=64, blocks=13, type=VRS 
start=77, blocks=19, type=ERASE 
start=96, blocks=16, type=MVDS 
start=112, blocks=16, type=ERASE 
start=128, blocks=16, type=LVID 
start=144, blocks=112, type=ERASE 
start=256, blocks=1, type=ANCHOR 
start=257, blocks=30313982, type=PSPACE 
start=30314239, blocks=1, type=ANCHOR 
start=30314240, blocks=96, type=ERASE 
start=30314336, blocks=16, type=RVDS 
start=30314352, blocks=143, type=ERASE 
start=30314495, blocks=1, type=ANCHOR 
mkudffs: Warning: Creating new UDF filesystem on partition, and not on whole disk device
mkudffs: Warning: UDF filesystem on partition cannot be read on Apple systems
Please wait for sync (flushing file system buffers to the device)
Partprobe: 
Umount: 
umount: /dev/sdc3: ingen monteringspunkt angavs.
umount: /dev/sdc4: ingen monteringspunkt angavs.
umount: /dev/sdc5: ingen monteringspunkt angavs.

MODEL            NAME   FSTYPE  LABEL                      MOUNTPOINT  SIZE
Extreme          sdc                                                  29,8G
                 ├─sdc1 udf     home-rw                               14,5G
                 ├─sdc2                                                  1M
                 ├─sdc3 vfat    usbboot                                244M
                 ├─sdc4 iso9660 Xubuntu Core 18.04 - amd64             682M
                 └─sdc5 ext4    casper-rw                             14,5G
 Created UDF file system on home-rw partition '/dev/sdc1' :-)

3. Boot a computer from the modified persistent live drive and test that things work.

4. Connect the modified persistent live drive to Windows, and check that Windows can 'see' it and use it (read/write).

---

It works to boot and run a persistent live drive with the UDF file system in the casper-rw partition, but some important tasks do not work. For example, it is not possible to install programs with apt (and apt-get and other tools from the repositories), so it is not a useful alternative.

Instead I think this shellscript is a good alternative, but there is still a problem, that Windows wants to format the casper-rw partition.

[sudodus]

I have downloaded the script and purchased a new flash drive. will test in the morning.

The new drive is a Kingston DT50 16GB, not that fast ~100R ~35W, but boots USB3.

I've been a little horrified reading about TLC drive, kinda changes my perspective on preserving writes.

1. Hardware discussion

We have several threads going on here and at AskUbuntu discussing how to use USB pendrives to boot linux looking at wear as well as other issues.

I think you are right about the hardware (including the memory cells and the programmed electronic systems in the pendrives for input/output and wear leveling).

There can be big differences, and it is hard for end users to find out what is actually inside a particular pendrive, and how many write cycles it can be expected to last.

By the way, the 'best' USB pendrives with high end memory cells and programmed electronic systems are more expensive than standard SSD SATA drives of the same size. (I would say more than the double price. You can buy an external box or adapter too for the same money.) But the 'best' USB pendrives are more convenient to use.

2. UDF

I'm looking forward to your test of the script to convert partition #1 to a home-rw partition with UDF.

- Does it work correctly for what you [want to] do with persistent a live drive?

- If it works, would it be worthwhile to provide a means for backup and restore (similar to what I made some weeks ago for /home in casper-rw)?

- Is it worthwhile, when the casper-rw partition needs an ext file system anyway, and Windows 10 is eager to format it?

3. USB pendrives versus persistent live systems

- How much are you into creating installed systems in USB pendrives instead of persistent live systems? Do you think we should focus on that, maybe a special feature of mkusb to prepare a USB pendrive for it, or a separate shellscript/program?

[C.S.Cameron]

The script runs well from another Persistent drive but not from a Full install drive.
Attached screen shots, a red warning and gparted.
How to find how much space left in home?
Script seems to move everything from home in the casper-rw partition to the home partition.
I'm still playing will let you know if I notice anything else.
I'm not sure I am too hot on UDF though.
It is only a small pain for me when windows wants to reformat, except if I have a lot of partitions then it takes a while.

I really like the idea of having an flexible sdx4 that could be used for ISO9660 as is, or for a Full installed system, or maybe for multibootin ISO's. I'm wondering if there should be a basic page in mkusb and maybe a pro page or advanced page.
But I would hate to see mkusb messed up, it is pretty good as is. There are lots of uses for Persistent and Live installs.

I think mkusb would be the first USB tool to make Full installs, which is the future.

Edit:
There is a new unknown partition where usbdata used to be but stuff dumped on the desktop is still going to home in casper-rw, I think.