gpg/ssh info

[TheFu]

I have to be honest: although LXD is a wonder once it is mastered and up and running, it is a royal pain getting it set up properly.

Very true. Most of the issues I had getting it setup came down to 3 issues.

* The LXD team **really** likes ZFS. Using any other back-end storage is counterproductive. I don't use ZFS due to some circumstances here, but I ended up creating a file-based block device to place ZFS on, just for LXD needs. I would have rather used LVM, which I'm more familiar with and have been using 20 yrs. Alas, I was unable to figure out how to convince LXD to use LVM.

* Networking. This wasn't really much of an issue for me, but I have network bridges setup on most of my physical machines for use by VMs. LXD can be hooked into any of those bridges, but most new users wouldn't know what a bridge is or how to set them up. I think by default, LXD sets up a new NAT subnet for containers. I don't recall today. It is a common solution for VM desktops and makes sense. But for "servers" - say a container running a small internal-only website - having bridged network is more sensible. In Linux, networking isn't part of the container or VM tools. It is separate, so if the admin setting things up (this applies to all hypervisors too), they need to setup at least 1 bridge too.

* Snap package. LXD is only available as a snap package with all the great and terrible things about that constrained package format and running in that environment. The other issues, above, would be there regardless of the snap or no snap. I have to setup different userids to run snaps, since my normal user cannot due to some decisions by the snap package team around where home directories must be located and which storage can be used. They have valid reasons for this, but other, similar, package tools allow local overrides. Snaps to not. Also, I'd like more control over where VMs are stored. Due to snap constraints, access to storage locations they haven't 'blessed' and actually hard coded, we cannot.

I'll probably never use LXD for anything with lots of data. It will only be used for 1-off, small, internal, services where most of the data isn't tied to the snap.

Though documentation is a little odd for lxd, it is there. They just split up the documents for 1 command into 10 other manpages, which have lots of details, but lack the caveats and 'why' certain options should be used or not. LXD is the name of the system and daemon, but lxc (this can be confusing) is the main program used post-install of the LXD system. There are manpages for .... well .... here are some of them.

lxc                                lxc.image.alias                    lxc.profile.edit
lxc.alias                          lxc.image.alias.create             lxc.profile.get
lxc.alias.add                      lxc.image.alias.delete             lxc.profile.list
lxc.alias.list                     lxc.image.alias.list               lxc.profile.remove
lxc.alias.remove                   lxc.image.alias.rename             lxc.profile.rename
lxc.alias.rename                   lxc.image.copy                     lxc.profile.set
lxc.cluster                        lxc.image.delete                   lxc.profile.show
lxc.cluster.enable                 lxc.image.edit                     lxc.profile.unset
lxc.cluster.list                   lxc.image.export                   lxc.publish
lxc.cluster.remove                 lxc.image.import                   lxc.remote
lxc.cluster.rename                 lxc.image.info                     lxc.remote.add
lxc.cluster.show                   lxc.image.list                     lxc.remote.get-default
lxc.config                         lxc.image.refresh                  lxc.remote.list
lxc.config.device                  lxc.image.show                     lxc.remote.remove
lxc.config.device.add              lxc.info                           lxc.remote.rename
lxc.config.device.get              lxc.launch                         lxc.remote.set-default
lxc.config.device.list             lxc.list                           lxc.remote.set-url
lxc.config.device.override         lxc.move                           lxc.rename
lxc.config.device.remove           lxc.network                        lxc.restart
lxc.config.device.set              lxc.network.attach                 lxc.restore
lxc.config.device.show             lxc.network.attach-profile         lxc.snapshot
lxc.config.device.unset            lxc.network.create                 lxc.start
lxc.config.edit                    lxc.network.delete                 lxc.stop
lxc.config.get                     lxc.network.detach                 lxc.storage
lxc.config.metadata                lxc.network.detach-profile         lxc.storage.create
lxc.config.metadata.edit           lxc.network.edit                   lxc.storage.delete
lxc.config.metadata.show           lxc.network.get                    lxc.storage.edit
lxc.delete                         lxc.profile.delete                 lxc.storage.volume.get
lxc.exec                           lxc.profile.device                 lxc.storage.volume.list
lxc.file                           lxc.profile.device.add             lxc.storage.volume.move
lxc.file.delete                    lxc.profile.device.get             lxc.storage.volume.rename
lxc.file.edit                      lxc.profile.device.list            lxc.storage.volume.set
lxc.file.pull                      lxc.profile.device.remove          lxc.storage.volume.show
lxc.file.push                      lxc.profile.device.set             lxc.storage.volume.unset
lxcfs                              lxc.profile.device.show            lxc.version
lxc.image                          lxc.profile.device.unset

More documentation than anyone would read.

Most of us would find a walk-thru guide and follow that, trusting it does "the right thing." Once the networking, storage, and minimal understand for LXD are solved, getting from idea to having a new server is, perhaps 5 minutes. LXD containers are treated very much like VMs, which no other linux containers can be done that way.

lxc - is a terrible choice for the LXD client program. This is because the first set of scripts to create Linux Containers 15 yrs ago was named "lxc" and this was used by everyone else with their on containers, including Docker, for years. lxc (lxd's version) is not that lxc. None of those manpages listed above are either.

Confusion.

Canonical has and end-user KVM VM tool called MultiPass. I've never gotten it to work, but that's more about their choice to make it a constrained snap package. Besides lxd, I've struggled to get most snaps working. Snaps are not flexible and THAT is a huge issue for Linux systems.

[coley9225]

OK..status update.
It took me better part of the day, but I now have kvm/qemu installed, and working lubuntu 21.10 on it. Static ip set. I was able to ssh into the VM. I edited my ssh_config so all I have to type is

ssh coley

and I am ssh'ed into lubuntu VM. So far so good...HOWEVER...
I noticed that the VM is located in my / partition. Is there a way to make it go to my /home partition, or even a seperate partition. I have a large amount of room on my ssd, but unsure how to resize or move /home, and the resize /.
I have limited space left in my / partition, and don't want to fill it up, so moving to /home or a separate partition would be tremendous help. Ideas?
I can post returns of any commands you may need to help me with this.
Thanks

[TheFu]

lubuntu 21.10 support ends in less than 2 months. Should have started with 22.04 or 20.04, which are LTS releases with 3yrs support. 20.04 Lubuntu support will end next April, but you'll have many months to consider the migration.

KVM is an enterprise virtual machine tool. If you use libvirt to manage VMs (and I wouldn't do it any other way), it expects VMs to be something that isn't per user, but managed by a team.
Don't put shared things under your HOME, especially not VMs. Put them somewhere else or mount some extra storage specifically for VMs where libvirt expects.

There are multiple solutions to adding storage where it is needed.

• If you used LVM with your OS at installation time, you can tell libvirt about that and have new VMs use LVM-LVs. This assumes that the VG has unallocated storage, which is an LVM best practice. These are all LVM-LVs, each assigned to a virtual machine:
  

  $ sudo lvs
    LV             VG       Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
    lv-blog44-1804 hadar-vg -wi-ao----  16.20g                                                    
    lv-regulus     hadar-vg -wi-ao----  30.00g                                                    
    lv-regulus-2   hadar-vg -wi-ao----  10.00g                                                    
    lv-tp-lxd      hadar-vg twi-a-tz--  32.23g             0.00   10.06                           
    lv-vpn09-2004  hadar-vg -wi-ao----   7.50g                                                    
    lv-xen41-1804  hadar-vg -wi-ao----  12.50g                                                    
    lv-zcs45-1804  hadar-vg -wi-ao----  25.00g

• Mount a separate partition or LVM-LV directly to the libvirt area, sized as needed.
  

  $ sudo lvs
    LV             VG       Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
    libvirt-lv     hadar-vg -wi-ao---- 175.00g                                                    
  
  $ df -Th /var/lib/libvirt
  Filesystem                        Type  Size  Used Avail Use% Mounted on
  /dev/mapper/hadar--vg-libvirt--lv ext4  173G  132G   33G  81% /var/lib/libvirt

• Tell libvirt about a different storage area. Either virt-manager or virsh can be used for this. The virsh manpage has a section called "STORAGE POOL COMMANDS" which goes into how to do this. In virt-manager, there is a Details for the KVM server, then Storage is a tag. The left-hand columns are the major storage groups. At the bottom of the current list, there is a "+" to add more storage groups. I have 6 different storage groups, but almost always, I choose the LVM-VG. It is crazy convenient to use and since the storage isn't mounted to the file system, there is a bit of safety from people accidentally deleting it. Also, no /etc/fstab issues, since they aren't in the fstab. Creating and deleting LVM-LVs through this interface are nearly instant. You can add either a "Filesystem Directory" location or point it a an "LVM Volume Group" or ... well, there are 12 different back-end storage options, like Sheepdog, Gluster, NFS, direct disks, SCSI, iSCSI, and others. I'd warn against using any storage under /home/ or any storage on USB connected storage.

Those are the options.

[coley9225]

I want to again thank you guys for the help.
I pretty much got what I was looking for here. Great references, some tips, and some advise. Some I have already started using, and some(LXD, LVM) I will look into further.

I handled my storage issue, for now, by creating a 300GB partition, made a mount point, /home/charles/KVM_pool, and added an fstab entry for that. That is working great. The 'guts' of the app remains at the default location, and the images get place on the seperate partition.
I have run into a new issue though. However, since I have worked this thread for what I needed, and the new issue(s) are a support type of thing, I will start a new thread in the networking sub-forum to address those.

You guys(community in general) don't get thanked nearly enough. I want to emphasize that you are not unnoticed, or unappreciated. Please, continue the great things you do for us new guys.

I will continue to monitor this thread in case someone has more ideas or references for me to consider.

[TheFu]

Thanks for the kind words.

I would caution against using this location for VMs. /home/charles/KVM_pool. Placing mounts under a user's HOME is a bad idea. There are a number of issues that we have seen with doing this. I'd rather see you mount that partition to /var/lib/libvirt where libvirt expects the default storage pool to be.

But it is your system. You've been warned.

[coley9225]

TheFu, thanks for the tip. I mounted in /home/charles because I thought it would be safer, as it isn't touched during a release upgrade. I think I know how to do this, but I want to make sure.

First, unmount the partition;

sudo umount /home/charles/KVM_pool

Second, edit fstab;

## current entry for KVM_pool
UUID=c23e8deb-98eb-437a-834f-7a9287937dc0  /home/charles/KVM_pool ext4 auto,rw,async,user 0 0
## edit to read
UUID=c23e8deb-98eb-437a-834f-7a9287937dc0  /var/lib/libvirt/images ext4 auto,rw,async,user >

Third, edit default pool as follows;

## existing entry
    <path>/home/charles/KVM_pool/images</path>
## edit to
<path>/var/lib/libvirt/images</path>

Finally, remount the partition;

sudo mount /var/lib/libvirt/images

If this is not correct, please let me know. I will hold back on making the changes stated above until I'm sure this is correct.

[TheFu]

Just be certain that you stop all VMs and KVM and get any directories/files in /var/lib/libvirt/images/ moved to the new storage first.

The fstab above isn't complete. Looks like the copy/paste missed the last 2 fields which are critical.
I wouldn't use those options. This is my exact fstab line:

/dev/hadar-vg/libvirt-lv  /var/lib/libvirt    ext4     noatime,errors=remount-ro 0 2

I use LVM, so the first field points to the LVM-LV. You'd use the UUID= method.

[coley9225]

Thanks. I only have the images stored on the separate partition, all config files, binaries, etc. are store at default locations. I'm not moving the storage itself, just changing the mount point. I have all VMs stopped, and closed virt-manager. I'll double check to see if anything else is running and stop those. God willing and the creeks don't rise, this should go smoothly. If not, I'll be back looking for more help.

[TheFu]

Thanks. I only have the images stored on the separate partition, all config files, binaries, etc. are store at default locations. I'm not moving the storage itself, just changing the mount point. I have all VMs stopped, and closed virt-manager. I'll double check to see if anything else is running and stop those. God willing and the creeks don't rise, this should go smoothly. If not, I'll be back looking for more help.

I think it should work. When I did it, I hadn't migrated any VMs to the new machine yet, so I really don't know if that is sufficient. The kvm+libvirt subsystem may have open files even when they aren't used. A reboot after changing it in the main install should be sufficient. Worst case, boot into a Try Ubuntu environment from flash media and move the mount. Always have a flash drive with the same version of Ubuntu (flavor doesn't matter) available to fix still stuff like this.

[coley9225]

It worked, with a little work. Kept changing paths, and still couldn't get 'virsh pool-start default' to work. also couldn't start any VM. I finally solved that. I had to edit each VM xml file to the new path, after that, it's working fine. I was just using ssh to update and upgrade each of them.
By the way, you were curious as to why I installed lubuntu 21.10 when support ends soon. I wanted to see if I could do a release upgrade via ssh. That went very well, it is now a full 22.04 OS. It also let me 'test run' the process so I could be sure I understood it, so I can reduce any chance of surprises when I do the release upgrade on my bare metal install. That was a new install with almost no added apps whereas I have added a great deal to my computer, but now I know what to expect.

Again, many thanks.

ps.. I have 2 bootable usb drives with the installers, of multiple distros. All I need to do is get more familiar with chroot, but enough knowledge to get the most common issues handled.
Plus, there's alway the forums to help.