No LUKS password prompt for Ubuntu installation with manual partitioning

[Paddy Landau]

If I install Ubuntu with LUKS, asking it to encrypt the entire drive, I can get it to boot properly (after adding startup.nsh and setting nosplash in Grub).

However, when I use manual partitioning so that the installer doesn't overwrite specific partitions, I cannot get the boot sequence to prompt for the LUKS password. Instead, after a minute or so, the boot falls into initramfs (whatever that is :().

My test setup is similar to the default whole-disk encryption:

/dev/sda1 EFI System Partition
/dev/sda2 Boot partition
/dev/sda3 Unused space
/dev/sda4 LUKS partition containing an LVM group, which contains a swap partition and a root partition.

All partitions start off empty, so that the installer can format and fill them as needed.

I ran the installer. I don't know where to place the bootloader, so I've tried both /dev/sda and the default /dev/dm-0 (which points to the encrypted, unlocked partition sda4_crypt).

Once I have added startup.nsh (bug #1665329 (https://bugs.launchpad.net/ubuntu/+source/ubiquity/+bug/1665329)) and fixed Grub (another bug (https://askubuntu.com/questions/574296/cannot-enter-password-to-start-ubuntu)), I can boot into the system as far as the Grub prompt.

Then, after I select Ubuntu, it sits for about a minute before dropping into a prompt for initramfs, something about which I know nothing. (Using the installer's default whole-disk encryption does not have this problem.)

Do you have any idea what I can do to fix this so that I can enter the LUKS password and continue to boot?

More information:

Installing in VirtualBox
Using Ubuntu 16.10 64-bit

[DuckHook]

Hello Paddy,

If using whole disk encryption, everything that is in /boot must be unencrypted for obvious reasons. When you customize your install, do you make sure to set up /boot on an unencrypted partition?

NOTE:

I do not use whole disk encryption, so am of limited help. But the above is one of the usual problems.

[Paddy Landau]

… do you make sure to set up /boot on an unencrypted partition?
Thanks for your reply, @DuckHook. Yes, it is unencrypted. As described in my setup, only the final partition is encrypted.

If I understood how the EFI and boot partitions are set up to request the LUKS password, I might be able to figure out what's going on, but I have failed to understand what little I have managed to find.

[DuckHook]

Old-ish tutorial, no longer maintained, but perhaps it will help: http://thesimplecomputer.info/full-disk-encryption-with-ubuntu

**NOTE**

Untested by me. While I do use encryption, I don't do it whole disk. Too many bad vibes from trying to help people out on these forums. I encrypt only a ~/Private directory, throw all my sensitive stuff in there and symlink back to where the system expects to find it. But I do get the fact that in some situations only full disk will do. I just don't have any personal experience implementing it.

[TheFu]

No help from me, but ...

IMHO, every portable device requires as much encryption as possible. I've had devices brazenly stolen in front of me and via pick pockets. At least through encryption, I can give myself time. With hole disk encryption, they have to wipe the entire system before using it and I don't have to worry about any data or credentials getting out.

I do whole disk encryption and every attempt to make it work manually with booting has failed. I've setup the crypttab, fstab, rebuilt the initrd using the standard tools, etc ... never got it working on OS devices. I can mount LUKS encrypted partition and the LVs inside them no problem. I've pulled disks from properly working encrypted setups and accessed them on other machines too.

It is just finding the magic incantation and which dog I need to make sacrifices towards which have eluded me. ;)

It is really frustrating when the default LV sizes generated by the setup are completely wrong for a specific system.

I'll be lurking hoping someone can make some steps to do this. BTW, CentOS makes this sort of thing trivial. The RPM-based distros all have excellent disk setup tools during install.

[Paddy Landau]

Old-ish tutorial, no longer maintained, but perhaps it will help: http://thesimplecomputer.info/full-disk-encryption-with-ubuntu
Thank you. I had come across something similar, but this one seems more comprehensive. I shall try it when I have a little time.

… every attempt to make it work manually with booting has failed.
…
I'll be lurking hoping someone can make some steps to do this. BTW, CentOS makes this sort of thing trivial. The RPM-based distros all have excellent disk setup tools during install.
Yes, it's frustrating! If I find the solution, I'll post it here. If not, I think that I'll head over to AskUbuntu (https://askubuntu.com/) and ask there, because every proposed solution that I've found so far doesn't work.

[TheFu]

Thanks for the link. After I'm done migrating my remaining 12.04 server to something newer, need to take a look. The link says the default for LUKS in Ubuntu is SHA1 in 2015? Ouch.


$ sudo cryptsetup status sda5_crypt
/dev/mapper/sda5_crypt is active and is in use.
type: LUKS1
cipher: aes-xts-plain64
keysize: 512 bits
device: /dev/sda5
offset: 4096 sectors
size: 116224000 sectors
mode: read/write
flags: discards


LUKS1: aes-xts-plain64, Key: 256 bits, LUKS header hashing: sha1, RNG: /dev/urandom is the defaults on my 16.04 box. Ouch. It is just for the header, but those are the keys to the universe. Don't see how to change that to sha512.

[Paddy Landau]

The link says the default for LUKS in Ubuntu is SHA1 in 2015? Ouch.
Versions 16.04 and 16.10 use SHA256. Enter cryptsetup --help to see the defaults. Scroll right to the end of the help, and you see this (Ubuntu 16.04 and 16.10):

Default compiled-in device cipher parameters:
loop-AES: aes, Key 256 bits
plain: aes-cbc-essiv:sha256, Key: 256 bits, Password hashing: ripemd160
LUKS1: aes-xts-plain64, Key: 256 bits, LUKS header hashing: sha256, RNG: /dev/urandom
Use cryptsetup luksDump to view a specific partition. In my case, /dev/sda3 was created by the default Ubuntu whole-disk encryption installer. The relevant results are:

Cipher name: aes
Cipher mode: xts-plain64
Hash spec: sha256
Payload offset: 4096
MK bits: 512
If your current setup uses SHA1, you can re-encrypt (I've used /dev/sda3 in my example). Warning: do a full backup first in case it fails! Also, it takes a long, long time to re-encrypt, but at least it gives you an ongoing progress report.

sudo cryptsetup-reencrypt --key-size=512 --hash=sha512 /dev/sda3

[TheFu]

Default compiled-in device cipher parameters:
loop-AES: aes, Key 256 bits
plain: aes-cbc-essiv:sha256, Key: 256 bits, Password hashing: ripemd160
LUKS1: aes-xts-plain64, Key: 256 bits, LUKS header hashing: sha1, RNG: /dev/urandom

16.04 install. Looks like different builds have different options?

But ...

$ sudo cryptsetup luksDump /dev/sda5
LUKS header information for /dev/sda5

Version: 1
Cipher name: aes
Cipher mode: xts-plain64
Hash spec: sha256
Payload offset: 4096
MK bits: 512

So all isn't THAT bad.

[Paddy Landau]

16.04 install. Looks like different builds have different options?
It certainly does! How strange :confused:


But ...

Hash spec: sha256
You must be relieved! It seems that the installer overrides the SHA1 default.

[oldfred]

I also do not use LVM nor encryption.

But all the UEFI installs have a separate ESP - efi system partition that looks just like any standard install and an unencrypted /boot partition that is typical /boot.

But whether UEFI or BIOS you always install grub to the drive like sda, never to a partition.
The only exception seems to be FakeRAID, or BIOS RAID where then grub is installed to the root of /mapper which the BIOS RAID uses for booting.

[Paddy Landau]

But whether UEFI or BIOS you always install grub to the drive like sda, never to a partition.
I'm no expert in these things — I thought that the bootloader was loaded to sda whereas Grub is usually installed to /boot (for encrypted drives)?

I have discovered that it is possible to encrypt everything, including /boot, except for the bootloader and the EFI System Partition. However, I still don't know how to do that!

[TheFu]

Other distros allow /boot encryption, but I've never heard of Ubuntu doing it. Links?

The UEFI standard probably supports some specific type of encryption, but I don't have a clue about that. No UEFI here. In my paranoid state, I boot from a USB2 flash drive that never leaves my person when on travel. That's the only way I know to ensure the boot hasn't been tampered with when I step away from the computer for 3 minutes.

[Paddy Landau]

Other distros allow /boot encryption, but I've never heard of Ubuntu doing it. Links?
Ah ha. Maybe Ubuntu's installer somehow disables the ability?

I shall keep seeking, and if I find a solution, I'll post here.

My problem is that I don't understand how the setup works, nor have I managed to find something that I understand (I'm not highly technical).

[oldfred]

When you install grub, you are just installing the boot loader to the drive.
With BIOS boot, boot loader must be in MBR.
With UEFI boot the boot loader must be in the ESP - efi system partition. It auto finds ESP on drive seen as sda.

When you specify the drive to install grub, you are saying to install boot loader. But parts of grub are in several places with a standard install.

GRUB 2 - GRand Unified Bootloader has three main parts plus a boot loader:




/etc/default/grub - the file containing GRUB 2 menu settings.
/etc/grub.d/ - the directory containing GRUB 2 menu creating scripts.And a place for totally custom entries 40_custom.
/boot/grub/grub.cfg - the GRUB 2 configuration file, not editable.


I have not seen any way to encrypt an ESP.
But have seen some links on moving kernels & necessary boot files into ESP and directly booting from that.
http://www.rodsbooks.com/efi-bootloaders/index.html

[Paddy Landau]

But parts of grub are in several places with a standard install.
That explains a lot to me, thank you. I was wondering why I could find Grub on both the ESP and /boot partitions.




/etc/default/grub - the file containing GRUB 2 menu settings.
/etc/grub.d/ - the directory containing GRUB 2 menu creating scripts.And a place for totally custom entries 40_custom.
/boot/grub/grub.cfg - the GRUB 2 configuration file, not editable.


And yet, /etc is not available to Grub until after the encrypted partition has been decrypted. I imagine that is what prevents the non-standard Ubuntu installation from working, because for whatever reason, it does not prompt for the LUKS password.

I have not seen any way to encrypt an ESP.
I imagine not, because something has to be unencrypted for the computer to be able to boot. Presumably the ESP is it.

But have seen some links on moving kernels & necessary boot files into ESP and directly booting from that.
http://www.rodsbooks.com/efi-bootloaders/index.html
Wouldn't that cancel the point of encrypting the application area? I would think that the idea is to have as little as possible unencrypted, which is why encrypting /boot would be such a good idea.

By the way, you can edit /boot/grub/grub.cfg and rebuild Grub (from a Live CD). It is what allows you to overcome a bug in the standard Ubuntu installer, where it doesn't prompt for the password. I have had personal experience of this.

[oldfred]

Booting does not access any of the grub configuration files nor scripts, only grub.cfg.
Running sudo update-grub then reads the settings and creates the grub.cfg for booting.

And if you just want to install only grub to a device, you have to manually maintain a grub.cfg as then you do not have scripts & settings.
I do this for grub on a flash drive to directly boot ISO. Does work for both BIOS & UEFI.

[Paddy Landau]

I do this for grub on a flash drive to directly boot ISO. Does work for both BIOS & UEFI.
If I get time, I'll test this on a flash drive. However, I don't need paranoid mode, so it isn't a priority for me. If I sometime get a laptop and travel, then I shall reconsider.

[francoislepage]

Hi,

I'm having exactly the same problem as described here :

https://ubuntuforums.org/showthread.php?t=2353942

I tried to specify explicitly the cryptdevice in grub with no luck.

GRUB_CMDLINE_LINUX="cryptdevice=/dev/disk/by-uuid/ad3923cd-1667-4c66-a515-b9ebc6e0185e:cryptroot root=/dev/mapper/cryptroot

I also do not understand clearly what is the process through which the password prompt is kicked off... do the initramrd need any tools in order to support encryption? Is it required to load specific modules at boot time? What is the exact role of Plymouth (I have disabled quiet splash).

In my case, kernel 4.8.0-32 works perfectly (the one installed initially by the installer) but subsequent versions installed by apt are not functional.

Please advise,

Thanks

[TheFu]

I don't know and haven't figured it out myself, but I think initrd needs to be build with whatever tools are necessary to handle LVM and LUKS. The normal update-grub process doesn't handle it when we manually try to do the install. I've looked at all the config files I could find last time I tried to get all this stuff working. That was about a year ago. Think I ended up writing a little script to build the initrd needed - clearly, I missed something important too.

The last how-to I found with sufficient details for me to follow (I've only been a Unix C/C++ dev and system admin for 20+ yrs), it was from 2012-ish and extremely out of date for the current build systems.

Perhaps if we look at this a different way completely? Are there any detailed logs of what happened during the install? With those, assuming enough detail is provided, perhaps we'd uncover the config files and tools necessary? I haven't looked for those logs yet myself. Just a thought while I'm at a different system.

[Paddy Landau]

I also do not understand clearly what is the process through which the password prompt is kicked off...
I don't understand it at all!

However, I have been making great progress with my tests, and I hope to post something here by next week (due to illness I cannot work on this anywhere near as speedily as I'd like to). I hope that whatever I find will help you.

[francoislepage]

Hi,

I also though about initrd but the one of the kernel that works and the one of the kernel that doesn't work are identical in content.

[DuckHook]

I don't understand it at all!

However, I have been making great progress with my tests, and I hope to post something here by next week (due to illness I cannot work on this anywhere near as speedily as I'd like to). I hope that whatever I find will help you.Paddy, I know you've been deciphering this on your own and are currently working on a community wiki article, but I just ran across this which might help: http://m.linuxjournal.com/content/preseeding-full-disk-encryption

[Paddy Landau]

Paddy, I know you've been deciphering this on your own and are currently working on a community wiki article, but I just ran across this which might help: http://m.linuxjournal.com/content/preseeding-full-disk-encryption
Thank you for the link, DuckHook.

That particular article assumes that Boot needs to be unencrypted and that the installation forces prefilling the disk with random data, but in fact neither is true for Ubuntu. Also, the solutions are at a level that is too technical for me!

I have managed to decypher (pun intended) the entire process, and I'm busy documenting it. It's taking a long time, partly because I'm unwell and partly because it isn't straightforward.

[TheFu]

Have you figured out how to dual boot - perhaps a tiny 10G OS and the rest be totally encrypted OS?

With border controls, I'd like to have something like ChromiumOS to show without having to unlock/decrypt a HDD with my passphrase/2FA.

[DuckHook]

It's taking a long time, partly because I'm unwell and partly because it isn't straightforward.There's no rush on these things Paddy. Look after yourself first.

[Paddy Landau]

Have you figured out how to dual boot - perhaps a tiny 10G OS and the rest be totally encrypted OS?
Everything Ubuntu-related is encrypted. The ESP (EFI System Partition) is unencrypted, because that's what the UEFI uses to initiate the boot. An existing Windows system (including its recovery partitions) are left alone, therefore unencrypted. I haven't tried this on a Chromebook, but I presume that the same will apply.

The Ubuntu system is fully encrypted with LUKS. This includes /boot, swap and root (which must be all on the same physical partition or it won't boot), and any additional partitions that you choose to have such as /home.

However…

Because boot is encrypted, you will have to enter the LUKS passphrase to boot.

If you're trying to fool border control, I suspect that you'll find their IT people to be cleverer than that!

I'm wondering if this has something to do with my current problem with un-encrypting drives after installing VirtualBox?
I've just had a look at your thread, and it has nothing to do with this topic, sorry. I shall respond to your thread shortly.


There's no rush on these things Paddy. Look after yourself first.
Thanks, DuckHook, I have to look after myself otherwise I land in hospital :(

[TheFu]

Can't use UEFI on my Chromebooks. Legacy boot only.

I'm not trying to fool anyone, just want to prove the device works without having to unlock encrypted storage. A quick boot is all I've ever had to show, nothing more. Chaining boot loaders isn't a big deal usually.

Windows won't boot on chromebooks, at least not outside a VM. Zero interest in running Windows.

[Paddy Landau]

Can't use UEFI on my Chromebooks. Legacy boot only.
I have only tested this with EFI. I shall test it with non-EFI, but it will take time to get an answer.


I'm not trying to fool anyone, just want to prove the device works without having to unlock encrypted storage.
Ah, I forgot that this was a new security requirement. I don't travel any more.


Chaining boot loaders isn't a big deal usually.
That's something where I have zero knowledge, sorry!

[Paddy Landau]

In case you are interested, I have completed the documentation for full-system encryption (https://help.ubuntu.com/community/ManualFullSystemEncryption).

Testers would be welcome.

[TheFu]

Read the main page. Very nice.

I'd alter - there are 8 slots to unlock each encrypted partition. That means 8 different passphrases can be used, so if there are fewer than 8 physical-access users of the system, sharing of the disk unlock passphrase isn't necessary.

man cryptsetup explains:

Each passphrase, also called a key in this document, is associated with
one of 8 key-slots. Key operations that do not specify a slot affect
the first slot that matches the supplied passphrase or the first empty
slot if a new passphrase is added.
I've used multiple slots. They work.

It will take some time to make it through the entire thing.

[Paddy Landau]

I'd alter - there are 8 slots to unlock each encrypted partition…
Thanks, TheFu. I hadn't considered this. There are six slots available, because one is used for the automatic decryption and another for the manual passphrase. I'll wait for any other feedback before making changes.

[Paddy Landau]

I'd alter - there are 8 slots to unlock each encrypted partition.
I've added this to the instructions. Thanks for the good idea.

[Sami_Mattila]

I don't remember ever smiling before while reading a manual.
Thx 4 that

:popcorn:

Paranoid mode
"Encryption can be taken a little further, which might be an idea if you deal with huge volumes of sensitive customer data; government secrets or spying; confidential proprietary business research; or conspiracy theories and aliens."

[Paddy Landau]

I don't remember ever smiling before while reading a manual.
Glad to be of service :D