16.04.01 LTS on HP Proliant Microserver Gen8 G1610T setup woes (ID10T error likely)

[JusticeEmpire]

Hi All,

Firstly, thanks for such a wonderful forum full of useful information. I will try to be brief with my problem to save everyone's time reading.

What I have

I have an HP Microserver (ProLiant Gen8 G1610T), if firmware etc is needed then I can provide that. It was previous running using Ubuntu Server (12 I think) that was setup remotely by a friend. It wasn't using RAID, and the OS and Data were installed/stored on a single 250gb drive, with the 3 other 250gb drives not doing a lot. I have replaced the drives with 4 x 2TB WD Red, and have managed to install Ubuntu Server 16 on a 16gb SD card, which seems to boot and run fine. However, I have VERY limited knowledge with Ubuntu/Linux, apart from what I have learnt from forums.

What I'd like to achieve

I am looking to use this as a server, not only for normal home media type data, but business data too. As such, I want it fully backed up which is partly the reason I bought the box, as I wanted to run RAID1 (I think). I think this should give me 4TB of fully backed up data, and if I need more, I'll upgrade drives or duplicate the box. I think this means setting up SAMBA, due to mainly windows access and for streaming, but I think I would need to install some media software too. I have the data currently sitting on multiple external drives, and I am thinking about transferring to the server via a windows laptop, although I would be up for doing it from CLI with drives connected via USB, if it was relatively simple and quicker.

Where I am stuck and need help

Primarily, I think it's confusion that's hindering me. My friend who set it up before isn't available, and I have a very basic knowledge/understanding of servers, so far! I did try to setup the RAID via the HP Smart Array Management console prior to OS install (possibly my big mistake?). Initially, after I installed the OS on the SD Card, using various guides on the net, I installed and tried to setup a SAMBA share, which eventually I seemed to get working. I was about to start data transfer until it clicked that I had created the share on the SD Card rather than the 2TB drives. After a bit more digging, I thought that all I had to do was mount the logical 4TB volume I'd created, create a folder and map that in SAMBA and it would be sorted. But when I enter

parted -l

, I can see the OS card but it is listed at /dev/sdb. /dev/sda is showing one of the 2TB drives with a partition table, and the other 3 drives are showing up at /dev/sdc, /dev/sdd and /dev/sde with no/unknown partition tables.

Do I need to somehow swap the OS drive from /dev/sdb to /dev/sda? If so, how?

I really don't want to have to switch OS, as I've made the decision to go with Ubuntu and would much rather learn my way around the problem. So how can I create my 4TB of RAID space, if it's not picking up the 4tb logical drive I created?

[howefield]

Thread moved to the "Server Platforms" forum, probably a better fit.

[JusticeEmpire]

Ok thanks, was struggling to decided between 'Server Platforms' and 'Installation'....

[darkod]

How you use and set up your server is of course your own decosion...

But I would go a step back. If you used the bios raid setup, I would delete the raid arrays you created and leave the disks as simple AHCI mode, no raid from the onboard controller. Linux works quite great with its own software raid and that is not dependent on HW.

If you decide to do that, we will set up again all 4 disks with parted, and then create the raid arrays with mdadm.

One thing you need to know and plan in advance, is that with SW raid there is no partitioning of the arrays. You create the arrays the size you want them, no splitting afterwards... So, if you want 2TB for data1 and 2TB for data2, the disks have to be partitioned accordingly. But we can discuss that in details if you decide to do it.

[bab1]

Adding to darko's post....

RAID of any sort is not a backup solution. It's for high availability (HA) only. You need a true backup solution if you want your data protected. I use scripts to run rsync to created full and versioned backups on a regular basis.

[JusticeEmpire]

Thanks for the response.

darkod - What would be your reasons for that? Simplicity? More efficient?
I'm not sure I fully understand the partitioning. Ideally, I want maximum storage possibilities out of my 4 x 2TB drives. If they could appear as one 8TB drive, that would be very handy. Obviously with RAID, I know that would be 4TB, but I've only got about 1.5TB data at the moment anyway. When I was using the BIOS RAID setup, it gave me the option of having a 4TB logical drive and I thought that is what I had setup. I'm imagining that I'd have a 'Media Folder' with sub folders for Films, Music, Photos etc, and another 'Business' folder, again with different subfolders.

bab1 - That is a very valid point, and there is probably some data that I should backup & store elsewhere, even 'offline'. I've got a lot of music stored, and I obviously don't want to loose it, but it's also a little unpractical/uneconomical to backup all of that offline.

[bab1]

Thanks for the response.

bab1 - That is a very valid point, and there is probably some data that I should backup & store elsewhere, even 'offline'. I've got a lot of music stored, and I obviously don't want to loose it, but it's also a little unpractical/uneconomical to backup all of that offline.

Why would you say that about impractical to back up? I don't see why you should even use RAID. I would use LVM so I have all the capacity of all the disks. I would use one or two of the disks as the backup destination. After the initial backup it should take little time to backup. Typically it takes less than five minutes to backup 150GB.

[JusticeEmpire]

Am I right in thinking that with LVM, I could create a 'combined' space of say 6TB, giving me 2TB to backup to? In all fairness, I probably haven't thoroughly understood my backup needs, so I probably should work out what I need to backup and how.

Is the /dev/sdb with the OS drive, just a cosmetic issue? Or is that something that I need to sort out before I start playing with LVM or parted/mdadm?

[darkod]

If you mean whether it's ok the OS disk to be sdb, yes it is. It doesn't matter at all. It doesn't have to be the first disk (sda).

And yes, LVM will let you combine any physical disks (devices) into any logical storage you desire. So, 3x 2TB disks will give you 6TB continuous storage.

In fact, with the raid1 you still had to use LVM, because raid1 mirror of 2x 2TB disks will give you 2TB device, and then the other 2x 2TB disks would give you again 2TB device. To combine that into continuous 4TB you need to use LVM (mdadm arrays can serve as physical devices in LVM terms).

Also, here is another idea to think about: You can think about whether to continue running the OS from the SD card or put it on the HDDs. You can "slice" a small part of them (like 16GB or 32GB) and create one raid1 with 4x members (all the disks) and use that as your root partition for the OS. That way even if one disk fails, the OS keeps running. In fact, even if three disks fail the OS will keep running. Of course, the data won't be there with so many disks failed, but the OS will be there.

The rest of the disks can be partition into one small swap partition (again on all disks same size to keep symmetry) and one big partition to use as you see fit. It can be two raid1 arrays combined into LVM, or 3x partition into LVM for a big data partition, and 1x for backup.

[JusticeEmpire]

Ok, I'll leave /dev/sdb as is for now, considering it is operational.

I'm happy to keep OS on the SD card, as from what I can gather, that is less likely to fail compared to a standard HDD. The idea of the 'box' is for a single place to store all my data, so it sounds like using LVM and mdadm is the way to go. I guess I need to decide on a structure and see how it goes. I've got around 1.5TB data to store at the moment, so it seem logical to me to have 2 x 4TB drives, one for the data dump and the other for backup/RAID.

Would I need to delete the array using the bios setup? Then disable the RAID controller and set to AHCI, and then look at LVM/mdadm?

Once I have everything in place, I think I will then prioritise the data, and backup elsewhere accordingly. I've attempted to use rsync before, so I think I would probably go with that again, but that would be a phase 2 mission.