zfs-fuse gone crazy, won't export, won't destroy pool

[mrluohua]

tl;dr zfs-fuse wont export nor destory my pool claiming 'busy', even when it's disks are used by mdadm raid!

So I had 4x3tb drives that I created a raidz pool on it for some temporary testing. When done, I was unable to export or destroy my pool

# zpool export -f rstoreb1
cannot export 'rstoreb1': pool is busy
# zpool destroy -f rstoreb1
cannot destroy 'rstoreb1': pool is busy

I can find nothing using it with lsof -n nor with ps -ef nor with fuser.

At this point I'm annoyed, and stop it with /etc/init.d/zfs-fuse stop

I partition the 4x3tb drives with parted, and create my raid with mdadm, and then create filesystem with mkfs.ext4, and mounted,

At this point mdadm is creating the raid, the new raid is mounted and I can copy files to it just fine.

I do a final reboot to make sure everything boots well and mounts well ... and low and behold ... zfs-fuse starts with boot, and mounts rstoreb1 again !

/dev/md127 7.9T 1.7G 7.9T 1% /extra/mdstore2
rstoreb1 8.1T 8.0T 89G 99% /extra/rstoreb1

these two raids are using the same hard drives !!

zpool stays it's healthy (i copied a 1gb file to the md127 raid device that shares the same disks with zfs raidz)

# zpool status
pool: rstoreb1
state: ONLINE
scrub: resilver completed after 0h0m with 0 errors on Sat Jan 5 17:55:15 2013
config:

NAME STATE READ WRITE CKSUM
rstoreb1 ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
disk/by-id/wwn-0x5000c5004e5454fe ONLINE 0 0 0
disk/by-id/ata-ST3000DM001-1CH166_Z1F16Q3S ONLINE 0 0 0 10K resilvered
disk/by-id/ata-ST3000DM001-1CH166_Z1F16QB0 ONLINE 0 0 0
disk/by-id/ata-ST3000DM001-1CH166_Z1F16QDX ONLINE 0 0 0

errors: No known data errors

at this point I am confused. I like fuse-zfs and I want to (need to) continue using it with my external e-sata drives but this internal set of disks I want mdadm, and zfs just won't let go.

How can I coax zfs to nicely let go of this volume?

# dpkg -l | grep zfs
ii zfs-fuse 0.6.9-1build1 ZFS on FUSE

I'm running Ubuntu 12.04.1 LTS.

note: I had uninstalled nfs-kernel-server at one point, and reading that it might be necessary I just reinstalled it (didn't start it), but I still can't destroy or export pool rstoreb1.

Any ideas? Thanks so much !!

mrluohua

edit: change zfs-kernel-server to nfs-kernel-server. There is no zfs-kernel-server, I typo'd it. I meant nfs.

[tgalati4]

How much RAM do you have?

free

Bad things happen in zfs when you are low on RAM.

[mrluohua]

about 1gb free

# free -m
total used free shared buffers cached
Mem: 4026 3040 986 0 103 2522
-/+ buffers/cache: 413 3613
Swap: 1906 0 1906

edit: formatting

[mrluohua]

I should add that I've rebooted 2x and stopped/started zfs-fuse multiple times.

[tgalati4]

I have zfs running on my freenas home server, and I recall reading that you need 1 GB per TB of disk space. You have 12 TB running on a 4 GB machine. Perhaps you should monitor the RAM to see what is going on.

Also, uninstalling and reinstalling a zfs-enabled kernel AND expecting your pools to continue to exist is expecting a lot. ZFS was developed under BSD. It has some neat features (like being 128-bit, which means HUGE file system sizes). I don't know how stable zfs is under linux.

It sounds like zfs has striped your internal drives, which might require a wipe to remove the striping to get zfs to release them. It's also possible that a bad interaction with mdadm and zfs on the same devices caused a partition table problem. The data is still there and the RAID is intact, but the pool architecture is messed up in a nontrivial way.

Running the system with and without the zfs kernel may have contributed to the problem. I don't know what else to say. Proceed carefully.

[mrluohua]

Running the system with and without the zfs kernel may have contributed to the problem. I don't know what else to say. Proceed carefully.

Sorry, I had a horrible typo, which I just corrected in my main post. I removed the "nfs-kernel-server" package... nfs, not zfs.

I read in some post somewhere that the zfs-fuse made use of the nfs-kernel-server which is why I re-installed it.

As an aside, the 4 x 3tb raidz 'temp' setup worked great for about 2 weeks. I was moving data around and I filled it all the way up, swapped hardware, and then moved my data back. Worked wonderfully.

It almost seems like a bug with the specific version of zfs that I have. All I want to do is export/destroy it. Why does it say 'busy'? ps / lsof / fuser tell me that it's not busy. Just confusing.

Anyway, thanks for your advice !!