I guess I'm not going to worry about it. If I have to delete that directory manually I will. It's hardly worth doing a clean install.
Just glad I can use this command sudo rm -r -v /lib/modules/3.2.0-64-generic/kernel/drivers/char because it complains about deleting a folder with something in it otherwise.
I do everything though CLI so I can see if the folder is not deleted when the kernel is.
Bookmarks