I had the same problem. I dragged the layers dialogue outside the paths, channels, layers, undo window so it became its own seperate window and this solved the problem. I'm using Win7 so I don't know...