authorized_keys needs to be world readable, because it is owned by the user, so that's fine.
As far as the delay, is that between the time you enter the username and password or before the logon prompt?
I've had a delay after login while processes are running, but if the delay is priory to actually logging in, it might be either slow name resolution or a slow network.
Bookmarks