Page 1 of 2 12 LastLast
Results 1 to 10 of 13

Thread: Error when trying to log in using SSH: /bin/bash: No such file or directory

  1. #1
    Join Date
    Sep 2011
    Beans
    5

    Exclamation Error when trying to log in using SSH: /bin/bash: No such file or directory

    Hi all,

    I have a dedicated server running Ubuntu Server 12.04 64-bit. For a little more than a week now, I have been unable to log into it using SSH. The output I get is:

    Code:
    $ ssh xxxxxxx.com
    Welcome to Ubuntu 12.04 LTS (GNU/Linux 3.2.0-26-generic x86_64)
    
     * Documentation:  https://help.ubuntu.com/
    
      System information as of Sat Jul 14 21:49:44 CDT 2012
    
      System load:  0.46                Processes:           198
      Usage of /:   32.1% of 457.24GB   Users logged in:     0
      Memory usage: 64%                 IP address for eth0: xxx.xxx.xxx.xx
      Swap usage:   3%
    
      Graph this data and manage this system at https://landscape.canonical.com/
    
    1 package can be updated.
    0 updates are security updates.
    
    Last login: Sun Jul 22 11:42:17 2012 from xxx.xx.xxx.xx
    /bin/bash: No such file or directory
    Connection to xxxxxxx.com closed.
    Note how the "System information" hasn't been updated since Sat Jul 14 21:49:44 CDT 2012 (which is probably about when this problem started).

    The websites hosted on this server (using Apache, PHP and MySQL) are still running fine, and I can still do things on it with Virtualmin/Webmin/Usermin.

    Using Webmin's built-in Java-based file manager, I have determined that /bin/bash does indeed exist, and that its permissions are 755 (allowing reading and execution by all users) and that the permissions for the /bin directory are 6755 (allowing reading and listing by all users).

    I have also tried uploading the bash executable from my home desktop (also running Ubuntu 12.04 64-bit) to my home directory on the server and setting the shell to that by editing /etc/passwd, but that resulted in the same "No such file or directory" error for the bash path in my home directory.

    Attempting to have SSH execute a command on the server rather than a login shell also yields the same error:

    Code:
    $ ssh xxxxxxx.com pwd
    /bin/bash: No such file or directory
    Trying to log in as other users also didn't work.

    Frankly, I'm at a loss.

    If anyone has any ideas, please let me know! Thanks in advance.

    -BZ
    Last edited by BlueZenith; July 24th, 2012 at 06:09 PM.

  2. #2
    Join Date
    Feb 2009
    Location
    Dallas, TX
    Beans
    7,735
    Distro
    Ubuntu 16.04 Xenial Xerus

    Re: Error when trying to log in using SSH: /bin/bash: No such file or directory

    Hi BlueZenith. Welcome to the forums

    It looks like you tried to do chroot configuration, but something went wrong.

    Could you post the content of this file?
    Code:
    /etc/ssh/sshd_config
    Regards.

  3. #3
    Join Date
    Sep 2011
    Beans
    5

    Re: Error when trying to log in using SSH: /bin/bash: No such file or directory

    Quote Originally Posted by papibe View Post
    Hi BlueZenith. Welcome to the forums
    Thank you papibe, and thanks for replying

    Quote Originally Posted by papibe View Post
    It looks like you tried to do chroot configuration, but something went wrong.

    Could you post the content of this file?
    Code:
    /etc/ssh/sshd_config
    I can imagine that it looks that way, but I've never used chroot before in my life.

    These are the contents of /etc/ssh/sshd_config on the server:
    Code:
    # Package generated configuration file
    # See the sshd_config(5) manpage for details
    
    # What ports, IPs and protocols we listen for
    Port 22
    # Use these options to restrict which interfaces/protocols sshd will bind to
    #ListenAddress ::
    #ListenAddress 0.0.0.0
    Protocol 2
    # HostKeys for protocol version 2
    HostKey /etc/ssh/ssh_host_rsa_key
    HostKey /etc/ssh/ssh_host_dsa_key
    HostKey /etc/ssh/ssh_host_ecdsa_key
    #Privilege Separation is turned on for security
    UsePrivilegeSeparation yes
    
    # Lifetime and size of ephemeral version 1 server key
    KeyRegenerationInterval 3600
    ServerKeyBits 768
    
    # Logging
    SyslogFacility AUTH
    LogLevel INFO
    
    # Authentication:
    LoginGraceTime 120
    PermitRootLogin yes
    StrictModes yes
    
    RSAAuthentication yes
    PubkeyAuthentication yes
    #AuthorizedKeysFile	%h/.ssh/authorized_keys
    
    # Don't read the user's ~/.rhosts and ~/.shosts files
    IgnoreRhosts yes
    # For this to work you will also need host keys in /etc/ssh_known_hosts
    RhostsRSAAuthentication no
    # similar for protocol version 2
    HostbasedAuthentication no
    # Uncomment if you don't trust ~/.ssh/known_hosts for RhostsRSAAuthentication
    #IgnoreUserKnownHosts yes
    
    # To enable empty passwords, change to yes (NOT RECOMMENDED)
    PermitEmptyPasswords no
    
    # Change to yes to enable challenge-response passwords (beware issues with
    # some PAM modules and threads)
    ChallengeResponseAuthentication no
    
    # Change to no to disable tunnelled clear text passwords
    #PasswordAuthentication yes
    
    # Kerberos options
    #KerberosAuthentication no
    #KerberosGetAFSToken no
    #KerberosOrLocalPasswd yes
    #KerberosTicketCleanup yes
    
    # GSSAPI options
    #GSSAPIAuthentication no
    #GSSAPICleanupCredentials yes
    
    X11Forwarding yes
    X11DisplayOffset 10
    PrintMotd no
    PrintLastLog yes
    TCPKeepAlive yes
    #UseLogin no
    
    #MaxStartups 10:30:60
    #Banner /etc/issue.net
    
    # Allow client to pass locale environment variables
    AcceptEnv LANG LC_*
    
    Subsystem sftp /usr/lib/openssh/sftp-server
    
    # Set this to 'yes' to enable PAM authentication, account processing,
    # and session processing. If this is enabled, PAM authentication will
    # be allowed through the ChallengeResponseAuthentication and
    # PasswordAuthentication.  Depending on your PAM configuration,
    # PAM authentication via ChallengeResponseAuthentication may bypass
    # the setting of "PermitRootLogin without-password".
    # If you just want the PAM account and session checks to run without
    # PAM authentication, then enable this but set PasswordAuthentication
    # and ChallengeResponseAuthentication to 'no'.
    UsePAM yes

  4. #4
    Join Date
    Sep 2011
    Beans
    5

    Re: Error when trying to log in using SSH: /bin/bash: No such file or directory

    This has just got even more confusing, I think.

    Setting /bin/bash as shell for my account, I get the following when I connect through SSH (omitting the system information and disconnect message which are the same every time):

    Code:
    /bin/bash: No such file or directory
    Similarly, with /bin/dash set as shell, I get:

    Code:
    /bin/dash: No such file or directory
    However, setting /bin/busybox as shell results in BusyBox's normal output when connecting through SSH:

    Code:
    BusyBox v1.18.5 (Ubuntu 1:1.18.5-1ubuntu4) multi-call binary.
    Copyright (C) 1998-2009 Erik Andersen, Rob Landley, Denys Vlasenko
    and others. Licensed under GPLv2.
    See source distribution for full notice.
    
    Usage: busybox [function] [arguments]...
       or: busybox --list[-full]
       or: function [arguments]...
    
        BusyBox is a multi-call binary that combines many common Unix
        utilities into a single executable.  Most people will create a
        link to busybox for each function they wish to use and BusyBox
        will act like whatever it was invoked as.
    
    Currently defined functions:
        [, [[, acpid, addgroup, adduser, adjtimex, ar, arping, ash,
        awk, basename, blockdev, brctl, bunzip2, bzcat, bzip2, cal,
        cat, chgrp, chmod, chown, chroot, chvt, clear, cmp, cp, cpio,
        crond, crontab, cut, date, dc, dd, deallocvt, delgroup,
        deluser, df, diff, dirname, dmesg, dnsdomainname, dos2unix,
        dpkg, dpkg-deb, du, dumpkmap, dumpleases, echo, ed, egrep,
        eject, env, expand, expr, false, fbset, fdflush, fdisk, fgrep,
        find, fold, free, freeramdisk, fsck.minix, ftpget, ftpput,
        getopt, getty, grep, gunzip, gzip, head, hexdump, hostid,
        hostname, httpd, hwclock, id, ifconfig, ifdown, ifup, init,
        ionice, ip, ipcalc, kill, killall, klogd, last, length, less,
        linuxrc, ln, loadfont, loadkmap, logger, login, logname,
        logread, losetup, ls, lzcat, lzma, makedevs, md5sum, mdev,
        mesg, microcom, mkdir, mkfifo, mkfs.minix, mknod, mkswap,
        mktemp, more, mount, mt, mv, nameif, nc, netstat, nslookup,
        od, openvt, passwd, patch, pidof, ping, ping6, pivot_root,
        printf, ps, pwd, rdate, readlink, realpath, renice, reset,
        rev, rm, rmdir, route, rpm, rpm2cpio, run-parts, sed, seq,
        setkeycodes, sh, sha1sum, sha256sum, sha512sum, sleep, sort,
        start-stop-daemon, static-sh, strings, stty, su, sulogin,
        swapoff, swapon, switch_root, sync, sysctl, syslogd, tac,
        tail, tar, tee, telnet, telnetd, test, tftp, time, timeout,
        top, touch, tr, traceroute, traceroute6, true, tty, tunctl,
        udhcpc, udhcpd, umount, uname, uncompress, unexpand, uniq,
        unix2dos, unlzma, unxz, unzip, uptime, usleep, uudecode,
        uuencode, vconfig, vi, vlock, watch, watchdog, wc, wget,
        which, who, whoami, xargs, xz, xzcat, yes, zcat
    And symbolically linking /bin/sh to /bin/busybox and then setting /bin/sh as the shell results in a working BusyBox shell:

    Code:
    BusyBox v1.18.5 (Ubuntu 1:1.18.5-1ubuntu4) built-in shell (ash)
    Enter 'help' for a list of built-in commands.
    
    $
    The built-in BusyBox commands (like ls and which) work fine, but I have so far been unable to find any executable that will run, except for the BusyBox executable itself:

    Code:
    $ which bash
    /bin/bash
    $ ls -l /bin/bash
    -rwxr-xr-x    1 root     root        955024 Apr  3 10:58 /bin/bash
    $ /bin/bash
    -sh: /bin/bash: not found
    Code:
    $ which ls
    /bin/ls
    $ ls -l /bin/ls
    -rwxr-xr-x    1 root     root        105840 Mar 31 22:09 /bin/ls
    $ /bin/ls
    -sh: /bin/ls: not found
    Code:
    $ which mysql
    /usr/bin/mysql
    $ ls -l /usr/bin/mysql
    -rwxr-xr-x    1 root     root       3499664 Jun 21 18:29 /usr/bin/mysql
    $ /usr/bin/mysql
    -sh: /usr/bin/mysql: not found
    Code:
    $ which busybox
    /bin/busybox
    $ ls -l /bin/busybox
    -rwxr-xr-x    1 root     root       1827920 Apr 13 14:53 /bin/busybox
    $ /bin/busybox
    BusyBox v1.18.5 (Ubuntu 1:1.18.5-1ubuntu4) multi-call binary.
    Copyright (C) 1998-2009 Erik Andersen, Rob Landley, Denys Vlasenko
    and others. Licensed under GPLv2.
    See source distribution for full notice.
    
    Usage: busybox [function] [arguments]...
       or: busybox --list[-full]
       or: function [arguments]...
    
        BusyBox is a multi-call binary that combines many common Unix
        utilities into a single executable.  Most people will create a
        link to busybox for each function they wish to use and BusyBox
        will act like whatever it was invoked as.
    
    Currently defined functions:
        [, [[, acpid, addgroup, adduser, adjtimex, ar, arping, ash,
        awk, basename, blockdev, brctl, bunzip2, bzcat, bzip2, cal,
        cat, chgrp, chmod, chown, chroot, chvt, clear, cmp, cp, cpio,
        crond, crontab, cut, date, dc, dd, deallocvt, delgroup,
        deluser, df, diff, dirname, dmesg, dnsdomainname, dos2unix,
        dpkg, dpkg-deb, du, dumpkmap, dumpleases, echo, ed, egrep,
        eject, env, expand, expr, false, fbset, fdflush, fdisk, fgrep,
        find, fold, free, freeramdisk, fsck.minix, ftpget, ftpput,
        getopt, getty, grep, gunzip, gzip, head, hexdump, hostid,
        hostname, httpd, hwclock, id, ifconfig, ifdown, ifup, init,
        ionice, ip, ipcalc, kill, killall, klogd, last, length, less,
        linuxrc, ln, loadfont, loadkmap, logger, login, logname,
        logread, losetup, ls, lzcat, lzma, makedevs, md5sum, mdev,
        mesg, microcom, mkdir, mkfifo, mkfs.minix, mknod, mkswap,
        mktemp, more, mount, mt, mv, nameif, nc, netstat, nslookup,
        od, openvt, passwd, patch, pidof, ping, ping6, pivot_root,
        printf, ps, pwd, rdate, readlink, realpath, renice, reset,
        rev, rm, rmdir, route, rpm, rpm2cpio, run-parts, sed, seq,
        setkeycodes, sh, sha1sum, sha256sum, sha512sum, sleep, sort,
        start-stop-daemon, static-sh, strings, stty, su, sulogin,
        swapoff, swapon, switch_root, sync, sysctl, syslogd, tac,
        tail, tar, tee, telnet, telnetd, test, tftp, time, timeout,
        top, touch, tr, traceroute, traceroute6, true, tty, tunctl,
        udhcpc, udhcpd, umount, uname, uncompress, unexpand, uniq,
        unix2dos, unlzma, unxz, unzip, uptime, usleep, uudecode,
        uuencode, vconfig, vi, vlock, watch, watchdog, wc, wget,
        which, who, whoami, xargs, xz, xzcat, yes, zcat
    So BusyBox and Webmin can see these executables and show that they have the correct permissions, but when they are supposed to get executed, they somehow can't be found — except busybox, which must be special in some way that I can't identify.

    I wondered what would happen if I tried to execute one of these (e.g., bash) without the executable permission set. This was the result:

    Code:
    $ cp /bin/bash .
    $ ./bash
    -sh: ./bash: not found
    $ chmod -x bash
    $ ./bash
    -sh: ./bash: Permission denied
    $ chmod +x bash
    $ ./bash
    -sh: ./bash: not found
    The problem doesn't just affect SSH, as shown by the mail error log (/var/log/mail.err), the last 10 lines of which are:

    Code:
    Jul 24 11:20:56 chicago master[2156]: fatal: master_spawn: exec /usr/lib/postfix/smtpd: No such file or directory
    Jul 24 11:20:56 chicago master[2157]: fatal: master_spawn: exec /usr/lib/postfix/pickup: No such file or directory
    Jul 24 11:21:57 chicago master[2196]: fatal: master_spawn: exec /usr/lib/postfix/smtpd: No such file or directory
    Jul 24 11:21:57 chicago master[2197]: fatal: master_spawn: exec /usr/lib/postfix/pickup: No such file or directory
    Jul 24 11:22:58 chicago master[2274]: fatal: master_spawn: exec /usr/lib/postfix/smtpd: No such file or directory
    Jul 24 11:22:58 chicago master[2275]: fatal: master_spawn: exec /usr/lib/postfix/pickup: No such file or directory
    Jul 24 11:23:59 chicago master[2672]: fatal: master_spawn: exec /usr/lib/postfix/smtpd: No such file or directory
    Jul 24 11:23:59 chicago master[2673]: fatal: master_spawn: exec /usr/lib/postfix/pickup: No such file or directory
    Jul 24 11:25:00 chicago master[2779]: fatal: master_spawn: exec /usr/lib/postfix/smtpd: No such file or directory
    Jul 24 11:25:00 chicago master[2780]: fatal: master_spawn: exec /usr/lib/postfix/pickup: No such file or directory
    (Like the other executables mentioned before, /usr/lib/postfix/smtpd and /usr/lib/postfix/pickup do in fact exist and have the appropriate permissions.)

    I am not aware of any change I have made that could have anything to do with this problem or was correlated with its beginning.

    The package-manager log shows that the most recent package actions were upgrades of OpenJDK 6 and IcedTea 6, which happened days before the problem first occurred.

    I'm scared to restart the machine for fear that it won't properly boot at all with whatever problem is going on. My websites (one of which is absolutely mission-critical) are fortunately still running fine, but things clearly can't keep going on like this.

    Any ideas, anyone?
    Last edited by BlueZenith; July 24th, 2012 at 06:01 PM.

  5. #5
    Join Date
    Aug 2010
    Location
    Lancs, United Kingdom
    Beans
    1,308
    Distro
    Ubuntu Mate 16.04 Xenial Xerus

    Re: Error when trying to log in using SSH: /bin/bash: No such file or directory

    This is a bit of a guess. busybox is statically linked whereas bash, dash and other similar programs are dynamically linked. So it could be that some fundamental library has gone missing.

    Does
    Code:
    ldd /bin/bash
    give any clues? (Assuming ldd works!)

  6. #6
    Join Date
    May 2010
    Location
    uk
    Beans
    9,324
    Distro
    Xubuntu 14.04 Trusty Tahr

    Re: Error when trying to log in using SSH: /bin/bash: No such file or directory

    Hi
    So BusyBox ... can see these executables and show that they have the correct permissions, but when they are supposed to get executed, they somehow can't be found — except busybox, which must be special in some way that I can't identify.
    This is not actually how busy bux works.

    BusyBox is a multi-call binary that combines many common Unix utilities into a single executable
    Busybox contains all the commands to run (ls, chown, chmod....) within the executable itself.

    Busybox knows which commands to run because, usually, each call in busybox is a symlink. Obviously at the busybox command prompt each call into busybox will also work.

    Internally, busybox will read the name of the symlink that called into it and, therefore, know which command to run. ie. which function within the busybox executable to run.

    Your problem is not busybox being able to run the executables. The busybox is one exectuable containingt functions that implement the commands.

    Check your $PATH.
    Check PAM.
    Check out ldd as posted above.
    Commands will not run if libc cannot be loaded by an executable. (check permissions)
    Check you library paths are accessible.
    Check the libc symbolic links are intact.

    EDIT: To make the above commentry on busybox clearer, read this...

    (Ubuntu is on a different partition than the one i am currently using.....)

    Code:
    [matthew@matthew-aspire7450]~% ln -s /mnt/bin/busybox ls
    [matthew@matthew-aspire7450]~% ls -l ~/ls
    lrwxrwxrwx. 1 matthew matthew 16 Jul 24 18:31 /home/matthew/ls -> /mnt/bin/busybox
    [matthew@matthew-aspire7450]~% ~/ls
    Desktop              Dropbox              Public               index.html           my_storage           sshfs_media          test
    Documents            Music                Templates            ls                   my_virtual_machines  storage
    Downloads            Pictures             Videos               my_documents         sshfs_home           t
    [matthew@matthew-aspire7450]~%
    Kind regards
    Last edited by matt_symes; July 24th, 2012 at 07:15 PM. Reason: Added another couple of things to check. Redited to state the obvious.
    If you believe everything you read, you better not read. ~ Japanese Proverb

    If you don't read the newspaper, you're uninformed. If you read the newspaper, you're mis-informed. - Mark Twain

    Thinking about becoming an Ubuntu Member?

  7. #7
    Join Date
    Sep 2011
    Beans
    5

    Re: Error when trying to log in using SSH: /bin/bash: No such file or directory

    Thank you both for your advice! I've got it back to working.

    I tried spjackson's suggestion, which didn't work on the server:
    Code:
    $ ldd
    -sh: ldd: not found
    (presumably because ldd itself has dynamically linked dependencies), but running it locally on my home desktop (which is set up mostly the same way) gave four file paths:

    Code:
    $ ldd /bin/bash
        linux-vdso.so.1 =>  (0x00007fff2a7ff000)
        libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x00007f2754708000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f2754504000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f2754146000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f2754953000)
    The first three were present on the server, but the last one was not. It turned out that the lib64 directory (as well as the lib32 directory) that was present on my home system didn't exist on the server. So I tarred my local lib32 and lib64 directories and uploaded/unpacked them using Webmin's file manager, and now everything seems to be back to normal.

    I'd sure like to know how those directories got deleted (or the lib64 one at least — I copied the lib32 one along just in case).

    But more than that, I find it puzzling that the error messages seemed to suggest that the executables themselves could not be found, rather than mentioning the library that was actually missing. A message like
    Code:
    /bin/bash: Missing dynamic library: /lib64/ld-linux-x86-64.so.2
    would have been infinitely more useful than
    Code:
    /bin/bash: No such file or  directory
    or
    Code:
    fatal: master_spawn: exec /usr/lib/postfix/smtpd: No such file or directory
    and would have saved many hours of troubleshooting in this case.

    Would it be reasonable to suggest this as an idea for improvement somewhere?
    Last edited by BlueZenith; July 24th, 2012 at 07:39 PM. Reason: Grammar

  8. #8
    Join Date
    Feb 2009
    Location
    Dallas, TX
    Beans
    7,735
    Distro
    Ubuntu 16.04 Xenial Xerus

    Re: Error when trying to log in using SSH: /bin/bash: No such file or directory

    May be a missconfigured or corrupted libc6 ?

    Could you post the result of these commands?
    Code:
    apt-cache policy libc6
    
    dpkg -L libc6
    Regards.

  9. #9
    Join Date
    May 2010
    Location
    uk
    Beans
    9,324
    Distro
    Xubuntu 14.04 Trusty Tahr

    Re: Error when trying to log in using SSH: /bin/bash: No such file or directory

    Hi

    But more than that, I find it puzzling that the error messages seemed to suggest that the executables themselves could not be found, rather than mentioning the library that was actually missing
    It means that either the executable cannot be found or a dependant dynamically loaded library. (ofcourse ther are other reasons for it as well)

    Maybe not the best error message in the world

    Kind regards
    Last edited by matt_symes; July 24th, 2012 at 09:18 PM.
    If you believe everything you read, you better not read. ~ Japanese Proverb

    If you don't read the newspaper, you're uninformed. If you read the newspaper, you're mis-informed. - Mark Twain

    Thinking about becoming an Ubuntu Member?

  10. #10
    Join Date
    May 2010
    Location
    uk
    Beans
    9,324
    Distro
    Xubuntu 14.04 Trusty Tahr

    Re: Error when trying to log in using SSH: /bin/bash: No such file or directory

    Hi

    Just wanted to say...

    Nice work spjackson. Good call !!

    Kind regards
    If you believe everything you read, you better not read. ~ Japanese Proverb

    If you don't read the newspaper, you're uninformed. If you read the newspaper, you're mis-informed. - Mark Twain

    Thinking about becoming an Ubuntu Member?

Page 1 of 2 12 LastLast

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •