Transparent multi-hop SSH

Connecting from outside/home via a proxy server a.k.a. jump host

Most of our servers can only be accessed using a jumphost (a.k.a. proxy) server. This requires port 22 to be open on all machines between your computer and our jumphost server.
The jumphost for the Calculon cluster is: lobby.hpc.rug.nl
Its ED25519 key fingerprint is: SHA256:YBt5nyd2ragxOXmIoQ+2kkoCjBsbdFWbdpN2PfqxKCs

Update 2021-08-04

• The old lobby.hpc.rug.nl jumphost was replaced with a new one with the same name.
• You can skip this update section if you never used the old jumphost.
• It may take some time for the change to propagate to all corners of the internet Domain Name System (DNS): if the name lobby.hpc.rug.nl resolves to IP address 195.169.22.135, you are connecting to the new lobby.hpc.rug.nl
• Previously there was also a jumphost named foyer.hpc.rug.nl, but this one has been deprecated.
• Furthermore the old example config listed many other machines like flexo and bender: Please remove all entries for machines that could be accessed via the old lobby or foyer jumphosts as well as the entries for these jumphosts themselves from your ~/.ssh/config file:
• The only line that should remain in your ~/.ssh/config for the UMCG Research HPC machines that must be accessed via a jumphost is:

  Include conf.d/*
  

• You should keep lines for machines that are not accessed via our jumphost. E.g. lines for the cher-ami data staging server, for the RUG peregrine cluster or for machines from other sites/services.
• Additional lines that are required to access UMCG Research HPC machines are added to separate config files inside the ~/.ssh/conf.d/ directory. E.g. a ~/.ssh/conf.d/lobby file for machines accessed via lobby.hpc.rug.nl.
• If you previously used the old lobby.hpc.rug.nl, you may get a warning that the host key (fingerprint) has changed and you may not be able to login until you remove the old host key from your list of known hosts.

Users on Linux / Unix / Mac OS X

To setup transparent multi-hop SSH:

• On the machine from which you want to connect
  • Make sure you have a ~/.ssh folder with generated SSH keys linked to your account. See RequestAccount? for instructions.
  • Create ~/.ssh/tmp and ~/.ssh/conf.d subfolders and configuration files if they did not already exist and make sure they have the right permissions. Open a terminal and type the following command:

    mkdir -p -m 700 "${HOME}/.ssh/"
    mkdir -p -m 700 "${HOME}/.ssh/tmp/"
    mkdir -p -m 700 "${HOME}/.ssh/conf.d/"
    touch "${HOME}/.ssh/config"
    touch "${HOME}/.ssh/conf.d/lobby"
    chmod -R go-rwx "${HOME}/.ssh"
    

  • Add the following line to your ~/.ssh/config:

    Include conf.d/*
    

    Important: this Include directive must precede any lines containing Host or Match directives, otherwise the Include will only apply to a specific set of hosts.
    
    
  • Add the following block to your ~/.ssh/conf.d/lobby:

    #
    # Generic stuff: only for MacOS clients.
    #
    IgnoreUnknown UseKeychain
    	UseKeychain yes
    IgnoreUnknown AddKeysToAgent
    	AddKeysToAgent yes
    
    #
    # Host settings.
    #
    Host lobby*
        #
        # Default account name when not specified explicitly.
        #
        User youraccount
        #
        # Prevent timeouts
        #
        ServerAliveInterval 60
        ServerAliveCountMax 5
        #
        # We use public-private key pairs for authentication.
        # Optionally: specify the path to your RSA private key it is not in the default location.
        # Do not use password based authentication as fallback,
        # which may be confusing and won't work anyway.
        #
        IdentityFile "~/.ssh/id_rsa"
        PasswordAuthentication No
        #
        # Multiplex connections to
        #   * reduce lag when logging in to the same host in a second terminal
        #   * reduce the amount of connections that are made to prevent excessive DNS lookups
        #     and to prevent getting blocked by a firewall, because it thinks we are executing a DoS attack.
        #
        # Name/location of sockets for connection multiplexing are configured using the ControlPath directive.
        # In the ControlPath directive %C expands to a hashed value of %l_%h_%p_%r, where:
        #    %l = local hostname
        #    %h = remote hostname
        #    %p = remote port
        #    %r = remote username
        # This makes sure that the ControlPath is
        #   * a unique socket that is local to machine on which the sessions are created,
        #     which means it works with home dirs from a shared network file system.
        #     (as sockets cannot be shared by servers.)
        #   * not getting to long as the hash has a fixed size no matter how long %l_%h_%p_%r was.
        #
        ControlMaster auto
        ControlPath ~/.ssh/tmp/%C
        ControlPersist 1m
    #
    # Expand short jumphost names to FQDN or IP address.
    #
    Host lobby !*.hpc.rug.nl
        HostName %h.hpc.rug.nl
    #
    # Universal jumphost settings for triple-hop SSH.
    #
    Host *+*+*
        ProxyCommand ssh -x -q $(echo %h | sed 's/+[^+]*$//') -W $(echo %h | sed 's/^[^+]*+[^+]*+//'):%p
    #
    # Double-hop SSH settings to connect via specific jumphosts.
    #
    Host lobby+* 
        ProxyCommand ssh -x -q $(echo "${JUMPHOST_USER:-%r}")@$(echo %h | sed 's/+[^+]*$//').hpc.rug.nl -W $(echo %h | sed 's/^[^+]*+//'):%p
    #
    # Sometimes port 22 for the SSH protocol is blocked by firewalls; in that case you can try to use SSH on port 443 as fall-back.
    # Do not use port 443 by default for SSH as it is officially assigned to HTTPS traffic
    # and some firewalls will cause problems with SSH traffic over port 443.
    #
    Host lobby443+* 
        ProxyCommand ssh -x -q $(echo "${JUMPHOST_USER:-%r}")@$(echo %h | sed 's/443+[^+]*$//').hpc.rug.nl -W $(echo %h | sed 's/^[^+]*+//'):%p -p 443
    

    Replace youraccount with the accountname you received from the UMCG HPC helpdesk.
    
    If you are not on a Mac or on a very old one you may have to comment the # Generic stuff: only for MacOS clients section at the top of the example ~/.ssh/conf.d/lobby
    
    
• You can now for example connect to the User Interface of the Calculon cluster named calculon with the account as specified in the User directive of your ~/.ssh/conf.d/lobby file via the lobby jumphost using the alias lobby+calculon. Type the following command in a terminal:

  ssh lobby+calculon
  

  In order to override the account name specified in your ~/.ssh/conf.d/lobby you can use:

  ssh youraccount@lobby+calculon
  

  You can also transfer data with scp (secure copy) to copy files to your home dir on the cluster like this:

  scp some_file lobby+calculon:
  

  or even better use rsync to copy data to your home dir on the cluster like this:

  rsync -av some_directory lobby+calculon:
  

  Note the colon at the end of the scp and rsync commands. Without the colon you would copy to a local file named lobby+calculon. You can add additional jumphost+* aliases for other User Interface (UI) servers depending on your computing preferences or authorization. If you also added the Host *+*+* code block you can do tripple hops starting with a jumphost like this:

  ssh jumphost+intermediate_server+destination_server
  

  In case you are on a network where the default port for SSH (22) is blocked by a firewall you can try to setup SSH over port 443 using an alias like this:

  ssh lobby443+calculon
  

Frequent Asked Questions:

• Q: Why do I get the error muxserver_listen bind(): No such file or directory.?
  A: You may have failed to create the ~/.ssh/tmp folder in the first step or the permissions on this folder are wrong.
  
  
• Q: Why do I get the error ControlPath too long?
  A: The ControlPath ~/.ssh/tmp/%C line in your ~/.ssh/conf.d/lobby file expands to a path that is too long. Change the ControlPath line in your ~/.ssh/conf.d/lobby file to create a shorter path for the automagically created sockets. You can use for example ControlPath /tmp/%h_%p_%r to create sockets in /tmp.
  
  
• Q: Why do I get the error nc: getaddrinfo: Name or service not known. ssh_exchange_identification: Connection closed by remote host?
  A: This server does not exist (anymore); You have a typo in the name of the server you are trying to connect to. Check both the command you typed as well as your ~/.ssh/conf.d/lobby for typos in server names.
  
  
• Q: Why do I get the error Permission denied (publickey).?
  A: This error can be caused by various configuration issues:
  

Either you are using the wrong account name
or you are using the wrong private key file
or the permissions on your home ~/.ssh/ dir and/or on its content are wrong
or your account is misconfigured on our account server.

Check your account name, private key and permissions. If that did not resolve the issue, then increase the verbosity to debug connection problems (see below).

• Q: Can I increase the verbosity to debug connection problems?
  A: Yes try adding -vvv like this

ssh -vvv prefix-youraccount@lobby+calculon

If that does not help to figure out what is wrong please send an email to the HPC helpdesk and include:

1 The command you used for your failed login attempt
2 The output of that failed login attempt with -vvv debugging enabled
3 A copy of both your ~/.ssh/config and ~/.ssh/conf.d/lobby files.

(Never ever send us your private key; It does not help to debug your connection problems, but will render the key useless as it is no longer private.)

Users on Windows

• On the machine from which you want to connect
  1. Make sure you have generated SSH keys linked to your account. See RequestAccount? for instructions.
  2. Download and install ​Putty. Make sure to also download the key manager Pageant.
  3. Start Pageant
  4. Load your private key into Pageant
  5. Start Putty
  6. Go to Connection -> Ssh -> Auth and select Allow agent forwarding
  7. Go to Connection -> Data and fill in your user name in the auto-login username option

• You can now connect to for example calculon via for example jumphost server lobby.hpc.rug.nl using a double hop like this
  • In Putty connect to lobby.hpc.rug.nl
  • Once the connection is established type the following command in a terminal:

    ssh youraccount@calculon
    

    Change the name of the destination host to connect to other User Interface (UI) servers.

On order to automate this double hop on Windows please have a look at these instructions: ​http://mikelococo.com/2008/01/multihop-ssh/