= Transparent multi-hop SSH=

== Connecting from outside/home via a proxy server a.k.a. jump host ==

[[TOC()]]

Most of our servers can only be accessed using a jumphost (a.k.a. proxy) server. This requires port 22 to be open on all machines between your computer and our jumphost server. [[BR]]
The jumphost for the Calculon cluster is: {{{lobby.hpc.rug.nl}}} [[BR]]
Its ED25519 key fingerprint is: {{{SHA256:YBt5nyd2ragxOXmIoQ+2kkoCjBsbdFWbdpN2PfqxKCs}}}

=== Update 2021-08-04 ===
 * The old {{{lobby.hpc.rug.nl}}} jumphost was replaced with a new one with the same name.
 * You can skip this update section if you never used the old jumphost.
 * It may take some time for the change to propagate to all corners of the internet Domain Name System (DNS): if the name {{{lobby.hpc.rug.nl}}} resolves to IP address 195.169.22.135, you are connecting to the new {{{lobby.hpc.rug.nl}}}
 * Previously there was also a jumphost named {{{foyer.hpc.rug.nl}}}, but this one has been deprecated.
 * Furthermore the old example config listed many other machines like {{{flexo}}} and {{{bender}}}: Please remove all entries for machines that could be accessed via the old ''lobby'' or ''foyer'' jumphosts as well as the entries for these jumphosts themselves from your {{{~/.ssh/config}}} file:
 * The only line that should remain in your {{{~/.ssh/config}}} for the UMCG Research HPC machines that must be accessed via a jumphost is:
   {{{
   Include conf.d/*
   }}}
 * You should keep lines for machines that are not accessed via our jumphost. E.g. lines for the {{{cher-ami}}} data staging server, for the RUG {{{peregrine}}} cluster or for machines from other sites/services.
 * Additional lines that are required to access UMCG Research HPC machines are added to separate config files inside the {{{~/.ssh/conf.d/}}} directory. E.g. a {{{~/.ssh/conf.d/lobby}}} file for machines accessed via {{{lobby.hpc.rug.nl}}}.
 * If you previously used the old {{{lobby.hpc.rug.nl}}}, you may get a warning that the host key (fingerprint) has changed and you may not be able to login until you **''remove the old host key from your list of known hosts''**.

== Users on Linux / Unix / Mac OS X ==

To setup transparent multi-hop SSH:
 * On the machine from which you want to connect
   * Make sure you have a **''~/.ssh''** folder with generated SSH keys linked to your account. See [wiki:RequestAccount] for instructions.
   * Create **''~/.ssh/tmp''** and **''~/.ssh/conf.d''** subfolders and configuration files if they did not already exist and make sure they have the right permissions. Open a terminal and type the following command:
{{{
mkdir -p -m 700 "${HOME}/.ssh/"
mkdir -p -m 700 "${HOME}/.ssh/tmp/"
mkdir -p -m 700 "${HOME}/.ssh/conf.d/"
touch "${HOME}/.ssh/config"
touch "${HOME}/.ssh/conf.d/lobby"
chmod -R go-rwx "${HOME}/.ssh"
}}}
   * Add the following line to your **''~/.ssh/config''**:
{{{
Include conf.d/*
}}}
     Important: this **''Include''** directive must precede any lines containing **''Host''** or **''Match''** directives, otherwise the **''Include''** will only apply to a specific set of hosts.[[BR]][[BR]]
   * Add the following block to your **''~/.ssh/conf.d/lobby''**:
{{{
#
# Generic stuff: only for MacOS clients.
#
IgnoreUnknown UseKeychain
	UseKeychain yes
IgnoreUnknown AddKeysToAgent
	AddKeysToAgent yes

#
# Host settings.
#
Host lobby*
    #
    # Default account name when not specified explicitly.
    #
    User youraccount
    #
    # Prevent timeouts
    #
    ServerAliveInterval 60
    ServerAliveCountMax 5
    #
    # We use public-private key pairs for authentication.
    # Optionally: specify the path to your RSA private key it is not in the default location.
    # Do not use password based authentication as fallback,
    # which may be confusing and won't work anyway.
    #
    IdentityFile "~/.ssh/id_rsa"
    PasswordAuthentication No
    #
    # Multiplex connections to
    #   * reduce lag when logging in to the same host in a second terminal
    #   * reduce the amount of connections that are made to prevent excessive DNS lookups
    #     and to prevent getting blocked by a firewall, because it thinks we are executing a DoS attack.
    #
    # Name/location of sockets for connection multiplexing are configured using the ControlPath directive.
    # In the ControlPath directive %C expands to a hashed value of %l_%h_%p_%r, where:
    #    %l = local hostname
    #    %h = remote hostname
    #    %p = remote port
    #    %r = remote username
    # This makes sure that the ControlPath is
    #   * a unique socket that is local to machine on which the sessions are created,
    #     which means it works with home dirs from a shared network file system.
    #     (as sockets cannot be shared by servers.)
    #   * not getting to long as the hash has a fixed size no matter how long %l_%h_%p_%r was.
    #
    ControlMaster auto
    ControlPath ~/.ssh/tmp/%C
    ControlPersist 1m
#
# Expand short jumphost names to FQDN or IP address.
#
Host lobby !*.hpc.rug.nl
    HostName %h.hpc.rug.nl
#
# Universal jumphost settings for triple-hop SSH.
#
Host *+*+*
    ProxyCommand ssh -x -q $(echo %h | sed 's/+[^+]*$//') -W $(echo %h | sed 's/^[^+]*+[^+]*+//'):%p
#
# Double-hop SSH settings to connect via specific jumphosts.
#
Host lobby+* 
    ProxyCommand ssh -x -q $(echo "${JUMPHOST_USER:-%r}")@$(echo %h | sed 's/+[^+]*$//').hpc.rug.nl -W $(echo %h | sed 's/^[^+]*+//'):%p
#
# Sometimes port 22 for the SSH protocol is blocked by firewalls; in that case you can try to use SSH on port 443 as fall-back.
# Do not use port 443 by default for SSH as it is officially assigned to HTTPS traffic
# and some firewalls will cause problems with SSH traffic over port 443.
#
Host lobby443+* 
    ProxyCommand ssh -x -q $(echo "${JUMPHOST_USER:-%r}")@$(echo %h | sed 's/443+[^+]*$//').hpc.rug.nl -W $(echo %h | sed 's/^[^+]*+//'):%p -p 443
}}}
   Replace '''youraccount''' with the accountname you received from the UMCG HPC helpdesk.[[BR]][[BR]]
   If you are **not** on a Mac or on a very old one you may have to comment the {{{# Generic stuff: only for MacOS clients}}} section at the top of the example **''~/.ssh/conf.d/lobby''**[[BR]][[BR]]
 * You can now for example connect to the User Interface of the Calculon cluster named ''calculon'' with the account as specified in the ''User'' directive of your ''~/.ssh/conf.d/lobby'' file via the ''lobby'' jumphost using the alias ''lobby+calculon''. Type the following command in a terminal:
{{{
ssh lobby+calculon
}}}
   In order to override the account name specified in your ''~/.ssh/conf.d/lobby'' you can use:
{{{
ssh youraccount@lobby+calculon
}}}
   You can also transfer data with scp (secure copy) to copy files to your home dir on the cluster like this:
{{{
scp some_file lobby+calculon:
}}}
   or even better use rsync to copy data to your home dir on the cluster like this:
{{{
rsync -av some_directory lobby+calculon:
}}}
   Note the colon at the end of the ''scp'' and ''rsync'' commands. Without the colon you would copy to a local file named ''lobby+calculon''.
   You can add additional ''jumphost+*'' aliases for other User Interface (UI) servers depending on your computing preferences or authorization. If you also added the {{{Host *+*+*}}} code block you can do tripple hops starting with a jumphost like this:
{{{
ssh jumphost+intermediate_server+destination_server
}}}
   In case you are on a network where the default port for SSH (22) is blocked by a firewall you can try to setup SSH over port 443 using an alias like this:
{{{
ssh lobby443+calculon
}}}

Frequent Asked Questions:
 * Q: Why do I get the error {{{muxserver_listen bind(): No such file or directory.}}}? [[BR]]
   A: You may have failed to create the **''~/.ssh/tmp''** folder in the first step or the permissions on this folder are wrong. [[BR]][[BR]]
 * Q: Why do I get the error {{{ControlPath too long}}}? [[BR]]
   A: The {{{ControlPath ~/.ssh/tmp/%C}}} line in your ''~/.ssh/conf.d/lobby'' file expands to a path that is too long. Change the ''!ControlPath'' line in your ''~/.ssh/conf.d/lobby'' file to create a shorter path for the automagically created sockets. You can use for example {{{ControlPath /tmp/%h_%p_%r}}} to create sockets in /tmp. [[BR]][[BR]]
 * Q: Why do I get the error {{{nc: getaddrinfo: Name or service not known. ssh_exchange_identification: Connection closed by remote host}}}? [[BR]]
   A: This server does not exist (anymore); You have a typo in the name of the server you are trying to connect to. Check both the command you typed as well as your ''~/.ssh/conf.d/lobby'' for typos in server names. [[BR]][[BR]]
 * Q: Why do I get the error {{{Permission denied (publickey).}}}? [[BR]]
   A: This error can be caused by various configuration issues: [[BR]]
         Either you are using the wrong account name [[BR]]
         or you are using the wrong private key file [[BR]]
         or the permissions on your home **''~/.ssh/''** dir and/or on its content are wrong [[BR]]
         or your account is misconfigured on our account server. [[BR]]
      Check your account name, private key and permissions. If that did not resolve the issue, then increase the verbosity to debug connection problems (see below). [[BR]][[BR]]
 * Q: Can I increase the verbosity to debug connection problems? [[BR]]
   A: Yes try adding ''-vvv'' like this
      {{{ssh -vvv prefix-youraccount@lobby+calculon}}} [[BR]][[BR]]
   If that does not help to figure out what is wrong please [wiki:Contact send an email to the HPC helpdesk] and include: [[BR]]
    1 The command you used for your failed login attempt [[BR]]
    2 The output of that failed login attempt with ''-vvv'' debugging enabled [[BR]]
    3 A copy of both your ''~/.ssh/config'' and ''~/.ssh/conf.d/lobby'' files. [[BR]]
   (Never ever send us your private key; It does not help to debug your connection problems, but will render the key useless as it is no longer private.)

== Users on Windows ==

 * On the machine from which you want to connect
   1. Make sure you have generated SSH keys linked to your account. See [wiki:RequestAccount RequestAccount] for instructions.
   1. Download and install **''[http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html Putty]''**. Make sure to also download the key manager **''Pageant''**.
   1. Start **''Pageant''**
   1. Load your private key into **''Pageant''**
   1. Start **''Putty''**
   1. Go to **''Connection''** -> **''Ssh''** -> **''Auth''** and select **''Allow agent forwarding''**
   1. Go to **''Connection''** -> **''Data''** and fill in your user name in the **auto-login username''** option

 * You can now connect to for example ''calculon'' via for example jumphost server ''lobby.hpc.rug.nl'' using a double hop like this
   * In **''Putty''** connect to **''lobby.hpc.rug.nl''**
   * Once the connection is established type the following command in a terminal:
{{{
ssh youraccount@calculon
}}}
   Change the name of the destination host to connect to other User Interface (UI) servers.
On order to automate this double hop on Windows please have a look at these instructions: http://mikelococo.com/2008/01/multihop-ssh/