wiki:TransparentMultiHopSSHNewLobby

Version 1 (modified by Pieter Neerincx, 3 years ago) (diff)

--

Transparent multi-hop SSH

Connecting from outside/home via a proxy server a.k.a. jump host

Table of Contents

  1. Connecting from outside/home via a proxy server a.k.a. jump host
  2. Users on Linux / Unix / Mac OS X
  3. Users on Windows

Our servers can only be accessed using a jumphost (a.k.a. proxy) server. This requires port 22 to be open on all machines between your computer and our jumphost server.
The existing lobby.hpc.rug.nl will shortly be replaced by a new machine with the same name. This new machine currently has no DNS entry yet; the example config below is for the new lobby jumphost and is based on its IP address only.

Note that previously there was also a jumphost named lobby.hpc.rug.nl, but this one has been deprecated. Furthermore there many other machines specified in the example config like flexo and bender. Please remove all entries for these machines from your ~/.ssh/config file:

  • The only line that should remain in your ~/.ssh/config for our machines is
  • Additional lines that are required to access our machines are added to a separate ~/.ssh/conf.d/calculon file now.

Users on Linux / Unix / Mac OS X

To setup transparent multi-hop SSH:

  • On the machine from which you want to connect
    • Make sure you have a ~/.ssh folder with generated SSH keys linked to your account. See RequestAccount? for instructions.
    • Create ~/.ssh/tmp and ~/.ssh/conf.d folders. Open a terminal and type the following command: 
      mkdir -p ~/.ssh/tmp
mkdir -p ~/.ssh/conf.d
    • Create a ~/.ssh/config file if it does not exist yet.
    • Add to your ~/.ssh/config something like the following: 
      #
# Generic stuff: only for MacOS clients.
#
IgnoreUnknown UseKeychain
	UseKeychain yes
IgnoreUnknown AddKeysToAgent
	AddKeysToAgent yes

#
# Generic stuff: prevent timeouts 
#
Host *
	ServerAliveInterval 60
	ServerAliveCountMax 5

#
# Generic stuff: share existing connections to reduce lag when logging into the same host in a second shell
#
ControlMaster auto
ControlPath ~/.ssh/tmp/%h_%p_%r

#
##
### RUG HPC v2 hosts in *.hpc.rug.nl domain with DNS.
##
#
Host *peregrine pg-interactive !*.hpc.rug.nl
	HostName %h.hpc.rug.nl
	User prefix-youraccount

#
##
### UMCG Research IT HPC v2 hosts in *.hpc.rug.nl domain.
##
#
#  A. With DNS entry.
#
Host foyer lobby *calculon *cher-ami !*.hpc.rug.nl
	HostName %h.hpc.rug.nl
	User prefix-youraccount

#
##
### GCC HPC v2 hosts in *.gcc.rug.nl domain.
##
#
Host *flexo *bender *gattaca* !*.gcc.rug.nl
	HostName %h.gcc.rug.nl
	User prefix-youraccount

#
##
### Proxy settings for multi-hop SSH.
##
# 
# The syntax in all the ProxyCommand rules below assumes your private key is in the default location.
# The default location is:
#  ~/.ssh/id_rsa for keys generated with the RSA algorithm.
#  ~/.ssh/id_dsa for keys generated with the DSA algorithm.
# In case your private key file is NOT in the default location you must:
#  1. Specify the path to your private key file on the command line when logging in with SSH.
#     For example:
#         $> ssh -i ~/.ssh/some_other_key.file prefix-youraccount@proxy_server+destination_server
#  2. Add the path to your private key file in the ProxyCommand rules below.
#     For example:
#         Host proxy_server+*
#             PasswordAuthentication No
#             ProxyCommand ssh -X -q -i ~/.ssh/some_other_key.file prefix-youraccount@$(echo %h | sed 's/+[^+]*$//').some.sub.domain -W $(echo %h | sed 's/^[^+]*+//'):%p
#

#
# Universal proxy settings for triple-hop SSH.
#
Host *+*+*
        ProxyCommand ssh -X -q $(echo %h | sed 's/+[^+]*$//') -W $(echo %h | sed 's/^[^+]*+[^+]*+//'):%p

#
# Double-hop proxy settings for HPC V2 & V3 environment servers in *.hpc.rug.nl or *.umcg.nl domain.
#
Host lobby+* foyer+*
        PasswordAuthentication No
        ProxyCommand ssh -X -q prefix-youraccount@$(echo %h | sed 's/+[^+]*$//').hpc.rug.nl -W $(echo %h | sed 's/^[^+]*+//'):%p
Host passage+* gate+*
        PasswordAuthentication No
        ProxyCommand ssh -X -q prefix-youraccount@$(echo %h | sed 's/+[^+]*$//').umcg.nl -W $(echo %h | sed 's/^[^+]*+//'):%p
#
# Sometimes port 22 for the SSH protocol is blocked by firewalls; in that case you can try to use SSH on port 80 as fall-back.
# Do not use port 80 by default for SSH as it officially assigned to HTTP traffic and some firewalls will cause problems when trying to route SSH over port 80.
#
Host lobby80+* foyer80+*
        PasswordAuthentication No
        ProxyCommand ssh -X -q prefix-youraccount@$(echo %h | sed 's/+[^+]*$//').hpc.rug.nl -W $(echo %h | sed 's/^[^+]*+//'):%p -p 80
      Replace all occurences of prefix-youraccount with:
      prefix = based on your organization. Usually either umcg or lifelines
      youraccount = your account on calculon.hpc.rug.nl = your account on umcg.hpc.rug.nl = etc.

      If you are not on a Mac or on a very old one you may have to comment the # Generic stuff: only for MacOS clients section at the top of example ~/.ssh/config

    • Make sure you are the only one who can access your ~/.ssh folder. Type the following command in a terminal: 
      chmod -R go-rwx ~/.ssh
  • You can now for example connect to calculon.hpc.rug.nl with the account as specified by User via for example proxy server lobby.hpc.rug.nl using the alias lobby+calculon. Type the following command in a terminal: 
    ssh lobby+calculon
    In order to override the accountname specified in your ~/.ssh/config you can use: 
    ssh prefix-youraccount@lobby+calculon
    You can also transfer data with scp (secure copy) to copy files to your home dir on the cluster like this: 
    scp some_file lobby+calculon:
    or even better use rsync to copy data to your home dir on the cluster like this: 
    rsync -av some_directory lobby+calculon:
    Note the colon at the end of the scp and rsync commands. Without the colon you would copy to a local file named lobby+calculon. You can add additional proxy+* aliases for other User Interface (UI) servers and other proxy servers depending on your computing preferences or authorization. If you also added the Host *+*+* code block you can do tripple hops starting with a proxy like this: 
    ssh proxy+intermediate_server+destination_server
    In case you are on a network where the default port for SSH (22) is blocked by a firewall you can try to setup SSH over port 80 using an alias like this: 
    ssh lobby80+calculon

Frequent Asked Questions:

  • Q: Why do I get the error muxserver_listen bind(): No such file or directory.?
    A: You may have failed to create the ~/.ssh/tmp folder in the first step or the permissions on this folder are wrong.

  • Q: Why do I get the error ControlPath too long?
    A: The ControlPath ~/.ssh/tmp/%h_%p_%r line in your ~/.ssh/config file expands to a path that is too long. Change the ControlPath line in your ~/.ssh/config file to create a shorter path for the automagically created sockets. You can use for example ControlPath /tmp/%h_%p_%r to create sockets in /tmp.

  • Q: Why do I get the error nc: getaddrinfo: Name or service not known. ssh_exchange_identification: Connection closed by remote host?
    A: This server does not exist (anymore); You have a typo in the name of the server you are trying to connect to. Check both the command you typed as well as your ~/.ssh/config for typos in server names.

  • Q: Why do I get the error Permission denied (publickey).?
    A: This error can be caused by various configuration issues:

Either you are using the wrong account name
or you are using the wrong private key file
or the permissions on your home ~/.ssh/ dir and/or on its content are wrong
or your account is misconfigured on our account server.

Check your account name, private key and permissions. If that did not resolve the issue, then increase the verbosity to debug connection problems (see below).

  • Q: Can I increase the verbosity to debug connection problems?
    A: Yes try adding -vvv like this

ssh -vvv prefix-youraccount@lobby+calculon

If that does not help to figure out what is wrong please send an email to the HPC helpdesk and include:

1 The command you used for your failed login attempt
2 The output of that failed login attempt with -vvv debugging enabled
3 A copy of your ~/.ssh/config file.

(Never ever send us your private key; It does not help to debug your connection problems, but will render the key useless as it is no longer private.)

Users on Windows

  • On the machine from which you want to connect
    1. Make sure you have generated SSH keys linked to your account. See RequestAccount? for instructions.
    2. Download and install ​Putty. Make sure to also download the key manager Pageant.
    3. Start Pageant
    4. Load your private key into Pageant
    5. Start Putty
    6. Go to Connection -> Ssh -> Auth and select Allow agent forwarding
    7. Go to Connection -> Data and fill in your user name in the auto-login username option
  • You can now connect to for example calculon.hpc.rug.nl via for example proxy server lobby.hpc.rug.nl using a double hop like this
    • In Putty connect to lobby.hpc.rug.nl (or choose another proxy from the list)
    • Once the connection is established type the following command in a terminal: 
      ssh prefix-youraccount@calculon.hpc.rug.nl
      Change the name of the destination host to connect to other User Interface (UI) servers.

On order to automate this double hop on Windows please have a look at these instructions: ​http://mikelococo.com/2008/01/multihop-ssh/