Introduction

SSH is method of securely connect to remote server and execute command on that server. To provide user access to remote server either password or SSH key needs to be shared. But this becomes cumbersome when there are many users as well as servers to manage.

Let's list down the key problems with SSH key and username/password approaches

Risks with traditional SSH authentication

• Storing and sharing passwords of all machines securely is tedious. This is same with private key also, exposing them to other user and storing insecurely.

• Revoking password or key is again tedious job. How to ensure all users private keys are revoked?

• Keys management becomes more tedious when removing or adding new team member. When user removed form the team their key needs to be removed from all remote machines similarly while adding new user his/her key needs to be added in all remote machines.

We can solve all above the problems with the help of HashiCorp Vault and SSH certificate authentication.

HashiCorp Vault

HashiCorp Vault is an identity-based secrets and encryption management system. A secret is anything that you want to tightly control access to, such as API encryption keys, passwords, and certificates.

It provides following features so it can help to solve all above problems listed

• Enable and enforce identity based security, where user must authenticate first before granting the ability to SSH into machine

• Enable Role Based Access Control (RBAC) for SSH access, where user has given specific role and its permissions to perform limited or restrict action on machine.

• Short lived SSH credentials that expire

SSH Certificate Authentication

SSH certificates work in a way similar to SSL certificates. SSH certs are simply a public key signed by a trusted entity called the certificate authority (CA). It provides following features

• SSH certificates are signed with a valid time and will automatically expire once the certificate has reached its expiry time.

• Valid SSH certificates can only be signed by the trusted CA’s private key.

SSH Configuration

There are many SSH host configurations. Some are not used commonly since they do not typically apply to SSH key authentication methods.

One such configuration which we use here is sshd AuthorizedPrincipalsFile configuration.

SSH Certificate Authority (CA) - HashiCorp Vault SSH Secrets Engine

SSH Secrets Engine, it functions as SSH CA here. It also provides granular access controls to SSH certificate parameters and signing, which is enforced by Vault policies.

The Architecture

The numbers in the diagram represent the following steps:

1. User creates a personal SSH key pair.

2. User authenticates to Vault with their Identity Provider (IDP) credentials.

3. Once authenticated, the user sends their SSH public key to Vault for signing.

4. Vault signs the SSH key and return the SSH certificate to the user.

5. User initiates SSH connection using the SSH certificate.

6. Host verifies the client SSH certificate is signed by the trusted SSH CA and allows connection.

The Vault SSH Secret Engine can contain multiple Vault roles, where each role will contain the parameters that will be used during the SSH key signing. This allows different SSH certificates to be signed with different parameters and principals depending on the Vault role configurations.

Once a user successfully authenticates to Vault, a Vault token will be dynamically generated with an attached policy that dictates which services and secrets can be accessed by the user.

Prerequisites

• One remote machine where Vault running and unsealed. Let's call it Vault Server Machine

• One remote machine whose access we want from our laptop using SSH and our Vault server . Lets call it host machine.

• Our laptop from where we will SSH. Let's call it client machine

You can install Vault by following instructions from the getting started with Vault guide

The following steps will be used to configure vault. I will use Vault CLI for Vault Configuration

Enable UserPass Authentication and Create users with policies

Here we will UserPass authentication method which is simple, Vault supports other authentication methoda such as LDAP or OIDC.

To enable UserPass authentication use following command

vault auth enable userpass

Create one user with policy with the following command, replace {user name} with specific user name and password also. Attach policy ''administrator-policy" to user which we will created later

vault write auth/userpass/users/{user name} password="passw0rd" policies="administrator-policy"

Enable SSH Certificate Secret Engine and Generate SSH CA Key Pair

Use following command to enable secret engine

vault secrets enable -path=ssh-client-signer ssh

To generate signing key use following command

 vault write ssh-client-signer/config/ca generate_signing_key=true

You should get the above output, showing the SSH CA public key, which will be used later on in the host machine.

Create Vault Roles for signing client SSH keys

To sign the client keys, we will configure Vault roles to sign and issue SSH certificates with specific configurations based on users’ functional roles.

Use following command to create `administrator-role` role.

vault write ssh-client-signer/roles/administrator-role -<<EOH
{
 "allow_user_certificates": true,
 "allowed_users": "administrator",
 "allowed_extensions": "",
 "default_extensions": [
 {
 "permit-pty": ""
 }
 ],
 "key_type": "ca",
 "default_user": "administrator",
 "ttl": "30m0s"
}
EOH

A few important notes regarding Vault SSH role configurations:

allowed_users: This is the list of allowed users this CA role will sign for. If the requester attempts to get a key signed by specifying a different user not in the allowed_users list for that role, it will fail.

ttl: This is where certificate expiry is set when signing an SSH key. In this example it is set for 30 minutes. Once the certificate expires, a user must authenticate to Vault and request another signed SSH certificate.

Create Vault Policies

The policies will restrict each user’s access to their authorized Vault SSH role for key signing. For example, User is an administrator and will require the SSH certificate signed with the administrator principal. He will have permissions to get her SSH public key signed by the administrator-role. He will not be able to use any other Vault SSH role as per the policy. There is an implicit deny on everything else.

Use following command to crate 'administrator-policy'

vault policy write administrator-policy - << EOF
"# List available SSH roles",
"path", "ssh-client-signer/roles/*", {
 "capabilities =": ["list"]
},
"# Allow access to SSH role",
"path", "ssh-client-signer/sign/administrator-role", {
 "capabilities =": ["create","update"]
}
EOF

Host Machine Configurations

The following steps will be used to complete the SSH configurations on the host machine

Create local user

Use following command to create local user

sudo useradd -m admin

This Linux account will be used for administrator user login.

Update Trusted SSH CA Public key

Navigate to the SSH directory and create a file that contains the SSH CA public key, which was created in Generate SSH CA Key Pair section

cd /etc/ssh

sudo echo ‘ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDthG9+wjEvCgqBVlBpifXK2PhjXzmjt2+3kN6J6CGsZpeevdlMtxutaAmALfREGYKGol4pqwzJrT5iGp3LAenuiC97x1skItXWrM+BYwyiaU0f8XjBIOPrviBN0v+H6XgoFlujxvdKgJ4+diORkaqtW4gz3Fhe/Gq/3s5WZ5MnB+UZsvYPhd0zuQorEoaYAqarwxq30XPSYEFaH6zqefddJaejMd8PiKGuFdhLHjnZ5jU0r+vpZfrcuQ+81RhVxD4wtiKmouO3zW8bBVrYoURKLap5UlPu8RzPwbJ3PtqtNDkV7SxeNvbXP9mmfkwyBqsU4zaenDRlFqc8UG0SyU6JMM4nL6pLpicC8EC/tHb434U/WwmuHrYhKmpXt25G6cCAHyWxu/9WXqrj4C4Cn2w495WgxGs3EqM+DotSRPw7GkfyCujnpCddcjPc1L5RdHk/tGI26UrnxMHOEBa+zXVmTlEeRyqg3HCSwBejAk3AT1lk1I+D89ANWDgipnNY7UZG+lSsRZHyiKKbfyhaoUFo8JW0KR6vujqflPrzmOHHHyW6zYmBKO+EYTlaiwWAV8nUMl5XfnARmq4Lv6WL3dbdhur6LWY1WiDNviByntYaEDViRXl6lyqrYhrXONJbi0PLB9HuKL7z4m3K9OYgYS5ofFjoCVb9HGytGgEko1wQ==’ > trusted-CA.pem

Create AuthorizedPrincipalsFile

For client authentication to be successful, the principal in the signed SSH certificate must appear in the AuthorizedPrincipalsFile file. Let’s set up AuthorizedPrincipalsFile for the administrator

cd /etc/ssh

mkdir auth_principals/

sudo echo ‘administrator’ > admin

Update sshd_config Configuration

Update the SSH to enable SSH certificate authentication

AuthorizedPrincipalsFile /etc/ssh/auth_principals/%u
ChallengeResponseAuthentication no
PasswordAuthentication no
TrustedUserCAKeys /etc/ssh/trusted-CA.pem

The AuthorizedPrincipalsFile is the path containing the files listing the accepted principal names. The %u signifies that the file name will be the username of the Linux user. Here the admin user will contain the administrator principal.

Don’t forget to restart the sshd service.

sudo service sshd restart

Client Machine Configurations

The user will only need to create an SSH key pair. The user’s SSH public key will be signed by the Vault SSH CA and returned to the user. This signed SSH certificate will then be used to connect to the target host.

Let's complete this with following steps

Create SSH Key Pair

ssh-keygen -t ed25519 -C "user@email.com"  -f ~/.ssh/vault-admin-key
ssh-add ~/.ssh/vault-admin-key

Note : If you are using a legacy system that doesn't support the Ed25519 algorithm, use following command

ssh-keygen -t rsa -b 4096 -C "your_email@example.com" -f ~/.ssh/vault-admin-key
-f ~/.ssh/vault-admin-key

Login to Vault

Let's login to Vault using CLI to authenticate with the UserPass authentication method then notice assigned policy in the output like below

Request Signing of User SSH Public Key

With following command request to vault for signing of ssh public key.

 vault write -field=signed_key ssh-client-signer/sign/administrator-role \
 public_key={SSH_PUBLIC_KEY_PATH} valid_principals=administrator > {SIGNED_PUBLIC_KEY_PATH}

{SSH_PUBLIC_KEY_PATH} replace with actual user ssh public key path generated in Generate SSH Key Pair and {SIGNED_PUBLIC_KEY_PATH} replace where you want to save signed key.

Login with Signed Public Key

Now you should be able to SSH into the host using the signed SSH certificate:

ssh -i {SIGNED_PUBLIC_KEY_PATH} admin@server "whoami"
admin

Normally we keep all ssh keys in ~/.ssh/ folder. When try to login to server you might get error saying Too many authentication, this is because if no identity file is specified, and ssh can find identity files, then it will try all of them.

For this you need to pass IdentitiesOnly=yes to ssh command with SSH private key also like below

In the output you will get Linux user name, that means user has successfully logged in the host machine.

Conclusion

In this article we have created only one role, but in real world example we can create multiple roles like dev-team, support-team and give limited access of remote server to such users using AuthorizedPrincipalsFile in conjunction with Vault SSH roles for security. We can revoke access of usres any time from Vault itself.