Introduction
SSH is method of securely connect to remote server and execute command on that server. To provide user access to remote server either password or SSH key needs to be shared. But this becomes cumbersome when there are many users as well as servers to manage.
Let's list down the key problems with SSH key and username/password approaches
Risks with traditional SSH authentication
Storing and sharing passwords of all machines securely is tedious. This is same with private key also, exposing them to other user and storing insecurely.
Revoking password or key is again tedious job. How to ensure all users private keys are revoked?
Keys management becomes more tedious when removing or adding new team member. When user removed form the team their key needs to be removed from all remote machines similarly while adding new user his/her key needs to be added in all remote machines.
We can solve all above the problems with the help of HashiCorp Vault and SSH certificate authentication.
HashiCorp Vault
HashiCorp Vault is an identity-based secrets and encryption management system. A secret is anything that you want to tightly control access to, such as API encryption keys, passwords, and certificates.
It provides following features so it can help to solve all above problems listed
Enable and enforce identity based security, where user must authenticate first before granting the ability to SSH into machine
Enable Role Based Access Control (RBAC) for SSH access, where user has given specific role and its permissions to perform limited or restrict action on machine.
Short lived SSH credentials that expire
SSH Certificate Authentication
SSH certificates work in a way similar to SSL certificates. SSH certs are simply a public key signed by a trusted entity called the certificate authority (CA). It provides following features
SSH Configuration
There are many SSH host configurations. Some are not used commonly since they do not typically apply to SSH key authentication methods.
One such configuration which we use here is sshd AuthorizedPrincipalsFile configuration.
SSH Certificate Authority (CA) - HashiCorp Vault SSH Secrets Engine
SSH Secrets Engine, it functions as SSH CA here. It also provides granular access controls to SSH certificate parameters and signing, which is enforced by Vault policies.
The Architecture
The numbers in the diagram represent the following steps:
User creates a personal SSH key pair.
User authenticates to Vault with their Identity Provider (IDP) credentials.
Once authenticated, the user sends their SSH public key to Vault for signing.
Vault signs the SSH key and return the SSH certificate to the user.
User initiates SSH connection using the SSH certificate.
Host verifies the client SSH certificate is signed by the trusted SSH CA and allows connection.
The Vault SSH Secret Engine can contain multiple Vault roles, where each role will contain the parameters that will be used during the SSH key signing. This allows different SSH certificates to be signed with different parameters and principals depending on the Vault role configurations.
Once a user successfully authenticates to Vault, a Vault token will be dynamically generated with an attached policy that dictates which services and secrets can be accessed by the user.
Prerequisites
One remote machine where Vault running and unsealed. Let's call it Vault Server Machine
One remote machine whose access we want from our laptop using SSH and our Vault server . Lets call it host machine.
Our laptop from where we will SSH. Let's call it client machine
You can install Vault by following instructions from the getting started with Vault guide
The following steps will be used to configure vault. I will use Vault CLI for Vault Configuration
Enable UserPass Authentication and Create users with policies
Here we will UserPass authentication method which is simple, Vault supports other authentication methoda such as LDAP or OIDC.
To enable UserPass authentication use following command
vault auth enable userpass
Create one user with policy with the following command, replace {user name} with specific user name and password also. Attach policy ''administrator-policy" to user which we will created later
vault write auth/userpass/users/{user name} password="passw0rd" policies="administrator-policy"
Enable SSH Certificate Secret Engine and Generate SSH CA Key Pair
Use following command to enable secret engine
vault secrets enable -path=ssh-client-signer ssh
To generate signing key use following command
vault write ssh-client-signer/config/ca generate_signing_key=true
You should get the above output, showing the SSH CA public key, which will be used later on in the host machine.
Create Vault Roles for signing client SSH keys
To sign the client keys, we will configure Vault roles to sign and issue SSH certificates with specific configurations based on users’ functional roles.
Use following command to create `administrator-role` role.
vault write ssh-client-signer/roles/administrator-role -<<EOH
{
"allow_user_certificates": true,
"allowed_users": "administrator",
"allowed_extensions": "",
"default_extensions": [
{
"permit-pty": ""
}
],
"key_type": "ca",
"default_user": "administrator",
"ttl": "30m0s"
}
EOH
A few important notes regarding Vault SSH role configurations:
allowed_users: This is the list of allowed users this CA role will sign for. If the requester attempts to get a key signed by specifying a different user not in the allowed_users list for that role, it will fail.
ttl: This is where certificate expiry is set when signing an SSH key. In this example it is set for 30 minutes. Once the certificate expires, a user must authenticate to Vault and request another signed SSH certificate.
Create Vault Policies
The policies will restrict each user’s access to their authorized Vault SSH role for key signing. For example, User is an administrator and will require the SSH certificate signed with the administrator principal. He will have permissions to get her SSH public key signed by the administrator-role. He will not be able to use any other Vault SSH role as per the policy. There is an implicit deny on everything else.
Use following command to crate 'administrator-policy'
vault policy write administrator-policy - << EOF
"# List available SSH roles",
"path", "ssh-client-signer/roles/*", {
"capabilities =": ["list"]
},
"# Allow access to SSH role",
"path", "ssh-client-signer/sign/administrator-role", {
"capabilities =": ["create","update"]
}
EOF
Host Machine Configurations
The following steps will be used to complete the SSH configurations on the host machine
Create local user
Use following command to create local user
sudo useradd -m admin
This Linux account will be used for administrator user login.
Update Trusted SSH CA Public key
Navigate to the SSH directory and create a file that contains the SSH CA public key, which was created in Generate SSH CA Key Pair section
cd /etc/ssh
sudo echo ‘ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDthG9+wjEvCgqBVlBpifXK2PhjXzmjt2+3kN6J6CGsZpeevdlMtxutaAmALfREGYKGol4pqwzJrT5iGp3LAenuiC97x1skItXWrM+BYwyiaU0f8XjBIOPrviBN0v+H6XgoFlujxvdKgJ4+diORkaqtW4gz3Fhe/Gq/3s5WZ5MnB+UZsvYPhd0zuQorEoaYAqarwxq30XPSYEFaH6zqefddJaejMd8PiKGuFdhLHjnZ5jU0r+vpZfrcuQ+81RhVxD4wtiKmouO3zW8bBVrYoURKLap5UlPu8RzPwbJ3PtqtNDkV7SxeNvbXP9mmfkwyBqsU4zaenDRlFqc8UG0SyU6JMM4nL6pLpicC8EC/tHb434U/WwmuHrYhKmpXt25G6cCAHyWxu/9WXqrj4C4Cn2w495WgxGs3EqM+DotSRPw7GkfyCujnpCddcjPc1L5RdHk/tGI26UrnxMHOEBa+zXVmTlEeRyqg3HCSwBejAk3AT1lk1I+D89ANWDgipnNY7UZG+lSsRZHyiKKbfyhaoUFo8JW0KR6vujqflPrzmOHHHyW6zYmBKO+EYTlaiwWAV8nUMl5XfnARmq4Lv6WL3dbdhur6LWY1WiDNviByntYaEDViRXl6lyqrYhrXONJbi0PLB9HuKL7z4m3K9OYgYS5ofFjoCVb9HGytGgEko1wQ==’ > trusted-CA.pem
Create AuthorizedPrincipalsFile
For client authentication to be successful, the principal in the signed SSH certificate must appear in the AuthorizedPrincipalsFile file. Let’s set up AuthorizedPrincipalsFile for the administrator
cd /etc/ssh
mkdir auth_principals/
sudo echo ‘administrator’ > admin
Update sshd_config Configuration
Update the SSH to enable SSH certificate authentication
AuthorizedPrincipalsFile /etc/ssh/auth_principals/%u
ChallengeResponseAuthentication no
PasswordAuthentication no
TrustedUserCAKeys /etc/ssh/trusted-CA.pem
The AuthorizedPrincipalsFile is the path containing the files listing the accepted principal names. The %u signifies that the file name will be the username of the Linux user. Here the admin user will contain the administrator principal.
Don’t forget to restart the sshd service.
sudo service sshd restart
Client Machine Configurations
The user will only need to create an SSH key pair. The user’s SSH public key will be signed by the Vault SSH CA and returned to the user. This signed SSH certificate will then be used to connect to the target host.
Let's complete this with following steps
Create SSH Key Pair
ssh-keygen -t ed25519 -C "user@email.com" -f ~/.ssh/vault-admin-key
ssh-add ~/.ssh/vault-admin-key
Note : If you are using a legacy system that doesn't support the Ed25519 algorithm, use following command
ssh-keygen -t rsa -b 4096 -C "your_email@example.com" -f ~/.ssh/vault-admin-key
-f ~/.ssh/vault-admin-key
Login to Vault
Let's login to Vault using CLI to authenticate with the UserPass authentication method then notice assigned policy in the output like below
Request Signing of User SSH Public Key
With following command request to vault for signing of ssh public key.
vault write -field=signed_key ssh-client-signer/sign/administrator-role \
public_key={SSH_PUBLIC_KEY_PATH} valid_principals=administrator > {SIGNED_PUBLIC_KEY_PATH}
{SSH_PUBLIC_KEY_PATH} replace with actual user ssh public key path generated in Generate SSH Key Pair and {SIGNED_PUBLIC_KEY_PATH} replace where you want to save signed key.
Login with Signed Public Key
Now you should be able to SSH into the host using the signed SSH certificate:
ssh -i {SIGNED_PUBLIC_KEY_PATH} admin@server "whoami"
admin
Normally we keep all ssh keys in ~/.ssh/ folder. When try to login to server you might get error saying Too many authentication, this is because if no identity file is specified, and ssh can find identity files, then it will try all of them.
For this you need to pass IdentitiesOnly=yes to ssh command with SSH private key also like below
In the output you will get Linux user name, that means user has successfully logged in the host machine.
Conclusion
In this article we have created only one role, but in real world example we can create multiple roles like dev-team, support-team and give limited access of remote server to such users using AuthorizedPrincipalsFile in conjunction with Vault SSH roles for security. We can revoke access of usres any time from Vault itself.
