Hi! I'm Liz, a Developer Advocate at honeycomb.io, and I spent my first weeks at the company doing security hardening of our infrastructure. I'd like to share what I'd learned with you, so that you can benefit from my reading of dozens of scattered pages of documentation and my ruling out of numerous dead ends.
# Why you should take security and usability seriously
Developers and administrators have historically used SSH keys to provide authentication between hosts. By adding passphrase encryption, the private keys become resistant to theft when at rest. But what about when in use? Unfortunately, the usability challenges of re-entering the passphrase on every connection means that engineers began caching keys unencrypted in memory of their workstations, and worse yet, forwarding the agent to allow remote hosts to use the cached keys without further confirmation. The [recent breach at Matrix](https://web.archive.org/web/20190412143908/https://github.com/matrix-org/matrix.org/issues/357) underscores how dangerous it is to allow authenticated sessions to propagate across hosts and environments without a human in the loop.
Thus, we need solutions that prevent key theft from the systems we connect to, while maintaining ease of use. Two-factor authentication stops malicious automated propagation in its tracks by having a second factor protect use of our keys. There are two primary ways of preventing an attacker from misusing our credentials: either using a separate device that generates, using a shared secret, numerical codes that we can transfer over out of band and enter alongside our key, or having the separate device perform all the cryptography for us only when physically authorized by us.
Google, where I previously worked, employs short-lived SSH certificates issued by a central piece of infrastructure, stored on secure hardware tokens. But this is a serious change to developer workflow, and requires extensive infrastructure to set up. What will work for a majority of developers who are used to simply loading their SSH key into the agent at the start of their login session and SSHing everywhere?
# Design considerations & threat models
I'm assuming that you have a publicly exposed bastion host for each environment that intermediates accesses to the rest of each environment's VPC, and use SSH keys to authenticate from laptops to the bastion and from the bastion to each VM/container in the VPC. If you don't yet have a bastion host and a VPC, start there!
It was important to me to make Honeycomb safe from compromise, even if malicious worm-like code were executed on a developer's laptop while SSH keys were unlocked, or if a developer accidentally forwarded an SSH agent to a hostile remote system. I also thought it important to build on existing work to disk encrypt all endpoints by ensuring the loss of physical control over a phone or hardware token could not itself grant production access. However, I consider it out of scope to prevent active local intervention and session hijacking (since someone who controls your active console or keyboard has you pretty well pwned).
I'm also assuming you have a mix of operating systems, hardware, and preferences about carrying dongles vs. wanting to use phones for second factor, etc.
# How to get started!
First, start by enabling numerical time-based one time password (TOTP) for SSH authentication. Is it perfect? No, since a malicious host could impersonate the real bastion (if strict host checking isn't on), intercept your OTP, and then use it to authenticate to the real bastion. But it's better than being wormed or compromised because you forgot to take basic measures against even a passive adversary.
## Server-side setup
You'll want a root shell open just in case, and the following snippets added to your Chef cookbooks (from this gist):
* `metadata.rb`
* `attributes/default.rb` (from `attributes.rb`)
* `files/sshd`
* `recipes/default.rb` (copy from `recipe.rb`)
* `templates/default/users.oath.erb`
Okay, now we can set this running on our hosts… and go through the client setup for ourselves at least.
## Client-side setup
Now, each user authenticating needs a shared key to be present, encrypted, in SSM (or equivalent for your choice of cloud provider). Have each user install an OTP app such as Google Authenticator, Authy, Duo, or Lastpass, then do the following on their laptop:
**Install dependencies:**
`brew install oath-toolkit` OR `apt install oathtool openssl`
**Generate a random base16 string to use as your key:**
➜ openssl rand -hex 10
22ea2966afefd82660e1
##### ^^^ that's an example output used here - don't use it!
**Convert it and put it into a phone-based authenticator app:**
Run `oathtool -v [key]` to convert it to the format (“Base32 secret”) that mobile authenticators use.
➜ oathtool -v 22ea2966afefd82660e1
Hex secret: 22ea2966afefd82660e1
Base32 secret: ELVCSZVP57MCMYHB
... more stuff down here we don't need
* For *1Password*, add a one time password and enter the “Base32 secret” output from oathtool -v [key]
* For *Duo*, select “other” and use the Base32 secret.
* for *Authy* click “Enter key manually” and use the Base32 secret
**Verify that generated codes are correct:**
Run `oathtool --totp [key]` and check that it returns the same value as your authenticator application.
➜ oathtool --totp 22ea2966afefd82660e1
693439
**Store our key into the cloud secrets manager:**
Run `aws ssm put-parameter --name /2fa/totp/$USER --value [key] --type SecureString --key-id alias/parameter_store_key` to put your key into SSM Parameter Store. `$USER` should be the same as the username you use when you log in to a bastion. If you are updating the key instead of pushing it for the first time, add the `--overwrite` flag to the end of the command.
➜ aws ssm put-parameter --name /2fa/totp/lizf --value 22ea2966afefd82660e1 --type SecureString --key-id alias/parameter_store_key
{
"Version": 1
}
**Log in for the first time:**
Now, when we ssh to the bastion host, we can ensure that the SSH agent can only be trampolined to other hosts within the VPC, but any attempt to programatically use from the outside the forwarded agent (or loaded in-memory keys) to access a bastion will fail because no TOTP from the separate mobile device was provided.
Let's check that we're asking for TOTPs:
➜ ssh -A bastion
Enter passphrase for key '[snip]':
One-time password (OATH) for '[user]':
Welcome to Ubuntu 18.04.1 LTS...
# Now there's a value proposition for hardware auth…
People might get sick and tired of entering a numerical OTP every time they have to log into the bastion! It's almost like the old days of passphrase-encrypted SSH keys that motivated us to use agents! So let's leverage this inherent laziness to get people more, rather than less, secure!
## Server-side setup
Change the beginning of `files/sshd` in your Chef module to begin as follows:
auth required pam_permit.so
auth optional pam_cap.so
# If it's a hardware or secure enclave SSH key, no need for a numerical OTP.
auth sufficient pam_ssh_agent_auth.so file=/etc/2fa_token_keys
# Check a TOTP code as a second resort, using a time slip of +/- 150 seconds.
auth sufficient pam_oath.so usersfile=/etc/users.oath digits=6 window=5
# People without OTPs will need to add an OTP secret to AWS SSM and wait an hour.
auth requisite pam_deny.so
...
And add the following additional lines to `recipes/default.rb` (a note to the nervous: my source modifications to openssh-server and libpam-ssh-agent-auth are available [from Launchpad](https://launchpad.net/~honeycomb.io/+archive/ubuntu/ssh-2fa/+packages)):
apt_repository 'openssl-pam-bindings' do
uri 'ppa:honeycomb.io/ssh-2fa'
end
packages = %w{ openssh-server libpam-ssh-agent-auth }
packages.each do |p|
r = package p do
action :upgrade
end
end
service 'sshd' do
subscribes :reload, 'package[openssh-server]'
end
Now you'll need to use Chef to populate `/etc/2fa_token_keys` with keys that you know are generated and stored securely (e.g. using one of the below methods). I don't know how you maintain your lists of ssh key mappings to users, nor how you add ssh keys to your `~/.ssh/authorized_keys` files, so I can't provide general advice.
## Mac client setup
People with Touchbar Macs should use TouchID to authenticate logins, as they'll have their laptop and their fingers with them anyways. [sekey](https://github.com/sekey/sekey) lets us support this.
**Install the binary:**
`brew cask install sekey`
**Add to `~/.ssh/config` on your local machine:**
`IdentityAgent ~/.sekey/ssh-agent.ssh`
**Generate a key and export it:**
sekey --generate-keypair "bastion key"
sekey --export-key $(sekey --list-keys|grep "bastion key"|grep --only-matching -E '[a-f0-9]{40}')
And then store the resulting key to `/etc/2fa_token_keys` and `~/.ssh/authorized_keys` in Chef.
## Krypt.co setup for iOS and Android
Instead of generating OTPs and sending them over manually with our fingers, our mobile devices can securely store our SSH keys and only remotely authorize usage (and send the signed challenge to the remote server) if a human presses a button on the phone.
This is the theory behind krypt.co, and is even more secure than a TOTP app so long as you supply [appropriate parameters](https://krypt.co/docs/security/privacy-policy.html#private-key-storage) to force hardware coprocessor storage (NIST P-256 for iOS, and 3072-bit RSA for Android, on new enough devices). Make sure people use screen locks!
Follow the instructions here: https://krypt.co/docs/start/upload-your-ssh-publickey.html and then supply the generated key to both `~/.ssh/authorized_keys` and `/etc/2fa_token_keys` in your Chef automation, and you won't be prompted for a TOTP.
## YubiKey hardware token & Linux/ChromeOS client setup
### Initial per-YubiKey setup
Follow these instructions from a Linux host to set up a basic working hardened YubiKey SSH key:
**Install Dependencies**
sudo apt-add-repository ppa:yubico/stable && sudo apt-get update
sudo apt-get install gpg yubikey-manager-qt pinentry-curses scdaemon pcscd
echo "reader-port Yubico YubiKey" > .gnupg/scdaemon.conf
**Hardening to prevent a rogue host from authenticating without your permission**
ykman openpgp touch sig on
ykman openpgp touch aut on
ykman openpgp touch enc on
**Hardening in case your security key is stolen**
`gpg --change-pin`
Default user pin is `123456` and admin pin is `12345678`, change both of them to something more secure; they can both be the same PIN.
Generate a random 24-byte hex-encoded reset key and save it somewhere, GPG encrypted with your normal daily use keys (`ykman-gui` can generate a 24-byte string for you in “PIV → Configure PINs → Change Management Key”)
**Generating the keys:**
gpg --card-edit
admin
generate
* Input `4096` for all three modes (you'll need to enter the admin and user pins)
* Don't back up the stubs when prompted.
* Enter your full name and email address; make sure you leave a comment (e.g. `desk computer`) so you know which stub key is which in your GPG keyring.
Wait a minute, then enter the user PIN one more time, then wait about 5-10 minutes for the generation process to complete. It will print the UID of the master key before returning you to the card-edit prompt.
quit
`gpg --export-ssh-key UID_of_master_key`
This will print out the ssh pubkey string you'll need to add to the remote `~/.ssh/authorized_keys` and `/etc/2fa_token_keys` in Chef.
### Usage for authentication
#### Linux
Once the per-key setup is done, the configured Yubikey can be used in a Linux machine configured like so:
`gpg --with-keygrip -K`
Save the keygrip of the master key you just generated to `.gnupg/sshcontrol`
**Ensure that you have `gpg-agent` configured correctly:**
Set curses pinentry. Why? So you don't randomly get X passphrase/passcode prompts all over the place (esp remotely):
Edit `~/.gnupg/gpg-agent.conf` to contain:
pinentry-program /usr/bin/pinentry-curses
enable-ssh-support
You'll update your `~/.bashrc` to contain the following lines:
export SSH_AUTH_SOCK=$(gpgconf --list-dirs agent-ssh-socket)
export GPG_TTY=$(tty)
gpg-connect-agent updatestartuptty /bye >/dev/null
Run `ssh-add -l` to confirm you see your key in the list (it'll show `4096 SHA256:... cardno:... (RSA)` in the listing).
When you ssh from a terminal into a bastion (remember to `ssh -A` for agent forwarding!), it'll prompt in the terminal that you most recently opened for your PIN on initial usage of the key. You'll complete that step, then tap your key to confirm. You're in!
#### ChromeOS
Install these two Chrome apps:
* https://chrome.google.com/webstore/detail/secure-shell-app/pnhechapfaindjhompbnflcldabbghjo
* https://chrome.google.com/webstore/detail/smart-card-connector/khpfeaanjngmcnplbdlpegiifgpfgdco
Then open the Secure Shell App (this won't work yet from the Crostini Terminal app because Crostini doesn't have USB pass-through yet, although it's coming in Chrome 75!)
Within the secure shell app's configuration screen for the bastion host:
* relay server option: `--ssh-agent=gsc`
* ssh option: `-A`
You'll then enter the user PIN when prompted, and tap the security key to confirm when logging into the bastion.
# Further reading
The folks at krypt.co have written some [fantastic blogs on securing SSH](https://krypt.co/docs/ssh/using-a-bastion-host.html) that go beyond the basic hardening I recommend here.
Hope this helps! Send me a Twitter DM (@lizthegrey) or email (lizf@honeycomb.io) if you have improvements to suggest!
metadata.rb
name "bastion"
description "special hardening for bastions"
version "0.0.1"
depends "aws"
depends "sshd"
recipe.rb
package "libpam-oath" do
action :upgrade
end
aws_ssm_parameter_store 'getOTPsecrets' do
path '/2fa/totp/' # or your own choice of SSM path.
recursive true
with_decryption true
return_key 'totp_secrets'
action :get_parameters_by_path
# No need for aws_access_key and aws_secret_access_key due to implicit EC2 grant.
sensitive true
end
# Populate the oath file.
template '/etc/users.oath' do
source 'users.oath.erb'
owner 'root'
group 'root'
mode '0600'
variables(
:users => lazy { node.run_state['totp_secrets'] }
)
sensitive true
end
cookbook_file '/etc/pam.d/sshd' do
source 'sshd'
owner 'root'
group 'root'
mode '0644'
action :create
end
# Force ssh to consult PAM as well as using SSH keys for primary auth..
include_recipe 'sshd'
sshd
auth required pam_permit.so
auth optional pam_cap.so
# Check a TOTP code, using a time slip of +/- 150 seconds.
auth sufficient pam_oath.so usersfile=/etc/users.oath digits=6 window=5
# People without OTPs will need to add an OTP secret to AWS SSM and wait an hour.
auth requisite pam_deny.so
account required pam_nologin.so
@include common-account
session [success=ok ignore=ignore module_unknown=ignore default=bad] pam_selinux.so close
session required pam_loginuid.so
session optional pam_keyinit.so force revoke
@include common-session
session optional pam_motd.so motd=/run/motd.dynamic
session optional pam_motd.so noupdate
session optional pam_mail.so standard noenv
session required pam_limits.so
session required pam_env.so
session required pam_env.so user_readenv=1 envfile=/etc/default/locale
session [success=ok ignore=ignore module_unknown=ignore default=bad] pam_selinux.so open
@include common-password
users.oath.erb
# To update this file, generate a SSM parameter in /2fa/totp.
# See this URL for examples:
# https://console.aws.amazon.com/systems-manager/parameters/?region=us-east-1#list_parameter_filters=Path:Recursive:%2F2fa%2F
<% @users.each do |key, value| %>
HOTP/T30 <%= key %> - <%= value %>
<% end %>
class UsersController < ApplicationController
# ユーザ情報登録画面
def new
@user = User.new
end
# ユーザの情報が送られてきたら、データベースに保存
def create
@user = User.new(user_params)
if @user.save
redirect_to login_path
else
render 'new'
end
end
private
# 検証用
def user_params
params.require(:user).permit(:name, :mail, :password, :password_confirmation)
end
end
routes.rb
> rails g controller Sessions new
# route ルーティングの設定を行う
# ログイン / ログアウト
get 'login', to: 'sessions#new'
post 'login', to: 'sessions#create'
delete 'logout', to: 'sessions#destroy'
- Create a Rails 6 app
```
# In terminal run
rails new my_tails_app -d postgresql
```
- Change into the app directory
```
cd my_tails_app
```
- Install Tailwind
```
yarn add tailwindcss --dev
```
- Create the `app/javascript/css` directory
```
mkdir app/javascript/css
```
- Generate the Tailwind config file
```
yarn tailwind init app/javascript/css/tailwind.js
```
- Create the `tailwind.css` & `application.css` files
```
touch app/javascript/css/tailwind.css app/javascript/css/application.css
```
- Paste the following into `app/javascript/css/tailwind.css`
```css
/**
* This injects Tailwind's base styles, which is a combination of
* Normalize.css and some additional base styles.
*
* You can see the styles here:
* https://github.com/tailwindcss/tailwindcss/blob/master/css/preflight.css
*
* If using `postcss-import`, use this import instead:
*
* @import "tailwindcss/preflight";
*/
@tailwind preflight;
/**
* This injects any component classes registered by plugins.
*
* If using `postcss-import`, use this import instead:
*
* @import "tailwindcss/components";
*/
@tailwind components;
/**
* Here you would add any of your custom component classes; stuff that you'd
* want loaded *before* the utilities so that the utilities could still
* override them.
*
* Example:
*
* .btn { ... }
* .form-input { ... }
*
* Or if using a preprocessor or `postcss-import`:
*
* @import "components/buttons";
* @import "components/forms";
*/
/**
* This injects all of Tailwind's utility classes, generated based on your
* config file.
*
* If using `postcss-import`, use this import instead:
*
* @import "tailwindcss/utilities";
*/
@tailwind utilities;
/**
* Here you would add any custom utilities you need that don't come out of the
* box with Tailwind.
*
* Example :
*
* .bg-pattern-graph-paper { ... }
* .skew-45 { ... }
*
* Or if using a preprocessor or `postcss-import`:
*
* @import "utilities/background-patterns";
* @import "utilities/skew-transforms";
*/
```
- Add the requires to the `postcss.config.js` file found in your app root.
Add these lines.
```
require('tailwindcss')('./app/javascript/css/tailwind.js'),
require('autoprefixer'),
```
The `postcss.config.js` file should now look something like this.
```
module.exports = {
plugins: [
require('postcss-import'),
require('postcss-flexbugs-fixes'),
require('tailwindcss')('./app/javascript/css/tailwind.js'),
require('autoprefixer'),
require('postcss-preset-env')({
autoprefixer: {
flexbox: 'no-2009'
},
stage: 3
})
]
};
```
- Generate a home page to check the setup.
```
rails g controller home index
```
Add a root path to `config/routes.rb`
```
root 'home#index'
```
Before starting the Rails Server create the db.
```
rails db:create
```
Start the Rails Server
```
rails s
```
> You should see the `Home#index` page.
Replace `app/views/home/index.html.erb` with
```
<div class="bg-white mx-auto max-w-sm shadow-lg rounded-lg overflow-hidden">
<div class="sm:flex sm:items-center px-6 py-4">
<img class="block h-16 sm:h-24 rounded-full mx-auto mb-4 sm:mb-0 sm:mr-4 sm:ml-0" src="https://avatars2.githubusercontent.com/u/4323180?s=400&u=4962a4441fae9fba5f0f86456c6c506a21ffca4f&v=4" alt="">
<div class="text-center sm:text-left sm:flex-grow">
<div class="mb-4">
<p class="text-xl leading-tight">Adam Wathan</p>
<p class="text-sm leading-tight text-grey-dark">Developer at NothingWorks Inc.</p>
</div>
<div>
<button class="text-xs font-semibold rounded-full px-4 py-1 leading-normal bg-white border border-purple text-purple hover:bg-purple hover:text-white">Message</button>
</div>
</div>
</div>
</div>
```
You should see a formated card.
application.css
/**
* This injects Tailwind's base styles, which is a combination of
* Normalize.css and some additional base styles.
*
* You can see the styles here:
* https://github.com/tailwindcss/tailwindcss/blob/master/css/preflight.css
*
* If using `postcss-import`, use this import instead:
*
* @import "tailwindcss/preflight";
*/
@tailwind preflight;
/**
* This injects any component classes registered by plugins.
*
* If using `postcss-import`, use this import instead:
*
* @import "tailwindcss/components";
*/
@tailwind components;
/**
* Here you would add any of your custom component classes; stuff that you'd
* want loaded *before* the utilities so that the utilities could still
* override them.
*
* Example:
*
* .btn { ... }
* .form-input { ... }
*
* Or if using a preprocessor or `postcss-import`:
*
* @import "components/buttons";
* @import "components/forms";
*/
/**
* This injects all of Tailwind's utility classes, generated based on your
* config file.
*
* If using `postcss-import`, use this import instead:
*
* @import "tailwindcss/utilities";
*/
@tailwind utilities;
/**
* Here you would add any custom utilities you need that don't come out of the
* box with Tailwind.
*
* Example :
*
* .bg-pattern-graph-paper { ... }
* .skew-45 { ... }
*
* Or if using a preprocessor or `postcss-import`:
*
* @import "utilities/background-patterns";
* @import "utilities/skew-transforms";
*/
module ApplicationHelper
def bootstrap_class_for flash_type
case flash_type
when :success
"alert-success"
when :error
"alert-error"
when :alert
"alert-block"
when :notice
"alert-info"
else
flash_type.to_s
end
end
end
Kui meil on koodiplokk，mis kordub mitmetel lehtedel，võibseléstastaeraldi lehele（partial）ning sellele vastavalt vajadusele viidata <br/> <br/> oluline on pidada silmas partial'i faili nimetust，mis algab alakriipsuga
new.html.erb
Siia tuleb see viide
<%= render 'form'%>
_form.html.erb
Siia tuleb koodiplokk
<% if @article.errors.any? %>
<ul>
<% @article.errors.full_messages.each do |msg|%>
<li><%= msg %></li>
<% end %>
</ul>
<% end %>
<%= form_for @article do |n| %>
<p>
<%= n.label :title%><br>
<%= n.text_field :title%>
</p>
<p>
<%= n.label :description%><br>
<%= n.text_area :description%>
</p>
<p>
<%= n.label :content%><br>
<%= n.text_area :content%>
</p>
<p>
<p>
<%= n.submit %>
</p>
</p>
<% end %>
<%= link_to 'Back to articles', articles_path %>