AWS credentials are very powerful – in the wrong hands you could lose data or incur large costs – so you need to manage them carefully. I don’t think there’s a one size fits all here and as usual security and convenience tend to pull in opposite directions, but I think there are at least some general guideslines:

• You should very rarely be using the master account, instead use IAM to create users with specific permissions
• Try to attach policies to groups rather than users – this makes them easier to manage
• Make groups task focussed rather than service focussed. For example I’d have a backup group that had relevant access to S3 and glacier rather than having an S3 group and a glacier group
• Only give people access to what they need. Your accountant for example only needs access to the billing data, not the api or console

The downside of this is that you now have multiple sets of credentials to manage for all these IAM users. For me, credentials fall into 3 categories:

• Credentials that the app uses itself
• Credentials for using the web console
• Credentials for management scripts (provisioning new instances etc.)

Credentials for the app

All of our code runs on EC2, so we use IAM roles to manage credentials used by the application. In a nutshell each instance is assigned a role and roles can be assigned permissions. Code running on an instance can query the instance metadata service to get a temporary set of credentials. Most of the time you don’t need to worry about the details of this as the AWS library you use will probably handle this for you (fog and the official AWS SDKs certainly do). The big advantage of this is that you don’t need to worry about distributing credentials to your instances. Of course this only works if you’re running on EC2.

You can’t add roles to groups or anything like that so our management scripts generate the policies for each role. Changing the permissions a role has is simply a matter of re-running the policy generation script. As with groups, I also tend to organize permissions by purpose so that it’s easy to see what’s going on. Instead of having all the permissions related to S3 in one place, all the permissions related to SQS in another, I’ll have one chunk which has all the permissions for our image thumbnailing in one section, all the permissions for backup scripts in another section and so on.

I find this makes it easier to keep on top of things - as functionality changes or is removed it’s more obvious which sets of permissions should be dropped. You do end up with some redundancy in your policies, but you avoid policy changes to one area of the app impacting other areas of the app.

Credentials for the console

Human beings using the console are subject to exactly the same permission checks as api calls. I tend to allow humans at the console fairly wide ranging permissions, but denying dangerous actions (such as terminating instances) as it is all too easy to misclick and end up doing something terrible. If you need to perform such a task, write some code that does it: code that can be checked in, tested, reviewed by others, etc.

In order for a user to be able to use the web console, it needs a login profile. With fog you can set this up like so

1
2
3

iam = Fog::AWS::IAM.new(...)
user = iam.users.create :id => 'fred'
iam.create_login_profile user.id, 'aPassword'

The user can then signin at https://<amazon account id>.signin.aws.amazon.com/. You can also setup a friendly alias, eg https://my-company.signin.aws.amazon.com/.

One permission I always give is the ability to change your own password. Obviously you don’t want to just give people the change password permission or they could change anyone’s password, so you should make the IAM user the resource for the policy:

1
2
3
4
5
6
7
8
9
10

iam.put_user_policy user.id, 'change_password', {
  "Version"=>"2008-10-17",
  "Statement"=>[
    {
      "Effect"=>"Allow",
      "Action"=>["iam:UpdateLoginProfile"],
      "Resource"=>[user.arn]
    }
  ]
}

This allows that user to only change their own password. As far as keeping those passwords safe, this is just a browser based password so there are plenty of tools to assist with this.

API credentials

I normally create a separate IAM user for api users (eg fred-api). Much like console users can change their password, I give users permission to rotate their own access keys. The policy I use is

1
2
3
4
5
6
7
8
9
10

iam.put_user_policy user.id, 'manage_key', {
  "Version"=>"2008-10-17",
  "Statement"=>[
    {
      "Effect"=>"Allow",
      "Action"=>["iam:*AccessKey*"],
      "Resource"=>[user.arn]
    }
  ]
}

Key rotation is built into our management scripts - if this sort of stuff isn’t really easy then people end up not doing it.

One problem with api keys is where to store them. Unlike passwords, they’re never going to be memorable, and tools such as 1Password aren’t so useful =- they tend to integrate really nicely with browsers but not with my ruby scripts. It would be a real hassle if I had to fire up 1Password and copy api credentials each time I wanted to deploy the app, list instances, start our staging instances and so on. It also feels unpleasant to store the credentials in plain text (although I did that for a long time - it’s probably not too bad if you use some form of full disk encryption).

In the end I settled on the OS X keychain. The platform requirement wasn’t an issue as the whole dev team works on OS X anyway. I wrote some ruby bindings for the keychain that I hope are reusable and of use to others. I store the AWS credentials in their own private credentials file, so the code for getting credentials looks like

1
2
3

keychain = Keychain.open('/path/to/credentials')
item = keychain.generic_passwords.where(:service => 'AWS').first
#You can now use item.password and item.account to access the api

If there is no keychain then one is created on demand.

Using a separate keychain allows me to have separate access policies: this keychain locks itself on sleep or 30 minutes after it was unlocked. When the keychain is locked, the OS X security subsystem takes care of presenting an unlock dialog. One slight niggle is that although you can grant access to a keychain item on a per app level, as far as OS X is concerned the app is ruby - all ruby scripts are considered equal.

Non AWS credentials

Sometimes you might require credentials for a non AWS service, for example you might be sending your emails using a service such as sendgrid. One way to fit those credentials into this scheme is to put those credentials in an S3 bucket. You can then write a very specific IAM policy, along the lines of

1
2
3
4
5
6
7
8
9
10

{
  "Version"=>"2008-10-17",
  "Statement"=>[
    {
      "Effect"=>"Allow",
      "Action"=>["s3:Get*"],
      "Resource"=>["arn:aws:s3:::fred-credentials/sendgrid.json"]
    }
  ]
}

This policy grants the role or user it is applied to the right to read that credentials file. You then write your app to fetch those credentials from S3. This is best wrapped up in a small abstraction layer so that the calling code just asks for a credential by name. This enables production apps to use the S3 mechanism, use different credentials for staging and just read from a local file in development.