Setting up Encryption at REST with Amazon RDS and PostgreSQL

In order to make sure our customer data is as safe as possible, we decided to implement encryption at rest. Although we tried a few different approaches, we finally settled on encrypting the database file system and everything related to it. As we are already using an Amazon PostgreSQL instance, and Amazon RDS supports database encryption at rest, we chose that option. As it often is in life, you can’t really flip a switch and encrypt a running instance. So we decided to dive into the documentation and find out how to do it as painlessly as possible.

_Mandatory disclaimer: this guide is intended for educational purposes only and if you somehow mess up your production data by following it, it’s not my fault._

How do you encrypt an unencrypted Amazon RDS instance then? Well, it’s a pretty simple and quite short procedure (this does depend on the amount of data you have, but it shouldn’t take you more than an hour for a 500GB instance)

Take a few minutes and make yourself a cup of coffee/tea/matcha, then come back and follow along this simple procedure.

1. To ensure that you don’t have any “new” data from users or background jobs you should put the application into maintenance mode and turn off all web and worker processes. It is also nice to announce a planned downtime weeks in advance, and do it when there are practically no users using the app. This probably won’t be a Wednesday morning, during business hours. You have your analytics, plan the thing in advance.
2. After the maintenance mode is on, and there is no chance of any database connections coming to the old unencrypted database, you can take a snapshot of it. Select the desired database from Amazon RDS \> Instances then go to Instance actions \> Take snapshot, be sure to give the snapshot a nice name (Daisy does sound nice, but I would go with app-production-decrypted or something descriptive).
3. Go to Snapshots and select the snapshot you’ve just created
4. Select Snapshot actions then Copy Snapshot
5. Select Enable Encryption in the encryption box and select the KMS key for the database you are encrypting (or let the system create an auto managed key, which I don’t recommend, because you will be using a master KMS key for all instances here, and not a per-instance one.)
6. Rename the old database instance to a new name, so the name won’t collide with the restored instance.
7. Restore the snapshot in the same way as you would do with a normal unencrypted snapshot by going to the Snapshots page, selecting a snapshot and selecting Restore Snapshot under Snapshot Actions. Obviously you want to give it the same name that the old database instance had (you don’t want to update those pesky configuration files with database connection strings).
8. Run your tests and assert that everything works, and only then turn on the web/worker services on. Take down the maintenance page, and reap the benefits of having encryption at rest.

While there are many ways to set up encryption at rest, and many variations on what falls under that term, this is just one of them that fits our model best. We did explore a few more encryption options, but all came with certain drawbacks when implemented within our architecture. Stay tuned for more AWS articles in the future.

A good setup goes a long way

I’ve been limited to a 13” screen to do my work on for ages and ages, well since July 2013 at least. The only person who did the limiting was myself. One of the things that contributed to the issue was working from home, and never having proper office space. DHH also praised working from his 11” MacBook Air on the go all the time. Although that is doable, and I first went with the 13” Air, then a 13” pro, there were certain times when something was missing.
Those certain times increased in rate and frequency when I started wearing different hats, working on everything from OPS, through backend Ruby on Rails, to front-end JavaScript development. Handling complex, multi-language projects is pretty tricky on a 13” screen. It’s not impossible, because I’ve been doing this for two years now. But if you can solve the pain you are feeling by buying a screen/mouse/keyboard combo, then there is no reason to suffer that pain.
Dissecting big problems into smaller chunks can also help a lot. But that also takes a lot of time, and you need to be able to look at the big picture once in a while. Although my eyes are still serving me well, there is no sense in destroying them on purpose. It’s also nice to be able to pass the Joel Test
So I faced another of those complex issues, where you have to look at the big picture while developing something. As it happens often, the lack of top level overview caused me to follow the wrong path a couple of times. Now, I don’t suggest impulse buying, and I investigated all the equipment beforehand. Luckily the computer shop near me had everything that I needed.
After a 10 minute setup, everything was up and running, I’m still getting used to my mechanical keyboard. It differs a lot from the MacBook keyboard I’ve been using for the last few years. But as I’m writing this article, I’m liking the feedback and the writing simplicity more and more. It kinda reminds me of the IBM model M keyboard that I had ages ago. I’d use it today, if my mother didn’t throw it away.
Having a 27” screen, alongside the 13” MacBook, is awesome. It takes me back to the time when I had to buy a new video card with my own cash. Then install it in my work computer on the previous job. Then I took an old 15” lcd from the company’s service room, putting it alongside my 17” screen. It was pure awesomeness back then, more or less.
Having an awesome chair also means a lot. I know people that praise their Aerons and Embodys and I don’t know what else is cool these days. I’ve sat in both of the mentioned chairs, and they are awesome. But I just won’t pay more money for a chair than I paid for my computer. Yes, you can buy a refurbished chair, but it’s still worth a hefty amount of cash. A few friends suggested that I buy Markus from IKEA, and I didn’t regret it at all. I can buy 10 of these chairs (maybe even more) for the price of one Aeron. And it’s an awesome super adjustable, comfortable chair.
What are your pain points with your computer setup? You have something that has been itching you for a long time. It’s easy to solve, and there is often an inexpensive solution to the problem. So go ahead and do it. Pull the trigger and buy the thing you need to flourish. You will feel the benefits immediately.

Automated testing will add value to your software project

We have read enough about TDD and it’s demise in the last year. Since David published his post about how TDD is dead there have been a couple of flame wars concerning TDD and testing in general.

I believe that TDD is a good thing, but I don’t always practice it, as sometimes you don’t have the time to do it. I know, some of you reading this will say that there you must make time for TDD and that TDD is the only way. Maybe you are correct, but in a startup world there is rarely any time for testing at all.

With deadlines and churning new features each week, one can’t make the time to do proper TDD. And sometimes it seems that TDD is some relic from the past, from the really distant past. There is a nice report called Why Most Unit Testing is Waste that sums it up fairly good.

However, I believe in automated testing, at least having a full integration suite, following the application happy path, and any edge case you find later on. Also I’m not against unit testing, if it makes sense. Payment processing code, of course you will test it. Some code deciding if the label class is blue or red, well, you can probably skip that test if you have no time to write it. Rails controller tests are a great example of procrastination in tests. I don’t have anything smart to work on, let’s write a couple useless controller tests.

Unit testing external libraries is another thing, they should be properly tested, to ensure that their API behaves as it claims. Especially if your library is public, then you have to test it.

Having a thorough test suite increases the application value, and decreases the breakability because any subsequent change you make on an application that isn’t tested is like walking through a minefield. You never know what will break.

I would have liked that I learned this lesson the easy way, by listening to other people having issues when some of their code wasn’t tested and they had to change just one little thing, and something completely unrelated broke. However, that wasn’t the case, I learned it the hard way. With a really bad client, who constantly changed their mind about features (another red flag) we were implementing a lot of stuff, and changing it on a daily basis. Having no tests meant that you expected something else will break after deploy, because you just don’t know what can go wrong.

After I got burned by that, I started writing tests, trying to do TDD, but at least covering the process with integration tests as I went along. And it helped a lot, the sheer confidence when deploying the app that nothing will go wrong is really enough. And the client is better off in the long run, because there is no chance that the code breaks, and no one notices it.

Lesson learned: Don’t obsess with TDD or the proper way to test, but try to test the code as well as you can, have an integration/acceptance suite that you run before deploying, and try to cover as much of the app as possible with it. Don’t overdo it, and don’t test the language or framework you are using. Shaving Yaks is really fun sometimes, but don’t do it on a production application, because someone will read it later on, and think that you have to test every little thing.

I attended my first Coderetreat

As the title says, I attended my first (and definitely not last) Coderetreat. And we also made history, because it was the first one in Croatia. It was organized by Željko Filipin and Aljoša Mohorović, and we also had a great facilitator, Péter Zsoldos, who really made us think differently in solving an easy problem, at least it was easy on first sight. Also, his insights into our code and the ways we should think outside of the box, even if it doesn’t make sense in a given situation, really helped. Big thanks to Tentamen for sponsoring the event, and Aljoša for being a great caterer who transformed the sponsorship into enough food and drinks for everyone.

You are probably wondering what a Coderetreat is now. It is an activity that serves to teach us (the programmers) better ways of thinking, while doing TDD, and pair programming all the time. There were about 15 of us, and that was an ideal number. Not to crowded, and everyone had another pair for each session. Working with people you regularly don’t work with, switching between development environments, and sometimes languages can be really hard, if you don’t open yourself to accepting every possible way of stepping out of your comfort zone, so you can learn a thing or two. If you have never been to an event like this, I won’t spoil the fun, but you can always go and read about the Structure of a code retreat on their official website.

The problem to solve is always the same, it’s Conway’s Game of life. A simple mathematical problem, that only has four rules you need to obey. By using a simple enough, but not a trivial problem, one can focus on sharpening the skills as a programmer, which we tend to skip in our day to day lives. We “know” how to do our work, and rarely step out of the comfort zone, and do a thing differently, not because it should be done that way, but for fun, trying different things, and learning new tricks.

I learned a lot there, and had fun pair programming with people I never had a chance to work with, which is great by itself. All of us are quirky in our own way, and adapting to that takes some time, but it makes you a better person, not just a better programmer. I’m looking forward to attending the next event.

Learn a new programming language (basics) fast

While doing my best to understand Emacs, that I have been using as a primary development, writing, and play environment, I have felt the need to learn Emacs Lisp, or elisp. Because I didn’t know anything about lisps, aside from the basics learned while reading the SICP book, which I sadly have not finished yet. I could make a ton of excuses here, but I just found another, more interesting thing to read, or do.

What I wanted to do with elisp, as it is crucial to know it, before you can do any reasonable customization of Emacs, is to learn the most basic elements fast enough, without diving really deep into the language semantics and rules. I just wanted to be able to hack up a few functions, change a thing or two, and not feel like an idiot while looking at other people’s .emacs configuration files.

There is a really nice website I have found, to accomplish just that. To teach you the basics of almost any programming language (even brainfuck), in a reasonable amount of time. One pomodoro should be a reasonable amount of time for that. As there are only a couple of things you really have to know in the start, to grasp a language, and see if it is worth for you to dig in deeper, and explore it.

The site is called Learn X in Y Minutes, and it is a community site, which means, that if your favorite language is not included there, you can submit a pull request with the tutorial, and they will gladly accept it. I have only gone through the elisp tutorial, and it is well made, but as i glanced through some of the other ones, they seem good too.

In the start you should know how to assign variables, create methods, get some feedback in the REPL or the console, and maybe create classes if the language is one of the Object Oriented languages. Of course, I might be missing something, but those are the basics for me personally, to see if the language is worth learning, for me of course, not for you.

The site has made me wonder, if I could maybe come back to the dreadful C, and master it this time, after about a dozen failed trials over the years. I probably won’t be using it to program anything in it, but it is a magical unicorn for me, the thing i never did have the time to learn properly, maybe I will never need it, but there is no 32 year old who can say he won’t be needing a certain skill for the rest of his life. So I won’t say that either.

Here is the site link again, in case you don’t want to look for it in the text above: Learn X in Y Minutes