Use the database replica you are already paying for

I just completed a long data migration on production and while monitoring the pending migration, I noticed an interesting trend in our database usage metrics. Our database read replica was using around half of the CPU compared to the primary. And there are few things I hate more than not using stuff I pay for to the full extent. Especially if it’s a recurring cost.

You know, a responsible thing is to have redundancy in as many systems as possible, especially when it comes to data. And read only replicas (in a multi-zone setup) are one of the easy ways to achieve it. Another benefit of using a managed database provider like AWS RDS for us is that you can add and remove multiple read replicas very fast when necessary. Especially worthy if you have a seasonal business that gets 1000X more traffic only at a specific period of the year.

Disclaimer: the following stuff is Ruby on Rails related, because lo and behold, I use Ruby on Rails. But the approach works in any framework you are using.

And because we are using Ruby on Rails for the backend, a lot of the multi-database setup comes baked in. It even has the automatic role switching which routes all queries coming to POST, PUT, PATCH, or DELETE methods to the read replica, and the GET and HEAD ones go to the reader. If I started a new Rails application today, I would use this from the start and possibly wrap writes on the non-modifying endpoints into a deliberate action.

Like many products out there, there are times when we write things to the database during GET requests. I know some people are very strict about REST verbs, but we are being pragmatic and although it would be convenient in certain situations, in some others it’s not necessary to be super strict about it. Sometimes you just want to log things for audit or analytics purposes. But this is easily solvable with a helper in ApplicationRecord that allows you to wrap a piece of code and send it directly to the read or write replica.

class ApplicationRecord << ActiveRecord::Base
  connects_to database: { writing: :writer,  reading: :reader }

  def self.run_on_db_reader(&)
    ActiveRecord::Base.connected_to(role: :reading, prevent_writes: true, &)
  end
  
  def self.run_on_db_writer(&)
    ActiveRecord::Base.connected_to(role: :writing, &)
  end
end

Now you are able to call this from any part of the codebase and move the database queries to the preferred database. Since the majority of traffic happening on any web application is read, you can use this approach to increase throughput just by adding as many read replicas as you need/can afford. Again very easy to do on AWS.

One aside here is that we made the deliberate choice to have the reader as the named target, while the writer is default for everything. Although in most situations we are just reading stuff from the database, the person adding new code has to clearly state that this goes to the reader. While I’m writing this, I can see the logic flaw behind this, and it would be better if we had to consciously say that this little piece of code will write to the db and everything else just reads from it. Luckily when using an ORM you can read the object from the reader and then fire an update on the writer if necessary. I don’t want to go into concurrent updates, race conditions or multiple people working on the same record.

To prevent this from blowing up in your face on production, you have to set it up in development and test environments. With that, you can quickly figure out which endpoints could be moved to the reader, either by having a good spec coverage or by making the change and testing the app manually. I highly suggest having a great coverage. You can also use your AI harness of choice to do the same for you. Since there can only be one primary instance, we are always bottlenecked on the amount of writes that are happening on that instance. However, we don’t have to suffocate this database with all of the read traffic if we are already paying for the second (third, fourth) database replica. And AWS or whatever is running behind your cluster will load balance read requests over the read replicas. This way we are achieving a few things, our traffic doesn’t only go to one database instance and we use all the resources that we paid for. Database replicas are a type of insurance policy, but compared to most insurance you are paying for and not using, you can and should be using them to their full extent.

The only thing I want to mention here to be careful with is the replication latency. So you shouldn’t read from the replica immediately after a write happens. This can happen in some situations when the user does a write action, and is immediately redirected to a read action. If your server is faster than the database replicates, the user might receive a stale record. Luckily for us, the automated rails setup already adds a 2-second delay to allow this replication to happen. In my experience 2 seconds is more than enough because the replication latency is measured in single and lower double-digit millisecond values, so the safety net is huge.

If you liked what you read, sign up below to be notified when I write again on this subject. If you would like a free 30 minute consultation on how to best use the expensive resources you are already paying for, or pay less for the same performance, book a free call here