Your users are out to get you

How many times did you create a UI with a specific user flow in mind, and most of the time your users did something else? Did you ever experience users ignoring clear warnings and proceeding to do something dangerous to their data. Can we prevent this?

A user will always try to find shortcuts and model the app usage to what they seem fit. And it rarely happens that the end user uses the app in the exact way that you planned. If your app does allow users to skip any step in the flow, and do anything out of the boxed process you imagined, they will do it. Not all the time, but in your professional life you will encounter what I like to call creative users.

Now, I’m not saying that all users are ignorant, stupid, or criminal. But you must have that in mind when creating an application flow. Ideal user input that I can think of would be one button on an otherwise blank page. But even that imposes a couple of questions: How do we know who pressed the button? Did we state what the button does? Does the end user understand the language? If we do know who pressed the button, are they allowed to press it? What does the button do? Questions, questions…

There are also hackers, benevolent and malicious. And there are idiots that download hacking tools from various forums, playing stupid games. Now, I think that in a broader sense, benevolent hackers are doing great work for the community. You can’t plug all leaking holes in the thing you are building. It goes on a much greater scale when talking about libraries or web servers, ssl and encryption. While malicious hackers are after the money, they also expose holes that you learn how to plug. To be honest, no education is free, sometimes you pay less, sometimes more.

This is why having a good security practices works in the long term. Using a framework like Ruby on Rails solves a lot of that stuff for you. But don’t forget that you still have to have at least a grain of common sense. Even Rails allows you to shoot yourself in the foot, if you don’t follow the guidelines. Also don’t forget to upgrade to the latest security fix, or a maintained branch. Upgrading the frameworks or libraries to the supported versions might seem like a tedious thing to do, it’s the only smart option out there.

There are some drawbacks in using open source software. The number of solutions out there is huge. It’s hard to pick and choose the correct solution. I could suggest going with Ruby on Rails here, but that’s just my pick. Open source is also a great pool for learning new things. If you don’t seem to like one approach, nothing is forcing you not to create and publish your own solution. Just be sure it’s documented and tested, you created it because you weren’t satisfied with some other solution out there, right?

Back to (non-)malicious users again, that form you made to enter the email twice to verify it, it doesn’t work like that. I’ve experienced someone that used my own personal email to set up their Facebook account. My email wasn’t hacked (I hope), Facebook’s email verification sucks). I’ve since hijacked and archived the account, so no one can reuse my email to register again.

Now imagine a benevolent user, registering for your web service, and they enter a wrong email. I’ve worked with applications that have non computer savvy users, and that happens a lot. You make a typo, whatever, just while entering your email to register. Even my father who has been using a computer at work for the last 16 years or so still has troubles with typo’s. It’s not second nature to him, as it is to (most of the) computer professionals out there. And I believe you, the reader, are one of them.

Don’t judge a man until you have walked a mile in their shoes. Almost every culture has a variant of this proverb, and for a reason. The main premise is to do your best to see and feel what the other person is seeing, or feeling. Try to replicate their mindset while using your application. It’s easy creating applications for IT professionals. But when you realize that this is only a small percentage of the general population, the light bulb in your head might go off. Then you can think about a better solution to do the same once difficult task that only a few people understand.

Billionaire empires are created on simplifying complex stuff. Google simplified search and email. Apple simplified smartphones, and obtaining music before that (well at least legally). but there are lots of smaller examples, and you just have to look around to notice them. Complex UI’s have no place anywhere in the world, not even in airplanes or space shuttles. You need a lot of them, and sometimes you must have knobs and gouges somewhere where you wouldn’t like to have them. But if 99% of your users aren’t using them, maybe you can at least hide them from plain sight. You could even remove them, and get on with the fact that 1% of your users might leave. But the other 99% will have a much better experience.

Issue with Rails's restrict_with_exception and counter_cache

When building Rails applications, a lot of focus goes into optimisation. One of those optimizations is using Rails counter_cache columns. The reasoning behind it is simple, you have a relation e.g. Post to Comments and you want to display the comments count for each post. One option is to always count the comments for each post with something like @post.comments.count of course, this is a really expensive operation, and if you are displaying more than one post on the page, it will cause an N+1 issue. Also going to the database just to count how many rows a relation has, isn’t really the best idea.

Luckily Rails has its own optimisation, called counter_cache. It is very simple to include in the model, and Rails manages all of the magic for you behind the scenes. Below is the example of Post and the cached comments count. The only prerequisite that you have to do, is to add a comments_count integer column, with default value of 0.

#in a migration
def change
  add_column :posts, :comments_count, :integer, default: 0
end

class Post < ActiveRecord::Base
  has_many :comments
end

class Comment < ActiveRecord::Base
  belongs_to :post, counter_cache: true
end

This way, via the Rails Magic™, you will get the count column incremented whenever a new comment is posted. And decremented when a comment is destroyed (if you support such a thing). Now you don’t have to count all of the comments when showing the number next to the post, which will be a pretty big performance boost (relatively speaking).

That is the first part of the equation, as the title clearly states this post has something to do about restrict_with_exception. That is the dependency management type, which disallows you to destroy a record that has child records, with that orphaning those records, and creating database inconsistencies. In our case, a post with comments.

We have to modify the Post class to accomplish this:

class Post < ActiveRecord::Base
  has_many :comments, dependent: :restrict_with_exception
end

Now every time you call destroy on a Post model that has comments, Rails won’t let you do that, and it will raise an exception, which is the desired behaviour.

Now, we avoid rails and use regular SQL when the situation needs us to. Let’s say moving comments from one post to another. It can be done with

@post.comments.each do |comment|
  comment.update_attributes(post_id: @new_post.id)
end

But for a post with a lot of comments this operation can be too slow. Moving comments with a sql update is much easier and faster way to accomplish the same thing, but not without it’s own pitfalls. @post.comments.update_all(post_id: @new_post.id) will update your comments and set them to the new post, and you will probably want to refresh the counter_cache on the @new_post, so you have to do something like this Post.reset_counters(@new_post.id, :comments).

Now if you forget doing the same thing on the old post, and just try to destroy it, you will fins yourself chasing ghosts. Rails somehow takes the counter cache column as an authority, before even looking if there are associated records there. This is a cool performance hack, and by going only through Rails, and not pure SQL, this situation wouldn’t happen at all.

Sometimes we can’t or don’t need Rails call backs and validations, just to transfer the ownership from one model to another. Then weird things start to happen, and you don’t know why, so you lose a day debugging what should be a simple thing.

Send PDF attachments from Rails with WickedPdf and ActionMailer

In almost any web application you create, the question of generating PDF files will pop up pretty soon. There are a couple of options while using Ruby on Rails. First one is Prawn which is a pure PDF generator library, and the other options are mostly wrappers around the very popular wkhtmltopdf unix library that converts html pages to pdf. There is also an option that uses PrinceXML but I would initially exclude it because it is pretty pricey for an SME.

While using Prawn gives you all of the formatting power when creating PDF files, it’s learning curve is pretty steep, and the option of converting html to a PDF seems pretty good. This is especially the case when you already have html templates that only need some small modifications to be able to convert them to PDF.

There are a couple of gems that wrap around the wkhtmltopdf library, and ruby-toolbox pdf generation section shows some more viable options that can be used when generating PDF files from ruby.

I’ll be using WickedPdf in this example, as that is the one I have set up this system with a couple of times already, so I’m pretty confident it will work in any other Rails application.

Setup

As always, using a ruby gem in rails is pretty simple, you just add a couple of lines to the Gemfile

gem 'wicked_pdf'
#we need the new binary here, so that we can be OS independent
gem 'wkhtmltopdf-binary', github: 'pallymore/wkhtmltopdf-binary-edge', tag: 'v0.12.2'

Usage

This setup will work pretty straightforward in the controllers, because WickedPdf registers :pdf request format, and you can respond to it in the same fashion as html or js in a respond_to block. Code below is copied from the WickedPdf Readme page.

class ThingsController < ApplicationController
  def show
    respond_to do |format|
      format.html
      format.pdf do
        render pdf: "file_name" # Excluding ".pdf" extension.
      end
    end
  end
end

That part was pretty easy, and you can configure the pdf render with a lot of options that are mentioned in the advanced configuration section of the WickedPdf Readme

Generating PDF from ActionMailer

The issue when trying to generate the PDF from ActionMailer is that there is no request, so you have to render the html in a different way. In a nutshell, you want to render the template to a string, and then forward that string (html) to the wkhtmltopdf. This is the same way the gem work when rendering pdf from the controller, but it is worth mentioning once more.

We will use a method that exists on the AbstractController::Rendering and it is called render_to_string So we can do something like this in the mailer:

class TodoMailer < ActionMailer::Base
  #...
  def pdf_attachment_method(todo_id)
    todo = Todo.find(todo_id)
    attachments["todo_#{todo.id}.pdf"] = WickedPdf.new.pdf_from_string(
      render_to_string(pdf: 'todo', template: 'todo.pdf.erb', layout: 'pdf.html'), { :hash_with_wickedpdf_options }
    )
    mail(to: todo.owner.email, subject: 'Your todo PDF is attached', todo: todo)
  end
end

By using this approach, we can easily send a PDF attachment from an ActionMailer method. You can do this for any Rails template you want, but be sure to test everything before putting it into production.

Notify an Ember.js app that a deployment has happened

The approach with refreshing regular web applications after deployment is pretty straightforward, you click on a link, it goes to the server, server renders a page. If the deployment has happened in the meantime, server renders you the current(new) version of the page. The issue with Ember.js is a pretty nice example of how the old approach doesn’t work anymore. You get the whole application when you first request the page, then everything happens in the browser. The only thing that goes between server and the client is the JSON being passed here and there.

Luckily we are already using Sam Saffron’s awesome gem message_bus, to notify the front-end that an update has happened in a model we are working on. So I decided to implement deployment notification using a similar approach.

First, we want to be able for the Ember app to know that a deployment has happened, so registering MessageBus and subscribing it to a channel is the first thing we will do. We chose the ApplicationController for this in Ember, because it is always loaded, regardless of the route. We are using ember-cli-growl component to provide us with growl like notifications, so you might need to adapt the code to suit your needs.

/* global MessageBus */
import Ember from 'ember';

export default Ember.Controller.extend({
  initMessageBus: function() {
   MessageBus.start();
   MessageBus.callbackInterval = 50;
  },
  subscribe: function() {
    var channel = '/deployment';
    var controller = this;
    MessageBus.subscribe(channel, function() {
      controller.growl.info('A new application version has been deployed, please reload the browser page.', { clickToDismiss: true });
    });
  },
  init: function() {
   this._super();
   this.initMessageBus();
   this.subscribe();
  }
});

In Rails, include the message_bus gem in the Gemfile, bundle install and run the console. Now you can try running MessageBus.publish('/deployment', 'foo') and you should see a notification pop up in your ember app. Ok, but we need to be able to call it somehow, and what better way to do it than using Rake.

desc "notifies Ember that a deploy has happened"
task notify_ember: :environment
  MessageBus.publish('/deployment', '')
end

Now you only have to call that task on the server when your deployment has finished and everything will be fine and dandy. There are a lot of options to do this, depending of your deployment strategies, and infrastructure. We are using Capistrano, so a nice after hook, running after the app server has restarted did the job quite well. If you are having trouble setting it up, feel free to ping me in the contact form.

Speed up Rails on Heroku by using Rack::Zippy

I really like Heroku for fast deployments and everything they offer for a small Ruby on Rails or node.js apps. It’s so easy to deploy a web app there, and it’s pretty much free unless you are doing some serious stuff, by when you should already be paying either Heroku, or have your own dedicated solution when you’re all grown up.

There is one thing that I personally dislike about Heroku, and that is their asset serving. As we are all sitting behind a desktop(laptop) computer, and have pretty good network connections, we don’t see how slow the app is for someone on 3G mobile network, or something even slower. And in a well written app, the main culprit for large data transfers are the assets.

We have a lot of JavaScript, Css and images in our apps, because we want to make them beautiful, but that takes it’s toll on network bandwidth. Luckily most browsers support compressed assets(js and css) and know how to decompress them. And when you build your own server, there is a very high probability that assets won’t be served through Rails at all. You will either use NGINX or Apache httpd to act as a reverse proxy, and also serve your assets for you. The web server is smart enough to serve compressed assets to browsers that support them, and uncompressed to others.

Rails has a pretty good ActionDispatch::Static rack middleware that does all the work for us while in development, or on heroku by changing a config flag, or using their rails_12factor gem in production. To get it running you have to change the static flag inside the config/environments/production.rb

 # rails 4.1 and below
 config.serve_static_assets = true
 # rails 4.2 and above
 config.serve_static_files = true

For those on a budget, and not able to afford to serve assets from Amazon S3 or another CDN (will update here when my post on how to do that is finished), there is an option on serving the compressed assets from rails. They are already generated by heroku on deploy, but just aren’t being served, which considerably increases your initial page load time, and every additional one, if the assets aren’t being properly cached. ` You can solve this pretty big issue with including one gem that will serve compressed assets for you. It’s called Rack::Zippy and it’s really easy to configure. You have to add it to the Gemfile first:

gem 'rack-zippy'

Run the bundle command and create the initializer for Rack::Zippy in config/initializers/rack_zippy.rb like this:

Rails.application.config.middleware.swap(ActionDispatch::Static, Rack::Zippy::AssetServer)

This will use Rack::Zippy to server gzipped assets instead of uncompressed ones and really reduce your initial load time, especially on slow connections. It also adds pretty sane cache expiration headers which can also help with the browser caching.

A not so funny orphaning issue or Rails's has_one relationships

If you are using ActiveRecord and keeping your models pretty thin and properly normalized, you will surely reach for the very nice has_one relationship. It’s basically a one to many relationship, but Rails thinks for you and keeps only one child model available for use. It’s pretty easy to set up, given that you have two models, let’s say Post and Post::Metadata

class Post < ActiveRecord::Base
 has_one :post_metadata
end
class Post::Metadata
 belongs_to :post
end

OK, this part is pretty straightforward, you create your records either by using Post::Metadata.create(post_id: 42, other_attrs) or by using Post.find(42).create_post_metadata(attrs). If there already is a has_one child record, and you create a new one, the old one will be orphaned, which is a default option, and you should be thinking of it before you start messing around. This can also be the desired option, maybe you are storing some photo urls that aren’t being used anymore, and have a callback on them to destroy the record after you remove the file from the third-party storage. There is one tiny kink that cost me a few hours of life today, although you create a child record in a has_one relationship, the mere action of building an association (just instantiating it, not creating) is enough to orphan the previous record, without it rolling back if the newly built record is destroyed.

Use Rack::Deflater to get faster first time loads of your app

Is your Ruby on Rails application slow for end users? Maybe you are sending a lot of data through the network. As we rarely test performance on our basic server to client setup, and I don’t mean the maxed out broadband speeds we like to have at our places of work, but regular DSL, or mobile internet user, with the minimal internet speed available.
We also have the false security while browser testing the application on the same machine we develop it on. And it’s really fast to fetch 10MB from localhost:3000 into the browser.
Do you think that pulling even a 1MB response is nice when the DSL speed is 2mbps and that means around 4 seconds to fetch it. Only your assets can grow to that size if you are not careful, and this is far from the response size of larger websites. For example, youtube.com has a payload of ~1.5MB (at least at the time of writing, my front page). And it takes around 4.55 seconds to render on my DSL which is around 10mbps. For a regular DSL user (~2mbps) the time it takes is 8 seconds, and the difference is, of course, the time it takes to receive the 1.5MB of data. For a mobile user it takes even longer because let’s be honest, no one can achieve those claimed 150mbps LTE speeds.
There is a very quick solution to reduce the payload of your Rails (and other Rack based apps) by including just one line of code in yours config/application.rb.

config.middleware.use Rack::Deflater

That will automatically deflate your server responses (which is a fancy word for compression that the browser knows how to uncompress), and you will be serving substantially smaller responses. Of course, your web server must be configured to enable compression, and there is a great guide for that here.

Learning Ruby and Rails in 2014

In the last few weeks, I have been approached by a couple of people who are interested in learning Ruby and/or Ruby on Rails. And for their sake, and for my own interest, I investigated what are the best resources at the moment. Ruby and Rails ecosystem is a one that evolves really fast, and while I like it a lot, because we have new toys to play with all of the time, it is not a great thing for new developers. Also Rails has grown to this big behemoth, and although I like Rails a lot, and will continue using it as long as I can, I’m not sure it is newbie friendly. But I’ll address that one later.

Before i list the resources, and their brief descriptions, there are a couple of rules you should follow, to have an easy ride with Ruby on rails. A basic understanding of HTML/CSS is needed, because, you are learning a web framework, which incidentally runs in the browser, which incidentally uses HTML/CSS to render the pages. I’m not telling you that you need to be a frontend developer/designer, just to have a basic knowledge of it. Ok, no more philosophy from now on. You should also be acquainted with JavaScript, not that you need to be an expert in it, but just have a basic knowledge of it. You are going to use it if you want to have any dynamic interactions in your web applications, but it is not as crucial as HTML/CSS. After you have your prerequisites in order (and it will probably take you a couple of days to get acquainted with HTML/CSS and JavaScript), you can start learning a new programming language, and a web framework. The basics will come in handy if decide to use any other programming language and framework, if your goal is to create web applications.

Learning Ruby on Rails is mostly done backwards, at least that is the way I’ve done it myself. I first “learned” Ruby on Rails, then started learning Ruby, because it was pretty stupid not being skilled in a language that runs your favorite web framework. Who would of thought that by learning the language, you get to understand the framework better. You should not be a fool that I was, and learn the language first. Also, learning Ruby on Rails is a really big bite to chew all at once. Here is a brief overview of the things you are expected to learn all at once:

  • A new language
  • A whole web framework and its abstractions
  • HTML/CSS (if you haven’t done web development before)
  • JavaScript (or CoffeeScript and its abstractions)
  • Probably much more stuff that I don’t remember anymore

Learning Ruby

First thing you want to know is how Ruby syntax looks like, and how it behaves. I think that the best possible way to find out is to Try Ruby. This is a great interactive tutorial, which helps you to learn the language basics. There is also one more site, which I’ve been using lately, to get an insight into other programming languages: Learn X in Y minutes and specifically their Ruby tutorial. After you have done the ruby tutorials, and at least grasped the language basics, you will be able to decide if you like it or not. If you chose not to experience the awesomeness of Ruby, and everything that comes with it, that is OK. To each their own. If you want to continue learning ruby, this is probably the best book there is: Programming Ruby or colloquially called “The Pickaxe book”, from which you will not only learn about the Ruby programming language, but there are also three Regex chapters from which you can learn to be a wizard.

Learning Ruby web frameworks

After you have learned Ruby, the next step on your itinerary is learning Ruby on Rails. I would like you to make a short break here, and learn something a bit closer to the metal. Sinatra is a great small Ruby web framework which will teach you how to build small and simple sites or API endpoints. Also, it is possible to mount Sinatra apps in Rails, and the Rails learning curve will be easier for you if you already know Sinatra. And maybe you decide that Sinatra, with some added libraries is enough for your project. So as a next step, I want you to look into Padrino which is built upon Sinatra, and it also gives you a predefined way to build your web applications.

Learning Ruby on Rails

Learning Ruby on Rails is a big task, and if you set out to do it, it can take you 15 minutes to build a blog, and probably half a decade to master everything that Rails can (and will be able to) do. In my opinion, the best resource to get you on the way is The Ruby on Rails Tutorial by Michael Hartl. A book (and videos) that have helped mold plenty of Ruby on Rails developers. There is also a free online version, if you are on a budget. There is of course the official Ruby on Rails Guides, and one of my favorite books on Rails, The Rails 4 Way which is a great resource, and each time I get through it, I find another new thing I should be using. There is (was) a great screencast resource packaged in Railscasts. Although it isn’t updated anymore (I really hope Ryan gets back soon), it is still a great asset for solving most of the specific problems you will encounter as a Rails developer.

I have focused mostly on free resources, although some of the books I recommend come with a price tag attached. The price, although it might seem a lot to you, is really reasonable, and is much lower than the value these books provide. Even if you don’t stick with Ruby, owning The Pickaxe Book, and going through the first part of it, will teach you some stuff you can probably employ in your language of choice.

I however, chose Ruby, and I’m happy with it. And it doesn’t mean I am, or will only be doing Ruby for the rest of my life (I probably could, if you don’t count JavaScript), but that Ruby on Rails solves the database backed web application problem in the best way I can imagine. And if that isn’t your game, you probably didn’t get to this part of the post. If you have, please try Ruby, it is a beautiful programming language that was designed to make developers happy.

Rails No-No's: Not using background processing

Have you encountered a website with a very slow response? How do you feel when you have to wait for 5 or more seconds, for the page to even start responding? How do you feel when the app you are waiting for ages, is something you are paying for on a monthly basis? Fuck this, let’s try something different? Yep, that is the same thing your users (customers) are feeling when your app is waiting for something. And users have the tendency of voting with their wallets, which in this case, can make your much thinner than it is at the moment.

First and foremost issue is sending emails, when the user registers or when they ask for a password renewal. Those situations can even stand sending emails in sync, but you are blocking the web process thread for new requests while the email processing ends. Let’s consider a situation where you have some larger data massaging operation, that hides between a single form submit. Take for example large business reports, which have to collect data from a number of different data sources, both internal and external. Now, this process can take from a couple of seconds, to hours if your data is not well optimized. You are basically wasting that person’s time on waiting for the report to finish. Now multiply the average time that a person waits, with their hourly rate and that with the average number of reports someone waits for, and with the number of people running such reports in a given company. That is a serious waste of money right there, now if you could somehow optimize this process, you could stand to easily save $10,000+ on lost time, and more over the lifespan of the application.

Imagine if the users abandoned your shopping cart, while it was processing something that could have ran in the background. Now you are losing real sales, and your client is on their way of either firing you, or going broke.

OK, enough melodrama for now. Let us now consider a different option from doing everything in one process in sync. You have the option of threading, but that won’t get you very far, because you are still blocking the user’s flow, while they wait for the response. Enter background workers, which are best run on a separate machine, and they have queue of tasks that need to be performed. A set number of processes which take handle of those tasks, and report back(if needed) when the task is finished. Here are a few options for background jobs in Ruby on Rails:

  • Sidekiq – Multi threaded background processing built on Celluloid::IO and actor based model
  • Delayed::Job – One of the first, if not the first background workers for rails, extracted from Shopify
  • Resque – Background worker using redis instead of the database for queuing jobs

I’ve been using all of them over the years, but as of a year ago, I have been using only Sidekiq on new and migrated some old applications from other workers to it. The benefit of Sidekiq is using actor model and threads to process jobs concurrently, it has a low memory footprint and can be used to boost your application performance, and responsiveness. If you are on Heroku, this will cost you one worker instance per month, as that should be enough to process a very large number of background jobs. If you are running your own stack, one server with the minimum hardware requirements will suffice. No one cares if their email is delivered in 3 minutes instead of right now. But they do care if they can’t do anything for that 3 minutes while waiting for “something to finish”. Here is a basic example of how it’s done with Sidekiq:

class UserGeoLocator
  include Sidekiq::Worker

  def perform(user_id)
    user = User.find(user_id)
    user.perform_long_geolocation_api_call!
  end
end

You can find out all the things it can be used for in the Sidekiq Wiki on Github.

Rails No-No's: Using inline CoffeeScript with Haml

I’ve always had a love-hate relationship with HAML, partly because if the css is clean, it can be a great thing, and if it isn’t then your markup will look really ugly. But that is not the thing I’m writing about now. This is about something else, using inline CoffeeScript inside Haml templates. As you know, you really shouldn’t be using inline JavaScript at all, but you can get by, especially if you have some weird use cases. The thing with CoffeeScript is that  it has to get compiled each time into JavaScript, which is fine, if you are using something like Sprockets to compile it on deployment, but if you are using the feature Haml provides you to shoot yourself in the foot, you can cause some serious performance issues for your site.
Consider the two following snippet:

#content
  %h1 Some nice headline

:coffeescript
  $('h1').click ->
  alert('You clicked on the headline')

If you put that in the template, you are causing yourself with a hit of about 100-200ms that takes to compile that coffescript into javascript, and then inline it. But if you try doing this:

#content
  %h1 Some nice headline

:javascript
  $('h1').click(function() {
    alert('You clicked on the headline');
  };

Then the performance hit is mostly gone. You still have a code smell in your views, but not a one that will cause a drastic performance loss if you don’t remove it right away.
But if you consider a situation in which you render nested partials, and each one of them has some inline coffeescript, which I myself have done unknowingly before, that will multiply the issue with the amount of rendered partials. You can of course count on rails caching to save your ass, and it will most of the times, but not always. And then you will have that one customer who waits for more than 10 seconds to get their page.

If by any means you want to reduce the maintenance nightmare, you will be very fast to remove inline javascript from the code all together, which will most probably also optimise your javascript code, and make it more maintainable in the future.