Saturday, October 20, 2012

Simply Explained: Multithreaded vs. Async as the hamburger restaurant



Recently I have been lurking around the Node.js IRC channel and I saw that many people start using Node.js without realizing the main differences between the multithreaded approach and the asynchronous approach. This post will explain this difference through a simple (non computer oriented) example.

Suppose you are starting a hamburger fast food restaurant. In your restaurant you have only one dish: A hamburger. It is made by strictly following these steps:

  1. When a customer comes in the clerk takes his order (10 sec).
  2. The clerk puts a meatball on the grill and roasts it (30 sec).
  3. The clerk puts the buns in the oven and warms them (30 sec).
  4. The clerk assembles the hamburger and serves it to the customer (10 sec).

This recipe may not result in the best of hamburgers but it’s a start. In our example we have two critical resources (marked bold), the grill and the oven. When each of these devices is in use, no one else can use them. The clerk represents our thread and the recipe our code. The clerk follows the recipe to the letter in the same way our code is executing on a CPU.

--- What happens in a single threaded environment?

We have one clerk. When he puts the meatball in the grill he has to wait near the grill (doing nothing) while the meatball is fully roasted, then he takes the meatball out and can proceed to the oven. He puts the buns in the oven and again… waits… Until the buns are warm. The entire hamburger making process takes 80 seconds (10 + 30 + 30 + 10).

This is not bad for a single customer. In fact, the fastest any customer can be served a hamburger is 80 seconds so in the case when only one customer comes to the restaurant we are doing a perfect job serving him fast. The problem starts when there are several customers coming in. Consider the case when two customers are coming in together. The clerk has to follow the recipe for both customers, the first customer  gets a hamburger after 80 seconds while the second customer after 160 seconds (since the clerk can start making the second hamburger only after he is done with the first). Each additional customer would have to wait additional 80 seconds for any other customer standing before him in the line. The next diagram depicts serving one customer by a single clerk.


To solve the aforementioned problem we usually use a multithreaded model. In our example we simply add another clerk. In this case when two customers come into the restaurant each customer will be served by a different clerk. On first glance you might think that now each customer gets a hamburger after 80 seconds but this is not the case. Remember the critical resources? Both clerks take the order at the same time but only one of the clerks can use the grill (while the other waits for the grill to be available). The diagram for two clerks serving two customers looks like this:


The first customer still gets the hamburger after 80 seconds but the second customer gets his after 110 seconds, 50 seconds faster than in the single threaded case. From this example we can deduce two things:

  1. If more than two customers come we need more clerks. Clearly, there is a limit to the number of clerks we can have (and that limit is probably not very high). More clerks mean a crowded kitchen and at some point they start bumping into each other and even taking the wrong meatballs from the grill. This happens in software too. The more threads you have the more time you computer spends on switching between them and the more time you spend on debugging various race conditions .
  2. No matter how many clerks we hire we still have only one grill and one oven so at some point adding clerks will have no added value (in fact it will just cause more problems, see item 1). This happens in software too. So what do we do? We buy more grills… errr… RAM, CPUs, storage, bandwidth etc... Our handling capacity is directly linked to the hardware upgrades we are making.

--- Where does the asynchronous model comes in?

In the asynchronous model we make a small change to our kitchen equipment. Instead of hiring more clerks we have only the one clerk but he doesn’t wait at the critical resources. Instead, the grill and the oven have an input box and an output box each. When the clerk wants to use the grill, for example, he puts the meatball in the input box and goes away to do anything else. When the meatball is ready (after 30 seconds) it drops into the output box and the grill rings a bell. The clerk can pickup the meatball and continue the recipe. The diagram for serving two customers in this model looks like this:

Drawing3 (3)

Note the colors: White – the clerk is actually doing some work, Grey – the clerk is not doing anything, Yellow – the machines are doing the actual work.

We can see that the second customer still gets his hamburger after 110 seconds with only one clerk. We can also see that during these 110 seconds the clerk was occupied for 40 seconds and the rest of the time he was waiting for additional customers to come in (we didn’t have that slack time in the multithreaded model).

--- Wait! What? Making the algorithm async didn’t make it faster?

No! Async will not make your algorithm run faster, if anything it will make it run slower.

--- Then why people use Node.js which works in the async model?

The answer is in those Grey areas in the last diagram. Think about the second example (with the multithreaded model), if another customer comes in and wants to order ice-cream (which takes 10 seconds to make and requires the ice-cream machine). The customer would have to wait 120 seconds to get the ice-cream simply because there is no clerk to take his order. In the async model it would take 30 seconds because the single clerk can take the order 20 seconds into handling the first two customers.

The main point here is that you can’t take the clerk from the second example and put it to work as the clerk in the third example.  He simply would not understand why the machines are making all those sounds. In the software world, you can’t take code written for a multithreaded server, translate it to JavaScript and let it run on Node.js because it doesn’t work in the async model. Doing so means that you are back to the single threaded model of the first example which would certainly cause severe degradation in the response times of your server.

So what should you do to make your Node.js server as responsive as possible? (Not as fast as possible!)

  1. Use the async framework that Node.js provides for you and avoid using the sync methods.
  2. Make your code async by implementing your own async calls or using the calls of Node.js APIs.
  3. Make the non async code section as short as possible. Look at the ice-cream example. If we could reduce the “Take order” step into 5 steps that take 2 second each then the third customer could get the ice-cream faster (14 seconds).

Conclusion, Node.js with its async nature is a great tool but as with any other tool it requires understanding of how and when it should be used. Node.js is not a magic solution that makes any server run faster but a framework that allows you to build a server with improved response times for high number of users without considerably increasing the hardware capacity.

Thank you for reading. Please write your thoughts in the comments.