async and await in Node.js — 100m
This is the second in a three part series on the async and await keywords in a Node.js application. In each part we’ll dive deeper into what these keywords do with the idea being that:
- 1000m view is the stuff you need to keep in your head for day-to-day coding
- 100m view is more in depth and might be useful for debugging certain problems
- 10m+ view is for masochists who really want to know what happens under-the-hood
This is the 100m view so at this stage we’ll dive into some details that are unlikely to help you regularly but which might make you more confident in using the feature (or help to debug something tricky).
Article goals
For the previous post in this series my goal was to give the reader all the information they’d need to work with async/await day to day. In this one the aim is a bit fuzzier: “a more in depth view”.
So let’s make it a bit more concrete — by the end of the article I’d like you to understand how async and await keywords allow you to avoid blocking in a Node.js application. This will mean a quick dive into the Node.js Event Loop and then a brief tour through some async and await code.
Node.js Threading
Node.js applications are single-threaded. Kinda.
What do I mean by threaded? When an application has multiple threads available to it, it can do several things at the same time. If you have two threads available and you receive three requests, you can expect your threads to handle the first two requests simultaneously while the third has to wait in line. Applications with a single thread (single-threaded applications) cannot service multiple requests in parallel in the same way.
Multiple threads can improve an application’s performance and multi-threading is present in many modern programming languages. But it has a big downside: race conditions.
Race conditions arise because each thread in an application accesses a shared pool of memory. A given variable can be written to by two threads simultaneously and when this happens it can be tricky to work out whose value should “win”. Rather than guard against this, Node.js uses a single thread to handle all JavaScript code.
From the time node starts, up until the point it exits, a Node.js application runs run a single infinite loop. This loop is called the Event Loop.
Node.js Event Loop
The Event Loop has a bunch of queues available to it each of which contain tasks. There are various mechanisms for adding tasks onto each of these queues. On every iteration of the loop, it will cycle through these queues and resolve all present tasks. It will handle all tasks in Queue 1, then all tasks in Queue 2, then Queue 3, then Queue 4 then back to Queue 1…
These queues have different jobs: one handles setTimeout() callbacks while a different one handles on-demand service handlers. They are resolved in a specific order. All that said, the details of queue order and which queue does what isn’t really important for understanding async/await.
All you need to know is that any JavaScript you write will be executed on the event loop. If you run an express server and receive a request, that will be fulfilled as an action on this loop. If that server also logs a health check message every minute, that will be done on the very same thread.
This means any set of received requests will be resolved in a predictable sequence dependent on their source and delivery time. With this in mind it’s easy to see how blocking could be a huge issue for Node developers; if your code blocks the only thread then none of your JS code will run during the time it is blocked. If some part of your application is slow then it can hold up your entire application.
Multi Threaded Operations in Node.js
Any code that you write will usually be run on the main thread, but that doesn’t mean that all code is run on the main thread. About 1/3 of Node is written in C++ and when you run a function which is async and backed by C++ this may be run off the main thread. There are two different mechanisms available to Node for doing this: Thread-Pools and kernel-operations.
Thread Pools
When you hash something using the crypto module, the hashing will happen off the main thread using something called the Thread Pool. Let’s consider 5 cases all running on a dual core processor.
- Single hash request
If we make a single hash request we’ll say that takes 1 second for the sake of easy discussion. - Two hash requests run synchronously
If one hash request takes 1 second, then running 2 synchronously will take about 2 seconds. The first request will be processed before work begins on the second. - Two hash requests run asynchronously
In this case both requests will be run in parallel and will be passed off to separate cores on the processor. We can expect both operations to complete in a second as both requests are processed at the same time. - Four hash requests run asynchronously
This time all 4 requests are handled in parallel but there’s only 2 cores available. This means that 2 threads are assigned to core A and 2 threads are assigned to core B. Each core will ping pong back and forth between executing the two tasks assigned to it. Each task takes 2 seconds to complete because it gets 50% of the core’s attention each second. All tasks finish at the same time. - Six hash requests run asynchronously
Something kind of interesting happens here. The first 4 requests are handled as above but the remaining 2 don’t start until the first 2 tasks are complete. These two tasks are split across both cores as in scenario 3 and so each takes 1 second to complete. So at the 2 second mark we can expect the first 4 tasks to complete, and at the 3 second mark we can expect the remaining two tasks to complete.
It turns out that Node has some threads available on which it can run its async C++ code. By default there are 4 threads in this pool though this number can be adjusted through config. When certain asynchronous operations come through Node pushes them onto the thread pool to be handled there. This frees up the main thread to do something else.
Note that there are 4 threads (configurable at startup). It’s not the case that Node spins up a new one each time an async operation comes through, instead it’s stuck with the threads it already has available. This explains why we experienced queueing behaviour in scenario 5.
In addition to hashing, file system interactions are also typically handled on the thread pool along with DNS lookups.
Kernel Operations
Node has another async option available to it. At times it’s able to push tasks onto the kernel for asynchronous operations. For context, the kernel is the low level program one step up from the computer’s hardware.
Node will always opt for kernel execution over Thread Pools where available as the kernel doesn’t have the same limits on thread count. The kernel therefore rarely acts as a bottleneck in the way that the Thread Pool can.
You can see this with network requests (which are asynchronously run on the kernel). If we have a network request that takes 1 second to run, we can expect 6 network requests triggered at the same time to all complete within a second as well.
request 1: {triggered: 15:00:01.000, finished: 15:00:02.000}
request 2: {triggered: 15:00:01.000, finished: 15:00:02.000}
request 3: {triggered: 15:00:01.000, finished: 15:00:02.000}
request 4: {triggered: 15:00:01.000, finished: 15:00:02.000}
request 5: {triggered: 15:00:01.000, finished: 15:00:02.000}
request 6: {triggered: 15:00:01.000, finished: 15:00:02.000}// all requests in a single second, NOT one per second
Our execution speed here will be limited by network bandwidth rather than thread count.
MacroTask vs MicroTask
Earlier I talked about the Event Loop and explained it was made up of a bunch of queues. I didn’t go into much detail when it came to these queues.
The details of queue ordering and separation aren’t important to understand, but it is important to know what happens when a message is dequeued. If we were to zoom in on one of those queues in the diagram we’d find something like this:
Or, if you’d rather see it as pseudocode:
So you have a set of MacroTasks. These can come from multiple sources but for an Express application a common source would be API requests.
We also have something called the MicroTask queue. Whenever a Promise moves to a Fulfilled state, the corresponding callback is added to the MicroTask queue. The MicroTask queue is shared amongst all MacroTask queues and callbacks can be added from multiple sources:
- The Thread Pool
- The kernel
- The MacroTask callback
- A MicroTask callback
One distinction of note between the MicroTask and MacroTask queues is the way newly added tasks are handled.
Assume there are two tasks currently in the MacroTask queue. When we come to handle this queue we dequeue these two tasks and commit to resolving them on the current iteration of the Event Loop. In my diagram/pseudocode I’ve interpreted this as a MacroTask snapshot but I’m not actually sure of the real implementation details here. Of note though is that if new requests come through they won’t be handled until the next Event Loop iteration.
The MicroTask queue works differently. MicroTask callbacks are continuously resolved until the queue is empty. This means it’s possible to set up an infinite loop in the MicroTask queue as we’ll see in the second example below.
Example 1
A simple demo to illustrate the relationship between MacroTasks and MicroTasks:
Explanation
This example prints 1, 2, 3, 4. With what we’ve learned about the Event Loop we should now be able to understand the mechanics behind this output.
Step 1:
My code is added as a callback to one of the Event Loop’s MacroTask queues. When we come to resolve this callback we start executing asyncFunction() on line 16. This starts executing right away and prints 1 to the console on line 8.
Step 2:
When we reach line 9 the promise body starts executing straight away and 2 is printed to the console. The promise then resolves immediately. This is an important point, the body of a promise executes as part of the current task, only the callback is deferred.
Step 3:
Now things are kind of interesting. The promise has resolved already but we awaited it. When a Promise we are awaiting is resolved a callback is added to the MicroTask queue and that’s just what happens now. The remainder of asyncFunction() is added to the MicroTask queue to be executed after the MacroTask is done. In this case the remainder of asyncFunction() is just line 13.
Step 4:asyncFunction() yields control to the parent at this point. Execution will continue in the MicroTask loop. We’re not awaiting asyncFunction() on line 16 so we’re free to move on. We do so and print 3 on line 17
Step 5:
The MacroTask has finished executing so we can move on to resolve MicroTasks. Our callback from step 3 is resolved here and 4 is printed to the console.
Example 2
A slightly more complex example to show that the MicroTask loop does not exit until that queue is empty
Explanation
Here I’ve triggered an infinite loop of MicroTask resolution. In callStuff()we resolve the “Hold my beer” promise. That adds a callback to the MicroTask queue but because we’re not awaiting that promise we can move on to log “Something bad is about to happen…”.
Next we resolve the “Hold my beer too” Promise in callStuffAsync() this adds another callback to the MicroTask queue. But because we await that latter Promise in callStuffAsync Node moves on to try to empty the MicroTask queue.
It’s not able to do this because both promises add copies of themselves to the MicroTask queue as part of the callback. So we end up in an infinite loop of:
Hold my beer
Hold my beer too
Hold my beer
Hold my beer too
Hold my beer
Hold my beer too
...The value of async/await
So with what we’ve now learned about Node we should be able to see how Promises and async/await are useful.
- Node has a single thread available for JS code so it’s important to keep it active whenever possible
- When handling certain long-running operations like network requests or hashing, Node can push the work off the main thread onto the kernel or Thread Pool.
- While Node is awaiting the completion of these actions it moves on from the awaiting task to handle other tasks
- When the outsourced work is complete, a new task is added to the MicroTask queue for the main thread to pick up
Other Articles in this series
Async and await in Node.js — 1000m
