Nodejs serial and parallel processing

Nodejs serial and parallel processing

Some time ago i ran into some requirements while working on a nodejs server. The server received json data which had to get processed (in this case saved to a mongodb) one part serial(synchronous) and one part parallel(asynchronous). This should be a common usecase in a nodejs application. Here is a solution by using the asyncjs library.

Consider the following nested json response from a http request.

var kids = {  
    "kids": [{
        "name": "Max",
        "father": {
            "name": "Tom"
        }
    }, ....
    , {
        "name": "Max",
        "father": {
            "name": "Tom"
        }
    }]
    "schools":[{...}]
}

Now we want to save the KIDs and its FATHER if it doesn't already exist. Without proper control we would create multiple fathers with the name 'Tom'. I used mongoosejs in my application which makes use of callbacks and therefore works asynchronous. So when saving a KID or a FATHER, we first check if it does already exist (update) otherwise we create a new one.

kids.forEach(  
    Kid.findOne({name : this.name},function (e, kid) {
        // create or update
        Father.findOne({name : ...name}, function (e, kid) {
            // create or update
        });
    });

//schools saving stuff    

Because the mongoose queries are running in the background, it can happen that the same KIDs and FATHERs are saved to the database twice.

Asyncjs

This is where we will use asyncjs. The library provides easy to use methods to control the flow of an application.

Synchronous Control

async.eachSeries(  
    kids,
    function (kid, callback) {
        //create or update
        callback();
    },
    function done() {
        console.log('finished');
    }
);

async.eachSeries iterates over the 'kids' array and calls our function on every element. We can now do our database stuff and will call the 'callback' when we are finished and the next element will be processed.

Parallel Control

Now we also want to process our kids list and schools list parallel instead of serial.

async.parallel([  
    function (callback) {
        //KIDS create or update
        callback();
    },
    function (callback) {
        //SCHOOLS create or update
        callback();
    }, callback
]);

By using asyncjs we have good control over async and serial behavior.

Thomas Sattlecker

Thomas Sattlecker

View Comments
Navigation