My Exprience using ElasticSearch NodeJS and ExpressJS

Idan FridmanAugust 3rd, 2014Last Updated: August 2nd, 2014

2 160 3 minutes read

General requirement

Part of a system I am currently working on is manipulating raw data. The data being sent from multiple client devices (Android/Iphone devices, etc..) ~ 100,000 requests per day.
The raw data needs to be collected and ready for later use. It gotta be searchable and aggregated for calculations.

I had little experience with Solr but this time I wanted to try ElasticSearch(ES). I read about ES out of the box clustering and shards capabilities so I gave it a shot.

ES

The installation is simple and quick. ES ships with comfortable UI dashboard(Marvel) and a command line tool to execute queries.

After playing with the framework I created my first Index and Mapping types (MyIndex, MyType).

I didn`t know at that point how exactly my data was going to be structured, so I went into “all-fields-querying’. Meaning that for a single input searching the ES engine will go through all documents and search in each document’s field. The matching documents will return.

We may achieve this by using the field _all. That field includes flatten text of one or more other fields within the document indexed. The field is populated by default (if enabled).

The query is pretty easy:

GET /_search
{
    "match": {
        "_all": "Android Galaxy S5 device"
    }
}

ES dashboard also ships with a tool called Sense. With this tool you can Index and execute queries straight into ES without using any client (It’s good for testing and developing purposes).

I suggest you going throw mapping, analyzing, boosting, aliasing,etc…

NodeJS

So after playing with ES I decided to write the server side layer to be as my ES facade.

My intention was to write simple client with a single search line that will send requests to the ES via the server layer and render the results to the screen.

ES is working with Json’s as its input and output. I thought about using javascript which working comfortably with json’s.

Nodejs is server side layer also using JS syntax. That’s why JavaScript makes Node a perfect language for interfacing with ElasticSearch. Dig it?

I grabbed the ES client implementation( elasticsearchclient) for NodeJS and after coding I completed the integration between node and ES. Here is a sample code from the Node side for querying the ES:

var index = "myindex";
var type = "mytype";
ElasticSearchClient = require('elasticsearchclient');
var Q = require('q');


var serverOptions = {
    host: 'localhost',
    port: 9200
};

var elasticSearchClient = new ElasticSearchClient(serverOptions);

function performSearch(termToSearch) {

    var deferred = Q.defer();
    console.log("Request handler 'search' was called.");
    var qryObj = {
        "query" : {
            "term" : { "name":termToSearch }
        }
    };
    elasticSearchClient.search(index, type, qryObj).
        on('data', function (data) {
            // console.log(data)
            deferred.resolve(JSON.parse(data));
        })
        .on('error', function (err) {
            console.log(err);
            return deferred.resolve(err);
        })
        .exec();
    return  deferred.promise;
}

exports.performSearch = performSearch;

NodeJS is single threaded that implemented with event looper. Therefore the “heavy” logic needs to be asynchronously executed. I am using Q in my code.
Q is a library for making and composing asynchronous Promises in JavaScript.

This way I can “free” the event looper and create the asynchronous requests to my ES engine and get clean callbacks through the code layers using Q.

After a while I found the need of having extensive API. I used ExpressJS for this purpose.

ExpressJS

I wanted MVC framework to provide a rest-api-like layer that will handle all API requests in my server side(NodjeJS).

I went to ExpressJS which appears to be the most popular web framework for Node.js among web developers (We as experienced open-source developers know that community matters!).

I am obsessed with project structure. I am coming from the Java world and sometimes javascript projects seems to me like a Jungle. I was looking for a popular and conventional way to structure my NodeJS project together with ExpressJS.

So after playing with the framework I found out that with Yeoman and generator-express you quickly generate project structure of Nodejs including all necessary expressJS libraries to get you going.

My project tree looked like this:

Example of ExpressJS controller:

var express = require('express');
var router = express.Router();
var esService = require('../services/esService');
var Q = require('q');



router.get("/search", function (req, res) {
    var termToSearch = req.query.termToSearch;
    console.log("termToSearch=" + termToSearch);
    Q(esService.performSearch(termToSearch)
    ).then(function (data) {
            res.send("Session: %j", data);
        });
})

The controller invoking the perfomSearch method using esService and returning the results back to the JS client asynchronously using promises.

This is the JS client code which rendering the results into HTML element:

* I used prettyPrint library to print the json result nicely on screen:

function handleClick() {
    var inputParam = document.querySelector("#myinput").value;
    $.get('http://localhost:3000/search?termToSearch=' + inputParam, function (responseText) {
        console.log(responseText);
        $("#resultlist").append(prettyPrint(responseText));
    });

    return false;
}