Creating a Push Notification Server with Node.js

Creating a Push Notification Server with Node.js

At Shockoe we’re used to integrating our clients’ complicated backend systems into our apps, and this often includes push notifications. However, to get that initial proof that our app is playing nicely with GCM and APNS we’ll sometimes rig up a sandbox server of our own. Since we’re used to JavaScript, Node is an obvious go-to. Here are the steps for rigging up a simple push notification server in Node.js.

A push notification server consists of two parts: storing deviceIds and sending push notifications. To achieve this we’ll create two endpoints: register and send. Register will utilize the mongodb node module mongoose and Send will leverage two platform-specific node modules: apns and node-gcm. We’ll also be using restify to set up our endpoints.

Register

For a basic implementation we only need two pieces of information: the deviceId and platform (Android or iOS). Technically–in some mongoose schema code golf scenario–we just need the deviceId, and could make both calls with one always failing.

So our mongoose schema, with the proper requirements, would look something like this:

var mongoose = require('mongoose');
var Schema = mongoose.Schema;

var Device = new Schema({
    deviceId : String,
    platform : String
});

var DeviceSchema = mongoose.model('Device', Device);

Mongoose is generally a lot more powerful than this implementation, with validations and enumerations that allow for easy error checking, but for the purposes of this demo we’ll use a basic schema. After making sure mongodb is installed and running on your machine, our restify endpoint would look something like this:

var restify = require('restify');

var db = mongoose.connection;

db.on('error', console.error.bind(console, 'connection error:'));

db.once('open', function(){
    console.log('db open');
});

mongoose.connect('mongodb://localhost/pushserver');

var server = restify.createServer({
    name : 'pushServer'
});

server.use(restify.plugins.bodyParser());

server.post('/register', (req, res, next) => {
    let body = JSON.parse(req.body);

    if (body) {
        let newDevice = new DeviceSchema(body);
        newDevice.save(err => {
            if (!err) {
                res.send(200);
            } else {
                res.send(500);
            }
        }
    }
}

Ideally we would perform a find on DeviceSchema to make sure we’re not adding duplicate devices.

Send

For Send we’ll implement a basic endpoint that will send a push notification to every device in the database. We can use a simple GET, grab the devices from mongo, and call functions to send the push, based on whether the target is an iOS or an Android device. The Android push module can take an array of deviceId’s instead of one at a time, so we’ll push those ids onto an array and call the sendAndroid function on all of them.

server.get('/send', (req, res) => {
    DeviceSchema.find( (err, devices) => {
        if (!err && devices) {
            let androidDevices = [];
            devices.forEach(device => {
                if (device.platform === 'ios') {
                    sendIos(device.deviceId);
                } else if (device.platform === 'android') {
                    androidDevices.push(device.deviceId);
                }
            });
            sendAndroid(androidDevices);
            res.send(200);
        } else {
            res.send(500);
        }
    });
});

Ideally we would only send the 200 response once we were sure that every push had been sent successfully, but for now it will indicate that the send functions were called.

Our sendIos function will leverage the apns module, which takes an options object. Make sure to include your push key and certificate .pem files at the root of your project:

var apns = require('apns');

var options = {
    keyFile  : 'key.pem',
    certFile : 'cert.pem',
    debug    : true,
    gateway  : 'gateway.sandbox.push.apple.com',
    errorCallback : function(num, err) {
        console.error(err);
    }
};

function sendIos(deviceId) {
    let connection = new apns.Connection(options);

    let notification = new apns.Notification();
    notification.device = new apns.Device(deviceId);
    notification.alert = 'Hello World !';

    connection.sendNotification(notification);
}

For android we’ll use node-gcm. Make sure you have your api key:

var gcm = require('node-gcm');

function sendAndroid(devices) {
    let message = new gcm.Message({
        notification : {
            title : 'Hello, World!'
        }
    });

    let sender = new gcm.sender('<YOUR_API_KEY_HERE>');

    sender.send(message, {
        registrationTokens : devices
    }, function(err, response) {
        if (err) {
            console.error(err);
        } else {
            console.log(response);
        }
    });
}

After that it’s just a simple matter of running your project and you’re ready to hit these endpoints and test push notifications on your app!

 

Node.js – Storing data with MongoDB

Node.js – Storing data with MongoDB

The last few weeks we have focused on web service design using Node.js with Express. So far we have covered getting started with express, using EJS to create web pages, and creating a push notification server. Today we will go over how to save data on your node project by using MongoDB.

This post will assume that you have MongoDB installed on your machine already. Check out the MongoDB Installation page if you do not have it setup.

First, let’s go over MongoDB a bit. MongoDB is a document oriented database. This allows you to just shove whatever JSON you want into it. You don’t have to declare a table structure like you do in SQL. You don’t even need to declare your collections at all. If you try to use a collection that doesn’t exist it just gets made for you immediately. Documents are stored in collections that you define, but documents within a collection don’t have to be similar at all. MongoDB also features an amazing query system. Imagine we want to store information about fruit. We could do the following:

db.fruit.insert({name : "apple", color : ["red", "yellow"], source : "tree"})

We now have a fruit collection containing one fruit: an apple. Let’s add a few more:

db.fruit.insert({name : "strawberry", color : ["red"], source : "bush"})
db.fruit.insert({name : "orange", color : ["orange"], source : "tree"})

Now we have a strawberry and an orange in our fruit collection. To get all fruits in our collection that come from trees we would do:

db.fruit.find({source : "tree"})

If we wanted to find all fruits that are red we would do:

db.fruit.find({color : "red"})

And to find all fruits that are yellow or orange we would do:

db.fruit.find({color : {$in : ["yellow", "orange"]}})

Now apples can be green too. Let’s modify our apple entry to reflect that:

db.fruit.update({name : "apple"}, {$push : {color : "green"}})

There are a few interesting things to be aware of when updating a document. By default, mongodb will only update the first entry it finds. You can set the multi option to force it to update every entry that matches the query. Also, note the $push operator I have included in the update object. That is telling mongo it will be adding an entry to an array on the document. If you left off push and just did:

db.fruit.update({name : "apple"}, {color : "green"})

You would be very disappointed. Setting no update operator will replace the document completely:

{
    "color" : "green"
}

So what if you wanted to update a field that isn’t an array? You use the $set update operator:

db.fruit.update({name : "apple"}, {$set : {name : "pear"}}

MongoDB is very powerful and an excellent document store, but it lacks the join power that SQL brings to the table. You can join documents together by their _id field using MonoDB’s populate, but it’s not nearly as fast as what you can do with a few SQL joins. I encourage you to explore MongoDB as an option for your databases, but it’s important to approach your database design differently with MongoDB than with SQL. Be sure to check out the documentation for all you can do with MongoDB.

So how to we start using Mongo with Node? There are several libraries available on npm for connecting to Mongo, but I will be focusing on Mongoskin and Mongoose.

Let’s go ahead and get Mongoskin out of the way. Mongoskin is what you want if you just want to run basic MongoDB commands and nothing more. To get started add Mongoskin as a dependency:

npm install mongoskin --save

And then in node:

var mongo = require('mongoskin');
var db = mongo.db("mongodb://localhost:27017/integration_tests", {native_parser:true});
db.collection('fruit).find({name : "orange"}).toArray(function(err, fruits){
    console.log("I have "+fruits.length+" fruits in my collection:");
    for(var i=0; i<fruits.length; i++){
        console.log(fruit.name);
    }
});

Mongoskin is very easy to use and most MongoDB commands are supported. It has its issues, but if you want something simple to connect to a mongo database look no further. If you want the best MongoDB experience you can find with Node, keep reading!

On to Mongoose. As they say on their website: “Mongoose provides a straight-forward, schema-based solution to modeling your application data and includes built-in type casting, validation, query building, business logic hooks and more, out of the box.” Mongoose is a delight. To get started add Mongoose as a dependency:

npm install mongoose --save

It’s nice that MongoDB doesn’t need to have schemas, but it’s great that Mongoose gives them to you. Let’s put together a schema for our fruit collection:

var mongoose = require('mongoose');

var FruitSchema = new mongoose.Schema({
    name : String,
    color : [String],
    source : {
        type : String,
        default : 'tree'
    }
});

FruitSchema.virtual('numberOfColors').get(function(){
    return this.color.length;
});

mongoose.model('Fruit', EventSchema);

Most of the schema above should be straight forward. Name must be a string or Mongoose will remove the property from the object before using it as a model. Color must be an array of strings. Source must be a string, and if it’s not present Mongoose will automatically set it to “tree”. Default values will not get stored in the actual database, but they will be applied when Mongoose pulls them out. The virtual “numberOfColors” will add a property automatically. Virtuals are great for properties that can be calculated based on the existing data on the model.

To use the fruits model:

mongoose.model('Fruit').find({name : "orange"}, function(err, fruits){
    console.log("I have "+fruits.length+" fruits in my collection:");
    for(var i=0; i<fruits.length; i++){
        console.log(fruit.name + " comes in " + fruit.numberOfColors + " colors");
    }
});

The examples above barely scrape the surface of what you can do with MongoDB and Mongoose. Do yourself a favor and explore the possibilities of a document oriented database backed by MongoDB.

Creating Dynamic Web Pages with EJS

Creating Dynamic Web Pages with EJS

    If you have been following along with our series on how to create your own web server, you should now be familiar with node.js and Express.  Node.js lays the groundwork for the web server and Express builds upon it.  If you haven’t already, I recommend that you take a look at the previous blog posts on Node.js and Express before we get started.

    The next step in building an awesome web server, is using EJS to render dynamic web pages with embedded JavaScript.  You can check out the EJS home page to view more documentation.  After your Express project has been set up, take a look at the app.js file.  In this file, you will see the line:

app.use('/', routes);

    This use statement means that when you visit your project in a web browser, Express will look in the routes folder for a file called index.js that will tell the browser what to do.  Right now, the index.js file should look something like this:

var express = require('express');

var router = express.Router();

/* GET home page. */

router.get('/', function(req, res, next) {

  res.render('index', { title: 'Express' });

});

module.exports = router;

    Right now, the Express router has one interaction that will render the home page.  Currently, when there is a GET request to the route ‘/‘, Express will render the view found in views/index.ejs.  The first argument for the render function is the name of the view file, and the second is an object that will be passed to the view file when it is rendered.  This object can be used by EJS to change what is rendered on the page, without having to write an entirely new view.  Right now the view will only render two lines of text.  We are going to expand on this in order to show some dynamic content in the browser.

Rendering with EJS

    With EJS, JavaScript between <% %> is executed, and JavaScript between <%= %> adds HTML to the result.  Lets give this a try.  First, in our Express file, index.js, we will send a list of items we want show on the web page.  Then, we will use EJS to iterate through this list and display the contents.

index.js

router.get('/', function(req, res, next) {

  res.render('index', { title: 'Express' , items : [‘red’, ‘blue’, ‘green’]});

});

    We simply add a new property to the object we are sending to the render function.  The property items is an array containing three strings.  Now we will print these strings in our EJS file.

index.ejs

<h1><%= title %></h1>

<ul>

<% for(var i=0; i<items.length; i++) {%>

<li style="color:<%=items[i]%>;"><%= items[i] %></li>

<% } %>

</ul>

    We create a for loop that will iterate through the array we passed from our Express function.  The loop is places inside <% %> tags because we do not want the for loop itself to be added to the web page.  Inside the loop, we have a <li> tag that will place an item on the web page for each item in the array.  To display the item from the array, use the tag <%= items[i] %>.  This will look up the element in the items, array, and add it to the web page.  As you can see, these tags can be place anywhere inside normal html.  To view these changes, just restart the server and reload the page in your browser.  You should see something like this:

Express

  • red
  • green
  • blue

If you view the source for the page in your browser, you might be wondering why you don’t see any of the EJS tags that are in the view file.  That is because Express will evaluate the EJS tags before the page is shown, and only the results will be displayed.  EJS gives you flexibility when creating views for you website.  You will be able to use powerful embedded JavaScript to render views for different situations.

Express HTTP servers with Node.js

Express HTTP servers with Node.js

This week we continue our series on building better web services with Node.js by taking a look at the Express web application framework. Unfamiliar with Node? Take a look at last week’s blog post to find out what you’re missing.

Why Express?

Express provides all of the tools you need to immediately be productive working on web applications in Node.js.  While Node provides a number of built-in networking APIs, they are as a group cumbersome and somewhat unintuitive to work with, forcing you to do a large amount of repeated setup any time you want to set up another endpoint, route, port, or HTTP verb.  This is where Express comes in.  We can use Express to handle all of the repeat setup tasks associated with building a HTTP server, and spend more time focus on getting the core logic and UI of our applications correct.

Getting set up

To start using Express, just do a normal NPM install:

yourname@domain > npm install express -g

Optionally (but highly encouraged!), you can also install the Express-generator package to provide a convenient terminal command to set up a well-structured Express project. Again, NPM is the bomb.

yourname@domain > npm install express-generator -g

We’ll probably have to install some more dependencies for the generated project later, but this is enough to start rolling.

Making your Express project

Now that Express and Express-generator have been set up, let’s generate an Express project. Express-generator comes with a number of command line options, but for the most part uses sensible defaults. One notable point of contention is the default template engine: Jade. Express-generator can also generate projects using EJS, Handlebars, or Hogan.js. While I personally prefer Handlebars for most purposes (it does a great job of separating logic and templates in a clean fashion), for the purposes of this article I’ll be using EJS in interest of staying as close to Javascript as possible. To generate an Express project using EJS templates, navigate to your project’s intended parent directory and run:

yourname@domain > express --ejs {{YOUR PROJECT NAME GOES HERE}}

This will place a simple Express project configured to template its web pages using EJS in the specified subdirectory. Next, you’ll need to navigate to that directory and run NPM install to resolve your dependencies.

yourname@domain > cd {{YOUR PROJECT NAME GOES HERE}}
yourname@domain > npm install

Note that you don’t provide any arguments to npm install this time. This means that npm will look inside the Node project’s configuration file, package.json, for a list of dependencies to install. This is the same way that you would download dependencies if one of your friends or coworkers gave you a link to a Node.js project to collaborate on. Specifying dependencies like this allows you not to include your dependencies on the project itself, and gives other developers an easy way to resolve these dependencies.

 

Cool… So what just happened?

Express generator sets up a very simple Express project, configured to perform common backend tasks like parsing requests, setting up routes and complex endpoints, and serving a public directory for client-side assets. Let’s take a look at the Express application’s entry point, app.js.

var express      = require('express');
var path         = require('path');
var favicon      = require('serve-favicon');
var logger       = require('morgan');
var cookieParser = require('cookie-parser');
var bodyParser   = require('body-parser');

var routes = require('./routes/index');
var users  = require('./routes/users');

var app = express();

// view engine setup
app.set('views', path.join(__dirname, 'views'));
app.set('view engine', 'ejs');

// uncomment after placing your favicon in /public
//app.use(favicon(__dirname + '/public/favicon.ico'));
app.use(logger('dev'));
app.use(bodyParser.json());
app.use(bodyParser.urlencoded({ extended: false }));
app.use(cookieParser());
app.use(express.static(path.join(__dirname, 'public')));

app.use('/', routes);
app.use('/users', users);

// catch 404 and forward to error handler
app.use(function(req, res, next) {
    var err    = new Error('Not Found');
    err.status = 404;
    next(err);
});

// error handlers

// development error handler
// will print stacktrace
if (app.get('env') === 'development') {
    app.use(function(err, req, res, next) {
        res.status(err.status || 500);
        res.render('error', {
            message: err.message,
            error: err
        });
    });
}

// production error handler
// no stacktraces leaked to user
app.use(function(err, req, res, next) {
    res.status(err.status || 500);
    res.render('error', {
        message: err.message,
        error: {}
    });
});


module.exports = app;

There’s a lot going on here, so let’s break it down by chunks. In the spirit of providing an overview of Node.js, let’s look at the require statements first.

var express      = require('express');
var path         = require('path');
var favicon      = require('serve-favicon');
var logger       = require('morgan');
var cookieParser = require('cookie-parser');
var bodyParser   = require('body-parser');

var routes = require('./routes/index');
var users  = require('./routes/users');

Require is an extraordinarily powerful tool for looking up libraries and external dependencies within a Node.js project. When you pass a string to require, it will try to look up a properly packaged CommonJS or native Node.js module, keep a reference to the loaded module, and return it to the caller. Then, if you try to load the library again, the cached reference will be returned rather than attempting to load a fresh instance. This lets you efficiently manage dependencies and script loading with almost no effort.

Taking a closer look at the require block, you will notice it is split into two halves, based on the type of require that is happening. The first group is loading pre-packaged Node modules, installed using npm. Require first checks your project’s node_modules folder for these files, and then checks the global npm cache. Express is a good example of a library that will be installed globally (we supplied the option -g to npm install earlier, meaning the library was installed at a system level), whereas the other modules were downloaded to the node_modules folder by NPM install, and will be loaded from there.

The second group of requires loads libraries local to the current project. This type of require uses normal relative file paths (. represents the current directory, .. is the parent), and will attempt to load javascript files relative to the current file’s parent directory. These statements load a pair of router files which will define a set of endpoints. We’ll take a closer look at these later.

 

Now let’s start serving

Now that we’re done resolving our dependencies, we can actually start initializing our Express project. Let’s take a look at the next couple of lines in app.js.

var app = express();

// view engine setup
app.set('views', path.join(__dirname, 'views'));
app.set('view engine', 'ejs');

// uncomment after placing your favicon in /public
//app.use(favicon(__dirname + '/public/favicon.ico'));
app.use(logger('dev'));
app.use(bodyParser.json());
app.use(bodyParser.urlencoded({ extended: false }));
app.use(cookieParser());
app.use(express.static(path.join(__dirname, 'public')));

app.use('/', routes);
app.use('/users', users);

We do a couple of important set up tasks here. The obvious big one is creating our Express instance, but we also set up how we will be rendering our web pages and where to find them (in the views subdirectory), and begin to set up the actual logic of the Express application.

Express is built on the concept of middleware functions, small little functions that are each capable of processing or responding to a request, or sending the request to the next piece of middleware capable of handling your request. You can use middleware to set up generic handling for all endpoints, specific handling for endpoints matching some route, or specialized handling for an individual endpoint. This is done primarily via the use function.

use has a couple of nuances that allow you to set up complex server-wide handling easily. Looking at the use calls, you can see that some calls are passed a single argument, while others receive two. The single argument calls apply the selected piece of middleware to every request that reaches the server, while the two args version allows you to restrict the middleware to operating on a single route. We set up our pre-generated routes at the end of this call.

Next, we set up a couple of custom middleware functions.

// catch 404 and forward to error handler
app.use(function(req, res, next) {
    var err    = new Error('Not Found');
    err.status = 404;
    next(err);
});

// error handlers

// development error handler
// will print stacktrace
if (app.get('env') === 'development') {
    app.use(function(err, req, res, next) {
        res.status(err.status || 500);
        res.render('error', {
            message: err.message,
            error: err
        });
    });
}

// production error handler
// no stacktraces leaked to user
app.use(function(err, req, res, next) {
    res.status(err.status || 500);
    res.render('error', {
        message: err.message,
        error: {}
    });
});

Again, there’s a lot going on in this series of calls, so let’s break down what’s happening. Middleware is ultimately some function that will be executed, and these functions always receive the same three arguments, in the same order. The first argument is the request object. The request object is where you will look to get information about an individual request coming to your server. The request can be preproccessed by middleware, and since all pieces of middleware receive the same object reference for a given request, all other pieces of middleware will receive this information. That’s how the bodyParser library from earlier works. It allows JSON payloads to automatically be parsed and turned into a request body object, rather than plain text.

The second argument is the response object. The response object is used to do things like set HTTP status codes or response headers, report request progress back to the client, and actually send your response to the client when you are done handling the request.

The third argument is the next object. This is a reference to the next function in the middleware stack. You can call next to indicate that you are done attempting to handle this request, and that the next piece of middleware should make its attempt at handling. Middleware is run in order of execution, so the first thing that you app.use that matches the request’s path will be executed first, proceeding until either a response is sent or we run out of middleware to attempt to handle the request with.

There is a special case of middleware, namely error handlers. Error handlers must accept exactly four arguments (e.g. the function’s arity must be four). The last three arguments are the same as a normal piece of middleware, but an error object must be accepted as an additional first argument

Taking a look at the middleware added at the end of app.js, we have a single piece of standard middleware that will handle any routes that do not have middleware set up to respond, and will return a 404: not found message instead. The remaining middleware defines the development and production error handlers. As you can see these are four-argument functions, and are as such error handling middleware. Since the first piece of error middleware sends a response, in the dev environment we’ll return error messages with full stack traces, whereas in production we will merely notify the user that an error occurred.

Finally, on the last line of app.js, we export our Express application using the normal commonjs module.exports syntax.

 

So, about those route things…

Routes are a powerful tool that Express gives you to organize your code in a sensible fashion. Express wouldn’t make for a very good framework if we had to put everything in app.js, now would it? Let’s take a look at one of the generated route files that we required earlier.

var express = require('express');
var router = express.Router();

/* GET home page. */
router.get('/', function(req, res) {
  res.render('index', { title: 'Express' });
});

module.exports = router;

As you can see, the way a router works is very similar to the way to how your base Express application works. You initialize an instance of your router, register some number of pieces of middleware, and export it for your consumer (almost always your Express application). One notable difference is that instead of using use to register middleware, we are using a function named get. get is a convenience function for registering middleware that only executes for requests with the HTTP verb GET. Similarly, there is a function for PUT, POST, DELETE, and the other HTTP verbs as well. The second nuance of routers is that they inherit the base route of where they are used by the application. For the users route for instance, all endpoints will start with /users, and then have any additional route tokens appended to the end. Additionally, you can nest routes as deeply as you need to to achieve your desired code organization.

 

Cool, but how do I start it?

NPM comes to the rescue again. Express-generator set up the project’s package.json with a pre-defined start command, which will set up your application to run on a port defined by your shell’s environment variables, or the default port of 3000. Take a look at bin/www to see how this works, or just run npm start to get your application running! Note that npm start will take control of your current console and use it for logging, you may want to direct your output to a file by running something of the form npm start >> log.txt

 

So where do I go from here?

So far we’ve covered setting up your node instance and configuring it to serve a web service using Express, but there’s still a lot of learning to be done! In the coming weeks, we’ll go over how to take this basic Node.js HTTP server and configure it to serve templated web pages using EJS, we’ll go over how to configure a simple noSQL database for Node.js using MongoDB, and we’ll pull it all together to write a simple push notification server, controlled through a RESTful API.

Node.js – A JavaScript Developer’s Dream

Node.js – A JavaScript Developer’s Dream

Here at Shockoe we are happiest when working with JavaScript. There are plenty of projects that call for other languages, like Objective-C, Java, or PHP, but the goal is to work with JavaScript whenever possible. For a decade I thought of JavaScript as this insecure toy language used to do miscellaneous effects on the web. I gave JavaScript another look when AJAX came into the vogue, but again dismissed it as a toy language with no real use. Then I started working at Shockoe, where I was very surprised to learn that JavaScript could be used for writing mobile applications.

Despite working for a company that primarily deals with mobile applications, I consider myself a web developer. I’ve been using PHP for a very long time and mostly just tolerated its inconsistencies and occasional strange syntax. PHP was just the way you did server side code for the web. Then I discovered Node.js. Now the goal is to write everything in JavaScript.

Node.js is advertised as being “for easily building fast, scalable network applications.” By default, Node.js performs very little I/O so blocking is not a concern. There’s an absurd amount of Node.js modules available from Node Package Manager (npm) that allow you to connect to almost anything imaginable. Most, if not all, modules for connecting to I/O provide async methods so that blocking remains of little to no concern.

To get started with Node, head on over to nodejs.org There are builds available for Windows, OSX, and Linux. Then head over to npmjs.com to see what modules Node Package Manager has to offer. There’s web frameworks, build harnesses, database drivers, and much more. Any developer who loves JavaScript owes it to themselves to explore the world of Node.js.

I’m at the point where I use Node.js for anything from giant web projects to minor fun tasks. At Shockoe we have taken an initiative to improve our health. One concern is that we spend all day looking at monitors which puts a lot of strain on our eyes. To combat this we have implemented a system known as the twenty twenty twenty rule. Every twenty minutes we take a twenty second break while looking at something twenty feet away. To alert everyone of the time we have a node server, running on our continuous integration machine in the corner, that turns a hue light purple for twenty seconds every twenty minutes. It’s enough to catch everyone’s attention without being extremely interrupting. This task could have been accomplished with cron, but I don’t want to schedule cron jobs. The goal is to write JavaScript.

In the upcoming weeks we at Shockoe will be expanding upon this post and, step by step, creating a project using Node.js.