Deploying Nodejs Nginx Upstart Monit Redis on Ec2 With Puppet and Vagrant

In a previous post I wrote about automating deployments on AWS Cloud instances with vagrant and puppet. Today I will be describing how we deploy Node.js with nginx,redis, upstart, monit on ec2 using vagrant and puppet.

Our developers are rewriting an existing application in Node.js and we basically needed a consistent environment to play with, which would mirror the target production environment as much as possible. So we picked out technology stack as follows:

  • node.js – obviously
  • nginx – As a reverse proxy server. We are using nginx 1.5.0 which has websocket support. You could also use the stable nginx version that comes with your linux distribution in combination with varnish or haproxy to handle websocket connection requests. If you are using nginx development builds, make sure you upgrade to nginx 1.5.0 or 1.4.1 to address the recent buffer overflow vulnerability security issue.
  • upstart – to daemonize the node app
  • monit – to proactively check that the app is actually humming nicely.
  • redis – for handling session data

0ur approach is simply to start with a basic configuration which works well and fine-tune as we go along. The description below doesn’t apply strictly to Node.js deployments; it could really be adapted for other web-apps/frameworks e.g. python/django/gunicorn behind nginx.

Setup

First of all clone the nodejs_deployment repo as follows:–

clone the node.js deployment repolink
1
$ git clone git@github.com:pidah/nodejs_deployment.git

switch into the nodejs_deployment directory and have a look at the contents –

review the contents of nodejs_deploymentlink
1
2
3
$ cd nodejs_deployment/
$ ls
README.md Vagrantfile app.js      package.json    puppet

Node.js app

app.js shown below is a very simple node application listening on port 3000. The package.json file contains the node app dependencies. This app only depends on express.

node.js applink
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
$ cat app.js
var express = require("express");
var app = express();
app.use(express.logger());

app.get('/', function(request, response) {
  response.send('My awesome node app!');
});

var port = process.env.PORT || 3000;
app.listen(port, function() {
  console.log("Listening on " + port);
});

$ cat package.json
{
  "author": "Peter Idah",
  "name": "awesome-app",
  "version": "0.0.1",
  "dependencies": {
    "express": "~3.1.0"
  }
}

Vagrantfile

The Vagrantfile holds the vagrant configuration –

Vagrantfilelink
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu_aws"
  config.vm.box_url = "https://github.com/mitchellh/vagrant-aws/raw/master/dummy.box"
  config.vm.synced_folder ".", "/vagrant", id: "vagrant-root"
  config.vm.provider :aws do |aws, override|
    aws.keypair_name = "development"
    override.ssh.private_key_path = "~/.ssh/development.pem"
    aws.instance_type = "t1.micro"
    aws.security_groups = "development"
    aws.ami = "ami-c5afc2ac"
    override.ssh.username = "ubuntu"
    aws.tags = {
      'Name' => 'Nodejs App',
     }
  end

  config.vm.provision :puppet do |puppet|
    puppet.manifests_path = "puppet/manifests"
    puppet.manifest_file  = "init.pp"
    puppet.options = ["--fileserverconfig=/vagrant/puppet/fileserver.conf"]
  end

The Vagrantfile will need to be updated with your AWS details. I am using a custom AMI with puppet baked in. Then you need to setup your AWS keys in your local ~/.profile file as follows

Configuring AWS Credentials in ~/.profile
1
2
 export AWS_ACCESS_KEY="AKXXXXXXXXXXXXXXX"
 export AWS_SECRET_KEY="XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"

Then source the ~/.profile to make the variables available in the current shell session

source the ~/.profile file
1
 source ~/.profile

Every AWS EC2 instance would have an ssh key pair. I keep my ssh private key in ~/.ssh/development.pem

For more details on the Vagrantfile, have a look at automated deployments of ec2 instances with vagrant and puppet.

puppet configuration overview

The puppet configuration layout is shown below –

puppet manifests and fileslink
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
$ ls puppet/manifests
init.pp       nginx.pp    redis.pp
monit.pp  nodejs.pp   upstart.pp

$ cat puppet/manifests/init.pp
include nodejs
include monit
include nginx
include upstart
include redis

$ ls puppet/files
app.conf  deploy_node.sh  monitrc     nginx.conf

$ cat puppet/fileserver.conf
# Puppet template files directory
[files]
    path /opt/node/puppet/files
    allow *
$ cat puppet/manifests/init.pp

Each component is in a seperate manitfest file. We include all of them init.pp as shown above. puppet/fileserver.conf tells puppet to serve files from our custom mount point /opt/node/puppet/files. More information on this is available on puppetlabs website. Let’s go through each manifest configuration file:

node.js config

The following shows the node.js puppet configuration files

node.js puppet configurationlink
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
$ cat puppet/manifests/nodejs.pp
class nodejs {

    file { "/opt/node":
        ensure  => "link",
        target  => "/vagrant",
        force   => true,
    }

    exec { "apt-get update":
        path => "/usr/bin",
        require => File["/opt/node"]
    }

    $nodejs_deps = [ "python-software-properties", "g++", "make", "git", "vim" ]
        package { $nodejs_deps:
        ensure => installed,
        require => Exec["apt-get update"],
    }

    file { "/tmp/deploy_node.sh":
        ensure  => present,
        mode    => '0775',
        source  => "puppet:///files/deploy_node.sh",
        require => Package[$nodejs_deps]
    }

    exec { "install_node":
        command => "/bin/bash /tmp/deploy_node.sh",
        path => "/usr/bin:/usr/local/bin:/bin:/usr/sbin:/sbin",
        timeout => 0,
        unless => "ls /usr/local/bin/node ",
        require => File["/tmp/deploy_node.sh"]
    }

    exec { "npm_install":
        cwd => "/opt/node",
        command => "npm install",
        path => "/usr/bin:/usr/local/bin:/bin:/usr/sbin:/sbin",
        require => Exec["install_node"]
    }

}

$ cat puppet/files/deploy_node.sh
#!/bin/bash -x
version=0.10.5
mkdir /tmp/nodejs && cd $_
wget -N http://nodejs.org/dist/v${version}/node-v${version}.tar.gz
tar xzvf node-v${version}.tar.gz && cd node-v${version}
./configure
make install

The app is deployed to /opt/node which is sym-linked to /vagrant on the ec2 instance. The node.js package provided by ubuntu is really old, so we decided to build node.js from source as shown in the deploy_node.sh file above. The make build takes longer than 5 minutes which is the default timeout for the puppet exec resource, so I set timeout =>0 in exec {"install_node":} above to prevent a timeout. Subsequent vagrant provision runs would just do a quick check to confirm that node is already installed. Alternatively you could use the node.js packages available at https://launchpad.net/~chris-lea/+archive/node.js/

upstart config

upstart configuration filelink
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
$ cat puppet/manifests/upstart.pp
class upstart {

    file { "/etc/init/app.conf":
        ensure  => file,
        source  => "puppet:///files/app.conf",
        require => Class["Nodejs"],
    }

    service { 'app':
        ensure => running,
        provider => 'upstart',
        require => File['/etc/init/app.conf'],
    }

}

  $ cat puppet/files/app.conf
description "node.js app server"
author      "Peter Idah"

env PROGRAM_NAME="awesome-app"
env FULL_PATH="/opt/node"
env FILE_NAME="app.js"
env NODE_PATH="/usr/local/bin/node"

start on startup
stop on shutdown

script

    echo $$ > /var/run/$PROGRAM_NAME.pid
    cd $FULL_PATH
    exec $NODE_PATH $FULL_PATH/$FILE_NAME >> $FULL_PATH/node_app.log 2>&1
end script

pre-start script
    # Date format same as (new Date()).toISOString() for consistency
    echo "[`date -u +%Y-%m-%dT%T.%3NZ`] (sys) Starting" >> $FULL_PATH/node_app.log
end script

pre-stop script
    rm /var/run/$PROGRAM_NAME.pid
    echo "[`date -u +%Y-%m-%dT%T.%3NZ`] (sys) Stopping" >> $FULL_PATH/node_app.log
end script

The upstart.pp manifest above ensures the app is running with the configuration sourced from /etc/init/app.conf. This file daemonizes the application, specifies the process id location, logfile and the start/stop control scripts.

monit config

monit config filelink
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
$ cat puppet/manifests/monit.pp
class monit {

    package { "monit":
        ensure  => latest,
        require => Class["Nodejs"],
    }

    file { '/etc/monit/monitrc':
        ensure  => present,
        mode    => '0600',
        owner   => 'root',
        group   => 'root',
        source  => "puppet:///files/monitrc",
        notify  => Service['monit'],
        require => Package["monit"],
    }

    service { 'monit':
        ensure     => running,
        enable     => true,
        hasrestart => true,
        require    => File['/etc/monit/monitrc'],
        subscribe  => File['/etc/monit/monitrc'],
        notify     => Service['app'],
    }

}

$ cat puppet/files/monitrc
#!monit
set logfile /opt/node/monit_app.log

check process nodejs with pidfile "/var/run/awesome-app.pid"
    start program = "/sbin/start app"
    stop program  = "/sbin/stop app"
    if failed port 3000 protocol HTTP
        request /
        with timeout 2 seconds
        then restart
    if cpu > 80% for 10 cycles then restart
    if 3 restarts within 10 cycles then timeout

The monit.pp manifest above ensures the latest monit package is installed and running. It will also trigger a restart if it detects changes to the contents of /etc/monit/monitrc config file. The monitrc config file sets the location of the monit log file, checks the process id and specifies the path to the start/stop scripts for the node app. Then it gets a bit more interesting with a few rules – in the 1st rule monit checks if there is a failed request to the root of the node app listening on port 3000 for 2 seconds, monit will restart the application. The 2nd rule checks for cpu usage above 80% for 10 cpu cycles and trigers a restart of the app. There are several monit rules you can add, but that’s the general idea.

nginx config

nginx config filelink
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
$ cat puppet/manifests/nginx.pp
class nginx {

    exec { "add_nginx_repo":
            command => "add-apt-repository ppa:nginx/development --yes && apt-get update ",
            path => "/usr/bin",
            require => Class[Nodejs]
    }

    exec { "install_nginx":
        command => "/usr/bin/apt-get install nginx -y --force-yes",
        path => "/usr/bin:/usr/local/bin:/bin:/usr/sbin:/sbin",
        unless => "ls /usr/sbin/nginx ",
        require => Exec["add_nginx_repo"]
    }
    service { 'nginx':
        ensure     => running,
        enable     => true,
        hasrestart => true,
        require    => Exec['install_nginx'],
    }

    file { "/etc/nginx/nginx.conf":
        ensure  => present,
        mode    => '0644',
        source  => "puppet:///files/nginx.conf",
        notify => Service['nginx'],
        require => Exec["install_nginx"],
    }

}

$ cat puppet/files/nginx.conf
user www-data;
worker_processes 4;
pid /var/run/nginx.pid;



events {
        worker_connections 768;
}

http {

        ##
        # Basic Settings
        ##

        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 65;
        types_hash_max_size 2048;

        include /etc/nginx/mime.types;
        default_type application/octet-stream;

        ##
        # Logging Settings
        ##

        access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;

        ##
        # Gzip Settings
        ##

        gzip on;
        gzip_disable "msie6";


        include /etc/nginx/conf.d/*.conf;

upstream nodejs_backend {
    server 127.0.0.1:3000;
}

# the nginx server instance

    server {
        listen 80 default;
        server_name 127.0.0.1;
        access_log /opt/node/nginx_app.log;

    # pass the request to the node.js server with the correct headers
    location / {
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_set_header X-NginX-Proxy true;
        proxy_pass http://nodejs_backend/;
        proxy_redirect off;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        }
    }
}

The nginx.pp manifest above installs nginx 1.5.0 from the nginx ppa development repo, ensures the service is running as specified in nginx.conf and would trigger a restart if changes are detected in nginx.conf. The nginx.conf file above specifies the web server should listen on port 80 and hands requests to the upstream nodejs_backend server listening on port 3000. The following three lines are required for websocket connections in nginx version 1.4+

nginx config filelink
1
2
3
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";

redis config

The redis config file below ensures that the redis-server is installed and running

redis config filelink
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
class redis {

    package { "redis-server":
            ensure  => installed,
            require => Class["Nodejs"],
    }

    service { 'redis-server':
        ensure     => running,
        enable     => true,
        hasrestart => true,
        require    => Package['redis-server'],
    }

}

Deploying the instance

As described in the README.md file, the next step is to launch the instance as follows

launching the instance
1
$ vagrant up --provider=aws

You can then use your regular vagrant commands as usual e.g, to ssh into your instance

login to the instance
1
$ vagrant ssh

You can get the public DNS name of the instance with the vagrant ssh-config command:

get the public dns name of ec2 instance
1
2
3
4
5
6
7
8
9
10
11
12
$ vagrant ssh-config

Host nodejs
  HostName ec2-184-73-111-79.compute-1.amazonaws.com
  User ubuntu
  Port 22
  UserKnownHostsFile /dev/null
  StrictHostKeyChecking no
  PasswordAuthentication no
  IdentityFile "/Users/Peter/.ssh/development.pem"
  IdentitiesOnly yes
  LogLevel FATAL

You can then access the app from your browser e.g http://ec2-184-73-111-79.compute-1.amazonaws.com or using curl as shown below:

testing the app with curl
1
2
$ curl http://ec2-184-73-111-79.compute-1.amazonaws.com
My awesome node app!$ 

That’s all folks. Thanks for dropping by!

Comments