10. Node.js: HTTP, HTTPS

This chapter covers the HTTP and HTTPS modules.

Just a reminder: the coverage is limited to common use cases; I will go into more depth in the chapter on Controllers about specific topics such as routing and sessions.

10.1 HTTP server

HTTP server
Methods http.createServer([requestListener]) server.listen(port, [hostname], [callback]) server.listen(path, [callback]) server.close() Events request connection close checkContinue upgrade clientError
Server request Methods setEncoding(encoding=null) pause() resume() Events data end close Properties method url headers trailers httpVersion connection Server response Methods writeContinue() writeHead(statusCode, [reasonPhrase], [headers]) setHeader(name, value) getHeader(name) removeHeader(name) write(chunk, encoding='utf8') addTrailers(headers) end([data], [encoding]) Properties * statusCode

Creating an HTTP server is simple: after requiring the http module, you call createServer, then instruct the server to listen on a particular port:

var http = require('http');
var server = http.createServer(function(request, response) {
  // Read the request, and write back to the response
});
server.listen(8080, 'localhost');

The callback function you pass to http.createServer is called every time a client makes a request to the server. The callback should take two parameters - a request and a response - and send back HTML or some other output to the client.

The request is used to determine what should be done (e.g. what path on the server was requested, what GET and POST parameters were sent). The response allows us to write output back to the client.

The other API functions related to starting the HTTP server and receiving requests and closing the server are:

http.createServer(requestListener)Returns a new web server object. The requestListener is a function which is automatically added to the 'request' event.
server.listen(port, [hostname], [callback])Begin accepting connections on the specified port and hostname. If the hostname is omitted, the server will accept connections directed to any IPv4 address (INADDR_ANY). To listen to a unix socket, supply a filename instead of port and hostname. This function is asynchronous. The last parameter callback will be called when the server has been bound to the port.
server.on([eventname], [callback])Allows you to bind callbacks to events such as "request", "upgrade" and "close".
server.close()Stops the server from accepting new connections.

The server object returned by http.createServer() is an EventEmitter (see the previous chapter) - so you can also bind new request handlers using server.on():

// create a server with no callback bound to 'request'
var server = http.createServer().listen(8080, 'localhost');
// bind a listener to the 'request' event
server.on('request', function(req, res) {
  // do something with the request
});

The other events that the HTTP server emits are not particularly interesting for daily use, so let's look at the Request and Response objects.

10.1.1 The Server Request object - http.ServerRequest

The first parameter of the request handler callback is a ServerRequest object. ServerRequests are Readable Streams, so we can bind to the "data" and "end" events to access the request data (see further below).

The Request object contains three interesting properties:

request.methodThe request method as a string. Read only. Example: 'GET', ‘POST’, 'DELETE'.
request.urlRequest URL string. Example: ‘/’, ‘/user/1’, /‘post/new/?param=value’
request.headersA read only object, indexed by the name of the header (converted to lowercase), containing the values of the headers. Example: see below.

request.url The most important property is request.url, which is used to determine which page was requested and what to do with the request.

In the simple messaging application example, we briefly saw how this property was used to determine what to do with client requests. We will go into further depth when we discuss routing in web applications in the chapter on Controllers.

Checking request.method tells us what HTTP method was used to retrieve the page, the most common of which are GET and POST. GET requests retrieve a resource, and may have additional parameters passed as part of request.url. POST requests are generally the result of form submissions and their data must be read from the request separately (see below).

request.headers allows us read-only access to the headers. HTTP cookies are transmitted in the headers, so we need to parse the headers to access the cookies. User agent information is also passed in the request headers.

You can access all of this information through the first parameter of your request handler callback:

var http = require('http');
var server = http.createServer(function(request, response) {
  console.log(request);
});
server.listen(8080, 'localhost');

Once you point your browser to http://localhost:8080/, this will print out all the different properties of the current HTTP request, which includes a number of more advanced properties:

{
  socket: { … },
  connection: { … },
  httpVersion: '1.1',
  complete: false,
  headers:
    {
      host: 'localhost:8080',
      connection: 'keep-alive',
      'cache-control': 'max-age=0',
      'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) ...',
      accept: 'application/xml,application/xhtml+xml ...',
      'accept-encoding': 'gzip,deflate,sdch',
      'accept-language': 'en-US,en;q=0.8',
      'accept-charset': 'ISO-8859-1,utf-8;q=0.7,*;q=0.3'
     },
  trailers: {},
  readable: true,
  url: '/',
  method: 'GET',
  statusCode: null,
  client:  { … },
  httpVersionMajor: 1,
  httpVersionMinor: 1,
  upgrade: false
}

10.1.1.1 Parsing data

There are several different formats through which an HTTP server can receive requests. The most commonly used formats are:

  • HTTP GET - passed via request.url
  • HTTP POST requests - passed as “data” events
  • cookies - passed via request.headers.cookies

10.1.1.2 Parsing GET requests

The request.url parameter contains the URL for the current request. GET parameters are passed as a part of this string. The URL module provides three functions which can be used to work with URLs:

  • url.parse(urlStr, parseQueryString = false): Parses a URL string and returns an object which contains the various parts of the URL.
  • url.format(urlObj): Accepts a parsed URL object and returns the string. Does the reverse of url.parse().
  • url.resolve(from, to): Resolves a given URL relative to a base URL as a browser would for an anchor tag.

The url.parse() function can be used to parse a URL:

var url = require('url');
var url_parts = url.parse(req.url, true);

By passing true as the second parameter (parseQueryString), you get an additional "query" key that contains the parsed query string. For example:

var url = require('url');
console.log( url.parse(
    'http://user:[email protected]:8080/p/a/t/h?query=string#hash', true
  ));

Returns the following object:

{
  href: 'http://user:[email protected]:8080/p/a/t/h?query=string#hash',
  protocol: 'http:',
  host: 'user:[email protected]:8080',
  auth: 'user:pass',
  hostname: 'host.com',
  port: '8080',
  pathname: '/p/a/t/h',
  search: '?query=string',
  query: { query: 'string' },
  hash: '#hash',
  slashes: true
}

Of these result values, there are three are most relevant for data prosessing in the controller: pathname (the URL path), query (the query string) and hash (the hash fragment).

10.1.1.3 Parsing POST requests

Post requests transmit their data as the body of the request. To access the data, you can buffer the data to a string in order to parse it. The data is accessible through the “data” events emitted by the request. When all the data has been received, the “end” event is emitted:

function parsePost(req, callback) {
  var data = '';
  req.on('data', function(chunk) {
    data += chunk;
  });
  req.on('end', function() {
    callback(data);
  });
}

POST requests can be in multiple different encodings. The two most common encodings are: application/x-www-form-urlencoded and multipart/form-data.

application/x-www-form-urlencoded

name=John+Doe&gender=male&family=5&city=kent&city=miami&other=abc%0D%0Adef&nickname=J%26D

application/x-www-form-urlencoded data is encoded like a GET request. It's the default encoding for forms and used for most textual data.

The QueryString module provides two functions:

  • querystring.parse(str, sep=’&’, eq=’=’): Parses a GET query string and returns an object that contains the parameters as properties with values. Example: qs.parse(‘a=b&c=d’) would return {a: ‘b’, c: ‘d’}.
  • querystring.stringify(obj, sep=’&’, eq=’=’): Does the reverse of querystring.parse(); takes an object with properties and values and returns a string. Example: qs.stringify({a: ‘b’}) would return ‘a=b’.

You can use querystring.parse to convert POST data into an object:

var qs = require('querystring');
var data = '';
req.on('data', function(chunk) {
  data += chunk;
});
req.on('end', function() {
  var post = qs.parse(data);
  console.log(post);
});

multipart/form-data

Content-Type: multipart/form-data; boundary=AaB03x

--AaB03x
Content-Disposition: form-data; name="submit-name"

Larry
--AaB03x
Content-Disposition: form-data; name="files"; filename="file1.txt"
Content-Type: text/plain

... contents of file1.txt ...
--AaB03x--

multipart/form-data is used for binary files. This encoding is somewhat complicated to decode, so I won't provide a snippet. Instead, have a look at how it's done in:

10.1.2 The Server Response object - http.ServerResponse

The second parameter of the request handler callback is a ServerResponse object. ServerResponses are Writable Streams, so we write() data and call end() to finish the response.

10.1.2.1 Writing response data

The code below shows how you can write back data from the server.

var http = require('http'),
    url = require('url');
var server = http.createServer().listen(8080, 'localhost');
server.on('request', function(req, res) {
  var url_parts = url.parse(req.url, true);
  switch(url_parts.pathname) {
    case '/':
    case '/index.html':
      res.write('<html><body>Hello!</body></html>');
      break;
    default:
      res.write('Unknown path: ' + JSON.stringify(url_parts));
  }
  res.end();
});

Note that response.end() must be called on each response to finish the response and close the connection.

10.1.2.2 Common response headers

Some common uses forHTTP headers include:

I will only cover the first two use cases, since the focus here is on how to use the HTTP API rather than on HTTP 1.1 itself.

Headers and write()

HTTP headers have to be sent before the request data is sent (that's why they are called headers).

Headers can be written in two ways:

  • Explicitly using response.writeHead(statusCode, [reasonPhrase], [headers]). In this case, you have to specify all the headers in one go, along with the HTTP status code and an optional human-readable reasonPhrase.
  • Implicitly: the first time response.write() is called, the currently set implicit headers are sent.

The API has the details:

response.writeHead(statusCode, [reasonPhrase], [headers])Sends a response header to the request. The status code is a 3-digit HTTP status code, like 404. The last argument, headers, are the response headers. Optionally one can give a human-readable reasonPhrase as the second argument.
response.statusCodeWhen using implicit headers (not calling response.writeHead() explicitly), this property controls the status code that will be send to the client when the headers get flushed.
response.setHeader(name, value)Sets a single header value for implicit headers. If this header already exists in the to-be-sent headers, it's value will be replaced. Use an array of strings here if you need to send multiple headers with the same name.
response.getHeader(name)Reads out a header that's already been queued but not sent to the client. Note that the name is case insensitive. This can only be called before headers get implicitly flushed.
response.removeHeader(name)Removes a header that's queued for implicit sending.

Generally, using implicit headers is simpler since you can change the individual headers up until the point when the first call to response.write() is made.

Setting the content/type header

Browsers expect to receive a content-type header for all content. This header contains the MIME type for the content/file that is sent, which is used to determine what the browser should do with the data (e.g. display it as an image).

In our earlier examples, we did not set a content-type header, because the examples did not serve content other than HTML and plaintext. However, in order to support binary files like images and in order to send formatted data such as JSON and XML back, we need to explicitly specify the content type.

Usually, the mime type is determined by the server based on the file extension:

var map = {
  '.ico': 'image/x-icon',
  '.html': 'text/html',
  '.js': 'text/javascript',
  '.json': 'application/json',
  '.css': 'text/css',
  '.png': 'image/png'
};
var ext = '.css';
if(map[ext]) {
  console.log('Content-type', map[ext]);
}

There are ready-made libraries that you can use to determine the mime type of a file, such as:

To set the content-type header from a filename:

var ext = require('path').extname(filename);
if(map[ext]) {
  res.setHeader('Content-type', map[ext]);
}

We will look at how to read files in the next chapter.

Redirecting to a different URL

Redirects are performed using the Location: header. For example, to redirect to /index.html:

res.statusCode = 302;
res.setHeader('Location', '/index.html');
res.end();

10.2 HTTP client

There is also a HTTP client API, which allows you to make HTTP requests and read content from other websites.

### HTTP client Methods http.request(options, callback) http.get(options, callback)
Client request Methods write(chunk, encoding='utf8') end([data], [encoding]) abort() setTimeout(timeout, [callback]) setNoDelay(noDelay=true) setSocketKeepAlive(enable=false, [initialDelay]) Events response socket upgrade continue Client response Methods setEncoding(encoding=null) pause() resume() Events data end close Properties statusCode httpVersion headers trailers

10.2.1 Issuing a simple GET request

http.get(options, callback)A convinience method to make HTTP GET requests.

http.get() returns a http.ClientRequest object, which is a Writable Stream.

The callback passed to http.get() will receive a http.ClientResponse object when the request is made. The ClientResponse is a Readable Stream.

To send a simple GET request, you can use http.get. You need to set the following options:

  • host: the domain or IP address of the server
  • port: the port (e.g. 80 for HTTP)
  • path: the request path, including the query string (e.g. 'index.html?page=12')

To read the response data, you should attach a callback to the 'data' and 'end' events of the returned object. You will most likely want to store the data somewhere from the 'data' events, then process it as a whole on the 'end' event.

The following code issues a GET request to www.google.com/, reads the 'data' and 'end' events and outputs to the console.

var http = require('http');
var options = {
    host: 'www.google.com',
    port: 80,
    path: '/'
  };
var req = http.get(options, function(response) {
  // handle the response
  var res_data = '';
  response.on('data', function(chunk) {
    res_data += chunk;
  });
  response.on('end', function() {
    console.log(res_data);
  });
});
req.on('error', function(e) {
  console.log("Got error: " + e.message);
});

To add GET query parameters from an object, use the querystring module:

var qs = require('querystring');
var options = {
    host: 'www.google.com',
    port: 80,
    path: '/'+'?'+qs.stringify({q: 'hello world'})
  };
// .. as in previous example

As you can see above, GET parameters are sent as a part of the request path.

10.2.2 Issuing POST, DELETE and other methods

http.request(options, callback)Issue an HTTP request. Host, port and path are specified in the options object parameter. Calls the callback with an http.ClientRequest object with the new request.

To issue POST, DELETE or other requests, you need to use http.request and set the method in the options explicitly:

var opts = {
    host: 'www.google.com',
    port: 80,
    method: 'POST'
    path: '/',
    headers: {}
  };

To send the data along with the POST request, call req.write() with the data you want to send along with the request before calling req.end(). To ensure that the receiving server can decode the POST data, you should also set the content-type.

There are two common encodings used to encode POST request data: application/x-www-form-urlencoded

// POST encoding
opts.headers['Content-Type'] = 'application/x-www-form-urlencoded';
req.data = qs.stringify(req.data);
opts.headers['Content-Length'] = req.data.length;

and application/json:

// JSON encoding
opts.headers['Content-Type'] = 'application/json';
req.data = JSON.stringify(req.data);
opts.headers['Content-Length'] = req.data.length;

Making a request is very similar to making a GET request:

var req = http.request(opts, function(response) {
  response.on('data', function(chunk) {
    res_data += chunk;
  });
  response.on('end', function() {
    callback(res_data);
  });
});
req.on('error', function(e) {
  console.log("Got error: " + e.message);
});
// write the data
if (opts.method != 'GET') {
  req.write(req.data);
}
req.end();

Note, however, that you need to call req.end() after http.request(). This is because http.ClientRequest supports sending a request body (with POST or other data) and if you do not call req.end(), the request remains "pending" and will most likely not return any data before you end it explicitly.

10.2.3 Writing a simple HTTP proxy

Since the HTTP server and client expose Readable/Writable streams, we can write a simple HTTP proxy simply by pipe()ing the two together.

The ServerRequest is Readable, while the ClientRequest is Writable. Similarly, the ClientResponse is Readable, while the ServerResponse is Writable.

var server = http.createServer(function(sreq, sres) {
  var url_parts = url.parse(sreq.url);
  var opts = {
    host: 'google.com',
    port: 80,
    path: url_parts.pathname,
    method: sreq.method,
    headers: sreq.headers
  };
  var creq = http.request(opts, function(cres) {
    sres.writeHead(cres.statusCode, cres.headers);
    cres.pipe(sres); // pipe client to server response
  });
  sreq.pipe(creq); // pipe server to client request
});
server.listen(80, '0.0.0.0');
console.log('Server running.');

The code above causes any requests made to http://localhost:80/ to be proxied to Google. You can pass queries too, such as http://localhost:80/search?q=Node.js

10.3 HTTPS server and client

The HTTPS server and client API is almost identical to the HTTP API, so pretty much everything said above applies to them. In fact, the client API is the same, and the HTTPS server only differs in that it needs a certificate file.

The HTTPS server library allows you to serve files over SSL/TLS. To get started, you need to have a SSL certificate from a certificate authority or you need to generate one yourself. Of course, self-generated certificates will generally trigger warnings in the browser.

10.3.1 Configuration: generating your own certificate

Here is how you can generate a self-signed certificate:

openssl genrsa -out privatekey.pem 1024
openssl req -new -key privatekey.pem -out certrequest.csr
openssl x509 -req -in certrequest.csr -signkey privatekey.pem -out certificate.pem

Note that this certificate will trigger warnings in your browser, since it is self-signed.

10.3.2 Starting the server

To start the HTTPS server, you need to read the private key and certificate. Note that readFileSync is used in this case, since blocking to read the certificates when the server starts is acceptable:

// HTTPS
var https = require('https');
// read in the private key and certificate
var pk = fs.readFileSync('./privatekey.pem');
var pc = fs.readFileSync('./certificate.pem');
var opts = { key: pk, cert: pc };
// create the secure server
var serv = https.createServer(opts, function(req, res) {
  console.log(req);
  res.end();
});
// listen on port 443
serv.listen(443, '0.0.0.0');

Note that on Linux, you may need to run the server with higher privileges to bind to port 443. Other than needing to read a private key and certificate, the HTTPS server works like the HTTP server.

blog comments powered by Disqus