I Dream of Node Streams

Written by Katy Farmer and edited by me! (Not that it needed editing. Katy is a former writer and editor like me, which more or less makes us the coolest developers ever. 😂)

I’ve been thinking about streams.


They exist in various incarnations in many languages, often used in the workings of our code that we take for granted. At first, I wanted to learn what they were and how they worked. Then, I fell in love--okay, but I definitely fell in intrigue at the very least.

There are a few things I try to ensure about my code: it's modular, readable, and single responsibility. These characteristics help me to write code that someone else (or most likely me, some months later) can easily access.

Following that logic, my first inclination is to write my Javascript like I write Ruby: encapsulated objects that only interact under my supervision, like teenagers from that town in "Footloose". I know what they're doing and where they're doing it. All is well. End of blog post.

Except pushing my Javscript to be more object-oriented wasn't necessarily helping me write modular, readable, single responsibility code. It was just one pattern that I was familiar with, but I realized it wasn't what Javascript did best. What Javascript does with aplomb is streams.

What is a stream?

First, let's talk about what a stream is. It is, as simply as I can put, a flow of data. It helps manage memory by sending the data in chunks, meaning if I have a 10GB file full of my favorite mythical creatures (starting with the chupacabra all the way down to satyrs), I don't have to fit 10GB of information in to memory. Instead, the stream will send along chunks until it reaches the end, without the need for my system to get overriden by an army of flying monkeys or centaurs.

I started wondering how I could write my best code using streams. How could I take the principles of modularity and single responsibility and apply them not to emulate another language, but to embrace Javascript?

Let's say I keep all of my most important secrets in a text file, but lately, I'm worried about my privacy. I heard my text file is vulnerable.


I want to encrypt my file and store a new, encrypted version of it. I think streams can do all of this.

Let's make some streams

Readable Streams

Streams were added to node core, so we can use the fs library to make a readable stream.

   let read = fs.createReadStream('./original.txt')

Take a moment. Wipe the sweat from your brow.

Now that we have a readable stream, we can do something like this:

   read.on('data', (chunk) => {
      console.log(`Received ${chunk.length} bytes of data`)

Look at us! We're reading streams already because Javascript is good at it. We are but the shepherds of bytes.

But the power of streams isn't in making or using just one. The power of streams is that streams can be connected to each other. If you read up on this, you'll see plumbing metaphors. Hoses get connected to allow water (or hot fudge if you're in a chocolate factory) to pass through to its destination. After many hours of watching talks and discussing streams, here is the diagram that finally helped me understand.


Writable Streams

Maybe there's a reason most people understand plumbing. The point is that we can make something much more effective and powerful if we connect streams. Let's give a try.

   let write = fs.createWriteStream('./copy.txt')

We're getting there. A writeable stream is just what is sounds like: a place we can send data. In this case, I'm sending my secrets from the original file to this copy. Right now, we're using streams to file-hop, which is fun, but my secrets aren't encrypted yet. We need to add a transformation step. Hold on to your butts.

Transform Streams

   const fs = require('fs')
   const split = require('split2')
   const through2 = require('through2')

   let read = fs.createReadStream('./original.txt')
   let write = fs.createWriteStream('./copy.txt')

     .pipe(through2(function (chunk, enc, callback) {
       for (var i = 0; i < chunk.length; i++)
         chunk[i] = chunk[i] + 3


    .on('finish', function () {
      console.log('Created new encrypted file')

Follow me here. I'm using a couple small libraries: split2, which splits text on new lines, and through2, which brings us to the meat of the transformation. Through2 creates a transform stream, which is both readable and writable (similar to a Duplex stream if you're reading the docs). This means that we can modify the data while we read it and write it. In this example, the anonymous function is getting the byte code from each chunk (at this point, each character) and adding three, performing a mutated version of the Caesar Cipher, and putting into the new file all of my protected, gibberish secrets.

    My secrets: DOO#KDLO#K\SQRWRDGDovr/#orrn#dw#|rx#dqg#|rxu#elj#eudlq1


Now imagine that you wanted to read data from a CSV, turn it into the right object, and send it to your database. We could use the same architecture to accomplish that. A readable stream can retrieve the CSV data, a transform stream can modify the interface, and a write stream could insert the record into the DB. Imagine using streams to read and write your HTTP requests.

I can still write functions with encapsulated logic; in fact, this architecture lends itself to small, modular pieces of code that can be assembled in the pipeline easily. Honestly, that's how I want to solve problems. A great teacher once told me that when I'm stuck on a problem, I should ask myself: "What is the next smallest step I can take?" That advice has never let me down, and writing my code in a way that reflects that has helped me solve pretty hard problems.

I'd love to know how you've used streams, or where you'd like to see them used. In the meantime, I'll be exploring streams more, trying to write smarter, more useful code as I go.


About Katy

I'm a Developer Advocate at InfluxData (InfluxDB), a pretty sweet time series platform and a kick-ass place to work. I have great swag if you're into sock and stickers, and I'd love to talk about Node.js, Ruby, the abstract concept of time, cake, or Star Wars.

Twitter/GitHub: TheKaterTot
Email: Katy@influxdb

The contributors to JavaScript January are passionate engineers, designers and teachers. Emily Freeman is a developer advocate at Kickbox and curates the articles for JavaScript January.