Question

When I came through readable stream part of stream-handbook,

with NodeJS code,

var Readable = require('stream').Readable;
var rs = Readable();

var c = 97 - 1;

rs._read = function () {
    if (c >= 'z'.charCodeAt(0)) return rs.push(null);

    setTimeout(function () {
        rs.push(String.fromCharCode(++c));
    }, 100);
};

rs.pipe(process.stdout);

process.on('exit', function () {
    console.error('\n_read() called ' + (c - 97) + ' times');
});
process.stdout.on('error', process.exit);

piped by head -c5, will only generate 5 characters.

$ node read2.js | head -c5
abcde
_read() called 5 times

I was confused because I think data always generated by process before pipe,

node -> data -> head
     1       2

But it seemed that it's possible data was generated in demands(just like lazy evaluation).

After googling for a while, I begin to understand how pipe works internally.

Solution

After the shell command node read2.js | head -c5 executed, the following incident happened in sequence

Unix shell arrange file descriptors fd open for reading and writing
node and head processes start at the same time
node call write(2), add data "a" to fd
head call read(2), return data "a" from fd
head call read(2), block because fd don't have enough data for read
rs._read is triggered, node call write(2), add data "b" to fd ...
head read 5 lines and exit
pipe is shut down
node exit and receive SIGPIPE signal

A general version of how pipe works could quote Leif Walsh's answer

All processes in the pipeline start at the same time. When one in front of a pipe calls write(2), the data is added to a stream of data buffered internally by the kernel. When one after a pipe calls read(2), anything in the stream (up to the max asked for) is returned. If there is not enough data in the stream, read(2) blocks until there is. If the stream buffer is full when the first process calls write(2), then that call blocks until the stream gets emptied enough.

To shut down a pipe, either process can call close(2) on the corresponding file descriptor, or can simply exit (forcing an implicit close). The next time a "source" process (as opposed to a "sink" process) attempts to write to the pipe, it will receive a SIGPIPE. The default signal handler for SIGPIPE typically kills the process, but a programmer can handle SIGPIPE in some other way appropriate to his application.

Referrences

How does unix pipeline work?
Unix Pipes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!