Updated 2016-04-07 14:42:47 by pooryorick

Reading and writing to a piped command

See Also  edit

How can I run data through an external filter?

Description  edit

The methods for accomplishing some task involving writing to and then reading from a piped command depend primarily on whether the communication will be interactive or non-interactive. The non-interactive case is fairly straight-forward and robust. The interactive case can be problematic, since external programs may buffer incoming and outgoing data in arbitrary ways. Expect is the best all-purpose tool for interactive communication with another program.

Example: Non-Interactive: gzip  edit

proc gzip {buf} {
    set chan [open "|gzip -c" r+]
    fconfigure $chan -translation binary -encoding binary
    puts $chan $buf
    flush $chan
    chan close $chan write
    set buf [read $chan]
    close $chan
    return $buf

The write side of $chan must be closed so that gzip reads any remaining data in the buffers processes it, and sends the result to its stdtout.

For versions of Tcl that don't have [chan close ?direction?], try the following workaround:
proc gzip {buf} {
    return [exec gzip -c << $buf]

Example: Interactive Conversation with another Program  edit

#! /bin/env tclsh

proc communicate {chan msg pump respond eof } {
    fileevent $chan readable [list apply [list {chan respond eof} {
        set res [gets $chan]
        if {$res ne {}} {
            fileevent $chan readable {}
            {*}$respond $res
        if {[eof $chan]} {
            fileevent $chan readable {}
    }] $chan $respond $eof]

    fileevent $chan writable [list apply [list {chan msg pump} {
        catch {puts $chan $msg}
        #something like this may be needed if the child program can be configured
        #with reasonable output buffering
        #fileevent $chan writable [list apply [list {chan pump} {
        #       puts $chan $pump
        #}] $chan $pump]
        fileevent $chan writable {}
    }] $chan $msg $pump]

proc process {chan count args} {
    puts "sed output: $args"
    if {$count < 2} {
        communicate $chan hello \n [namespace code [list process $chan [incr count]]] \
            [namespace code [list closed $chan]]
    } else {
        catch {close $chan}
        set ::done 0

proc closed {chan} {
    #flush first to catch any pipe error, getting it out of the way in order to grab
    #the exist status with [close]
    if {[catch {flush $chan} eres eopts]} {
        set status [catch {close $chan} eres eopts]
        set ::done $status
    set ::done 0

#try this line, which causes a sed error, to see how that's handled
#set chan [open {|sed -l {s/hello/goodbye} 2>@stderr} r+]

set chan [open {|sed -l s/hello/goodbye/ 2>@stderr} r+]
fconfigure $chan -buffering none

communicate $chan hello \n [namespace code [list process $chan 1]] \
    [namespace code [list closed $chan]]

vwait ::done
return -code $::done

It's up to the individual program when to print out its results. The -l option to BSD sed configures sed to print output whenever at least one line of output is ready. -u accomplishes the same for some other versions of sed. The pump feature of this example can be used pump some program-specifc value through the channel until the desired output is collected. It's a hack, but depending on the program, might be the only way to accomplish the task. Expect is the fully-featured tool for this type of task. Another approach would be to fork the current process into a producer and a consumer.

Neil Madden  edit

Neil Madden - here is a real-life example of interacting with a program through a pipe. The program in question is ispell - a UNIX spell-checking utility. I use it to spell-check the contents of a text widget containing LaTeX markup. There are a number of issues to deal with:

  • Keeping track of the position of the word in the widget.
  • Filtering out useless (blank) lines from ispell
  • Filtering out the version info that my version of ispell dumps out.
  • When passing in a word which is a TeX command (e.g. \maketitle), ispell returns nothing at all.
  • Careful handling of blocking.

The example does not use [fileevent], as this would complicate this particular example. The options passed to ispell are -a (which makes it non-interactive) and -t (which makes it recognize TeX input).
set contents [split [$text get 1.0 end] \n]
set pipe [open [list | ispell -a -t] r+]
fconfigure $pipe -blocking 0 -buffering line
set ver [gets $pipe] ;# Ignore the initial version line
set linenum 1
foreach line $contents {
    set wordnum 1
    foreach word [split $line] {
        puts $pipe $word   ;# Feed word to ispell
        while 1 {
            set len [gets $pipe res]
            if {$len > 0} {
                # A valid result
                # do stuff
            } else {
                if {[fblocked $pipe]} {
                    # No output
                } elseif {[eof $pipe]} {
                    # Pipe closed
                    catch {close $pipe}
                # A blank line - skip
        incr wordnum
    incr linenum

Thanks to Kevin Kenny for helping me figure this out.

To Sort: Arjen Markus  edit

Arjen Markus: I have experimented a bit with plain Tcl driving another program. As this needs to work on Windows 95 (NT, ...) as well as UNIX in four or five flavours, I wanted to use plain Tcl, not Expect (however much I would appreciate the chance to do something really useful with Expect - apart from Android :-).

I think it is worth a page of its own, but here is a summary:

  • Open the pipeline and make sure buffering is minimal via
set inout [open |[list myprog] r+]
fconfigure $inout -buffering line

Buffering might be out of your hands, though, for "real 16-bit commandline applications", which apparently don't have the ability to flush (reliably), except on close.

  • Set up [fileevent] handlers for reading from the process and reading from stdin.
  • Make sure the process ("myprog" above) does not buffer its output to stdout!