There are some interesting ideas here. One thing that occurred to me while reading it was using a message queue to implement the pipeline functionality. I've been interested in Apache ActiveMQ/Camel for a while now and excited about some of the things it can do. With that model, each 'line' could have it's own set of headers, allowing for transformation by aware tools, but transparent by default. This would also blur the line between running commands and having background filters. It would also work nicely over the network.
I did some experiments with clojure/camel at one point, and came up with stuff like this:
Which is basically a continuous file copy. I could see different flags for "do now and exit", "do at HH:MM:SS", "do until I tell you to stop", "do when system is idle", etc.
I'm not sure what you'd really gain from using a message queue. Pipes with streams seems to be lower overhead and could be extended pretty trivially to get most of the functionality you'd want.
I did some experiments with clojure/camel at one point, and came up with stuff like this:
(defroute context (from "file:/home/jw/scratch/inbox?noop=true") (to "file:/home/jw/scratch/outbox")) ; from http://codeabout.blogspot.com/2010/06/using-apache-camel-fro...
Which is basically a continuous file copy. I could see different flags for "do now and exit", "do at HH:MM:SS", "do until I tell you to stop", "do when system is idle", etc.