In my blog post The uselessness of bash I made a tool to improve pipes in shell, to assemble a better pipeline.

It solves the problem, but it’s a bit too different, with its own language.

While complaining with some people at work that one of the main features of shell (the pipe operator) is broken, someone joked that it should be replaced by a protobuf based protocol.

But on second thought it’s not really a joke.

How about instead of this:

$ goodpipe <<EOF
[
  ["gsutil", "cat", "gs://example/input-unsorted.txt"],
  ["sort", "-S300M", "-n"],
  ["gzip", "-9"],
  ["gsutil", "cp", "-", "gs://example/input-sorted-numerically.txt.gz"]
]
EOF

how about this:

$ wp -o gsutil cat gs://example/input-unsorted.txt \
  | wp -io sort -S300M -n \
  | wp -io gzip -9 \
  | wp -i gsutil cp - gs://example/input-sorted-numerically.txt.gz

It doesn’t use protobufs, but a simpler regular protocol. This in order to avoid well known bugs types. Before implementing any protocol also see formal theory and science of insecurity.

First I hacked it together in Go, but I think the main implementation I’ll maintain is the one I made while porting it to Rust, as a way to learn Rust. The two implementations are compatible with each other.

I’ll write up my thoughts on Rust in a future blog post, but the summary is that it’s great, and ChatGPT was a helpful mentor in learning a new language.