C++ – Most elegant way to implement Pipe and Filter Pattern


I want to create a Pipe and Filter based data handler that should manipulate incoming data sets like so, but not neccessarily limited to:

source pipe(could be a data set from a db) <-sink-source-> filter(add an additional field) <-sink-source-> filter(manipulate some more data / remove ie nullify data set)

I have an idea how such architecture will look like in C/C++. But given all the goodies that come with C++11's functional aspects, I hope this task can be done in an elegant way so that it is able to:

  • easily scale and use it in a multithreaded environment (e.g. by performing filter tasks as lambda funcions, thus probably avoiding at least some thread handling)
  • easily add and remove filters at runtime
  • handle input streams lazily

Best Solution

There is a draft in the upcoming C++14 standard that covers this area:

C++ Pipelines - ISO/IEC JTC1 SC22 WG21 N3534 = 2013-03-15

And here is an implementation for it: