Introduction

The UNIX pipes system allows one to quickly compose computations from existing building blocks. However, at times it can be aggravating that one cannot ‘get at the stream’ that a program is using. With that ability, we could, for example, watch a video stream at the same time that it moves to long-term storage. We could compute statistics on how often we write versus read particular files, which might inform an archival storage policy. We could provide visualizations of simulation data as it streams to disk.

Such uses are difficult in modern pipeline systems. One can pre-create a pipe as the target of the operations given above, and then write a multiplexing program that implements the tee(1)-like nature we require. This hack, however, fails to provide a useful solution when data are moving out as opposed to in. Finally, it is hard to envision a named-pipe-based solution providing useful semantics in a parallel environment that cannot guarantee an ordering between writers or any way to distinguish them.

Freeprocessing is a system for injecting code into these hard to reach areas. All of these use cases and many more can be realized by utilizing the appropriate freeprocessor. These freeprocessor modules plug-in to what we call the symbiont to implement the desired processing at the desired time. In short, Freeprocessing gives you access to any data I/O in any program, allowing you to change any data movement operation into an opportunity for further processing.

Your first freeprocessor

A freeprocessor is a module that plugs in to the Freeprocessing symbiont.
[We use the terms “freeprocessor” and “module” interchangeably in this document.]
 The module’s code is executed whenever a data movement operation is detected. Operation of the original program is suspended while the freeprocessor executes.

Using the term ‘module’ is perhaps too grandiose: a freeprocessor is actually just a library, in the native format of the target system. The following shell commands generate a “freeprocessor” on a Linux system:

touch nothing.c
gcc -fPIC -shared nothing.c -o libfp.so

Of course, this “module” is not very interesting. Let’s start with the classic example: Hello, world. In Freeprocessing, the ‘standard’ entry point is not called "main", it is "exec":

#include <stdio.h>
void
exec(const char* fn, const void* buf, size_t n) {
  (void)fn; (void)buf; (void)n;
  printf("Hello, world!\n");
}

The (void) nonsense is a trick to convince C compilers not to warn us about unused variables.

Now that we have defined the exec function, we have a valid freeprocessor. First, compile it the same way we compiled our nothing.c example. Then, let’s load it up and see it running.

Using freeprocessors

We need three things to execute a freeprocessor:

  1. the freeprocessor in library form, of course;

  2. a program to inject our code into; and

  3. a configuration file that specifies how the first two connect together.

For the program, we can use almost any program on our system. However, for the sake of demonstration it will be clearer to define our own program. Here is a simple C program that opens a file and writes an (integer) argument into it.
[It is worth noting that most of the code from this document is available in the Freeprocessing source tree. See the processors/hello/ directory for the code in this introductory tutorial.]

#include <stdio.h>
#include <stdlib.h>

int
main(int argc, char* argv[]) {
  if(argc != 2) {
    fprintf(stderr, "expecting integer argument!\n");
    return EXIT_FAILURE;
  }
  const int value = atoi(argv[1]);
  FILE* fp = fopen("test-int", "wb");
  if(!fp) {
    fprintf(stderr, "could not open 'test-int' file for writing!\n");
    return EXIT_FAILURE;
  }
  if(fwrite(&value, sizeof(int), 1, fp) != 1) {
    fprintf(stderr, "oh noes!  write failed!\n");
    fclose(fp);
    return EXIT_FAILURE;
  }
  if(fclose(fp) != 0) {
    fprintf(stderr, "file close failed; data didn't make it to disk\n");
    return EXIT_FAILURE;
  }
  return EXIT_SUCCESS;
}

Compile this as you would any C program; we will refer to it here as the program hw. You may need to put your compiler into C99 mode for it to understand the code.

Now let’s run hw instrumented with Freeprocessing. You could have installed the main Freeprocessing library, libsitu.so, anywhere, so I’ll just refer to it as /path/to/freeprocessing/libsitu.so. You should substitute the path to your Freeprocessing install here.

$ LD_PRELOAD=/path/to/freeprocessing/libsitu.so ./hw 42
[8660](fp_init) could not find a 'situ.cfg'; will not apply any processing.
Note
Mac OS X
On Mac, the environment variable is DYLD_INSERT_LIBRARIES, not LD_PRELOAD. Often, one needs to set DYLD_FORCE_NAMESPACE to 1 as well.

Your output will be slightly different. Here, my hw process happened to get a process ID (pid) of 8660. However, during its initialization, it complained that it could not find a situ.cfg file, and refused to do anything! We’ll need to define that before Freeprocessing will do anything.

situ.cfg maps stream names to processing code. It supports an arbitrary number of mappings, but for now we just want one. Copy this into a situ.cfg in your current directory:

test-int { exec: ./.libs/libhello.so }

This configuration file simply says that any stream acting on the file test-int should utilize the freeprocessor compiled to ./.libs/libhello.so. We could have also used a wildcard to refer to the filename, such as test-* or even just *.

Now we can finally execute our freeprocessor:

$ LD_PRELOAD=/path/to/freeprocessing/libsitu.so ./hw 42
Hello, world!
Tip
Debugging
Freeprocessing includes a number of debug channels. By default, these channels are fairly quiet. However, when debugging issues with a freeprocessor or simply because we’d like to know what’s going on under the hood, we can ask Freeprocessing to enable these with the LIBSITU_DEBUG environment variable. Try setting this to "opens=+trace" and re-running the above example. More information is available on the website.

Congratulations, you have now developed and executed your first freeprocessor! In the next section, we will describe how to get at the data as it flows through your freeprocessor.

Min/Max freeprocessor

To demonstrate the handling of data, we will expand our freeprocessor to compute the minimum and maximum values of a stream passed through it. It is tempting to implement this by simply pulling the data out of our stream, computing the min/max, and reporting it. Create a new file named minmax.c with the following contents:

00001: /* Calculate the min/max of integer data that flows through it.
00002:  * WARNING: this has a subtle bug, do not use directly! */
00003: #include <limits.h>
00004: #include <stdio.h>
00005:
00006: void
00007: exec(const char* fn, const void* buf, size_t n) {
00008:   (void)fn;
00009:   const int* data = (const int*)buf;
00010:   const size_t elems = n / sizeof(int);
00011:   int minmax[2] = { INT_MAX, INT_MIN };
00012:   for(size_t i=0; i < elems; ++i) {
00013:     minmax[0] = data[i] < minmax[0] ? data[i] : minmax[0];
00014:     minmax[1] = data[i] > minmax[1] ? data[i] : minmax[1];
00015:   }
00016:   printf("The data range is: [%d:%d]\n", minmax[0], minmax[1]);
00017: }

In lines 9 and 10 we see how to get at the underlying data from the stream: the buf and n parameters in this case. Here, we assume our data will come in the form of integers; if this assumption is false, then our freeprocessor will compute incorrect values. 11 initializes our minimum and maximum values, our loop computes these values, and finally our output comes on line 16.

Important
Metadata
Obtaining metadata (such as the underlying stream type int we used here) for our stream information is a thorny issue in Freeprocessing. The general mantra is to let the user do whatever they want, even if it’s dangerous. Thus, the system does not impose any particular method for obtaining metadata, or rather, does not provide any judgement on whether "read metadata from a file" is any better or worse than "assume we have IEEE 754 floating-point values". This gives flexibility and power, but can be aggravating when one desires to produce idiomatic freeprocessors. For the beginning stages of this tutorial, we will embed any metadata into the structure of the code itself, for simplicity. Later we will develop and advocate standardized mechanisms to obtain and utilize metadata.

The trouble with this approach is due to the model that Freeprocessing uses for the user’s exec function. exec is executed every time the instrumented program performs any kind of write statement. The fwrite in the hw program from the previous section translates directly into a single call of exec. If we were to call fwrite in a loop, then exec would execute as many times as the loop does. In that case, we’d be calculating the minimum and maximum of each write, as opposed to the minimum and maximum over all writes.

To fix this, we need a mechanism to only perform our printf at the end of the computation, instead of each intermediate value. Furthermore, we need to initialize our minimum and maximum once, before any processing takes place; if we initialize it during exec, then we will kill any intermediate computation from a previous exec invocation.

Freeprocessors implement these two ideas using the functions file and finish. Add these functions to your minmax.c implementation so that it reads:

static int minmax[2];

void
file(const char* fn) {
  (void)fn;
  minmax[0] = INT_MAX;
  minmax[1] = INT_MIN;
}
void
finish(const char* fn) {
  printf("The data range of %s is: [%d:%d]\n", fn, minmax[0], minmax[1]);
}
void
exec(const char* fn, const void* buf, size_t n) {
  (void)fn;
  const int* data = (const int*)buf;
  const size_t elems = n / sizeof(int);
  for(size_t i=0; i < elems; ++i) {
    minmax[0] = data[i] < minmax[0] ? data[i] : minmax[0];
    minmax[1] = data[i] > minmax[1] ? data[i] : minmax[1];
  }
}

file is invoked whenever a new file matching the situ.cfg mapping is opened. This is therefore a good place to perform any initialization actions, such as reading metadata or obtaining external resources. finish, on the other hand, is invoked when the instrumented program will no longer utilize a given stream, and is therefore the place to clean up any needed resources.

Note
C++
The three aforementioned functions closely follow the "initialize", "use" and "cleanup" methodology common in object-oriented programming. For those of us who prefer to use C++, you might prefer to create a new object in file, manipulate that object in exec, and delete the object in finish.

Now we have a valid freeprocessor which can compute the min/max of all the data pushed through it. Congratulations! This is the first freeprocessor that is conceivably useful in practice. The full code for this processor is available in the processors/minmax/ directory of the Freeprocessing git repository.

Example: real world Freeprocessing with scp

Caution not yet written, check back soon!

We’ll have an example of using Freeprocessing with scp here.

Example: no situ example (producer/consumer model)

Freeprocessing was designed to make a variety of in situ visualization scenarios easy to implement. However, it can also be used as a simple mechanism for chaining dependent tasks. In this scenario, a program creates many different outputs in sequence; each output, in turn, requires some post-processing by a separate program. You might also think of this scenario as ‘poor man’s in situ’.

Implementing this in the standard UNIX paradigm is difficult. The problem is that the producers and consumers need some method of synchronization: the consumers cannot start until the producers create a data set. Furthermore, the production of N outputs is sequentially performed by a single process. One solution is to add code to the producing program such that it creates and manages consumers. A second solution is to add code to consumers to (heuristically) identify when a producer has created an output. Neither solution is desirable.

The crux of the issue is that every producer/consumer system has a "hidden" third entity: the synchronization. This issue infects both sides of the producer/consumer problem: producers must create notifications and consumers must ingest them. It would be much better if we could develop the three components: producers; consumers; and synchronization mechanisms; in isolation.

This can be achieved with Freeprocessing. The essence of the idea is quite simple:

void
finish(const char* fn) {
  char cmd[1024];
  snprintf(cmd, 1024, "pvbatch --use-offscreen-rendering script.py %s", fn);
  consumer(cmd);
}

That is, all we need is a finish function which fires off the consumer process. We can attach this freeprocessor to the producer, and then whenever it finishes with a file, it will automatically create a new consumer.

The devil is, of course, in the details. The missing consumer function here must implement the standard UNIX bookkeeping for creating and managing processes. Furthermore, in practice one would want to maintain a queue or pool of produced resources, and fire off a limited number of consumers.

A ready-made freeprocessor for this use case is available in the source tree, under processors/nositu/.

Metadata

The data passed through a freeprocessor is one-dimensional, untyped binary data. This is much like when a collaborator sends a binary data file: the data is not useful until you receive some additional metadata that explains what the data’s format is. One needs to impose the data’s organization on the binary stream to make it useful.

Freeprocessing lets you do this however you want. However, we espouse a particular method here in the interests of quickly getting a working system. That method is to have a secondary file which contains this metadata, and parse out the information you need. For generality, we recommend JSON for this. Specifically, we’ll seek to parse files such as this:

{
"dims": [ {"idx":3}, {"idx":3}, {"idx":3} ],
"type": "char"
}

The verbosity of the "idx"s is unfortunately necessary to conform to JSON.

There is sample code in the repository to get you started using this kind of JSON. This is located in contrib/json-starter/; we recommend you copy all of them into your freeprocessor’s directory and use them that way. The json.[ch] files are a third-party JSON parser; documentation for it is outside the scope of this document. The jsdd.[ch] files provide a number of functions that make parsing out our form of JSON easier. They can also be customized to define/parse your own additions to the format.

You will first have to read and parse the JSON file. You can do this with the aptly-named json_parse function:

char* config = slurp(filename);
json_value* js = json_parse(config, strlen(config));
free(config);

Here slurp is a hypothetical function which reads an entire file into an appropriately-sized string. As seen, once the JSON data is parsed, we can throw away any of the memory it needed during the parse; all data is self-contained in the returned js value.

The type information parses out the information from the "type" component of the JSON, and converts that into a dtype enumeration:

enum dtype type = js_datatype(js);
if(GARBAGE == type) {
  fprintf(stderr, "Error getting type from JSON!\n");
  exit(EXIT_FAILURE);
}

There could be any number of dimensions to the data. Thus, the js_dimensions function takes a second argument that tells us how many dimensions it read back. The data we get back is allocated memory, so make sure to free it when you are done.

size_t ndims = 0;
size_t* dims = js_dimensions(js, &ndims);
printf("dimensions: [ ");
for(size_t i=0; i < ndims; ++i) {
  printf("%zu ", dims[i]);
}
printf("]\n");
free(dims);

Finally, when you are completely done reading information from the JSON data, make sure to free that memory too:

json_value_free(js);

The file jsdd.c contains a number of routines useful for parsing out any additional properties.

Python

One of the many freeprocessors which comes with the Freeprocessing distribution is a Python processor named freepython. Note this is only built if you pass the --enable-python option when you configure the package.

The module does a few things:

  1. Creates a Python module named freeprocessing and imports it.

  2. Exports a number of values from the freeprocessing module:

    • field: the name of the field

    • rank: MPI process rank of running process

    • stream: the stream data as a numpy array

    • timestep: the number of files opened thus far

  3. Executes the script vis.py in the current directory.

The python freeprocessor works slightly different than the others. First, it expects to be run on HDF5 files. HDF5 provides additional metadata that the freeprocessor uses to provide the correct shape for freeprocessing.stream. Furthermore, since HDF5 output data is always "interesting" from a visualization standpoint, the patterns listed in situ.cfg filter on the dataset name (as used in HDF5) instead of the file name. This makes it considerably easier to pull out just a particular field in a multi-field HDF5 file.