This post will cover a simple way to compress simulation data in memory using ZLib. The goal is to write out compressed data directly without having to deal with binary data formats.
I do a lot of rendering with PovRay which means that I need to store comma separated ascii files because that is the only data format supported. The problem is that when I’m saving data for millions of objects the files can become several hundred MB in size.
I propose one solution where all simulation data is stored compressed and a simple program can be used to decompress it and write it out as povray compatible files when needed.
My data is stored in a std::stringstream which I normally stream out to a file. I would like to directly compress this stream and when I need the data, decompress it back into a string.
ZLib is a library that is found on pretty much every platform. Most package managers will have it, windows users can get it here
My build environment uses CMake and adding ZLib is trivial
In your program include ZLib
Writing out compressed data
Reading in compressed data
At this point the string data holds the data that was in the string stream and can be written into a temporary file for povray or processed.
There are better compression algorithms for text but I chose to use ZLib because of it’s simplicity and availability