Redis compression: LogicMonitor's 85% reduction

LogicMonitor cut Redis model storage by 85% using float16 arrays, Blosc compression and Base64 encoding. Redis compression happens in the application, not through a Redis server setting. I think the useful part is the comparison behind that result: how the encoded arrays stack up against LZ4 and decimal rounding, and whether the saved storage is worth the precision and processing costs.

If your service caches large numerical models as JSON, that comparison is relevant to your own storage decisions. In LogicMonitor's engineering write-up, a stateless anomaly-detection microservice needed to share training data across pods through centralized Redis clusters. As its numerical lists grew, so did the storage demand. Compressing those lists raised a separate question: how much processing would the application spend encoding and decoding each value?

Why LZ4 stopped delivering enough reduction

LZ4 worked well for Redis compression on the original model because there was redundancy to compress. LogicMonitor reports that lossless compression took that model from approximately 39,739 bytes to 16,208 bytes, a saving it describes as 60%. Much of the benefit came from repeated structure in the model's keys and values.

Then the workload changed. Longer floating-point lists more than doubled the model's size, but those lists had less statistical redundancy for the compressor to use. LogicMonitor also traced some of the stored string size to the long decimal representations associated with Python's default double-precision values.

The team was no longer just choosing a compressor. It also needed to examine the numbers being passed into that compressor. Rounding every float to four decimal places produced roughly the same storage saving as a more complicated rounding scheme that varied with magnitude. That gave the team a simpler representation to compare, rather than another compressor setting to adjust.

I'd treat this as a NoSQL data-modeling decision, not a Redis configuration change. The application has to produce the representation Redis stores, and it has to be able to consume that representation when the data comes back.

Redis compression results need separate baselines

LogicMonitor reports results for different model sizes. LZ4 used a smaller model, so its percentage saving is not directly comparable to the later results.

These percentages are not additive. LZ4 was tested on the earlier, smaller model. The later results describe the larger numerical workload, not successive reductions of that original payload. Reading the table as a sequence of savings against an unchanged model would give the wrong comparison.

For rounding, the team chose the uniform four-decimal rule over the magnitude-dependent rule. Their storage savings were similar, while the simpler rule avoided the additional complexity and reduced precision of magnitude-dependent rounding. The team also checked the effect on its algorithm and found it insignificant. Rounding was not the final choice, though: the later encoded-array approach produced a substantially smaller stored model.

Benchmark your Redis model before changing its format

The target for your evaluation is a useful comparison on your own model, not a reproduction of LogicMonitor's percentage. Its write-up describes the transformations, but it does not supply client calls or compressor APIs.

1. Record the existing model and transaction behaviour. Keep an uncompressed reference and record its size. Measure the elapsed time for Redis Get and Set, along with the application's Redis compression and decompression work. LogicMonitor compared transactions with and without prior compression. It also warns that the extra processing can increase latency and reduce throughput during peak load, so stored size is not the only measurement in this comparison.

   Test every alternative against the same captured model and access pattern. Without that consistency, a change in the data could pass for an improvement in compression effectiveness, even though the comparison itself had changed.

2. Evaluate LZ4 and decimal rounding separately before combining changes. LZ4 is the lossless-compression candidate; rounding tests whether the algorithm can accept fewer decimal digits. A smaller JSON value answers the storage question, but it does not establish that the analytical result is acceptable.

   This valid JSON fragment isolates a training-data field using numerical examples from LogicMonitor's discussion for the rounding comparison.

   {
     "algorithm-1-training-data": {
       "training-data-value": [12.43, 0.4312444]
     }
   }

   Compare the original with this four-decimal fixture, which uses the same training-data field.

   {
     "algorithm-1-training-data": {
       "training-data-value": [12.43, 0.4312]
     }
   }

   Run both the original and the rounded data through your algorithm, then decide whether the difference is acceptable. LogicMonitor validated rounding against its own algorithm. That result does not automatically validate rounding for yours.

3. Evaluate the encoded-array representation before converting the NumPy arrays into lists. LogicMonitor first converted those arrays to float16, then compressed them with Blosc using its internal ZLib codec, and then applied Base64 encoding. It decoded the encoded result as UTF-8 text before putting it into JSON.

   Keep the order of those operations explicit in the evaluation. The reported saving came from changing the numeric representation as well as compressing it. It was not a storage improvement attributable to Base64 alone.

4. Validate retrieval and operation under load before settling on a format. Check that the stored data can be recovered, then test the recovered values with the algorithm. LogicMonitor recommends error checking, backups and regular validation of compressed data. It also recommends monitoring performance as data volume and access patterns change, rather than relying only on the initial comparison.

   If the processing cost becomes the constraint, LogicMonitor suggests a lightweight compressor such as LZ4 as one alternative. Another is selective compression: compress only certain data types, or data that is accessed less frequently.

The 85% result leaves precision and peak-load questions

The write-up establishes the storage saving, but it does not give a numerical latency benchmark. LogicMonitor says compressed transactions showed substantial elapsed-time improvements. It does not publish the measurements needed to estimate an equivalent improvement for another service, so the storage figure cannot answer that performance question for you.

My main reservation is the accuracy evidence for the final representation. The article explicitly reports that decimal rounding had an insignificant effect on its algorithm, but it does not quantify accuracy for the final float16 pipeline. Those are different numerical representations. A successful rounding test should not take the place of validating the values recovered from the float16 pipeline.

Redis compression also gives the application another operational failure path. LogicMonitor warns that an altered bit could leave a compressed block unreadable, and that managing compressed data can become more complicated as a database grows. Its recommendations include error-detecting compression, backups and retrieval testing, as well as distributing load through Redis clustering.

That puts processing capacity beside stored bytes in the decision. An application whose memory use is acceptable but whose spare processing capacity is limited may make a different choice from an application dominated by large numerical payloads. The same storage saving does not settle both cases.

Choose the representation your model can tolerate

I'd start with LZ4 if preserving the existing numerical values is the priority. Rounding and float16 belong in the comparison only with an explicit accuracy check. For a workload resembling LogicMonitor's large training arrays, the encoded-array route deserves testing: it produced the smallest reported result, even though that result does not answer every deployment question.

Before choosing that route, run the captured model through it and test the recovered values with your algorithm. Measure the Get and Set path under load, and check that the application can reliably recover the stored representation.