Managing dictionary-based compression in Amazon DocumentDB 8.0

Amazon DocumentDB 8.0 introduces a new document compression algorithm, Zstd, as an improved alternative to LZ4. When you enable dictionary compression on an Amazon DocumentDB 8.0 collection by selecting Zstd as the compression algorithm, documents from your collection are sampled to create a custom compression dictionary. New and updated documents are compressed using this dictionary and the Zstd algorithm. This approach typically achieves higher compression ratios than standard compression methods, especially for collections with consistent document schemas or repeated field names.

LZ4 is an algorithm designed for fast compression and decompression. It is known to be light on the CPU while achieving noticeable compression. Zstd is a general-purpose algorithm that with default settings uses more CPU but achieves better compression ratios than LZ4. The usage of dictionaries improves compression even further for most JSON documents. Some benefits of the Zstd algorithm are:

Performance considerations

Zstd compression involves the following trade-offs:

Zstd compression is particularly effective for collections with small documents, document arrays and repeated field names.

Enabling dictionary-based compression

For new collections you can use the below command to enable Zstd compression:

db.createCollection("myCollection", 
    { 
        storageEngine: { 
            documentDB: { 
                compression: {
                    enable: true,
                    algorithm: "zstd"
                } 
            }
        }
    }
 )
        

You can also enable or modify compression on an existing collection:

db.runCommand({ 
    collMod: "myCollection", 
    storageEngine: { 
        documentDB: { 
            compression: {
                enable: true,
                algorithm: "zstd"
            } 
        }
    }
 })
        

To enable the Zstd algorithm for all new collections in your cluster, set the default_collection_compression parameter in your cluster parameter group to zstd.

Getting Started

In Amazon DocumentDB 8.0, new collections use Zstd compression by default. To turn off compression for new collections, set the default_collection_compression parameter in your cluster parameter group to disabled.

Starting with Amazon DocumentDB 8.0, enabled is no longer a valid value for default_collection_compression. The valid values are zstd, lz4, and disabled.

Monitoring

You can view compression information for a collection using one of the following commands:

These commands return key statistics that you can use to compute compression ratio:

The following CloudWatch metrics can be helpful while evaluating dictionary-based compression:

collStats metrics:

In addition, it can be useful to keep track of metrics specific to your application, like latency and throughput per query type or API.

Compressor statistics

Starting with Amazon DocumentDB 8.0.3, the collStats command and the db.collection.stats() method return a compressor_stats object inside the compression field. It reports how much data the compression algorithm has processed for the collection, and works for both Zstd and LZ4.

Use compressor statistics to see how effectively compression reduces the size of the data that you write to a collection. For example, you can use them to compare Zstd and LZ4 on your workload, or to evaluate a change to the compression threshold.

To view compressor statistics for a collection, run the following command:

db.runCommand({ collStats: "myCollection" })
            

For a collection that uses Zstd, the output includes:

"compression": {
    "enable": true,
    "threshold": 128,
    "algorithm": "zstd",
    "usingDictionary": true,
    "compressor_stats": {
        "input_bytes": 10000000,
        "output_bytes": 2500000,
        "compression_operations": 10000,
        "compression_ratio": 4
    }
}
            

For a collection that uses LZ4, the output doesn't include usingDictionary:

"compression": {
    "enable": true,
    "threshold": 128,
    "algorithm": "lz4",
    "compressor_stats": {
        "input_bytes": 10000000,
        "output_bytes": 2500000,
        "compression_operations": 10000,
        "compression_ratio": 4
    }
}
            

The following table describes the fields in the compression object.

In these examples, compression reduces 10,000 documents totaling 10 MB to 2.5 MB, a 4:1 compression ratio.

How compressor statistics accumulate

Compressor statistics are cumulative totals of compression operations, and they don't represent the collection's current data size. The following table describes how operations on the collection affect these statistics.

Choosing the right metric

Compressor statistics and collection size statistics measure different things, so their ratios usually don't match. compression_ratio covers only data that the compressor processed. size/storageSize covers the whole collection, including uncompressed documents and page fragmentation.

The following table shows which metric to use, depending on your goal.