You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
133 lines
4.8 KiB
133 lines
4.8 KiB
# Compression Guide [](https://slack.min.io)
|
|
|
|
MinIO server allows streaming compression to ensure efficient disk space usage.
|
|
Compression happens inflight, i.e objects are compressed before being written to disk(s).
|
|
MinIO uses [`klauspost/compress/s2`](https://github.com/klauspost/compress/tree/master/s2)
|
|
streaming compression due to its stability and performance.
|
|
|
|
This algorithm is specifically optimized for machine generated content.
|
|
Write throughput is typically at least 500MB/s per CPU core,
|
|
and scales with the number of available CPU cores.
|
|
Decompression speed is typically at least 1GB/s.
|
|
|
|
This means that in cases where raw IO is below these numbers
|
|
compression will not only reduce disk usage but also help increase system throughput.
|
|
Typically, enabling compression on spinning disk systems
|
|
will increase speed when the content can be compressed.
|
|
|
|
## Get Started
|
|
|
|
### 1. Prerequisites
|
|
|
|
Install MinIO - [MinIO Quickstart Guide](https://docs.min.io/docs/minio-quickstart-guide).
|
|
|
|
### 2. Run MinIO with compression
|
|
|
|
Compression can be enabled by updating the `compress` config settings for MinIO server config.
|
|
Config `compress` settings take extensions and mime-types to be compressed.
|
|
|
|
```bash
|
|
~ mc admin config get myminio compression
|
|
compression extensions=".txt,.log,.csv,.json,.tar,.xml,.bin" mime_types="text/*,application/json,application/xml"
|
|
```
|
|
|
|
Default config includes most common highly compressible content extensions and mime-types.
|
|
|
|
```bash
|
|
~ mc admin config set myminio compression extensions=".pdf" mime_types="application/pdf"
|
|
```
|
|
|
|
To show help on setting compression config values.
|
|
```bash
|
|
~ mc admin config set myminio compression
|
|
```
|
|
|
|
To enable compression for all content, no matter the extension and content type
|
|
(except for the default excluded types) set BOTH extensions and mime types to empty.
|
|
|
|
```bash
|
|
~ mc admin config set myminio compression enable="on" extensions="" mime_types=""
|
|
```
|
|
|
|
The compression settings may also be set through environment variables.
|
|
When set, environment variables override the defined `compress` config settings in the server config.
|
|
|
|
```bash
|
|
export MINIO_COMPRESS="on"
|
|
export MINIO_COMPRESS_EXTENSIONS=".txt,.log,.csv,.json,.tar,.xml,.bin"
|
|
export MINIO_COMPRESS_MIME_TYPES="text/*,application/json,application/xml"
|
|
```
|
|
|
|
### 3. Compression + Encryption
|
|
|
|
Combining encryption and compression is not safe in all setups.
|
|
This is particularly so if the compression ratio of your content reveals information about it.
|
|
See [CRIME TLS](https://en.wikipedia.org/wiki/CRIME) as an example of this.
|
|
|
|
Therefore, compression is disabled when encrypting by default, and must be enabled separately.
|
|
|
|
Consult our security experts on [SUBNET](https://min.io/pricing) to help you evaluate if
|
|
your setup can use this feature combination safely.
|
|
|
|
To enable compression+encryption use:
|
|
|
|
```bash
|
|
~ mc admin config set myminio compression allow_encryption=on
|
|
```
|
|
|
|
Or alternatively through the environment variable `MINIO_COMPRESS_ALLOW_ENCRYPTION=on`.
|
|
|
|
### 4. Excluded Types
|
|
|
|
- Already compressed objects are not fit for compression since they do not have compressible patterns.
|
|
Such objects do not produce efficient [`LZ compression`](https://en.wikipedia.org/wiki/LZ77_and_LZ78)
|
|
which is a fitness factor for a lossless data compression.
|
|
|
|
Pre-compressed input typically compresses in excess of 2GiB/s per core,
|
|
so performance impact should be minimal even if precompressed data is re-compressed.
|
|
Decompressing incompressible data has no significant performance impact.
|
|
|
|
Below is a list of common files and content-types which are typically not suitable for compression.
|
|
|
|
- Extensions
|
|
|
|
| `gz` | (GZIP)
|
|
| `bz2` | (BZIP2)
|
|
| `rar` | (WinRAR)
|
|
| `zip` | (ZIP)
|
|
| `7z` | (7-Zip)
|
|
| `xz` | (LZMA)
|
|
| `mp4` | (MP4)
|
|
| `mkv` | (MKV media)
|
|
| `mov` | (MOV)
|
|
|
|
- Content-Types
|
|
|
|
| `video/*` |
|
|
| `audio/*` |
|
|
| `application/zip` |
|
|
| `application/x-gzip` |
|
|
| `application/zip` |
|
|
| `application/x-bz2` |
|
|
| `application/x-compress` |
|
|
| `application/x-xz` |
|
|
|
|
All files with these extensions and mime types are excluded from compression,
|
|
even if compression is enabled for all types.
|
|
|
|
### 5. Notes
|
|
|
|
- MinIO does not support compression for Gateway (Azure/GCS/NAS) implementations.
|
|
|
|
## To test the setup
|
|
|
|
To test this setup, practice put calls to the server using `mc` and use `mc ls` on
|
|
the data directory to view the size of the object.
|
|
|
|
## Explore Further
|
|
|
|
- [Use `mc` with MinIO Server](https://docs.min.io/docs/minio-client-quickstart-guide)
|
|
- [Use `aws-cli` with MinIO Server](https://docs.min.io/docs/aws-cli-with-minio)
|
|
- [Use `s3cmd` with MinIO Server](https://docs.min.io/docs/s3cmd-with-minio)
|
|
- [Use `minio-go` SDK with MinIO Server](https://docs.min.io/docs/golang-client-quickstart-guide)
|
|
- [The MinIO documentation website](https://docs.min.io)
|
|
|