minio/pkg/storage/erasure/doc.go

// Package erasure is a Go wrapper for the Intel Intelligent Storage
// Acceleration Library (Intel ISA-L).  Intel ISA-L is a CPU optimized
// implementation of erasure coding algorithms.
//
// For more information on Intel ISA-L, please visit:
// https://01.org/intel%C2%AE-storage-acceleration-library-open-source-version
//
// Usage:
//
// Encode encodes a block of data. The input is the original data. The output
// is a 2 tuple containing (k + m) chunks of erasure encoded data and the
// length of the original object.
//
// Decode decodes 2 tuple data containing (k + m) chunks back into its original form.
// Additionally original block length should also be provided as input.
//
// Decoded data is exactly similar in length and content as the original data.
//
// Encoding data may be performed in 3 steps.
//
//  1. Create a parse set of encoder parameters
//  2. Create a new encoder
//  3. Encode data
//
// Decoding data is also performed in 3 steps.
//
//  1. Create a parse set of encoder parameters for validation
//  2. Create a new encoder
//  3. Decode data
//
// Encoder parameters contain three configurable elements:
//  ParseEncoderParams(k, m, technique int) (EncoderParams, error)
//  k - Number of rows in matrix
//  m - Number of colums in matrix
//  technique - Matrix type, can be either Cauchy (recommended) or Vandermonde
//  constraints: k + m < Galois Field (2^8)
//
// Choosing right parity and matrix technique is left for application to decide.
//
// But here are the few points to keep in mind
//
//  Techniques:
//     - Vandermonde is most commonly used method for choosing coefficients in erasure
//       encoding but does not guarantee invertable for every sub matrix.
//       Users may want to adjust for k > 5. (k is data blocks)
//     - Whereas Cauchy is our recommended method for choosing coefficients in erasure coding.
//       Since any sub-matrix of a Cauchy matrix is invertable.
//
//  Total blocks:
//     - Data blocks and Parity blocks should not be greater than 'Galois Field' (2^8)
//
// Example
//
// Creating and using an encoder
//  var bytes []byte
//  params := erasure.ParseEncoderParams(10, 5, erasure.Cauchy)
//  encoder := erasure.NewEncoder(params)
//  encodedData, length := encoder.Encode(bytes)
//
// Creating and using a decoder
//  var encodedData [][]byte
//  var length int
//  params := erasure.ParseEncoderParams(10, 5, erasure.Cauchy)
//  encoder := erasure.NewEncoder(params)
//  originalData, err := encoder.Decode(encodedData, length)
//
package erasure