118270d76f
Vendor the upstream changes with Avx512 ( #7225 )
...
Thanks to @fwessels we have Avx512 support with 4x improvement
6 years ago
3265112d04
Remove gateway implementations for manta, sia and b2 ( #7115 )
6 years ago
def04f01cf
Update reedsolomon/highwayhash to start using ppc64le support ( #7003 )
...
Thanks to @fwessels for the upstream work on reedsolomon and
highwayhash which has resulted in 10x performance improvement
on ppc64 architecture.
6 years ago
f162d7bd97
Performance improvements by re-using record buffer ( #6622 )
...
Avoid unnecessary pointer reference allocations
when not needed, for example
- *SelectFuncs{}
- *Row{}
6 years ago
7c14cdb60e
S3 Select API Support for CSV ( #6127 )
...
Add support for trivial where clause cases
7 years ago
c0721164be
Automatically set goroutines based on shardSize ( #5346 )
...
Update reedsolomon library to enable feature to automatically
set number of go-routines based on the input shard size,
since shard size is sort of a constant in Minio for
objects > 10MiB (default blocksize)
klauspost reported around 15-20% improvement in performance
numbers on older systems such as AVX and SSE3
```
name old speed new speed delta
Encode10x2x10000-8 5.45GB/s ± 1% 6.22GB/s ± 1% +14.20% (p=0.000 n=9+9)
Encode100x20x10000-8 1.44GB/s ± 1% 1.64GB/s ± 1% +13.77% (p=0.000 n=10+10)
Encode17x3x1M-8 10.0GB/s ± 5% 12.0GB/s ± 1% +19.88% (p=0.000 n=10+10)
Encode10x4x16M-8 7.81GB/s ± 5% 8.56GB/s ± 5% +9.58% (p=0.000 n=10+9)
Encode5x2x1M-8 15.3GB/s ± 2% 19.6GB/s ± 2% +28.57% (p=0.000 n=9+10)
Encode10x2x1M-8 12.2GB/s ± 5% 15.0GB/s ± 5% +22.45% (p=0.000 n=10+10)
Encode10x4x1M-8 7.84GB/s ± 1% 9.03GB/s ± 1% +15.19% (p=0.000 n=9+9)
Encode50x20x1M-8 1.73GB/s ± 4% 2.09GB/s ± 4% +20.59% (p=0.000 n=10+9)
Encode17x3x16M-8 10.6GB/s ± 1% 11.7GB/s ± 4% +10.12% (p=0.000 n=8+10)
```
7 years ago
6e6aeb6a9e
Updated version of klauspost/reedsolomon using proper AVX2 instructions as well a providing support for Cauchy matrices. ( #5215 )
7 years ago
02af37a394
optimize memory allocs during reconstruct ( #4964 )
...
The reedsolomon library now avoids allocations during reconstruction.
This change exploits that to reduce memory allocs and GC preasure during
healing and reading.
8 years ago
93f126364e
Updated version of klauspost/reedsolomon with NEON support for ARM ( #4865 )
8 years ago
85fcee1919
erasure: simplify XL backend operations ( #4649 ) ( #4758 )
...
This change provides new implementations of the XL backend operations:
- create file
- read file
- heal file
Further this change adds table based tests for all three operations.
This affects also the bitrot algorithm integration. Algorithms are now
integrated in an idiomatic way (like crypto.Hash).
Fixes #4696
Fixes #4649
Fixes #4359
8 years ago
fffe4ac7e6
Prevent unnecessary verification of parity blocks while reading ( #4683 )
...
* Prevent unnecessary verification of parity blocks while reading erasure
coded file.
* Update klauspost/reedsolomon and just only reconstruct data blocks while
reading (prevent unnecessary parity block reconstruction)
* Remove Verification of (all) reconstructed Data and Parity blocks since
in our case we are protected by bit rot protection. And even if the
verification would fail (essentially impossible) there is no way to
definitively say whether the data is still correct or not, so this call
make no sense for our use case.
8 years ago
faa6b1e925
vendorize deps for snappy, blake2b and sha256 ( #3476 )
...
Bring in new optimization and portability changes.
Fixes https://github.com/minio/minio-go/issues/578
8 years ago
8e6e9301ce
Add support for Kafka as a notifications target ( #2869 ) ( #3439 )
8 years ago
ed676667d0
vendor: update reedsolomon package with new changes. ( #2870 )
...
- Cached inverse matrices for better reconstruct performance.
- New error reconstruction required is returned, helpful in
initiating healing.
8 years ago
35d438e0ff
vendorize: update all vendorized packages. ( #2206 )
...
Bring in new changes from upstream for all the packages.
Important ones include
- gorilla/mux
- logrus
- jwt
9 years ago
7f38f46e20
vendor: update klauspost/reedsomon package with upstream changes. ( #1912 )
9 years ago
ba7a55c321
xl: ReedSolomon code fix small file erasure bug. ( #1431 )
...
For files less than 'dataBlocks', erasure encoding would fail
with short data due to a bug in the implementation itself.
Relax the error return, even a single byte can be properly
erasure coded without issues.
Fixes #1413
9 years ago
a9935f886c
vendor: update reedsolomon package with new perm improvements. ( #1422 )
9 years ago
a98a7fb1ad
Implement XL layer - preliminary work.
9 years ago
9e10ee7e47
cpu: Remove pkg/cpu in favor of better klauspost/cpuid.
...
Fixes #1128
9 years ago
Harshavardhana