f503ac3db8
XL/Erasure: Make bit-rot verification based on xl.json algo. ( #2299 )
...
Currently `xl.json` saves algorithm information for bit-rot
verification. Since the bit-rot algo's can change in the
future make sure the erasureReadFile doesn't default to
a particular algo. Instead use the checkSumInfo.
9 years ago
a2b6f0524d
XL/erasure: Remove deprecated copyN function. ( #2288 )
9 years ago
18728a0b59
XL/erasure-read: refactor erasure read and add tests ( #2232 )
9 years ago
ddf3245677
xl/fs: offset and length cannot be negative. ( #2121 )
...
Fixes #2119
9 years ago
44ae7a037b
fix: allocate buffer to required size than readSizeV1 ( #2095 )
...
Refer #2077
9 years ago
d484157d67
XL/bitrot: Migrate to new blake2b-simd SIMD optimized implementation. ( #2031 )
...
Thanks for Frank Wessels <fwessels@xs4all.nl> for all the heavy lifting work.
Comparative benchmarks are as below.
```
benchmark old ns/op new ns/op delta
BenchmarkHash64-4 742 411 -44.61%
BenchmarkHash128-4 681 346 -49.19%
BenchmarkWrite1K-4 4239 1497 -64.69%
BenchmarkWrite8K-4 33633 11514 -65.77%
BenchmarkWrite32K-4 134091 45947 -65.73%
BenchmarkWrite128K-4 537976 183643 -65.86%
benchmark old MB/s new MB/s speedup
BenchmarkHash64-4 86.18 155.51 1.80x
BenchmarkHash128-4 187.96 369.10 1.96x
BenchmarkWrite1K-4 241.55 683.87 2.83x
BenchmarkWrite8K-4 3897.06 11383.41 2.92x
BenchmarkWrite32K-4 977.48 2852.63 2.92x
BenchmarkWrite128K-4 243.64 713.73 2.93x
```
Fixes #2030
9 years ago
ce7d5eddbc
Misspell warnings fix ( #2001 )
9 years ago
ed2fdd90b0
fs: Fix GetObject failure to read large blocks. ( #1982 )
...
Add relevant test cases as well for verifying this
part of the codebase.
Fixes #1979
9 years ago
e8990e42c2
XL: Make allocations simpler avoid redundant allocs. ( #1961 )
...
- Reduce 10MiB buffers for loopy calls to use 128KiB.
- start using 128KiB buffer where needed.
9 years ago
e10934a88e
bitrot: Start using blake2b algorithm and remove sha512 usage. ( #1957 )
...
Fixes #1952
9 years ago
9b82e64a11
XL/erasure-read: Avoid memory copy, write to writer directly all the dataBlocks.
9 years ago
d4bea5fbf8
XL/erasure-read: Add Comments and enable bitrot detection.
9 years ago
17efaaa902
XL/erasure-read: Support parallel reads from disks.
9 years ago
8559e89494
XL: fix getBlockInfo() to return correct end block ( #1941 )
...
If requested offset/length of an object is equal to
erasureInfo.BlockSize, getBlockInfo() returns one more block added to
actual end block. This patch fixes the issue.
This patch also adds unit test for get objects with big files.
9 years ago
1ea1dba528
erasure-readfile: write to given Writer than returning buffer. ( #1910 )
...
Fixes #1889
9 years ago
2eb6fa3fce
cleanup: remove unused waitCloser.
9 years ago
ae311aa53b
XL: Cleanup, comments and all the updated functions. ( #1830 )
9 years ago
c493ab5d0d
XL: Bring in sha512 checksum support. ( #1797 )
9 years ago
a4a0ea605b
XL: Fix GetObject erasure decode issues. ( #1793 )
9 years ago
feb337098d
XL: bring in new storage API. ( #1780 )
...
Fixes #1771
9 years ago
a97230dd56
XL/erasure: Reset dataBlocks to reduce the memory usage. ( #1749 )
...
Fixes #1748
9 years ago
b38b9fea79
XL/erasure: fix for skipping 0 padding. ( #1737 )
...
Fixes #1736
9 years ago
293d246f95
XL/FS: Rewrite in new format.
9 years ago
0b74f5624e
xl: Fix how we deal with read offsets at erasure layer. ( #1511 )
...
Requires skipping necessary parts of dataBlocks during
decoding phase and requires us to properly skip the
entries as needed.
Thanks to Karthic for reproducing this important issue.
Fixes #1503
9 years ago
4e34e03dd4
xl/fs: Split object layer into interface. ( #1415 )
9 years ago
a1a667ae5d
xl: Change fileMetadata to xlMetadata. ( #1404 )
...
Finalized backend format
```
{
"version": "1.0.0",
"stat": {
"size": 24256,
"modTime": "2016-04-28T00:11:37.843Z"
},
"erasure": {
"data": 5,
"parity": 5,
"blockSize": 4194304
],
"minio": {
"release": "RELEASE.2016-04-28T00-09-47Z"
}
}
```
9 years ago
8c85815106
xl: refactor functions to xl-v1-common.go xl-v1-utils.go. ( #1357 )
9 years ago
becc814531
Xl layer selfheal quorum2
...
* xl/selfheal: selfheal based on read quorum on GET
* xl: getReadableDisks() also returns whether self-heal is needed so that this info can be used by ReadFile/SelfHeal/StatFile.
* xl: trigger selfheal from StatFile.
9 years ago
9bd9441107
xl: Simplify reading metadata and add a new fileMetadata type. ( #1346 )
9 years ago
5c33b68318
xl: code refactor, cleanup ReadFile and CreateFile.
9 years ago
Harshavardhana