This patch fixes majority of discrepant messages and responses
previously reported.
There are few discrepancies observed
- S3 is not honoring 'If-Modified-Since' header.
- We do not implement object policy, S3 returns a different response in this category.
- Adding new headers causes signature mismatch, but Minio server is fine for example
TestCopyObject() to be fixed by moving the signature logic out.
Relevant bug - https://github.com/minio/minio/issues/2097Fixes#1955
Change brings in a new signVerifyReader which provides a io.Reader
compatible reader, additionally implements Verify() function.
Verify() function validates the signature present in the incoming
request. This approach is choosen to avoid complexities involved
in using io.Pipe().
Thanks to Krishna for his inputs on this.
Fixes#2058Fixes#2054Fixes#2087
Previously checkDiskFree() checks for free available space. This
patch enables checkDiskFree() also checks for free inodes in linux and
free clusters in windows.
Fixes#2075
The object cache implementation is XL cache, which defaults
to 8GB worth of read cache. Currently GetObject() transparently
writes to this cache upon first client read and then subsequently
serves reads from the same cache.
Currently expiration is not implemented.
PR contains,
- New setup utilities for running object handler tests. Here is why they are essential,
- Unit tests have to be run in isolation without being have to be associated with other functionalities which are not under test.
- The integration tests follows the philosophy of running a Test Server and registers all handlers and fires HTTP requests over the socket to simulate the system functionality under usual work load scenarios and test for correctness. But this philosophy cannot be adopted for running unit tests for HTTP handlers.
- Running Unit tests for API handlers,
- Shouldn't run a test server. Should purely call the handlers `ServeHTTP` under isolation.
- Shouldn't register all handlers, should only register handlers under test and so that the system is close to be in an isolated setup.
- As an example PutBucketPolicy test is illustrated using the new setup. Exhaustive cases has to be added and has been listen in TODO for now.
This is needed so that we only write data which was requested
for, using a limit reader avoids spurious reads on the incoming
client data. Additionally using limit reader provides server
safety from rogue clients sending copious amounts of data (for
example a denial of service attack).
This patch also caters for size == -1 when content encoding from
a client is set as chunked, we happily read till io.EOF
* Sent XML header before the first of whitespace chars
XML parsing fails in aws cli due to unexpected whitespace character. To
fix this, we send the xml header before we send the first whitespace
character, if any.
* Fix race between sendWhiteSpaceChars and completeMultiUploadpart
Deletion of tmp files where xl metadata was saved before the commit
operation doesn't change the error returned to the caller. So, it is to
be ignored.
An in-place update to xl.json amidst concurrent PutObjectPart operations
lead to racy updates to xl.json making it un-parseable. To avoid this,
we create a unique tmp file where updates to xl.json are staged before
renaming it to the final location.
This change co-incides with another patch set which
reduces the writeQuorum requirement. With the
write quorum change it is now possible to support
6 disk configuration.
* Added unit-tests for treeWalkPool
* Added unit tests for tree-walk-fs
* Added period at the end of all comments.
* FS/XL: Unified tree walk tests for both backends
* Added disk failure related tests for treewalk
Replaced removeRandomDisks with removeDiskN. There is no need to
randomize disks that fail while the distribution of chunks in XL during
erasure coding data is random.
This function is added to extend the standard error responses.
Which is needed in some cases for example CompleteMultipartUpload
should respond with ErrPartTooSmall error when parts uploaded are
lesser than 5MB (i.e minimum allowed size per part).
Fixes#1536
Previously, checkETag didn't handle ETags with leading and trailing
double quotes. e.g "abcdef1234" == "\"abcdef1234\"" would return false.
Now, checkETag function canonicalizes the ETags passed as arguments by
removing one leading/trailing double quote.
Krishna Srinivas