8c767218a4
URL Encode X-Amz-Copy-Source as per the spec ( #2114 )
...
The documents for COPY state that the X-Amz-Copy-Source must be URL encoded.
http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectCOPY.html
9 years ago
a51bb1d728
http: handle request range like Amazon S3. ( #2112 )
...
Fixes #2098
9 years ago
8ddf52021a
server: Bring in s3 compatibility fixes. ( #2099 )
...
This patch fixes majority of discrepant messages and responses
previously reported.
There are few discrepancies observed
- S3 is not honoring 'If-Modified-Since' header.
- We do not implement object policy, S3 returns a different response in this category.
- Adding new headers causes signature mismatch, but Minio server is fine for example
TestCopyObject() to be fixed by moving the signature logic out.
Relevant bug - https://github.com/minio/minio/issues/2097
Fixes #1955
9 years ago
8a028a9efb
handler/PUT: Handle signature verification through a custom reader. ( #2066 )
...
Change brings in a new signVerifyReader which provides a io.Reader
compatible reader, additionally implements Verify() function.
Verify() function validates the signature present in the incoming
request. This approach is choosen to avoid complexities involved
in using io.Pipe().
Thanks to Krishna for his inputs on this.
Fixes #2058
Fixes #2054
Fixes #2087
9 years ago
1ec86dac2c
server-tests: unify XL and FS tests into common code. server_test.go contains common test code. server_xl_test.go contains XL tests specific to XL. ( #2089 )
9 years ago
d64c3fd464
posix: Return errDiskNotWritable during disk initialization. ( #2048 )
...
It can happen that minio server might not have
writable permissions on the export paths command line.
Fixes #2035
9 years ago
8e8f6f90a4
adding detailed comments to server_test ( #2032 )
9 years ago
ed2fdd90b0
fs: Fix GetObject failure to read large blocks. ( #1982 )
...
Add relevant test cases as well for verifying this
part of the codebase.
Fixes #1979
9 years ago
ba5bb4a127
TestServer introduction and revamp of functional tests. ( #1940 )
...
Allows for easy creation of Test server with temp backend.
changes
9 years ago
ad779a8ba4
XL: Enable tests for large GetObject. ( #1947 )
...
Ref #1946
9 years ago
8559e89494
XL: fix getBlockInfo() to return correct end block ( #1941 )
...
If requested offset/length of an object is equal to
erasureInfo.BlockSize, getBlockInfo() returns one more block added to
actual end block. This patch fixes the issue.
This patch also adds unit test for get objects with big files.
9 years ago
4ee2136b28
Unit tests for PUT object when object already exists ( #1904 )
...
* fs/xl tests for multiple put object requests
* xl fix for put object on directory
* Unit tests fix windows test issue.
9 years ago
c41bf26712
Unit tests: add unit tests for listv1/v2 for list bucket handler. ( #1933 )
...
fixes #1818
9 years ago
365f80efa3
XL/DeleteObject: delete call on a prefix should not delete the entire tree structure. ( #1916 )
...
fixes #1915
9 years ago
f51d34cedd
Do not guess content-type for objects with no extension ( #1918 )
9 years ago
ed4fe689b4
posix: Support UNC paths on windows. ( #1887 )
...
This allows us to now use 32K paths names on windows.
Fixes #1620
9 years ago
feb337098d
XL: bring in new storage API. ( #1780 )
...
Fixes #1771
9 years ago
498ce1e9bb
handler: Add a waitgroup to avoid expect100Continue crash. ( #1623 )
...
This waitgroup allows for safe blocking operation where we can cleanly
control the flow of the writes and the underlying pipe altogether.
Fixes #1553
9 years ago
5b29cefd40
api: DeleteObject should always return 204. ( #1645 )
...
Fixes #1643
9 years ago
d4745c7d6a
object: PutObjectHandler should set the md5Sum properly. ( #1604 )
...
Additionally add a test case as well for validating for us
to reply BadDigest properly.
Fixes #1603
9 years ago
04a5b25929
Multipart: Minimum part size limit does not apply to the last part during CompleteMultipartUpload. ( #1518 ) ( #1538 )
9 years ago
0625c050e6
xl/tests: Enable server handler tests over XL. ( #1512 )
...
Fixes #1513
9 years ago
82fbe908a3
object: DeleteBucket should return proper error for BucketNotEmpty. ( #1489 )
...
Fixes #1488
9 years ago
bba5468368
minio: Replace 'bucket already exists' error by 'bucket already owned by you'. ( #1465 )
...
S3 API returns BucketAlreadyExists error when some another user has such bucket.
If user that creates the bucket already has it, s3 returns BucketAlreadyOwnedByYou.
As minio has only one user, it should behave accordingly.
Otherwise it causes failures in the applications that ignore creation of already existing bucket in the account, but fail when bucket name is used by someone else.
9 years ago
ac7a7cec20
bucket-policy: Delete policy should remove policy properly. ( #1456 )
9 years ago
984903cce1
server: Add global namespace lock. ( #1398 )
...
Fixes #1393
9 years ago
8457af5708
fs: Add proper volume and path validation.
9 years ago
30b0b4deba
storage/server/client: Enable storage server, enable client storage.
9 years ago
01a439f95b
refactor: add multipart code to the object layer.
9 years ago
3c48537f20
refactor: refactor code to separate fs into object-layer and fs layer. ( #1305 )
9 years ago
33cd910d3a
backend/fs: More cleanup and start using checkBuckeArg. ( #1306 )
...
backend/fs: More cleanup and start using checkBucketArg.
9 years ago
0479d4976b
objectAPI: Fix object API interface, remove unnecessary structs.
...
ObjectAPI changes.
```
ListObjects(bucket, prefix, marker, delimiter string, maxKeys int) (ListObjectsInfo, *probe.Error)
ListMultipartUploads(bucket, objectPrefix, keyMarker, uploadIDMarker, delimiter string, maxUploads int) (ListMultipartsInfo, *probe.Error)
ListObjectParts(bucket, object, uploadID string, partNumberMarker, maxParts int) (ListPartsInfo, *probe.Error)
CompleteMultipartUpload(bucket string, object string, uploadID string, parts []completePart) (ObjectInfo, *probe.Error)
```
9 years ago
484ba91b08
config: Migrate to the new version. Remove backend details.
...
Migrate to new config format v4.
```
{
"version": "4",
"credential": {
"accessKey": "WLGDGYAQYIGI833EV05A",
"secretKey": "BYvgJM101sHngl2uzjXS/OBF/aMxAN06JrJ3qJlF"
},
"region": "us-east-1",
"logger": {
"console": {
"enable": true,
"level": "fatal"
},
"file": {
"enable": false,
"fileName": "",
"level": "error"
},
"syslog": {
"enable": false,
"address": "",
"level": "debug"
}
}
}
```
This patch also updates [minio cli spec](./minio.md)
9 years ago
efc80343e3
fs: Break fs package to top-level and introduce ObjectAPI interface.
...
ObjectAPI interface brings in changes needed for XL ObjectAPI layer.
The new interface for any ObjectAPI layer is as below
```
// ObjectAPI interface.
type ObjectAPI interface {
// Bucket resource API.
DeleteBucket(bucket string) *probe.Error
ListBuckets() ([]BucketInfo, *probe.Error)
MakeBucket(bucket string) *probe.Error
GetBucketInfo(bucket string) (BucketInfo, *probe.Error)
// Bucket query API.
ListObjects(bucket, prefix, marker, delimiter string, maxKeys int) (ListObjectsResult, *probe.Error)
ListMultipartUploads(bucket string, resources BucketMultipartResourcesMetadata) (BucketMultipartResourcesMetadata, *probe.Error)
// Object resource API.
GetObject(bucket, object string, startOffset int64) (io.ReadCloser, *probe.Error)
GetObjectInfo(bucket, object string) (ObjectInfo, *probe.Error)
PutObject(bucket string, object string, size int64, data io.Reader, metadata map[string]string) (ObjectInfo, *probe.Error)
DeleteObject(bucket, object string) *probe.Error
// Object query API.
NewMultipartUpload(bucket, object string) (string, *probe.Error)
PutObjectPart(bucket, object, uploadID string, partID int, size int64, data io.Reader, md5Hex string) (string, *probe.Error)
ListObjectParts(bucket, object string, resources ObjectResourcesMetadata) (ObjectResourcesMetadata, *probe.Error)
CompleteMultipartUpload(bucket string, object string, uploadID string, parts []CompletePart) (ObjectInfo, *probe.Error)
AbortMultipartUpload(bucket, object, uploadID string) *probe.Error
}
```
9 years ago
30fc970eab
Changes to TestPutBucket to catch the race
9 years ago
9dca46e156
signature: Use a layered approach for signature verification.
...
Signature calculation has now moved out from being a package to
top-level as a layered mechanism.
In case of payload calculation with body, go-routines are initiated
to simultaneously write and calculate shasum. Errors are sent
over the writer so that the lower layer removes the temporary files
properly.
9 years ago
aaf97ea02c
config/main: Re-write config files - add to new config v3
...
- New config format.
```
{
"version": "3",
"address": ":9000",
"backend": {
"type": "fs",
"disk": "/path"
},
"credential": {
"accessKey": "WLGDGYAQYIGI833EV05A",
"secretKey": "BYvgJM101sHngl2uzjXS/OBF/aMxAN06JrJ3qJlF"
},
"region": "us-east-1",
"logger": {
"file": {
"enable": false,
"fileName": "",
"level": "error"
},
"syslog": {
"enable": false,
"address": "",
"level": "debug"
},
"console": {
"enable": true,
"level": "fatal"
}
}
}
```
New command lines in lieu of supporting XL.
Minio initialize filesystem backend.
~~~
$ minio init fs <path>
~~~
Minio initialize XL backend.
~~~
$ minio init xl <url1>...<url16>
~~~
For 'fs' backend it starts the server.
~~~
$ minio server
~~~
For 'xl' backend it waits for servers to join.
~~~
$ minio server
... [PROGRESS BAR] of servers connecting
~~~
Now on other servers execute 'join' and they connect.
~~~
....
minio join <url1> -- from <url2> && minio server
minio join <url1> -- from <url3> && minio server
...
...
minio join <url1> -- from <url16> && minio server
~~~
9 years ago
902aa05021
main: Remove all the dead/unused code.
...
This patch removes some dead and unused code.
9 years ago
fc72a0362f
api: ListMultipartUploads and ListParts responded more entries.
...
Issue is empty entries were added since allocating an array was
followed by an append. Keep the index and copy the right entries
precisely.
Fixes an issue reported at - https://github.com/minio/mc/issues/1642
9 years ago
88714e7c8e
bucketpolicy: Improve bucket policy validation, avoid nested rules.
...
Bucket policy validation is more stricter now, to avoid nested
rules. The reason to do this is keep the rules simpler and more
meaningful avoiding conflicts.
This patch implements stricter checks.
Example policy to be generally avoided.
```
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"s3:GetObject",
"s3:DeleteObject"
],
"Effect": "Allow",
"Principal": {
"AWS": [
"*"
]
},
"Resource": [
"arn:aws:s3:::jarjarbing/*"
]
},
{
"Action": [
"s3:GetObject",
"s3:DeleteObject"
],
"Effect": "Deny",
"Principal": {
"AWS": [
"*"
]
},
"Resource": [
"arn:aws:s3:::jarjarbing/restic/key/*"
]
}
]
}
```
9 years ago
d5057b3c51
accessPolicy: Implement Put, Get, Delete access policy.
...
This patch implements Get,Put,Delete bucket policies
Supporting - http://docs.aws.amazon.com/AmazonS3/latest/dev/access-policy-language-overview.html
Currently supports following actions.
"*": true,
"s3:*": true,
"s3:GetObject": true,
"s3:ListBucket": true,
"s3:PutObject": true,
"s3:CreateBucket": true,
"s3:GetBucketLocation": true,
"s3:DeleteBucket": true,
"s3:DeleteObject": true,
"s3:AbortMultipartUpload": true,
"s3:ListBucketMultipartUploads": true,
"s3:ListMultipartUploadParts": true,
following conditions for "StringEquals" and "StringNotEquals"
"s3:prefix", "s3:max-keys"
9 years ago
aed62788d9
api: Implement multiple objects Delete api - fixes #956
...
This API takes input XML input in following form.
```
<?xml version="1.0" encoding="UTF-8"?>
<Delete>
<Quiet>true</Quiet>
<Object>
<Key>Key</Key>
</Object>
<Object>
<Key>Key</Key>
</Object>
...
</Delete>
```
and responds the list of successful deletes, list of errors
for all the deleted objects.
```
<?xml version="1.0" encoding="UTF-8"?>
<DeleteResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/ ">
<Deleted>
<Key>sample1.txt</Key>
</Deleted>
<Error>
<Key>sample2.txt</Key>
<Code>AccessDenied</Code>
<Message>Access Denied</Message>
</Error>
</DeleteResult>
```
9 years ago
ee1b86e517
api: Implement support for additional request headers.
...
Now GetObject and HeadObject both support
- If-Modified-Since, If-Unmodified-Since
- If-Match, If-None-Match
request headers.
These headers are used to further handle the responses for GetObject
and HeadObject API.
Fixes #1098
9 years ago
3ff8a1b719
api: Implement CopyObject s3 API, doing server side copy.
...
Fixes #1172
9 years ago
ae6e774377
api: Return NotImplemented for MultiDelete and CopyObject APIs
9 years ago
91a092792a
presigned: Fix a bug in presigned request verification.
...
Additionally add Docker proxy configuration.
9 years ago
dd9aaa855c
web/rpc: Merge ports with API server.
...
Fixes #1081 and #1130
9 years ago
5a9333a67b
signature: Rewrite signature handling and move it into a library.
9 years ago
8df201ef30
multipart: Multipart session map now is based on uploadID.
...
- Fixes initiating parallel uploads, and configs being quickly
re-written by another incoming request.
- Parallel uploads work smoothly now and return expected behavior.
9 years ago
7a3409c309
fs: Cleanup Golang errors to be called 'e' and probe to be called as 'err'
...
- Replace the ACL checks back, remove them when bucket
policy is implemented.
- Move FTW (File Tree Walk) into ioutils package.
9 years ago
Nick Craig-Wood