9b82e64a11
XL/erasure-read: Avoid memory copy, write to writer directly all the dataBlocks.
9 years ago
ba5bb4a127
TestServer introduction and revamp of functional tests. ( #1940 )
...
Allows for easy creation of Test server with temp backend.
changes
9 years ago
409b4ddecb
api: MakeBucket should set proper bucket location. ( #1948 )
...
Fixes #1942
9 years ago
ad779a8ba4
XL: Enable tests for large GetObject. ( #1947 )
...
Ref #1946
9 years ago
4ee2136b28
Unit tests for PUT object when object already exists ( #1904 )
...
* fs/xl tests for multiple put object requests
* xl fix for put object on directory
* Unit tests fix windows test issue.
9 years ago
c41bf26712
Unit tests: add unit tests for listv1/v2 for list bucket handler. ( #1933 )
...
fixes #1818
9 years ago
8c0942bf0d
XL: Remove usage of reduceErr and make it isQuorum verification. ( #1909 )
...
Fixes #1908
9 years ago
365f80efa3
XL/DeleteObject: delete call on a prefix should not delete the entire tree structure. ( #1916 )
...
fixes #1915
9 years ago
ed4fe689b4
posix: Support UNC paths on windows. ( #1887 )
...
This allows us to now use 32K paths names on windows.
Fixes #1620
9 years ago
967c2b2940
Handled possible short writes to httpResponseWriter ( #1804 )
...
* XL: Handled possible short writes to httpResponseWriter
* Added tests for Range Header combinations
9 years ago
5a4b074ca0
XL: PutObject incorrectly returned after deleting multipart object. ( #1715 )
...
Fixes #1714
9 years ago
9fdb69563d
handler: CopyObject should save metadata. ( #1698 )
...
- Content-Type
- Content-Encoding
- ETag
Fixes #1682
9 years ago
00d0558131
XL: Enable tests for content-type for Head and Get
...
Fixes #1674
9 years ago
498ce1e9bb
handler: Add a waitgroup to avoid expect100Continue crash. ( #1623 )
...
This waitgroup allows for safe blocking operation where we can cleanly
control the flow of the writes and the underlying pipe altogether.
Fixes #1553
9 years ago
adbcafefad
xl/CreateFile: handle errFileNameTooLong error properly ( #1523 )
...
When errFileNameTooLong error is returned from posix, xl.CreateFile()
treats the error specially by returning the same error immediately.
Fixes #1501
9 years ago
04a5b25929
Multipart: Minimum part size limit does not apply to the last part during CompleteMultipartUpload. ( #1518 ) ( #1538 )
9 years ago
75320f70d0
multipart: reject part upload if size is less than 5MB. ( #1518 )
9 years ago
0625c050e6
xl/tests: Enable server handler tests over XL. ( #1512 )
...
Fixes #1513
9 years ago
82fbe908a3
object: DeleteBucket should return proper error for BucketNotEmpty. ( #1489 )
...
Fixes #1488
9 years ago
bba5468368
minio: Replace 'bucket already exists' error by 'bucket already owned by you'. ( #1465 )
...
S3 API returns BucketAlreadyExists error when some another user has such bucket.
If user that creates the bucket already has it, s3 returns BucketAlreadyOwnedByYou.
As minio has only one user, it should behave accordingly.
Otherwise it causes failures in the applications that ignore creation of already existing bucket in the account, but fail when bucket name is used by someone else.
9 years ago
ac7a7cec20
bucket-policy: Delete policy should remove policy properly. ( #1456 )
9 years ago
984903cce1
server: Add global namespace lock. ( #1398 )
...
Fixes #1393
9 years ago
8457af5708
fs: Add proper volume and path validation.
9 years ago
30b0b4deba
storage/server/client: Enable storage server, enable client storage.
9 years ago
01a439f95b
refactor: add multipart code to the object layer.
9 years ago
3c48537f20
refactor: refactor code to separate fs into object-layer and fs layer. ( #1305 )
9 years ago
33cd910d3a
backend/fs: More cleanup and start using checkBuckeArg. ( #1306 )
...
backend/fs: More cleanup and start using checkBucketArg.
9 years ago
0479d4976b
objectAPI: Fix object API interface, remove unnecessary structs.
...
ObjectAPI changes.
```
ListObjects(bucket, prefix, marker, delimiter string, maxKeys int) (ListObjectsInfo, *probe.Error)
ListMultipartUploads(bucket, objectPrefix, keyMarker, uploadIDMarker, delimiter string, maxUploads int) (ListMultipartsInfo, *probe.Error)
ListObjectParts(bucket, object, uploadID string, partNumberMarker, maxParts int) (ListPartsInfo, *probe.Error)
CompleteMultipartUpload(bucket string, object string, uploadID string, parts []completePart) (ObjectInfo, *probe.Error)
```
9 years ago
484ba91b08
config: Migrate to the new version. Remove backend details.
...
Migrate to new config format v4.
```
{
"version": "4",
"credential": {
"accessKey": "WLGDGYAQYIGI833EV05A",
"secretKey": "BYvgJM101sHngl2uzjXS/OBF/aMxAN06JrJ3qJlF"
},
"region": "us-east-1",
"logger": {
"console": {
"enable": true,
"level": "fatal"
},
"file": {
"enable": false,
"fileName": "",
"level": "error"
},
"syslog": {
"enable": false,
"address": "",
"level": "debug"
}
}
}
```
This patch also updates [minio cli spec](./minio.md)
9 years ago
efc80343e3
fs: Break fs package to top-level and introduce ObjectAPI interface.
...
ObjectAPI interface brings in changes needed for XL ObjectAPI layer.
The new interface for any ObjectAPI layer is as below
```
// ObjectAPI interface.
type ObjectAPI interface {
// Bucket resource API.
DeleteBucket(bucket string) *probe.Error
ListBuckets() ([]BucketInfo, *probe.Error)
MakeBucket(bucket string) *probe.Error
GetBucketInfo(bucket string) (BucketInfo, *probe.Error)
// Bucket query API.
ListObjects(bucket, prefix, marker, delimiter string, maxKeys int) (ListObjectsResult, *probe.Error)
ListMultipartUploads(bucket string, resources BucketMultipartResourcesMetadata) (BucketMultipartResourcesMetadata, *probe.Error)
// Object resource API.
GetObject(bucket, object string, startOffset int64) (io.ReadCloser, *probe.Error)
GetObjectInfo(bucket, object string) (ObjectInfo, *probe.Error)
PutObject(bucket string, object string, size int64, data io.Reader, metadata map[string]string) (ObjectInfo, *probe.Error)
DeleteObject(bucket, object string) *probe.Error
// Object query API.
NewMultipartUpload(bucket, object string) (string, *probe.Error)
PutObjectPart(bucket, object, uploadID string, partID int, size int64, data io.Reader, md5Hex string) (string, *probe.Error)
ListObjectParts(bucket, object string, resources ObjectResourcesMetadata) (ObjectResourcesMetadata, *probe.Error)
CompleteMultipartUpload(bucket string, object string, uploadID string, parts []CompletePart) (ObjectInfo, *probe.Error)
AbortMultipartUpload(bucket, object, uploadID string) *probe.Error
}
```
9 years ago
30fc970eab
Changes to TestPutBucket to catch the race
9 years ago
9dca46e156
signature: Use a layered approach for signature verification.
...
Signature calculation has now moved out from being a package to
top-level as a layered mechanism.
In case of payload calculation with body, go-routines are initiated
to simultaneously write and calculate shasum. Errors are sent
over the writer so that the lower layer removes the temporary files
properly.
9 years ago
aaf97ea02c
config/main: Re-write config files - add to new config v3
...
- New config format.
```
{
"version": "3",
"address": ":9000",
"backend": {
"type": "fs",
"disk": "/path"
},
"credential": {
"accessKey": "WLGDGYAQYIGI833EV05A",
"secretKey": "BYvgJM101sHngl2uzjXS/OBF/aMxAN06JrJ3qJlF"
},
"region": "us-east-1",
"logger": {
"file": {
"enable": false,
"fileName": "",
"level": "error"
},
"syslog": {
"enable": false,
"address": "",
"level": "debug"
},
"console": {
"enable": true,
"level": "fatal"
}
}
}
```
New command lines in lieu of supporting XL.
Minio initialize filesystem backend.
~~~
$ minio init fs <path>
~~~
Minio initialize XL backend.
~~~
$ minio init xl <url1>...<url16>
~~~
For 'fs' backend it starts the server.
~~~
$ minio server
~~~
For 'xl' backend it waits for servers to join.
~~~
$ minio server
... [PROGRESS BAR] of servers connecting
~~~
Now on other servers execute 'join' and they connect.
~~~
....
minio join <url1> -- from <url2> && minio server
minio join <url1> -- from <url3> && minio server
...
...
minio join <url1> -- from <url16> && minio server
~~~
9 years ago
902aa05021
main: Remove all the dead/unused code.
...
This patch removes some dead and unused code.
9 years ago
fc72a0362f
api: ListMultipartUploads and ListParts responded more entries.
...
Issue is empty entries were added since allocating an array was
followed by an append. Keep the index and copy the right entries
precisely.
Fixes an issue reported at - https://github.com/minio/mc/issues/1642
9 years ago
88714e7c8e
bucketpolicy: Improve bucket policy validation, avoid nested rules.
...
Bucket policy validation is more stricter now, to avoid nested
rules. The reason to do this is keep the rules simpler and more
meaningful avoiding conflicts.
This patch implements stricter checks.
Example policy to be generally avoided.
```
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"s3:GetObject",
"s3:DeleteObject"
],
"Effect": "Allow",
"Principal": {
"AWS": [
"*"
]
},
"Resource": [
"arn:aws:s3:::jarjarbing/*"
]
},
{
"Action": [
"s3:GetObject",
"s3:DeleteObject"
],
"Effect": "Deny",
"Principal": {
"AWS": [
"*"
]
},
"Resource": [
"arn:aws:s3:::jarjarbing/restic/key/*"
]
}
]
}
```
9 years ago
d5057b3c51
accessPolicy: Implement Put, Get, Delete access policy.
...
This patch implements Get,Put,Delete bucket policies
Supporting - http://docs.aws.amazon.com/AmazonS3/latest/dev/access-policy-language-overview.html
Currently supports following actions.
"*": true,
"s3:*": true,
"s3:GetObject": true,
"s3:ListBucket": true,
"s3:PutObject": true,
"s3:CreateBucket": true,
"s3:GetBucketLocation": true,
"s3:DeleteBucket": true,
"s3:DeleteObject": true,
"s3:AbortMultipartUpload": true,
"s3:ListBucketMultipartUploads": true,
"s3:ListMultipartUploadParts": true,
following conditions for "StringEquals" and "StringNotEquals"
"s3:prefix", "s3:max-keys"
9 years ago
aed62788d9
api: Implement multiple objects Delete api - fixes #956
...
This API takes input XML input in following form.
```
<?xml version="1.0" encoding="UTF-8"?>
<Delete>
<Quiet>true</Quiet>
<Object>
<Key>Key</Key>
</Object>
<Object>
<Key>Key</Key>
</Object>
...
</Delete>
```
and responds the list of successful deletes, list of errors
for all the deleted objects.
```
<?xml version="1.0" encoding="UTF-8"?>
<DeleteResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/ ">
<Deleted>
<Key>sample1.txt</Key>
</Deleted>
<Error>
<Key>sample2.txt</Key>
<Code>AccessDenied</Code>
<Message>Access Denied</Message>
</Error>
</DeleteResult>
```
9 years ago
ee1b86e517
api: Implement support for additional request headers.
...
Now GetObject and HeadObject both support
- If-Modified-Since, If-Unmodified-Since
- If-Match, If-None-Match
request headers.
These headers are used to further handle the responses for GetObject
and HeadObject API.
Fixes #1098
9 years ago
3ff8a1b719
api: Implement CopyObject s3 API, doing server side copy.
...
Fixes #1172
9 years ago
ae6e774377
api: Return NotImplemented for MultiDelete and CopyObject APIs
9 years ago
91a092792a
presigned: Fix a bug in presigned request verification.
...
Additionally add Docker proxy configuration.
9 years ago
dd9aaa855c
web/rpc: Merge ports with API server.
...
Fixes #1081 and #1130
9 years ago
5a9333a67b
signature: Rewrite signature handling and move it into a library.
9 years ago
8df201ef30
multipart: Multipart session map now is based on uploadID.
...
- Fixes initiating parallel uploads, and configs being quickly
re-written by another incoming request.
- Parallel uploads work smoothly now and return expected behavior.
9 years ago
7a3409c309
fs: Cleanup Golang errors to be called 'e' and probe to be called as 'err'
...
- Replace the ACL checks back, remove them when bucket
policy is implemented.
- Move FTW (File Tree Walk) into ioutils package.
9 years ago
df91661ec6
flags: Remove anonymous, ratelimit, json and web-address flags.
...
- Web address now uses the port + 1 from the API address port directly.
- Remove ratelimiting, ratelimiting will be achieved if necessary through
iptables.
- Remove json flag, not needed anymore.
- Remove anonymous flag, server will be no more anonymous for play.minio.io
we will use demo credentials.
9 years ago
2e311168ee
multipart: NewMultipartUpload shouldn't return empty UploadID
...
Existing code
```
{
if os.IsNotExist(e) {
e = os.MkdirAll(objectDir, 0700)
if e != nil {
return "", probe.NewError(e)
}
}
return "", probe.NewError(e) ---> Error was here.
}
```
For a successful 'MkdirAll' it would still return an empty uploadID,
but the 'error' would be nil. This would succeed the request but
client would fail.
Fix is to re-arrange the logic. Thanks to Alexander Neumann @fd0 , for
reporting this problem.
9 years ago
682020ef2f
listObjects: Channel based changes.
...
Supports:
- prefixes
- marker
9 years ago
836f5204af
minio: Add config-folder option.
...
Fixes #997
9 years ago
Harshavardhana