4e34e03dd4
xl/fs: Split object layer into interface. ( #1415 )
9 years ago
f3784d1087
xl: Handle read quorum for StatVol, ListVols
9 years ago
560c3bd153
Adding return statement after error response in the lastest commit to verify location constraint ( #1348 )
9 years ago
cb1116725b
api: verify Location constraint for make bucket. ( #1342 )
9 years ago
e0f8fed011
object: handle Error responses and handle errDiskFull. ( #1331 )
9 years ago
be002ac01e
fs/object: Fix issues from review comments.
9 years ago
30b0b4deba
storage/server/client: Enable storage server, enable client storage.
9 years ago
3c48537f20
refactor: refactor code to separate fs into object-layer and fs layer. ( #1305 )
9 years ago
188bb92d8a
bucket-policy parset tests, and bug fixes ( #1317 )
9 years ago
b182e94acc
signature: Handle presigned payload if set.
...
Validate payload with incoming content.
Fixes #1288
9 years ago
e8cd1aad8d
accessPolicy: prevent backdoor ListBucket via brute-force 404s, per docs + small fixes
...
* accessPolicy: copy object should require PutObject
* accessPolicy: cite mpu perms doc only for relevant operations
* accessPolicy: prevent backdoor ListBucket via brute-force 404s, per docs
9 years ago
8b4a5f07b4
accessPolicy: allow anonymous HEAD for Getable objects
...
* accessPolicy: allow anonymous HEAD for Getable objects
* accessPolicy: allow anonymous HEAD of Listable Buckets
9 years ago
3fcc60de91
Move the files and rename some functions.
...
- Rename dir.go as 'fs-multipart-dir.go'
- Move the push/pop to fs-multipart.go and rename them as save/lookup.
- Rename objectInfo instances in fs-multipart as multipartObjInfo.
9 years ago
c69fdf0cf2
listObjects: Cleanup and naming conventions.
...
- Marker should be escaped outside in handlers.
- Delimiter should be handled outside in handlers.
- Add missing comments and change the function names.
- Handle case of 'maxKeys' when its set to '0', its a valid
case and should be treated as such.
9 years ago
0479d4976b
objectAPI: Fix object API interface, remove unnecessary structs.
...
ObjectAPI changes.
```
ListObjects(bucket, prefix, marker, delimiter string, maxKeys int) (ListObjectsInfo, *probe.Error)
ListMultipartUploads(bucket, objectPrefix, keyMarker, uploadIDMarker, delimiter string, maxUploads int) (ListMultipartsInfo, *probe.Error)
ListObjectParts(bucket, object, uploadID string, partNumberMarker, maxParts int) (ListPartsInfo, *probe.Error)
CompleteMultipartUpload(bucket string, object string, uploadID string, parts []completePart) (ObjectInfo, *probe.Error)
```
9 years ago
a6a4e7e297
signature: No need to validate region for getBucketLocation and listBuckets.
...
This type of check is added for making sure that we can support
custom regions.
ListBuckets and GetBucketLocation are always "us-east-1" rest
should look for the configured region.
Fixes #1278
9 years ago
efc80343e3
fs: Break fs package to top-level and introduce ObjectAPI interface.
...
ObjectAPI interface brings in changes needed for XL ObjectAPI layer.
The new interface for any ObjectAPI layer is as below
```
// ObjectAPI interface.
type ObjectAPI interface {
// Bucket resource API.
DeleteBucket(bucket string) *probe.Error
ListBuckets() ([]BucketInfo, *probe.Error)
MakeBucket(bucket string) *probe.Error
GetBucketInfo(bucket string) (BucketInfo, *probe.Error)
// Bucket query API.
ListObjects(bucket, prefix, marker, delimiter string, maxKeys int) (ListObjectsResult, *probe.Error)
ListMultipartUploads(bucket string, resources BucketMultipartResourcesMetadata) (BucketMultipartResourcesMetadata, *probe.Error)
// Object resource API.
GetObject(bucket, object string, startOffset int64) (io.ReadCloser, *probe.Error)
GetObjectInfo(bucket, object string) (ObjectInfo, *probe.Error)
PutObject(bucket string, object string, size int64, data io.Reader, metadata map[string]string) (ObjectInfo, *probe.Error)
DeleteObject(bucket, object string) *probe.Error
// Object query API.
NewMultipartUpload(bucket, object string) (string, *probe.Error)
PutObjectPart(bucket, object, uploadID string, partID int, size int64, data io.Reader, md5Hex string) (string, *probe.Error)
ListObjectParts(bucket, object string, resources ObjectResourcesMetadata) (ObjectResourcesMetadata, *probe.Error)
CompleteMultipartUpload(bucket string, object string, uploadID string, parts []CompletePart) (ObjectInfo, *probe.Error)
AbortMultipartUpload(bucket, object, uploadID string) *probe.Error
}
```
9 years ago
02ad48466d
error: Signature errors should be returned with APIErrorCode.
...
The reasoning is that we can reply back with wide range of
S3 error responses, which would provide more richer context
to S3 client.
Fixes #1267
9 years ago
9dca46e156
signature: Use a layered approach for signature verification.
...
Signature calculation has now moved out from being a package to
top-level as a layered mechanism.
In case of payload calculation with body, go-routines are initiated
to simultaneously write and calculate shasum. Errors are sent
over the writer so that the lower layer removes the temporary files
properly.
9 years ago
aaf97ea02c
config/main: Re-write config files - add to new config v3
...
- New config format.
```
{
"version": "3",
"address": ":9000",
"backend": {
"type": "fs",
"disk": "/path"
},
"credential": {
"accessKey": "WLGDGYAQYIGI833EV05A",
"secretKey": "BYvgJM101sHngl2uzjXS/OBF/aMxAN06JrJ3qJlF"
},
"region": "us-east-1",
"logger": {
"file": {
"enable": false,
"fileName": "",
"level": "error"
},
"syslog": {
"enable": false,
"address": "",
"level": "debug"
},
"console": {
"enable": true,
"level": "fatal"
}
}
}
```
New command lines in lieu of supporting XL.
Minio initialize filesystem backend.
~~~
$ minio init fs <path>
~~~
Minio initialize XL backend.
~~~
$ minio init xl <url1>...<url16>
~~~
For 'fs' backend it starts the server.
~~~
$ minio server
~~~
For 'xl' backend it waits for servers to join.
~~~
$ minio server
... [PROGRESS BAR] of servers connecting
~~~
Now on other servers execute 'join' and they connect.
~~~
....
minio join <url1> -- from <url2> && minio server
minio join <url1> -- from <url3> && minio server
...
...
minio join <url1> -- from <url16> && minio server
~~~
9 years ago
76bda0d8f1
routers: Fix order of PostPolicyHandlers and headers.
9 years ago
88714e7c8e
bucketpolicy: Improve bucket policy validation, avoid nested rules.
...
Bucket policy validation is more stricter now, to avoid nested
rules. The reason to do this is keep the rules simpler and more
meaningful avoiding conflicts.
This patch implements stricter checks.
Example policy to be generally avoided.
```
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"s3:GetObject",
"s3:DeleteObject"
],
"Effect": "Allow",
"Principal": {
"AWS": [
"*"
]
},
"Resource": [
"arn:aws:s3:::jarjarbing/*"
]
},
{
"Action": [
"s3:GetObject",
"s3:DeleteObject"
],
"Effect": "Deny",
"Principal": {
"AWS": [
"*"
]
},
"Resource": [
"arn:aws:s3:::jarjarbing/restic/key/*"
]
}
]
}
```
9 years ago
52751d81cb
cleanup: Rename ObjectMetadata as ObjectInfo.
...
Fixes #1215
9 years ago
fdf3d64793
error: Add proper prefixes for s3Error codes.
...
This patch adds 'Err' prefix for all standard API
error codes and also adds a proper type for them.
9 years ago
d5057b3c51
accessPolicy: Implement Put, Get, Delete access policy.
...
This patch implements Get,Put,Delete bucket policies
Supporting - http://docs.aws.amazon.com/AmazonS3/latest/dev/access-policy-language-overview.html
Currently supports following actions.
"*": true,
"s3:*": true,
"s3:GetObject": true,
"s3:ListBucket": true,
"s3:PutObject": true,
"s3:CreateBucket": true,
"s3:GetBucketLocation": true,
"s3:DeleteBucket": true,
"s3:DeleteObject": true,
"s3:AbortMultipartUpload": true,
"s3:ListBucketMultipartUploads": true,
"s3:ListMultipartUploadParts": true,
following conditions for "StringEquals" and "StringNotEquals"
"s3:prefix", "s3:max-keys"
9 years ago
aed62788d9
api: Implement multiple objects Delete api - fixes #956
...
This API takes input XML input in following form.
```
<?xml version="1.0" encoding="UTF-8"?>
<Delete>
<Quiet>true</Quiet>
<Object>
<Key>Key</Key>
</Object>
<Object>
<Key>Key</Key>
</Object>
...
</Delete>
```
and responds the list of successful deletes, list of errors
for all the deleted objects.
```
<?xml version="1.0" encoding="UTF-8"?>
<DeleteResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/ ">
<Deleted>
<Key>sample1.txt</Key>
</Deleted>
<Error>
<Key>sample2.txt</Key>
<Code>AccessDenied</Code>
<Message>Access Denied</Message>
</Error>
</DeleteResult>
```
9 years ago
adf74ffdb0
api: DRY code and add new test
...
This commit makes code cleaner and reduces the repetitions in the code
base. Specifically, it reduces the clutter in setObjectHeaders. It also
merges encodeSuccessResponse and encodeErrorResponse together because
they served no purpose differently. Finally, it adds a simple test for
generateRequestID.
9 years ago
f111997184
multipart: remove proper MD5, rather create MD5 based on parts to be s3 compatible.
...
This increases the performance phenominally.
9 years ago
17d145df3a
signature: Fix signature handling of parallel requests.
...
Signature struct should be immutable, this fixes an issue
with AWS cli not being able to do multipart put operations.
9 years ago
653ceee9ee
signV4: Move pkg/signature to pkg/s3/signature4
...
Cleanup and move this to relevant path.
9 years ago
91a092792a
presigned: Fix a bug in presigned request verification.
...
Additionally add Docker proxy configuration.
9 years ago
dd9aaa855c
web/rpc: Merge ports with API server.
...
Fixes #1081 and #1130
9 years ago
5a9333a67b
signature: Rewrite signature handling and move it into a library.
9 years ago
62f6ffb6db
xl: Moved to minio/minio - fixes #1112
9 years ago
a4c005ce30
multipart: Code cleanup
...
- More locking cleanup. Fix naming convention.
- Simplify concatenation and blocking calls.
9 years ago
d79fcb1800
fix: handle Transfer-Encoding for make bucket
...
In case of make bucket, there is a chance of Transfer-Encoding is sent
where Content-Length is missing. This patch fixes the problem by
checking whether Transfer-Encoding: chunked is set along with
Content-Length.
9 years ago
7a3409c309
fs: Cleanup Golang errors to be called 'e' and probe to be called as 'err'
...
- Replace the ACL checks back, remove them when bucket
policy is implemented.
- Move FTW (File Tree Walk) into ioutils package.
9 years ago
df91661ec6
flags: Remove anonymous, ratelimit, json and web-address flags.
...
- Web address now uses the port + 1 from the API address port directly.
- Remove ratelimiting, ratelimiting will be achieved if necessary through
iptables.
- Remove json flag, not needed anymore.
- Remove anonymous flag, server will be no more anonymous for play.minio.io
we will use demo credentials.
9 years ago
0aedb67de0
contentType: Reply back proper contentTypes based on the file extension.
...
Currently the server would set 'application/octet-stream' for all
objects, set this value based on the file extension transparently.
This is useful in case of minio browser to facilitate displaying
proper icons for the different mime data types.
9 years ago
682020ef2f
listObjects: Channel based changes.
...
Supports:
- prefixes
- marker
9 years ago
9e18bfa60e
listObjects: Channel based ftw - initial implementation.
9 years ago
cb7b2762f9
serverConfig: Add a new region config entry.
...
To change default region from 'us-east-1' to 'custom'.
Add a region value in your 'config.json'.
"version": "2",
"credentials": {
"accessKeyId": "****************",
"secretAccessKey": "***************",
"region": "my-region"
},
9 years ago
0c6a6dc380
http: Enable Transfer-Encoding chunked transfer
...
Fixes #1020
9 years ago
2f67559684
handlers: read ContentLength value directly from http.Request.
...
Do not look for Content-Length in headers and try to convert them into
integer representations use ContentLength field from *http.Request*.
If Content-Length is understood to be as '-1' then treat it as an error
condition, since it could be a malformed body to crash the server.
Fixes #1011
9 years ago
0345c8fffb
bucket-location: Implement bucket location response.
9 years ago
e1a33deabf
acl: Handle readonly buckets properly
9 years ago
5b2fa33bdb
Implementing min-free-disk
9 years ago
704fa420a3
Reply back proper statuses for DeleteBucket/DeleteObject
9 years ago
0eb7f078f9
Implement Bucket ACL support
9 years ago
9a01026a78
If anonymous mode is set avoid verifying signature at lower level
9 years ago
Harshavardhana