79ba4d3f33
refactor ObjectLayer PutObject and PutObjectPart ( #4925 )
...
This change refactor the ObjectLayer PutObject and PutObjectPart
functions. Instead of passing an io.Reader and a size to PUT operations
ObejectLayer expects an HashReader.
A HashReader verifies the MD5 sum (and SHA256 sum if required) of the object.
This change updates all all PutObject(Part) calls and removes unnecessary code
in all ObjectLayer implementations.
Fixes #4923
7 years ago
18c4e5d357
Enable browser support for gateway ( #4425 )
8 years ago
155a90403a
fs/erasure: Rename meta 'md5Sum' as 'etag'. ( #4319 )
...
This PR also does backend format change to 1.0.1
from 1.0.0. Backward compatible changes are still
kept to read the 'md5Sum' key. But all new objects
will be stored with the same details under 'etag'.
Fixes #4312
8 years ago
fa3f6d75b6
fs: Verify if parent is an object before i/o. ( #4304 )
...
PutObject() needs to verify and fail.
Fixes #4301
8 years ago
a5e60706a2
xl,fs: Return 404 if object ends with a separator ( #3897 )
...
HEAD Object for FS and XL was returning invalid object name when
an object name has a trailing slash separator, this PR changes the
behavior and will always return 404 object not found, this guarantees
a better compatibility with S3 spec.
8 years ago
47ac410ab0
Code cleanup - simplify server side code. ( #3870 )
...
Fix all the issues reported by `gosimple` tool.
8 years ago
77a192a7b5
Implement CopyObjectPart API ( #3663 )
...
This API is implemented to allow copying data from an
existing source object to an ongoing multipart operation
http://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadUploadPartCopy.html
Fixes #3662
8 years ago
98a6a2bcab
obj: Return objectInfo for CompleteMultipartUpload(). ( #3587 )
...
This patch avoids doing GetObjectInfo() in similar way
how we did for PutOject().
8 years ago
1c699d8d3f
fs: Re-implement object layer to remember the fd ( #3509 )
...
This patch re-writes FS backend to support shared backend sharing locks for safe concurrent access across multiple servers.
8 years ago
825000bc34
Use humanize constants for KiB, MiB and GiB units. ( #3322 )
8 years ago
bef0a50bc1
Cleanup and fixes ( #3273 )
...
* newRequestID() (previously generateUploadID()) returns string than byte array.
* Remove unclear comments and added appropriate comments.
* SHA-256, MD5 Hash functions return Hex/Base64 encoded string than byte array.
* Remove duplicate MD5 hasher functions.
* Rename listObjectsValidateArgs() into validateListObjectsArgs()
* Remove repeated auth check code in all bucket request handlers.
* Remove abbreviated names in bucket-metadata
* Avoid nested if in bucketPolicyMatchStatement()
* Use ioutil.ReadFile() instead of os.Open() and ioutil.ReadAll()
* Set crossDomainXML as constant.
8 years ago
c91d3791f9
heal: Add healing support for bucket, bucket metadata files. ( #3252 )
...
This patch implements healing in general but it is only used
as part of quickHeal().
Fixes #3237
8 years ago
61a18ed48f
sha256: Verify sha256 along with md5sum, signature is verified on the request early. ( #2813 )
8 years ago
b4e4846e9f
PutObject: object layer now returns ObjectInfo instead of md5sum to avoid extra GetObjectInfo call. ( #2599 )
...
From the S3 layer after PutObject we were calling GetObjectInfo for bucket notification. This can
be avoided if PutObjectInfo returns ObjectInfo.
fixes #2567
8 years ago
1ce339abeb
Fixing ineffssaign errors ( #2608 )
8 years ago
9358ee011b
logging: Print stack trace in case of errors.
...
fixes #1827
8 years ago
bccf549463
server: Move all the top level files into cmd folder. ( #2490 )
...
This change brings a change which was done for the 'mc'
package to allow for clean repo and have a cleaner
github drop in experience.
8 years ago
2c837128ef
Object layer tests revamp for individual execution ( #2134 )
8 years ago
6dcfa7b046
Fix for tests leaving out temp directories ( #2025 )
9 years ago
8c0942bf0d
XL: Remove usage of reduceErr and make it isQuorum verification. ( #1909 )
...
Fixes #1908
9 years ago
f51d34cedd
Do not guess content-type for objects with no extension ( #1918 )
9 years ago
1b9db9ee6c
FS/PutObject: Read() data should be handled even in case of EOF. ( #1864 )
...
Fixes #1710
9 years ago
feb337098d
XL: bring in new storage API. ( #1780 )
...
Fixes #1771
9 years ago
293d246f95
XL/FS: Rewrite in new format.
9 years ago
7d6ed50fc2
objects: Save all the incoming metadata properly. ( #1688 )
...
For both multipart and single put operation
9 years ago
d267696110
Validation: Reject object names with trailing "/". ( #1619 )
...
Fixes #1616
9 years ago
75320f70d0
multipart: reject part upload if size is less than 5MB. ( #1518 )
9 years ago
ac2933c799
windows: Enable erasure test for windows. ( #1432 )
...
Fixes #1363
9 years ago
4e34e03dd4
xl/fs: Split object layer into interface. ( #1415 )
9 years ago
1813e9c070
Cleanup - Comments and readability fixes ( #1386 )
9 years ago
4333e529e6
xl/ListFiles: return as many objects as requested. ( #1383 )
...
* xl/ListFiles: return as many objects as requested and take care of eof (#1361 )
* xl/ListFiles: fix review comments.
* xl/ListFiles: Add windows filepath translation.
* xl/ListFiles: Use slashSeparator instead of "/". Remove filepath.FromSlash() as golang-windows takes care of it automatically.
9 years ago
57f35c2bcc
xl: Introduce new blocking writer to make CreateFile atomic. ( #1362 )
...
Creates a new write closer that must be released
by the read consumer. This is necessary so that
while commiting the underlying writers in erasure
coding we need to make sure we reply success only if
we have committed to disk.
This in turn also fixes plethora of bugs related to
subsequent PutObject() races with namespace locking.
This patch also enables most of the tests, other than
ListObjects paging which has some issues still.
Fixes #1358 , #1360
9 years ago
cab6805f09
xl: Enable a subset of tests for XL branch. ( #1359 )
9 years ago
8c85815106
xl: refactor functions to xl-v1-common.go xl-v1-utils.go. ( #1357 )
9 years ago
6e372f83b4
Tests: object api multipart tests and bug fixes.
9 years ago
d63d17012d
tests: Add API suite tests back for object api. ( #1352 )
9 years ago
be002ac01e
fs/object: Fix issues from review comments.
9 years ago
30b0b4deba
storage/server/client: Enable storage server, enable client storage.
9 years ago
01a439f95b
refactor: add multipart code to the object layer.
9 years ago
3c48537f20
refactor: refactor code to separate fs into object-layer and fs layer. ( #1305 )
9 years ago
33cd910d3a
backend/fs: More cleanup and start using checkBuckeArg. ( #1306 )
...
backend/fs: More cleanup and start using checkBucketArg.
9 years ago
9632c94e7a
Fix list objects test and remove all the old unnecessary files.
...
- Fix tests for new changes.
- Change Golang err as 'e' for the time being, before we bring in
probe removal change.
- Remove old structs and temporary files.
9 years ago
0479d4976b
objectAPI: Fix object API interface, remove unnecessary structs.
...
ObjectAPI changes.
```
ListObjects(bucket, prefix, marker, delimiter string, maxKeys int) (ListObjectsInfo, *probe.Error)
ListMultipartUploads(bucket, objectPrefix, keyMarker, uploadIDMarker, delimiter string, maxUploads int) (ListMultipartsInfo, *probe.Error)
ListObjectParts(bucket, object, uploadID string, partNumberMarker, maxParts int) (ListPartsInfo, *probe.Error)
CompleteMultipartUpload(bucket string, object string, uploadID string, parts []completePart) (ObjectInfo, *probe.Error)
```
9 years ago
efc80343e3
fs: Break fs package to top-level and introduce ObjectAPI interface.
...
ObjectAPI interface brings in changes needed for XL ObjectAPI layer.
The new interface for any ObjectAPI layer is as below
```
// ObjectAPI interface.
type ObjectAPI interface {
// Bucket resource API.
DeleteBucket(bucket string) *probe.Error
ListBuckets() ([]BucketInfo, *probe.Error)
MakeBucket(bucket string) *probe.Error
GetBucketInfo(bucket string) (BucketInfo, *probe.Error)
// Bucket query API.
ListObjects(bucket, prefix, marker, delimiter string, maxKeys int) (ListObjectsResult, *probe.Error)
ListMultipartUploads(bucket string, resources BucketMultipartResourcesMetadata) (BucketMultipartResourcesMetadata, *probe.Error)
// Object resource API.
GetObject(bucket, object string, startOffset int64) (io.ReadCloser, *probe.Error)
GetObjectInfo(bucket, object string) (ObjectInfo, *probe.Error)
PutObject(bucket string, object string, size int64, data io.Reader, metadata map[string]string) (ObjectInfo, *probe.Error)
DeleteObject(bucket, object string) *probe.Error
// Object query API.
NewMultipartUpload(bucket, object string) (string, *probe.Error)
PutObjectPart(bucket, object, uploadID string, partID int, size int64, data io.Reader, md5Hex string) (string, *probe.Error)
ListObjectParts(bucket, object string, resources ObjectResourcesMetadata) (ObjectResourcesMetadata, *probe.Error)
CompleteMultipartUpload(bucket string, object string, uploadID string, parts []CompletePart) (ObjectInfo, *probe.Error)
AbortMultipartUpload(bucket, object, uploadID string) *probe.Error
}
```
9 years ago
9dca46e156
signature: Use a layered approach for signature verification.
...
Signature calculation has now moved out from being a package to
top-level as a layered mechanism.
In case of payload calculation with body, go-routines are initiated
to simultaneously write and calculate shasum. Errors are sent
over the writer so that the lower layer removes the temporary files
properly.
9 years ago
b55922effe
Fix for Istruncated set to true under certain conditions.
...
Optimizing List Objects by using binary sort to discard entries in cases
where prefix or marker is set.
Adding test coverage to ListObjects.
Adding benchmark to ListObjects.
9 years ago
5282a79eda
cleanup: Remove unecessary packages and tests. Simplify.
9 years ago
52751d81cb
cleanup: Rename ObjectMetadata as ObjectInfo.
...
Fixes #1215
9 years ago
d5057b3c51
accessPolicy: Implement Put, Get, Delete access policy.
...
This patch implements Get,Put,Delete bucket policies
Supporting - http://docs.aws.amazon.com/AmazonS3/latest/dev/access-policy-language-overview.html
Currently supports following actions.
"*": true,
"s3:*": true,
"s3:GetObject": true,
"s3:ListBucket": true,
"s3:PutObject": true,
"s3:CreateBucket": true,
"s3:GetBucketLocation": true,
"s3:DeleteBucket": true,
"s3:DeleteObject": true,
"s3:AbortMultipartUpload": true,
"s3:ListBucketMultipartUploads": true,
"s3:ListMultipartUploadParts": true,
following conditions for "StringEquals" and "StringNotEquals"
"s3:prefix", "s3:max-keys"
9 years ago
f111997184
multipart: remove proper MD5, rather create MD5 based on parts to be s3 compatible.
...
This increases the performance phenominally.
9 years ago
Andreas Auernhammer