* posix: Avoid using getAllVolumeInfo() in getVolumeDir()
This is necessary compromise to avoid significant slowness this
causes under load. The compromise is also substantial in a way
so that to avoid penalizing common cases v/s special cases.
For buckets with Caps on Unixes, we filter buckets based on the
latest anyways, so this is completely acceptable.
* XL/fs: Change the usage of verification of existance of buckets.
Optimize calling isBucketExists, it is not needed for all call
paths. isBucketExist should be called only for calls which use
temporary volume location for operations, for the rest rely on
the errors returned on their original call path.
Remove usage of filtering as well across all volume names.
This patch brings in the removal of debug logging altogether, instead
we bring in the functionality of being able to trace the errors properly
pointing back to the origination of the problem.
To enable tracing you need to enable "MINIO_TRACE" set to "1" or "true"
environment variable which would print back traces whenever there is an
error which is unhandled or at the handler layer.
By default this tracing is turned off and only user level logging is
provided.
Since there is a good amount of overlap, each code has to lock
properly for the operation they are going to perform.
- MakeVol create vols in a routine on all disks, hold locks.
- DeleteVol delete vols in a routine on all disks, hold locks.
- StatVol stat vols in a routine on all disks, hold locks.
Fixes#1588
All transactions happen through this directory inside ".minio/temp".
Adding this allows us to remove any temporary files which were not
committed before.
Fixes#1462Fixes#1444
* xl/ListFiles: return as many objects as requested and take care of eof (#1361)
* xl/ListFiles: fix review comments.
* xl/ListFiles: Add windows filepath translation.
* xl/ListFiles: Use slashSeparator instead of "/". Remove filepath.FromSlash() as golang-windows takes care of it automatically.
It is the bucket and volumes which needs to have this
value rather than the DiskInfo API itself. Eventually
this can be extended to show disk usage per
Buckets/Volumes whenever we have that functionality.
For now since buckets/volumes are thinly provisioned
this is the right approach.
Previously scanMultipartDir() returns object info channel and timer
channel where timer channel is used to check whether object info
channel is alive or not. This causes a race condition that timeout
may occur while object info channel in use.
This patch fixes the issue by removing timer channel and uses object
info channel directly where each object info has End bool field
indicates whether received object info is end or not.
- Rename dir.go as 'fs-multipart-dir.go'
- Move the push/pop to fs-multipart.go and rename them as save/lookup.
- Rename objectInfo instances in fs-multipart as multipartObjInfo.
- Fix tests for new changes.
- Change Golang err as 'e' for the time being, before we bring in
probe removal change.
- Remove old structs and temporary files.
This patch modifies multipart upload related functions as below
* New multipart upload call creates file
EXPORT_DIR/.minio/BUCKET/PATH/TO/OBJECT/UPLOAD_ID.uploadid
* Put object part call creates file
EXPORT_DIR/.minio/BUCKET/PATH/TO/OBJECT/UPLOAD_ID.PART_NUMBER.MD5SUM_STRING
* Abort multipart call removes all files matching
EXPORT_DIR/.minio/BUCKET/PATH/TO/OBJECT/UPLOAD_ID.*
* Complete multipart call does
1. creates a staging file
EXPORT_DIR/.minio/BUCKET/PATH/TO/OBJECT/UPLOAD_ID.complete.TEMP_NAME
then renames to
EXPORT_DIR/.minio/BUCKET/PATH/TO/OBJECT/UPLOAD_ID.complete
2. rename staging file
EXPORT_DIR/.minio/BUCKET/PATH/TO/OBJECT/UPLOAD_ID.complete
to EXPORT_DIR/BUCKET/PATH/TO/OBJECT
- Marker should be escaped outside in handlers.
- Delimiter should be handled outside in handlers.
- Add missing comments and change the function names.
- Handle case of 'maxKeys' when its set to '0', its a valid
case and should be treated as such.
* listObjects: improve response time by not doing stat during readDir() operation.
* listObjects: Add windows support.
* listObjects: Readdir() in batches to conserve memory. Add solaris build.
* listObjects: cleanup code.
When list object is invoked, it creates a goroutine if not available
for given parameters else uses existing goroutine. These goroutines
are alive for 15 seconds for further continuation list object request
else they exit.
Fixes#1076
Golang 1.6 is default version for the build now.
Additionally set 'GODEBUG=cgocheck=0' for now, until
we fix the erasure coding package.
Readmore here https://tip.golang.org/doc/go1.6#cgo
- Fixes initiating parallel uploads, and configs being quickly
re-written by another incoming request.
- Parallel uploads work smoothly now and return expected behavior.
Harshavardhana