97deba2a7c
GetKVS should add new keys automatically, preserve order ( #8612 )
5 years ago
3df7285c3c
Add Support for Cache and S3 related metrics in Prometheus endpoint ( #8591 )
...
This PR adds support below metrics
- Cache Hit Count
- Cache Miss Count
- Data served from Cache (in Bytes)
- Bytes received from AWS S3
- Bytes sent to AWS S3
- Number of requests sent to AWS S3
Fixes #8549
5 years ago
d2dc964cb5
Support TLS auth for Kafka notification target ( #8609 )
5 years ago
d8e3de0cae
Ensure comment is always a valid key ( #8604 )
...
Also fix LDAP leaky connection
5 years ago
c9940d8c3f
Final changes to config sub-system ( #8600 )
...
- Introduces changes such as certain types of
errors that can be ignored or which need to
go into safe mode.
- Update help text as per the review
5 years ago
794eb54da8
Export command prints turned-off sub-sys as comments ( #8594 )
...
This PR also tries to
- Preserve the order of keys printed in export command
- Fix cache to be enabled with _STATE env to keep
backward compatibility
5 years ago
2ab8d5e47f
Enable build verification with race ( #8583 )
5 years ago
947bc8c7d3
Update Azure Gateway to azure-storage-blob SDK ( #8537 )
...
The azure-sdk-for-go/storage package has been in maintenance-
only mode since February 2018 (see [1]) and will be deprecated in the future.
5 years ago
5d3d57c12a
Start using error wrapping with fmt.Errorf ( #8588 )
...
Use fatih/errwrap to fix all the code to use
error wrapping with fmt.Errorf()
5 years ago
0bfd20a8e3
Add client_id support for OpenID ( #8579 )
...
- One click OpenID authorization on Login page
- Add client_id help, config keys etc
Thanks to @egorkaru @ihostage for the
original work and testing.
5 years ago
db3dbcce3a
Print goroutines when shutdown hangs ( #8574 )
5 years ago
b21835f195
Honor DurationSeconds properly for WebIdentity ( #8581 )
...
Also cleanup code to add various constants for
verbatim strings across the code base.
Fixes #8482
5 years ago
c7844fb1fb
posix: cache disk ID for a short while ( #8564 )
...
`*posix.getDiskID()` takes up to 30% of all CPU due to the `os.Stat` call on `GET` calls.
Before:
```
Operation: GET - Concurrency: 12
* Average: 1333.97 MB/s, 1365.99 obj/s, 1365.98 ops ended/s (4m59.975s)
* First Byte: Average: 7.801487ms, Median: 7.9974ms, Best: 1.9822ms, Worst: 110.0021ms
Aggregated, split into 299 x 1s time segments:
* Fastest: 1453.50 MB/s, 1488.38 obj/s, 1492.00 ops ended/s (1s)
* 50% Median: 1360.47 MB/s, 1393.12 obj/s, 1393.00 ops ended/s (1s)
* Slowest: 978.68 MB/s, 1002.17 obj/s, 1004.00 ops ended/s (1s)
```
After:
```
Operation: GET - Concurrency: 12
* Average: 1706.07 MB/s, 1747.02 obj/s, 1747.01 ops ended/s (4m59.985s)
* First Byte: Average: 5.797886ms, Median: 5.9959ms, Best: 996.3µs, Worst: 84.0007ms
Aggregated, split into 299 x 1s time segments:
* Fastest: 1830.03 MB/s, 1873.96 obj/s, 1872.00 ops ended/s (1s)
* 50% Median: 1735.04 MB/s, 1776.68 obj/s, 1776.00 ops ended/s (1s)
* Slowest: 994.94 MB/s, 1018.82 obj/s, 1018.00 ops ended/s (1s)
```
TLDR; `os.Stat` is not free.
5 years ago
2ff8132e2d
Fix the regression introduced in #8580
5 years ago
30e80d0a86
Add ReadFrom,WriteTo helpers for server config ( #8580 )
5 years ago
87443af49e
Cache Windows mount point info ( #8572 )
...
It seems like every upload creates a new StorageInfo object.
This makes it very sensitive to slow code paths.
A particular slow function is `IsLikelyMountPoint` on Windows since `windows.GetVolumePathName` is slow.
We cache the result of this operation.
Before:
```
Operation: PUT - Concurrency: 12
* Average: 70.46 MB/s, 70.46 obj/s, 70.46 ops ended/s (59.58s)
Aggregated, split into 59 x 1s time segments:
* Fastest: 77.67 MB/s, 77.67 obj/s, 79.00 ops ended/s (1s)
* 50% Median: 70.84 MB/s, 70.84 obj/s, 70.00 ops ended/s (1s)
* Slowest: 61.51 MB/s, 61.51 obj/s, 61.00 ops ended/s (1s)
```
After:
```
Operation: PUT - Concurrency: 12
* Average: 166.80 MB/s, 166.80 obj/s, 166.74 ops ended/s (59.8s)
Aggregated, split into 59 x 1s time segments:
* Fastest: 183.83 MB/s, 183.83 obj/s, 186.00 ops ended/s (1s)
* 50% Median: 167.47 MB/s, 167.47 obj/s, 166.00 ops ended/s (1s)
* Slowest: 139.30 MB/s, 139.30 obj/s, 141.00 ops ended/s (1s)
```
5 years ago
5d65428b29
Handle localhost distributed setups properly ( #8577 )
...
Fixes an issue reported by @klauspost and @vadmeste
This PR also allows users to expand their clusters
from single node XL deployment to distributed mode.
5 years ago
78eb3b78bb
Repurpose Get/SetConfig as import/export support ( #8578 )
5 years ago
720442b1a2
Add lock expiry handler to expire state locks ( #8562 )
5 years ago
e542084c37
Add etcd path prefix for all IAM assets ( #8569 )
...
Currently, we use the top-level prefix "config/"
for all our IAM assets, instead of to provide
tenant-level separation bring 'path_prefix'
to namespace the access properly.
Fixes #8567
5 years ago
f931fc7bfb
Fix retention enforcement in Compliance mode ( #8556 )
...
In compliance mode, the retention date can be extended with
governance bypass permissions
5 years ago
0a56e33ce1
Preserve client sent config appropriately ( #8566 )
5 years ago
c3771df641
Add bootstrap REST handler for verifying server config ( #8550 )
5 years ago
890b493a2e
Use random file name for write check ( #8563 )
...
Since there may be multiple writes going on concurrently
Use a random file name for the write check to avoid collisions.
5 years ago
fb708b6b64
minimum Go version 1.13 ( #8561 )
5 years ago
f96e902f63
Do not rely on quorum for StorageInfo() ( #8557 )
...
StorageInfo() call is supposed to give each
server/disk information independently, rely
on this appropriately so that `mc admin info server`
gets correct information all the time.
5 years ago
06bd1e582a
Log in with OIDC not work with MINIO_DOMAIN ( #8558 ) ( #8559 )
5 years ago
fb43d64dc3
Fix healing on multiple zones ( #8555 )
...
It is expected in zone healing underlying
callers should return appropriate errors
5 years ago
fd0fa4e5c5
Add NTP retention time ( #8548 )
5 years ago
4e9de58675
Avoid pointer based copy, instead use Clone() ( #8547 )
...
This PR adds functional test to test expanded
cluster syntax.
5 years ago
9565641b9b
Enhance ListObjectsV2 API to return UserDefined metadata ( #8539 )
5 years ago
4da68cfcfc
Handle indexes correctly in DeleteMultipleObjectsHandler ( #8544 )
...
Regression from #8509 which changes objectsToDelete entry
from a list to map. This will cause index out of range
panic if object is not selected for delete.
5 years ago
a9b87c0a16
Fix typo in event name ( #8545 )
5 years ago
5ac4b517c9
Order all keys in config ( #8541 )
...
New changes
- return default values when sub-sys is
not configured.
- state is hidden parameter now
- remove worm mode to be saved in config
5 years ago
ca96560d56
Add object retention at the per object ( #8528 )
...
level - this PR builds on #8120 which
added PutBucketObjectLockConfiguration and
GetBucketObjectLockConfiguration APIS
This PR implements PutObjectRetention,
GetObjectRetention API and enhances
PUT and GET API operations to display
governance metadata if permissions allow.
5 years ago
cc1a84b62e
Fix heal result item output to properly count drives and sets ( #8543 )
5 years ago
8392d2f510
Preserve same deploymentID on all zones ( #8542 )
5 years ago
347b29d059
Implement bucket expansion ( #8509 )
5 years ago
3a34d98db8
Initialize local nsLocker for gateway instances ( #8540 )
5 years ago
7cdb67680e
Add help with order of keys ( #8535 )
5 years ago
929951fd49
Add support for multiple admins ( #8487 )
...
Also define IAM policies for administering
MinIO server
5 years ago
13a3d17321
Do not add comments after migration ( #8530 )
...
Also filter out empty comments from being
printed.
5 years ago
a8e156d6a5
Fix cache locking to use local namespace locking ( #8529 )
5 years ago
c9be601988
NATS TLS specify CA and client TLS authentication ( #8389 )
...
- added ability to specify CA for self-signed certificates
- added option to authenticate using client certificates
- added unit tests for nats connections
5 years ago
13e2b97ad9
Fix regression in caching on single PUT ( #8526 )
...
Regression caused by #8120
5 years ago
95e5d7a9c3
Improve access and secret key validation error, sync with implementation ( #8516 )
5 years ago
32c200fe12
Fix console logger crash in gateway mode ( #8525 )
...
This PR also fixes config migration only
for credentials and region which are valid
and set.
Also fix implicit `state="on"` behavior
5 years ago
1dd38750f7
Remove read-ahead for small files ( #8522 )
...
We should only read ahead if we are reading big files. We enable it for files >= 16MB.
Benchmark on 64KB objects.
Before:
```
Operation: GET
Errors: 0
Average: 59.976s, 87.13 MB/s, 1394.07 ops ended/s.
Fastest: 1s, 90.99 MB/s, 1455.00 ops ended/s.
50% Median: 1s, 87.53 MB/s, 1401.00 ops ended/s.
Slowest: 1s, 81.39 MB/s, 1301.00 ops ended/s.
```
After:
```
Operation: GET
Errors: 0
Average: 59.992s, 207.99 MB/s, 3327.85 ops ended/s.
Fastest: 1s, 219.20 MB/s, 3507.00 ops ended/s.
50% Median: 1s, 210.54 MB/s, 3368.00 ops ended/s.
Slowest: 1s, 179.14 MB/s, 2865.00 ops ended/s.
```
The 64KB buffer is actually a small disadvantage for this case, but I believe it will be better in general than no buffer.
5 years ago
e3273bc5bf
Fix runtime panic in BackendDisks.Merge() ( #8524 )
5 years ago
26a866a202
Fix review comments and new changes in config ( #8515 )
...
- Migrate and save only settings which are enabled
- Rename logger_http to logger_webhook and
logger_http_audit to audit_webhook
- No more pretty printing comments, comment
is a key=value pair now.
- Avoid quotes on values which do not have space in them
- `state="on"` is implicit for all SetConfigKV unless
specified explicitly as `state="off"`
- Disabled IAM users should be disabled always
5 years ago
Harshavardhana