c3d4c1f584
add minio/keys KMS integration ( #8631 )
...
This commit adds support for the minio/kes KMS.
See: https://github.com/minio/kes
In particular you can configure it as KMS by:
- `export MINIO_KMS_KES_ENDPOINT=` // Server URL
- `export MINIO_KMS_KES_KEY_FILE=` // TLS client private key
- `export MINIO_KMS_KES_CERT_FILE=` // TLS client certificate
- `export MINIO_KMS_KES_CA_PATH=` // Root CAs issuing server cert
- `export MINIO_KMS_KES_KEY_NAME=` // The name of the (default)
master key
5 years ago
471a3a650a
fix: Don't allow to set unconfigured notification ARNs ( #8643 )
...
Fixes #8642
5 years ago
cc02bf0442
Remove old ListenBucketNotification API ( #8645 )
5 years ago
39e8e4f4aa
Allow empty target KVS for notification targets ( #8644 )
...
This is allowed with enable=off arg value
5 years ago
80558e839d
Clear cache if reverting to backend ( #8637 )
...
Clear cached entry before reverting to backend for
encrypted objects or those under retention to avoid
stale objects remaining in cache.
5 years ago
ca62ac65d4
Reject mandatory KVS if not set for any sub-sys ( #8641 )
5 years ago
f5abe4e1f1
Support ListenBucketNotificationV2 streaming ( #8622 )
5 years ago
3211cb5df6
Add encryption buffer ( #8626 )
...
Quite hard to measure difference:
```
λ warp cmp put-before.csv.zst put-after2.csv.zst
Operation: PUT
Operations: 340 -> 353
* Average: +4.11% (+22.7 MB/s) throughput, +4.11% (+0.2) obj/s
* 50% Median: +1.58% (+7.3 MB/s) throughput, +1.58% (+0.1) obj/s
```
Difference is likely bigger on Intel platforms due to higher syscall costs.
5 years ago
abc266caa1
Add bucket and object count along with total object size ( #8639 )
5 years ago
c364f0af6c
Start using custom HTTP transport for webhook endpoints ( #8630 )
...
Use a more performant http transport for webhook
endpoints with proper connection pooling, appropriate
timeouts etc.
5 years ago
555969ee42
Add data usage collect with its new admin API ( #8553 )
...
Admin data usage info API returns the following
(Only FS & XL, for now)
- Number of buckets
- Number of objects
- The total size of objects
- Objects histogram
- Bucket sizes
5 years ago
e2c5d29017
Bucket,Object count & Usage removed if set to default ( #8638 )
5 years ago
fa00a84709
Avoid crashes on peers if IAMSys is not initialized ( #8636 )
5 years ago
d266b3a066
Admin Info: Modify Uptime to return seconds ( #8635 )
5 years ago
db3f41fcb4
Doc: change url for Total Population CSV ( #8633 )
5 years ago
24fb1bf258
New Admin Info ( #8497 )
5 years ago
8b803491af
fix: CacheOpts parsing tests ( #8632 )
5 years ago
10b2f15f6f
Add randomize sleep times for lock checkers ( #8628 )
5 years ago
3e9ab5f4a9
Fix k8s replica set deployment ( #8629 )
...
In replica sets, hosts resolve to localhost
IP automatically until the deployment fully
comes up. To avoid this issue we need to
wait for such resolution.
5 years ago
46b4dd8e20
Add command to list added lifecycle configurations ( #8627 )
5 years ago
bf3a97d3aa
S3 Select: Concurrent LINES delimited json parsing ( #8610 )
...
The speedup is ~5x on a 6 core CPU
5 years ago
3b67f629a4
Retry peer notification of events ( #8621 )
5 years ago
3c30e4503d
Cache only the range requested for range GETs ( #8599 )
5 years ago
8390bc26db
Fix cache hit metrics. ( #8617 )
5 years ago
24ad59316d
Use atomic.Uint64 for gateway metrics count instead of mutex ( #8615 )
5 years ago
be0c8b1ec0
Add support for missing Cache-Control directives ( #8619 )
...
no-cache, only-if-cached and no-store directives are
being enforced in this PR.
5 years ago
476111968a
Update help messages with new wording ( #8616 )
...
Final update to all messages across sub-systems
after final review, the only change here is that
NATS now has TLS and TLSSkipVerify to be consistent
for all other notification targets.
5 years ago
f1e2e1cc9e
S3 Select: Mismatched types don't match ( #8608 )
...
When comparing for equality, if types cannot be matched, they don't match.
5 years ago
97deba2a7c
GetKVS should add new keys automatically, preserve order ( #8612 )
5 years ago
3df7285c3c
Add Support for Cache and S3 related metrics in Prometheus endpoint ( #8591 )
...
This PR adds support below metrics
- Cache Hit Count
- Cache Miss Count
- Data served from Cache (in Bytes)
- Bytes received from AWS S3
- Bytes sent to AWS S3
- Number of requests sent to AWS S3
Fixes #8549
5 years ago
d2dc964cb5
Support TLS auth for Kafka notification target ( #8609 )
5 years ago
d8e3de0cae
Ensure comment is always a valid key ( #8604 )
...
Also fix LDAP leaky connection
5 years ago
c9940d8c3f
Final changes to config sub-system ( #8600 )
...
- Introduces changes such as certain types of
errors that can be ignored or which need to
go into safe mode.
- Update help text as per the review
5 years ago
794eb54da8
Export command prints turned-off sub-sys as comments ( #8594 )
...
This PR also tries to
- Preserve the order of keys printed in export command
- Fix cache to be enabled with _STATE env to keep
backward compatibility
5 years ago
2ab8d5e47f
Enable build verification with race ( #8583 )
5 years ago
947bc8c7d3
Update Azure Gateway to azure-storage-blob SDK ( #8537 )
...
The azure-sdk-for-go/storage package has been in maintenance-
only mode since February 2018 (see [1]) and will be deprecated in the future.
5 years ago
5d3d57c12a
Start using error wrapping with fmt.Errorf ( #8588 )
...
Use fatih/errwrap to fix all the code to use
error wrapping with fmt.Errorf()
5 years ago
0bfd20a8e3
Add client_id support for OpenID ( #8579 )
...
- One click OpenID authorization on Login page
- Add client_id help, config keys etc
Thanks to @egorkaru @ihostage for the
original work and testing.
5 years ago
db3dbcce3a
Print goroutines when shutdown hangs ( #8574 )
5 years ago
b21835f195
Honor DurationSeconds properly for WebIdentity ( #8581 )
...
Also cleanup code to add various constants for
verbatim strings across the code base.
Fixes #8482
5 years ago
c7844fb1fb
posix: cache disk ID for a short while ( #8564 )
...
`*posix.getDiskID()` takes up to 30% of all CPU due to the `os.Stat` call on `GET` calls.
Before:
```
Operation: GET - Concurrency: 12
* Average: 1333.97 MB/s, 1365.99 obj/s, 1365.98 ops ended/s (4m59.975s)
* First Byte: Average: 7.801487ms, Median: 7.9974ms, Best: 1.9822ms, Worst: 110.0021ms
Aggregated, split into 299 x 1s time segments:
* Fastest: 1453.50 MB/s, 1488.38 obj/s, 1492.00 ops ended/s (1s)
* 50% Median: 1360.47 MB/s, 1393.12 obj/s, 1393.00 ops ended/s (1s)
* Slowest: 978.68 MB/s, 1002.17 obj/s, 1004.00 ops ended/s (1s)
```
After:
```
Operation: GET - Concurrency: 12
* Average: 1706.07 MB/s, 1747.02 obj/s, 1747.01 ops ended/s (4m59.985s)
* First Byte: Average: 5.797886ms, Median: 5.9959ms, Best: 996.3µs, Worst: 84.0007ms
Aggregated, split into 299 x 1s time segments:
* Fastest: 1830.03 MB/s, 1873.96 obj/s, 1872.00 ops ended/s (1s)
* 50% Median: 1735.04 MB/s, 1776.68 obj/s, 1776.00 ops ended/s (1s)
* Slowest: 994.94 MB/s, 1018.82 obj/s, 1018.00 ops ended/s (1s)
```
TLDR; `os.Stat` is not free.
5 years ago
2ff8132e2d
Fix the regression introduced in #8580
5 years ago
30e80d0a86
Add ReadFrom,WriteTo helpers for server config ( #8580 )
5 years ago
87443af49e
Cache Windows mount point info ( #8572 )
...
It seems like every upload creates a new StorageInfo object.
This makes it very sensitive to slow code paths.
A particular slow function is `IsLikelyMountPoint` on Windows since `windows.GetVolumePathName` is slow.
We cache the result of this operation.
Before:
```
Operation: PUT - Concurrency: 12
* Average: 70.46 MB/s, 70.46 obj/s, 70.46 ops ended/s (59.58s)
Aggregated, split into 59 x 1s time segments:
* Fastest: 77.67 MB/s, 77.67 obj/s, 79.00 ops ended/s (1s)
* 50% Median: 70.84 MB/s, 70.84 obj/s, 70.00 ops ended/s (1s)
* Slowest: 61.51 MB/s, 61.51 obj/s, 61.00 ops ended/s (1s)
```
After:
```
Operation: PUT - Concurrency: 12
* Average: 166.80 MB/s, 166.80 obj/s, 166.74 ops ended/s (59.8s)
Aggregated, split into 59 x 1s time segments:
* Fastest: 183.83 MB/s, 183.83 obj/s, 186.00 ops ended/s (1s)
* 50% Median: 167.47 MB/s, 167.47 obj/s, 166.00 ops ended/s (1s)
* Slowest: 139.30 MB/s, 139.30 obj/s, 141.00 ops ended/s (1s)
```
5 years ago
5d65428b29
Handle localhost distributed setups properly ( #8577 )
...
Fixes an issue reported by @klauspost and @vadmeste
This PR also allows users to expand their clusters
from single node XL deployment to distributed mode.
5 years ago
78eb3b78bb
Repurpose Get/SetConfig as import/export support ( #8578 )
5 years ago
720442b1a2
Add lock expiry handler to expire state locks ( #8562 )
5 years ago
e542084c37
Add etcd path prefix for all IAM assets ( #8569 )
...
Currently, we use the top-level prefix "config/"
for all our IAM assets, instead of to provide
tenant-level separation bring 'path_prefix'
to namespace the access properly.
Fixes #8567
5 years ago
f931fc7bfb
Fix retention enforcement in Compliance mode ( #8556 )
...
In compliance mode, the retention date can be extended with
governance bypass permissions
5 years ago
0a56e33ce1
Preserve client sent config appropriately ( #8566 )
5 years ago
Andreas Auernhammer