This PR is the first set of changes to move the config
to the backend, the changes use the existing `config.json`
allows it to be migrated such that we can save it in on
backend disks.
In future releases, we will slowly migrate out of the
current architecture.
Fixes#6182
When an S3 client issues a GET request with range specified, Minio
server returns some partial data with 206 http code. The latter
is sent in MINIO_HTTP_TRACE output which is incorrect. This PR
fixes the issue.
No locks are ever left in memory, we also
have a periodic interval of clearing stale locks
anyways. The lock instrumentation was not complete
and was seldom used.
Deprecate this for now and bring it back later if
it is really needed. This also in-turn seems to improve
performance slightly.
With CoreDNS now supporting etcdv3 as the DNS backend, we
can update our federation target to etcdv3. Users will now be
able to use etcdv3 server as the federation backbone.
Minio will update bucket data to etcdv3 and CoreDNS can pick
that data up and serve it as bucket style DNS path.
Commit 0d52126023 caused a regression in setting
a new bucket policy in a distributed setup. The reason is that gob is not able
to encode fields declared as interfaces unless we provide GobEncode() and GobDecode()
This PR adds them by using json marshaller and unmarshaller that are already
implemented for Functions interface.
disk usage crawling is not needed when a tenant
is not sharing the same disk for multiple other
tenants. This PR adds an optimization when we
see a setup uses entire disk, we simply rely on
statvfs() to give us total usage.
This PR also additionally adds low priority
scheduling for usage check routine, such that
other go-routines blocked will be automatically
unblocked and prioritized before usage.
This PR adds CopyObject support for objects residing in buckets
in different Minio instances (where Minio instances are part of
a federated setup).
Also, added support for multiple Minio domain IPs. This is required
for distributed deployments, where one deployment may have multiple
nodes, each with a different public IP.
Buckets already present on a Minio server before it joins a
bucket federated deployment will now be added to etcd during
startup. In case of a bucket name collision, admin is informed
via Minio server console message.
Added configuration migration for configuration stored in etcd
backend.
Also, environment variables are updated and ListBucket path style
request is no longer forwarded.
This commit adds a check to the server's admin-API such that it only
accepts Admin-API requests with authenticated bodies. Further this
commit updates the `madmin` package to always add the
`X-Amz-Content-Sha256` header.
This change improves the Admin-API security since the server does not
accept unauthenticated request bodies anymore.
After this commit `mc` must be updated to the new `madmin` api because
requests over TLS connections will fail.
This commit fixes a DoS vulnerability for certain APIs using
signature V4 by verifying the content-md5 and/or content-sha56 of
the request body in a streaming mode.
The issue was caused by reading the entire body of the request into
memory to verify the content-md5 or content-sha56 checksum if present.
The vulnerability could be exploited by either replaying a V4 request
(in the 15 min time frame) or sending a V4 presigned request with a
large body.
This PR fixes a regression where the remote bucket policies
were not updated due to policy.Resource not having exported
fields, gob fails with unexported fields while marshalling.
This commit ensures that all tickers are stopped using defer ticker.Stop()
style. This will also fix one bug seen when a client starts to listen to
event notifications and that case will result a leak in tickers.
Previously we used allow bucket policies without
`Version` field to be set to any given value, but
this behavior is inconsistent with AWS S3.
PR #5790 addressed this by making bucket policies
stricter and cleaner, but this causes a breaking
change causing any existing policies perhaps without
`Version` field or the field to be empty to fail upon
server startup.
This PR brings a code to migrate under these scenarios
as a one time operation.
- remove old bucket policy handling
- add new policy handling
- add new policy handling unit tests
This patch brings support to bucket policy to have more control not
limiting to anonymous. Bucket owner controls to allow/deny any rest
API.
For example server side encryption can be controlled by allowing
PUT/GET objects with encryptions including bucket owner.
This is an effort to remove panic from the source.
Add a new call called CriticialIf, that calls LogIf and exits.
Replace panics with one of CriticalIf, FatalIf and a return of error.
As we move to multiple config backends like local disk and etcd,
config file should not be read from the disk, instead the quick
package should load and verify for duplicate entries.
fixes a regression introduced in 0e4431725c
when removing a previously applied notification configuration.
event.ParseConfig() was stricter in terms of handling notification
configuration, we need to allow when notification configuration is
sent empty, this is the way to remove notification configuration.
Current code didn't implement the logic to support
decrypting encrypted multiple parts, this PR fixes
by supporting copying encrypted multipart objects.
This PR implements an object layer which
combines input erasure sets of XL layers
into a unified namespace.
This object layer extends the existing
erasure coded implementation, it is assumed
in this design that providing > 16 disks is
a static configuration as well i.e if you started
the setup with 32 disks with 4 sets 8 disks per
pack then you would need to provide 4 sets always.
Some design details and restrictions:
- Objects are distributed using consistent ordering
to a unique erasure coded layer.
- Each pack has its own dsync so locks are synchronized
properly at pack (erasure layer).
- Each pack still has a maximum of 16 disks
requirement, you can start with multiple
such sets statically.
- Static sets set of disks and cannot be
changed, there is no elastic expansion allowed.
- Static sets set of disks and cannot be
changed, there is no elastic removal allowed.
- ListObjects() across sets can be noticeably
slower since List happens on all servers,
and is merged at this sets layer.
Fixes#5465Fixes#5464Fixes#5461Fixes#5460Fixes#5459Fixes#5458Fixes#5460Fixes#5488Fixes#5489Fixes#5497Fixes#5496
in-memory caching cannot be cleanly implemented
without the access to GC which Go doesn't naturally
provide. At times we have seen that object caching
is more of an hindrance rather than a boon for
our use cases.
Removing it completely from our implementation
related to #5160 and #5182
This is a generic minimum value. The current reason is to support
Azure blob storage accounts name whose length is less than 5. 3 is the
minimum length for Azure.
Harshavardhana