minio

Commit Graph

Author	SHA1	Message	Date
Klaus Post	ddea0bdf11	Concurrent CSV parsing and reduce S3 select allocations (#8200 ) ``` CSV parsing, BEFORE: BenchmarkReaderBasic-12 2842 407533 ns/op 397860 B/op 957 allocs/op BenchmarkReaderReplace-12 2718 429914 ns/op 397844 B/op 957 allocs/op BenchmarkReaderReplaceTwo-12 2718 435556 ns/op 397855 B/op 957 allocs/op BenchmarkAggregateCount_100K-12 171 6798974 ns/op 16667102 B/op 308077 allocs/op BenchmarkAggregateCount_1M-12 19 65657411 ns/op 168057743 B/op 3146610 allocs/op BenchmarkSelectAll_10M-12 1 20882119900 ns/op 2758799896 B/op 41978762 allocs/op CSV parsing, AFTER: BenchmarkReaderBasic-12 3721 312549 ns/op 101920 B/op 338 allocs/op BenchmarkReaderReplace-12 3776 318810 ns/op 101993 B/op 340 allocs/op BenchmarkReaderReplaceTwo-12 3610 330967 ns/op 102012 B/op 341 allocs/op BenchmarkAggregateCount_100K-12 295 4149588 ns/op 3553623 B/op 103261 allocs/op BenchmarkAggregateCount_1M-12 30 37746503 ns/op 33827931 B/op 1049435 allocs/op BenchmarkSelectAll_10M-12 1 17608495800 ns/op 1416504040 B/op 21007082 allocs/op ~ benchcmp old.txt new.txt benchmark old ns/op new ns/op delta BenchmarkReaderBasic-12 407533 312549 -23.31% BenchmarkReaderReplace-12 429914 318810 -25.84% BenchmarkReaderReplaceTwo-12 435556 330967 -24.01% BenchmarkAggregateCount_100K-12 6798974 4149588 -38.97% BenchmarkAggregateCount_1M-12 65657411 37746503 -42.51% BenchmarkSelectAll_10M-12 20882119900 17608495800 -15.68% benchmark old allocs new allocs delta BenchmarkReaderBasic-12 957 338 -64.68% BenchmarkReaderReplace-12 957 340 -64.47% BenchmarkReaderReplaceTwo-12 957 341 -64.37% BenchmarkAggregateCount_100K-12 308077 103261 -66.48% BenchmarkAggregateCount_1M-12 3146610 1049435 -66.65% BenchmarkSelectAll_10M-12 41978762 21007082 -49.96% benchmark old bytes new bytes delta BenchmarkReaderBasic-12 397860 101920 -74.38% BenchmarkReaderReplace-12 397844 101993 -74.36% BenchmarkReaderReplaceTwo-12 397855 102012 -74.36% BenchmarkAggregateCount_100K-12 16667102 3553623 -78.68% BenchmarkAggregateCount_1M-12 168057743 33827931 -79.87% BenchmarkSelectAll_10M-12 2758799896 1416504040 -48.66% ``` ``` BenchmarkReaderHuge/97K-12 2200 540840 ns/op 184.32 MB/s 1604450 B/op 687 allocs/op BenchmarkReaderHuge/194K-12 1522 752257 ns/op 265.04 MB/s 2143135 B/op 1335 allocs/op BenchmarkReaderHuge/389K-12 1190 947858 ns/op 420.69 MB/s 3221831 B/op 2630 allocs/op BenchmarkReaderHuge/778K-12 806 1472486 ns/op 541.61 MB/s 5201856 B/op 5187 allocs/op BenchmarkReaderHuge/1557K-12 426 2575269 ns/op 619.36 MB/s 9101330 B/op 10233 allocs/op BenchmarkReaderHuge/3115K-12 286 4034656 ns/op 790.66 MB/s 12397968 B/op 16099 allocs/op BenchmarkReaderHuge/6230K-12 172 6830563 ns/op 934.05 MB/s 16008416 B/op 26844 allocs/op BenchmarkReaderHuge/12461K-12 100 11409467 ns/op 1118.39 MB/s 22655163 B/op 48107 allocs/op BenchmarkReaderHuge/24922K-12 66 19780395 ns/op 1290.19 MB/s 35158559 B/op 90216 allocs/op BenchmarkReaderHuge/49844K-12 34 37282559 ns/op 1369.03 MB/s 60528624 B/op 174497 allocs/op ```	5 years ago
Krishnan Parthasarathi	6ba323b009	Add ability to test drive speeds on a MinIO setup (#7664 ) - Extends existing Admin API to measure disk performance	5 years ago
Harshavardhana	5512baab21	Fix go mod tidy errors and fix platform independent downloads (#8222 ) Refer https://github.com/gin-gonic/gin/issues/1673	5 years ago
Minio Trusted	3d65dc8d94	Update yaml files to latest version RELEASE.2019-09-11T19-53-16Z	5 years ago
Aditya Manthramurthy	a0456ce940	LDAP STS API (#8091 ) Add LDAP based users-groups system This change adds support to integrate an LDAP server for user authentication. This works via a custom STS API for LDAP. Each user accessing the MinIO who can be authenticated via LDAP receives temporary credentials to access the MinIO server. LDAP is enabled only over TLS. User groups are also supported via LDAP. The administrator may configure an LDAP search query to find the group attribute of a user - this may correspond to any attribute in the LDAP tree (that the user has access to view). One or more groups may be returned by such a query. A group is mapped to an IAM policy in the usual way, and the server enforces a policy corresponding to all the groups and the user's own mapped policy. When LDAP is configured, the internal MinIO users system is disabled.	5 years ago
Harshavardhana	94e5cb7576	Migrate to go1.13 to avail all new features (#8203 ) Read more https://blog.golang.org/go1.13	5 years ago
Klaus Post	33aec08e8c	Clean up go. mod (#8197 ) Golang proposes to clean up these and is fairly insisting about it ;)	5 years ago
Minio Trusted	5bde31d021	Update yaml files to latest version RELEASE.2019-09-05T23-24-38Z	5 years ago
Harshavardhana	b52a3e523c	Avoid using fastjson parser pool, move back to jsoniter (#8190 ) It looks like from implementation point of view fastjson parser pool doesn't behave the same way as expected when dealing many `xl.json` from multiple disks. The fastjson parser pool usage ends up returning incorrect xl.json entries for checksums, with references pointing to older entries. This led to the subtle bug where checksum info is duplicated from a previous xl.json read of a different file from different disk.	5 years ago
Minio Trusted	daa57e8e5a	Update yaml files to latest version RELEASE.2019-08-29T00-25-01Z	5 years ago
Minio Trusted	de736f2864	Update yaml files to latest version RELEASE.2019-08-21T19-40-07Z	5 years ago
Harshavardhana	9ca7470ccc	Avoid using jsoniter, move to fastjson (#8063 ) This is to avoid using unsafe.Pointer type code dependency for MinIO, this causes crashes on ARM64 platforms Refer #8005 collection of runtime crashes due to unsafe.Pointer usage incorrectly. We have seen issues like this before when using jsoniter library in the past. This PR hopes to fix this using fastjson	5 years ago
Minio Trusted	26640033b0	Update yaml files to latest version RELEASE.2019-08-14T20-37-41Z	5 years ago
Minio Trusted	55b385beee	Update yaml files to latest version RELEASE.2019-08-07T01-59-21Z	5 years ago
kannappanr	930943f058	Fix IAM users migration regression in etcd (#8029 ) PR #8008 did not migrate user data stored in etcd. This PR fixes that.	5 years ago
Minio Trusted	dfa8835720	Update yaml files to latest version RELEASE.2019-08-01T22-18-54Z	5 years ago
Minio Trusted	c301f5882d	Update yaml files to latest version RELEASE.2019-07-31T18-57-56Z	5 years ago
Harshavardhana	6f2b4675fa	Add krb5 support for HDFS gateway (#7933 )	5 years ago
Minio Trusted	5b71c21330	Update yaml files to latest version RELEASE.2019-07-24T02-02-23Z	5 years ago
Krishnan Parthasarathi	559a59220e	Add initial support for bucket lifecycle (#7563 ) This PR is based off @sinhaashish's PR for object lifecycle management, which includes support only for, - Expiration of object - Filter using object prefix (_not_ object tags) N B the code for actual expiration of objects will be included in a subsequent PR.	5 years ago
Minio Trusted	b058e32348	Update yaml files to latest version RELEASE.2019-07-17T22-54-12Z	5 years ago
Harshavardhana	16a45e5aff	Fix dynamic help vars for sub-commands (#7925 ) The fix in #7646 introduced a regression which was left unnoticed, the fix didn't work for sub-commands unfortunately. This fixes it by moving v1.21.0 version of the minio/cli package. Fixes #7924	5 years ago
Praveen raj Mani	bba562235b	Enable persistent event store in elasticsearch (#7564 )	5 years ago
Minio Trusted	2d96745156	Update yaml files to latest version RELEASE.2019-07-10T00-34-56Z	5 years ago
mzukowski-reef	9d49688c87	Switch to kurin/blazer from minio/blazer fork for b2 gateway (#7879 )	5 years ago
Minio Trusted	22bc15d89b	Update yaml files to latest version RELEASE.2019-07-05T21-20-21Z	5 years ago
Krishna Srinivas	a2e904b966	Support any string as delimiter for listing (#7882 )	5 years ago
Krishna Srinivas	338e9a9be9	Put object client disconnect (#7824 ) Fail putObject and postpolicy in case client prematurely disconnects Use request's context to cancel lock requests on client disconnects	6 years ago
Minio Trusted	5db60a6c59	Update yaml files to latest version RELEASE.2019-06-27T21-13-50Z	6 years ago
Minio Trusted	e43d3a075c	Update yaml files to latest version RELEASE.2019-06-19T18-24-42Z	6 years ago
Minio Trusted	59f7266081	Update yaml files to latest version RELEASE.2019-06-15T23-07-18Z	6 years ago
Harshavardhana	99bf4d0c42	[sec] Match ${aws:username} exactly instead of prefix match (#7791 ) This PR fixes a security issue where an IAM user based on his policy is granted more privileges than restricted by the users IAM policy. This is due to an issue of prefix based Matcher() function which was incorrectly matching prefix based on resource prefixes instead of exact match.	6 years ago
Minio Trusted	da2887f914	Update yaml files to latest version RELEASE.2019-06-13T01-41-13Z	6 years ago
Harshavardhana	c22439c82e	Update minio-go v6.0.29 (#7778 ) Bring improved retry logic	6 years ago
Harshavardhana	b4ab778cb2	Fix user IAM policy regression, reload policy appropriately (#7770 ) Introduce in commit `7e4c9a9e1e` Fixes #7769	6 years ago
Minio Trusted	d3a2efbf91	Update yaml files to latest version RELEASE.2019-06-11T00-44-33Z	6 years ago
Harshavardhana	cb1566c6e6	S3 Gateway: Handle restricted access credentials (#7757 )	6 years ago
Minio Trusted	df19546cc8	Update yaml files to latest version RELEASE.2019-06-04T01-15-58Z	6 years ago
Harshavardhana	ff7799452b	Fix build failure due to nats project migration	6 years ago
Minio Trusted	48ffb7731a	Update yaml files to latest version RELEASE.2019-06-01T03-46-14Z	6 years ago
Harshavardhana	2c0b3cadfc	Update go mod with sem versions of our libraries (#7687 )	6 years ago
Minio Trusted	7a02faab72	Update yaml files to latest version RELEASE.2019-05-23T00-29-34Z	6 years ago
Harshavardhana	39b3e4f9b3	Avoid using io.ReadFull() for WriteAll and CreateFile (#7676 ) With these changes we are now able to peak performances for all Write() operations across disks HDD and NVMe. Also adds readahead for disk reads, which also increases performance for reads by 3x.	6 years ago
Andrei Mikhalenia	59e847aebe	Signature v4: Allow signed headers from GET parameters	6 years ago
Harshavardhana	c5f26d5cdd	Fix hdfsReader fd leak upon GetObject() (#7596 ) Also migrate to minio/hdfs/v3@v3.0.0	6 years ago
Krishna Srinivas	a3ec71bc28	Use O_DIRECT while writing to disk (#7479 ) - Use O_DIRECT while writing to disk - Remove MINIO_DRIVE_SYNC option	6 years ago
Minio Trusted	ab711fe1a2	Update yaml files to latest version RELEASE.2019-04-23T23-50-36Z	6 years ago
Harshavardhana	f767a2538a	Optimize listing with leaf check offloaded to posix (#7541 ) Other listing optimizations include - remove double sorting while filtering object entries - improve error message when upload-id is not in quorum - use jsoniter for full unmarshal json, instead of gjson - remove unused code	6 years ago
Minio Trusted	d42496cc74	Update yaml files to latest version RELEASE.2019-04-18T21-44-59Z	6 years ago
Minio Trusted	7686340621	Update yaml files to latest version RELEASE.2019-04-18T01-15-57Z	6 years ago

1 2

57 Commits (975134e42b5897daca4928704c8573df6905b16c)