History

Harshavardhana 76e2713ffe fix: use buffers only when necessary for io.Copy() (#11229 ) Use separate sync.Pool for writes/reads Avoid passing buffers for io.CopyBuffer() if the writer or reader implement io.WriteTo or io.ReadFrom respectively then its useless for sync.Pool to allocate buffers on its own since that will be completely ignored by the io.CopyBuffer Go implementation. Improve this wherever we see this to be optimal. This allows us to be more efficient on memory usage. ``` 385 // copyBuffer is the actual implementation of Copy and CopyBuffer. 386 // if buf is nil, one is allocated. 387 func copyBuffer(dst Writer, src Reader, buf []byte) (written int64, err error) { 388 // If the reader has a WriteTo method, use it to do the copy. 389 // Avoids an allocation and a copy. 390 if wt, ok := src.(WriterTo); ok { 391 return wt.WriteTo(dst) 392 } 393 // Similarly, if the writer has a ReadFrom method, use it to do the copy. 394 if rt, ok := dst.(ReaderFrom); ok { 395 return rt.ReadFrom(src) 396 } ``` From readahead package ``` // WriteTo writes data to w until there's no more data to write or when an error occurs. // The return value n is the number of bytes written. // Any error encountered during the write is also returned. func (a *reader) WriteTo(w io.Writer) (n int64, err error) { if a.err != nil { return 0, a.err } n = 0 for { err = a.fill() if err != nil { return n, err } n2, err := w.Write(a.cur.buffer()) a.cur.inc(n2) n += int64(n2) if err != nil { return n, err } ```	3 years ago
..
DESIGN.md	fix: use buffers only when necessary for io.Copy() (#11229 )	3 years ago
README.md	feat: migrate to ROOT_USER/PASSWORD from ACCESS/SECRET_KEY (#11185 )	3 years ago

fix: use buffers only when necessary for io.Copy() (#11229 )

Use separate sync.Pool for writes/reads

Avoid passing buffers for io.CopyBuffer()
if the writer or reader implement io.WriteTo or io.ReadFrom
respectively then its useless for sync.Pool to allocate
buffers on its own since that will be completely ignored
by the io.CopyBuffer Go implementation.

Improve this wherever we see this to be optimal.

This allows us to be more efficient on memory usage.
```
   385  // copyBuffer is the actual implementation of Copy and CopyBuffer.
   386  // if buf is nil, one is allocated.
   387  func copyBuffer(dst Writer, src Reader, buf []byte) (written int64, err error) {
   388  	// If the reader has a WriteTo method, use it to do the copy.
   389  	// Avoids an allocation and a copy.
   390  	if wt, ok := src.(WriterTo); ok {
   391  		return wt.WriteTo(dst)
   392  	}
   393  	// Similarly, if the writer has a ReadFrom method, use it to do the copy.
   394  	if rt, ok := dst.(ReaderFrom); ok {
   395  		return rt.ReadFrom(src)
   396  	}
```

From readahead package
```
// WriteTo writes data to w until there's no more data to write or when an error occurs.
// The return value n is the number of bytes written.
// Any error encountered during the write is also returned.
func (a *reader) WriteTo(w io.Writer) (n int64, err error) {
	if a.err != nil {
		return 0, a.err
	}
	n = 0
	for {
		err = a.fill()
		if err != nil {
			return n, err
		}
		n2, err := w.Write(a.cur.buffer())
		a.cur.inc(n2)
		n += int64(n2)
		if err != nil {
			return n, err
		}
```

3 years ago

DESIGN.md

fix: use buffers only when necessary for io.Copy() (#11229 )

3 years ago

README.md

feat: migrate to ROOT_USER/PASSWORD from ACCESS/SECRET_KEY (#11185 )

3 years ago

README.md

Shared Backend MinIO Quickstart Guide

MinIO shared mode lets you use single NAS (like NFS, GlusterFS, and other distributed filesystems) as the storage backend for multiple MinIO servers. Synchronization among MinIO servers is taken care by design. Read more about the MinIO shared mode design here.

MinIO shared mode is developed to solve several real world use cases, without any special configuration changes. Some of these are

You have already invested in NAS and would like to use MinIO to add S3 compatibility to your storage tier.
You need to use NAS with an S3 interface due to your application architecture requirements.
You expect huge traffic and need a load balanced S3 compatible server, serving files from a single NAS backend.

With a proxy running in front of multiple, shared mode MinIO servers, it is very easy to create a Highly Available, load balanced, AWS S3 compatible storage system.

Get started

If you're aware of stand-alone MinIO set up, the installation and running remains the same.

1. Prerequisites

Install MinIO - MinIO Quickstart Guide.

2. Run MinIO on Shared Backend

To run MinIO shared backend instances, you need to start multiple MinIO servers pointing to the same backend storage. We'll see examples on how to do this in the following sections.

Note

All the nodes running shared MinIO need to have same access key and secret key. To achieve this, we export access key and secret key as environment variables on all the nodes before executing MinIO server command.
The drive paths below are for demonstration purposes only, you need to replace these with the actual drive paths/folders.

MinIO shared mode on Ubuntu 16.04 LTS.

You'll need the path to the shared volume, e.g. /path/to/nfs-volume. Then run the following commands on all the nodes you'd like to launch MinIO.

export MINIO_ROOT_USER=<ACCESS_KEY>
export MINIO_ROOT_PASSWORD=<SECRET_KEY>
minio gateway nas /path/to/nfs-volume

MinIO shared mode on Windows 2012 Server

You'll need the path to the shared volume, e.g. \\remote-server\smb. Then run the following commands on all the nodes you'd like to launch MinIO.

set MINIO_ROOT_USER=my-username
set MINIO_ROOT_PASSWORD=my-password
minio.exe gateway nas \\remote-server\smb\export

Windows Tip

If a remote volume, e.g. \\remote-server\smb is mounted as a drive, e.g. M:\. You can use net use command to map the drive to a folder.

set MINIO_ROOT_USER=my-username
set MINIO_ROOT_PASSWORD=my-password
net use m: \\remote-server\smb\export /P:Yes
minio.exe gateway nas M:\export

3. Test your setup

To test this setup, access the MinIO server via browser or mc. You’ll see the uploaded files are accessible from the all the MinIO shared backend endpoints.

README.md

Shared Backend MinIO Quickstart Guide

Get started

1. Prerequisites

2. Run MinIO on Shared Backend

MinIO shared mode on Ubuntu 16.04 LTS.

MinIO shared mode on Windows 2012 Server

3. Test your setup

Explore Further