Kubernetes concepts like Deployments and StatefulSets provide perfect platform to deploy MinIO server in standalone, distributed or gateway mode. There are multiple options to deploy MinIO on Kubernetes, you can choose the one that best suits your requirements.
MinIO is a high performance distributed object storage server, designed for large-scale private cloud infrastructure. MinIO is designed in a cloud-native manner to scale sustainably in multi-tenant environments. Orchestration platforms like Kubernetes provide perfect cloud-native environment to deploy and scale MinIO.
- Helm Chart: MinIO Helm Chart offers customizable and easy MinIO deployment with a single command. Refer [MinIO Helm Chart repository documentation](https://github.com/helm/charts/tree/master/stable/minio) for more details.
## MinIO Deployment on Kubernetes
- YAML File: MinIO can be deployed with `yaml` files via `kubectl`. This document outlines steps required to deploy MinIO using `yaml` files.
There are multiple options to deploy MinIO on Kubernetes:
## Table of Contents
- MinIO-Operator: Operator offers seamless way to create and update highly available distributed MinIO clusters. Refer [MinIO Operator documentation](https://github.com/minio/minio-operator/blob/master/README.md) for more details.
- [Prerequisites](#Prerequisites)
- [MinIO Standalone Server Deployment](#minio-standalone-server-deployment)
- [MinIO Distributed Server Deployment](#minio-distributed-server-deployment)
- [Monitoring MinIO in Kubernetes](#monitoring-minio)
- Helm Chart: MinIO Helm Chart offers customizable and easy MinIO deployment with a single command. Refer [MinIO Helm Chart documentation](https://github.com/helm/charts/tree/master/stable/minio) for more details.
<aname="Prerequisites"></a>
## Prerequisites
To run this example, you need Kubernetes version >=1.4 cluster installed and running, and that you have installed the [`kubectl`](https://kubernetes.io/docs/tasks/kubectl/install/) command line tool in your path. Please see the [getting started guides](https://kubernetes.io/docs/setup/) for installation instructions for your platform.
<aname="minio-standalone-server-deployment"></a>
## MinIO Standalone Server Deployment
The following section describes the process to deploy standalone [MinIO](https://min.io/) server on Kubernetes. The deployment uses the [official MinIO Docker image](https://hub.docker.com/r/minio/minio/~/dockerfile/) from Docker Hub.
This section uses following core components of Kubernetes:
MinIO needs persistent storage to store objects. If there is no persistent storage, the data stored in MinIO instance will be stored in the container file system and will be wiped off as soon as the container restarts.
Create a persistent volume claim (PVC) to request storage for the MinIO instance. Kubernetes looks out for PVs matching the PVC request in the cluster and binds it to the PVC automatically. Create the PersistentVolumeClaim
A deployment encapsulates replica sets and pods. If a pod goes down, replication controller makes sure another pod comes up automatically. This way you won’t need to bother about pod failures and will have a stable MinIO service available. Create the Deployment using the following command
Now that you have a MinIO deployment running, you may either want to access it internally (within the cluster) or expose it as a Service onto an external (outside of your cluster, maybe public internet) IP address, depending on your use case. You can achieve this using Services. There are 3 major service types—default type is ClusterIP, which exposes a service to connection from inside the cluster. NodePort and LoadBalancer are two types that expose services to external traffic.
In this example, we expose the MinIO Deployment by creating a LoadBalancer service. Create the MinIO service using the following command
You can update an existing MinIO deployment to use a newer MinIO release. To do this, use the `kubectl set image` command:
```sh
kubectl set image deployment/minio-deployment minio=<replace-with-new-minio-image>
```
Kubernetes will restart the deployment to update the image. You will get a message as shown below, on successful update:
```
deployment "minio-deployment" image updated
```
### Standalone Resource cleanup
You can cleanup the cluster using
```sh
kubectl delete deployment minio \
&& kubectl delete pvc minio-pv-claim \
&& kubectl delete svc minio-service
```
<aname="minio-distributed-server-deployment"></a>
## MinIO Distributed Server Deployment
The following document describes the process to deploy [distributed MinIO](https://docs.min.io/docs/distributed-minio-quickstart-guide) server on Kubernetes. This example uses the [official MinIO Docker image](https://hub.docker.com/r/minio/minio/~/dockerfile/) from Docker Hub.
This example uses following core components of Kubernetes:
Headless Service controls the domain within which StatefulSets are created. The domain managed by this Service takes the form: `$(service name).$(namespace).svc.cluster.local` (where “cluster.local” is the cluster domain), and the pods in this domain take the form: `$(pod-name-{i}).$(service name).$(namespace).svc.cluster.local`. This is required to get a DNS resolvable URL for each of the pods created within the Statefulset. Create the Headless Service using the following command
A StatefulSet provides a deterministic name and a unique identity to each pod, making it easy to deploy stateful distributed applications. To launch distributed MinIO you need to pass drive locations as parameters to the minio server command. Then, you’ll need to run the same command on all the participating pods. StatefulSets offer a perfect way to handle this requirement. Create the Statefulset using the following command
Now that you have a MinIO statefulset running, you may either want to access it internally (within the cluster) or expose it as a Service onto an external (outside of your cluster, maybe public internet) IP address, depending on your use case. You can achieve this using Services. There are 3 major service types—default type is ClusterIP, which exposes a service to connection from inside the cluster. NodePort and LoadBalancer are two types that expose services to external traffic. Create the MinIO service using the following command
On successful update, you should see the output below
```
statefulset "minio" patched
```
Then delete all the pods in your StatefulSet one by one as shown below. Kubernetes will restart those pods for you, using the new image.
```sh
kubectl delete minio-0
```
### Resource cleanup
You can cleanup the cluster using
```sh
kubectl delete statefulset minio \
&& kubectl delete svc minio \
&& kubectl delete svc minio-service
```
### Deploying on cluster nodes with local host path
If your cluster does not have a storage solution or PV abstraction, you must explicitly define what nodes you wish to run MinIO on, and define a homogeneous path to a local fast block device available on every host.
This must be changed in the example daemonset: [minio-distributed-daemonset.yaml](https://github.com/minio/minio/blob/master/docs/orchestration/kubernetes/minio-distributed-daemonset.yaml)
Specifically the hostpath:
```yaml
hostPath:
path: /data/minio/
```
And the list of hosts:
```yaml
- http://hostname{1...4}/data/minio
```
Once deployed, tag the defined host with the `minio-server=true` label:
```bash
kubectl label node hostname1 -l minio-server=true
kubectl label node hostname2 -l minio-server=true
kubectl label node hostname3 -l minio-server=true
kubectl label node hostname4 -l minio-server=true
```
<aname="minio-gcs-gateway-deployment"></a>
## MinIO GCS Gateway Deployment
The following section describes the process to deploy [MinIO](https://min.io/) GCS Gateway on Kubernetes. The deployment uses the [official MinIO Docker image](https://hub.docker.com/r/minio/minio/~/dockerfile/) from Docker Hub.
This section uses following core components of Kubernetes:
Create the Google Cloud Service credentials file using the steps mentioned [here](https://github.com/minio/minio/blob/master/docs/gateway/gcs.md#create-service-account-key-for-gcs-and-get-the-credentials-file).
Use the path of file generated above to create a Kubernetes `secret`.
A `secret` is intended to hold sensitive information, such as passwords, OAuth tokens, and ssh keys. Putting this information in a secret is safer and more flexible than putting it verbatim in a pod definition or in a docker image.
Create the Google Cloud Service credentials file using the steps mentioned [here](https://github.com/minio/minio/blob/master/docs/gateway/gcs.md#create-service-account-key-for-gcs-and-get-the-credentials-file).
Use the path of file generated above to create a Kubernetes `secret`.
A deployment encapsulates replica sets and pods—so, if a pod goes down, replication controller makes sure another pod comes up automatically. This way you won’t need to bother about pod failures and will have a stable MinIO service available.
MinIO Gateway uses GCS as its storage backend and need to use a GCP `projectid` to identify your credentials. Update the section `gcp_project_id` with your
GCS project ID. Create the Deployment using the following command
Now that you have a MinIO deployment running, you may either want to access it internally (within the cluster) or expose it as a Service onto an external (outside of your cluster, maybe public internet) IP address, depending on your use case. You can achieve this using Services. There are 3 major service types—default type is ClusterIP, which exposes a service to connection from inside the cluster. NodePort and LoadBalancer are two types that expose services to external traffic. Create the MinIO service using the following command
You can update an existing MinIO deployment to use a newer MinIO release. To do this, use the `kubectl set image` command:
```sh
kubectl set image deployment/minio-deployment minio=<replace-with-new-minio-image>
```
Kubernetes will restart the deployment to update the image. You will get a message as shown below, on successful update:
```
deployment "minio-deployment" image updated
```
### GCS Gateway Resource Cleanup
You can cleanup the cluster using
```sh
kubectl delete deployment minio-deployment \
&& kubectl delete secret gcs-credentials
```
<aname="monitoring-minio"></a>
- YAML File: MinIO can be deployed with `YAML` files via `kubectl`. Refer to the [MinIO `YAML` file documentation](https://github.com/minio/minio/blob/master/docs/orchestration/kubernetes/k8s-yaml.md) for more details.
## Monitoring MinIO in Kubernetes
@ -327,6 +19,7 @@ MinIO server exposes un-authenticated readiness and liveness endpoints so Kubern
_Note_ : Readiness check is not allowed in distributed MinIO deployment. This is because Kubernetes doesn't allow any traffic to containers whose Readiness checks fail, and in a distributed setup, MinIO server can't respond to Readiness checks until all the nodes are reachable. So, Liveness checks are recommended native Kubernetes monitoring approach for distributed MinIO StatefulSets. Read more about Kubernetes recommendations for [container probes](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-probes).
- [Monitoring MinIO in Kubernetes](#monitoring-minio)
<aname="Prerequisites"></a>
## Prerequisites
To run this example, you need Kubernetes version >=1.4 cluster installed and running. You will also need to have installed and configured the [`kubectl`](https://kubernetes.io/docs/tasks/kubectl/install/) command line tool in your path. Please see the [getting started guides](https://kubernetes.io/docs/setup/) for installation instructions for your platform.
<aname="minio-standalone-server-deployment"></a>
## MinIO Standalone Server Deployment
The following section describes the process to deploy standalone MinIO server on Kubernetes. The deployment uses the [official MinIO Docker image](https://hub.docker.com/r/minio/minio/) from Docker Hub.
This section uses following core components of Kubernetes:
MinIO needs persistent storage to store objects. If there is no persistent storage, the data stored in MinIO instance will be stored in the container file system and will be destroyed as soon as the container restarts.
Create a persistent volume claim (PVC) to request storage for the MinIO instance. Kubernetes looks out for PVs matching the PVC request in the cluster and binds it to the PVC automatically. Run the following to create the PersistentVolumeClaim:
A deployment encapsulates replica sets and pods. If a pod goes down, the replication controller makes sure another pod comes up automatically. This way, you won’t need to bother about pod failures and will have a stable MinIO service available. Create the Deployment using the following command:
Now that you have a MinIO deployment running, you may either access it internally (within the cluster) or expose it as a Service onto an external (outside of your cluster, for example public internet) IP address, depending on your use case. You can achieve this using Services. There are 3 major service types—the default type is ClusterIP, which exposes a service to connection from inside the cluster. NodePort and LoadBalancer are two types that expose services to external traffic.
In this example, we expose the MinIO Deployment by creating a LoadBalancer service. Create the MinIO service using the following command
You can update an existing MinIO deployment to use a newer MinIO release. To do this, use the `kubectl set image` command:
```sh
kubectl set image deployment/minio-deployment minio=<replace-with-new-minio-image>
```
Kubernetes will restart the deployment to update the image. You will get a message similar to the one shown below, on successful update:
```
deployment "minio-deployment" image updated
```
### Standalone Resource cleanup
You can cleanup the cluster using
```sh
kubectl delete deployment minio \
&& kubectl delete pvc minio-pv-claim \
&& kubectl delete svc minio-service
```
<aname="minio-distributed-server-deployment"></a>
## MinIO Distributed Server Deployment
The following document describes the process to deploy [distributed MinIO](https://docs.min.io/docs/distributed-minio-quickstart-guide) server on Kubernetes. This example uses the [official MinIO Docker image](https://hub.docker.com/r/minio/minio/~/dockerfile/) from Docker Hub.
This example uses following core components of Kubernetes:
Headless Service controls the domain within which StatefulSets are created. The domain managed by this Service takes the form: `$(service name).$(namespace).svc.cluster.local` (where “cluster.local” is the cluster domain), and the pods in this domain take the form: `$(pod-name-{i}).$(service name).$(namespace).svc.cluster.local`. This is required to get a DNS resolvable URL for each of the pods created within the Statefulset. Create the Headless Service using the following command
A StatefulSet provides a deterministic name and a unique identity to each pod, making it easy to deploy stateful distributed applications. To launch distributed MinIO you need to pass drive locations as parameters to the minio server command. Then, you’ll need to run the same command on all the participating pods. StatefulSets offer a perfect way to handle this requirement. Create the Statefulset using the following command:
Now that you have a MinIO statefulset running, you may either access it internally (within the cluster) or expose it as a Service onto an external (outside of your cluster, maybe public internet) IP address, depending on your use case. You can achieve this using Services. There are 3 major service types—default type is ClusterIP, which exposes a service to connection from inside the cluster. NodePort and LoadBalancer are two types that expose services to external traffic. Create the MinIO service using the following command
On successful update, you should see the output below
```
statefulset "minio" patched
```
Then delete all the pods in your StatefulSet one by one using the command shown below.
```sh
kubectl delete minio-0
```
As you delete each pod, Kubernetes will restart that pod for you, using the new image. Once all the pods are restarted, your upgrade process is
complete.
### Resource cleanup
You can cleanup the cluster using
```sh
kubectl delete statefulset minio \
&& kubectl delete svc minio \
&& kubectl delete svc minio-service
```
### Deploying on cluster nodes with local host path
If your cluster does not have a storage solution or PV abstraction, you must explicitly define what nodes you wish to run MinIO on, and define a homogeneous path to a local fast block device available on every host.
This must be changed in the example daemonset: [minio-distributed-daemonset.yaml](https://github.com/minio/minio/blob/master/docs/orchestration/kubernetes/minio-distributed-daemonset.yaml)
Specifically the hostpath:
```yaml
hostPath:
path: /data/minio/
```
And the list of hosts:
```yaml
- http://hostname{1...4}/data/minio
```
Once deployed, tag the defined host with the `minio-server=true` label:
```bash
kubectl label node hostname1 -l minio-server=true
kubectl label node hostname2 -l minio-server=true
kubectl label node hostname3 -l minio-server=true
kubectl label node hostname4 -l minio-server=true
```
<aname="minio-gcs-gateway-deployment"></a>
## MinIO GCS Gateway Deployment
The following section describes the process to deploy MinIO GCS Gateway on Kubernetes. The deployment uses the [official MinIO Docker image](https://hub.docker.com/r/minio/minio/) from Docker Hub.
This section uses following core components of Kubernetes:
Create the Google Cloud Service credentials file using the steps mentioned [here](https://github.com/minio/minio/blob/master/docs/gateway/gcs.md#create-service-account-key-for-gcs-and-get-the-credentials-file).
Use the path of file generated above to create a Kubernetes `secret`.
A `secret` is intended to hold sensitive information, such as passwords, OAuth tokens, and ssh keys. Putting this information in a secret is safer and more flexible than putting it verbatim in a pod definition or in a docker image.
Create the Google Cloud Service credentials file using the steps mentioned [here](https://github.com/minio/minio/blob/master/docs/gateway/gcs.md#create-service-account-key-for-gcs-and-get-the-credentials-file).
Use the path of file generated above to create a Kubernetes `secret`.
A deployment encapsulates replica sets and pods—so, if a pod goes down, the replication controller makes sure another pod comes up automatically. This way you won’t need to bother about pod failures and will have a stable MinIO service available.
MinIO Gateway uses GCS as its storage backend and needs to use a GCP `projectid` to identify your credentials. Update the section `gcp_project_id` with your
GCS project ID. Create the Deployment using the following command
Now that you have a MinIO deployment running, you may either access it internally (within the cluster) or expose it as a Service onto an external (outside of your cluster, for example via public internet) IP address, depending on your use case. You can achieve this using Services. There are 3 major service types—default type is ClusterIP, which exposes a service to connection from inside the cluster. NodePort and LoadBalancer are two types that expose services to external traffic. Create the MinIO service using the following command
Nitish Tiwari
5 years ago
committed by