Use client tools like `mc` to set and listen for event notifications using the [`event` sub-command](https://docs.min.io/docs/minio-client-complete-guide#events). MinIO SDK's [`BucketNotification` APIs](https://docs.min.io/docs/golang-client-api-reference#SetBucketNotification) can also be used. The notification message MinIO sends to publish an event is a JSON message with the following [structure](https://docs.aws.amazon.com/AmazonS3/latest/dev/notification-content-structure.html).
Use client tools like `mc` to set and listen for event notifications using the [`event` sub-command](https://docs.min.io/docs/minio-client-complete-guide#events). MinIO SDK's [`BucketNotification` APIs](https://docs.min.io/docs/golang-client-api-reference#SetBucketNotification) can also be used. The notification message MinIO sends to publish an event is a JSON message with the following [structure](https://docs.aws.amazon.com/AmazonS3/latest/dev/notification-content-structure.html).
Bucket events can be published to the following targets:
Bucket events can be published to the following targets:
* Install and configure MinIO Server from [here](https://docs.min.io/docs/minio-quickstart-guide).
- Install and configure MinIO Server from [here](https://docs.min.io/docs/minio-quickstart-guide).
* Install and configure MinIO Client from [here](https://docs.min.io/docs/minio-client-quickstart-guide).
- Install and configure MinIO Client from [here](https://docs.min.io/docs/minio-client-quickstart-guide).
<aname="AMQP"></a>
<aname="AMQP"></a>
## Publish MinIO events via AMQP
## Publish MinIO events via AMQP
Install RabbitMQ from [here](https://www.rabbitmq.com/).
Install RabbitMQ from [here](https://www.rabbitmq.com/).
@ -33,20 +34,20 @@ Install RabbitMQ from [here](https://www.rabbitmq.com/).
The MinIO server configuration file is stored on the backend in json format. The AMQP configuration is located in the `amqp` key under the `notify` top-level key. Create a configuration key-value pair here for your AMQP instance. The key is a name for your AMQP endpoint, and the value is a collection of key-value parameters described in the table below.
The MinIO server configuration file is stored on the backend in json format. The AMQP configuration is located in the `amqp` key under the `notify` top-level key. Create a configuration key-value pair here for your AMQP instance. The key is a name for your AMQP endpoint, and the value is a collection of key-value parameters described in the table below.
| `enable` | _bool_ | (Required) Is this server endpoint configuration active/enabled? |
| `enable`| _bool_ | (Required) Is this server endpoint configuration active/enabled? |
| `url` | _string_ | (Required) AMQP server endpoint, e.g. `amqp://myuser:mypassword@localhost:5672` |
| `url`| _string_ | (Required) AMQP server endpoint, e.g. `amqp://myuser:mypassword@localhost:5672` |
| `exchange` | _string_ | Name of the exchange. |
| `exchange`| _string_ | Name of the exchange. |
| `routingKey` | _string_ | Routing key for publishing. |
| `routingKey`| _string_ | Routing key for publishing. |
| `exchangeType` | _string_ | Kind of exchange. |
| `exchangeType` | _string_ | Kind of exchange. |
| `deliveryMode` | _uint8_ | Delivery mode for publishing. 0 or 1 - transient; 2 - persistent. |
| `deliveryMode` | _uint8_| Delivery mode for publishing. 0 or 1 - transient; 2 - persistent. |
| `mandatory` | _bool_ | Publishing related bool. |
| `mandatory`| _bool_| Publishing related bool. |
| `immediate` | _bool_ | Publishing related bool. |
| `immediate`| _bool_| Publishing related bool. |
| `durable` | _bool_ | Exchange declaration related bool. |
| `durable`| _bool_ | Exchange declaration related bool. |
| `internal` | _bool_ | Exchange declaration related bool. |
| `internal`| _bool_| Exchange declaration related bool. |
| `noWait` | _bool_ | Exchange declaration related bool. |
| `noWait`| _bool_| Exchange declaration related bool. |
| `autoDeleted` | _bool_ | Exchange declaration related bool. |
| `autoDeleted`| _bool_ | Exchange declaration related bool. |
An example configuration for RabbitMQ is shown below:
An example configuration for RabbitMQ is shown below:
@ -68,22 +69,26 @@ An example configuration for RabbitMQ is shown below:
}
}
}
}
```
```
To update the configuration, use `mc admin config get` command to get the current configuration file for the minio deployment in json format, and save it locally.
To update the configuration, use `mc admin config get` command to get the current configuration file for the minio deployment in json format, and save it locally.
```sh
```sh
$ mc admin config get myminio/ > /tmp/myconfig
$ mc admin config get myminio/ > /tmp/myconfig
```
```
After updating the AMQP configuration in /tmp/myconfig , use `mc admin config set` command to update the configuration for the deployment.Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:amqp` at start-up if there were no errors.
After updating the AMQP configuration in /tmp/myconfig , use `mc admin config set` command to update the configuration for the deployment.Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:amqp` at start-up if there were no errors.
```sh
```sh
$ mc admin config set myminio < /tmp/myconfig
$ mc admin config set myminio < /tmp/myconfig
```
```
MinIO supports all the exchanges available in [RabbitMQ](https://www.rabbitmq.com/). For this setup, we are using ``fanout`` exchange.
Note that, you can add as many AMQP server endpoint configurations as needed by providing an identifier (like "1" in the example above) for the AMQP instance and an object of per-server configuration parameters.
MinIO supports all the exchanges available in [RabbitMQ](https://www.rabbitmq.com/). For this setup, we are using `fanout` exchange.
Note that, you can add as many AMQP server endpoint configurations as needed by providing an identifier (like "1" in the example above) for the AMQP instance and an object of per-server configuration parameters.
### Step 2: Enable bucket notification using MinIO client
### Step 2: Enable bucket notification using MinIO client
We will enable bucket event notification to trigger whenever a JPEG image is uploaded or deleted ``images`` bucket on ``myminio`` server. Here ARN value is ``arn:minio:sqs::1:amqp``. To understand more about ARN please follow [AWS ARN](http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) documentation.
We will enable bucket event notification to trigger whenever a JPEG image is uploaded or deleted `images` bucket on `myminio` server. Here ARN value is `arn:minio:sqs::1:amqp`. To understand more about ARN please follow [AWS ARN](http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) documentation.
The python program below waits on the queue exchange ``bucketevents`` and prints event notifications on the console. We use [Pika Python Client](https://www.rabbitmq.com/tutorials/tutorial-three-python.html) library to do this.
The python program below waits on the queue exchange `bucketevents` and prints event notifications on the console. We use [Pika Python Client](https://www.rabbitmq.com/tutorials/tutorial-three-python.html) library to do this.
```py
```py
#!/usr/bin/env python
#!/usr/bin/env python
@ -131,7 +136,7 @@ Execute this example python program to watch for RabbitMQ events on the console.
python rabbit.py
python rabbit.py
```
```
Open another terminal and upload a JPEG image into ``images`` bucket.
Open another terminal and upload a JPEG image into `images` bucket.
```
```
mc cp myphoto.jpg myminio/images
mc cp myphoto.jpg myminio/images
@ -145,6 +150,7 @@ python rabbit.py
```
```
<aname="MQTT"></a>
<aname="MQTT"></a>
## Publish MinIO events MQTT
## Publish MinIO events MQTT
Install an MQTT Broker from [here](https://mosquitto.org/).
Install an MQTT Broker from [here](https://mosquitto.org/).
@ -153,17 +159,16 @@ Install an MQTT Broker from [here](https://mosquitto.org/).
The MinIO server configuration file is stored on the backend in json format. The MQTT configuration is located in the `mqtt` key under the `notify` top-level key. Create a configuration key-value pair here for your MQTT instance. The key is a name for your MQTT endpoint, and the value is a collection of key-value parameters described in the table below.
The MinIO server configuration file is stored on the backend in json format. The MQTT configuration is located in the `mqtt` key under the `notify` top-level key. Create a configuration key-value pair here for your MQTT instance. The key is a name for your MQTT endpoint, and the value is a collection of key-value parameters described in the table below.
| `enable` | _bool_ | (Required) Is this server endpoint configuration active/enabled? |
| `enable` | _bool_ | (Required) Is this server endpoint configuration active/enabled? |
| `broker` | _string_ | (Required) MQTT server endpoint, e.g. `tcp://localhost:1883` |
| `broker` | _string_ | (Required) MQTT server endpoint, e.g. `tcp://localhost:1883` |
| `topic` | _string_ | (Required) Name of the MQTT topic to publish on, e.g. `minio` |
| `topic` | _string_ | (Required) Name of the MQTT topic to publish on, e.g. `minio` |
| `qos` | _int_ | Set the Quality of Service Level |
| `qos` | _int_ | Set the Quality of Service Level |
| `username` | _string_ | Username to connect to the MQTT server (if required) |
| `username` | _string_ | Username to connect to the MQTT server (if required) |
| `password` | _string_ | Password to connect to the MQTT server (if required) |
| `password` | _string_ | Password to connect to the MQTT server (if required) |
| `queueDir` | _string_ | Persistent store for events when MQTT broker is offline |
| `queueDir` | _string_ | Persistent store for events when MQTT broker is offline |
| `queueLimit` | _int_ | Set the maximum event limit for the persistent store. The default limit is 10000 |
| `queueLimit` | _int_ | Set the maximum event limit for the persistent store. The default limit is 10000 |
An example configuration for MQTT is shown below:
An example configuration for MQTT is shown below:
@ -181,25 +186,28 @@ An example configuration for MQTT is shown below:
}
}
}
}
```
```
MinIO supports persistent event store. The persistent store will backup events when the MQTT broker goes offline and replays it when the broker comes back online. The event store can be configured by setting the directory path in `queueDir` field and the maximum limit of events in the queueDir in `queueLimit` field. For eg, the `queueDir` can be `/home/events` and `queueLimit` can be `1000`. By default, the `queueLimit` is set to 10000.
MinIO supports persistent event store. The persistent store will backup events when the MQTT broker goes offline and replays it when the broker comes back online. The event store can be configured by setting the directory path in `queueDir` field and the maximum limit of events in the queueDir in `queueLimit` field. For eg, the `queueDir` can be `/home/events` and `queueLimit` can be `1000`. By default, the `queueLimit` is set to 10000.
To update the configuration, use `mc admin config get` command to get the current configuration file for the minio deployment in json format, and save it locally.
To update the configuration, use `mc admin config get` command to get the current configuration file for the minio deployment in json format, and save it locally.
```sh
```sh
$ mc admin config get myminio/ > /tmp/myconfig
$ mc admin config get myminio/ > /tmp/myconfig
```
```
After updating the MQTT configuration in /tmp/myconfig , use `mc admin config set` command to update the configuration for the deployment.Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:mqtt` at start-up if there were no errors.
After updating the MQTT configuration in /tmp/myconfig , use `mc admin config set` command to update the configuration for the deployment.Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:mqtt` at start-up if there were no errors.
```sh
```sh
$ mc admin config set myminio < /tmp/myconfig
$ mc admin config set myminio < /tmp/myconfig
```
```
MinIO supports any MQTT server that supports MQTT 3.1 or 3.1.1 and can connect to them over TCP, TLS, or a Websocket connection using ``tcp://``, ``tls://``, or ``ws://`` respectively as the scheme for the broker url. See the [Go Client](http://www.eclipse.org/paho/clients/golang/) documentation for more information.
MinIO supports any MQTT server that supports MQTT 3.1 or 3.1.1 and can connect to them over TCP, TLS, or a Websocket connection using `tcp://`, `tls://`, or `ws://` respectively as the scheme for the broker url. See the [Go Client](http://www.eclipse.org/paho/clients/golang/) documentation for more information.
Note that, you can add as many MQTT server endpoint configurations as needed by providing an identifier (like "1" in the example above) for the MQTT instance and an object of per-server configuration parameters.
Note that, you can add as many MQTT server endpoint configurations as needed by providing an identifier (like "1" in the example above) for the MQTT instance and an object of per-server configuration parameters.
### Step 2: Enable bucket notification using MinIO client
### Step 2: Enable bucket notification using MinIO client
We will enable bucket event notification to trigger whenever a JPEG image is uploaded or deleted ``images`` bucket on ``myminio`` server. Here ARN value is ``arn:minio:sqs::1:mqtt``.
We will enable bucket event notification to trigger whenever a JPEG image is uploaded or deleted `images` bucket on `myminio` server. Here ARN value is `arn:minio:sqs::1:mqtt`.
The python program below waits on mqtt topic ``/minio`` and prints event notifications on the console. We use [paho-mqtt](https://pypi.python.org/pypi/paho-mqtt/) library to do this.
The python program below waits on mqtt topic `/minio` and prints event notifications on the console. We use [paho-mqtt](https://pypi.python.org/pypi/paho-mqtt/) library to do this.
```py
```py
#!/usr/bin/env python3
#!/usr/bin/env python3
@ -243,7 +251,7 @@ Execute this example python program to watch for MQTT events on the console.
python mqtt.py
python mqtt.py
```
```
Open another terminal and upload a JPEG image into ``images`` bucket.
Open another terminal and upload a JPEG image into `images` bucket.
@ -277,12 +286,12 @@ MinIO requires a 5.x series version of Elasticsearch. This is the latest major r
The MinIO server configuration file is stored on the backend in json format. The Elasticsearch configuration is located in the `elasticsearch` key under the `notify` top-level key. Create a configuration key-value pair here for your Elasticsearch instance. The key is a name for your Elasticsearch endpoint, and the value is a collection of key-value parameters described in the table below.
The MinIO server configuration file is stored on the backend in json format. The Elasticsearch configuration is located in the `elasticsearch` key under the `notify` top-level key. Create a configuration key-value pair here for your Elasticsearch instance. The key is a name for your Elasticsearch endpoint, and the value is a collection of key-value parameters described in the table below.
| `enable` | _bool_ | (Required) Is this server endpoint configuration active/enabled? |
| `enable`| _bool_ | (Required) Is this server endpoint configuration active/enabled? |
| `format` | _string_ | (Required) Either `namespace` or `access`. |
| `format`| _string_ | (Required) Either `namespace` or `access`. |
| `url` | _string_ | (Required) The Elasticsearch server's address, with optional authentication info. For example: `http://localhost:9200` or with authentication info `http://elastic:MagicWord@127.0.0.1:9200`. |
| `url`| _string_ | (Required) The Elasticsearch server's address, with optional authentication info. For example: `http://localhost:9200` or with authentication info `http://elastic:MagicWord@127.0.0.1:9200`. |
| `index` | _string_ | (Required) The name of an Elasticsearch index in which MinIO will store documents. |
| `index`| _string_ | (Required) The name of an Elasticsearch index in which MinIO will store documents. |
An example of Elasticsearch configuration is as follows:
An example of Elasticsearch configuration is as follows:
@ -300,10 +309,13 @@ An example of Elasticsearch configuration is as follows:
If Elasticsearch has authentication enabled, the credentials can be supplied to MinIO via the `url` parameter formatted as `PROTO://USERNAME:PASSWORD@ELASTICSEARCH_HOST:PORT`.
If Elasticsearch has authentication enabled, the credentials can be supplied to MinIO via the `url` parameter formatted as `PROTO://USERNAME:PASSWORD@ELASTICSEARCH_HOST:PORT`.
To update the configuration, use `mc admin config get` command to get the current configuration file for the minio deployment in json format, and save it locally.
To update the configuration, use `mc admin config get` command to get the current configuration file for the minio deployment in json format, and save it locally.
```sh
```sh
$ mc admin config get myminio/ > /tmp/myconfig
$ mc admin config get myminio/ > /tmp/myconfig
```
```
After updating the Elasticsearch configuration in /tmp/myconfig , use `mc admin config set` command to update the configuration for the deployment.Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:elasticsearch` at start-up if there were no errors.
After updating the Elasticsearch configuration in /tmp/myconfig , use `mc admin config set` command to update the configuration for the deployment.Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:elasticsearch` at start-up if there were no errors.
@ -408,6 +420,7 @@ This output shows that a document has been created for the event in Elasticsearc
Here we see that the document ID is the bucket and object name. In case `access` format was used, the document ID would be automatically generated by Elasticsearch.
Here we see that the document ID is the bucket and object name. In case `access` format was used, the document ID would be automatically generated by Elasticsearch.
<aname="Redis"></a>
<aname="Redis"></a>
## Publish MinIO events via Redis
## Publish MinIO events via Redis
Install [Redis](http://redis.io/download) server. For illustrative purposes, we have set the database password as "yoursecret".
Install [Redis](http://redis.io/download) server. For illustrative purposes, we have set the database password as "yoursecret".
@ -424,13 +437,13 @@ The steps below show how to use this notification target in `namespace` and `acc
The MinIO server configuration file is stored on the backend in json format.The Redis configuration is located in the `redis` key under the `notify` top-level key. Create a configuration key-value pair here for your Redis instance. The key is a name for your Redis endpoint, and the value is a collection of key-value parameters described in the table below.
The MinIO server configuration file is stored on the backend in json format.The Redis configuration is located in the `redis` key under the `notify` top-level key. Create a configuration key-value pair here for your Redis instance. The key is a name for your Redis endpoint, and the value is a collection of key-value parameters described in the table below.
| `key` | _string_ | (Required) The name of the redis key under which events are stored. A hash is used in case of `namespace` format and a list in case of `access` format.|
| `key`| _string_ | (Required) The name of the redis key under which events are stored. A hash is used in case of `namespace` format and a list in case of `access` format.|
An example of Redis configuration is as follows:
An example of Redis configuration is as follows:
@ -445,11 +458,15 @@ An example of Redis configuration is as follows:
}
}
}
}
```
```
To update the configuration, use `mc admin config get` command to get the current configuration file for the minio deployment in json format, and save it locally.
To update the configuration, use `mc admin config get` command to get the current configuration file for the minio deployment in json format, and save it locally.
```sh
```sh
$ mc admin config get myminio/ > /tmp/myconfig
$ mc admin config get myminio/ > /tmp/myconfig
```
```
After updating the Redis configuration in /tmp/myconfig , use `mc admin config set` command to update the configuration for the deployment.Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:redis` at start-up if there were no errors.
After updating the Redis configuration in /tmp/myconfig , use `mc admin config set` command to update the configuration for the deployment.Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:redis` at start-up if there were no errors.
```sh
```sh
$ mc admin config set myminio < /tmp/myconfig
$ mc admin config set myminio < /tmp/myconfig
```
```
@ -481,7 +498,7 @@ redis-cli -a yoursecret
OK
OK
```
```
Open another terminal and upload a JPEG image into ``images`` bucket.
Open another terminal and upload a JPEG image into `images` bucket.
```
```
mc cp myphoto.jpg myminio/images
mc cp myphoto.jpg myminio/images
@ -501,13 +518,14 @@ Here we see that MinIO performed `HSET` on `minio_events` key.
In case, `access` format was used, then `minio_events` would be a list, and the MinIO server would have performed an `RPUSH` to append to the list. A consumer of this list would ideally use `BLPOP` to remove list items from the left-end of the list.
In case, `access` format was used, then `minio_events` would be a list, and the MinIO server would have performed an `RPUSH` to append to the list. A consumer of this list would ideally use `BLPOP` to remove list items from the left-end of the list.
<aname="NATS"></a>
<aname="NATS"></a>
## Publish MinIO events via NATS
## Publish MinIO events via NATS
Install NATS from [here](http://nats.io/).
Install NATS from [here](http://nats.io/).
### Step 1: Add NATS endpoint to MinIO
### Step 1: Add NATS endpoint to MinIO
The NATS configuration block in ``config.json`` is as follows:
The NATS configuration block in `config.json` is as follows:
```
```
"nats": {
"nats": {
@ -530,14 +548,19 @@ The NATS configuration block in ``config.json`` is as follows:
}
}
},
},
```
```
To update the configuration, use `mc admin config get` command to get the current configuration file for the minio deployment in json format, and save it locally.
To update the configuration, use `mc admin config get` command to get the current configuration file for the minio deployment in json format, and save it locally.
```sh
```sh
$ mc admin config get myminio/ > /tmp/myconfig
$ mc admin config get myminio/ > /tmp/myconfig
```
```
After updating the NATS configuration in /tmp/myconfig , use `mc admin config set` command to update the configuration for the deployment.Restart MinIO server to reflect config changes. ``bucketevents`` is the subject used by NATS in this example.
After updating the NATS configuration in /tmp/myconfig , use `mc admin config set` command to update the configuration for the deployment.Restart MinIO server to reflect config changes. `bucketevents` is the subject used by NATS in this example.
```sh
```sh
$ mc admin config set myminio < /tmp/myconfig
$ mc admin config set myminio < /tmp/myconfig
```
```
MinIO server also supports [NATS Streaming mode](http://nats.io/documentation/streaming/nats-streaming-intro/) that offers additional functionality like `Message/event persistence`, `At-least-once-delivery`, and `Publisher rate limiting`. To configure MinIO server to send notifications to NATS Streaming server, update the MinIO server configuration file as follows:
MinIO server also supports [NATS Streaming mode](http://nats.io/documentation/streaming/nats-streaming-intro/) that offers additional functionality like `Message/event persistence`, `At-least-once-delivery`, and `Publisher rate limiting`. To configure MinIO server to send notifications to NATS Streaming server, update the MinIO server configuration file as follows:
```
```
@ -561,11 +584,12 @@ MinIO server also supports [NATS Streaming mode](http://nats.io/documentation/st
}
}
},
},
```
```
Read more about sections `clusterID`, `clientID` on [NATS documentation](https://github.com/nats-io/nats-streaming-server/blob/master/README.md). Section `maxPubAcksInflight` is explained [here](https://github.com/nats-io/go-nats-streaming#publisher-rate-limiting).
Read more about sections `clusterID`, `clientID` on [NATS documentation](https://github.com/nats-io/nats-streaming-server/blob/master/README.md). Section `maxPubAcksInflight` is explained [here](https://github.com/nats-io/go-nats-streaming#publisher-rate-limiting).
### Step 2: Enable bucket notification using MinIO client
### Step 2: Enable bucket notification using MinIO client
We will enable bucket event notification to trigger whenever a JPEG image is uploaded or deleted from ``images`` bucket on ``myminio`` server. Here ARN value is ``arn:minio:sqs::1:nats``. To understand more about ARN please follow [AWS ARN](http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) documentation.
We will enable bucket event notification to trigger whenever a JPEG image is uploaded or deleted from `images` bucket on `myminio` server. Here ARN value is `arn:minio:sqs::1:nats`. To understand more about ARN please follow [AWS ARN](http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) documentation.
```
```
mc mb myminio/images
mc mb myminio/images
@ -614,13 +638,13 @@ go run nats.go
2016/10/12 06:39:18 Subscribing to subject 'bucketevents'
2016/10/12 06:39:18 Subscribing to subject 'bucketevents'
```
```
Open another terminal and upload a JPEG image into ``images`` bucket.
Open another terminal and upload a JPEG image into `images` bucket.
```
```
mc cp myphoto.jpg myminio/images
mc cp myphoto.jpg myminio/images
```
```
The example ``nats.go`` program prints event notification to console.
The example `nats.go` program prints event notification to console.
```
```
go run nats.go
go run nats.go
@ -664,19 +688,21 @@ go run nats.go
2017/07/07 11:47:40 Connected
2017/07/07 11:47:40 Connected
2017/07/07 11:47:40 Subscribing to subject 'bucketevents'
2017/07/07 11:47:40 Subscribing to subject 'bucketevents'
```
```
Open another terminal and upload a JPEG image into ``images`` bucket.
Open another terminal and upload a JPEG image into `images` bucket.
```
```
mc cp myphoto.jpg myminio/images
mc cp myphoto.jpg myminio/images
```
```
The example ``nats.go`` program prints event notification to console.
The example `nats.go` program prints event notification to console.
```
```
Received a message: {"EventType":"s3:ObjectCreated:Put","Key":"images/myphoto.jpg","Records":[{"eventVersion":"2.0","eventSource":"minio:s3","awsRegion":"","eventTime":"2017-07-07T18:46:37Z","eventName":"s3:ObjectCreated:Put","userIdentity":{"principalId":"minio"},"requestParameters":{"sourceIPAddress":"192.168.1.80:55328"},"responseElements":{"x-amz-request-id":"14CF20BD1EFD5B93","x-minio-origin-endpoint":"http://127.0.0.1:9000"},"s3":{"s3SchemaVersion":"1.0","configurationId":"Config","bucket":{"name":"images","ownerIdentity":{"principalId":"minio"},"arn":"arn:aws:s3:::images"},"object":{"key":"myphoto.jpg","size":248682,"eTag":"f1671feacb8bbf7b0397c6e9364e8c92","contentType":"image/jpeg","userDefined":{"content-type":"image/jpeg"},"versionId":"1","sequencer":"14CF20BD1EFD5B93"}},"source":{"host":"192.168.1.80","port":"55328","userAgent":"MinIO (linux; amd64) minio-go/2.0.4 mc/DEVELOPMENT.GOGET"}}],"level":"info","msg":"","time":"2017-07-07T11:46:37-07:00"}
Received a message: {"EventType":"s3:ObjectCreated:Put","Key":"images/myphoto.jpg","Records":[{"eventVersion":"2.0","eventSource":"minio:s3","awsRegion":"","eventTime":"2017-07-07T18:46:37Z","eventName":"s3:ObjectCreated:Put","userIdentity":{"principalId":"minio"},"requestParameters":{"sourceIPAddress":"192.168.1.80:55328"},"responseElements":{"x-amz-request-id":"14CF20BD1EFD5B93","x-minio-origin-endpoint":"http://127.0.0.1:9000"},"s3":{"s3SchemaVersion":"1.0","configurationId":"Config","bucket":{"name":"images","ownerIdentity":{"principalId":"minio"},"arn":"arn:aws:s3:::images"},"object":{"key":"myphoto.jpg","size":248682,"eTag":"f1671feacb8bbf7b0397c6e9364e8c92","contentType":"image/jpeg","userDefined":{"content-type":"image/jpeg"},"versionId":"1","sequencer":"14CF20BD1EFD5B93"}},"source":{"host":"192.168.1.80","port":"55328","userAgent":"MinIO (linux; amd64) minio-go/2.0.4 mc/DEVELOPMENT.GOGET"}}],"level":"info","msg":"","time":"2017-07-07T11:46:37-07:00"}
```
```
<aname="PostgreSQL"></a>
<aname="PostgreSQL"></a>
## Publish MinIO events via PostgreSQL
## Publish MinIO events via PostgreSQL
Install [PostgreSQL](https://www.postgresql.org/) database server. For illustrative purposes, we have set the "postgres" user password as `password` and created a database called `minio_events` to store the events.
Install [PostgreSQL](https://www.postgresql.org/) database server. For illustrative purposes, we have set the "postgres" user password as `password` and created a database called `minio_events` to store the events.
@ -697,17 +723,17 @@ MinIO requires PostgreSQL version 9.5 or above. MinIO uses the [`INSERT ON CONFL
The MinIO server configuration file is stored on the backend in json format. The PostgreSQL configuration is located in the `postgresql` key under the `notify` top-level key. Create a configuration key-value pair here for your PostgreSQL instance. The key is a name for your PostgreSQL endpoint, and the value is a collection of key-value parameters described in the table below.
The MinIO server configuration file is stored on the backend in json format. The PostgreSQL configuration is located in the `postgresql` key under the `notify` top-level key. Create a configuration key-value pair here for your PostgreSQL instance. The key is a name for your PostgreSQL endpoint, and the value is a collection of key-value parameters described in the table below.
| `enable` | _bool_ | (Required) Is this server endpoint configuration active/enabled? |
| `enable`| _bool_ | (Required) Is this server endpoint configuration active/enabled? |
| `format` | _string_ | (Required) Either `namespace` or `access`. |
| `format`| _string_ | (Required) Either `namespace` or `access`. |
| `connectionString` | _string_ | (Optional) [Connection string parameters](https://godoc.org/github.com/lib/pq#hdr-Connection_String_Parameters) for the PostgreSQL server. Can be used to set `sslmode` for example. |
| `connectionString` | _string_ | (Optional) [Connection string parameters](https://godoc.org/github.com/lib/pq#hdr-Connection_String_Parameters) for the PostgreSQL server. Can be used to set `sslmode` for example. |
| `table` | _string_ | (Required) Table name in which events will be stored/updated. If the table does not exist, the MinIO server creates it at start-up.|
| `table`| _string_ | (Required) Table name in which events will be stored/updated. If the table does not exist, the MinIO server creates it at start-up.|
| `host` | _string_ | (Optional) Host name of the PostgreSQL server. Defaults to `localhost`|
| `host`| _string_ | (Optional) Host name of the PostgreSQL server. Defaults to `localhost`. IPv6 host should be enclosed with `[` and `]`|
| `port` | _string_ | (Optional) Port on which to connect to PostgreSQL server. Defaults to `5432`. |
| `port`| _string_ | (Optional) Port on which to connect to PostgreSQL server. Defaults to `5432`. |
| `user` | _string_ | (Optional) Database user name. Defaults to user running the server process. |
| `user`| _string_ | (Optional) Database user name. Defaults to user running the server process. |
An example of PostgreSQL configuration is as follows:
An example of PostgreSQL configuration is as follows:
@ -729,15 +755,18 @@ An example of PostgreSQL configuration is as follows:
Note that for illustration here, we have disabled SSL. In the interest of security, for production this is not recommended.
Note that for illustration here, we have disabled SSL. In the interest of security, for production this is not recommended.
To update the configuration, use `mc admin config get` command to get the current configuration file for the minio deployment in json format, and save it locally.
To update the configuration, use `mc admin config get` command to get the current configuration file for the minio deployment in json format, and save it locally.
```sh
```sh
$ mc admin config get myminio/ > /tmp/myconfig
$ mc admin config get myminio/ > /tmp/myconfig
```
```
After updating the Postgres configuration in /tmp/myconfig , use `mc admin config set` command to update the configuration for the deployment.Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:postgresql` at start-up if there were no errors.
After updating the Postgres configuration in /tmp/myconfig , use `mc admin config set` command to update the configuration for the deployment.Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:postgresql` at start-up if there were no errors.
```sh
```sh
$ mc admin config set myminio < /tmp/myconfig
$ mc admin config set myminio < /tmp/myconfig
```
```
Note that, you can add as many PostgreSQL server endpoint configurations as needed by providing an identifier (like "1" in the example above) for the PostgreSQL instance and an object of per-server configuration parameters.
Note that, you can add as many PostgreSQL server endpoint configurations as needed by providing an identifier (like "1" in the example above) for the PostgreSQL instance and an object of per-server configuration parameters.
### Step 3: Enable bucket notification using MinIO client
### Step 3: Enable bucket notification using MinIO client
Open another terminal and upload a JPEG image into ``images`` bucket.
Open another terminal and upload a JPEG image into `images` bucket.
```
```
mc cp myphoto.jpg myminio/images
mc cp myphoto.jpg myminio/images
@ -779,6 +808,7 @@ key | value
```
```
<aname="MySQL"></a>
<aname="MySQL"></a>
## Publish MinIO events via MySQL
## Publish MinIO events via MySQL
Install MySQL from [here](https://dev.mysql.com/downloads/mysql/). For illustrative purposes, we have set the root password as `password` and created a database called `miniodb` to store the events.
Install MySQL from [here](https://dev.mysql.com/downloads/mysql/). For illustrative purposes, we have set the root password as `password` and created a database called `miniodb` to store the events.
@ -799,17 +829,17 @@ MinIO requires MySQL version 5.7.8 or above. MinIO uses the [JSON](https://dev.m
The MinIO server configuration file is stored on the backend in json format. The MySQL configuration is located in the `mysql` key under the `notify` top-level key. Create a configuration key-value pair here for your MySQL instance. The key is a name for your MySQL endpoint, and the value is a collection of key-value parameters described in the table below.
The MinIO server configuration file is stored on the backend in json format. The MySQL configuration is located in the `mysql` key under the `notify` top-level key. Create a configuration key-value pair here for your MySQL instance. The key is a name for your MySQL endpoint, and the value is a collection of key-value parameters described in the table below.
| `enable` | _bool_ | (Required) Is this server endpoint configuration active/enabled? |
| `enable`| _bool_ | (Required) Is this server endpoint configuration active/enabled? |
| `format` | _string_ | (Required) Either `namespace` or `access`. |
| `format`| _string_ | (Required) Either `namespace` or `access`. |
| `dsnString` | _string_ | (Optional) [Data-Source-Name connection string](https://github.com/go-sql-driver/mysql#dsn-data-source-name) for the MySQL server. If not specified, the connection information specified by the `host`, `port`, `user`, `password` and `database` parameters are used. |
| `dsnString` | _string_ | (Optional) [Data-Source-Name connection string](https://github.com/go-sql-driver/mysql#dsn-data-source-name) for the MySQL server. If not specified, the connection information specified by the `host`, `port`, `user`, `password` and `database` parameters are used. |
| `table` | _string_ | (Required) Table name in which events will be stored/updated. If the table does not exist, the MinIO server creates it at start-up.|
| `table`| _string_ | (Required) Table name in which events will be stored/updated. If the table does not exist, the MinIO server creates it at start-up.|
| `host` | _string_ | Host name of the MySQL server (used only if `dsnString` is empty). |
| `host`| _string_ | Host name of the MySQL server (used only if `dsnString` is empty). |
| `port` | _string_ | Port on which to connect to the MySQL server (used only if `dsnString` is empty). |
| `port`| _string_ | Port on which to connect to the MySQL server (used only if `dsnString` is empty). |
| `user` | _string_ | Database user-name (used only if `dsnString` is empty). |
| `user`| _string_ | Database user-name (used only if `dsnString` is empty). |
| `password` | _string_ | Database password (used only if `dsnString` is empty). |
| `password`| _string_ | Database password (used only if `dsnString` is empty). |
| `database` | _string_ | Database name (used only if `dsnString` is empty). |
| `database`| _string_ | Database name (used only if `dsnString` is empty). |
An example of MySQL configuration is as follows:
An example of MySQL configuration is as follows:
@ -827,16 +857,20 @@ An example of MySQL configuration is as follows:
}
}
}
}
```
```
To update the configuration, use `mc admin config get` command to get the current configuration file for the minio deployment in json format, and save it locally.
To update the configuration, use `mc admin config get` command to get the current configuration file for the minio deployment in json format, and save it locally.
```sh
```sh
$ mc admin config get myminio/ > /tmp/myconfig
$ mc admin config get myminio/ > /tmp/myconfig
```
```
After updating the MySQL configuration in /tmp/myconfig , use `mc admin config set` command to update the configuration for the deployment.Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:mysql` at start-up if there were no errors.
After updating the MySQL configuration in /tmp/myconfig , use `mc admin config set` command to update the configuration for the deployment.Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:mysql` at start-up if there were no errors.
```sh
```sh
$ mc admin config set myminio < /tmp/myconfig
$ mc admin config set myminio < /tmp/myconfig
```
```
Note that, you can add as many MySQL server endpoint configurations as needed by providing an identifier (like "1" in the example above) for the MySQL instance and an object of per-server configuration parameters.
Note that, you can add as many MySQL server endpoint configurations as needed by providing an identifier (like "1" in the example above) for the MySQL instance and an object of per-server configuration parameters.
### Step 3: Enable bucket notification using MinIO client
### Step 3: Enable bucket notification using MinIO client
Open another terminal and upload a JPEG image into ``images`` bucket:
Open another terminal and upload a JPEG image into `images` bucket:
```
```
mc cp myphoto.jpg myminio/images
mc cp myphoto.jpg myminio/images
@ -879,6 +913,7 @@ mysql> select * from minio_images;
```
```
<aname="apache-kafka"></a>
<aname="apache-kafka"></a>
## Publish MinIO events via Kafka
## Publish MinIO events via Kafka
Install Apache Kafka from [here](http://kafka.apache.org/).
Install Apache Kafka from [here](http://kafka.apache.org/).
@ -889,7 +924,7 @@ MinIO requires Kafka version 0.10 or 0.9. Internally MinIO uses the [Shopify/sar
### Step 2: Add Kafka endpoint to MinIO
### Step 2: Add Kafka endpoint to MinIO
The MinIO server configuration file is stored on the backend in json format. Update the kafka configuration block in ``config.json`` as follows:
The MinIO server configuration file is stored on the backend in json format. Update the kafka configuration block in `config.json` as follows:
```
```
"kafka": {
"kafka": {
@ -900,17 +935,22 @@ The MinIO server configuration file is stored on the backend in json format. Upd
}
}
}
}
```
```
To update the configuration, use `mc admin config get` command to get the current configuration file for the minio deployment in json format, and save it locally.
To update the configuration, use `mc admin config get` command to get the current configuration file for the minio deployment in json format, and save it locally.
```sh
```sh
$ mc admin config get myminio/ > /tmp/myconfig
$ mc admin config get myminio/ > /tmp/myconfig
```
```
After updating the Kafka configuration in /tmp/myconfig , use `mc admin config set` command to update the configuration for the deployment.Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:kafka` at start-up if there were no errors.``bucketevents`` is the topic used by kafka in this example.
After updating the Kafka configuration in /tmp/myconfig , use `mc admin config set` command to update the configuration for the deployment.Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:kafka` at start-up if there were no errors.`bucketevents` is the topic used by kafka in this example.
```sh
```sh
$ mc admin config set myminio < /tmp/myconfig
$ mc admin config set myminio < /tmp/myconfig
```
```
### Step 3: Enable bucket notification using MinIO client
### Step 3: Enable bucket notification using MinIO client
We will enable bucket event notification to trigger whenever a JPEG image is uploaded or deleted from ``images`` bucket on ``myminio`` server. Here ARN value is ``arn:minio:sqs::1:kafka``. To understand more about ARN please follow [AWS ARN](http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) documentation.
We will enable bucket event notification to trigger whenever a JPEG image is uploaded or deleted from `images` bucket on `myminio` server. Here ARN value is `arn:minio:sqs::1:kafka`. To understand more about ARN please follow [AWS ARN](http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) documentation.
```
```
mc mb myminio/images
mc mb myminio/images
@ -927,13 +967,13 @@ We used [kafkacat](https://github.com/edenhill/kafkacat) to print all notificati
kafkacat -C -b localhost:9092 -t bucketevents
kafkacat -C -b localhost:9092 -t bucketevents
```
```
Open another terminal and upload a JPEG image into ``images`` bucket.
Open another terminal and upload a JPEG image into `images` bucket.
```
```
mc cp myphoto.jpg myminio/images
mc cp myphoto.jpg myminio/images
```
```
``kafkacat`` prints the event notification to the console.
`kafkacat` prints the event notification to the console.
[Webhooks](https://en.wikipedia.org/wiki/Webhook) are a way to receive information when it happens, rather than continually polling for that data.
[Webhooks](https://en.wikipedia.org/wiki/Webhook) are a way to receive information when it happens, rather than continually polling for that data.
### Step 1: Add Webhook endpoint to MinIO
### Step 1: Add Webhook endpoint to MinIO
The MinIO server configuration file is stored on the backend in json format. Update the Webhook configuration block in ``config.json`` as follows
The MinIO server configuration file is stored on the backend in json format. Update the Webhook configuration block in `config.json` as follows
```
```
"webhook": {
"webhook": {
@ -956,17 +997,22 @@ The MinIO server configuration file is stored on the backend in json format. Upd
"endpoint": "http://localhost:3000/"
"endpoint": "http://localhost:3000/"
}
}
```
```
To update the configuration, use `mc admin config get` command to get the current configuration file for the minio deployment in json format, and save it locally.
To update the configuration, use `mc admin config get` command to get the current configuration file for the minio deployment in json format, and save it locally.
```sh
```sh
$ mc admin config get myminio/ > /tmp/myconfig
$ mc admin config get myminio/ > /tmp/myconfig
```
```
After updating the webhook configuration in /tmp/myconfig , use `mc admin config set` command to update the configuration for the deployment.Here the endpoint is the server listening for webhook notifications. Save the file and restart the MinIO server for changes to take effect. Note that the endpoint needs to be live and reachable when you restart your MinIO server.
After updating the webhook configuration in /tmp/myconfig , use `mc admin config set` command to update the configuration for the deployment.Here the endpoint is the server listening for webhook notifications. Save the file and restart the MinIO server for changes to take effect. Note that the endpoint needs to be live and reachable when you restart your MinIO server.
```sh
```sh
$ mc admin config set myminio < /tmp/myconfig
$ mc admin config set myminio < /tmp/myconfig
```
```
### Step 2: Enable bucket notification using MinIO client
### Step 2: Enable bucket notification using MinIO client
We will enable bucket event notification to trigger whenever a JPEG image is uploaded to ``images`` bucket on ``myminio`` server. Here ARN value is ``arn:minio:sqs::1:webhook``. To learn more about ARN please follow [AWS ARN](http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) documentation.
We will enable bucket event notification to trigger whenever a JPEG image is uploaded to `images` bucket on `myminio` server. Here ARN value is `arn:minio:sqs::1:webhook`. To learn more about ARN please follow [AWS ARN](http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) documentation.
Then open the Thumbnailer config file at ``config/webhook.json`` and add the configuration for your MinIO server and then start Thumbnailer by
Then open the Thumbnailer config file at `config/webhook.json` and add the configuration for your MinIO server and then start Thumbnailer by
```
```
NODE_ENV=webhook node thumbnail-webhook.js
NODE_ENV=webhook node thumbnail-webhook.js
```
```
Thumbnailer starts running at ``http://localhost:3000/``. Next, configure the MinIO server to send notifications to this URL (as mentioned in step 1) and use ``mc`` to set up bucket notifications (as mentioned in step 2). Then upload a JPEG image to MinIO server by
Thumbnailer starts running at `http://localhost:3000/`. Next, configure the MinIO server to send notifications to this URL (as mentioned in step 1) and use `mc` to set up bucket notifications (as mentioned in step 2). Then upload a JPEG image to MinIO server by
Wait a few moments, then check the bucket’s contents with mc ls—you will see a thumbnail appear.
Wait a few moments, then check the bucket’s contents with mc ls—you will see a thumbnail appear.
```
```
@ -1015,6 +1062,7 @@ mc ls myminio/images-thumbnail
```
```
<aname="NSQ"></a>
<aname="NSQ"></a>
## Publish MinIO events to NSQ
## Publish MinIO events to NSQ
Install an NSQ Daemon from [here](https://nsq.io/). Or use the following Docker
Install an NSQ Daemon from [here](https://nsq.io/). Or use the following Docker
@ -1043,21 +1091,24 @@ An example configuration for NSQ is shown below:
}
}
}
}
```
```
To update the configuration, use `mc admin config get` command to get the current configuration file for the minio deployment in json format, and save it locally.
To update the configuration, use `mc admin config get` command to get the current configuration file for the minio deployment in json format, and save it locally.
```sh
```sh
$ mc admin config get myminio/ > /tmp/myconfig
$ mc admin config get myminio/ > /tmp/myconfig
```
```
After updating the NSQ configuration in /tmp/myconfig , use `mc admin config set` command to update the configuration for the deployment.Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:nsq` at start-up if there were no errors.
After updating the NSQ configuration in /tmp/myconfig , use `mc admin config set` command to update the configuration for the deployment.Restart the MinIO server to put the changes into effect. The server will print a line like `SQS ARNs: arn:minio:sqs::1:nsq` at start-up if there were no errors.
```sh
```sh
$ mc admin config set myminio < /tmp/myconfig
$ mc admin config set myminio < /tmp/myconfig
```
```
Note that, you can add as many NSQ daemon endpoint configurations as needed by providing an identifier (like "1" in the example above) for the NSQ instance and an object of per-server configuration parameters.
Note that, you can add as many NSQ daemon endpoint configurations as needed by providing an identifier (like "1" in the example above) for the NSQ instance and an object of per-server configuration parameters.
### Step 2: Enable bucket notification using MinIO client
### Step 2: Enable bucket notification using MinIO client
We will enable bucket event notification to trigger whenever a JPEG image is uploaded or deleted ``images`` bucket on ``myminio`` server. Here ARN value is ``arn:minio:sqs::1:nsq``.
We will enable bucket event notification to trigger whenever a JPEG image is uploaded or deleted `images` bucket on `myminio` server. Here ARN value is `arn:minio:sqs::1:nsq`.
```
```
mc mb myminio/images
mc mb myminio/images
@ -1074,7 +1125,7 @@ The simplest test is to download `nsq_tail` from [nsq github](https://github.com
_NOTE_ If you are running [distributed MinIO](https://docs.min.io/docs/distributed-minio-quickstart-guide), modify `~/.minio/config.json` on all the nodes with your bucket event notification backend configuration.
*NOTE* If you are running [distributed MinIO](https://docs.min.io/docs/distributed-minio-quickstart-guide), modify ``~/.minio/config.json`` on all the nodes with your bucket event notification backend configuration.
Praveen raj Mani
6 years ago
committed by