diff --git a/docs/bucket-notifications/README.md b/docs/bucket/notifications/README.md similarity index 80% rename from docs/bucket-notifications/README.md rename to docs/bucket/notifications/README.md index 6d3fab028..20539dbad 100644 --- a/docs/bucket-notifications/README.md +++ b/docs/bucket/notifications/README.md @@ -1,50 +1,49 @@ -[![Slack](https://slack.minio.io/slack?type=svg)](https://slack.minio.io) +# Minio Bucket Notification Reference Guide [![Slack](https://slack.minio.io/slack?type=svg)](https://slack.minio.io) -Minio server supports Amazon S3 compatible bucket event notification for the following targets [AMQP](https://www.amqp.org/about/what), [Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started.html) , [Redis](http://redis.io/documentation), [nats.io](http://nats.io/), [PostgreSQL](https://www.postgresql.org) and [kafka](https://kafka.apache.org/). +Minio server supports Amazon S3 compatible bucket event notification for the following targets -## Prerequisites +| Notification Targets| +|:---| +| [`AMQP`](#AMQP) | +| [`Elasticsearch`](#Elasticsearch) | +| [`Redis`](#Redis) | +| [`NATS`](#NATS) | +| [`PostgreSQL`](#PostgreSQL) | +| [`Apache Kafka`](#apache-kafka) | + +## Prerequisites * Install and configure Minio Server from [here](http://docs.minio.io/docs/minio). * Install and configure Minio Client from [here](https://docs.minio.io/docs/minio-client-quickstart-guide). -| Notification Targets| -|:-- -| [`RabbitMQ`](#RabbitMQ) | -| [`Elasticsearch`](#Elasticsearch) | -| [`Redis`](#Redis) | -| [`NATS`](#NATS) | -| [`PostgreSQL`](#PostgreSQL) | -| [`kafka`](#kafka) | - - -## Publish Minio events via RabbitMQ + +## Publish Minio events via AMQP Install RabbitMQ from [here](https://www.rabbitmq.com/). -### Recipe steps +### Step 1: Add AMQP endpoint to Minio -### Step 1: Add RabbitMQ endpoint to Minio +The default location of Minio server configuration file is ``~/.minio/config.json``. Update the AMQP configuration block in ``config.json`` as follows: -The default location of Minio server configuration file is ``~/.minio/config.json``. Update the RabbitMQ configuration block in ``config.json`` as follows: - -``` +```json "amqp": { - "1": { - "enable": true, - "url": "amqp://myuser:mypassword@localhost:5672", - "exchange": "bucketevents", - "routingKey": "bucketlogs", - "exchangeType": "fanout", - "mandatory": false, - "immediate": false, - "durable": false, - "internal": false, - "noWait": false, - "autoDeleted": false - } - } -``` -Restart Minio server to reflect config changes. Minio supports all the exchange available in [RabbitMQ](https://www.rabbitmq.com/). For this setup, we are using ``fanout`` exchange. + "1": { + "enable": true, + "url": "amqp://myuser:mypassword@localhost:5672", + "exchange": "bucketevents", + "routingKey": "bucketlogs", + "exchangeType": "fanout", + "mandatory": false, + "immediate": false, + "durable": false, + "internal": false, + "noWait": false, + "autoDeleted": false + } +} +``` + +Restart Minio server to reflect config changes. Minio supports all the exchanges available in [RabbitMQ](https://www.rabbitmq.com/). For this setup, we are using ``fanout`` exchange. ### Step 2: Enable bucket notification using Minio client @@ -52,14 +51,14 @@ We will enable bucket event notification to trigger whenever a JPEG image is upl ``` mc mb myminio/images -mc events add myminio/images arn:minio:sqs:us-east-1:1:amqp --suffix .jpg +mc events add myminio/images arn:minio:sqs:us-east-1:1:amqp --suffix .jpg mc events list myminio/images arn:minio:sqs:us-east-1:1:amqp s3:ObjectCreated:*,s3:ObjectRemoved:* Filter: suffix=”.jpg” ``` ### Step 3: Test on RabbitMQ -The python program below waits on the queue exchange ``bucketevents`` and prints event notifications on the console. We use [Pika Python Client](https://www.rabbitmq.com/tutorials/tutorial-three-python.html) library to do this. +The python program below waits on the queue exchange ``bucketevents`` and prints event notifications on the console. We use [Pika Python Client](https://www.rabbitmq.com/tutorials/tutorial-three-python.html) library to do this. ```py #!/usr/bin/env python @@ -108,8 +107,9 @@ You should receive the following event notification via RabbitMQ once the upload python rabbit.py ‘{“Records”:[{“eventVersion”:”2.0",”eventSource”:”aws:s3",”awsRegion”:”us-east-1",”eventTime”:”2016–09–08T22:34:38.226Z”,”eventName”:”s3:ObjectCreated:Put”,”userIdentity”:{“principalId”:”minio”},”requestParameters”:{“sourceIPAddress”:”10.1.10.150:44576"},”responseElements”:{},”s3":{“s3SchemaVersion”:”1.0",”configurationId”:”Config”,”bucket”:{“name”:”images”,”ownerIdentity”:{“principalId”:”minio”},”arn”:”arn:aws:s3:::images”},”object”:{“key”:”myphoto.jpg”,”size”:200436,”sequencer”:”147279EAF9F40933"}}}],”level”:”info”,”msg”:””,”time”:”2016–09–08T15:34:38–07:00"}\n ``` + -## Publish Minio events via Elasticsearch +## Publish Minio events via Elasticsearch Install Elasticsearch 2.4 from [here](https://www.elastic.co/downloads/past-releases/elasticsearch-2-4-0). @@ -119,15 +119,16 @@ Install Elasticsearch 2.4 from [here](https://www.elastic.co/downloads/past-rele The default location of Minio server configuration file is ``~/.minio/config.json``. Update the Elasticsearch configuration block in ``config.json`` as follows: -``` +```json "elasticsearch": { - "1": { - "enable": true, - "url": "http://127.0.0.1:9200", - "index": "bucketevents" - } - }, + "1": { + "enable": true, + "url": "http://127.0.0.1:9200", + "index": "bucketevents" + } +}, ``` + Restart Minio server to reflect config changes. ``bucketevents`` is the index used by Elasticsearch. ### Step 2: Enable bucket notification using Minio client @@ -213,6 +214,7 @@ curl -XGET '127.0.0.1:9200/bucketevents/_search?pretty=1' } } ``` + ``curl`` output above states that an Elasticsearch index has been successfully created with notification contents. @@ -220,22 +222,21 @@ curl -XGET '127.0.0.1:9200/bucketevents/_search?pretty=1' Install Redis from [here](http://redis.io/download). -## Recipe steps - ### Step 1: Add Redis endpoint to Minio The default location of Minio server configuration file is ``~/.minio/config.json``. Update the Redis configuration block in ``config.json`` as follows: -``` +```json "redis": { - "1": { - "enable": true, - "address": "127.0.0.1:6379", - "password": "yoursecret", - "key": "bucketevents" - } - } + "1": { + "enable": true, + "address": "127.0.0.1:6379", + "password": "yoursecret", + "key": "bucketevents" + } +} ``` + Restart Minio server to reflect config changes. ``bucketevents`` is the key used by Redis in this example. ### Step 2: Enable bucket notification using Minio client @@ -272,12 +273,11 @@ OK 1474321638.556108 [0 127.0.0.1:40190] "AUTH" "yoursecret" 1474321638.556477 [0 127.0.0.1:40190] "RPUSH" "bucketevents" "{\"Records\":[{\"eventVersion\":\"2.0\",\"eventSource\":\"aws:s3\",\"awsRegion\":\"us-east-1\",\"eventTime\":\"2016-09-19T21:47:18.555Z\",\"eventName\":\"s3:ObjectCreated:Put\",\"userIdentity\":{\"principalId\":\"minio\"},\"requestParameters\":{\"sourceIPAddress\":\"[::1]:39250\"},\"responseElements\":{},\"s3\":{\"s3SchemaVersion\":\"1.0\",\"configurationId\":\"Config\",\"bucket\":{\"name\":\"images\",\"ownerIdentity\":{\"principalId\":\"minio\"},\"arn\":\"arn:aws:s3:::images\"},\"object\":{\"key\":\"myphoto.jpg\",\"size\":23745,\"sequencer\":\"1475D7B80ECBD853\"}}}],\"level\":\"info\",\"msg\":\"\",\"time\":\"2016-09-19T14:47:18-07:00\"}\n" ``` - -## Publish Minio events via NATS -Install NATS from [here](http://nats.io/). + +## Publish Minio events via NATS -## Recipe steps +Install NATS from [here](http://nats.io/). ### Step 1: Add NATS endpoint to Minio @@ -285,18 +285,19 @@ The default location of Minio server configuration file is ``~/.minio/config.jso ``` "nats": { - "1": { - "enable": true, - "address": "0.0.0.0:4222", - "subject": "bucketevents", - "username": "yourusername", - "password": "yoursecret", - "token": "", - "secure": false, - "pingInterval": 0 - } - }, + "1": { + "enable": true, + "address": "0.0.0.0:4222", + "subject": "bucketevents", + "username": "yourusername", + "password": "yoursecret", + "token": "", + "secure": false, + "pingInterval": 0 + } +}, ``` + Restart Minio server to reflect config changes. ``bucketevents`` is the subject used by NATS in this example. ### Step 2: Enable bucket notification using Minio client @@ -319,27 +320,28 @@ package main // Import Go and NATS packages import ( - "runtime" - "log" - "github.com/nats-io/nats" + "log" + "runtime" + + "github.com/nats-io/nats" ) func main() { - // Create server connection - natsConnection, _ := nats.Connect("nats://yourusername:yoursecret@localhost:4222") - log.Println("Connected") + // Create server connection + natsConnection, _ := nats.Connect("nats://yourusername:yoursecret@localhost:4222") + log.Println("Connected") - // Subscribe to subject - log.Printf("Subscribing to subject 'bucketevents'\n") - natsConnection.Subscribe("bucketevents", func(msg *nats.Msg) { + // Subscribe to subject + log.Printf("Subscribing to subject 'bucketevents'\n") + natsConnection.Subscribe("bucketevents", func(msg *nats.Msg) { - // Handle the message - log.Printf("Received message '%s\n", string(msg.Data) + "'") - }) + // Handle the message + log.Printf("Received message '%s\n", string(msg.Data)+"'") + }) - // Keep the connection alive - runtime.Goexit() + // Keep the connection alive + runtime.Goexit() } ``` @@ -358,7 +360,7 @@ mc cp myphoto.jpg myminio/images The example ``nats.go`` program prints event notification to console. ``` -go run nats.go +go run nats.go 2016/10/12 06:51:26 Connected 2016/10/12 06:51:26 Subscribing to subject 'bucketevents' 2016/10/12 06:51:33 Received message '{"EventType":"s3:ObjectCreated:Put","Key":"images/myphoto.jpg","Records":[{"eventVersion":"2.0","eventSource":"aws:s3","awsRegion":"us-east-1","eventTime":"2016-10-12T13:51:33Z","eventName":"s3:ObjectCreated:Put","userIdentity":{"principalId":"minio"},"requestParameters":{"sourceIPAddress":"[::1]:57106"},"responseElements":{},"s3":{"s3SchemaVersion":"1.0","configurationId":"Config","bucket":{"name":"images","ownerIdentity":{"principalId":"minio"},"arn":"arn:aws:s3:::images"},"object":{"key":"myphoto.jpg","size":56060,"eTag":"1d97bf45ecb37f7a7b699418070df08f","sequencer":"147CCD1AE054BFD0"}}}],"level":"info","msg":"","time":"2016-10-12T06:51:33-07:00"} @@ -369,26 +371,25 @@ go run nats.go Install PostgreSQL from [here](https://www.postgresql.org/). -## Recipe steps - ### Step 1: Add PostgreSQL endpoint to Minio The default location of Minio server configuration file is ``~/.minio/config.json``. Update the PostgreSQL configuration block in ``config.json`` as follows: ``` "postgresql": { - "1": { - "enable": true, - "connectionString": "", - "table": "bucketevents", - "host": "127.0.0.1", - "port": "5432", - "user": "postgres", - "password": "mypassword", - "database": "bucketevents_db" - } - } + "1": { + "enable": true, + "connectionString": "", + "table": "bucketevents", + "host": "127.0.0.1", + "port": "5432", + "user": "postgres", + "password": "mypassword", + "database": "bucketevents_db" + } +} ``` + Restart Minio server to reflect config changes. ``bucketevents`` is the database table used by PostgreSQL in this example. ### Step 2: Enable bucket notification using Minio client @@ -415,32 +416,31 @@ Open PostgreSQL terminal to list the saved event notification logs. ``` bucketevents_db=# select * from bucketevents; -key | value +key | value --------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- images/myphoto.jpg | {"Records": [{"s3": {"bucket": {"arn": "arn:aws:s3:::images", "name": "images", "ownerIdentity": {"principalId": "minio"}}, "object": {"key": "myphoto.jpg", "eTag": "1d97bf45ecb37f7a7b699418070df08f", "size": 56060, "sequencer": "147CE57C70B31931"}, "configurationId": "Config", "s3SchemaVersion": "1.0"}, "awsRegion": "us-east-1", "eventName": "s3:ObjectCreated:Put", "eventTime": "2016-10-12T21:18:20Z", "eventSource": "aws:s3", "eventVersion": "2.0", "userIdentity": {"principalId": "minio"}, "responseElements": {}, "requestParameters": {"sourceIPAddress": "[::1]:39706"}}]} (1 row) -``` +``` - + ## Publish Minio events via kafka Install kafka from [here](http://kafka.apache.org/). -## Recipe steps - ### Step 1: Add kafka endpoint to Minio The default location of Minio server configuration file is ``~/.minio/config.json``. Update the kafka configuration block in ``config.json`` as follows: ``` "kafka": { - "1": { - "enable": true, - "brokers": ["localhost:9092"], - "topic": "bucketevents" - } + "1": { + "enable": true, + "brokers": ["localhost:9092"], + "topic": "bucketevents" + } } ``` + Restart Minio server to reflect config changes. ``bucketevents`` is the topic used by kafka in this example. ### Step 2: Enable bucket notification using Minio client