From 4dea4f3b891494cb07ace6fc6c9051435df8e07b Mon Sep 17 00:00:00 2001 From: koolhead17 Date: Wed, 1 Feb 2017 06:34:36 +0530 Subject: [PATCH] docs: added event-notification back end doc. (#3510) --- docs/bucket-notifications/README.md | 477 ++++++++++++++++++++++++++++ 1 file changed, 477 insertions(+) create mode 100644 docs/bucket-notifications/README.md diff --git a/docs/bucket-notifications/README.md b/docs/bucket-notifications/README.md new file mode 100644 index 000000000..6d3fab028 --- /dev/null +++ b/docs/bucket-notifications/README.md @@ -0,0 +1,477 @@ +[![Slack](https://slack.minio.io/slack?type=svg)](https://slack.minio.io) + +Minio server supports Amazon S3 compatible bucket event notification for the following targets [AMQP](https://www.amqp.org/about/what), [Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started.html) , [Redis](http://redis.io/documentation), [nats.io](http://nats.io/), [PostgreSQL](https://www.postgresql.org) and [kafka](https://kafka.apache.org/). + +## Prerequisites + +* Install and configure Minio Server from [here](http://docs.minio.io/docs/minio). +* Install and configure Minio Client from [here](https://docs.minio.io/docs/minio-client-quickstart-guide). + +| Notification Targets| +|:-- +| [`RabbitMQ`](#RabbitMQ) | +| [`Elasticsearch`](#Elasticsearch) | +| [`Redis`](#Redis) | +| [`NATS`](#NATS) | +| [`PostgreSQL`](#PostgreSQL) | +| [`kafka`](#kafka) | + + +## Publish Minio events via RabbitMQ + +Install RabbitMQ from [here](https://www.rabbitmq.com/). + +### Recipe steps + +### Step 1: Add RabbitMQ endpoint to Minio + +The default location of Minio server configuration file is ``~/.minio/config.json``. Update the RabbitMQ configuration block in ``config.json`` as follows: + +``` +"amqp": { + "1": { + "enable": true, + "url": "amqp://myuser:mypassword@localhost:5672", + "exchange": "bucketevents", + "routingKey": "bucketlogs", + "exchangeType": "fanout", + "mandatory": false, + "immediate": false, + "durable": false, + "internal": false, + "noWait": false, + "autoDeleted": false + } + } +``` +Restart Minio server to reflect config changes. Minio supports all the exchange available in [RabbitMQ](https://www.rabbitmq.com/). For this setup, we are using ``fanout`` exchange. + +### Step 2: Enable bucket notification using Minio client + +We will enable bucket event notification to trigger whenever a JPEG image is uploaded or deleted ``images`` bucket on ``myminio`` server. Here ARN value is ``arn:minio:sqs:us-east-1:1:amqp``. To understand more about ARN please follow [AWS ARN](http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) documentation. + +``` +mc mb myminio/images +mc events add myminio/images arn:minio:sqs:us-east-1:1:amqp --suffix .jpg +mc events list myminio/images +arn:minio:sqs:us-east-1:1:amqp s3:ObjectCreated:*,s3:ObjectRemoved:* Filter: suffix=”.jpg” +``` + +### Step 3: Test on RabbitMQ + +The python program below waits on the queue exchange ``bucketevents`` and prints event notifications on the console. We use [Pika Python Client](https://www.rabbitmq.com/tutorials/tutorial-three-python.html) library to do this. + +```py +#!/usr/bin/env python +import pika + +connection = pika.BlockingConnection(pika.ConnectionParameters( + host='localhost')) +channel = connection.channel() + +channel.exchange_declare(exchange='bucketevents', + type='fanout') + +result = channel.queue_declare(exclusive=False) +queue_name = result.method.queue + +channel.queue_bind(exchange='bucketevents', + queue=queue_name) + +print(' [*] Waiting for logs. To exit press CTRL+C') + +def callback(ch, method, properties, body): + print(" [x] %r" % body) + +channel.basic_consume(callback, + queue=queue_name, + no_ack=False) + +channel.start_consuming() +``` + +Execute this example python program to watch for RabbitMQ events on the console. + +```py +python rabbit.py +``` + +Open another terminal and upload a JPEG image into ``images`` bucket. + +``` +mc cp myphoto.jpg myminio/images +``` + +You should receive the following event notification via RabbitMQ once the upload completes. + +```py +python rabbit.py +‘{“Records”:[{“eventVersion”:”2.0",”eventSource”:”aws:s3",”awsRegion”:”us-east-1",”eventTime”:”2016–09–08T22:34:38.226Z”,”eventName”:”s3:ObjectCreated:Put”,”userIdentity”:{“principalId”:”minio”},”requestParameters”:{“sourceIPAddress”:”10.1.10.150:44576"},”responseElements”:{},”s3":{“s3SchemaVersion”:”1.0",”configurationId”:”Config”,”bucket”:{“name”:”images”,”ownerIdentity”:{“principalId”:”minio”},”arn”:”arn:aws:s3:::images”},”object”:{“key”:”myphoto.jpg”,”size”:200436,”sequencer”:”147279EAF9F40933"}}}],”level”:”info”,”msg”:””,”time”:”2016–09–08T15:34:38–07:00"}\n +``` + +## Publish Minio events via Elasticsearch + +Install Elasticsearch 2.4 from [here](https://www.elastic.co/downloads/past-releases/elasticsearch-2-4-0). + +## Recipe steps + +### Step 1: Add Elasticsearch endpoint to Minio + +The default location of Minio server configuration file is ``~/.minio/config.json``. Update the Elasticsearch configuration block in ``config.json`` as follows: + +``` +"elasticsearch": { + "1": { + "enable": true, + "url": "http://127.0.0.1:9200", + "index": "bucketevents" + } + }, +``` +Restart Minio server to reflect config changes. ``bucketevents`` is the index used by Elasticsearch. + +### Step 2: Enable bucket notification using Minio client + +We will enable bucket event notification to trigger whenever a JPEG image is uploaded or deleted from ``images`` bucket on ``myminio`` server. Here ARN value is ``arn:minio:sqs:us-east-1:1:elasticsearch``. To understand more about ARN please follow [AWS ARN](http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) documentation. + +``` +mc mb myminio/images +mc events add myminio/images arn:minio:sqs:us-east-1:1:elasticsearch --suffix .jpg +mc events list myminio/images +arn:minio:sqs:us-east-1:1:elasticsearch s3:ObjectCreated:*,s3:ObjectRemoved:* Filter: suffix=”.jpg” +``` + +### Step 3: Test on Elasticsearch + +Upload a JPEG image into ``images`` bucket, this is the bucket which has been configured for event notification. + +``` +mc cp myphoto.jpg myminio/images +``` + +Run ``curl`` to see new index name ``bucketevents`` in your Elasticsearch setup. + +``` +curl -XGET '127.0.0.1:9200/_cat/indices?v' +health status index pri rep docs.count docs.deleted store.size pri.store.size +yellow open bucketevents 5 1 1 0 7.8kb 7.8kb +``` + +Use curl to view contents of ``bucketevents`` index. + +``` +curl -XGET '127.0.0.1:9200/bucketevents/_search?pretty=1' +{ + "took" : 3, + "timed_out" : false, + "_shards" : { + "total" : 5, + "successful" : 5, + "failed" : 0 + }, + "hits" : { + "total" : 1, + "max_score" : 1.0, + "hits" : [ { + "_index" : "bucketevents", + "_type" : "event", + "_id" : "AVcRVOlwe-uNB1tfj6bx", + "_score" : 1.0, + "_source" : { + "Records" : [ { + "eventVersion" : "2.0", + "eventSource" : "aws:s3", + "awsRegion" : "us-east-1", + "eventTime" : "2016-09-09T23:42:39.977Z", + "eventName" : "s3:ObjectCreated:Put", + "userIdentity" : { + "principalId" : "minio" + }, + "requestParameters" : { + "sourceIPAddress" : "10.1.10.150:52140" + }, + "responseElements" : { }, + "s3" : { + "s3SchemaVersion" : "1.0", + "configurationId" : "Config", + "bucket" : { + "name" : "images", + "ownerIdentity" : { + "principalId" : "minio" + }, + "arn" : "arn:aws:s3:::images" + }, + "object" : { + "key" : "myphoto.jpg", + "size" : 200436, + "sequencer" : "1472CC35E6971AF3" + } + } + } ] + } + } ] + } +} +``` +``curl`` output above states that an Elasticsearch index has been successfully created with notification contents. + + +## Publish Minio events via Redis + +Install Redis from [here](http://redis.io/download). + +## Recipe steps + +### Step 1: Add Redis endpoint to Minio + +The default location of Minio server configuration file is ``~/.minio/config.json``. Update the Redis configuration block in ``config.json`` as follows: + +``` +"redis": { + "1": { + "enable": true, + "address": "127.0.0.1:6379", + "password": "yoursecret", + "key": "bucketevents" + } + } +``` +Restart Minio server to reflect config changes. ``bucketevents`` is the key used by Redis in this example. + +### Step 2: Enable bucket notification using Minio client + +We will enable bucket event notification to trigger whenever a JPEG image is uploaded or deleted from ``images`` bucket on ``myminio`` server. Here ARN value is ``arn:minio:sqs:us-east-1:1:redis``. To understand more about ARN please follow [AWS ARN](http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) documentation. + +``` +mc mb myminio/images +mc events add myminio/images arn:minio:sqs:us-east-1:1:redis --suffix .jpg +mc events list myminio/images +arn:minio:sqs:us-east-1:1:redis s3:ObjectCreated:*,s3:ObjectRemoved:* Filter: suffix=”.jpg” +``` + +### Step 3: Test on Redis + +Redis comes with a handy command line interface ``redis-cli`` to print all notifications on the console. + +``` +redis-cli -a yoursecret +``` + +Open another terminal and upload a JPEG image into ``images`` bucket. + +``` +mc cp myphoto.jpg myminio/images +``` + +``redis-cli`` prints event notification to the console. + +``` +redis-cli -a yoursecret +127.0.0.1:6379> monitor +OK +1474321638.556108 [0 127.0.0.1:40190] "AUTH" "yoursecret" +1474321638.556477 [0 127.0.0.1:40190] "RPUSH" "bucketevents" "{\"Records\":[{\"eventVersion\":\"2.0\",\"eventSource\":\"aws:s3\",\"awsRegion\":\"us-east-1\",\"eventTime\":\"2016-09-19T21:47:18.555Z\",\"eventName\":\"s3:ObjectCreated:Put\",\"userIdentity\":{\"principalId\":\"minio\"},\"requestParameters\":{\"sourceIPAddress\":\"[::1]:39250\"},\"responseElements\":{},\"s3\":{\"s3SchemaVersion\":\"1.0\",\"configurationId\":\"Config\",\"bucket\":{\"name\":\"images\",\"ownerIdentity\":{\"principalId\":\"minio\"},\"arn\":\"arn:aws:s3:::images\"},\"object\":{\"key\":\"myphoto.jpg\",\"size\":23745,\"sequencer\":\"1475D7B80ECBD853\"}}}],\"level\":\"info\",\"msg\":\"\",\"time\":\"2016-09-19T14:47:18-07:00\"}\n" +``` + +## Publish Minio events via NATS + +Install NATS from [here](http://nats.io/). + +## Recipe steps + +### Step 1: Add NATS endpoint to Minio + +The default location of Minio server configuration file is ``~/.minio/config.json``. Update the NATS configuration block in ``config.json`` as follows: + +``` +"nats": { + "1": { + "enable": true, + "address": "0.0.0.0:4222", + "subject": "bucketevents", + "username": "yourusername", + "password": "yoursecret", + "token": "", + "secure": false, + "pingInterval": 0 + } + }, +``` +Restart Minio server to reflect config changes. ``bucketevents`` is the subject used by NATS in this example. + +### Step 2: Enable bucket notification using Minio client + +We will enable bucket event notification to trigger whenever a JPEG image is uploaded or deleted from ``images`` bucket on ``myminio`` server. Here ARN value is ``arn:minio:sqs:us-east-1:1:nats``. To understand more about ARN please follow [AWS ARN](http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) documentation. + +``` +mc mb myminio/images +mc events add myminio/images arn:minio:sqs:us-east-1:1:nats --suffix .jpg +mc events list myminio/images +arn:minio:sqs:us-east-1:1:nats s3:ObjectCreated:*,s3:ObjectRemoved:* Filter: suffix=”.jpg” +``` + +### Step 3: Test on NATS + +Using this program below we can log the bucket notification added to NATS. + +```go +package main + +// Import Go and NATS packages +import ( + "runtime" + "log" + "github.com/nats-io/nats" +) + +func main() { + + // Create server connection + natsConnection, _ := nats.Connect("nats://yourusername:yoursecret@localhost:4222") + log.Println("Connected") + + // Subscribe to subject + log.Printf("Subscribing to subject 'bucketevents'\n") + natsConnection.Subscribe("bucketevents", func(msg *nats.Msg) { + + // Handle the message + log.Printf("Received message '%s\n", string(msg.Data) + "'") + }) + + // Keep the connection alive + runtime.Goexit() +} +``` + +``` +go run nats.go +2016/10/12 06:39:18 Connected +2016/10/12 06:39:18 Subscribing to subject 'bucketevents' +``` + +Open another terminal and upload a JPEG image into ``images`` bucket. + +``` +mc cp myphoto.jpg myminio/images +``` + +The example ``nats.go`` program prints event notification to console. + +``` +go run nats.go +2016/10/12 06:51:26 Connected +2016/10/12 06:51:26 Subscribing to subject 'bucketevents' +2016/10/12 06:51:33 Received message '{"EventType":"s3:ObjectCreated:Put","Key":"images/myphoto.jpg","Records":[{"eventVersion":"2.0","eventSource":"aws:s3","awsRegion":"us-east-1","eventTime":"2016-10-12T13:51:33Z","eventName":"s3:ObjectCreated:Put","userIdentity":{"principalId":"minio"},"requestParameters":{"sourceIPAddress":"[::1]:57106"},"responseElements":{},"s3":{"s3SchemaVersion":"1.0","configurationId":"Config","bucket":{"name":"images","ownerIdentity":{"principalId":"minio"},"arn":"arn:aws:s3:::images"},"object":{"key":"myphoto.jpg","size":56060,"eTag":"1d97bf45ecb37f7a7b699418070df08f","sequencer":"147CCD1AE054BFD0"}}}],"level":"info","msg":"","time":"2016-10-12T06:51:33-07:00"} +``` + + +## Publish Minio events via PostgreSQL + +Install PostgreSQL from [here](https://www.postgresql.org/). + +## Recipe steps + +### Step 1: Add PostgreSQL endpoint to Minio + +The default location of Minio server configuration file is ``~/.minio/config.json``. Update the PostgreSQL configuration block in ``config.json`` as follows: + +``` +"postgresql": { + "1": { + "enable": true, + "connectionString": "", + "table": "bucketevents", + "host": "127.0.0.1", + "port": "5432", + "user": "postgres", + "password": "mypassword", + "database": "bucketevents_db" + } + } +``` +Restart Minio server to reflect config changes. ``bucketevents`` is the database table used by PostgreSQL in this example. + +### Step 2: Enable bucket notification using Minio client + +We will enable bucket event notification to trigger whenever a JPEG image is uploaded or deleted from ``images`` bucket on ``myminio`` server. Here ARN value is ``arn:minio:sqs:us-east-1:1:postgresql``. To understand more about ARN please follow [AWS ARN](http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) documentation. + +``` +mc mb myminio/images +mc events add myminio/images arn:minio:sqs:us-east-1:1:postgresql --suffix .jpg +mc events list myminio/images +arn:minio:sqs:us-east-1:1:postgresql s3:ObjectCreated:*,s3:ObjectRemoved:* Filter: suffix=”.jpg” +``` + +### Step 3: Test on PostgreSQL + +Open another terminal and upload a JPEG image into ``images`` bucket. + +``` +mc cp myphoto.jpg myminio/images +``` + +Open PostgreSQL terminal to list the saved event notification logs. + +``` +bucketevents_db=# select * from bucketevents; + +key | value +--------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + images/myphoto.jpg | {"Records": [{"s3": {"bucket": {"arn": "arn:aws:s3:::images", "name": "images", "ownerIdentity": {"principalId": "minio"}}, "object": {"key": "myphoto.jpg", "eTag": "1d97bf45ecb37f7a7b699418070df08f", "size": 56060, "sequencer": "147CE57C70B31931"}, "configurationId": "Config", "s3SchemaVersion": "1.0"}, "awsRegion": "us-east-1", "eventName": "s3:ObjectCreated:Put", "eventTime": "2016-10-12T21:18:20Z", "eventSource": "aws:s3", "eventVersion": "2.0", "userIdentity": {"principalId": "minio"}, "responseElements": {}, "requestParameters": {"sourceIPAddress": "[::1]:39706"}}]} +(1 row) +``` + + +## Publish Minio events via kafka + +Install kafka from [here](http://kafka.apache.org/). + +## Recipe steps + +### Step 1: Add kafka endpoint to Minio + +The default location of Minio server configuration file is ``~/.minio/config.json``. Update the kafka configuration block in ``config.json`` as follows: + +``` +"kafka": { + "1": { + "enable": true, + "brokers": ["localhost:9092"], + "topic": "bucketevents" + } +} +``` +Restart Minio server to reflect config changes. ``bucketevents`` is the topic used by kafka in this example. + +### Step 2: Enable bucket notification using Minio client + +We will enable bucket event notification to trigger whenever a JPEG image is uploaded or deleted from ``images`` bucket on ``myminio`` server. Here ARN value is ``arn:minio:sqs:us-east-1:1:kafka``. To understand more about ARN please follow [AWS ARN](http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) documentation. + +``` +mc mb myminio/images +mc events add myminio/images arn:minio:sqs:us-east-1:1:kafka --suffix .jpg +mc events list myminio/images +arn:minio:sqs:us-east-1:1:kafka s3:ObjectCreated:*,s3:ObjectRemoved:* Filter: suffix=”.jpg” +``` + +### Step 3: Test on kafka + +We used [kafkacat](https://github.com/edenhill/kafkacat) to print all notifications on the console. + +``` +kafkacat -b localhost:9092 -t bucketevents +``` + +Open another terminal and upload a JPEG image into ``images`` bucket. + +``` +mc cp myphoto.jpg myminio/images +``` + +``kafkacat`` prints the event notification to the console. + +``` +kafkacat -b localhost:9092 -t bucketevents +{"EventType":"s3:ObjectCreated:Put","Key":"images/myphoto.jpg","Records":[{"eventVersion":"2.0","eventSource":"aws:s3","awsRegion":"us-east-1","eventTime":"2017-01-31T10:01:51Z","eventName":"s3:ObjectCreated:Put","userIdentity":{"principalId":"88QR09S7IOT4X1IBAQ9B"},"requestParameters":{"sourceIPAddress":"192.173.5.2:57904"},"responseElements":{"x-amz-request-id":"149ED2FD25589220","x-minio-origin-endpoint":"http://192.173.5.2:9000"},"s3":{"s3SchemaVersion":"1.0","configurationId":"Config","bucket":{"name":"images","ownerIdentity":{"principalId":"88QR09S7IOT4X1IBAQ9B"},"arn":"arn:aws:s3:::images"},"object":{"key":"myphoto.jpg","size":541596,"eTag":"04451d05b4faf4d62f3d538156115e2a","sequencer":"149ED2FD25589220"}}}],"level":"info","msg":"","time":"2017-01-31T15:31:51+05:30"} +``` +*NOTE* If you are running [distributed Minio](https://docs.minio.io/docs/distributed-minio-quickstart-guide), modify ``~/.minio/config.json`` on all the nodes with your bucket event notification backend configuration.