docs: added event-notification back end doc. (#3510)
koolhead17
8 years ago
committed by
Harshavardhana
parent
1b30a3be2b
commit
4dea4f3b89
@ -0,0 +1,477 @@ |
|||||||
|
[](https://slack.minio.io) |
||||||
|
|
||||||
|
Minio server supports Amazon S3 compatible bucket event notification for the following targets [AMQP](https://www.amqp.org/about/what), [Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started.html) , [Redis](http://redis.io/documentation), [nats.io](http://nats.io/), [PostgreSQL](https://www.postgresql.org) and [kafka](https://kafka.apache.org/). |
||||||
|
|
||||||
|
## Prerequisites |
||||||
|
|
||||||
|
* Install and configure Minio Server from [here](http://docs.minio.io/docs/minio). |
||||||
|
* Install and configure Minio Client from [here](https://docs.minio.io/docs/minio-client-quickstart-guide). |
||||||
|
|
||||||
|
| Notification Targets| |
||||||
|
|:-- |
||||||
|
| [`RabbitMQ`](#RabbitMQ) | |
||||||
|
| [`Elasticsearch`](#Elasticsearch) | |
||||||
|
| [`Redis`](#Redis) | |
||||||
|
| [`NATS`](#NATS) | |
||||||
|
| [`PostgreSQL`](#PostgreSQL) | |
||||||
|
| [`kafka`](#kafka) | |
||||||
|
|
||||||
|
<a name="RabbitMQ"></a> |
||||||
|
## Publish Minio events via RabbitMQ |
||||||
|
|
||||||
|
Install RabbitMQ from [here](https://www.rabbitmq.com/). |
||||||
|
|
||||||
|
### Recipe steps |
||||||
|
|
||||||
|
### Step 1: Add RabbitMQ endpoint to Minio |
||||||
|
|
||||||
|
The default location of Minio server configuration file is ``~/.minio/config.json``. Update the RabbitMQ configuration block in ``config.json`` as follows: |
||||||
|
|
||||||
|
``` |
||||||
|
"amqp": { |
||||||
|
"1": { |
||||||
|
"enable": true, |
||||||
|
"url": "amqp://myuser:mypassword@localhost:5672", |
||||||
|
"exchange": "bucketevents", |
||||||
|
"routingKey": "bucketlogs", |
||||||
|
"exchangeType": "fanout", |
||||||
|
"mandatory": false, |
||||||
|
"immediate": false, |
||||||
|
"durable": false, |
||||||
|
"internal": false, |
||||||
|
"noWait": false, |
||||||
|
"autoDeleted": false |
||||||
|
} |
||||||
|
} |
||||||
|
``` |
||||||
|
Restart Minio server to reflect config changes. Minio supports all the exchange available in [RabbitMQ](https://www.rabbitmq.com/). For this setup, we are using ``fanout`` exchange. |
||||||
|
|
||||||
|
### Step 2: Enable bucket notification using Minio client |
||||||
|
|
||||||
|
We will enable bucket event notification to trigger whenever a JPEG image is uploaded or deleted ``images`` bucket on ``myminio`` server. Here ARN value is ``arn:minio:sqs:us-east-1:1:amqp``. To understand more about ARN please follow [AWS ARN](http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) documentation. |
||||||
|
|
||||||
|
``` |
||||||
|
mc mb myminio/images |
||||||
|
mc events add myminio/images arn:minio:sqs:us-east-1:1:amqp --suffix .jpg |
||||||
|
mc events list myminio/images |
||||||
|
arn:minio:sqs:us-east-1:1:amqp s3:ObjectCreated:*,s3:ObjectRemoved:* Filter: suffix=”.jpg” |
||||||
|
``` |
||||||
|
|
||||||
|
### Step 3: Test on RabbitMQ |
||||||
|
|
||||||
|
The python program below waits on the queue exchange ``bucketevents`` and prints event notifications on the console. We use [Pika Python Client](https://www.rabbitmq.com/tutorials/tutorial-three-python.html) library to do this. |
||||||
|
|
||||||
|
```py |
||||||
|
#!/usr/bin/env python |
||||||
|
import pika |
||||||
|
|
||||||
|
connection = pika.BlockingConnection(pika.ConnectionParameters( |
||||||
|
host='localhost')) |
||||||
|
channel = connection.channel() |
||||||
|
|
||||||
|
channel.exchange_declare(exchange='bucketevents', |
||||||
|
type='fanout') |
||||||
|
|
||||||
|
result = channel.queue_declare(exclusive=False) |
||||||
|
queue_name = result.method.queue |
||||||
|
|
||||||
|
channel.queue_bind(exchange='bucketevents', |
||||||
|
queue=queue_name) |
||||||
|
|
||||||
|
print(' [*] Waiting for logs. To exit press CTRL+C') |
||||||
|
|
||||||
|
def callback(ch, method, properties, body): |
||||||
|
print(" [x] %r" % body) |
||||||
|
|
||||||
|
channel.basic_consume(callback, |
||||||
|
queue=queue_name, |
||||||
|
no_ack=False) |
||||||
|
|
||||||
|
channel.start_consuming() |
||||||
|
``` |
||||||
|
|
||||||
|
Execute this example python program to watch for RabbitMQ events on the console. |
||||||
|
|
||||||
|
```py |
||||||
|
python rabbit.py |
||||||
|
``` |
||||||
|
|
||||||
|
Open another terminal and upload a JPEG image into ``images`` bucket. |
||||||
|
|
||||||
|
``` |
||||||
|
mc cp myphoto.jpg myminio/images |
||||||
|
``` |
||||||
|
|
||||||
|
You should receive the following event notification via RabbitMQ once the upload completes. |
||||||
|
|
||||||
|
```py |
||||||
|
python rabbit.py |
||||||
|
‘{“Records”:[{“eventVersion”:”2.0",”eventSource”:”aws:s3",”awsRegion”:”us-east-1",”eventTime”:”2016–09–08T22:34:38.226Z”,”eventName”:”s3:ObjectCreated:Put”,”userIdentity”:{“principalId”:”minio”},”requestParameters”:{“sourceIPAddress”:”10.1.10.150:44576"},”responseElements”:{},”s3":{“s3SchemaVersion”:”1.0",”configurationId”:”Config”,”bucket”:{“name”:”images”,”ownerIdentity”:{“principalId”:”minio”},”arn”:”arn:aws:s3:::images”},”object”:{“key”:”myphoto.jpg”,”size”:200436,”sequencer”:”147279EAF9F40933"}}}],”level”:”info”,”msg”:””,”time”:”2016–09–08T15:34:38–07:00"}\n |
||||||
|
``` |
||||||
|
<a name="Elasticsearch"></a> |
||||||
|
## Publish Minio events via Elasticsearch |
||||||
|
|
||||||
|
Install Elasticsearch 2.4 from [here](https://www.elastic.co/downloads/past-releases/elasticsearch-2-4-0). |
||||||
|
|
||||||
|
## Recipe steps |
||||||
|
|
||||||
|
### Step 1: Add Elasticsearch endpoint to Minio |
||||||
|
|
||||||
|
The default location of Minio server configuration file is ``~/.minio/config.json``. Update the Elasticsearch configuration block in ``config.json`` as follows: |
||||||
|
|
||||||
|
``` |
||||||
|
"elasticsearch": { |
||||||
|
"1": { |
||||||
|
"enable": true, |
||||||
|
"url": "http://127.0.0.1:9200", |
||||||
|
"index": "bucketevents" |
||||||
|
} |
||||||
|
}, |
||||||
|
``` |
||||||
|
Restart Minio server to reflect config changes. ``bucketevents`` is the index used by Elasticsearch. |
||||||
|
|
||||||
|
### Step 2: Enable bucket notification using Minio client |
||||||
|
|
||||||
|
We will enable bucket event notification to trigger whenever a JPEG image is uploaded or deleted from ``images`` bucket on ``myminio`` server. Here ARN value is ``arn:minio:sqs:us-east-1:1:elasticsearch``. To understand more about ARN please follow [AWS ARN](http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) documentation. |
||||||
|
|
||||||
|
``` |
||||||
|
mc mb myminio/images |
||||||
|
mc events add myminio/images arn:minio:sqs:us-east-1:1:elasticsearch --suffix .jpg |
||||||
|
mc events list myminio/images |
||||||
|
arn:minio:sqs:us-east-1:1:elasticsearch s3:ObjectCreated:*,s3:ObjectRemoved:* Filter: suffix=”.jpg” |
||||||
|
``` |
||||||
|
|
||||||
|
### Step 3: Test on Elasticsearch |
||||||
|
|
||||||
|
Upload a JPEG image into ``images`` bucket, this is the bucket which has been configured for event notification. |
||||||
|
|
||||||
|
``` |
||||||
|
mc cp myphoto.jpg myminio/images |
||||||
|
``` |
||||||
|
|
||||||
|
Run ``curl`` to see new index name ``bucketevents`` in your Elasticsearch setup. |
||||||
|
|
||||||
|
``` |
||||||
|
curl -XGET '127.0.0.1:9200/_cat/indices?v' |
||||||
|
health status index pri rep docs.count docs.deleted store.size pri.store.size |
||||||
|
yellow open bucketevents 5 1 1 0 7.8kb 7.8kb |
||||||
|
``` |
||||||
|
|
||||||
|
Use curl to view contents of ``bucketevents`` index. |
||||||
|
|
||||||
|
``` |
||||||
|
curl -XGET '127.0.0.1:9200/bucketevents/_search?pretty=1' |
||||||
|
{ |
||||||
|
"took" : 3, |
||||||
|
"timed_out" : false, |
||||||
|
"_shards" : { |
||||||
|
"total" : 5, |
||||||
|
"successful" : 5, |
||||||
|
"failed" : 0 |
||||||
|
}, |
||||||
|
"hits" : { |
||||||
|
"total" : 1, |
||||||
|
"max_score" : 1.0, |
||||||
|
"hits" : [ { |
||||||
|
"_index" : "bucketevents", |
||||||
|
"_type" : "event", |
||||||
|
"_id" : "AVcRVOlwe-uNB1tfj6bx", |
||||||
|
"_score" : 1.0, |
||||||
|
"_source" : { |
||||||
|
"Records" : [ { |
||||||
|
"eventVersion" : "2.0", |
||||||
|
"eventSource" : "aws:s3", |
||||||
|
"awsRegion" : "us-east-1", |
||||||
|
"eventTime" : "2016-09-09T23:42:39.977Z", |
||||||
|
"eventName" : "s3:ObjectCreated:Put", |
||||||
|
"userIdentity" : { |
||||||
|
"principalId" : "minio" |
||||||
|
}, |
||||||
|
"requestParameters" : { |
||||||
|
"sourceIPAddress" : "10.1.10.150:52140" |
||||||
|
}, |
||||||
|
"responseElements" : { }, |
||||||
|
"s3" : { |
||||||
|
"s3SchemaVersion" : "1.0", |
||||||
|
"configurationId" : "Config", |
||||||
|
"bucket" : { |
||||||
|
"name" : "images", |
||||||
|
"ownerIdentity" : { |
||||||
|
"principalId" : "minio" |
||||||
|
}, |
||||||
|
"arn" : "arn:aws:s3:::images" |
||||||
|
}, |
||||||
|
"object" : { |
||||||
|
"key" : "myphoto.jpg", |
||||||
|
"size" : 200436, |
||||||
|
"sequencer" : "1472CC35E6971AF3" |
||||||
|
} |
||||||
|
} |
||||||
|
} ] |
||||||
|
} |
||||||
|
} ] |
||||||
|
} |
||||||
|
} |
||||||
|
``` |
||||||
|
``curl`` output above states that an Elasticsearch index has been successfully created with notification contents. |
||||||
|
|
||||||
|
<a name="Redis"></a> |
||||||
|
## Publish Minio events via Redis |
||||||
|
|
||||||
|
Install Redis from [here](http://redis.io/download). |
||||||
|
|
||||||
|
## Recipe steps |
||||||
|
|
||||||
|
### Step 1: Add Redis endpoint to Minio |
||||||
|
|
||||||
|
The default location of Minio server configuration file is ``~/.minio/config.json``. Update the Redis configuration block in ``config.json`` as follows: |
||||||
|
|
||||||
|
``` |
||||||
|
"redis": { |
||||||
|
"1": { |
||||||
|
"enable": true, |
||||||
|
"address": "127.0.0.1:6379", |
||||||
|
"password": "yoursecret", |
||||||
|
"key": "bucketevents" |
||||||
|
} |
||||||
|
} |
||||||
|
``` |
||||||
|
Restart Minio server to reflect config changes. ``bucketevents`` is the key used by Redis in this example. |
||||||
|
|
||||||
|
### Step 2: Enable bucket notification using Minio client |
||||||
|
|
||||||
|
We will enable bucket event notification to trigger whenever a JPEG image is uploaded or deleted from ``images`` bucket on ``myminio`` server. Here ARN value is ``arn:minio:sqs:us-east-1:1:redis``. To understand more about ARN please follow [AWS ARN](http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) documentation. |
||||||
|
|
||||||
|
``` |
||||||
|
mc mb myminio/images |
||||||
|
mc events add myminio/images arn:minio:sqs:us-east-1:1:redis --suffix .jpg |
||||||
|
mc events list myminio/images |
||||||
|
arn:minio:sqs:us-east-1:1:redis s3:ObjectCreated:*,s3:ObjectRemoved:* Filter: suffix=”.jpg” |
||||||
|
``` |
||||||
|
|
||||||
|
### Step 3: Test on Redis |
||||||
|
|
||||||
|
Redis comes with a handy command line interface ``redis-cli`` to print all notifications on the console. |
||||||
|
|
||||||
|
``` |
||||||
|
redis-cli -a yoursecret |
||||||
|
``` |
||||||
|
|
||||||
|
Open another terminal and upload a JPEG image into ``images`` bucket. |
||||||
|
|
||||||
|
``` |
||||||
|
mc cp myphoto.jpg myminio/images |
||||||
|
``` |
||||||
|
|
||||||
|
``redis-cli`` prints event notification to the console. |
||||||
|
|
||||||
|
``` |
||||||
|
redis-cli -a yoursecret |
||||||
|
127.0.0.1:6379> monitor |
||||||
|
OK |
||||||
|
1474321638.556108 [0 127.0.0.1:40190] "AUTH" "yoursecret" |
||||||
|
1474321638.556477 [0 127.0.0.1:40190] "RPUSH" "bucketevents" "{\"Records\":[{\"eventVersion\":\"2.0\",\"eventSource\":\"aws:s3\",\"awsRegion\":\"us-east-1\",\"eventTime\":\"2016-09-19T21:47:18.555Z\",\"eventName\":\"s3:ObjectCreated:Put\",\"userIdentity\":{\"principalId\":\"minio\"},\"requestParameters\":{\"sourceIPAddress\":\"[::1]:39250\"},\"responseElements\":{},\"s3\":{\"s3SchemaVersion\":\"1.0\",\"configurationId\":\"Config\",\"bucket\":{\"name\":\"images\",\"ownerIdentity\":{\"principalId\":\"minio\"},\"arn\":\"arn:aws:s3:::images\"},\"object\":{\"key\":\"myphoto.jpg\",\"size\":23745,\"sequencer\":\"1475D7B80ECBD853\"}}}],\"level\":\"info\",\"msg\":\"\",\"time\":\"2016-09-19T14:47:18-07:00\"}\n" |
||||||
|
``` |
||||||
|
<a name="NATS"></a> |
||||||
|
## Publish Minio events via NATS |
||||||
|
|
||||||
|
Install NATS from [here](http://nats.io/). |
||||||
|
|
||||||
|
## Recipe steps |
||||||
|
|
||||||
|
### Step 1: Add NATS endpoint to Minio |
||||||
|
|
||||||
|
The default location of Minio server configuration file is ``~/.minio/config.json``. Update the NATS configuration block in ``config.json`` as follows: |
||||||
|
|
||||||
|
``` |
||||||
|
"nats": { |
||||||
|
"1": { |
||||||
|
"enable": true, |
||||||
|
"address": "0.0.0.0:4222", |
||||||
|
"subject": "bucketevents", |
||||||
|
"username": "yourusername", |
||||||
|
"password": "yoursecret", |
||||||
|
"token": "", |
||||||
|
"secure": false, |
||||||
|
"pingInterval": 0 |
||||||
|
} |
||||||
|
}, |
||||||
|
``` |
||||||
|
Restart Minio server to reflect config changes. ``bucketevents`` is the subject used by NATS in this example. |
||||||
|
|
||||||
|
### Step 2: Enable bucket notification using Minio client |
||||||
|
|
||||||
|
We will enable bucket event notification to trigger whenever a JPEG image is uploaded or deleted from ``images`` bucket on ``myminio`` server. Here ARN value is ``arn:minio:sqs:us-east-1:1:nats``. To understand more about ARN please follow [AWS ARN](http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) documentation. |
||||||
|
|
||||||
|
``` |
||||||
|
mc mb myminio/images |
||||||
|
mc events add myminio/images arn:minio:sqs:us-east-1:1:nats --suffix .jpg |
||||||
|
mc events list myminio/images |
||||||
|
arn:minio:sqs:us-east-1:1:nats s3:ObjectCreated:*,s3:ObjectRemoved:* Filter: suffix=”.jpg” |
||||||
|
``` |
||||||
|
|
||||||
|
### Step 3: Test on NATS |
||||||
|
|
||||||
|
Using this program below we can log the bucket notification added to NATS. |
||||||
|
|
||||||
|
```go |
||||||
|
package main |
||||||
|
|
||||||
|
// Import Go and NATS packages |
||||||
|
import ( |
||||||
|
"runtime" |
||||||
|
"log" |
||||||
|
"github.com/nats-io/nats" |
||||||
|
) |
||||||
|
|
||||||
|
func main() { |
||||||
|
|
||||||
|
// Create server connection |
||||||
|
natsConnection, _ := nats.Connect("nats://yourusername:yoursecret@localhost:4222") |
||||||
|
log.Println("Connected") |
||||||
|
|
||||||
|
// Subscribe to subject |
||||||
|
log.Printf("Subscribing to subject 'bucketevents'\n") |
||||||
|
natsConnection.Subscribe("bucketevents", func(msg *nats.Msg) { |
||||||
|
|
||||||
|
// Handle the message |
||||||
|
log.Printf("Received message '%s\n", string(msg.Data) + "'") |
||||||
|
}) |
||||||
|
|
||||||
|
// Keep the connection alive |
||||||
|
runtime.Goexit() |
||||||
|
} |
||||||
|
``` |
||||||
|
|
||||||
|
``` |
||||||
|
go run nats.go |
||||||
|
2016/10/12 06:39:18 Connected |
||||||
|
2016/10/12 06:39:18 Subscribing to subject 'bucketevents' |
||||||
|
``` |
||||||
|
|
||||||
|
Open another terminal and upload a JPEG image into ``images`` bucket. |
||||||
|
|
||||||
|
``` |
||||||
|
mc cp myphoto.jpg myminio/images |
||||||
|
``` |
||||||
|
|
||||||
|
The example ``nats.go`` program prints event notification to console. |
||||||
|
|
||||||
|
``` |
||||||
|
go run nats.go |
||||||
|
2016/10/12 06:51:26 Connected |
||||||
|
2016/10/12 06:51:26 Subscribing to subject 'bucketevents' |
||||||
|
2016/10/12 06:51:33 Received message '{"EventType":"s3:ObjectCreated:Put","Key":"images/myphoto.jpg","Records":[{"eventVersion":"2.0","eventSource":"aws:s3","awsRegion":"us-east-1","eventTime":"2016-10-12T13:51:33Z","eventName":"s3:ObjectCreated:Put","userIdentity":{"principalId":"minio"},"requestParameters":{"sourceIPAddress":"[::1]:57106"},"responseElements":{},"s3":{"s3SchemaVersion":"1.0","configurationId":"Config","bucket":{"name":"images","ownerIdentity":{"principalId":"minio"},"arn":"arn:aws:s3:::images"},"object":{"key":"myphoto.jpg","size":56060,"eTag":"1d97bf45ecb37f7a7b699418070df08f","sequencer":"147CCD1AE054BFD0"}}}],"level":"info","msg":"","time":"2016-10-12T06:51:33-07:00"} |
||||||
|
``` |
||||||
|
|
||||||
|
<a name="PostgreSQL"></a> |
||||||
|
## Publish Minio events via PostgreSQL |
||||||
|
|
||||||
|
Install PostgreSQL from [here](https://www.postgresql.org/). |
||||||
|
|
||||||
|
## Recipe steps |
||||||
|
|
||||||
|
### Step 1: Add PostgreSQL endpoint to Minio |
||||||
|
|
||||||
|
The default location of Minio server configuration file is ``~/.minio/config.json``. Update the PostgreSQL configuration block in ``config.json`` as follows: |
||||||
|
|
||||||
|
``` |
||||||
|
"postgresql": { |
||||||
|
"1": { |
||||||
|
"enable": true, |
||||||
|
"connectionString": "", |
||||||
|
"table": "bucketevents", |
||||||
|
"host": "127.0.0.1", |
||||||
|
"port": "5432", |
||||||
|
"user": "postgres", |
||||||
|
"password": "mypassword", |
||||||
|
"database": "bucketevents_db" |
||||||
|
} |
||||||
|
} |
||||||
|
``` |
||||||
|
Restart Minio server to reflect config changes. ``bucketevents`` is the database table used by PostgreSQL in this example. |
||||||
|
|
||||||
|
### Step 2: Enable bucket notification using Minio client |
||||||
|
|
||||||
|
We will enable bucket event notification to trigger whenever a JPEG image is uploaded or deleted from ``images`` bucket on ``myminio`` server. Here ARN value is ``arn:minio:sqs:us-east-1:1:postgresql``. To understand more about ARN please follow [AWS ARN](http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) documentation. |
||||||
|
|
||||||
|
``` |
||||||
|
mc mb myminio/images |
||||||
|
mc events add myminio/images arn:minio:sqs:us-east-1:1:postgresql --suffix .jpg |
||||||
|
mc events list myminio/images |
||||||
|
arn:minio:sqs:us-east-1:1:postgresql s3:ObjectCreated:*,s3:ObjectRemoved:* Filter: suffix=”.jpg” |
||||||
|
``` |
||||||
|
|
||||||
|
### Step 3: Test on PostgreSQL |
||||||
|
|
||||||
|
Open another terminal and upload a JPEG image into ``images`` bucket. |
||||||
|
|
||||||
|
``` |
||||||
|
mc cp myphoto.jpg myminio/images |
||||||
|
``` |
||||||
|
|
||||||
|
Open PostgreSQL terminal to list the saved event notification logs. |
||||||
|
|
||||||
|
``` |
||||||
|
bucketevents_db=# select * from bucketevents; |
||||||
|
|
||||||
|
key | value |
||||||
|
--------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
||||||
|
images/myphoto.jpg | {"Records": [{"s3": {"bucket": {"arn": "arn:aws:s3:::images", "name": "images", "ownerIdentity": {"principalId": "minio"}}, "object": {"key": "myphoto.jpg", "eTag": "1d97bf45ecb37f7a7b699418070df08f", "size": 56060, "sequencer": "147CE57C70B31931"}, "configurationId": "Config", "s3SchemaVersion": "1.0"}, "awsRegion": "us-east-1", "eventName": "s3:ObjectCreated:Put", "eventTime": "2016-10-12T21:18:20Z", "eventSource": "aws:s3", "eventVersion": "2.0", "userIdentity": {"principalId": "minio"}, "responseElements": {}, "requestParameters": {"sourceIPAddress": "[::1]:39706"}}]} |
||||||
|
(1 row) |
||||||
|
``` |
||||||
|
|
||||||
|
<a name="kafka"></a> |
||||||
|
## Publish Minio events via kafka |
||||||
|
|
||||||
|
Install kafka from [here](http://kafka.apache.org/). |
||||||
|
|
||||||
|
## Recipe steps |
||||||
|
|
||||||
|
### Step 1: Add kafka endpoint to Minio |
||||||
|
|
||||||
|
The default location of Minio server configuration file is ``~/.minio/config.json``. Update the kafka configuration block in ``config.json`` as follows: |
||||||
|
|
||||||
|
``` |
||||||
|
"kafka": { |
||||||
|
"1": { |
||||||
|
"enable": true, |
||||||
|
"brokers": ["localhost:9092"], |
||||||
|
"topic": "bucketevents" |
||||||
|
} |
||||||
|
} |
||||||
|
``` |
||||||
|
Restart Minio server to reflect config changes. ``bucketevents`` is the topic used by kafka in this example. |
||||||
|
|
||||||
|
### Step 2: Enable bucket notification using Minio client |
||||||
|
|
||||||
|
We will enable bucket event notification to trigger whenever a JPEG image is uploaded or deleted from ``images`` bucket on ``myminio`` server. Here ARN value is ``arn:minio:sqs:us-east-1:1:kafka``. To understand more about ARN please follow [AWS ARN](http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html) documentation. |
||||||
|
|
||||||
|
``` |
||||||
|
mc mb myminio/images |
||||||
|
mc events add myminio/images arn:minio:sqs:us-east-1:1:kafka --suffix .jpg |
||||||
|
mc events list myminio/images |
||||||
|
arn:minio:sqs:us-east-1:1:kafka s3:ObjectCreated:*,s3:ObjectRemoved:* Filter: suffix=”.jpg” |
||||||
|
``` |
||||||
|
|
||||||
|
### Step 3: Test on kafka |
||||||
|
|
||||||
|
We used [kafkacat](https://github.com/edenhill/kafkacat) to print all notifications on the console. |
||||||
|
|
||||||
|
``` |
||||||
|
kafkacat -b localhost:9092 -t bucketevents |
||||||
|
``` |
||||||
|
|
||||||
|
Open another terminal and upload a JPEG image into ``images`` bucket. |
||||||
|
|
||||||
|
``` |
||||||
|
mc cp myphoto.jpg myminio/images |
||||||
|
``` |
||||||
|
|
||||||
|
``kafkacat`` prints the event notification to the console. |
||||||
|
|
||||||
|
``` |
||||||
|
kafkacat -b localhost:9092 -t bucketevents |
||||||
|
{"EventType":"s3:ObjectCreated:Put","Key":"images/myphoto.jpg","Records":[{"eventVersion":"2.0","eventSource":"aws:s3","awsRegion":"us-east-1","eventTime":"2017-01-31T10:01:51Z","eventName":"s3:ObjectCreated:Put","userIdentity":{"principalId":"88QR09S7IOT4X1IBAQ9B"},"requestParameters":{"sourceIPAddress":"192.173.5.2:57904"},"responseElements":{"x-amz-request-id":"149ED2FD25589220","x-minio-origin-endpoint":"http://192.173.5.2:9000"},"s3":{"s3SchemaVersion":"1.0","configurationId":"Config","bucket":{"name":"images","ownerIdentity":{"principalId":"88QR09S7IOT4X1IBAQ9B"},"arn":"arn:aws:s3:::images"},"object":{"key":"myphoto.jpg","size":541596,"eTag":"04451d05b4faf4d62f3d538156115e2a","sequencer":"149ED2FD25589220"}}}],"level":"info","msg":"","time":"2017-01-31T15:31:51+05:30"} |
||||||
|
``` |
||||||
|
*NOTE* If you are running [distributed Minio](https://docs.minio.io/docs/distributed-minio-quickstart-guide), modify ``~/.minio/config.json`` on all the nodes with your bucket event notification backend configuration. |
||||||
Loading…
Reference in new issue