minio 支援object搜尋方案
2023-09-08 12:00:28
minio支援上傳時對object打標籤，查詢時可以根據標籤做篩選。但是有ftp上傳檔案的需求，導致無法給object打標籤。並且也不清楚minio對於根據標籤的篩選效能如何，因此我們打算將object的物件的資料放到資料庫。在資料庫中對object進行篩選。
docker部署
mkdir -p ~/minio/data
docker run \
-d \
--network=host \
--name minio \
-v /opt/minio/data:/data \
-e "MINIO_ROOT_USER=ROOTNAME" \
-e "MINIO_ROOT_PASSWORD=CHANGEME123" \
quay.io/minio/minio server /data --console-address ":9090" \
--address=":9002" \
--ftp="address=:8021" \
--ftp="passive-port-range=30000-40000"
9090埠是minio console網頁的埠，登入此網頁使用者名稱密碼就是我們設定的ROOTNAME CHANGEME123
9002是我們程式呼叫minio sdk的埠，根據自己機器的情況按需設定。
ftp參數列示我們開啟ftp，並使用8021最為ftp伺服器埠。
登入9090網頁後，申請Access key，呼叫minio sdk時需要secret key、Access key等。
Bucket Notification
我們通過訂閱minio的object events來做資料同步。
minio提供了多種方式，我們最終使用的webhook方式，但是我們暫時先說一下mysql方式：
docker run \
-d \
--network=host \
--name minio \
-v /opt/minio/data:/data \
-e "MINIO_ROOT_USER=ROOTNAME" \
-e "MINIO_ROOT_PASSWORD=CHANGEME123" \
-e MINIO_NOTIFY_MYSQL_ENABLE_PRIMARY="on" \
-e MINIO_NOTIFY_MYSQL_DSN_STRING_PRIMARY="root:root@tcp(192.168.16.46:3306)/cogent-admin" \
-e MINIO_NOTIFY_MYSQL_TABLE_PRIMARY="minio_events" \
-e MINIO_NOTIFY_MYSQL_FORMAT_PRIMARY="namespace" \
quay.io/minio/minio server /data --console-address ":9090" \
--address=":9002" \
--ftp="address=:8021" \
--ftp="passive-port-range=30000-40000"
MINIO_NOTIFY_MYSQL_DSN_STRING_PRIMARY 設定我們的mysql資料庫，然後進入docker容器
docker exec -it minio bash
chmod 777 /opt/bin/mc
mc alias set myminio http://192.168.16.46:9002 ROOTNAME CHANGEME123
mc admin info --json myminio
mc event add myminio/cogent arn:minio:sqs::PRIMARY:mysql \
--event put
mc event add myminio/cogent arn:minio:sqs::PRIMARY:mysql \
--event delete
mc event ls myminio/cogent arn:minio:sqs::PRIMARY:mysql
注意事項：我們必須先至少建立一個bucket才能mc event add。進入容器後，我們要給/opt/bin/mc 執行許可權。mc event add 的設定好像會持久化，如果重新啟動一個容器，event仍然在，這時就不用在執行mc event add命令了，當然我們可以mc event ls去確定一下evnet是否新增成功
最後我們的結果，minio傳送給mysql中的資料：
value：
{
"Records": [
{
"s3": {
"bucket": {
"arn": "arn:aws:s3:::cogent",
"name": "cogent",
"ownerIdentity": {
"principalId": "ROOTNAME"
}
},
"object": {
"key": "jetbra.zip",
"eTag": "8943434aec7868e6e16d36209ca47fab",
"size": 172138,
"sequencer": "177F6C96E9864D53",
"contentType": "application/x-zip-compressed",
"userMetadata": {
"content-type": "application/x-zip-compressed"
}
},
"configurationId": "Config",
"s3SchemaVersion": "1.0"
},
"source": {
"host": "192.168.16.148",
"port": "",
"userAgent": "MinIO (linux; amd64) minio-go/v7.0.59 MinIO Console/(dev)"
},
"awsRegion": "",
"eventName": "s3:ObjectCreated:Put",
"eventTime": "2023-08-28T02:56:20.333Z",
"eventSource": "minio:s3",
"eventVersion": "2.0",
"userIdentity": {
"principalId": "ROOTNAME"
},
"responseElements": {
"x-amz-id-2": "fad4b3083214c3b0ad28cc0f83fd770a8fd5fb5e47b065bc7cae01b61e817e1a",
"x-amz-request-id": "177F6C96E763B761",
"x-minio-deployment-id": "d6a44a90-a62c-4605-8aa2-121a85f0d440",
"x-minio-origin-endpoint": "http://192.168.192.254:9002"
},
"requestParameters": {
"region": "",
"principalId": "ROOTNAME",
"sourceIPAddress": "192.168.16.148"
}
}
]
}
這種格式在mysql中不好查詢。因此我使用webhook方式訂閱event，這種方式更靈活，我可以提取任何我想要的資料存入資料庫。
以下是webhook的方式：
docker run \
-d \
--network=host \
--name minio \
-v /opt/minio/data:/data \
-e MINIO_ROOT_USER=ROOTNAME \
-e MINIO_ROOT_PASSWORD=CHANGEME123 \
-e MINIO_NOTIFY_WEBHOOK_ENABLE_PRIMARY="on" \
-e MINIO_NOTIFY_WEBHOOK_ENDPOINT_PRIMARY="http://127.0.0.1:8080/buckets/event" \
quay.io/minio/minio server /data --console-address ":9090" \
--address=":9002" \
--ftp="address=:8021" \
--ftp="passive-port-range=30000-40000"
mc alias set myminio http://192.168.16.46:9002 ROOTNAME CHANGEME123
mc admin info --json myminio
mc event add myminio/cogent arn:minio:sqs::PRIMARY:webhook --event put
mc event add myminio/cogent arn:minio:sqs::PRIMARY:webhook --event delete
mc event ls myminio/cogent arn:minio:sqs::PRIMARY:webhook
進入容器後，記得要給 /opt/bin/mc 賦予執行許可權
以下是我對minio中object進行刪除和新增時收到的json：
刪除時，webhook的引數：
{
"EventName": "s3:ObjectRemoved:Delete",
"Key": "cogent/jetbra (1).zip",
"Records": [
{
"eventVersion": "2.0",
"eventSource": "minio:s3",
"awsRegion": "",
"eventTime": "2023-08-28T05:55:05.275Z",
"eventName": "s3:ObjectRemoved:Delete",
"userIdentity": {
"principalId": "ROOTNAME"
},
"requestParameters": {
"principalId": "ROOTNAME",
"region": "",
"sourceIPAddress": "127.0.0.1"
},
"responseElements": {
"content-length": "160",
"x-amz-id-2": "fad4b3083214c3b0ad28cc0f83fd770a8fd5fb5e47b065bc7cae01b61e817e1a",
"x-amz-request-id": "177F765801436093",
"x-minio-deployment-id": "d6a44a90-a62c-4605-8aa2-121a85f0d440",
"x-minio-origin-endpoint": "http://192.168.192.254:9002"
},
"s3": {
"s3SchemaVersion": "1.0",
"configurationId": "Config",
"bucket": {
"name": "cogent",
"ownerIdentity": {
"principalId": "ROOTNAME"
},
"arn": "arn:aws:s3:::cogent"
},
"object": {
"key": "jetbra+%281%29.zip",
"sequencer": "177F765801E61109"
}
},
"source": {
"host": "127.0.0.1",
"port": "",
"userAgent": "MinIO (linux; amd64) minio-go/v7.0.59 MinIO Console/(dev)"
}
}
]
}
新增object時，webhook引數
{
"EventName": "s3:ObjectCreated:Put",
"Key": "cogent/jetbra (1).zip",
"Records": [
{
"eventVersion": "2.0",
"eventSource": "minio:s3",
"awsRegion": "",
"eventTime": "2023-08-28T05:56:00.485Z",
"eventName": "s3:ObjectCreated:Put",
"userIdentity": {
"principalId": "ROOTNAME"
},
"requestParameters": {
"principalId": "ROOTNAME",
"region": "",
"sourceIPAddress": "192.168.16.148"
},
"responseElements": {
"x-amz-id-2": "fad4b3083214c3b0ad28cc0f83fd770a8fd5fb5e47b065bc7cae01b61e817e1a",
"x-amz-request-id": "177F7664D9E2DAD5",
"x-minio-deployment-id": "d6a44a90-a62c-4605-8aa2-121a85f0d440",
"x-minio-origin-endpoint": "http://192.168.192.254:9002"
},
"s3": {
"s3SchemaVersion": "1.0",
"configurationId": "Config",
"bucket": {
"name": "cogent",
"ownerIdentity": {
"principalId": "ROOTNAME"
},
"arn": "arn:aws:s3:::cogent"
},
"object": {
"key": "jetbra+%281%29.zip",
"size": 172138,
"eTag": "8943434aec7868e6e16d36209ca47fab",
"contentType": "application/x-zip-compressed",
"userMetadata": {
"content-type": "application/x-zip-compressed"
},
"sequencer": "177F7664DCA24B86"
}
},
"source": {
"host": "192.168.16.148",
"port": "",
"userAgent": "MinIO (linux; amd64) minio-go/v7.0.59 MinIO Console/(dev)"
}
}
]
}
漏洞：如果我們Java程式重啟，或者說webhook的web程式重啟後，這時我們對minio刪除或新增修改物件資料，那麼此時將監聽不到這個事件。因此我們一種方式是停止web服務時，先停止minio。另一種方式是執行定時任務，同步資料，或者說在web程式啟動後一分鐘內，同步minio 的object資料。當然如果object非常多，web程式更新頻繁，我們可以將minio的event放到MQ中，web程式再去消費MQ中event