diff --git a/source/developers/transforms-with-object-lambda.rst b/source/developers/transforms-with-object-lambda.rst new file mode 100644 index 00000000..bc0f4b4b --- /dev/null +++ b/source/developers/transforms-with-object-lambda.rst @@ -0,0 +1,366 @@ +.. _developers-object-lambda: + +================================= +Transforms with Object Lambda +================================= + +.. default-domain:: minio + +.. contents:: Table of Contents + :local: + :depth: 2 + +MinIO's Object Lambda enables developers to programmatically transform objects on demand. +You can transform objects as needed for your use case, such as redacting personally identifiable information (PII), enriching data with information from other sources, or converting between formats. + +Overview +-------- + +An :ref:`Object Lambda handler ` is a small code module that transforms the contents of an object and returns the results. +Like :s3-docs:`Amazon S3 Object Lambda functions `, you trigger a MinIO Object Lambda handler function with a GET request from an application. +The handler retrieves the requested object from MinIO, transforms it, and returns the modified data back to MinIO to send to the original application. +The original object remains unchanged. + +Each handler is an independent process, and multiple handlers can transform the same data. +This allows you to use the same object for different purposes without maintaining different versions of the original. + +.. _minio_object_lambda_handers: + +Object Lambda Handlers +---------------------- + +You can write a handler function in any language capable of sending and receiving HTTP requests. +It must be able to: + +- Listen for an HTTP POST request. +- Retrieve the original object using a URL. +- Return the transformed contents and authorization tokens. + +Create a Function +~~~~~~~~~~~~~~~~~ + +A handler function should perform the following steps: + +#. Extract the object details from the incoming POST request. + + The ``getObjectContext`` property of the JSON request payload contains details about the original object. + To construct the response, you need the following values: + + .. list-table:: + :widths: 25 75 + :header-rows: 1 + + * - Value + - Description + + * - ``inputS3Url`` + - A `presigned URL `__ for the original object. + The calling application generates the URL and sends it in the original request. + This allows the handler to access the original object without the MinIO credentials usually required. + The URL is valid for one hour. + + * - ``outputRoute`` + - A token that allows MinIO to validate the destination for the transformed object. + Return this value with the response in an ``x-amz-request-route`` header. + + * - ``outputToken`` + - A token that allows MinIO to validate the response. + Return this value in the response in an ``x-amz-request-token`` header. + +#. Retrieve the original object from MinIO. + + Use the presigned URL to retrieve the object from the MinIO deployment. + The contents of the object are in the body of the response. + +#. Transform the object as desired. + + Perform any operations needed to generate a transformed object. + Since the calling application is waiting for a response, you may wish to avoid potentially long running operations. + +#. Construct a response containing the following information: + + - The transformed object contents. + - An ``x-amz-request-route`` header with the ``outputRoute`` token. + - An ``x-amz-request-token`` header with the ``outputToken`` token. + +#. Return the response back to Object Lambda. + + MinIO validates the response and sends the transformed data back to the original calling application. + + +.. admonition:: Response headers + :class: note + + Handlers **must** include the ``outputRoute`` and ``outputToken`` values in the appropriate response headers. + This allows MinIO to correctly validate the response from the handler. + + +Register the Handler +~~~~~~~~~~~~~~~~~~~~ + +To enable MinIO to call the handler, register the handler function as a webhook with the following :ref:`MinIO server Object Lambda environment variables `: + +:envvar:`MINIO_LAMBDA_WEBHOOK_ENABLE_functionname ` + Enable or disable Object Lambda for a handler function. + For multiple handlers, set this environment variable for each function name. + +:envvar:`MINIO_LAMBDA_WEBHOOK_ENDPOINT_functionname ` + Register an endpoint for a handler function. + For multiple handlers, set this environment variable for each function endpoint. + +Restart MinIO to apply the changes. + + +Trigger From an Application +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To request a transformed object from your application: + +#. Connect to the MinIO deployment. + +#. Set the Object Lambda target by adding a ``lambdaArn`` parameter with the ARN of the desired handler. + +#. Generate a `presigned URL `__ for the original object. + +#. Use the generated URL to retrieve the transformed object. + + MinIO sends the request to the target Object Lambda handler. + The handler returns the transformed contents back to MinIO, which validates the response and sends it back to the application. + + +Example +------- + +Transform the contents of an object using Python, Go, and ``curl``: + +- Create and register an Object Lambda handler. +- Create a bucket and an object to transform. +- Request and display the transformed object contents. + +Prerequisites: + +- An existing :ref:`MinIO ` deployment +- Working Python (3.8+) and Golang development environments +- :doc:`The MinIO Go SDK ` + + +Create a Handler +~~~~~~~~~~~~~~~~ + +The sample handler, written in Python, retrieves the target object using a `presigned URL `__ generated by the caller. +The handler then transforms the object's contents and returns the new text. +It uses the `Flask web framework `__ and Python 3.8+. + +The following command installs Flask and other needed dependencies: + +.. code-block:: shell + :class: copyable + + pip install flask requests + +The handler calls ``swapcase()`` to change the case of each letter in the original text. +It then sends the results back to MinIO, which returns it to the caller. + +.. code-block:: py + :class: copyable + + from flask import Flask, request, abort, make_response + import requests + + app = Flask(__name__) + @app.route('/', methods=['POST']) + def get_webhook(): + if request.method == 'POST': + # Get the request event from the 'POST' call + event = request.json + + # Get the object context + object_context = event["getObjectContext"] + + # Get the presigned URL + # Used to fetch the original object from MinIO + s3_url = object_context["inputS3Url"] + + # Extract the route and request tokens from the input context + request_route = object_context["outputRoute"] + request_token = object_context["outputToken"] + + # Get the original S3 object using the presigned URL + r = requests.get(s3_url) + original_object = r.content.decode('utf-8') + + # Transform the text in the object by swapping the case of each char + transformed_object = original_object.swapcase() + + # Return the object back to Object Lambda, with required headers + # This sends the transformed data to MinIO + # and then to the user + resp = make_response(transformed_object, 200) + resp.headers['x-amz-request-route'] = request_route + resp.headers['x-amz-request-token'] = request_token + return resp + + else: + abort(400) + + if __name__ == '__main__': + app.run() + + +Start the Handler ++++++++++++++++++ + +Use the following command to start the handler in your local development environment: + +.. code-block:: shell + :class: copyable + + python lambda_handler.py + +The output resembles the following: + +.. code-block:: shell + + * Serving Flask app 'lambda_handler' + * Debug mode: off + WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead. + * Running on http://127.0.0.1:5000 + Press CTRL+C to quit + + +Start MinIO ++++++++++++ + +Once the handler is running, start MinIO with the :envvar:`MINIO_LAMBDA_WEBHOOK_ENABLE` and :envvar:`MINIO_LAMBDA_WEBHOOK_ENDPOINT` environment variables to register the function with MinIO. +To identify the specific Object Lambda handler, append the name of the function to the name of the environment variable. + +The following command starts MinIO in your local development environment: + +.. code-block:: shell + :class: copyable + + MINIO_LAMBDA_WEBHOOK_ENABLE_myfunction=on MINIO_LAMBDA_WEBHOOK_ENDPOINT_myfunction=http://localhost:5000 minio server /data + +Replace ``myfunction`` with the name of your handler function and ``/data`` with the location of the MinIO directory for your local deployment. +The output resembles the following: + +.. code-block:: shell + + MinIO Object Storage Server + Copyright: 2015-2023 MinIO, Inc. + License: GNU AGPLv3 + Version: RELEASE.2023-03-24T21-41-23Z (go1.19.7 linux/arm64) + + Status: 1 Online, 0 Offline. + API: http://192.168.64.21:9000 http://127.0.0.1:9000 + RootUser: minioadmin + RootPass: minioadmin + Object Lambda ARNs: arn:minio:s3-object-lambda::myfunction:webhook + + +Test the Handler +~~~~~~~~~~~~~~~~ + +To test the Lambda handler function, first create an object to transform. +Then invoke the handler, in this case with ``curl``, using the presigned URL from a Go function. + +#. Create a bucket and object for the handler to transform. + + .. code-block:: shell + :class: copyable + + mc alias set myminio/ http://localhost:9000 minioadmin minioadmin + mc mb myminio/myfunctionbucket + cat > testobject << EOF + Hello, World! + EOF + mc cp testobject myminio/myfunctionbucket/ + +#. Invoke the Handler + + The following Go code uses the :doc:`The MinIO Go SDK ` to generate a presigned URL and print it to ``stdout``. + + .. code-block:: go + :class: copyable + + package main + + import ( + "context" + "log" + "net/url" + "time" + "fmt" + + "github.com/minio/minio-go/v7" + "github.com/minio/minio-go/v7/pkg/credentials" + ) + + func main() { + + // Connect to the MinIO deployment + s3Client, err := minio.New("localhost:9000", &minio.Options{ + Creds: credentials.NewStaticV4("my_admin_user", "my_admin_password", ""), + Secure: false, + }) + if err != nil { + log.Fatalln(err) + } + + // Set the Lambda function target using its ARN + reqParams := make(url.Values) + reqParams.Set("lambdaArn", "arn:minio:s3-object-lambda::myfunction:webhook") + + // Generate a presigned url to access the original object + presignedURL, err := s3Client.PresignedGetObject(context.Background(), "myfunctionbucket", "testobject", time.Duration(1000)*time.Second, reqParams) + if err != nil { + log.Fatalln(err) + } + + // Print the URL to stdout + fmt.Println(presignedURL) + } + + In the code above, replace the following values: + + - Replace ``my_admin_user`` and ``my_admin_password`` with user credentials for a MinIO deployment. + - Replace ``myfunction`` with the same function name set in the ``MINIO_LAMBDA_WEBHOOK_ENABLE`` and ``MINIO_LAMBDA_WEBHOOK_ENDPOINT`` environment variables. + + To retrieve the transformed object, execute the Go code with ``curl`` to generate a GET request: + + .. code-block:: shell + :class: copyable + + curl -v $(go run presigned.go) + + ``curl`` runs the Go code and then retrieves the object with a GET request to the presigned URL. + The output resembles the following: + + .. code-block:: shell + + * Trying 127.0.0.1:9000... + * Connected to localhost (127.0.0.1) port 9000 (#0) + > GET /myfunctionbucket/testobject?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=minioadmin%2F20230406%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230406T184749Z&X-Amz-Expires=1000&X-Amz-SignedHeaders=host&lambdaArn=arn%3Aminio%3As3-object-lambda%3A%3Amyfunction%3Awebhook&X-Amz-Signature=68fe7e03929a7c0da38255121b2ae09c302840c06654d1b79d7907d942f69915 HTTP/1.1 + > Host: localhost:9000 + > User-Agent: curl/7.81.0 + > Accept: */* + > + * Mark bundle as not supporting multiuse + < HTTP/1.1 200 OK + < Content-Security-Policy: block-all-mixed-content + < Strict-Transport-Security: max-age=31536000; includeSubDomains + < Vary: Origin + < Vary: Accept-Encoding + < X-Amz-Id-2: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 + < X-Amz-Request-Id: 17536CF16130630E + < X-Content-Type-Options: nosniff + < X-Xss-Protection: 1; mode=block + < Date: Thu, 06 Apr 2023 18:47:49 GMT + < Content-Length: 14 + < Content-Type: text/plain; charset=utf-8 + < + hELLO, wORLD! + * Connection #0 to host localhost left intact + + + diff --git a/source/index.rst b/source/index.rst index e9fd278f..a85dd4c9 100644 --- a/source/index.rst +++ b/source/index.rst @@ -172,6 +172,7 @@ Any file uploaded to ``play`` should be considered public and non-protected. Software Development Kits (SDK) Security Token Service (STS) + Object Lambda MinIO Client MinIO Admin Client Integrations @@ -184,6 +185,7 @@ Any file uploaded to ``play`` should be considered public and non-protected. /developers/minio-drivers /developers/security-token-service + /developers/transforms-with-object-lambda /reference/minio-mc /reference/minio-mc-admin /reference/minio-mc-deprecated @@ -202,4 +204,4 @@ Any file uploaded to ``play`` should be considered public and non-protected. :titlesonly: :hidden: - /glossary \ No newline at end of file + /glossary diff --git a/source/integrations/integrations.rst b/source/integrations/integrations.rst index 309c3ca9..17f06244 100644 --- a/source/integrations/integrations.rst +++ b/source/integrations/integrations.rst @@ -23,3 +23,4 @@ All provided guides assume familiarity with the third-party integration software /integrations/setup-nginx-proxy-with-minio /integrations/presigned-put-upload-via-browser.md /integrations/generate-lets-encrypt-certificate-using-certbot-for-minio.md + diff --git a/source/reference/minio-server/minio-server.rst b/source/reference/minio-server/minio-server.rst index 53979688..d9f0374f 100644 --- a/source/reference/minio-server/minio-server.rst +++ b/source/reference/minio-server/minio-server.rst @@ -621,13 +621,13 @@ audit log webhook endpoints: export MINIO_AUDIT_WEBHOOK_ENABLE_PRIMARY="on" export MINIO_AUDIT_WEBHOOK_AUTH_TOKEN_PRIMARY="TOKEN" - export MINIO_AUDIT_WEBHOOK_ENDPOINT_PRIMARY="http://webhook-1.example.net + export MINIO_AUDIT_WEBHOOK_ENDPOINT_PRIMARY="http://webhook-1.example.net" export MINIO_AUDIT_WEBHOOK_CLIENT_CERT_SECONDARY="/tmp/cert.pem" export MINIO_AUDIT_WEBHOOK_CLIENT_KEY_SECONDARY="/tmp/key.pem" export MINIO_AUDIT_WEBHOOK_ENABLE_SECONDARY="on" export MINIO_AUDIT_WEBHOOK_AUTH_TOKEN_SECONDARY="TOKEN" - export MINIO_AUDIT_WEBHOOK_ENDPOINT_SECONDARY="http://webhook-1.example.net + export MINIO_AUDIT_WEBHOOK_ENDPOINT_SECONDARY="http://webhook-1.example.net" export MINIO_AUDIT_WEBHOOK_CLIENT_CERT_SECONDARY="/tmp/cert.pem" export MINIO_AUDIT_WEBHOOK_CLIENT_KEY_SECONDARY="/tmp/key.pem" @@ -2238,6 +2238,35 @@ endpoints as ``PRIMARY`` and ``SECONDARY`` respectively: :mc-conf:`notify_webhook comment ` configuration setting. +.. _minio-server-envvar-object-lambda-webhook: + +Object Lambda +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The following section documents environment variables for configuring MinIO to publish data to an HTTP webhook endpoint and trigger an Object Lambda function. +See :ref:`developers-object-lambda` for more complete documentation and tutorials on using these environment variables. + +You can specify multiple webhook endpoints as Lambda targets by appending a unique identifier ``_FUNCTIONNAME`` for each Object Lambda function. +For example, the following command sets two distinct Object Lambda webhook endpoints: + +.. code-block:: shell + :class: copyable + + export MINIO_LAMBDA_WEBHOOK_ENABLE_myfunction="on" + export MINIO_LAMBDA_WEBHOOK_ENDPOINT_myfunction="http://webhook-1.example.net" + export MINIO_LAMBDA_WEBHOOK_ENABLE_yourfunction="on" + export MINIO_LAMBDA_WEBHOOK_ENDPOINT_yourfunction="http://webhook-2.example.net" + +.. envvar:: MINIO_LAMBDA_WEBHOOK_ENABLE + + Specify ``"on"`` to enable the Object Lambda webhook endpoint for a handler function. + + Requires specifying :envvar:`MINIO_LAMBDA_WEBHOOK_ENDPOINT`. + +.. envvar:: MINIO_LAMBDA_WEBHOOK_ENDPOINT + + The HTTP endpoint of the webhook for the handler function. + .. _minio-server-envvar-external-identity-management-ad-ldap: Active Directory / LDAP Identity Management