Skip to content

Azure Blob Storage source

This event source subscribes to blob events from an Azure Storage Account through an Event Grid subscription. Events are consumed from a dedicated Event Hubs instance.

With tmctl:

tmctl create source azureblobstorage --storageAccountID <accountID> --endpoint.eventHubs.namespaceID <namespaceID> --auth.servicePrincipal.tenantID <tenantID> --auth.servicePrincipal.clientID <clientID> --auth.servicePrincipal.clientSecret <clientSecret>

On Kubernetes:

apiVersion: sources.triggermesh.io/v1alpha1
kind: AzureBlobStorageSource
metadata:
  name: sample
spec:
  storageAccountID: /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/MyGroup/providers/Microsoft.Storage/storageAccounts/MyBlobStorage

  endpoint:
    eventHubs:
      namespaceID: /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/MyGroup/providers/Microsoft.EventHub/namespaces/MyNamespace

  # Available event types are documented at
  # https://docs.microsoft.com/en-us/azure/event-grid/event-schema-blob-storage
  eventTypes:
  - Microsoft.Storage.BlobCreated
  - Microsoft.Storage.BlobDeleted

  auth:
    servicePrincipal:
      tenantID:
        value: 00000000-0000-0000-0000-000000000000
      clientID:
        value: 00000000-0000-0000-0000-000000000000
      clientSecret:
        value: some_secret

  sink:
    ref:
      apiVersion: eventing.triggermesh.io/v1alpha1
      kind: RedisBroker
      name: triggermesh
  • Secret: Service Principal authentication credentials, as described in the previous sections.
  • Storage Account ID: Resource ID of the Storage Account.
  • Event Hub ID: Resource ID of either
    • an Event Hubs namespace (Event Hub managed by the event source)
    • an Event Hubs instance (Event Hub managed by the user)
  • Event types: (optional) List of event types to subscribe to. BlobCreated and BlobDeleted are enabled by default when no item is set.

Events produced have the following attributes:

  • types
    • Microsoft.Storage.BlobCreated
    • Microsoft.Storage.BlobDeleted
    • Microsoft.Storage.BlobRenamed
    • Microsoft.Storage.DirectoryCreated
    • Microsoft.Storage.DirectoryDeleted
    • Microsoft.Storage.DirectoryRenamed
    • Microsoft.Storage.BlobTierChanged
    • Microsoft.Storage.AsyncOperationInitiated
    • Microsoft.Storage.BlobInventoryPolicyCompleted
  • Schema of the data attribute: com.microsoft.azure.blobstorage.event.json

See the Kubernetes object reference for more details.

Prerequisite(s)

  • Storage Account
  • Service Principal
  • Event Hubs Namespace
  • Event Hubs Instance (optional)

Storage Account

If you didn't already do so, create a Storage Account of one of the following supported types: General-purpose V2, BlockBlobStorage or BlobStorage. Take note of its resource ID, it is a required input to be able to run an instance of the Azure Blob Storage event source.

A resource ID for a Storage Account has the following format:

/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Storage/storageAccounts/{storageAccountName}

Storage Account overview Storage Account resource ID

Resource IDs can also be obtained using the Azure CLI (az). The following command line uses values from the screenshots above:

$ az storage account show --resource-group blobstorage-source-dev --name eventsourcedev
{
  "id": "/subscriptions/15537daf-e607-4df8-b2ef-277248b205b3/resourceGroups/blobstorage-source-dev/providers/Microsoft.Storage/storageAccounts/eventsourcedev",
  "resourceGroup": "blobstorage-source-dev",
  "type": "Microsoft.Storage/storageAccounts",
  "kind": "BlobStorage",
  "name": "eventsourcedev",
  "location": "eastus",
  "provisioningState": "Succeeded",
  ...
}

Service Principal

A Service Principal is required in order to authenticate the event source against the Azure tenant that has authority over the Azure Subscription to monitor. You can create a Service Principal by following the instructions at How to: Use the portal to create an Azure AD application and service principal that can access resources.

The section called Assign a role to the application describes how to assign permissions to the Service Principal. Make sure you select a role which has at least the following permissions:

  • Microsoft.EventGrid/eventSubscriptions/read
  • Microsoft.EventGrid/eventSubscriptions/write
  • Microsoft.EventGrid/eventSubscriptions/delete
  • Microsoft.EventHub/namespaces/eventhubs/write

The following set of permissions is also required if you decide to delegate the management of the Event Hub to the event source. In case you prefer to use your own Event Hub, these can be safely be omitted. More details on that topic are provided in the Event Hubs Instance section below.

  • Microsoft.EventHub/namespaces/eventhubs/read (optional)
  • Microsoft.EventHub/namespaces/eventhubs/delete (optional)

Additionally, assign the built-in role Azure Event Hubs Data Receiver to the Service Principal to allow it to receive events from an Event Hubs instance.

In the example below, we create a custom IAM role that is dedicated to the TriggerMesh event source for Azure Blob Storage:

IAM roles Custom IAM role

The corresponding role JSON is given as a reference which you can replicate to create a similar custom IAM role:

{
    "properties": {
        "roleName": "TriggerMesh Event Grid subscriptions manager",
        "description": "Allows the usage of TriggerMesh event sources for Azure Blob Storage.",
        "assignableScopes": [
            "/subscriptions/15537daf-e607-4df8-b2ef-277248b205b3"
        ],
        "permissions": [
            {
                "actions": [
                    "Microsoft.EventGrid/eventSubscriptions/read",
                    "Microsoft.EventGrid/eventSubscriptions/write",
                    "Microsoft.EventGrid/eventSubscriptions/delete",
                    "Microsoft.EventHub/namespaces/eventhubs/read",
                    "Microsoft.EventHub/namespaces/eventhubs/write",
                    "Microsoft.EventHub/namespaces/eventhubs/delete"
                ],
                "notActions": [],
                "dataActions": [],
                "notDataActions": []
            }
        ]
    }
}

After the Service Principal is created and assigned suitable roles, take note of the following information:

  • Tenant ID and Client ID (see Get tenant and app ID values for signing in)
  • Client secret (see Create a new application secret)

Event Hubs Namespace

Follow the instructions at Quickstart: Create an Event Hub using Azure portal, and create a new Event Hubs namespace. This namespace will contain an Event Hubs instance which will be configured by the event source as the destination of events originating from the Azure Storage Account.

Event Hubs namespaces

Event Hubs Instance (optional)

This section can be skipped if you would like to let the event source manage its own Event Hub. In this case, please ensure you granted all necessary permissions to the Service Principal in the previous section.

If, however, you prefer to provide your own Event Hub for that purpose, follow the instructions at Quickstart: Create an Event Hub using Azure portal to create an Event Hubs instance. Take note of its resource ID, it is a required input to be able to run an instance of the Azure Blob Storage event source.

A resource ID for an Event Hub has the following format:

/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.EventHub/namespaces/{namespaceName}/eventHubs/{eventHubName}

Note

The resource ID of the corresponding Event Hubs namespace is obtained by simply omitting the /eventHubs/{eventHubName} part of the Event Hub's resource ID.

Event Hub overview Event Hub resource ID

Resource IDs can also be obtained using the Azure CLI (az). The following command line uses values from the screenshots above:

$ az eventhubs eventhub show --resource-group blobstorage-source-dev --namespace-name eventsourcedev --name my-event-hub
{
  "id": "/subscriptions/15537daf-e607-4df8-b2ef-277248b205b3/resourceGroups/blobstorage-source-dev/providers/Microsoft.EventHub/namespaces/eventsourcedev/eventhubs/my-event-hub",
  "resourceGroup": "blobstorage-source-dev",
  "type": "Microsoft.EventHub/Namespaces/EventHubs",
  "name": "my-event-hub",
  "location": "East US",
  "status": "Active",
  ...
}

Check that everything is in place

This can be confirmed by navigating back to the Azure Portal and ensuring that:

  • The Storage Account contains a new Event Subscription targeting Event Hubs.
  • The Resource Group contains an Event Grid System Topic with an Event Subscription matching the one from the Storage Account.

Event Subscription Event Grid system topic