Develop event-based solutions

Develop event-based solutions is part of Connect to and consume Azure services and third-party services topics. the total weight of this in the exam will be 25-30%. This training post is designed to help and provide readers with a better understanding of the topic mentioned.

Disclaimer: This is not a training article to help complete the Microsoft Azure AZ-204, but it provides a good insight and understanding of topics and subject matters within the specified. labs and hands-on work are essential to passing most Microsoft Azure exams.

Develop event-based solutions:
implement solutions that use Azure Event Grid

Develop event-based solutions

Event-driven architecture

In an event-driven architecture, events are delivered in nearly real time, so that consumers can respond immediately to events as they occur.

An event-driven architecture can use a pub/sub model or an event-stream model.

  • Pub/sub: The messaging infrastructure keeps track of subscriptions. When an event is published, it sends the event to each subscriber. After an event is received, it cannot be replayed, and new subscribers do not observe the event.
  • Event streaming: Events are written to a log. Events are strictly ordered (within a partition) and durable. Clients don’t subscribe to the stream. Instead, a client can read from any part of the stream. The client is responsible for advancing its position in the stream. That means that a client can join at any time, and can replay events.

On the consumer side, there are some common variations:

  • Simple event processing. An event immediately triggers an action in the consumer. For example, you could use Microsoft Azure Functions with a Service Bus trigger, so that a function executes whenever a message is published to a Service Bus topic.
  • Complex event processing. A consumer processes a series of events, observing patterns in the event data by using a technology such as Azure Stream Analytics or Apache Storm. For example, you could aggregate readings from an embedded device over a time window, and generate a notification if the moving average crosses a certain threshold.
  • Event stream processing. Use a data-streaming platform, such as Azure IoT Hub or Apache Kafka, as a pipeline to ingest events and feed them to stream processors. The stream processors act to process or transform the stream. There may be multiple stream processors for different subsystems of the application. This approach is a good fit for Internet of Things (IoT )workloads. •

The source of the events might be external to the system, such as physical devices in an IoT solution. In that case, the system must be able to ingest the data at the volume and throughput that is required by the data source.

Azure Event Grid

Azure Event Grid allows you to easily build applications with event-based architectures. First, select the Azure resource that you would like to subscribe to, and then provide the event handler or webhook endpoint to send the event to. Event Grid has built-in support for events coming from Azure services, like storage blobs and resource groups. Event Grid also has support for your own events, using custom topics.

You can use filters to route specific events to different endpoints, multicast to multiple endpoints, and make sure your events are reliably delivered.

Event Grid concepts

Events

An event is the smallest amount of information that fully describes something that happened in the system. Every event has common information, such as: source of the event, time the event took place, and the unique identifier. Every event also has specific information that is only relevant to the specific type of event. For example, an event about a new file being created in Azure Storage has details about the file, such as the lastTimeModified value. Or, an Event Hubs event has the URL of the Capture file.

Each event is limited to 64 kilobytes (KB) of data.

Event sources

An event source is where the event happens. Each event source is related to one or more event types. For example, Azure Storage is the event source for blob-created events. IoT Hub is the event source for device-created events. Your application is the event source for custom events that you define. Event sources are responsible for sending events to Event Grid.

Topics

The Event Grid topic provides an endpoint where the source sends events. The publisher creates the Event Grid topic, and decides whether an event source needs one topic or more than one topic. A topic is used for a collection of related events. To respond to certain types of events, subscribers decide which topics to subscribe to.

System topics are built-in topics provided by Azure services. You don’t observe system topics in your Azure subscription because the publisher owns the topics, but you can subscribe to them. To subscribe, you provide information about the resource that you want to receive events from. If you have access to the resource, you can subscribe to its events.

Custom topics are application and third-party topics. When you create or are assigned access to a custom topic, you observe that custom topic in your subscription.

When designing your application, you have flexibility when deciding how many topics to create. For large solutions, create a custom topic for each category of related events. For example, consider an application that sends events related to modifying user accounts and processing orders. It’s unlikely that any event handler wants both categories of events. Create two custom topics and let event handlers subscribe to the one that interests them. For small solutions, you might prefer to send all events to a single topic. Event subscribers can filter for the event types they want.

Event subscriptions

A subscription tells Event Grid which events on a topic you’re interested in receiving. When creating the subscription, you provide an endpoint for handling the event. You can filter the events that are sent to the endpoint. You can filter by event type or subject pattern.

Event handlers

From an Event Grid perspective, an event handler is the place where the event is sent. The handler takes some further action to process the event. Event Grid supports several handler types. You can use a supported Azure service or your own webhook as the handler. Depending on the type of handler, Event Grid follows different mechanisms to guarantee the delivery of the event. For HTTP webhook event handlers, the event is retried until the handler returns a status code of 200 – OK. For Azure Storage Queue, the events are retried until the Queue service successfully processes the message push into the queue.

Security

Three types of authentication:

Webhook event delivery

  • Event Grid will send HTTP Post requests to an endpoint of your choice
  • Requires you prove ownership unless it’s hosted in Azure Functions, Logic Apps, or Azure Automation

Event subscriptions

  • Uses role-based access control (RBAC) permissions to ensure that you have access to the resources needed to subscribe to events
  • Must have Microsoft.EventGrid/EventSubscriptions/Write permission on the resource that is the event source

Custom topic publishing

  • Uses SAS or key-based authentication

Develop event-based solutions:
implement solutions that use Azure Notification Hubs

Develop event-based solutions

Notification Hubs

Azure Notification Hubs provide an easy-to-use and scaled-out push engine that allows you to send notifications to any platform (iOS, Android, Windows, Kindle, Baidu, etc.) from any backend (cloud or on-premises).

Notification Hubs example

  1. The client app wants to receive notification. Hence, it contacts the corresponding PNS to retrieve its unique and temporary push handle. The handle type depends on the system (for example, WNS has URIs, while APNS has tokens).
  2. The client app stores this handle in the app backend or provider.
  3. To send a push notification, the app backend uses the Notification Hubs SDK to send a notification. The Notification Hubs service contacts the PNS by using the handle to target a specific client app.
  4. The PNS forwards the notification to the device specified by the handle.

Push notifications

Push notifications is a form of app-to-user communication where users of mobile apps are notified of certain desired information, usually in a pop-up or dialog box. Users can generally choose to view or dismiss the message. Choosing the former opens the mobile application that communicated the notification.

Push notification service

Push notifications are delivered through platform-specific infrastructures called Platform Notification Systems (PNSs). They offer basic push functionalities to deliver a message to a device with a provided handle, and have no common interface. To send a notification to all customers across the iOS, Android, and Windows versions of an app, the developer must work with Apple Push Notification Service (APNS), Firebase Cloud Messaging (FCM), and Windows Notification Service (WNS).

At a high level, here is how push works:

  1. The client app decides it wants to receive notification. Hence, it contacts the corresponding PNS to retrieve its unique and temporary push handle. The handle type depends on the system (for example, WNS has URIs while APNS has tokens).
  2. The client app stores this handle in the app backend or provider.
  3. To send a push notification, the app backend contacts the PNS by using the handle to target a specific client app.
  4. The PNS forwards the notification to the device specified by the handle.

Comparing cloud messaging options

RequirementSimple queuingEventing and PubSubBig data streamingEnterprise messaging
ProductQueue storageEvent GridEvent HubsService Bus
Supported advantages•Communication within an app •Individual message •Queue semantics / polling buffer •Simple and easy to use •Pay as you go•Communication between apps / orgs •Individual message •Push semantics •Filtering and routing •Pay as you go •Fan out•Many messages in a Stream (think in MBs) •Ease of use and operation •Low cost •Fan in •Strict ordering •Works with other tools•Instantaneous consistency •Strict ordering •Java Messaging Service •Non-repudiation and security •Geo-replication and availability •Rich features (such as deduplication and scheduling)
Weaknesses•Ordering of messaging •Instantaneous consistency•Ordering of messaging •Instantaneous consistency•Server-side cursor •Only once•Cost •Simplicity
TypeServerlessServerlessBig dataEnterprise
Develop event-based solutions:
implement solutions that use Azure Event Hub

Develop event-based solutions

Azure Event Hubs

Azure Event Hubs is a scalable event-processing service that ingests and processes large volumes of events and data, with low latency and high reliability.

Event publishers

An Event Hubs namespace provides a unique scoping container, referenced by its fully qualified domain name (FQDN), in which you create one or more event hubs or Kafka topics.


Event publishers

Any entity that sends data to an event hub is an event producer or event publisher. Event publishers can publish events by using HTTPS or AMQP 1.0 or Kafka 1.0 and later. Event publishers use a SAS token to identify themselves to an event hub, and can have a unique identity or use a common SAS token.

Publishing an event

You can publish an event via AMQP 1.0, Kafka 1.0 (and later), or HTTPS. Event Hubs provides client libraries and classes for publishing events to an event hub from .NET clients. For other runtimes and platforms, you can use any AMQP 1.0 client, such as Apache Qpid. You can publish events individually or batched. A single publication (event data instance) has a limit of 1 megabyte (MB), regardless of whether it is a single event or a batch. Publishing events larger than this threshold results in an error. It is a best practice for publishers to be unaware of partitions within the event hub and to only specify a partition key (introduced in the next section), or their identity via their SAS token.

The choice to use AMQP or HTTPS is specific to the usage scenario. AMQP requires the establishment of a persistent bidirectional socket in addition to Transport Layer Security (TLS) or SSL/TLS. AMQP has higher network costs when initializing the session, however HTTPS requires additional SSL overhead for every request. AMQP has higher performance for frequent publishers.

Event Hubs ensures that all events sharing a partition key value are delivered in order and to the same partition. If partition keys are used with publisher policies, then the identity of the publisher and the value of the partition key must match. Otherwise, an error occurs.

Partitions

Consumer groups

The publish/subscribe mechanism of Event Hubs is enabled through consumer groups. A consumer group is a view (state, position, or offset) of an entire event hub. Consumer groups enable multiple consuming applications to each have a separate view of the event stream, and to read the stream independently at their own pace and with their own offsets.

In a stream processing architecture, each downstream application equates to a consumer group. If you want to write event data to long-term storage, then that storage writer application is a consumer group. Complex event processing can then be performed by another, separate consumer group. You can only access partitions through a consumer group. There is always a default consumer group in an event hub, and you can create up to 20 consumer groups for a Standard tier Event Hub.

There can be at most five concurrent readers on a partition per consumer group; however, it is recommended that there be only one active receiver on a partition per consumer group. Within a single partition, each reader receives all the messages. If you have multiple readers on the same partition, then you process duplicate messages. You need to handle this in your code, which might not be a trivial task. However, it’s a valid approach in some scenarios.

Capture

Data can be automatically captured

  • Stored in Azure Blob storage or Azure Data Lake Storage
  • Capture-time or size intervals can be specified

You can specify a window to control capturing

  • Must specify a minimum size and time configuration
  • First trigger encountered causes a capture operation

Data is stored by using a naming convention:

{Namespace}/{EventHub}/{PartitionId}/{Year}/{Month}/{Day}/{Hour}/{Minute}/{Second}

https://mystorageaccount.blob.core.windows.net/mycontainer/mynamespace/myeventhub/0/2017/12/08/03/03/17.avro

Integration with Kafka

Event Hubs provides a Kafka endpoint that your existing Kafka-based applications can use. This Kafka endpoint serves as an alternative to running your own Kafka cluster. Event Hubs supports Apache Kafka protocol 1.0 and later, and it works with your existing Kafka applications, including MirrorMaker.

You can start using the Kafka endpoint from your applications with no code changes and only minimal configuration. You update the connection string in configurations to point to the Kafka endpoint that’s exposed by your event hub instead of pointing to your Kafka cluster. Then, you can start streaming events from your applications that use the Kafka protocol into Event Hubs.

Event Hubs and Apache Kafka mapping

Event HubsKafka
NamespaceCluster
Event HubTopic
PartitionPartition
Consumer GroupConsumer Group
OffsetOffset

More topics on Connect to and consume Azure services and third-party services:

Develop an App Service Logic App

Implement API Management

Develop message-based solutions

Microsoft Azure AZ-204 exam topics:

If you have covered the current topics in Connect to and consume Azure services and third-party services then you can have a look at the other topic areas:

View full documentation Microsoft Azure: AZ-204 exam content from Microsoft

Leave a Reply