Tuesday, February 11, 2020

Data Platform Tips 76 - Metrics to monitor on Azure Stream Analytics

Azure Stream Analytics Job need to be monitored for resource consumption and error handling scenarios and it is a good practice to monitor the following metrics that will assist in trouble shooting the jobs.

Backlogged Input Events
Number of input events that are backlogged. A non-zero value for this metric implies that your job can’t keep up with the number of incoming events. If this value is slowly increasing or consistently non-zero, you should scale out your job.
Data Conversion Errors
Number of errors that the Stream Analytics job encountered when attempting to convert data types.
Early Input Events
Events whose application timestamp is earlier than their arrival time by more than 5 minutes.
Failed Function Requests
Number of failed Azure Machine Learning function calls (if applicable).
Function Events
Number of events sent to the Azure Machine Learning function (if applicable).
Function Requests
Number of calls to the Azure Machine Learning function (if applicable).
Input Deserialization Errors
Number of input events that could not be deserialized.
Input Event Bytes
Amount of data received by the Stream Analytics job, in bytes. This can be used to validate that events are being sent to the input source.
Input Events
Number of records deserialized from the input events. This count does not include incoming events that result in deserialization errors.
Input Sources Received
Number of messages received by the job. For Event Hub, a message is a single EventData. For Blob, a message is a single blob.
Late Input Events
Events that arrived later than the configured late arrival tolerance window.
Out-of-Order Events
Number of events received out of order that were either dropped or given an adjusted timestamp, based on the Event Ordering Policy.
Output Events
Amount of data sent by the Stream Analytics job to the output target, in number of events.
Runtime Errors
Total number of errors related to query processing (excluding errors found while ingesting events or outputting results)
SU % Utilization
The utilization of the Streaming Unit(s) assigned to a job from the Scale tab of the job. Should this indicator reach 80%, or above, there is high probability that event processing may be delayed or stopped making progress.
Watermark Delay
The maximum watermark delay across all partitions of all outputs in the job.

No comments:

Post a Comment