Azure Monitor Alerts Series – Part 1

Alerts are important part of our monitoring and probably the most important one. Getting data and visualizing it is the foundation for alerts but in order to move to actual monitoring you need alerts. I can tell you nobody sits all day in front of dashboard and looks at visualized data. Alerts are also our knowledge of our applications and infrastructure gathered to help us when things are not going as planned. I wanted to write this blog post series for quite some time and I think this is the right time to do it. The reason for that is Classic Azure alerts are being deprecated and the vision of unified alerting capabilities is coming together and becoming more powerful… sort of. I will comment on parts that I think could and should be improved and hopefully they will be. I also expect some new features around Ignite as usually that is when Microsoft reveals some new stuff. They actually do it all the time it just the end development of some features matches Ignite conference time frame.

Although we have unified alerting capabilities there are different type of alerts present in Azure. The beauty of that is we can use whatever makes sense for us depending on the features offered. In this first blog post we will give summary the different type and sub-types of alerts and in later blog post we will have a deep dive on most of them. So let’s get started.

Activity Log Based Alert

As the name suggest these alerts are based on Azure Activity Log. You can find more information on Activity log alerts. Azure Activity Log is something you get for free. The activity logs are stored for free for 90 days and any write operations (POST, PUT, DELETE) will be visible there. The operations are scoped per subscription and recently per management group as now you have some actions happening on management group level. The activity log can be send to other stores including Log Analytics. When you create activity log alerts you are directly tapping into the activity log store so there is no need to send them to Log Analytics if you do not want to. As the activity log has different type of operations it is actually divided into 7 event categories that can be seen in the Azure Portal UI as shown below.

So when you create activity log alert it is always best practice to first scope it to specific category. Based on the category scope you are basically creating different sub-type alert. The reason for that is because also the information or data that you get into e-mail or other system via webhook will differ between those. It is important to know that each alert instance that will be created is per log record. So even if you have some log records that are registered at the same time you will have separate alert created for each record. Let’s deep dive into each sub-type now.

Administrative Activity Log Alert

The administrative category is basically for any write action you take on Azure resources. For example you can alert upon if Virtual Machine gets deleted.

Security Activity Log Alert

Alerts that are generated by Azure Security Center appear as record on activity log as well. The built-in ASC alerts are not very well integrated with Azure Monitor and because of that every ASC alert instance is logged to activity log. To put it in short for the built-in ASC alerts you will have to create another alert just to be able to send it properly to other system. The ASC capabilities for notifications are scoped to just only e-mails on high severity security alerts. It is not something you should use for production. Keep in mind that ASC integrates with other systems like Microsoft Defender ATP, etc. The alerts that come from these systems will also appear in activity log under security category. This is probably true for Azure Sentinel alerts which seems to be going in the same direction where you do not have native Azure Monitor Alerts integration but rather something custom like starting logic app from alert manually. I do not like that approach and I haven’t explored Azure Sentinel fully due to not being able to figure out what will stay in ASC and what will be in Azure Sentinel.

Service Health Alerts

From time to time Azure like any other cloud provider may have outages or degradation on certain services. Usually those issues appear for specific service, region and sometimes even parts of the region. Service Health will basically figure out what resources (services) you have deployed and when there is some outage or degradation will signal if they are affected. The signal appears as activity log under Service Health category. Service Health alerts are free so there is no charge when you scope your alerts to that category.

Alert Activity Log Alert

This is a new category that appeared recently. Usually you will not create alerts for this category. For now I have seen entries in this category related when activity log alert was activated.

Advisor Alerts

Azure Advisor is a service that gives you recommendations for your resources. Those recommendations are logged as activity log under Recommendation category. You can create activity log alerts scoped to that category to get alerted on new recommendations.

Policy Alerts

With Azure Policy you can see if your resources are compliant or not based on specific rules. Any audit events successful or not will appear under Policy category in activity log. You can create alerts that will signal on non-compliant resources.

Autoscale Alerts

Some resources in Azure have the ability to configure autoscale schedules. When that is configured and the resource scales up or scales down that is logged in the activity log under Autoscale cateogry. You can monitor your autoscale operations by creating alerts.

Resource Health Alerts

Resource Health is fairly new feature that is present on some resource. Resource health gives you basic information if your resource is working on not. This differs from Service Health as your resource for example VM might not be running due to BSOD or some other reason without Azure VM service having some outage. With alerts configured for this category you can alert upon such issues.

Metric Alerts

The next type of alert is metric alert. By default Azure provides certain metrics for almost all Azure services. The metrics vary depending on the service but they are stored for free for 90 days. You can also send the metric to Log Analytics but when you create metric alert you are not required as the metric alerts are taking the information directly from that metric store. There is also the possibility to send custom metrics to that metric store but than you pay for the stored data. In later blog series we will have a look closer at these alerts and the latest innovations there. It is important to mention that alerts created for Availability Tests in Application Insights underneath are metric alerts.

Azure Monitor for VMs Alerts

These alerts are quite new. Azure Monitor for VMs (sometimes even known as Azure Monitor VM Insights) is a new solution which is still in Preview at the time of writing this blog post. This new solution introduces new concept called Health. Health has some pre-defined rules to monitor Azure VMs. Such rules are for example High CPU Usage, DNS client service stopped, etc. When such issues occur alerts are logged in Azure Monitor. Basically think of this like pre-created alerts. By default these alerts do not have any actions assigned but you have capability to do it. Additionally you can also tweak these alerts to some degree.

Azure Monitor Log Alerts

These are the alerts that we create based on data coming from Log Analytics workspace or multiple Log Analytics workspaces. The alerts are based on having Kusto query language query that produces results. Based on those results you get alerted or not. For this alert type we also have two sub-types

• Number of results – In short for multiple results you get a signal alert. This is usually not the most desired result for most organizations as if you create some alert for CPU utilization you do not want to get one alert with 10 servers in it as those servers might belong to different teams for example.
• Metric Measurement – With this sub-type you get alert instance per result from Kusto query. Although this was initially created for creating alerts based on performance data in Log Analytics it could easily be used for event based data with some tricks.

These alerts are also used in Application Insights. Basically Log Analytics and Application Insights beneath has the same technology. Even the API for creating these alerts is the same. You can even do cross workspace and Application Insights instance alert that queries data from both. The only difference is in the webhook payload that is being send in case you integrate with action group. Even there the differences are small as the results are structured the same for both.

Cost Management Alerts

Cost Management recently released functionality that allows you to create alert when you create a budget. Technically these are not Azure Monitor alerts but they have some initial integration by being able to attach the alerts to action groups.

Azure Sentinel Alerts

Azure Sentinel is using Log Analytics workspace as platform for storing data. Azure Sentinel provides functionality for creating alerts based on that data but technically these are not Azure Monitor alerts. What is odd that for creating these alerts not even the Azure Monitor Log Alerts API is not used but instead a new one is created. Because of that there is also no integration with action groups. I never figured out why they basically decided to re-invent the rule instead of having the same API and may be having some sub-type in it in order to accommodate for the differences they want to make. Because of that we loose critical integration with action groups for example and instead there is manual integration with logic apps which is just one of the integrations available in action groups.

I hope you find this first part useful and hopefully I will be able to write soon the next parts.