Azure Monitor: The Enterprise Guide to Full-Stack Observability, Log Analytics, and KQL
You cannot manage what you cannot observe. Azure Monitor provides the complete observability stack for cloud-native and hybrid enterprise environments — collecting metrics, logs, traces, and dependencies from every layer of the infrastructure and application stack. This guide covers enterprise Azure Monitor architecture, Log Analytics workspace design, Application Insights for APM, alert strategy, KQL query patterns, Azure Workbooks, and cost optimization — based on EPC Group's experience across 200+ Azure enterprise deployments.
The Azure Monitor Observability Stack
Enterprise observability requires three pillars: metrics (what is happening right now), logs (what happened and why), and traces (how requests flow through distributed systems). Azure Monitor provides all three pillars in a unified platform with a common query language (KQL), common alerting engine, and common visualization layer.
Azure Monitor Observability Architecture
┌──────────────────────────────────────────────────────────────┐
│ Data Sources │
│ ├── Azure Resources (PaaS metrics, diagnostic logs) │
│ ├── Virtual Machines (Azure Monitor Agent) │
│ ├── Applications (Application Insights SDK / auto-instrument) │
│ ├── Containers (Container Insights / Prometheus) │
│ ├── Networks (Network Watcher, NSG flow logs) │
│ ├── On-premises (Arc-enabled servers, gateways) │
│ └── Custom sources (REST API, custom logs) │
└──────────────────────┬───────────────────────────────────────┘
│ Data Collection Rules (DCR)
┌──────────────────────▼───────────────────────────────────────┐
│ Data Platform │
│ ├── Metrics Store (time-series DB, 93-day retention) │
│ ├── Log Analytics Workspace (KQL-queryable logs, configurable)│
│ └── Change Analysis (resource configuration change tracking) │
└──────────────────────┬───────────────────────────────────────┘
│
┌──────────────────────▼───────────────────────────────────────┐
│ Analysis & Action │
│ ├── Metrics Explorer (real-time charts and dashboards) │
│ ├── Log Analytics (KQL query editor) │
│ ├── Application Insights (APM dashboards, smart detection) │
│ ├── Alerts (metric, log, activity log, smart detection) │
│ ├── Workbooks (interactive reports and dashboards) │
│ ├── Azure Dashboards (portal-level dashboards) │
│ └── Grafana (Azure Managed Grafana integration) │
└──────────────────────────────────────────────────────────────┘At EPC Group, our Azure cloud consulting practice implements Azure Monitor as part of every enterprise Azure deployment. Observability is not an afterthought — it is a foundational layer that must be designed alongside the Azure Landing Zone architecture.
Log Analytics: Architecture and Workspace Design
Log Analytics workspaces are the central data stores for Azure Monitor Logs. All log data — resource diagnostic logs, VM operating system logs, application logs, security events, and custom data — flows into Log Analytics workspaces where it is indexed, queryable via KQL, and retained for the configured period.
Workspace Architecture Patterns
- Centralized workspace (recommended): One workspace per region collects data from all resources. This enables cross-resource correlation (trace a problem from the application through the VM to the network), simplifies access control (RBAC at the workspace level with resource-context access for granularity), and qualifies for commitment tier pricing. Most enterprises use 1-2 workspaces (one per primary Azure region).
- Dedicated security workspace: A separate workspace for Microsoft Sentinel (SIEM) data, accessible only by the security operations team. This provides data isolation for security logs, independent retention policies (security data may need 1-2 year retention for compliance), and separate cost management. The security workspace ingests Azure AD sign-in logs, firewall logs, NSG flow logs, and Defender alerts.
- Workspace per environment (dev/staging/prod): Some organizations create separate workspaces for development and production to prevent development noise from polluting production monitoring dashboards and to enable cost chargeback by environment. However, this sacrifices cross-environment correlation.
Avoid Workspace Sprawl
The most common Azure Monitor architecture mistake is creating too many workspaces — per application, per team, or per subscription. This creates data silos, prevents cross-resource correlation, disqualifies commitment tier pricing (pricing applies per workspace), and increases management overhead. Consolidate to the minimum number of workspaces that meets your security isolation, compliance, and cost allocation requirements. EPC Group has migrated multiple enterprise environments from 20+ fragmented workspaces to 2-3 consolidated workspaces, reducing monitoring costs by 40% while improving observability coverage.
Data Retention and Archiving
Log Analytics provides three retention tiers: interactive retention (data is immediately queryable via KQL, first 31 days free, then $0.10/GB/month), long-term retention (data is archived and queryable with restore or search jobs, $0.02/GB/month), and export to storage (continuous export to Azure Storage for indefinite retention at blob storage pricing). Configure table-level retention policies: high-value tables like SecurityEvent and AppExceptions may need 365 days of interactive retention, while high-volume tables like Heartbeat or Perf may only need 31 days of interactive retention with 90 days of archive.
Data Collection Rules and Azure Monitor Agent
Data Collection Rules (DCRs) are the modern mechanism for controlling what data is collected, how it is transformed, and where it is sent. DCRs replace the legacy workspace-based agent configuration and provide granular control over data ingestion — critical for cost optimization.
Azure Monitor Agent (AMA)
The Azure Monitor Agent is the unified agent for collecting telemetry from Windows and Linux VMs (Azure, on-premises via Arc, and other clouds). It replaces the legacy Log Analytics Agent (MMA/OMS), Diagnostics extension, and Telegraf agent. AMA uses DCRs for configuration, supports multi-homing (send data to multiple workspaces), and provides improved security (managed identity authentication, no stored keys).
- Performance counters: CPU, memory, disk, and network metrics at configurable intervals. Collect at 60-second intervals for capacity planning; use 10-second intervals only when troubleshooting specific performance issues to avoid excessive data volume.
- Windows Event Logs: System, Application, and Security event logs. Filter by event level (Error, Warning, Critical) to exclude informational events that add volume without value. Always collect Security events for compliance audit trails.
- Syslog (Linux): Collect syslog messages from Linux VMs. Filter by facility and severity to control volume — collect auth and authpriv at all levels, but only warning and above for daemon and kern.
- Custom logs: Collect custom text log files or JSON log files from applications. Define the file path pattern and parsing rules in the DCR. This enables monitoring of legacy applications that write to flat log files.
Data Transformation in DCRs
DCRs support KQL-based transformations that filter, aggregate, or modify data before it reaches the workspace. This is one of the most impactful cost optimization features — you can drop unnecessary columns, filter out noisy log entries, or aggregate high-frequency data at the collection point, reducing ingestion volume by 30-60% without losing operational value.
Application Insights: Application Performance Monitoring
Application Insights is the APM (Application Performance Monitoring) component of Azure Monitor. It collects telemetry from web applications: requests, dependencies (database calls, HTTP calls, Azure service calls), exceptions, traces, page views, and custom events. Application Insights provides distributed tracing across microservices, smart detection of anomalies, and performance diagnostics.
Instrumentation Options
- Auto-instrumentation (recommended for Azure PaaS): Enable Application Insights on Azure App Service, Azure Functions, or AKS without code changes. Azure automatically instruments the application runtime and collects telemetry. Supported runtimes: .NET, Java, Node.js, and Python.
- SDK instrumentation: Add the Application Insights SDK to your application code for maximum control. The SDK enables custom telemetry (TrackEvent, TrackMetric, TrackTrace), dependency tracking, and request correlation. Use SDK instrumentation for applications with complex dependency chains or custom business metrics.
- OpenTelemetry: Application Insights supports OpenTelemetry, the vendor-neutral observability standard. Use the Azure Monitor OpenTelemetry Exporter to send traces, metrics, and logs from OpenTelemetry-instrumented applications to Application Insights. This is the recommended approach for multi-cloud or vendor-neutral observability strategies.
Key Application Insights Features
- Application Map: Visualizes the topology of your application — showing all components and their dependencies, call rates, failure rates, and latency between components. Essential for microservices architectures where understanding the dependency chain is critical for troubleshooting.
- Transaction Search: Search and inspect individual requests, dependencies, exceptions, and traces. Drill from a failed request to the exception, to the dependent service call that caused it, to the SQL query that timed out.
- Failures and Performance: Pre-built analysis views that aggregate failure rates by operation, exception type, and dependency, and identify performance bottlenecks by operation, dependency, and percentile distribution.
- Smart Detection: Machine learning that automatically detects performance anomalies (sudden increase in response time, failure rate spike, memory leak patterns) and sends proactive alerts. No configuration required — it learns normal behavior baselines and alerts on deviations.
- Availability tests: Synthetic monitoring that sends HTTP requests to your application from Azure data centers worldwide. Alerts when availability drops below threshold or response time exceeds limits. Use URL ping tests for basic availability and multi-step web tests for user flow validation.
Essential KQL Query Patterns for Enterprise
Kusto Query Language (KQL) is the query language for Log Analytics, Application Insights, Microsoft Sentinel, and Microsoft Defender. Proficiency in KQL is a core skill for Azure operations, security, and development teams.
Infrastructure Monitoring Queries
// VMs with CPU above 90% in the last hour
Perf | where TimeGenerated > ago(1h) | where ObjectName == "Processor" and CounterName == "% Processor Time" | where InstanceName == "_Total" | summarize AvgCPU = avg(CounterValue), MaxCPU = max(CounterValue) by Computer | where AvgCPU > 90 | order by AvgCPU desc
// Disk space below 10% free on any volume
Perf
| where TimeGenerated > ago(1h)
| where ObjectName == "LogicalDisk" and CounterName == "% Free Space"
| where InstanceName !in ("_Total", "HarddiskVolume1")
| summarize MinFreePercent = min(CounterValue) by Computer, InstanceName
| where MinFreePercent < 10
| order by MinFreePercent ascApplication Monitoring Queries
// Top 10 slowest API endpoints (95th percentile)
requests
| where timestamp > ago(24h)
| where success == true
| summarize P95_Duration = percentile(duration, 95),
AvgDuration = avg(duration),
RequestCount = count()
by name
| where RequestCount > 100
| order by P95_Duration desc
| take 10// Failed dependencies grouped by type and target
dependencies | where timestamp > ago(1h) | where success == false | summarize FailureCount = count(), AvgDuration = avg(duration) by type, target, resultCode | order by FailureCount desc
Cost Analysis Queries
// Data ingestion volume by table (last 30 days) for cost optimization
Usage | where TimeGenerated > ago(30d) | summarize IngestedGB = sum(Quantity) / 1024 by DataType | order by IngestedGB desc | take 20
Enterprise Alert Strategy
Alert fatigue is the enemy of effective monitoring. An enterprise alert strategy must balance coverage (detecting real issues) with noise reduction (suppressing non-actionable alerts). EPC Group implements a tiered alert framework for every Azure deployment.
Alert Severity Framework
| Severity | Criteria | Notification | Response Time |
|---|---|---|---|
| Sev 0 - Critical | Production outage, data loss, security breach | Phone call + SMS + email + Teams | 15 minutes |
| Sev 1 - Error | Service degradation, high error rate, capacity threshold | SMS + email + Teams channel | 1 hour |
| Sev 2 - Warning | Performance degradation, disk space low, certificate expiring | Email + Teams channel | 4 hours |
| Sev 3 - Informational | Autoscale event, deployment completed, backup success | Teams channel only | Next business day |
Alert Best Practices
- Alert on symptoms, not causes: Alert when users are impacted (response time exceeds SLA, error rate spikes) rather than on infrastructure metrics alone (CPU high). High CPU is only a problem if it impacts user experience.
- Use dynamic thresholds: Azure Monitor supports machine-learning-based dynamic thresholds that learn normal behavior patterns and alert on deviations. This eliminates the need to manually set and maintain static thresholds for every metric.
- Suppress during maintenance: Use alert processing rules to suppress notifications during planned maintenance windows. This prevents false alerts during deployments, patching, and planned failovers.
- Auto-remediation: Connect action groups to Azure Automation runbooks, Logic Apps, or Azure Functions to automatically remediate common issues. For example, auto-restart a stopped VM, auto-scale a resource, or auto-clear a temp folder when disk space is low.
Azure Workbooks: Interactive Dashboards
Azure Workbooks are interactive reports that combine KQL queries, metrics, parameters, and visualizations into shareable dashboards. Workbooks are the recommended visualization tool for Azure Monitor data because they support dynamic parameters, conditional visibility, and drill-down from summary to detail.
- Built-in workbook gallery: Azure Monitor includes 100+ pre-built workbooks for common scenarios: VM performance, AKS cluster monitoring, Application Insights overview, Key Vault diagnostics, and more. These provide immediate value and can be customized for your environment.
- Custom workbooks: Build custom workbooks combining KQL queries (tables, charts, tiles), Azure Resource Graph queries (resource inventory), and parameters (time range, subscription, resource group selectors). Use conditional visibility to show different sections based on parameter selections.
- Azure Managed Grafana: For teams already using Grafana, Azure Managed Grafana provides a fully managed Grafana instance with native Azure Monitor and Azure Data Explorer data source integration. This enables Grafana dashboards with KQL queries against Log Analytics data. See our Azure cost optimization guide for monitoring cost management with dashboards.
Cost Optimization for Azure Monitor
Azure Monitor costs are directly proportional to data ingestion volume. Without optimization, monitoring costs can grow unexpectedly as workloads scale. EPC Group implements these cost controls for every enterprise deployment:
Cost Reduction Strategies
- Commitment tiers: If your workspace ingests more than 100 GB/day, switch from pay-as-you-go to a commitment tier. The 100 GB/day tier provides a 38% discount; the 500 GB/day tier provides a 50% discount. Review your Usage table to determine the right tier.
- Data collection rules (filtering): Use DCR transformations to filter unnecessary data at the collection point. Drop verbose informational logs, exclude health check endpoints from Application Insights, and filter performance counters to only the metrics you actually monitor.
- Table-level retention: Configure shorter interactive retention for high-volume, low-value tables (Heartbeat, Perf, ContainerLog) and longer retention for high-value tables (SecurityEvent, AppExceptions, AuditLogs). Archive data to long-term retention ($0.02/GB/month vs. $0.10/GB/month) for data needed for compliance but not frequently queried.
- Sampling in Application Insights: Enable adaptive sampling to reduce telemetry volume from high-traffic applications while maintaining statistically accurate metrics. Adaptive sampling automatically adjusts the sampling rate based on telemetry volume, targeting a configured maximum events-per-second rate.
- Basic logs tier: For high-volume tables that are rarely queried (verbose diagnostic logs, container stdout logs), configure the Basic Logs tier at $0.50/GB ingestion (vs. $2.76/GB for Analytics tier). Basic Logs have limited KQL capabilities and 8-day retention but are ideal for data that is only queried during troubleshooting.
Hybrid and Multi-Cloud Monitoring
Most enterprises operate hybrid environments with workloads distributed across Azure, on-premises data centers, and potentially other cloud providers. Azure Monitor extends observability beyond Azure through Azure Arc, the Azure Monitor Agent, and Azure Managed Grafana.
- Azure Arc-enabled servers: Install the Azure Connected Machine agent on on-premises or multi-cloud VMs to project them into Azure as Arc-enabled resources. This enables Azure Monitor Agent installation, DCR-based data collection, and centralized monitoring through the same Log Analytics workspace that monitors Azure VMs.
- Azure Arc-enabled Kubernetes: Onboard non-Azure Kubernetes clusters (EKS, GKE, on-premises) to Azure Arc for centralized monitoring with Container Insights. This provides a single pane of glass for all Kubernetes workloads regardless of where they run.
- Network monitoring: Use Network Watcher for Azure network diagnostics, Connection Monitor for hybrid connectivity monitoring (Azure-to-on-premises latency, VPN tunnel health), and Traffic Analytics for NSG flow log analysis. For end-to-end network visibility, integrate with our Azure cloud services practice.
Partner with EPC Group
EPC Group was a Microsoft Gold Partner (2003-2022, the oldest in North America), and is currently a Microsoft Solutions Partner with 29 years of enterprise cloud infrastructure expertise. Our Azure cloud practice implements comprehensive observability solutions — from Log Analytics workspace architecture and data collection optimization through custom workbook development, alert strategy design, and ongoing monitoring operations. We specialize in hybrid enterprise environments where observability must span Azure, on-premises, and multi-cloud workloads.
Frequently Asked Questions
What is Azure Monitor and what does it include?
Azure Monitor is the comprehensive observability platform for Azure and hybrid environments. It collects, analyzes, and acts on telemetry data from cloud and on-premises resources. Azure Monitor includes: Log Analytics (log collection and KQL querying), Application Insights (application performance monitoring/APM), Metrics Explorer (real-time metric visualization), Alerts (proactive notification and automated response), Workbooks (interactive dashboards and reports), Azure Monitor Agent (data collection from VMs), and Network Insights (network monitoring and diagnostics). Together, these components provide full-stack observability across infrastructure, applications, and networks.
How much does Azure Monitor cost?
Azure Monitor pricing is primarily based on data ingestion and retention. Log Analytics charges approximately $2.76/GB ingested (pay-as-you-go) with the first 5 GB/month free. Commitment tiers provide discounts: 100 GB/day tier is ~$1.70/GB (38% savings). Data retention is free for the first 31 days; additional retention costs $0.10/GB/month. Application Insights charges the same Log Analytics ingestion rates since it stores data in a Log Analytics workspace. Alert rules cost $0.10-$1.50/rule/month depending on type. For most enterprise deployments (ingesting 50-200 GB/day), expect $4,000-$16,000/month. Using commitment tiers and data collection rules to filter unnecessary data reduces costs by 30-50%.
What is the difference between Azure Monitor Logs and Azure Monitor Metrics?
Azure Monitor Metrics are lightweight, time-series numerical values collected at regular intervals (typically 1-minute granularity). They are stored in a time-series database optimized for fast queries and real-time dashboards. Examples: CPU percentage, memory usage, request count, response time. Metrics are ideal for real-time alerting and dashboards. Azure Monitor Logs are detailed, structured or semi-structured records stored in Log Analytics workspaces. They include event logs, trace logs, performance counters, custom application logs, and security events. Logs are queried using KQL (Kusto Query Language) and support complex analysis, correlation, and long-term retention. Use Metrics for real-time monitoring and simple threshold alerts; use Logs for investigation, troubleshooting, and complex analytics.
What is KQL (Kusto Query Language) and why is it important?
KQL (Kusto Query Language) is the query language for Azure Monitor Logs, Microsoft Sentinel, Microsoft Defender, Azure Data Explorer, and Microsoft Fabric Real-Time Analytics. It is a read-only language optimized for querying large volumes of semi-structured telemetry data. KQL uses a pipe-based syntax where data flows through a series of operators: source table | filter | transform | aggregate | sort | render. For example: "Heartbeat | where TimeGenerated > ago(1h) | summarize count() by Computer | order by count_ desc" shows which VMs have reported in the last hour. KQL proficiency is essential for any Azure operations or security team because it is the query language for all Azure observability and security platforms.
How do I set up alerting in Azure Monitor?
Azure Monitor alerts consist of three components: the alert rule (defines the condition), the action group (defines who gets notified and what automated actions to take), and the alert processing rule (optional, controls notification routing and suppression). To create an alert: define the scope (resource, resource group, or subscription), select the signal type (metric, log, or activity log), configure the condition (threshold, frequency, aggregation), attach an action group (email, SMS, webhook, Logic App, Azure Function, ITSM), and set the severity (0-Critical through 4-Informational). Best practice: use log-based alerts for complex conditions (KQL queries) and metric-based alerts for simple threshold monitoring.
What is the recommended Log Analytics workspace architecture for enterprise?
For most enterprises, a centralized Log Analytics workspace architecture is recommended: one primary workspace per Azure region for all resources in that region, with Azure Lighthouse for multi-tenant scenarios. This simplifies cross-resource correlation, reduces management overhead, and enables cost optimization through commitment tiers. Exceptions that warrant separate workspaces include: compliance requirements that mandate data sovereignty (different workspaces per country/region), security isolation (a dedicated workspace for security logs accessed only by the SOC team), and cost allocation (separate workspaces for development and production when chargeback is required). Avoid per-resource or per-application workspaces — this approach creates data silos and prevents cross-resource correlation.