Monitoring AWS Machine using Azure Log Analytics - Cloud Computing

What is Azure Log Analytics?

Logs make your life easier and help everyone in the organization understand what you do and what's Actually going on with the software. Ensure that you follow best practices of Azure Log Analytics for monitoring AWS machines effectively.

Logs are created by network devices, applications, operating systems, and programmable or smart devices. They comprise several messages that are chronologically arranged and stored on a disk, in files, or in an application like a log collector. To know more, check out Cloud Computing certificate.

Firstly, we will try to understand why we need logs to monitor AWS services.

We need logs because, with Azure Log Analytics, you can gather and search log data from all your resources, whether they reside on Azure, on another cloud host, or on-premises. Then you can transform the gathered data into rich analytics with AI-enhanced insights into your environment.

How Log Analytics works

The log data from Virtual machines and other cloud resources, capture via some Agent which we need to install on VMs and Resource can be named as Connected Sources which further will be gathered in records and sent to OMS repository that is the part of Azure Cloud resource and stored in the Azure cloud.

After receiving logs to log analytics it further is available for log search and other activity like Set Alerts, Dashboard View, Power BI View, and Export into Excel and Data files.

Data collection in Azure Log Analytics

Data can be collected in azure Log Analytics in different ways –

• Agents on Windows and Linux virtual machines send telemetry from the guest operating system and applications to Log Analytics according to Data Sources that you configure.
• Connect a System Center Operations Manager management group to Log Analytics to collect data from its agents.
• Azure services such as Application Insights and Azure Security Center store their data directly in Log Analytics without any configuration.
• Write data from PowerShell command line or Azure Automation runbook using Log Analytics cmdlets.
• If you have custom requirements, then you can use the HTTP Data Collector API to write data to Log Analytics from any REST API client.

• Azure Monitor                   Data sources for events and performance
• Virtual Machines               Data sources for events and performance
• Operations Manager         Data from management group agents
• Application Insights           Application requests and exceptions
• Azure Security Center       Security events
• PowerShell                        PowerShell command line or runbook
• Data Collector API            Rest API for custom data

Incoming data automatically indexed. Data types and tables automatically created.

Log Analytics

Data available through log search and smart analytics to multiple channels.

• Design and test queries and analyze data         Analytics
• Visualize data in Azure portal                            Dashboards
• Workflows consuming Log Analytics data         Logic Apps
• Automatically respond critical conditions           Alerts
• Export for visualization with other sources        Power BI
• PowerShell command line or runbook               PowerShell
• Rest API for custom application                         Log Search API

Work flow of Log Analytics

Now let's find out the workflow of log analytics like how it collects data, analyze, visualize and alerts.

So first look into Collect part–

The logs can be collected via: -

• Event Logs
• Custom App Logs
• IIS Logs
• Crash Dumps
• Performance Data

We can also filter the type of logs by just checking and unchecking the log types.

Once the collection has done we need to Analyse the data which can be done by: -

• Filter based on attributes
• Analyze data with Kusto Query language
• Sort data
• Export log data to Excel and Power BI
• Conditional Filtering

After that, we can Visualize all the logs in an attractive Dashboard

Alerts can be configured on the basis of event conditions like once the value will be reached greater than or less than the limit threshold it will automatically generate the alerts and take configured actions like sending main, sending a message and start runbook (part of azure automation).

Steps to collect Data and log from AWS Machines

AWS VMs could be of Windows and Linux, so we can install the agent which can be download from azure log analytics page by just selecting the type of the OS and the Bit of the VMs.

After installing the Agent we need to configure the agent by entering the workspace id and key which will be provided by the azure after configuring. We can get the all the logs within some Hours in the portal.

Supported AWS OS and Versions

Let’s discuss the OS and versions of VMs that are supported on AWS Cloud.

If you are using Windows, the configuration should be:

• Windows Server 2008 Service Pack 1 (SP1) or later
• Windows 7 SP1 and later

If you are using Linux, the configuration should be:

• Amazon Linux 2012.09 to 2015.09 (x86/x64)
• CentOS Linux 5, 6, and 7 (x86/x64)
• Oracle Linux 5, 6, and 7 (x86/x64)
• Red Hat Enterprise Linux Server 5, 6 and 7 (x86/x64)
• Debian GNU/Linux 6, 7, and 8 (x86/x64)
• Ubuntu 12.04 LTS, 14.04 LTS, 16.04 LTS (x86/x64)
• SUSE Linux Enterprise Server 11 and 12 (x86/x64)

After verifying the supported configuration, we can successfully install the agent and receive the logs.

Kusto to query AWS Machine Logs

Kusto is a log analytics cloud platform optimized for ad-hoc big data queries.

Kusto Query Reference Portal: https://docs.loganalytics.io

The Kusto Query Language is used to query Azure services.

OMS (Operational Management Suite)

The Microsoft Operations Management Suite (OMS), previously known as Azure Operational Insights, is a software as a service platform that allows an administrator to manage on-premises and cloud IT assets from one console.

Azure OMS provides 4 types of services: -

• Log Analytics: Monitor and analyze the availability and performance of different resources including physical and virtual machines.
• Automation: Automate manual processes and enforce configurations for physical and virtual machines.
• Backup: Backup and restore critical data.
• Site Recovery: Provide high availability for critical applications.

Management Solutions

Management solutions leverage services in Azure provides additional insight into the operation of a particular application or service.

Here, we can manually select the required solution and add to the home page where we can only see the logs of that solution type.

Reports

The final reports of logs can be exported in Excel and Power BI that can be displayed in table and chart format.

Conclusion:

Azure Log Analytics is a very powerful tool to capture different types of system log. Kusto Query plays a very important role in extracting insights from the log file. Also, custom reports can be prepared using KUSTO Query,
which helps the organization in saving many man-hours.

Detailed reports and easy to export to excel and power helps to keep the troubleshooting and diagnosis handy.

Reports can be embedded in any website with live refresh data and code snipped can be generated within power BI.

Choose the right Azure logging service for AWS monitoring and use it to save many man-hours and reduce the time of troubleshooting and diagnosis.

Towards the end, I hope that you have gained a clear understanding of What is Azure Log Analytics and how Monitoring AWS Machine using Azure Log Analytics take place.

If this article has helped you clear your fundamentals and if you wish to learn more about it by getting certified, then you can undertake the Microsoft Azure Fundamental certification course offered by KnowledgeHut.