Diagnose webtraffic in Azure using KQL and Log Analytics

Diagnose webtraffic in Azure using KQL and Log Analytics

In an older post we discussed the benefits of using and protecting your web applications with Azure Web Application Firewall, you can find the post in full here

We touched on some best practices which were summarized in the post:

  • Best Practices: I recommend using Detection mode rather than Prevention mode (actively blocks requests) initially to catch false positives and use diagnostics logging to review traffic in a Log Analytics Workspace.

Today we will expand on the topic of reviewing traffic logs using Log Analytics and dive into some KQL-queries.

Pre-requisites

First off you need a log analytics workspace and a web application firewall instance with diagnostic settings forwarding logs to said workspace. If you want to quickly deploy this I have created the resources needed in this repository

The following objects will be deployed into one Resource Group so we can easily delete everything afterwards:

In order to deploy the code you need Terraform and the Azure CLI installed on your machine.

az login
az account set -s <sub-id>
git clone git@github.com:carlzxc71/waf-diagnostic-deployment.git
cd /waf-diagnostic-deployment/terraform
terraform init
terraform apply -auto-approve

Additional configuration

Once we have deployed our resources there is some additional configuration that we need to complete.

  • In the Azure Portal search for Application Gateway and select our new load balancer
  • Select Diagnostic Settings and select Add diagnostic setting
  • Select Save

It will take a few minutes for logs to start to ingest into the workspace. I should also note that this post is not aimed at configuring a proper loadbalancer infront of a website type of guide. This is merely to deploy resources and provide you some configuration out of the box to get to play with actual KQL-queries and see why they are useful with WAF.

Browse the logs

So after a few minutes you should see some logs starting to populate in your workspace if you search for log-prod-we-webdemo and expand LogManagement

As you can see I have run a simple AGWFirewallLogs command and I can see that I have a match or hit for a rule.

Match means that the web application firewall has matched an incoming request with a rule in the managed ruleset, in this case OWASP. If we search for the RuleId which in this case is 920350 in the WAF we can see what rule it is:

In this case it is complaining about the host header being an IP address. This makes sense in our case as I browsed to the public IP of the load balancer to get to the application service but in a production application you would not want this.

Action Anomaly score means that this client IP is adding upp points to their anomaly score for when it reaches a certain threshhold where it has added up enough points the requests will be blocked. This means that occassional anomalies will not be blocked but repeated offenses gets taken care of.

Once the score is 5 or greather and your WAF is in prevention mode as we have deployed the request is blocked

So if you are investigating the logs then, what are some useful queries you can use? Well here are some I use.

To find requests coming from a specific IP where the action is Blocked (You could change to Matched as well)

AGWFirewallLogs
| where ClientIp startswith "<IP>"
| where Action startswith "Blocked"

To search for hits for specific rules

AGWFirewallLogs
| where RuleId startswith "<rule-ID>"

You can then combine the two above

AGWFirewallLogs
| where ClientIp startswith "<IP>"
| where Action startswith "Blocked"
| where RuleId startswith "<rule-ID>"

Find hits generated from certain RequestUri (Useful if certain functions in your app services is not working)

AGWFirewallLogs
| where RequestUri startswith "/images"

There you go with some useful tips and trix. As you can see these are some pretty basic but very powerful examples of queries you can leverage, just imagine what more complex things you can do.

Of course this is not limited to Web Application Firewall you have resource logs for a whole host of Azure Resources such as Firewall, Virtual Machines, Storage Accounts and so forth..

References

Protect your web apps in the cloud with Web Application Firewall
Introduction Web Application Firewall (WAF) is something we have touched on briefly before on Lindbergtech but it is time to write a bit more in-depth into this topic. WAF is a service within Azure that you can add ontop of your Application Gateway or Frontdoor deployments to provide a centralized

Read more about Managed Rule sets here

About me

About me
If you have landed on my page you will have already understood my passion for tech, but obviously there is more to life than that. Here I will try and outline a few of my other hobbies. Strength training I am a person who loves to move around and challenge