Abstract
English is not my mother tongue. So often it takes a bit
of time for myself to understand few English statements; if it does not contain
regular words. Honestly “PURGE” was one of such word where I had to search for
dictionary.
//PURGE//
Means: To purify, getting rid of whatever that is impure
or undesirable and most important: when you want to start fresh.
How this is related in blog post? Well read on…
What is an APM tool?
One of my customers wanted to do application monitoring during
load test. They started configuring monitoring for their Virtual machines
hosting applications and database for CPU, Memory, network I/O, disk I/O and so
on. Configuration of their monitoring showed them all metrics of VMs like CPU,
memory etc. within 70% however application was failing consistently in load
test.
Then they called me and explained the same scenario and
said application is failing but there is no trace we are getting why it is
failing. So I told them, what you are doing currently is good however you are
doing only underlying “Infrastructure” monitoring. This is good but it can’t
tell always why application is failing. So you need “Application performance monitoring
(APM)” tool for application monitoring.
Folks, application monitoring and infrastructure
monitoring are two different things. If your VM is not healthy then for sure
your application will not work however if your application is not working still
your Virtual Machine might be healthy. Got it?
So when you say I am monitoring CPU, memory for
application you are actually monitoring only underlying infrastructure capacity/
performance and not the application behavior. To monitor application means you
want to understand -
- Application
failures
- Exceptions
and error raised
- User
telemetry and usage
- Availability
of application
- Live
metrics
- Response,
time, latency and application page load time
- Application
load time in client browser
- Dependencies,
slowest operation performed and so on.
This has nothing to do with your underlying
infrastructure performance metrics monitoring.
So you need a tool that can show all above mentioned
things in an application and this is where Application performance monitoring
APM tools are used.
There are many APM tools in the market like NewRelic,
Dynatrace, App Dynamics and “Azure Application Insights”.
So APM is generic concept and Application Insights is a
product from Azure for APM.
Customer Scenario
Some customer can be real pain in the …heart [I know what
you were thinking].
Customer configured application insights agent and SDK
both to monitor various aspects of their few java and .NET based applications for
POC. After running the POC for 15 days they were quite satisfied by application
insights features. They wanted to start using the same application insights
instance in production. However they wanted to start a fresh and hence wanted
to delete all the data captured by application insights till now. At the same
time they also wanted to retain the application insights service and post data
deletion they wanted to start capturing production data in the same service
instance of application insights.
Options for deleting application insights data
There 3 options to delete the application insights data –
Option 1 – Delete application insights
I know this is one of the crude ways of doing it. When
you delete application insights everything contained inside of it will be
deleted. This option would not fit into our scenario described above. When you
create a new application insights unfortunately you will have to redo the changes
of application insights key and setting again in the application wherever it is
referenced. My customer did not want to redo the settings. Customer is a lazy…heart
[I know what you were thinking].
Option 2 – Automatic deletion of data
This is more of configuring the data retention policy in
such a way that it deletes all the data over a time period. Default duration of
data retention is 90 days. So post 90 days all the data stored in application
insights is automatically removed. Also Application insights is charged as per the
amount of storage in GBs you are consuming. So it makes sense to store captured
data only for limited period and then delete it automatically. If you want to
store data longer than 90 days then one way is to do continuous export of
application insights data to Azure Blob Storage. Storage cost is cheaper however
you will not be able to query it using application insights UI. It will be just
a dump of data collected. So good for showing audit and compliance teams that
you are maintaining the data but it is not of much use.
The other option to retain beyond 90 days is preview
feature Microsoft is working on. Through this feature you would be able to even
store only till 30 days to 730 days. Refer
- https://feedback.azure.com/forums/357324-application-insights/suggestions/17454031
In our above scenario of our customer this will not help;
as to move into production customer will have to wait for 90 days.
Option 3 - PURGE
Microsoft Azure launched REST APIs for application
insight that can help you to delete/ remove/clear/ Purge data stored in application
insights based on filters you specify. This will exactly fill into our
scenario. Let us build those apis and
use it for deleting the application insights data.
Authorization token and Postman
For calling PURGE REST API I am going to use Postman. “Purge
application insights data” is a management operation. Hence we will need proper
authorized token for calling the APIs. This is the time where you should subscribe
to my blog. Because recently I wrote a blog post on how to use Azure AD service
principal for invoking the management API of Azure. Refer - https://sanganakauthority.blogspot.com/2019/04/how-to-create-service-principal-or-app.html
PURGE APIs
https://management.azure.com/subscriptions/YourSubscriptionID/resourceGroups/YourResourceGroup/providers/Microsoft.Insights/components/ApplicationInisghtsservice/purge?api-version=2015-05-01
This is post method. Add the body as below –
{
"table": "customEvents",
"filters": [
{
"column": "timestamp",
"operator": ">",
"value":
"2019-01-01T00:00:00"
}
]
}
In the above body we are specifying the table name and
timestamp filter. Means all the data collected after 01/01/2019 in customEvents
table will be deleted. So for deleting data from all tables you will need run
the API for those many tables with required date filter. Refer below
screenshots for setting up Params, Headers and Body for PURGE operation in
postman –
Where I can find other application insights tables?
Open your application insights in Azure portal. Then
click on “Analytics” option as shown below –
Left hand side blade shows all the tables of application
insights –
Status of PURGE operation
The above API starts the application insights data
deletion operation. The operation returns a GUID as shown below –
This id can be used to know status of PURGE operation. The
delete of application insights data will take upto 72 hours. To know the status
of PURGE operation refer to below REST API –
https://management.azure.com/subscriptions/YourSubscriptionId/resourceGroups/YourResourceGroupName/providers/Microsoft.Insights/components/YourApplicationInsights/operations/purge-_GuidReturnedbyPURGEoperation?api-version=2015-05-01
This is GET operation and will also require
Authorization header token. After 72 hours you will find that the data has been
deleted.
Conclusion
I hope this post have helped you to purge azure
application insights data.
Please provide your valuable comments.
Keep Purging!!
Great article...
ReplyDeleteHelped me a lot, thank you!
ReplyDelete