Azure Storage Analytics for Azure Storage –Blob, Queue and Tables - Logging example

Today I will be going to explain about turning on Logging and Metrics for Azure storage like Blob, Queue and Table. Logging and metrics are two parts of Storage Analytics in Windows Azure. I will be listing few code samples also.

Before going further I will highly recommend you to understand Need for Storage Analytics and What Storage Analytics is?.

Assuming you have gone through the above link and has a fair idea about storage analytics, let’s understand how Azure storage analytics can be used for azure storage.

Logging – Using logs to track storage request for Azure storage –

Create a sample cloud application with one web role in it. We will use this web role to test the azure storage analytics logging and metrics data.

To enable logging for azure storage we will first create a library which will contain all the necessary functions. The Microsoft azure storage dll - Microsoft.WindowsAzure.StorageClient does not have any specific method related to Storage analytics logging and metrics. Therefore it is mandatory for us to write extensions methods for all the storage services. MSDN blog has an excellent article listed here along with code. Create a library project with suitable name and copy the code for classes, AnalyticsSettings, AnalyticsSettingsExtensions, SettingsSerializableHelper. Just for sake of my understanding I have changed the name of class AnalyticsSettings to StorageAnalyticsSettings.

So your final project in solution explorer will look as follows along with library.

Add reference of storage extension library to web role. Now I have added a page named as Analytics.aspx in my web role to test the storage analytics library.

Add using statement for following libraries in addition to the existing one –

using System.Configuration;

using Microsoft.WindowsAzure.StorageClient;

using Microsoft.WindowsAzure.ServiceRuntime;

using Microsoft.WindowsAzure.Diagnostics;

using Microsoft.WindowsAzure;

using[your storage library name]AzureStorageExtensions;

using System.IO;

I have added a button named as Set Service Settings. On click of it I will be applying all the Azure Storage Analytics settings for Azure storage service.

protected void btnSetServiceSettings_Click(object sender, EventArgs e)

{

StorageCredentialsAccountAndKey storageCredentials = new StorageCredentialsAccountAndKey("YourStorageAccountName", "YourStorageAccountKey");

//alternatively you can read above key and account name from configuration file web.config of your web role project.

CloudStorageAccount storageAccount = new CloudStorageAccount(storageCredentials, true);

    CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();

    CloudTableClient tableClient = storageAccount.CreateCloudTableClient();

    CloudQueueClient queueClient = storageAccount.CreateCloudQueueClient();

StorageAnalyticsSettings settings = new StorageAnalyticsSettings()

{

//to TuRn on logging AND METRICS

LogType = LoggingLevel.Delete | LoggingLevel.Read | LoggingLevel.Write,

IsLogRetentionPolicyEnabled = true,//enable logging

       LogRetentionInDays = 4,

IsMetricsRetentionPolicyEnabled = true,//enable metrics

MetricsRetentionInDays = 7,

MetricsType = MetricsType.All

                //to turn off logging and metrics

                //LogType = LoggingLevel.None,               

                //IsLogRetentionPolicyEnabled = false,

                //IsMetricsRetentionPolicyEnabled = false,

                //MetricsType = MetricsType.None

};

//applying above settings to blob service service

StorageAnalyticsSettingsExtension.SetServiceSettings(blobClient, settings);

//applying above logging settings to table service

StorageAnalyticsSettingsExtension.SetServiceSettings(tableClient, settings);

//applying above logging settings to queue service            StorageAnalyticsSettingsExtension.SetServiceSettings(queueClient,storageAccount.QueueEndpoint, settings);

}

The settings above will get applied to blob storage service when above code is executed. Now If I want to retrieve the settings that are applied; you can use following code -

protected void btnGetServiceSettings_Click(object sender, EventArgs e)

        {

            StorageCredentialsAccountAndKey storageCredentials = new StorageCredentialsAccountAndKey("YourStorageAccountName", "YourStorageAccountKey");

//alternatively you can read above key and account name from configuration file web.config of your web role project.

            CloudStorageAccount storageAccount = new CloudStorageAccount(storageCredentials, true);

CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();

CloudTableClient tableClient = storageAccount.CreateCloudTableClient();

CloudQueueClient queueClient = storageAccount.CreateCloudQueueClient();

//retrieve blob settings

StorageAnalyticsSettings blobSettings = StorageAnalyticsSettingsExtension.GetServiceSettings(blobClient);

//retrieve table settings

StorageAnalyticsSettings tableSettings = StorageAnalyticsSettingsExtension.GetServiceSettings(tableClient);

//retrieve queue settings

StorageAnalyticsSettings queueSettings = StorageAnalyticsSettingsExtension.GetServiceSettings(queueClient, storageAccount.QueueEndpoint);

//alternatively you can write these settings to file or display in your web page or view while debugging in visual studio.

        }

Once you are done with above settings and verify that they are assigned to blob, queue and table with above get settings code, open up your storage account in any Cloud Storage Studio. I personally use Cerebrata as it is free for first 30 days. J

If you upload any file to blob storage or delete, or add an entity in table storage and delete and similarly if you perform any operation on Queue then these transactions will be recorded in Logging as per the current date and time.

You will see that, $logs container will have logs created under it as shown below -

To retrieve and show names of these log file I added a simple Listbox and populating all the logs in it on a button click. The code is as follows –

protected void btnListLoggingBlobs_Click(object sender, EventArgs e)

        {

            StorageCredentialsAccountAndKey storageCredentials = new StorageCredentialsAccountAndKey("YourStorageAccount", "YourStorageAccountKey");

            CloudStorageAccount storageAccount = new CloudStorageAccount(storageCredentials, true);

            CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();

            CloudBlobContainer container = blobClient.GetContainerReference("$logs");

            BlobRequestOptions blobRequest = new BlobRequestOptions();

            blobRequest.BlobListingDetails = BlobListingDetails.All;

            blobRequest.UseFlatBlobListing = true;

            ResultSegment<IListBlobItem> result = container.ListBlobsSegmented(blobRequest);          

           

            foreach (var blobItem in result.Results)

            {               

                lstBlobs.Items.Add(blobItem.Uri.AbsoluteUri);// lstBlobs is id of listbox I added

            }

        }

These logs of storage analytics then can be used for various purposes.

Hope this helps.

Cheers…

Azure Storage Analytics for Azure Storage –Blob, Queue and Tables - Metrics example.

Happy Analyzing!!