Using SAS, renew SAS and REST API to Upload large files to Azure blob storage in parallel and async

First of all thanks for overwhelming response to my earlier blog post related to upload of large files to azure block blob in parallel and async.

At the bottom of this post you will find the link for downloading the code sample.

In current post I will extend the same code library to support REST Api to perform azure blob upload. Also, as a best practice you should always use SAS (Shared Access Signature) for performing any operation on blob storage. Therefore in current post I will extend the code to support SAS as well.

Why we need to renew or extend Azure SAS –

Using SAS with blob storage is great option to provide one more level of security to azure storage operations however, anyone who gets access to SAS url can perform malicious operations. Therefore keeping SAS expiry time to minimum is always a good practice. But again, if we are performing upload of fairly large files then keeping SAS expiry to minimum will not help.

For example, let’s consider a scenario. Let’s say I will upload a file of size 100MB in azure blob storage using REST Api and SAS. As a best practice and already stated in my previous blog post of uploading large files to azure block blob in parallel, the files to be uploaded to azure need to be sliced in chunks.

Now if I have set the expiry time to 5 minutes. Of course upload of 100MB size will not be completed in 5 minutes and hence my block upload will fail in between.

Solution -

To continue to upload large files to azure block blob seamlessly after SAS expiry time, I need to renew the SAS token again. This is what exactly I am doing in this blog post and code sample.

So essentially, I check the response status code of every block upload. If response status code Created means current block is uploaded and continue to next. If the response status code is received as 403 Forbidden means the upload of block is failed due to invalid SAS. This is where I mark the particular failed block as Not Added.  Once the loop is over, I retrieve list of failed blocks and perform the same operation with new SAS. To perform same operation of upload with new SAS I am using popular and basic concept you also might know called as RECUSRSIVE FUNCTION.

JavaScript runtime error: '$' is undefined – Azure web role and jQuery

I wanted to use jQuery in my Azure ASP.Net web role. I created a cloud service project and then added one asp.net web role in it.

Then to use jQuery I had decided to use HTML page. Therefore I added HTML page in my wen role sample. I wrote some jQuery code and at runtime I received error as -

JavaScript runtime error: '$' is undefined

This error is encountered because the jQuery version is referenced in HTML page however it’s path is incorrect.

If you open the solution explorer you will find that jQuery is already added in your web role project under the folder Scripts as shown below –

However in HTML page it is referenced with incorrect path as shown below –

The folder js as referenced in above screenshot does not exists and file jQuery files are present in Scripts folder. Therefore I changed the path of jQuery file as below –

<script src="Scripts/jquery-1.10.2.min.js"></script>

And you are done. No runtime error was displayed after this change.

Hope this helps.

Cheers!!!

Session management using SQL Azure and Update on automatic session clearance

Session management using InProc in azure roles is dangerous thing!!!

Why InProc is dangerous?

Create an azure cloud service project and add a web role in it. Then I added a sample page named as Session demo in it. To test the session data saving and retrieval I added following control in the <form> control present on the page –

<asp:TextBox ID="TextBox1" runat="server"></asp:TextBox>

<asp:Button ID="btnAddToSession" runat="server" OnClick="btnAddToSession_Click" Text="Add To Session!!" />

            <br />

            <br />

            <asp:Button ID="btnReadFromSession" runat="server" OnClick="btnReadFromSession_Click" Text="Read From Session!!" />

            <asp:Label ID="Label1" runat="server" Text="Label"></asp:Label>

            <br />

            <br />

            <asp:Label ID="lblInstanceId" runat="server" Text="Label"></asp:Label>

On the code behind I added below simple code to add values in session and retrieve it. Also in page load event I am retrieving the id of role instance to check that, my current request is being served from which instance.

public partial class SessionDemo : System.Web.UI.Page

    {

        protected void Page_Load(object sender, EventArgs e)

        {

            lblInstanceId.Text = "You are being served from - "+ RoleEnvironment.CurrentRoleInstance.Id;       

 } 

        protected void btnAddToSession_Click(object sender, EventArgs e)

        {

            Session["MySession"] = TextBox1.Text;

        } 

        protected void btnReadFromSession_Click(object sender, EventArgs e)

        {

            if (Session["MySession"] != null)

                Label1.Text = Session["MySession"].ToString();

        }

    }

In web.config file as shown below, I have configured my session state as InProc under <system.web> tag –

<sessionState mode="InProc"/>

Now right click on cloud Service project, select Properties and set the application to run in full emulator and full IIS as shown below -

 

Note – The option to run the cloud service project locally in Full Emulator mode is applicable till the Azure SDK 2.3 only. In Azure SDK 2.4, full emulator is deprecated. I don’t know the reason why they have deprecated full emulator mode in Azure SDK 2.4.

Now we always should set the number of instances for an azure role as minimum of 2. Therefore right click on web role node present in cloud service project and in properties set the instance count as 2.

 

 

 

We all set to see how InProc session management in Azure web role is devil. Set the cloud service project as startup project and I have set the Sessiondemo.aspx page as startup page and here I click F5. And here what I see on my page and in my IIS –

So as you can see, in IIS there are two instances got created as expected and below that the UI shows simple design to test sessions. Also if you see in IE, my current request is being server from INSTACE 0.

Now let’s say I add the value to session as “kunal” by clicking on button “Add to Session” and retrieve it from session by clicking on button “Read from Session” and I found that, my requests are still getting served from INSTACE 0 and session value is being retrieved correctly. Now I right click on instance 0 website in IIS and then I select the options as Manage Web Site | Stop. This actually stops the 0^th instance running in IIs. Remember, my project is still running in debug mode. I just stopped one of the role instances of my azure application.

Once the IIS website is stopped you can view the black icon on the web site as shown. This confirms that my website is stopped.

Now I refresh my browser 2-3 times. This means now my request will start getting served from INSTANCE 1 of IIS website because instance 0 is stopped and here is what I see –

As you can see above, my request is being served from INSTACE 1 and my Session value is not retrieved on refresh. Therefore I click on button “Read from session” again and DAMN!!!

I am not receiving the value “kunal” that I had added in my session. This is the problem!!! Ideally I should have received my session values irrespective from which instance of role my request is getting served.

In Azure environment there is always possibility that, the instance of role gets recycled due to hardware failures or other errors. Means if you configure only 1 instance for your role then there is possibility that, your role instance is down and you DON”T GET HIGH AVAILABILITY for your application. Therefore it is always BEST PRACTICE TO HAVE ATLEAST 2 INSTANCE for your azure role.

But again, if you have 2 instance configured for your web role then session management using INPROC will not work as shown above!!!

What is the Solution?

You should always store the session state outside your role environment. How do I do that?

Azure Automation - shutdown azure virtual machine – step by step

Update - 21st Nov 2019

This document is old now. You can refer below article to know the concepts of Azure Automation and Azure VM Shutdown. If you plan to do Hands on and implement one then refer to updated and recommended guide here - https://www.sanganakauthority.com/2017/07/start-stop-multiple-azure-vms-on.html

Why?

Managing Azure resources using Management portal is cumbersome job for infrastructure guys. In one of my customers company (where I am supporting Azure Project as a consultant), the developer used start the Azure VM in the morning and many times they never stopped the azure VM while leaving from office. Ofcourse this used to incur huge cost for them. Therefore IT team guys used to check if their development Azure VM's are running, if yes then they used to shutdown it from azure portal to save on cost of running Azure VM overnight. The number of VM’s to stop was around 30 to 50. They had been performing this manually in combination of powershell script and few operations from management portal. This simple activity had consumed 25% of the IT people bandwidth and it was problem for them. Thanks to Azure Automation!!! IT guys don’t have to shutdown Azure VM anymore manually and they are happy now.

What?

So, Azure automation is the new service in preview. Well, Azure automation is not the only service that can automate the common activities of Azure. Powershell exists since long and in fact powershell is the basis behind Azure Automation service. You can automate the creation, deployment, monitoring, and maintenance of resources in your Microsoft Azure environment using runbooks, which ultimately uses Windows PowerShell workflows.

Ofcourse, Chef and Puppet are also doing the same automation job greatly however, I find them pretty complex. I know most of you may not agree, however, I feel Chef, Puppet is best for Linux, Unix OS based Azure VM. For Windows OS based Azure VM, Azure Automation with Powershell is your key.

In this post, I will be giving step by step approach to shutdown your Azure virtual machine using Powershell and Azure automation.  So let’s start!!

You may not have observed, but I feel the concept of Azure automation is very much similar to what Chef is doing for automation. See below comparison between Chef and Azure Automation –

Chef Jargons – Recipe, cookbook

Azure Automation Jargons – Job, Runbook

Cookbook – Runbook!!! Of course it is just an observation.

Runbook – Runbook is a set of powershell commands that gets executed based on schedule set in Azure automation. So the book has sentences (or commands) that run in Azure Automation service. In azure automation we always execute powershell scripts under runbook.

Activate

When I am writing this post, the Azure Automation is in preview mode and hence you need to enable is for your subscription from here - https://account.windowsazure.com/PreviewFeatures

If it becomes generally available then this step will not be required.

Automation Account

Create Automation Account first as shown in below screenshots –

 

As a preview feature, these are supported only in US region as of now.

Certificate Management

An Automation Credential is both a username and password that can be used with Windows PowerShell commands or a certificate that is uploaded to the server. We will use certificate based approach. Therefore we need certificate to authenticate azure subscription. Best way is to use self-signed certificate either created from makecert command or created from IIS itself.

Let’s see the way of using IIS. Open run Window and type INETMGR to open IIS window. Select the local machine name and double click on Server Certificates option as shown below –