Tuesday, December 31, 2019

Data Platform Tips 34 - Manage firewall settings in Azure SQL Database

To set the server level firewall settings for the Azure SQL Database, perform the following steps.

a) Logon to the Azure Portal.

b) Create a new Azure SQL Database.

c) Click on "Show Firewall Settings".






d) Click on "Add Client IP" to provide the current IP Address access to the SQL Server.










e) Once the Client IP is added it will be displayed as shown below and click on "Save".









f) Once saved, the IP Address will now get access to the SQL Server.


Monday, December 30, 2019

Data Platform Tips 33 - Azure Storage Explorer

Azure Storage Explorer is a stand alone tool that allow users to work with Azure Storage data on multiple operating systems.

Install Azure Storage Explorer

Once installed, you can sign in using your Azure Credentials and all the subscriptions and storage accounts associated with the subscriptions can be then accessed via the storage explorer.

a) To connect to Azure Storage, you can either connect by any one of the following ways.





















b) We will connect using the Azure account and you can provide your Azure credentials for Authentication.

c) Once authenticated, you will be able to see all your Azure Subscriptions.



















d) Now you can see all the storage services under all the subscriptions listed in Azure Storage Explorer.


























e) You will also be able to view all properties related to your Azure Storage services in Azure Storage Explorer.



f) Using Azure Storage Explorer, you will be able to do most of the operations over your Azure Storage services.


Sunday, December 29, 2019

Data Platform Tips 32 - Azure Data Lake Storage Gen2 and Azure Stream Analytics

In this post, we will be looking at how to store streaming data from IoT Hub to Azure Data Lake Storage Gen2 using Azure Stream Analytics.

Save streaming data to Azure Data Lake Storage Gen2 using Azure Portal

a) Logon to the Azure Portal.

b) First provision an Azure IoT Hub to ingest real time sensor data.



































c) Now your IoT Hub would have been successfully provisioned.














d) Add a new device named "Device1" and note down the connectionstring for Device1.


























e) Click on the "eye" icon to reveal the Primary Connection string of Device1 and save it for future use.










f) Next, let us provision an Azure Stream Analytics Job.



























g) You can now see the Azure Stream Analytics Job provisioned.













h) Add the IoTHub as the input for the Azure Stream Analytics Job.









i) Add the Azure Data Lake Storage Gen2 as output for the Azure Stream Analytics Job.










j) Now add the query to the Azure Stream Analytics Job.














k) Let us simulate some data to for the IoT Hub to ingest using https://azure-samples.github.io/raspberry-pi-web-simulator/

l) Replace the connectionstring with the Device1 connectionstring noted in step d) and click on Run to simulate the messages.














m) Start the Azure Stream Analytics Job.


n) Now you can check the Azure Data Lake Storage Gen2 using Azure Storage Explorer to see the newly created JSON file based on the streaming data.


Saturday, December 28, 2019

Data Platform Tips 31 - Azure Data Lake Storage Gen2 and Azure Data Factory v2

Azure Data Factory (ADF) v2 is a cloud based integration service that allows to populate Azure Data Lake Storage Gen2 with data from on-premises, cloud or SaaS data stores. In this post, let us look at how to upload data from on-premises file system to Azure Data Lake Storage Gen 2.

a) Logon to the Azure Portal.

b) Provision an Azure Data Lake Storage Gen2 service.

c) Look for "Azure Data Factory" in the market place and select "Data Factory".










d) Provision the "Azure Data Factory v2" service.























e) Once provisioned, you can now author your pipelines within Azure Data Factory by clicking "Author & Monitor".











f) Select "Copy Data" to copy data from an API to Azure Data Lake Storage Gen 2.










g) Provide a name for the "Copy Data" task.

























h) Select the Source data store as "Generic Protocol" and "OData" and create a new Linked Service.



























i) Select the Service URL as "https://services.odata.org/OData/OData.svc" and Authentication type as "Anonymous"










































j) Select the source data from the API.















h) Create an "Azure Data Lake Storage Gen2" Destination and also select the folder path under which the files needs to be uploaded.












































i) Select the file format as "Parquet" and compression type as "snappy".

























j) Now complete the settings and finish the configuration.





k) Now the pipeline would have run successfully and would have created the API output as txt files on Azure Data Lake Storage Gen2. You can check this by using the Azure Storage Explorer.