What is Azure Databricks? Features, Advantages, Limitations

sqlshack 

• Click on the Create button.

sqlshack 

• Now you will get a form like shown in the image below. It has the following fields:\

1. Subscription – Select your subscription.
2. Resource group – Create a new resource group by clicking on the Create button. The name will automatically appear here.
3. Workspace name – Pick any name for the Databricks service.
4. Location – Select the region where you want to deploy your Databricks service.
5. Pricing Tier – Select a suitable pricing tier for your service.

• After filling out all the details click on Review + Create button to review the values filled in the form. After reviewing click on the Create button to create the service.
• Now you'll get a message on the screen - "Deployment Succeeded" in case your deployment is successful. Click on the Go to Resource option to open the service that you have recently created. 

sqlshack 

• Now you will see all the details of the service that you have created. Click on Launch Workspace to open the Azure Databricks portal. Now you will have to sign in again to access the Databricks portal.

sqlshack 

• On the Workspace tab, you can create notebooks and manage your documents. The Data tab lets you create tables and databases. You can also work with various data sources like Cassandra, Kafka, Azure Blob Storage, etc.

sqlshack 

• After creating Databricks service we need to create a spark cluster. Click on Clusters in the left menu. Click on Create Cluster to create a cluster.

sqlshack 

• Use the below image to fill up the configurations of the cluster. And finally, click on Create Cluster

sqlshack 

• Now you will see the status of the creation of the cluster as Pending until it is created.
• Once it is active and running you will see the status as Running.

sqlshack 

• Now you can create a Notebook in a Spark cluster. A Notebook is a web-based code and visualization platform built to interact with Spark in various languages.
• Now to create a notebook, click on the Workspace option in the left menu. Click on Create and select the Notebook option.
• Provide the Notebook name, select Language and Cluster, and click on Create. This will create a Notebook.

sqlshack 

You have successfully created Azure Databricks service.

Databricks SQL

Just like any other data residing in a database can be queried via SQL, the same is true for the datasets handled by Databricks. Databricks SQL is a feature that allows users to perform SQL queries and analytics on their data. It extends the capabilities of the Apache Spark SQL module and helps data analysts and engineers to collaborate effectively in a unified environment. Using Databricks SQL on the data stored in the data lake makes it easier for the users to create dashboards to be consumed by business users. Below are certain key aspects of Databricks SQL:-

1. SQL Dialect Support - Databricks SQL supports ANSI SQL to allow users to write standard SQL queries and supports Spark SQL to handle complex data types.
2. Data Exploration and Visualization - It allows users to easily visualize their data using SQL queries.
3. Collaborative Notebooks - Users can create and share their code, and SQL queries ensuring collaboration between team members.
4. Performance Optimization - Databricks SQL uses Spark engine which is optimized for distributed computing and efficient processing of large datasets.
5. Connectivity to various data sources - Databricks SQL supports connectivity to various data sources, including data lakes, databases, and external file systems hence introducing flexible data integration.
6. Optimization and Tuning - Users can optimize and tune their SQL queries using the Databricks platform. This includes leveraging features such as query optimization, indexing, and caching to enhance the performance of SQL-based analytics.

Databricks Machine Learning

Databricks Machine Learning (DBML) is a Databricks component in the unified Databricks platform which provides an integrated and collaborative environment for developing, training, streamlining ML workflows, and deploying machine learning models. It leverages the power of Apache Spark and combines it with powerful machine-learning libraries to prepare a production-ready machine-learning solution. It provides below key aspects below:

1. Since Databricks ML is built on an open architecture with a foundation on Delta Lake, it simplifies all aspects of Data for ML and AI. It can turn features into production pipelines without much hassle.
2. The MLflow component of Databricks helps automate experiment tracking and governance. Once you have identified the best version of a model for production you can register it to the Model Registry to simplify handoffs along the deployment lifecycle.
3. It provides the capability to deploy ML models at scale and at low latency.
4. Databricks allows you to use Large Language Models (LLMs) which can be extended using techniques such as parameter-efficient fine-tuning (PEFT) or standard fine-tuning.
5. It can manage the full model lifecycle from data to production and back with model versions and other components.

Limitations of Azure Databricks

While Azure Databricks is a powerful and versatile platform to process and manage large data and analytics workloads it has certain limitations that a user must be aware of:-

1. Dependency on Azure - Since Azure Databricks is a service provided by Microsoft Azure, any issues or outages in Azure can reflect the impact on Databricks workloads.
2. Versioning Tool Integration - Azure Databricks does not integrate with Git or any other versioning tool at the moment.
3. Limited control over infrastructure - Azure Databricks is a managed service and hence user has little control over its infrastructure.
4. Costs - Azure Databricks can prove to be expensive, especially when dealing with large-scale data processing and compute-intensive workloads.

Final Words

In a data-driven world where insights are retrieved from large datasets that redefine business strategies, Azure Databricks seems like a compelling solution. It is a robust, collaborative, and scalable platform that lets data engineers, data analysts, and data scientists collaborate well and build end-to-end production-ready data processing and ML solutions. With all Azure Databricks components and Azure Databricks Storage, Azure Databricks becomes a great comprehensive platform to provide features that continue to harness the potential of big data to derive business successes. To learn more on Azure databricks Spark and Azure databricks components apart from the Azure Databricks example above you can checkout KnowledgeHut Azure certification courses.