Read Azure Synapse table with Spark

I'm looking for, with no success, how to read a Azure Synapse table from Scala Spark. I found in https://docs.microsoft.com connectors for others Azure Databases with Spark but nothing with the...

From Databricks to Azure Synapse Analytics: when we load a dataframe into table, why do we need to set parquet.writeLegacyFormat as True?

I am in need to load dataframes created in Databricks into Azure Synapse. Now, Microsoft's example requires setting Spark configuration for Parquet as...

Attempting to Read parcquet files on linked storage in Azure Synapse

I am attempting to give access to parquet files on a Gen2 Data Lake container. I have owner RBAC on the container but would prefer to limit access in the container for other users. My Query is...

PySaprk- Perform Merge in Synapse using Databricks Spark

We are having a tricky situation while performing ACID operation using Databricks Spark . We want to perform UPSERT on a Azure Synapse table over a JDBC connection using PySpark . We are aware of...

Azure Synapse Apache Spark to Synapse SQL connector error Login failed for user '<token-identified principal>'

Cannot connect Azure Synapse Apache Spark to Synapse SQL ``` %%spark import com.microsoft.spark.sqlanalytics.utils.Constants import...

How to handle white spaces in varchar not null column from azure synapse table to spark databricks

I have a problem when I read a table in spark (using azure databricks) from synapse database. The table is defined as follow: CREATE TABLE A ( [ID] [int] NOT NULL, [Value] [int] NOT NULL, ...

Load data from Databricks to Azure Analysis Services (AAS)

Objective I'm storing data as Delta Lake format at ADLS gen2. Also they are available through Hive catalog. It's important to notice that we're currently using PowerBI, but in future we may...

How to Insert Data into table with select query in Databricks using spark temp table

I would like to insert the results of a Spark table into a new SQL Synapse table using SQL within Azure Data Bricks. I have tried the following explanation...

Spark pool taking time to start in azure synapse Analytics

I have created 3 different notebook using pyspark code in Azure synapse Analytics. Notebook is running using spark pool. There is only one spark pool for all 3 notebook. when these 3 notebook run...

How can we parameterise Azure Synapse Spark Jobs?

The Spark Jobs UI in Azure Synapse has the option to pass command line arguments to the underlying code, but there doesn't appear to be any option to parameterise these arguments. Similarly the...

Azure Synapse: How to use Spark job reference files?

I created a Spark job definition following the PySpark section of this...

How do you import a custom python library onto an apache spark pool with Azure Synapse Analytics?

According to Microsoft's documentation it is possible to upload a python wheel file so that you can use custom libraries in Synapse Analytics. Here is that documentation:...

Azure Synapse Analytics Spark pool vnet integration solution

So I'm hoping to be able to move some ETL pipelines over from Azure Databricks over to Azure Synapse Analytics, but I'm running into some issues with a vnet integration. The case is as follows: We...

Connecting from Azure Synapse Analytics Spark Pool to Azure SQL Database

Did anyone had luck connecting from Azure Synapse Analytics provisioned Spark Pool to Azure SQL Database? Problem 1: I've uploaded the Spark SQL Connector...

How to handle exceptions in azure databricks notebooks?

I am new to Azure and Spark and request your help on writing the exception handling code for the below scenario. I have written HQL scripts (say hql1, hql2, hql3) in 3 different notebooks and...

Connecting from Azure Synapse spark notebook to SQL-Pool table

I'm looking for, with no success, how to read a Azure Synapse table from a SQL-Pool of another workspace using Scala Spark (since it is apparently the only option). I found in...

Working with pyspark in Azure Synapse Analytics how do I create a session that multiple notebooks can use

I'm creating a data pipeline in Azure Synapse. Basic flow: grab some CSV files of 837 EDI data. Put those data files on Azure Data Lake (Gen2). Foreach file put data into tabular database table...

Azure Synapse Studio ServerLess SQL Pool Access Error

I am not able to access any of my Server Less SQL Pool in Azure Synapse. I can create new Spark Pool but not SQL Pool. Error is : **You don't have access to the serverless SQL pool, Built-in....

List content of a directory in Spark code in Azure Synapse

In Databricks' Scala language, the command dbutils.fs.ls lists the content of a directory. However, I'm working on a notebook in Azure Synapse and it doesn't have dbutils package. What is a...

Write dataframe to Azure EventHub with Synapse Spark using Kafka

I want to write a dataframe with Azure Synapse Spark (in a Synapse Analytics Workspace) to a Kafka-enabled Azure Event Hub. Apache Spark version: 2.4 Python version: 3.6 Scala version:...

How to drop table or drop row on Databricks with Azure Synapse?

I want to drop my table using DROP TABLE, but I got an error JDBC on my Databricks, but if I use SELECT, it doesn't get any error, I have 2 function different that can read_from_synapse and...

Synapse Analytics sql on-demand sync with spark pool is very slow to query

I have files loaded into an azure storage account gen2, and am using Azure Synapse Analytics to query them. Following the documentation here:...

Apache Spark in Azure Synapse 'overwrite' method Function not working

I have a nice function let allows me to overwrite and rename a file when I save the results of query to ADLS, see following from pyspark.sql import SparkSession spark =...

Write Data to SQL DW from Apache Spark in Azure Synapse

When I write data to SQL DW in Azure from Databricks I use the following code: example1.write.format("com.databricks.spark.sqldw").option("url", sqlDwUrlSmall).option("dbtable",...

Is it possible to run Bash Commands in Apache Spark with Azure Synapse with Magic Commands

In databricks there is the following magic command $sh, that allows you run bash commands in a notebook. For example if I wanted to run the following code in Databrick: pip install...

Synapse Notebook reference - how to use %run

I'm trying to use the Azure Synapse notebook reference described...

Error when write spark dataframe from Databricks into Azure Synapse

I am trying to write the spark dataframe into Azure Syanpse database. My code: try: re_spdf.write \ .format("com.microsoft.sqlserver.jdbc.spark") \ .mode("append") \ .option("url",...

Spark: How to flatten data frame with dynamic nested structs / arrays

How to tidy data frame(json ) with dynamic nested structs / arrays in PySpark ? I have 10000 json files, each has static and dynamic fields as described below. Static names: data, label, units,...

Error accessing Key Vault while executing Azure Synapse Notebook via Synapse Pipelines

I'm trying to execute an Azure Synapse Notebook using Notebook Activity in Synapse Pipelines and it keeps coming up with errors while debuging the Pipelines, the Notebook is using...

Write dataframe to SQL dedicated database using Synapse Analytics

I would like to load a dataframe from my Azure Data Lake Storage Gen2 and write it to an SQL dedicated database that I created in Synapse. This is what I did: df =...