KeyVault getting a secret out fails

Here is my code, this was working a few days ago and now it throws the error when trying to fetch the secret out of the vault. I am using the latest version of Azure's Python SDK which was...

Problems creating HDInsight + Datalake cluster using azure cli and templates

I'm trying to create a HDInsight Spark cluster. I also have a Datalake storage which I want to use in the HDInsight cluster. I have generated a certificate to connect the HDInsight with Datalake....

Accessing AzureDataLake Gen2 from Databricks

I would like to read data from Azure DataLake Gen 2 from Databricks. I have an Owner role for both and read/write/execute rights. DataLake allows access from the Vnet where Databricks are added. I...

File system creation in Azure data lake storage account through ARM template

I have created an Azure Data Lake gen 2 with ARM templates. But now I am trying to figure out how to create Data Lake File systems in ARM but can't seem to find the API's to do this. Is this...

How to get the last modification time of each files present in azure datalake storage using python in databricks workspace?

I am trying to get the last modification time of each file present in azure data lake. files = dbutils.fs.ls('/mnt/blob') for fi in files: ...

partitionBy & overwrite strategy in an Azure DataLake using PySpark in Databricks

I have a simple ETL process in an Azure environment blob storage > datafactory > datalake raw > databricks > datalake curated > datwarehouse(main ETL). the datasets for this project are not very...

How to browse Azure Data lake gen 2 using GUI tool

First some background: I want to facilitate access to the different groups of data scientists in Azure Data Lake gen 2. However, we don’t want provide access to them to the entire data lake...

How to access captured data from Event Hub in Azure Data Lake Storage Gen2 using Python

I'm using the connection_string to access an Azure Data Lake Gen2 storage, in which lots of Avro files were stored by an Event Hubs Capture, under the typical directory structure containing...

Find Last modified timestamp of a files/folders in Azure Datalake through python script in Azure databricks that uses Credential passthrough

I have an Azure DataLake Storage Gen2 which contains a few Parquet files. My Organization has enabled credential passthrough and so I am able to create a python script in Azure Databricks and...

Copy n number of files from Azure Datalake to SFTP location using Logic Apps

I have a scenario- I have some files in Azure Datalake. A job is pushing these file at an ADLS location. This files needs to be uploaded at a SFTP location as a input data. An application will...

install python packages using init scripts in a databricks cluster

I have installed the databricks cli tool by running the following command pip install databricks-cli using the appropriate version of pip for your Python installation. If you are using Python 3,...

Trigger Azure data factory pipeline - Blob upload ADLS Gen2 (programmatically)

We are uploading files into Azure data lake storage using Azure SDK for java. After uploading a file, Azure data factory needs to be triggered. BLOB CREATED trigger is added in a pipeline. Main...

Create SQL table from parquet files

I am using R to handle large datasets (largest dataframe 30.000.000 x 120). These are stored in Azure Datalake Storage as parquet files, and we would need to query these daily and restore these in...

SSL errors while installing python packages from PyPI in a databricks cluster

I am trying to install azure-storage-file-datalake in a databricks cluster, but due to internal dependency I am facing the below SSL error. I am using python 3.7.3, pip 20 and was able to install...

How do you delete a file from an Azure Data Lake using the Python SDK?

I'm using the azure-storage-file-datalake plugin for Python 3.8. The SDK is described in great depth here...

For Python 3.8 Azure data lake Gen 2, how do I check if a file exists on a filesystem?

I'm using Python 3.8, Azure Data Lake gen 2 and the following plugins ... azure-storage-blob==12.4.0 azure-storage-file-datalake==12.1.1 How do I check if a specific path exists on a file...

How to connect and access Azure Datalake Gen1 storage using Azure Ad username and password only - c#

I want to connect and access Azure Datalake Gen1 storage using Azure Ad username and password only. I have a service account that has access to the Azure Datalake Gen1 storage. I am able to...

How to copy Azure storage files/directories using java azure-storage-file-datalake

I use azure-storage-file-datalake for java to make file system operations on my Azure storage account, I can open files, delete, and even rename/move files or directories. I can't find any way to...

Copy data transfer from sharepoint to datalake

EDIT: I have gone though "copy data from sharepoint to blob storage" under this thread. https://stackoverflow.com/questions/54868998/azure-data-factory-and-sharepoint But I am looking for a...

Why Java Azure Function App freezes when trying to access Azure datalake?

I am developing a Java Azure function that needs to download a file from Azure Datalake Gen2. When the function tries to read the file, it freezes and no exception is thrown, and nothing is...

Unable to run the Powershell Script using SQL Server Job Agent

I am trying to execute my PowerShell script using the SQL Server Job Agent but unable to do so. I am able to execute the script successfully via PowerShell Prompt. Here in the Agent I am Operating...

Azure Function in Python get schema of parquet file

It is possible get schema of parquet file using Azure Function in Python without download file from datalake ? I using BlobStorageClient to connect to data lake and get the files and containers...

Azure blob to Azure SQL Database: Cannot bulk load because the file "xxxx.csv" could not be opened. Operating system error code 5(Access is denied.)

I am trying to bulk load in azure sql database some data in an azure blob storage. The file content is: customer,age,gender 'C1093826151','4','M' 'C352968107','2','M' 'C2054744914','4','F' the...

Trying to open parquet in Synapse - cannot be opened because it does not exist or it is used by another process

I am trying to open a Parquet files that is generated by Stream Analytics and stored in Azure Datalake V2. I have connected datalake and Synapse successfully, but I keep getting...

Reading parquet file from ADLS gen2 using service principal

I am using azure-storage-file-datalake package to connect with ADLS gen2 from azure.identity import ClientSecretCredential # service principal credential tenant_id = 'xxxxxxx' client_id =...

Browse large number of files in Azure Storage Explorer

I have a large number of files in one of our datalake folders. I use Azure Storage Explorer to find\download specific files. If I need the latest files, which is usually the case, I go through all...

Cannot upgrade azure cli to the latest version

az --version showing an updated version available as shown below. [email protected]:~$ az --version azure-cli 2.18.0 * core 2.18.0 * telemetry ...

Using apache-airflow-providers-snowflake on airflow (no module named Snowflake)

I have installed package apache-airflow-providers-snowflake on airflow on docker and i am getting error No module named Snowflake Please refer attachment (check the error mentioned for the...

Stream Bytes chunks to csv rows in python

I need to process a large remote CSV line by line without downloading it entirely. Below is the closest I got. I iterate byte chunks from Azure, and have some code to handle truncated lines. But...

Cannot create Append Blobs in Azure Data Lake Gen2 using python azure-storage-file-datalake SDK

My use case requires me to continuously write incoming messages into files stored in an Azure Data Lake Gen2 storage account. I am able to create the files by triggering a function, which uses the...