How to improve the apache pool efficiency ?
I have roughly around 3000 jsons and i am creating a dataframe out of it and saving it to delta format and it is taking me hours for that even though my spark pool is of medium size. The spark pool runs at 33% utilization,
Writing binary data to ADLS in Synapse notebook
Hi all, Is it possible to write binary data to the ADLS in Synapse notebook using Pyspark? I retrieve data via an API call where the content type is application/excel (basically binary data) and want to save it to the ADLS, a specific location like…
How to save spark dataframe (with synaps) in data container (without making folder and SUCCES file)
I want to save a spark dataframe to my data container. It worked with this code: df.write.csv(path_name + "test5.csv") However, this makes a folder called test5.csv with 2 files in it. One which is my dataframe (but with a random generated…
reduce waiting time within a synapse pipeline/ improve performance synapse pipeline
A certain synapse pipeline takes a very long time. When I look at the various activities within this pipeline, the activities themselves don't take much time at all. It's mostly the waiting time between them that is consuming a lot of time. We are using…
Changing the time zone for a Synapse dedicated SQL pool
Hi, inside a Synapse dedicated SQL pool I need to use the GETDATE() T-SQL function for some queries, but it returns a time minor of two hours respect to the Italian time. This is the datetime returned by the GETDATE() using SSMS: The real time is…
In ADF, I am not able to merge data from multiple streams with different columns with different data types
Hello, I am not able to merge data from multiple streams in ADF DATAFLOWS. Stream 1) flow let - only 1 column. Stream 2) flow let - only 1 column. Stream 3) source columns (54 columns) Stream 4) regular…
Cannot edit an existing data factory in synapse
I have a synapse workspace with several pipelines with data factories. Today I have noticed that existing data factories that are running correctly I cannot see the details when I edit them I attach two examples, one that is working and the other that I…
Synapse Python/Spark Notebook code with SQL to query data from linked service to SQL Server
In Synpase, I have linked service to a SQL Server database. I'm looking for a sample notebook code, either in python or spark, that I can run a complex SQL qery within this code to get data from multiple SQL Server tables, put the rsults in a dataframe…
Join cosmos DB when processing streaming data from eventhub in Synapse
Hi Team, Good morning! Would like to check that when we using spark notebook in Synapse to process Eventhub streaming data, we join the streaming data from Eventhub with CosmosDB data. We try both way of connecting CosmosDB transcational store with OLTP,…
Integration
Can you provide detailed information on the factors that should be considered for hybrid cloud integration between an on-premises private cloud and Azure, with a focus on scalability, performance, and security?
How to translate database content with Azure Translator by ADF or Synapse notebook?
There is an Azure Database table. Some of columns need to be translated from one language to another into additional columns. Such as from English to Spanish, or Portuguese to English, etc. I am exploring how I can use ADF or Synapse notebook to…
Dedicated SQL Pool Performance issues
We're utilizing Tableau's live connection feature, linked to a dedicated SQL pool. We've taken all necessary steps from the SQL pool side, such as creating required indexes, statistics, assigning appropriate distribution models and partitioning tables,…
How to allow access for Azure Synapse pipeline to connect to logic app which has public access disabled with Vnet, NSG and user defined route configuration enabled
Hi, I need to call logic app through web activity from a synapse pipeline, get the error as "The web app you have attempted to reach has blocked your access". The synapse workspace has managed virtual network configured and the logic app has…
SAP CDC and SAP Tables connectors regarding SAP Note 3255746 ?
Dear Team, Is there any one who when through the SAP Note 3255746 ? What impact shroud we expect regarding the SAP CDC and SAP Tables connectors already implemented for our customers? Thanks for you input. Tarik
Is the 'tzoffset' function no longer compatible within Synapse?
Up until last night, executing the below script worked within our Synapse instance. However, since last night when I attempt execute the below script I receive the error message underneath. Is the 'tzoffset' function no longer compatible within…
Reducing Data Scanning Overhead in Delta Format in Azure Synapse Analytics SQL Pool
We've established a serverless SQL pool with a hierarchical folder structure partition and employed open row set to reduce data scanning when executing queries on Parquet file format from the serverless SQL pool. Now, as we endeavor to utilize the delta…
How can I scan views from the same Azure Synapse Analytics schema to multiple collections?
I have multiple collections in Microsoft Purview. I have registered an Azure Synapse Analytics source to a parent collection. Depending on the ownership of the datasets I need to scan in views from the same schema to different collections. When I scan…
How to enable writing for SH-IR via ADF Pipeliens
Hello community, I am facing an interesting problem when trying to write to a file share (on-prem) from a storage account via Self Hosted Integration Runtime (SHIR). Details: Rights on the share level are granted to the user and reading from the file…
Troubleshooting CSV file operations in Synapse Spark with R script
I'm encountering an issue I can't solve on my own. I'm running an R script in Synapse Spark, and I want to save the results of that script into a CSV file in Azure Data Lake Storage (ADLS). I've attempted to read a CSV file and then write it back into a…
Access Azure SQL DBs using Azure Synapse studio
I would like to understand how Azure Synapse Analytics can be used within the organization to facilitate data analysis and development. How will developers access Azure Datasets in a convenient way Is there a case for using Azure Synapse? I have a use…