pure gorge
Playing around with a pipeline, takes a container on a datalake and loads each subfolder and its relevant data. deals with incremental and full loads so i have a watermark setup.
Getting this error on my Copy Data Activity - The function 'substring' parameters are out of range: 'start index' and 'length' must be non-negative integers and their sum must be no larger than the length of the string.
pure gorge
Error occurs during Copy Activity
steady scaffold
can you share the expression that is using substring?
pure gorge
Sure
It's a parameter of a dataset
but im setting it dynamically when using the dataset in the pipeline
folderPath = @substring(item().SourcePath, 0, lastIndexOf(item().SourcePath, '/'))
fileName = @substring(item().SourcePath, add(lastIndexOf(item().SourcePath, '/'), 1), sub(length(item().SourcePath), add(lastIndexOf(item().SourcePath, '/'), 1)))
those parameters are set in the true condition of the if condition
under the copy data set
small comet
Playing around with a pipeline, takes a container on a datalake and loads each s...
Whats this tool?
regal heron
Whats this tool?
Looks like an export of the errorred pipeline in Ron’s Data Tool
It’s a CSV editor afaik
pure gorge
Hi
Any further advice on Parameterisation and building Pipeline logic
Really struggling to understand parameters and dynamic expressions. MS Learn doesn't have a resource to stuff like @dataset(), @activity(), @item() - I can't find any sort of docs on usage of these and tutorials about dynamic paramerisation...
small comet
What are you trying to achieve?
pure gorge
What are you trying to achieve?
I was trying to build an incremental pipeline to deal with full and incremental loads. I set up the following. A control table that hosts the table names, folder and file paths, which type of load it is, if the table is active with 1/0.
Linked Services - ADLS Gen2
Data Set - AzuresSQLTable and Generic CSV Dataset
small comet
So why do you need substring?
pure gorge
I left it on hold for now as I'm still getting some issues but I think I might have found the issue, in the Parameters in the Dataset definition, I was stupidly putting @dataset().paramName in the default column...
Instead of in the connection where it'll actually be called
I'm going to play around with that and see if that was the issue