Ndtyjky

I'm retrieving Azure Data Factory logs for analysis using Powershell.

I am successfully retrieving the top level log (the pipeline) and the log nested inside that (activities) and writing to a text file.

However I'm having issues flattening the activities file, which consists of a mix of flat records and fields containing json

Here is my abridged script. The issue is with the last cmdlet call. I need to work out how to flatten it out to what I need

$DateFrom = (new-object System.DateTime 2018, 07, 01)

$DateTo = Get-Date

$Pipeline="MyPipeline"

$Outputfile="C:SRCADF$Pipeline.TXT"

$OutputSubfile="C:SRCADF$Pipeline.Sub.TXT"

$DFname ="MyDataFactory"

$RG="MyRG"

$TenantId="a16xxxx-xxx-xxx"

$Subscription="d8xxx-xxx-xxx"



$Credential = Get-Credential



Connect-AzureRmAccount `

-TenantId $TenantId `

-Subscription $Subscription `

-Credential $Credential



$oADFLog = Get-AzureRmDataFactoryV2PipelineRun `

-ResourceGroupName $RG `

-DataFactoryName $DFname `

-LastUpdatedAfter $DateFrom `

-LastUpdatedBefore $DateTo `

-PipelineName $Pipeline



# This is the pipeline log - it works as required

$oADFLog | Export-Csv -Path $Outputfile -Delimiter "`t" -NoTypeInformation



# Delete the subtask file

Remove-Item -Path $oADFSubLog -Force -Recurse -ErrorAction Ignore



Foreach ($PipelineRun IN $oADFLog)

{

    # For each parent run ID, check the child tasks

    # File results in thispart need to be cleaned up

    $oADFSubLog = Get-AzureRmDataFactoryV2ActivityRun `

    -PipelineRunId $PipelineRun.RunId `

    -ResourceGroupName $PipelineRun.ResourceGroupName `

    -DataFactoryName $PipelineRun.DataFactoryName `

    -RunStartedAfter $DateFrom `

    -RunStartedBefore $DateTo



    # This is the activity log - it has nested data types and is ugly

    # I need to flatten the Json inside the message

    $oADFSubLog | Export-Csv -Append -Path $OutputSubfile -Delimiter "`t" -NoTypeInformation

}

Given $oADFSubLog in the above script, I've discovered that I can pull out some piece that I need like this:

(ConvertFrom-Json -InputObject $oADFSubLog[0].Input.ToString()).packageLocation

This pulls the property that I need out of the Json

But I'm not sure how to easily push that out to a file in conjunction with the other flat attributes

I've tried this, which is really just a stab in the dark

$oADFSubLog | Select-Object -Property ActivityName,@(ConvertFrom-Json -InputObject $oADFSubLog[0].Input.ToString()).packageLocation 

But I get

Select-Object : Cannot convert System.Management.Automation.PSObject
to one of the following types {System.String,
System.Management.Automation.ScriptBlock}.

I've seen a few examples of custom cmdlets and scripts that can be added but I don't want to go there just yet - I just want to understand how to do this.