Azure Data Factory V2 Copy activity Mapping deployment issue
Consider following test Mapping for Data Factory Copy activity:
"translator": {
"columnMappings": "@json('{"from":"to"}')",
"type": "TabularTranslator"
}
After deploying pipeline with the help of Set-AzureRmDataFactoryV2Pipeline PowerShell cmdlet we get normally deployed pipeline with the exact columnMappings value as specified in source code. But if you try to be more dynamic:
"translator": {
"columnMappings": "@json(pipeline().parameters.Mapping)",
"type": "TabularTranslator"
}
then after deployment you'll find that translator element is completely missing in pipeline. A workaround - set translator in Azure Portal Data Factory pipeline editing UI (either in Designer or JSON modes - both options work). But if after these manipulations you save pipeline JSON to the file and attempt to deploy it via Set-AzureRmDataFactoryV2Pipeline PowerShell cmdlet - bang, translator becomes missing. Expected result - deployment shall preserve translator element, because Portal JSON Editor preserves it.
We are doing automated deployment of pipelines (as you already figured out - with the help of Set-AzureRmDataFactoryV2Pipeline) and this bug breaks our automated deployment because it requires manual postdeployment pipeline editing on Azure Portal UI.
What may be the reason of such a buggy behavior? Can you suggest an idea how to work around this bug in automated way, or how to fix the code so it can be properly deployed with Set-AzureRmDataFactoryV2Pipeline?
azure deployment azure-data-factory-2 asked Nov 12 '18 at 16:06
Roman
2,19492960
add a comment |
Consider following test Mapping for Data Factory Copy activity:
"translator": {
"columnMappings": "@json('{"from":"to"}')",
"type": "TabularTranslator"
}
After deploying pipeline with the help of Set-AzureRmDataFactoryV2Pipeline PowerShell cmdlet we get normally deployed pipeline with the exact columnMappings value as specified in source code. But if you try to be more dynamic:
"translator": {
"columnMappings": "@json(pipeline().parameters.Mapping)",
"type": "TabularTranslator"
}
then after deployment you'll find that translator element is completely missing in pipeline. A workaround - set translator in Azure Portal Data Factory pipeline editing UI (either in Designer or JSON modes - both options work). But if after these manipulations you save pipeline JSON to the file and attempt to deploy it via Set-AzureRmDataFactoryV2Pipeline PowerShell cmdlet - bang, translator becomes missing. Expected result - deployment shall preserve translator element, because Portal JSON Editor preserves it.
We are doing automated deployment of pipelines (as you already figured out - with the help of Set-AzureRmDataFactoryV2Pipeline) and this bug breaks our automated deployment because it requires manual postdeployment pipeline editing on Azure Portal UI.
What may be the reason of such a buggy behavior? Can you suggest an idea how to work around this bug in automated way, or how to fix the code so it can be properly deployed with Set-AzureRmDataFactoryV2Pipeline?
azure deployment azure-data-factory-2 asked Nov 12 '18 at 16:06
Roman
2,19492960
For now I've resolved an issue by deploying this particular pipeline that hastranslatorproperty with the help of ARM templite andNew-AzureRmResourceGroupDeploymentPowerShell cmdlet. Moving deployment of all Data Factory objects from PowerShell to ARM is also an option, but not an easy.
– Roman
Nov 14 '18 at 10:13
add a comment |
Consider following test Mapping for Data Factory Copy activity:
"translator": {
"columnMappings": "@json('{"from":"to"}')",
"type": "TabularTranslator"
}
After deploying pipeline with the help of Set-AzureRmDataFactoryV2Pipeline PowerShell cmdlet we get normally deployed pipeline with the exact columnMappings value as specified in source code. But if you try to be more dynamic:
"translator": {
"columnMappings": "@json(pipeline().parameters.Mapping)",
"type": "TabularTranslator"
}
then after deployment you'll find that translator element is completely missing in pipeline. A workaround - set translator in Azure Portal Data Factory pipeline editing UI (either in Designer or JSON modes - both options work). But if after these manipulations you save pipeline JSON to the file and attempt to deploy it via Set-AzureRmDataFactoryV2Pipeline PowerShell cmdlet - bang, translator becomes missing. Expected result - deployment shall preserve translator element, because Portal JSON Editor preserves it.
We are doing automated deployment of pipelines (as you already figured out - with the help of Set-AzureRmDataFactoryV2Pipeline) and this bug breaks our automated deployment because it requires manual postdeployment pipeline editing on Azure Portal UI.
What may be the reason of such a buggy behavior? Can you suggest an idea how to work around this bug in automated way, or how to fix the code so it can be properly deployed with Set-AzureRmDataFactoryV2Pipeline?
azure deployment azure-data-factory-2 asked Nov 12 '18 at 16:06
Roman
2,19492960
Consider following test Mapping for Data Factory Copy activity:
"translator": {
"columnMappings": "@json('{"from":"to"}')",
"type": "TabularTranslator"
}
After deploying pipeline with the help of Set-AzureRmDataFactoryV2Pipeline PowerShell cmdlet we get normally deployed pipeline with the exact columnMappings value as specified in source code. But if you try to be more dynamic:
"translator": {
"columnMappings": "@json(pipeline().parameters.Mapping)",
"type": "TabularTranslator"
}
then after deployment you'll find that translator element is completely missing in pipeline. A workaround - set translator in Azure Portal Data Factory pipeline editing UI (either in Designer or JSON modes - both options work). But if after these manipulations you save pipeline JSON to the file and attempt to deploy it via Set-AzureRmDataFactoryV2Pipeline PowerShell cmdlet - bang, translator becomes missing. Expected result - deployment shall preserve translator element, because Portal JSON Editor preserves it.
We are doing automated deployment of pipelines (as you already figured out - with the help of Set-AzureRmDataFactoryV2Pipeline) and this bug breaks our automated deployment because it requires manual postdeployment pipeline editing on Azure Portal UI.
What may be the reason of such a buggy behavior? Can you suggest an idea how to work around this bug in automated way, or how to fix the code so it can be properly deployed with Set-AzureRmDataFactoryV2Pipeline?
azure deployment azure-data-factory-2 azure deployment azure-data-factory-2 asked Nov 12 '18 at 16:06
Roman
2,19492960
asked Nov 12 '18 at 16:06
Roman
2,19492960
asked Nov 12 '18 at 16:06
Roman
2,19492960
asked Nov 12 '18 at 16:06
Roman
2,19492960
asked Nov 12 '18 at 16:06
Roman
2,19492960
2,19492960
For now I've resolved an issue by deploying this particular pipeline that hastranslatorproperty with the help of ARM templite andNew-AzureRmResourceGroupDeploymentPowerShell cmdlet. Moving deployment of all Data Factory objects from PowerShell to ARM is also an option, but not an easy.
– Roman
Nov 14 '18 at 10:13
add a comment |
For now I've resolved an issue by deploying this particular pipeline that hastranslatorproperty with the help of ARM templite andNew-AzureRmResourceGroupDeploymentPowerShell cmdlet. Moving deployment of all Data Factory objects from PowerShell to ARM is also an option, but not an easy.
– Roman
Nov 14 '18 at 10:13
For now I've resolved an issue by deploying this particular pipeline that has
translator property with the help of ARM templite and New-AzureRmResourceGroupDeployment PowerShell cmdlet. Moving deployment of all Data Factory objects from PowerShell to ARM is also an option, but not an easy.– Roman
Nov 14 '18 at 10:13
For now I've resolved an issue by deploying this particular pipeline that has
translator property with the help of ARM templite and New-AzureRmResourceGroupDeployment PowerShell cmdlet. Moving deployment of all Data Factory objects from PowerShell to ARM is also an option, but not an easy.– Roman
Nov 14 '18 at 10:13
add a comment |
1 Answer
1
active
oldest
votes
You could try whether "Update-Module -Name AzureRm.DataFactoryV2" helps. It might be caused by that your powershell module is out of date.
answered Nov 17 '18 at 10:51
Fang Liu
1,175257
We tried to update PowerShell module. Most probably the fix to described problem is not published yet.
– Roman
Nov 20 '18 at 12:44
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53265932%2fazure-data-factory-v2-copy-activity-mapping-deployment-issue%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
You could try whether "Update-Module -Name AzureRm.DataFactoryV2" helps. It might be caused by that your powershell module is out of date.
answered Nov 17 '18 at 10:51
Fang Liu
1,175257
We tried to update PowerShell module. Most probably the fix to described problem is not published yet.
– Roman
Nov 20 '18 at 12:44
add a comment |
You could try whether "Update-Module -Name AzureRm.DataFactoryV2" helps. It might be caused by that your powershell module is out of date.
answered Nov 17 '18 at 10:51
Fang Liu
1,175257
We tried to update PowerShell module. Most probably the fix to described problem is not published yet.
– Roman
Nov 20 '18 at 12:44
add a comment |
You could try whether "Update-Module -Name AzureRm.DataFactoryV2" helps. It might be caused by that your powershell module is out of date.
answered Nov 17 '18 at 10:51
Fang Liu
1,175257
You could try whether "Update-Module -Name AzureRm.DataFactoryV2" helps. It might be caused by that your powershell module is out of date.
answered Nov 17 '18 at 10:51
Fang Liu
1,175257
answered Nov 17 '18 at 10:51
Fang Liu
1,175257
answered Nov 17 '18 at 10:51
Fang Liu
1,175257
answered Nov 17 '18 at 10:51
Fang Liu
1,175257
1,175257
We tried to update PowerShell module. Most probably the fix to described problem is not published yet.
– Roman
Nov 20 '18 at 12:44
add a comment |
We tried to update PowerShell module. Most probably the fix to described problem is not published yet.
– Roman
Nov 20 '18 at 12:44
We tried to update PowerShell module. Most probably the fix to described problem is not published yet.
– Roman
Nov 20 '18 at 12:44
We tried to update PowerShell module. Most probably the fix to described problem is not published yet.
– Roman
Nov 20 '18 at 12:44
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53265932%2fazure-data-factory-v2-copy-activity-mapping-deployment-issue%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
For now I've resolved an issue by deploying this particular pipeline that has
translatorproperty with the help of ARM templite andNew-AzureRmResourceGroupDeploymentPowerShell cmdlet. Moving deployment of all Data Factory objects from PowerShell to ARM is also an option, but not an easy.– Roman
Nov 14 '18 at 10:13