Jenkins Declarative Pipeline How to block parallel stage until all sub-stages have completed
I'm running Jenkins Declarative Pipeline controlles by a central Jenkins master and running on 2 slaves on 2 different sites, siteA and siteB.
I have one stage that needs to be run on both sites (ideally in parallel to save time) that waits until some resources are loaded. So this stage basically runs a script that checks if the resources are loaded and if not, waits X seconds and tries again until all resources are loaded.
What happens is that at one site, resource loading is faster than on the other site so when one site finishes the whole stage is done although the other site is not yet complete.
The pipeline for this stage looks like this:
stage('myStage') {
parallel {
stage('myStage-siteA') {
agent {
node {
label 'siteA'
}
}
steps {
waitForResourcesLoaded(siteA)
}
}
stage('myStage-siteB') {
agent {
node {
label 'siteB'
}
}
steps {
waitForResourcesLoaded(siteB)
}
}
}
}
Is there any way to "synchronize" each parallel stage so that the overall stage "myStage" will only be marked complete once each sub-stage has completed?
jenkins jenkins-pipeline jenkins-declarative-pipeline
asked Nov 12 '18 at 11:47
phivo
76113
|
show 1 more comment
I'm running Jenkins Declarative Pipeline controlles by a central Jenkins master and running on 2 slaves on 2 different sites, siteA and siteB.
I have one stage that needs to be run on both sites (ideally in parallel to save time) that waits until some resources are loaded. So this stage basically runs a script that checks if the resources are loaded and if not, waits X seconds and tries again until all resources are loaded.
What happens is that at one site, resource loading is faster than on the other site so when one site finishes the whole stage is done although the other site is not yet complete.
The pipeline for this stage looks like this:
stage('myStage') {
parallel {
stage('myStage-siteA') {
agent {
node {
label 'siteA'
}
}
steps {
waitForResourcesLoaded(siteA)
}
}
stage('myStage-siteB') {
agent {
node {
label 'siteB'
}
}
steps {
waitForResourcesLoaded(siteB)
}
}
}
}
Is there any way to "synchronize" each parallel stage so that the overall stage "myStage" will only be marked complete once each sub-stage has completed?
jenkins jenkins-pipeline jenkins-declarative-pipeline
asked Nov 12 '18 at 11:47
phivo
76113
What exactly is your "waitForResourcesLoaded" step doing?
– Fitzi
Nov 12 '18 at 11:56
Let's say its calling a URL and parsing the response for a certain keyword. If the keyword is not present it waits 10 seconds and polls again until the keyword is present.
– phivo
Nov 12 '18 at 13:51
How do you achieve this? Can you assure that these steps (especially the 10 second wait) are beeing executed in a blocking fashion?
– Fitzi
Nov 12 '18 at 15:17
That is the issue. Both parallel stages run completely independent on 2 different slave nodes and as soon as one slave finishes the whole stage is finished even if the stage on the second slave is not complete.
– phivo
Nov 13 '18 at 6:42
Or let's make an even easier example: Create a parallel stage with 2 sub-stages running on 2 different slave nodes. Have one sub-stage wait for 30 seconds and the other wait for 60 seconds. The whole stage will finish after 30 seconds, i.e. when the first slave finishes, and not wait for 60 seconds until the other slave has finished as well.
– phivo
Nov 13 '18 at 6:44
|
show 1 more comment
I'm running Jenkins Declarative Pipeline controlles by a central Jenkins master and running on 2 slaves on 2 different sites, siteA and siteB.
I have one stage that needs to be run on both sites (ideally in parallel to save time) that waits until some resources are loaded. So this stage basically runs a script that checks if the resources are loaded and if not, waits X seconds and tries again until all resources are loaded.
What happens is that at one site, resource loading is faster than on the other site so when one site finishes the whole stage is done although the other site is not yet complete.
The pipeline for this stage looks like this:
stage('myStage') {
parallel {
stage('myStage-siteA') {
agent {
node {
label 'siteA'
}
}
steps {
waitForResourcesLoaded(siteA)
}
}
stage('myStage-siteB') {
agent {
node {
label 'siteB'
}
}
steps {
waitForResourcesLoaded(siteB)
}
}
}
}
Is there any way to "synchronize" each parallel stage so that the overall stage "myStage" will only be marked complete once each sub-stage has completed?
jenkins jenkins-pipeline jenkins-declarative-pipeline
asked Nov 12 '18 at 11:47
phivo
76113
I'm running Jenkins Declarative Pipeline controlles by a central Jenkins master and running on 2 slaves on 2 different sites, siteA and siteB.
I have one stage that needs to be run on both sites (ideally in parallel to save time) that waits until some resources are loaded. So this stage basically runs a script that checks if the resources are loaded and if not, waits X seconds and tries again until all resources are loaded.
What happens is that at one site, resource loading is faster than on the other site so when one site finishes the whole stage is done although the other site is not yet complete.
The pipeline for this stage looks like this:
stage('myStage') {
parallel {
stage('myStage-siteA') {
agent {
node {
label 'siteA'
}
}
steps {
waitForResourcesLoaded(siteA)
}
}
stage('myStage-siteB') {
agent {
node {
label 'siteB'
}
}
steps {
waitForResourcesLoaded(siteB)
}
}
}
}
Is there any way to "synchronize" each parallel stage so that the overall stage "myStage" will only be marked complete once each sub-stage has completed?
jenkins jenkins-pipeline jenkins-declarative-pipeline
jenkins jenkins-pipeline jenkins-declarative-pipeline
asked Nov 12 '18 at 11:47
phivo
76113
asked Nov 12 '18 at 11:47
phivo
76113
asked Nov 12 '18 at 11:47
phivo
76113
asked Nov 12 '18 at 11:47
phivo
76113
asked Nov 12 '18 at 11:47
phivo
76113
76113
What exactly is your "waitForResourcesLoaded" step doing?
– Fitzi
Nov 12 '18 at 11:56
Let's say its calling a URL and parsing the response for a certain keyword. If the keyword is not present it waits 10 seconds and polls again until the keyword is present.
– phivo
Nov 12 '18 at 13:51
How do you achieve this? Can you assure that these steps (especially the 10 second wait) are beeing executed in a blocking fashion?
– Fitzi
Nov 12 '18 at 15:17
That is the issue. Both parallel stages run completely independent on 2 different slave nodes and as soon as one slave finishes the whole stage is finished even if the stage on the second slave is not complete.
– phivo
Nov 13 '18 at 6:42
Or let's make an even easier example: Create a parallel stage with 2 sub-stages running on 2 different slave nodes. Have one sub-stage wait for 30 seconds and the other wait for 60 seconds. The whole stage will finish after 30 seconds, i.e. when the first slave finishes, and not wait for 60 seconds until the other slave has finished as well.
– phivo
Nov 13 '18 at 6:44
|
show 1 more comment
What exactly is your "waitForResourcesLoaded" step doing?
– Fitzi
Nov 12 '18 at 11:56
Let's say its calling a URL and parsing the response for a certain keyword. If the keyword is not present it waits 10 seconds and polls again until the keyword is present.
– phivo
Nov 12 '18 at 13:51
How do you achieve this? Can you assure that these steps (especially the 10 second wait) are beeing executed in a blocking fashion?
– Fitzi
Nov 12 '18 at 15:17
That is the issue. Both parallel stages run completely independent on 2 different slave nodes and as soon as one slave finishes the whole stage is finished even if the stage on the second slave is not complete.
– phivo
Nov 13 '18 at 6:42
Or let's make an even easier example: Create a parallel stage with 2 sub-stages running on 2 different slave nodes. Have one sub-stage wait for 30 seconds and the other wait for 60 seconds. The whole stage will finish after 30 seconds, i.e. when the first slave finishes, and not wait for 60 seconds until the other slave has finished as well.
– phivo
Nov 13 '18 at 6:44
What exactly is your "waitForResourcesLoaded" step doing?
– Fitzi
Nov 12 '18 at 11:56
What exactly is your "waitForResourcesLoaded" step doing?
– Fitzi
Nov 12 '18 at 11:56
Let's say its calling a URL and parsing the response for a certain keyword. If the keyword is not present it waits 10 seconds and polls again until the keyword is present.
– phivo
Nov 12 '18 at 13:51
Let's say its calling a URL and parsing the response for a certain keyword. If the keyword is not present it waits 10 seconds and polls again until the keyword is present.
– phivo
Nov 12 '18 at 13:51
How do you achieve this? Can you assure that these steps (especially the 10 second wait) are beeing executed in a blocking fashion?
– Fitzi
Nov 12 '18 at 15:17
How do you achieve this? Can you assure that these steps (especially the 10 second wait) are beeing executed in a blocking fashion?
– Fitzi
Nov 12 '18 at 15:17
That is the issue. Both parallel stages run completely independent on 2 different slave nodes and as soon as one slave finishes the whole stage is finished even if the stage on the second slave is not complete.
– phivo
Nov 13 '18 at 6:42
That is the issue. Both parallel stages run completely independent on 2 different slave nodes and as soon as one slave finishes the whole stage is finished even if the stage on the second slave is not complete.
– phivo
Nov 13 '18 at 6:42
Or let's make an even easier example: Create a parallel stage with 2 sub-stages running on 2 different slave nodes. Have one sub-stage wait for 30 seconds and the other wait for 60 seconds. The whole stage will finish after 30 seconds, i.e. when the first slave finishes, and not wait for 60 seconds until the other slave has finished as well.
– phivo
Nov 13 '18 at 6:44
Or let's make an even easier example: Create a parallel stage with 2 sub-stages running on 2 different slave nodes. Have one sub-stage wait for 30 seconds and the other wait for 60 seconds. The whole stage will finish after 30 seconds, i.e. when the first slave finishes, and not wait for 60 seconds until the other slave has finished as well.
– phivo
Nov 13 '18 at 6:44
|
show 1 more comment
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53261523%2fjenkins-declarative-pipeline-how-to-block-parallel-stage-until-all-sub-stages-ha%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53261523%2fjenkins-declarative-pipeline-how-to-block-parallel-stage-until-all-sub-stages-ha%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
What exactly is your "waitForResourcesLoaded" step doing?
– Fitzi
Nov 12 '18 at 11:56
Let's say its calling a URL and parsing the response for a certain keyword. If the keyword is not present it waits 10 seconds and polls again until the keyword is present.
– phivo
Nov 12 '18 at 13:51
How do you achieve this? Can you assure that these steps (especially the 10 second wait) are beeing executed in a blocking fashion?
– Fitzi
Nov 12 '18 at 15:17
That is the issue. Both parallel stages run completely independent on 2 different slave nodes and as soon as one slave finishes the whole stage is finished even if the stage on the second slave is not complete.
– phivo
Nov 13 '18 at 6:42
Or let's make an even easier example: Create a parallel stage with 2 sub-stages running on 2 different slave nodes. Have one sub-stage wait for 30 seconds and the other wait for 60 seconds. The whole stage will finish after 30 seconds, i.e. when the first slave finishes, and not wait for 60 seconds until the other slave has finished as well.
– phivo
Nov 13 '18 at 6:44