Apache Flink : architecture question : backpressure and handling failure mode
I have just started reading about Flink and wanted to know more about how Flink handles backpressure and how it handles failures when there is backpressure. I have read this blog post by data-artisans https://data-artisans.com/blog/how-flink-handles-backpressure and kind of understand the notion of buffer pools and how they deal with backpressure.
My Questions are
(1) how does watermark help in case of remote exchange (when the tasks are on different nodes and buffers are copied to the wire TCP) -- read this in the blog above.
(2) if there is a failure when the system is facing backpressure how is it dealt with?
(3) Is it just the async checkpoints that are used to recover? What does this checkpoint include?
(4) How much of the buffer pool's data state is captured in a checkpoint?
apache-flink backpressure
asked Nov 15 '18 at 17:35
Lakshmi ManasaLakshmi Manasa
1
add a comment |
I have just started reading about Flink and wanted to know more about how Flink handles backpressure and how it handles failures when there is backpressure. I have read this blog post by data-artisans https://data-artisans.com/blog/how-flink-handles-backpressure and kind of understand the notion of buffer pools and how they deal with backpressure.
My Questions are
(1) how does watermark help in case of remote exchange (when the tasks are on different nodes and buffers are copied to the wire TCP) -- read this in the blog above.
(2) if there is a failure when the system is facing backpressure how is it dealt with?
(3) Is it just the async checkpoints that are used to recover? What does this checkpoint include?
(4) How much of the buffer pool's data state is captured in a checkpoint?
apache-flink backpressure
asked Nov 15 '18 at 17:35
Lakshmi ManasaLakshmi Manasa
1
add a comment |
I have just started reading about Flink and wanted to know more about how Flink handles backpressure and how it handles failures when there is backpressure. I have read this blog post by data-artisans https://data-artisans.com/blog/how-flink-handles-backpressure and kind of understand the notion of buffer pools and how they deal with backpressure.
My Questions are
(1) how does watermark help in case of remote exchange (when the tasks are on different nodes and buffers are copied to the wire TCP) -- read this in the blog above.
(2) if there is a failure when the system is facing backpressure how is it dealt with?
(3) Is it just the async checkpoints that are used to recover? What does this checkpoint include?
(4) How much of the buffer pool's data state is captured in a checkpoint?
apache-flink backpressure
asked Nov 15 '18 at 17:35
Lakshmi ManasaLakshmi Manasa
1
I have just started reading about Flink and wanted to know more about how Flink handles backpressure and how it handles failures when there is backpressure. I have read this blog post by data-artisans https://data-artisans.com/blog/how-flink-handles-backpressure and kind of understand the notion of buffer pools and how they deal with backpressure.
My Questions are
(1) how does watermark help in case of remote exchange (when the tasks are on different nodes and buffers are copied to the wire TCP) -- read this in the blog above.
(2) if there is a failure when the system is facing backpressure how is it dealt with?
(3) Is it just the async checkpoints that are used to recover? What does this checkpoint include?
(4) How much of the buffer pool's data state is captured in a checkpoint?
apache-flink backpressure
apache-flink backpressure
asked Nov 15 '18 at 17:35
Lakshmi ManasaLakshmi Manasa
1
asked Nov 15 '18 at 17:35
Lakshmi ManasaLakshmi Manasa
1
asked Nov 15 '18 at 17:35
Lakshmi ManasaLakshmi Manasa
1
asked Nov 15 '18 at 17:35
Lakshmi ManasaLakshmi Manasa
1
asked Nov 15 '18 at 17:35
Lakshmi ManasaLakshmi Manasa
1
1
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
The best resource for understanding how Flink's network stack currently behaves is this talk by Nico Kruber at Flink Forward Berlin 2018: Improving throughput and latency with Flink’s network stack.
As for checkpointing, Data Streaming Fault Tolerance gives a pretty thorough explanation of how it works, and what data is captured in each snapshot.
answered Nov 16 '18 at 9:28
David AndersonDavid Anderson
6,43921424
Thank you for the super useful links David. After going through these, I am still not clear as to how watermark helps in backpressure.
– Lakshmi Manasa
Nov 16 '18 at 23:48
I believe the term "watermark" is used in that blog post in a generic way, and is not referring to event-time watermarks. Furthermore, the network stack's flow control has been redesigned and reimplemented since then, and is now using the credit-based flow control described in the Flink Forward talk.
– David Anderson
Nov 17 '18 at 9:54
If you want to do a deep dive into how watermarks, checkpointing, and backpressure relate to one another, see this mailing list thread: apache-flink-mailing-list-archive.1008284.n3.nabble.com/….
– David Anderson
Nov 17 '18 at 10:00
Ah, I see. Thanks again David. Will go through these
– Lakshmi Manasa
Nov 18 '18 at 5:34
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53325038%2fapache-flink-architecture-question-backpressure-and-handling-failure-mode%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
The best resource for understanding how Flink's network stack currently behaves is this talk by Nico Kruber at Flink Forward Berlin 2018: Improving throughput and latency with Flink’s network stack.
As for checkpointing, Data Streaming Fault Tolerance gives a pretty thorough explanation of how it works, and what data is captured in each snapshot.
answered Nov 16 '18 at 9:28
David AndersonDavid Anderson
6,43921424
Thank you for the super useful links David. After going through these, I am still not clear as to how watermark helps in backpressure.
– Lakshmi Manasa
Nov 16 '18 at 23:48
I believe the term "watermark" is used in that blog post in a generic way, and is not referring to event-time watermarks. Furthermore, the network stack's flow control has been redesigned and reimplemented since then, and is now using the credit-based flow control described in the Flink Forward talk.
– David Anderson
Nov 17 '18 at 9:54
If you want to do a deep dive into how watermarks, checkpointing, and backpressure relate to one another, see this mailing list thread: apache-flink-mailing-list-archive.1008284.n3.nabble.com/….
– David Anderson
Nov 17 '18 at 10:00
Ah, I see. Thanks again David. Will go through these
– Lakshmi Manasa
Nov 18 '18 at 5:34
add a comment |
The best resource for understanding how Flink's network stack currently behaves is this talk by Nico Kruber at Flink Forward Berlin 2018: Improving throughput and latency with Flink’s network stack.
As for checkpointing, Data Streaming Fault Tolerance gives a pretty thorough explanation of how it works, and what data is captured in each snapshot.
answered Nov 16 '18 at 9:28
David AndersonDavid Anderson
6,43921424
Thank you for the super useful links David. After going through these, I am still not clear as to how watermark helps in backpressure.
– Lakshmi Manasa
Nov 16 '18 at 23:48
I believe the term "watermark" is used in that blog post in a generic way, and is not referring to event-time watermarks. Furthermore, the network stack's flow control has been redesigned and reimplemented since then, and is now using the credit-based flow control described in the Flink Forward talk.
– David Anderson
Nov 17 '18 at 9:54
If you want to do a deep dive into how watermarks, checkpointing, and backpressure relate to one another, see this mailing list thread: apache-flink-mailing-list-archive.1008284.n3.nabble.com/….
– David Anderson
Nov 17 '18 at 10:00
Ah, I see. Thanks again David. Will go through these
– Lakshmi Manasa
Nov 18 '18 at 5:34
add a comment |
The best resource for understanding how Flink's network stack currently behaves is this talk by Nico Kruber at Flink Forward Berlin 2018: Improving throughput and latency with Flink’s network stack.
As for checkpointing, Data Streaming Fault Tolerance gives a pretty thorough explanation of how it works, and what data is captured in each snapshot.
answered Nov 16 '18 at 9:28
David AndersonDavid Anderson
6,43921424
The best resource for understanding how Flink's network stack currently behaves is this talk by Nico Kruber at Flink Forward Berlin 2018: Improving throughput and latency with Flink’s network stack.
As for checkpointing, Data Streaming Fault Tolerance gives a pretty thorough explanation of how it works, and what data is captured in each snapshot.
answered Nov 16 '18 at 9:28
David AndersonDavid Anderson
6,43921424
answered Nov 16 '18 at 9:28
David AndersonDavid Anderson
6,43921424
answered Nov 16 '18 at 9:28
David AndersonDavid Anderson
6,43921424
answered Nov 16 '18 at 9:28
David AndersonDavid Anderson
6,43921424
6,43921424
Thank you for the super useful links David. After going through these, I am still not clear as to how watermark helps in backpressure.
– Lakshmi Manasa
Nov 16 '18 at 23:48
I believe the term "watermark" is used in that blog post in a generic way, and is not referring to event-time watermarks. Furthermore, the network stack's flow control has been redesigned and reimplemented since then, and is now using the credit-based flow control described in the Flink Forward talk.
– David Anderson
Nov 17 '18 at 9:54
If you want to do a deep dive into how watermarks, checkpointing, and backpressure relate to one another, see this mailing list thread: apache-flink-mailing-list-archive.1008284.n3.nabble.com/….
– David Anderson
Nov 17 '18 at 10:00
Ah, I see. Thanks again David. Will go through these
– Lakshmi Manasa
Nov 18 '18 at 5:34
add a comment |
Thank you for the super useful links David. After going through these, I am still not clear as to how watermark helps in backpressure.
– Lakshmi Manasa
Nov 16 '18 at 23:48
I believe the term "watermark" is used in that blog post in a generic way, and is not referring to event-time watermarks. Furthermore, the network stack's flow control has been redesigned and reimplemented since then, and is now using the credit-based flow control described in the Flink Forward talk.
– David Anderson
Nov 17 '18 at 9:54
If you want to do a deep dive into how watermarks, checkpointing, and backpressure relate to one another, see this mailing list thread: apache-flink-mailing-list-archive.1008284.n3.nabble.com/….
– David Anderson
Nov 17 '18 at 10:00
Ah, I see. Thanks again David. Will go through these
– Lakshmi Manasa
Nov 18 '18 at 5:34
Thank you for the super useful links David. After going through these, I am still not clear as to how watermark helps in backpressure.
– Lakshmi Manasa
Nov 16 '18 at 23:48
Thank you for the super useful links David. After going through these, I am still not clear as to how watermark helps in backpressure.
– Lakshmi Manasa
Nov 16 '18 at 23:48
I believe the term "watermark" is used in that blog post in a generic way, and is not referring to event-time watermarks. Furthermore, the network stack's flow control has been redesigned and reimplemented since then, and is now using the credit-based flow control described in the Flink Forward talk.
– David Anderson
Nov 17 '18 at 9:54
I believe the term "watermark" is used in that blog post in a generic way, and is not referring to event-time watermarks. Furthermore, the network stack's flow control has been redesigned and reimplemented since then, and is now using the credit-based flow control described in the Flink Forward talk.
– David Anderson
Nov 17 '18 at 9:54
If you want to do a deep dive into how watermarks, checkpointing, and backpressure relate to one another, see this mailing list thread: apache-flink-mailing-list-archive.1008284.n3.nabble.com/….
– David Anderson
Nov 17 '18 at 10:00
If you want to do a deep dive into how watermarks, checkpointing, and backpressure relate to one another, see this mailing list thread: apache-flink-mailing-list-archive.1008284.n3.nabble.com/….
– David Anderson
Nov 17 '18 at 10:00
Ah, I see. Thanks again David. Will go through these
– Lakshmi Manasa
Nov 18 '18 at 5:34
Ah, I see. Thanks again David. Will go through these
– Lakshmi Manasa
Nov 18 '18 at 5:34
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53325038%2fapache-flink-architecture-question-backpressure-and-handling-failure-mode%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
