Kafka stream app with internal rocksdb state store not able to get the data back for a key
I have a kafka stream app running with 36 different instance (one for each partition). And I am building rest service on top of the state to access the data.
Here some code to access the data
StreamsMetadata metadata = streams.metadataForKey(store, key, serializer); --> call this find ouy which host has the key
if (localSelf.host().equals(hostStoreInfo.getHost())) {
get the key from local store
}
else {
call the remote host using restTemplate
The problem now is metadata object returned has a different host/ip but the data is on a different node. I was able to see using some application logs I printed.
The allMetadata method on KafkaStreams says the value will be update as when the partition get reassigned. But its not happening in case. What could I do.
apache-kafka apache-kafka-streams stream-processing
asked Nov 15 '18 at 22:32
user3822232user3822232
1
add a comment |
I have a kafka stream app running with 36 different instance (one for each partition). And I am building rest service on top of the state to access the data.
Here some code to access the data
StreamsMetadata metadata = streams.metadataForKey(store, key, serializer); --> call this find ouy which host has the key
if (localSelf.host().equals(hostStoreInfo.getHost())) {
get the key from local store
}
else {
call the remote host using restTemplate
The problem now is metadata object returned has a different host/ip but the data is on a different node. I was able to see using some application logs I printed.
The allMetadata method on KafkaStreams says the value will be update as when the partition get reassigned. But its not happening in case. What could I do.
apache-kafka apache-kafka-streams stream-processing
asked Nov 15 '18 at 22:32
user3822232user3822232
1
If you can reproduce reliable, please file a bug report.
– Matthias J. Sax
Nov 16 '18 at 17:28
1
Thanks I have reported the issue here issues.apache.org/jira/browse/KAFKA-7640
– user3822232
Nov 16 '18 at 19:21
add a comment |
I have a kafka stream app running with 36 different instance (one for each partition). And I am building rest service on top of the state to access the data.
Here some code to access the data
StreamsMetadata metadata = streams.metadataForKey(store, key, serializer); --> call this find ouy which host has the key
if (localSelf.host().equals(hostStoreInfo.getHost())) {
get the key from local store
}
else {
call the remote host using restTemplate
The problem now is metadata object returned has a different host/ip but the data is on a different node. I was able to see using some application logs I printed.
The allMetadata method on KafkaStreams says the value will be update as when the partition get reassigned. But its not happening in case. What could I do.
apache-kafka apache-kafka-streams stream-processing
asked Nov 15 '18 at 22:32
user3822232user3822232
1
I have a kafka stream app running with 36 different instance (one for each partition). And I am building rest service on top of the state to access the data.
Here some code to access the data
StreamsMetadata metadata = streams.metadataForKey(store, key, serializer); --> call this find ouy which host has the key
if (localSelf.host().equals(hostStoreInfo.getHost())) {
get the key from local store
}
else {
call the remote host using restTemplate
The problem now is metadata object returned has a different host/ip but the data is on a different node. I was able to see using some application logs I printed.
The allMetadata method on KafkaStreams says the value will be update as when the partition get reassigned. But its not happening in case. What could I do.
apache-kafka apache-kafka-streams stream-processing
apache-kafka apache-kafka-streams stream-processing
asked Nov 15 '18 at 22:32
user3822232user3822232
1
asked Nov 15 '18 at 22:32
user3822232user3822232
1
asked Nov 15 '18 at 22:32
user3822232user3822232
1
asked Nov 15 '18 at 22:32
user3822232user3822232
1
asked Nov 15 '18 at 22:32
user3822232user3822232
1
1
If you can reproduce reliable, please file a bug report.
– Matthias J. Sax
Nov 16 '18 at 17:28
1
Thanks I have reported the issue here issues.apache.org/jira/browse/KAFKA-7640
– user3822232
Nov 16 '18 at 19:21
add a comment |
If you can reproduce reliable, please file a bug report.
– Matthias J. Sax
Nov 16 '18 at 17:28
1
Thanks I have reported the issue here issues.apache.org/jira/browse/KAFKA-7640
– user3822232
Nov 16 '18 at 19:21
If you can reproduce reliable, please file a bug report.
– Matthias J. Sax
Nov 16 '18 at 17:28
If you can reproduce reliable, please file a bug report.
– Matthias J. Sax
Nov 16 '18 at 17:28
1
1
Thanks I have reported the issue here issues.apache.org/jira/browse/KAFKA-7640
– user3822232
Nov 16 '18 at 19:21
Thanks I have reported the issue here issues.apache.org/jira/browse/KAFKA-7640
– user3822232
Nov 16 '18 at 19:21
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53328799%2fkafka-stream-app-with-internal-rocksdb-state-store-not-able-to-get-the-data-back%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53328799%2fkafka-stream-app-with-internal-rocksdb-state-store-not-able-to-get-the-data-back%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
If you can reproduce reliable, please file a bug report.
– Matthias J. Sax
Nov 16 '18 at 17:28
1
Thanks I have reported the issue here issues.apache.org/jira/browse/KAFKA-7640
– user3822232
Nov 16 '18 at 19:21