Surrogate Keys using Apache Spark
We are having event based framework where we use Kafka and Spark streaming and finally persist the data in database. I come from data warehousing world. I would like to use the surrogate keys generated for some of the dimensional data that we are using. How is it usually achieved in spark world? Is surrogate key a relevant topic in the modern big data architecture? Where can I get good information on the shift from data warehousing to big data architecture and how we do data models in big data architecture?
apache-spark spark-streaming data-warehouse
asked Nov 13 '18 at 11:15
RajDataRajData
214
add a comment |
We are having event based framework where we use Kafka and Spark streaming and finally persist the data in database. I come from data warehousing world. I would like to use the surrogate keys generated for some of the dimensional data that we are using. How is it usually achieved in spark world? Is surrogate key a relevant topic in the modern big data architecture? Where can I get good information on the shift from data warehousing to big data architecture and how we do data models in big data architecture?
apache-spark spark-streaming data-warehouse
asked Nov 13 '18 at 11:15
RajDataRajData
214
add a comment |
We are having event based framework where we use Kafka and Spark streaming and finally persist the data in database. I come from data warehousing world. I would like to use the surrogate keys generated for some of the dimensional data that we are using. How is it usually achieved in spark world? Is surrogate key a relevant topic in the modern big data architecture? Where can I get good information on the shift from data warehousing to big data architecture and how we do data models in big data architecture?
apache-spark spark-streaming data-warehouse
asked Nov 13 '18 at 11:15
RajDataRajData
214
We are having event based framework where we use Kafka and Spark streaming and finally persist the data in database. I come from data warehousing world. I would like to use the surrogate keys generated for some of the dimensional data that we are using. How is it usually achieved in spark world? Is surrogate key a relevant topic in the modern big data architecture? Where can I get good information on the shift from data warehousing to big data architecture and how we do data models in big data architecture?
apache-spark spark-streaming data-warehouse
apache-spark spark-streaming data-warehouse
asked Nov 13 '18 at 11:15
RajDataRajData
214
asked Nov 13 '18 at 11:15
RajDataRajData
214
asked Nov 13 '18 at 11:15
RajDataRajData
214
asked Nov 13 '18 at 11:15
RajDataRajData
214
asked Nov 13 '18 at 11:15
RajDataRajData
214
214
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
While surrogate keys are not common in "modern big data architecture" there are still use cases for them. Surrogate keys can still simplify the queries of business intelligence use cases such as Slowly Changing Dimensions. This is more important if you are migrating an existing data warehousing architecture with business users and tools to a new big data pipeline architecture while trying to maintain the same business tools and processes where the existing system uses surrogate keys.
Whether it is a good idea depends on your use case and actual architecture.
I don't use surrogate keys for developers. There is value still, but not worth the complexity of a surrogate key pipeline. Instead of persisting surrogate keys all that logic can usually be done on the fly at query time with enough processing power / longer queries.
Surrogate keys have many uses. Perhaps I have not covered your use case. If so, tell us more about what you goal is in using surrogate keys. I am very interested to hear about what others have done as well.
answered Nov 13 '18 at 15:08
Michael WestMichael West
646413
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53279817%2fsurrogate-keys-using-apache-spark%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
While surrogate keys are not common in "modern big data architecture" there are still use cases for them. Surrogate keys can still simplify the queries of business intelligence use cases such as Slowly Changing Dimensions. This is more important if you are migrating an existing data warehousing architecture with business users and tools to a new big data pipeline architecture while trying to maintain the same business tools and processes where the existing system uses surrogate keys.
Whether it is a good idea depends on your use case and actual architecture.
I don't use surrogate keys for developers. There is value still, but not worth the complexity of a surrogate key pipeline. Instead of persisting surrogate keys all that logic can usually be done on the fly at query time with enough processing power / longer queries.
Surrogate keys have many uses. Perhaps I have not covered your use case. If so, tell us more about what you goal is in using surrogate keys. I am very interested to hear about what others have done as well.
answered Nov 13 '18 at 15:08
Michael WestMichael West
646413
add a comment |
While surrogate keys are not common in "modern big data architecture" there are still use cases for them. Surrogate keys can still simplify the queries of business intelligence use cases such as Slowly Changing Dimensions. This is more important if you are migrating an existing data warehousing architecture with business users and tools to a new big data pipeline architecture while trying to maintain the same business tools and processes where the existing system uses surrogate keys.
Whether it is a good idea depends on your use case and actual architecture.
I don't use surrogate keys for developers. There is value still, but not worth the complexity of a surrogate key pipeline. Instead of persisting surrogate keys all that logic can usually be done on the fly at query time with enough processing power / longer queries.
Surrogate keys have many uses. Perhaps I have not covered your use case. If so, tell us more about what you goal is in using surrogate keys. I am very interested to hear about what others have done as well.
answered Nov 13 '18 at 15:08
Michael WestMichael West
646413
add a comment |
While surrogate keys are not common in "modern big data architecture" there are still use cases for them. Surrogate keys can still simplify the queries of business intelligence use cases such as Slowly Changing Dimensions. This is more important if you are migrating an existing data warehousing architecture with business users and tools to a new big data pipeline architecture while trying to maintain the same business tools and processes where the existing system uses surrogate keys.
Whether it is a good idea depends on your use case and actual architecture.
I don't use surrogate keys for developers. There is value still, but not worth the complexity of a surrogate key pipeline. Instead of persisting surrogate keys all that logic can usually be done on the fly at query time with enough processing power / longer queries.
Surrogate keys have many uses. Perhaps I have not covered your use case. If so, tell us more about what you goal is in using surrogate keys. I am very interested to hear about what others have done as well.
answered Nov 13 '18 at 15:08
Michael WestMichael West
646413
While surrogate keys are not common in "modern big data architecture" there are still use cases for them. Surrogate keys can still simplify the queries of business intelligence use cases such as Slowly Changing Dimensions. This is more important if you are migrating an existing data warehousing architecture with business users and tools to a new big data pipeline architecture while trying to maintain the same business tools and processes where the existing system uses surrogate keys.
Whether it is a good idea depends on your use case and actual architecture.
I don't use surrogate keys for developers. There is value still, but not worth the complexity of a surrogate key pipeline. Instead of persisting surrogate keys all that logic can usually be done on the fly at query time with enough processing power / longer queries.
Surrogate keys have many uses. Perhaps I have not covered your use case. If so, tell us more about what you goal is in using surrogate keys. I am very interested to hear about what others have done as well.
answered Nov 13 '18 at 15:08
Michael WestMichael West
646413
answered Nov 13 '18 at 15:08
Michael WestMichael West
646413
answered Nov 13 '18 at 15:08
Michael WestMichael West
646413
answered Nov 13 '18 at 15:08
Michael WestMichael West
646413
646413
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53279817%2fsurrogate-keys-using-apache-spark%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
