Does Insert Overwrite deletes the data and partition in hive?
I am creating a partition on year and month. suppose previous partition exist in target hive table is year = 2018 and month = 10 and month = 11. Now new data is coming having year = 2018 and month = 11 only.
So my question is will insert overwrite remove the data and partition for month = 10?
Please suggest me a solution as I want to remove data and partition for month = 10 too.
Thanks
hive partitioning overwrite
edited Nov 13 at 0:24
VIN
13511
asked Nov 12 at 8:38
Manish Pansari
17116
add a comment |
I am creating a partition on year and month. suppose previous partition exist in target hive table is year = 2018 and month = 10 and month = 11. Now new data is coming having year = 2018 and month = 11 only.
So my question is will insert overwrite remove the data and partition for month = 10?
Please suggest me a solution as I want to remove data and partition for month = 10 too.
Thanks
hive partitioning overwrite
edited Nov 13 at 0:24
VIN
13511
asked Nov 12 at 8:38
Manish Pansari
17116
Do you want to remove the old partitions everytime when a new month data arrives?
– Nishu Tayal
Nov 12 at 9:04
Yes , this is the requirement , FYI target table is
– Manish Pansari
Nov 13 at 14:20
add a comment |
I am creating a partition on year and month. suppose previous partition exist in target hive table is year = 2018 and month = 10 and month = 11. Now new data is coming having year = 2018 and month = 11 only.
So my question is will insert overwrite remove the data and partition for month = 10?
Please suggest me a solution as I want to remove data and partition for month = 10 too.
Thanks
hive partitioning overwrite
edited Nov 13 at 0:24
VIN
13511
asked Nov 12 at 8:38
Manish Pansari
17116
I am creating a partition on year and month. suppose previous partition exist in target hive table is year = 2018 and month = 10 and month = 11. Now new data is coming having year = 2018 and month = 11 only.
So my question is will insert overwrite remove the data and partition for month = 10?
Please suggest me a solution as I want to remove data and partition for month = 10 too.
Thanks
hive partitioning overwrite
hive partitioning overwrite
edited Nov 13 at 0:24
VIN
13511
asked Nov 12 at 8:38
Manish Pansari
17116
edited Nov 13 at 0:24
VIN
13511
asked Nov 12 at 8:38
Manish Pansari
17116
edited Nov 13 at 0:24
VIN
13511
edited Nov 13 at 0:24
VIN
13511
edited Nov 13 at 0:24
VIN
13511
13511
asked Nov 12 at 8:38
Manish Pansari
17116
asked Nov 12 at 8:38
Manish Pansari
17116
asked Nov 12 at 8:38
Manish Pansari
17116
17116
Do you want to remove the old partitions everytime when a new month data arrives?
– Nishu Tayal
Nov 12 at 9:04
Yes , this is the requirement , FYI target table is
– Manish Pansari
Nov 13 at 14:20
add a comment |
Do you want to remove the old partitions everytime when a new month data arrives?
– Nishu Tayal
Nov 12 at 9:04
Yes , this is the requirement , FYI target table is
– Manish Pansari
Nov 13 at 14:20
Do you want to remove the old partitions everytime when a new month data arrives?
– Nishu Tayal
Nov 12 at 9:04
Do you want to remove the old partitions everytime when a new month data arrives?
– Nishu Tayal
Nov 12 at 9:04
Yes , this is the requirement , FYI target table is
– Manish Pansari
Nov 13 at 14:20
Yes , this is the requirement , FYI target table is
– Manish Pansari
Nov 13 at 14:20
add a comment |
2 Answers
2
active
oldest
votes
When Hive tries to “INSERT OVERWRITE” to a partition of an external table under an existing directory, depending on whether the partition definition already exists in the metastore or not, Hive will behave differently:
a) if partition definition does not exist, it will not try to guess where the target partition directories are (either static or dynamic partitions), so it will not be able to delete existing files under those partitions that will be written to
b) if partition definition does exist, it will attempt to remove all files under the target partition directory before writing new data into those directories.
So In a nutshell, in your case, it will not delete the data.
answered Nov 12 at 21:33
VIN
13511
add a comment |
If you want to delete all existing partitions and keep only the new month data, you can use DROP PARTITION command with comparators.
Eg: If there are partitions for year 2018 and month 10 and before that. You can configure using either one
// If there is always only one partition
alter table part_t drop partition (year=2003,month=1);
// If there are multiple partitions, you can use < comparator
alter table part_t drop partition (year <2003,month<1);
answered Nov 13 at 22:21
Nishu Tayal
11.5k73481
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53258421%2fdoes-insert-overwrite-deletes-the-data-and-partition-in-hive%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
When Hive tries to “INSERT OVERWRITE” to a partition of an external table under an existing directory, depending on whether the partition definition already exists in the metastore or not, Hive will behave differently:
a) if partition definition does not exist, it will not try to guess where the target partition directories are (either static or dynamic partitions), so it will not be able to delete existing files under those partitions that will be written to
b) if partition definition does exist, it will attempt to remove all files under the target partition directory before writing new data into those directories.
So In a nutshell, in your case, it will not delete the data.
answered Nov 12 at 21:33
VIN
13511
add a comment |
When Hive tries to “INSERT OVERWRITE” to a partition of an external table under an existing directory, depending on whether the partition definition already exists in the metastore or not, Hive will behave differently:
a) if partition definition does not exist, it will not try to guess where the target partition directories are (either static or dynamic partitions), so it will not be able to delete existing files under those partitions that will be written to
b) if partition definition does exist, it will attempt to remove all files under the target partition directory before writing new data into those directories.
So In a nutshell, in your case, it will not delete the data.
answered Nov 12 at 21:33
VIN
13511
add a comment |
When Hive tries to “INSERT OVERWRITE” to a partition of an external table under an existing directory, depending on whether the partition definition already exists in the metastore or not, Hive will behave differently:
a) if partition definition does not exist, it will not try to guess where the target partition directories are (either static or dynamic partitions), so it will not be able to delete existing files under those partitions that will be written to
b) if partition definition does exist, it will attempt to remove all files under the target partition directory before writing new data into those directories.
So In a nutshell, in your case, it will not delete the data.
answered Nov 12 at 21:33
VIN
13511
When Hive tries to “INSERT OVERWRITE” to a partition of an external table under an existing directory, depending on whether the partition definition already exists in the metastore or not, Hive will behave differently:
a) if partition definition does not exist, it will not try to guess where the target partition directories are (either static or dynamic partitions), so it will not be able to delete existing files under those partitions that will be written to
b) if partition definition does exist, it will attempt to remove all files under the target partition directory before writing new data into those directories.
So In a nutshell, in your case, it will not delete the data.
answered Nov 12 at 21:33
VIN
13511
edited Nov 12 at 21:46
answered Nov 12 at 21:33
VIN
13511
answered Nov 12 at 21:33
VIN
13511
answered Nov 12 at 21:33
VIN
13511
13511
add a comment |
add a comment |
If you want to delete all existing partitions and keep only the new month data, you can use DROP PARTITION command with comparators.
Eg: If there are partitions for year 2018 and month 10 and before that. You can configure using either one
// If there is always only one partition
alter table part_t drop partition (year=2003,month=1);
// If there are multiple partitions, you can use < comparator
alter table part_t drop partition (year <2003,month<1);
answered Nov 13 at 22:21
Nishu Tayal
11.5k73481
add a comment |
If you want to delete all existing partitions and keep only the new month data, you can use DROP PARTITION command with comparators.
Eg: If there are partitions for year 2018 and month 10 and before that. You can configure using either one
// If there is always only one partition
alter table part_t drop partition (year=2003,month=1);
// If there are multiple partitions, you can use < comparator
alter table part_t drop partition (year <2003,month<1);
answered Nov 13 at 22:21
Nishu Tayal
11.5k73481
add a comment |
If you want to delete all existing partitions and keep only the new month data, you can use DROP PARTITION command with comparators.
Eg: If there are partitions for year 2018 and month 10 and before that. You can configure using either one
// If there is always only one partition
alter table part_t drop partition (year=2003,month=1);
// If there are multiple partitions, you can use < comparator
alter table part_t drop partition (year <2003,month<1);
answered Nov 13 at 22:21
Nishu Tayal
11.5k73481
If you want to delete all existing partitions and keep only the new month data, you can use DROP PARTITION command with comparators.
Eg: If there are partitions for year 2018 and month 10 and before that. You can configure using either one
// If there is always only one partition
alter table part_t drop partition (year=2003,month=1);
// If there are multiple partitions, you can use < comparator
alter table part_t drop partition (year <2003,month<1);
answered Nov 13 at 22:21
Nishu Tayal
11.5k73481
answered Nov 13 at 22:21
Nishu Tayal
11.5k73481
answered Nov 13 at 22:21
Nishu Tayal
11.5k73481
answered Nov 13 at 22:21
Nishu Tayal
11.5k73481
11.5k73481
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53258421%2fdoes-insert-overwrite-deletes-the-data-and-partition-in-hive%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Do you want to remove the old partitions everytime when a new month data arrives?
– Nishu Tayal
Nov 12 at 9:04
Yes , this is the requirement , FYI target table is
– Manish Pansari
Nov 13 at 14:20