Does interrupting keras training in a Jupyter notebook save the training?
So, I was working on a machine learning project using a Jupyter Notebook and Keras, and I started training. I came back a few hours later, only to realize that I had accidentally set the epochs to a really high number.
I'm wondering, if I stop running the cell (send a KeyboardInterrupt), will the whole training be canceled? Or will the weights from the epoch I was currently on still be saved?
I can still access the model from the next cell.
keras jupyter-notebook
asked Nov 15 '18 at 4:54
Oliver NiOliver Ni
1,41561936
add a comment |
So, I was working on a machine learning project using a Jupyter Notebook and Keras, and I started training. I came back a few hours later, only to realize that I had accidentally set the epochs to a really high number.
I'm wondering, if I stop running the cell (send a KeyboardInterrupt), will the whole training be canceled? Or will the weights from the epoch I was currently on still be saved?
I can still access the model from the next cell.
keras jupyter-notebook
asked Nov 15 '18 at 4:54
Oliver NiOliver Ni
1,41561936
Depends if you used a callback that saves the weights and the model, and you didn't include that information. In general, no.
– Matias Valdenegro
Nov 15 '18 at 6:32
add a comment |
So, I was working on a machine learning project using a Jupyter Notebook and Keras, and I started training. I came back a few hours later, only to realize that I had accidentally set the epochs to a really high number.
I'm wondering, if I stop running the cell (send a KeyboardInterrupt), will the whole training be canceled? Or will the weights from the epoch I was currently on still be saved?
I can still access the model from the next cell.
keras jupyter-notebook
asked Nov 15 '18 at 4:54
Oliver NiOliver Ni
1,41561936
So, I was working on a machine learning project using a Jupyter Notebook and Keras, and I started training. I came back a few hours later, only to realize that I had accidentally set the epochs to a really high number.
I'm wondering, if I stop running the cell (send a KeyboardInterrupt), will the whole training be canceled? Or will the weights from the epoch I was currently on still be saved?
I can still access the model from the next cell.
keras jupyter-notebook
keras jupyter-notebook
asked Nov 15 '18 at 4:54
Oliver NiOliver Ni
1,41561936
asked Nov 15 '18 at 4:54
Oliver NiOliver Ni
1,41561936
asked Nov 15 '18 at 4:54
Oliver NiOliver Ni
1,41561936
asked Nov 15 '18 at 4:54
Oliver NiOliver Ni
1,41561936
asked Nov 15 '18 at 4:54
Oliver NiOliver Ni
1,41561936
1,41561936
Depends if you used a callback that saves the weights and the model, and you didn't include that information. In general, no.
– Matias Valdenegro
Nov 15 '18 at 6:32
add a comment |
Depends if you used a callback that saves the weights and the model, and you didn't include that information. In general, no.
– Matias Valdenegro
Nov 15 '18 at 6:32
Depends if you used a callback that saves the weights and the model, and you didn't include that information. In general, no.
– Matias Valdenegro
Nov 15 '18 at 6:32
Depends if you used a callback that saves the weights and the model, and you didn't include that information. In general, no.
– Matias Valdenegro
Nov 15 '18 at 6:32
add a comment |
2 Answers
2
active
oldest
votes
The trained model will still be in memory, in the state it was in when the KeyboardInterrupt happened. As long as the Python kernel isn't stopped or the model isn't reinstantiated, you can continue to use the trained model. To test this, evaluate the model's prediction accuracy.
Note that, if you continue training the model, a KeyboardInterrupt restarts the epoch counter. That will effect any callbacks that rely on the epoch number.
answered Nov 16 '18 at 0:34
g-eojg-eoj
762
add a comment |
If you have not defined a ModelCheckpoint callback or some custom model saver callback then the answer is no.
Next time you should include the ModelCheckpoint callback, so at every epoch your model will be saved, and you can restore it
answered Nov 15 '18 at 11:27
GabeGabe
32229
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53312642%2fdoes-interrupting-keras-training-in-a-jupyter-notebook-save-the-training%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
The trained model will still be in memory, in the state it was in when the KeyboardInterrupt happened. As long as the Python kernel isn't stopped or the model isn't reinstantiated, you can continue to use the trained model. To test this, evaluate the model's prediction accuracy.
Note that, if you continue training the model, a KeyboardInterrupt restarts the epoch counter. That will effect any callbacks that rely on the epoch number.
answered Nov 16 '18 at 0:34
g-eojg-eoj
762
add a comment |
The trained model will still be in memory, in the state it was in when the KeyboardInterrupt happened. As long as the Python kernel isn't stopped or the model isn't reinstantiated, you can continue to use the trained model. To test this, evaluate the model's prediction accuracy.
Note that, if you continue training the model, a KeyboardInterrupt restarts the epoch counter. That will effect any callbacks that rely on the epoch number.
answered Nov 16 '18 at 0:34
g-eojg-eoj
762
add a comment |
The trained model will still be in memory, in the state it was in when the KeyboardInterrupt happened. As long as the Python kernel isn't stopped or the model isn't reinstantiated, you can continue to use the trained model. To test this, evaluate the model's prediction accuracy.
Note that, if you continue training the model, a KeyboardInterrupt restarts the epoch counter. That will effect any callbacks that rely on the epoch number.
answered Nov 16 '18 at 0:34
g-eojg-eoj
762
The trained model will still be in memory, in the state it was in when the KeyboardInterrupt happened. As long as the Python kernel isn't stopped or the model isn't reinstantiated, you can continue to use the trained model. To test this, evaluate the model's prediction accuracy.
Note that, if you continue training the model, a KeyboardInterrupt restarts the epoch counter. That will effect any callbacks that rely on the epoch number.
answered Nov 16 '18 at 0:34
g-eojg-eoj
762
answered Nov 16 '18 at 0:34
g-eojg-eoj
762
answered Nov 16 '18 at 0:34
g-eojg-eoj
762
answered Nov 16 '18 at 0:34
g-eojg-eoj
762
762
add a comment |
add a comment |
If you have not defined a ModelCheckpoint callback or some custom model saver callback then the answer is no.
Next time you should include the ModelCheckpoint callback, so at every epoch your model will be saved, and you can restore it
answered Nov 15 '18 at 11:27
GabeGabe
32229
add a comment |
If you have not defined a ModelCheckpoint callback or some custom model saver callback then the answer is no.
Next time you should include the ModelCheckpoint callback, so at every epoch your model will be saved, and you can restore it
answered Nov 15 '18 at 11:27
GabeGabe
32229
add a comment |
If you have not defined a ModelCheckpoint callback or some custom model saver callback then the answer is no.
Next time you should include the ModelCheckpoint callback, so at every epoch your model will be saved, and you can restore it
answered Nov 15 '18 at 11:27
GabeGabe
32229
If you have not defined a ModelCheckpoint callback or some custom model saver callback then the answer is no.
Next time you should include the ModelCheckpoint callback, so at every epoch your model will be saved, and you can restore it
answered Nov 15 '18 at 11:27
GabeGabe
32229
answered Nov 15 '18 at 11:27
GabeGabe
32229
answered Nov 15 '18 at 11:27
GabeGabe
32229
answered Nov 15 '18 at 11:27
GabeGabe
32229
32229
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53312642%2fdoes-interrupting-keras-training-in-a-jupyter-notebook-save-the-training%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Depends if you used a callback that saves the weights and the model, and you didn't include that information. In general, no.
– Matias Valdenegro
Nov 15 '18 at 6:32