How to monitor validation loss in the training of estimators in TensorFlow?
up vote
1
down vote
favorite
I want to ask a question about how to monitor validation loss in the training process of estimators in TensorFlow. I have checked a similar question (validation during training of Estimator) asked before, but it did not help much.
If I use estimators to build a model, I will give an input function to the Estimator.train() function. But there is no way to add another validation_x, and validation_y data in the training process. Therefore, when the training started, I can only see the training loss. The training loss is expected to decrease when the training process running longer. However, this information is not helpful to prevent overfitting. The more valuable information is validation loss. Usually, the validation loss is the U-shape with the number of epochs. To prevent overfitting, we want to find the number of epochs that the validation loss is minimum.
So this is my problem. How can I get validation loss for each epoch in the training process of using estimators?
python tensorflow machine-learning deep-learning
asked Nov 9 at 20:39
Han M
667
add a comment |
up vote
1
down vote
favorite
I want to ask a question about how to monitor validation loss in the training process of estimators in TensorFlow. I have checked a similar question (validation during training of Estimator) asked before, but it did not help much.
If I use estimators to build a model, I will give an input function to the Estimator.train() function. But there is no way to add another validation_x, and validation_y data in the training process. Therefore, when the training started, I can only see the training loss. The training loss is expected to decrease when the training process running longer. However, this information is not helpful to prevent overfitting. The more valuable information is validation loss. Usually, the validation loss is the U-shape with the number of epochs. To prevent overfitting, we want to find the number of epochs that the validation loss is minimum.
So this is my problem. How can I get validation loss for each epoch in the training process of using estimators?
python tensorflow machine-learning deep-learning
asked Nov 9 at 20:39
Han M
667
1
So, it looks like you need to manually control this. Define separate input functions for training and validation dataset. Train for X steps/epochs using train() and the training input, call get validation loss using evaluate() with the validation input. Decide whether or not you want to train more. Run train() again or quit.
– Mad Wombat
Nov 9 at 21:13
Hi Mad Wombat, yes. My goal is to get the right number of epochs to prevent overfitting.
– Han M
Nov 9 at 22:16
If you want to control the training on every step, you might want to skip the estimators and implement your own training cycle. If you are OK with a bit less granularity, you can implement a simple loop where you call train() for some preset number of steps or epochs (and you can adjust it as you go) and then call evaluate() to judge your progress. This is basic python we are talking about, nothing too complicated.
– Mad Wombat
Nov 10 at 5:00
add a comment |
up vote
1
down vote
favorite
up vote
1
down vote
favorite
I want to ask a question about how to monitor validation loss in the training process of estimators in TensorFlow. I have checked a similar question (validation during training of Estimator) asked before, but it did not help much.
If I use estimators to build a model, I will give an input function to the Estimator.train() function. But there is no way to add another validation_x, and validation_y data in the training process. Therefore, when the training started, I can only see the training loss. The training loss is expected to decrease when the training process running longer. However, this information is not helpful to prevent overfitting. The more valuable information is validation loss. Usually, the validation loss is the U-shape with the number of epochs. To prevent overfitting, we want to find the number of epochs that the validation loss is minimum.
So this is my problem. How can I get validation loss for each epoch in the training process of using estimators?
python tensorflow machine-learning deep-learning
asked Nov 9 at 20:39
Han M
667
I want to ask a question about how to monitor validation loss in the training process of estimators in TensorFlow. I have checked a similar question (validation during training of Estimator) asked before, but it did not help much.
If I use estimators to build a model, I will give an input function to the Estimator.train() function. But there is no way to add another validation_x, and validation_y data in the training process. Therefore, when the training started, I can only see the training loss. The training loss is expected to decrease when the training process running longer. However, this information is not helpful to prevent overfitting. The more valuable information is validation loss. Usually, the validation loss is the U-shape with the number of epochs. To prevent overfitting, we want to find the number of epochs that the validation loss is minimum.
So this is my problem. How can I get validation loss for each epoch in the training process of using estimators?
python tensorflow machine-learning deep-learning
python tensorflow machine-learning deep-learning
asked Nov 9 at 20:39
Han M
667
asked Nov 9 at 20:39
Han M
667
asked Nov 9 at 20:39
Han M
667
asked Nov 9 at 20:39
Han M
667
asked Nov 9 at 20:39
Han M
667
667
1
So, it looks like you need to manually control this. Define separate input functions for training and validation dataset. Train for X steps/epochs using train() and the training input, call get validation loss using evaluate() with the validation input. Decide whether or not you want to train more. Run train() again or quit.
– Mad Wombat
Nov 9 at 21:13
Hi Mad Wombat, yes. My goal is to get the right number of epochs to prevent overfitting.
– Han M
Nov 9 at 22:16
If you want to control the training on every step, you might want to skip the estimators and implement your own training cycle. If you are OK with a bit less granularity, you can implement a simple loop where you call train() for some preset number of steps or epochs (and you can adjust it as you go) and then call evaluate() to judge your progress. This is basic python we are talking about, nothing too complicated.
– Mad Wombat
Nov 10 at 5:00
add a comment |
1
So, it looks like you need to manually control this. Define separate input functions for training and validation dataset. Train for X steps/epochs using train() and the training input, call get validation loss using evaluate() with the validation input. Decide whether or not you want to train more. Run train() again or quit.
– Mad Wombat
Nov 9 at 21:13
Hi Mad Wombat, yes. My goal is to get the right number of epochs to prevent overfitting.
– Han M
Nov 9 at 22:16
If you want to control the training on every step, you might want to skip the estimators and implement your own training cycle. If you are OK with a bit less granularity, you can implement a simple loop where you call train() for some preset number of steps or epochs (and you can adjust it as you go) and then call evaluate() to judge your progress. This is basic python we are talking about, nothing too complicated.
– Mad Wombat
Nov 10 at 5:00
1
1
So, it looks like you need to manually control this. Define separate input functions for training and validation dataset. Train for X steps/epochs using train() and the training input, call get validation loss using evaluate() with the validation input. Decide whether or not you want to train more. Run train() again or quit.
– Mad Wombat
Nov 9 at 21:13
So, it looks like you need to manually control this. Define separate input functions for training and validation dataset. Train for X steps/epochs using train() and the training input, call get validation loss using evaluate() with the validation input. Decide whether or not you want to train more. Run train() again or quit.
– Mad Wombat
Nov 9 at 21:13
Hi Mad Wombat, yes. My goal is to get the right number of epochs to prevent overfitting.
– Han M
Nov 9 at 22:16
Hi Mad Wombat, yes. My goal is to get the right number of epochs to prevent overfitting.
– Han M
Nov 9 at 22:16
If you want to control the training on every step, you might want to skip the estimators and implement your own training cycle. If you are OK with a bit less granularity, you can implement a simple loop where you call train() for some preset number of steps or epochs (and you can adjust it as you go) and then call evaluate() to judge your progress. This is basic python we are talking about, nothing too complicated.
– Mad Wombat
Nov 10 at 5:00
If you want to control the training on every step, you might want to skip the estimators and implement your own training cycle. If you are OK with a bit less granularity, you can implement a simple loop where you call train() for some preset number of steps or epochs (and you can adjust it as you go) and then call evaluate() to judge your progress. This is basic python we are talking about, nothing too complicated.
– Mad Wombat
Nov 10 at 5:00
add a comment |
1 Answer
1
active
oldest
votes
up vote
1
down vote
accepted
You need to create a validation input_fn and either use estimator.train() and estimator.evaluate() alternatively or simpy use tf.estimator.train_and_evaluate()
x = ...
y = ...
...
# For example, if x and y are numpy arrays < 2 GB
train_dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train))
val_dataset = tf.data.Dataset.from_tensor_slices((x_val_, y_val))
...
estimator = ...
for epoch in n_epochs:
estimator.train(input_fn = train_dataset)
estimator.evaluate(input_fn = val_dataset)
estimator.evaluate() will compute the loss and any other metrics that are defined in your model_fn and will save the events in a new "eval" directory inside your job_dir.
answered Nov 10 at 17:03
Olivier Dehaene
65519
add a comment |
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
1
down vote
accepted
You need to create a validation input_fn and either use estimator.train() and estimator.evaluate() alternatively or simpy use tf.estimator.train_and_evaluate()
x = ...
y = ...
...
# For example, if x and y are numpy arrays < 2 GB
train_dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train))
val_dataset = tf.data.Dataset.from_tensor_slices((x_val_, y_val))
...
estimator = ...
for epoch in n_epochs:
estimator.train(input_fn = train_dataset)
estimator.evaluate(input_fn = val_dataset)
estimator.evaluate() will compute the loss and any other metrics that are defined in your model_fn and will save the events in a new "eval" directory inside your job_dir.
answered Nov 10 at 17:03
Olivier Dehaene
65519
add a comment |
up vote
1
down vote
accepted
You need to create a validation input_fn and either use estimator.train() and estimator.evaluate() alternatively or simpy use tf.estimator.train_and_evaluate()
x = ...
y = ...
...
# For example, if x and y are numpy arrays < 2 GB
train_dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train))
val_dataset = tf.data.Dataset.from_tensor_slices((x_val_, y_val))
...
estimator = ...
for epoch in n_epochs:
estimator.train(input_fn = train_dataset)
estimator.evaluate(input_fn = val_dataset)
estimator.evaluate() will compute the loss and any other metrics that are defined in your model_fn and will save the events in a new "eval" directory inside your job_dir.
answered Nov 10 at 17:03
Olivier Dehaene
65519
add a comment |
up vote
1
down vote
accepted
up vote
1
down vote
accepted
You need to create a validation input_fn and either use estimator.train() and estimator.evaluate() alternatively or simpy use tf.estimator.train_and_evaluate()
x = ...
y = ...
...
# For example, if x and y are numpy arrays < 2 GB
train_dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train))
val_dataset = tf.data.Dataset.from_tensor_slices((x_val_, y_val))
...
estimator = ...
for epoch in n_epochs:
estimator.train(input_fn = train_dataset)
estimator.evaluate(input_fn = val_dataset)
estimator.evaluate() will compute the loss and any other metrics that are defined in your model_fn and will save the events in a new "eval" directory inside your job_dir.
answered Nov 10 at 17:03
Olivier Dehaene
65519
You need to create a validation input_fn and either use estimator.train() and estimator.evaluate() alternatively or simpy use tf.estimator.train_and_evaluate()
x = ...
y = ...
...
# For example, if x and y are numpy arrays < 2 GB
train_dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train))
val_dataset = tf.data.Dataset.from_tensor_slices((x_val_, y_val))
...
estimator = ...
for epoch in n_epochs:
estimator.train(input_fn = train_dataset)
estimator.evaluate(input_fn = val_dataset)
estimator.evaluate() will compute the loss and any other metrics that are defined in your model_fn and will save the events in a new "eval" directory inside your job_dir.
answered Nov 10 at 17:03
Olivier Dehaene
65519
edited Nov 10 at 17:11
answered Nov 10 at 17:03
Olivier Dehaene
65519
answered Nov 10 at 17:03
Olivier Dehaene
65519
answered Nov 10 at 17:03
Olivier Dehaene
65519
65519
add a comment |
add a comment |
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53232969%2fhow-to-monitor-validation-loss-in-the-training-of-estimators-in-tensorflow%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
So, it looks like you need to manually control this. Define separate input functions for training and validation dataset. Train for X steps/epochs using train() and the training input, call get validation loss using evaluate() with the validation input. Decide whether or not you want to train more. Run train() again or quit.
– Mad Wombat
Nov 9 at 21:13
Hi Mad Wombat, yes. My goal is to get the right number of epochs to prevent overfitting.
– Han M
Nov 9 at 22:16
If you want to control the training on every step, you might want to skip the estimators and implement your own training cycle. If you are OK with a bit less granularity, you can implement a simple loop where you call train() for some preset number of steps or epochs (and you can adjust it as you go) and then call evaluate() to judge your progress. This is basic python we are talking about, nothing too complicated.
– Mad Wombat
Nov 10 at 5:00