Ndtyjky

I have a project using a canned estimator in Tensorflow and trying to produce train_and_evaluate method.

estimator = tf.estimator.DNNClassifier(

    feature_columns=my_feature_columns,

    hidden_units=hidden_units,

    model_dir=model_dir,

    optimizer=tf.train.ProximalAdagradOptimizer(learning_rate=0.01,l1_regularization_strength=0.001))

Every time that I see the console output it shows that the loss is always zero.

INFO:tensorflow:loss = 2.896826e-06, step = 875

INFO:tensorflow:global_step/sec: 5.96785

INFO:tensorflow:loss = 1.9453131e-05, step = 975 (16.756 sec)

INFO:tensorflow:global_step/sec: 7.2834

INFO:tensorflow:loss = 8.6957414e-05, step = 1075 (13.730 sec)

INFO:tensorflow:global_step/sec: 7.36042

INFO:tensorflow:loss = 0.0004585028, step = 1175 (13.586 sec)

INFO:tensorflow:global_step/sec: 7.38419

INFO:tensorflow:loss = 0.0012249642, step = 1275 (13.542 sec)

INFO:tensorflow:global_step/sec: 7.3658

INFO:tensorflow:loss = 0.002194246, step = 1375 (13.576 sec)

INFO:tensorflow:global_step/sec: 7.33054

INFO:tensorflow:loss = 0.0031063582, step = 1475 (13.641 sec)

This happened since I change my input_fn (I used to load my CSV into a pandas Dataframe and work from there, however my total Dataset is over 10GB (800x1500000 in dimensions), and every time that I used to save the model, the model folder size used to be crazy large (over 200GB), so I decided to use iterators instead (I found this input function in a tutorial somewhere and it works well):

def input_fn_train(filenames,

                        num_epochs=None,

                        shuffle=True,

                        skip_header_lines=0,

                        batch_size=200,

                        modeTrainEval=True):

    filename_dataset = tf.data.Dataset.from_tensor_slices(filenames)

    if shuffle:

        filename_dataset = filename_dataset.shuffle(len(filenames))

    dataset = filename_dataset.flat_map(lambda filename: tf.data.TextLineDataset(filename).skip(skip_header_lines))

    dataset = dataset.map(parse_csv)

    if shuffle:

        dataset = dataset.shuffle(buffer_size=batch_size * 10)

    dataset = dataset.repeat(num_epochs)

    dataset = dataset.batch(batch_size)

    iterator = dataset.make_one_shot_iterator()

    features = iterator.get_next()

    features, labels = features, features.pop(LABEL_COLUMN)

    if not modeTrainEval:

        return features, None

    return features, labels

Unfortunately, this change caused my loss being always be zero, and as a consequence the predictions are terribly bad (50% accuracy), and I can't find the reason why.

(github link with sample dataset and my code)