Ndtyjky

I am using the NER implementation in NER. with this configuration:

# embeddings

dim_word = 300

dim_char = 100



# training

train_embeddings = False

nepochs          = 15

dropout          = 0.5

batch_size       = 20

lr_method        = "adam"

lr               = 0.001

lr_decay         = 0.9

clip             = -1 # if negative, no clipping

nepoch_no_imprv  = 3



# model hyperparameters

hidden_size_char = 100 # lstm on chars

hidden_size_lstm = 300 # lstm on word embeddings

The size of the training data I have is 3000 sentences. When I just use ca. 2500 sentences the training is done without any issue. When I use all training sentences I got this error:

 Epoch 1 out of 15

 58/115 [==============>...............] - ETA: 734s - train loss: 15.6397Traceback (most recent call last):

  File "**/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call

    return fn(*args)

  File "**/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn

    options, feed_dict, fetch_list, target_list, run_metadata)

  File "**/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun

    run_metadata)

tensorflow.python.framework.errors_impl.InvalidArgumentError: indices[1,21,0] = 136 is not in [0, 136)

         [[{{node chars/char_embeddings}} = GatherV2[Taxis=DT_INT32, Tindices=DT_INT32, Tparams=DT_FLOAT, _class=["loc:@train_step/Adam/update_chars/_char_embeddings/AssignSub"], _device="/job:localhost/replica:0/task:0/device:CPU:0"](chars/_char_embeddings/read, _arg_char_ids_0_0, chars/char_embeddings/axis)]]



During handling of the above exception, another exception occurred:



Traceback (most recent call last):

  File "train.py", line 26, in <module>

    main()

  File "train.py", line 23, in main

    model.train(train, dev)

  File "**/base_model.py", line 121, in train

    score = self.run_epoch(train, dev, epoch)

  File "**/model/ner_model.py", line 287, in run_epoch

    [self.train_op, self.loss, self.merged], feed_dict=fd)

  File "**/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 929, in run

    run_metadata_ptr)

  File "**/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1152, in _run

    feed_dict_tensor, options, run_metadata)

  File "/***/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run

    run_metadata)

  File "**/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call

    raise type(e)(node_def, op, message)

tensorflow.python.framework.errors_impl.InvalidArgumentError: indices[1,21,0] = 136 is not in [0, 136)

         [[node chars/char_embeddings (defined at **/model/ner_model.py:128)  = GatherV2[Taxis=DT_INT32, Tindices=DT_INT32, Tparams=DT_FLOAT, _class=["loc:@train_step/Adam/update_chars/_char_embeddings/AssignSub"], _device="/job:localhost/replica:0/task:0/device:CPU:0"](chars/_char_embeddings/read, _arg_char_ids_0_0, chars/char_embeddings/axis)]]

Hardware infos:

x86_64 x86_64 x86_64 GNU/Linux

CPU MHz:               1200.671

CPU max MHz:           3500.0000

Memory                  503 GB