Nkululeko: How to evaluate a test set with a given best model

Since version 0.27.0, Nululeko has a concept for a test set, despite train and dev set.

Let's recap the concept of train/dev/test splits:

  • train is used to train a supervised model
  • dev is a set to evaluate this model, i.e. know when it is a good model (that doesn't overfit)
  • test is a set to be used ONLY once: for the real use of the model. If you would use the test as a dev set, you can't be sure if you're not overfitting again (because you used the dev set to adjust the meta parameters of your model).

So, in order to evaluate a third dataset ( beneath train and dev) you set a label_data entry in the configuration [DATA] section like so:

label_data = emovo
label_result = my_label_result.csv

and then run the experiment.

Leave a Reply

Your email address will not be published.