How to do cross validation with Nkululeko

Only for linear classifiers like XGB, SVM, SGR and SVR you have the possibility to disregard training and development splits and do a cross validation, i.e. validate one data set in a circular manner against itself.

The basic idea is that you take part of the data and evaluate against the rest, and in the next round take another part and so forth, until all data has been evaluated. Because the speaker identity is so strong in speech, this is done usually in a speaker exclusive manner, known under the term "leave one speaker out " (LOSO).

If you have too many speakers and/or each speaker really only one sample, you might want to split your speakers into groups and do a "leave one speaker group out" strategy (LOGO).

A related approach is known under the name k fold cross validation, where k usually equals 10.
When you only have one sample per speaker, this might make more sense.
So, how would you do that with Nkululeko?
First, you would define a training and development split for your data anyway, because Nkululeko is expecting it if there is only one database. You might set that to random, it's not used anyway:

mydata.split_strategy = random 

Then in your config file, you specify in the MODEL section either:

logo = 10 

to assign 10 groups to your speakers and then evaluate each group against all others.
If you want to do a leave-one-speaker_out experiment (LOSO), simply assign the number for logo the number of your speakers.

If there already is a fold column in your data, this will be used, otherwise Nkululeko will randomly assign folds to speakers.

Or you do

k_fold_cross = 5 

for instance to disregard speaker information and simply evaluate 5 times a fifth of the data against the rest.
We use stratified sets, i.e. the algorithm tries to balance the class data within each set.

Leave a Reply

Your email address will not be published. Required fields are marked *