With nkululeko since version 0.98.0 there is a new module that allows for the run of several classifier / feature combinations in one go, it's called flags.

Here is an example how to run this on a Polish emotional database:

[EXP]
root = /tmp/results/
name = exp_polish_flags1

[DATA]
databases = ['train', 'dev', 'test']
train = ./data/polish/polish_train.csv
train.type = csv
train.absolute_path = False
train.split_strategy = train
; train.audio_path = ./POLISH
dev = ./data/polish/polish_dev.csv
dev.type = csv
dev.absolute_path = False
dev.split_strategy = train
; dev.audio_path = ./POLISH
test = ./data/polish/polish_test.csv
test.type = csv
test.absolute_path = False
test.split_strategy = test
target = emotion

[FLAGS]
models = ['xgb', 'svm']
features = ['praat', 'os']   
balancing = ['none', 'ros', 'smote']  
scale = ['none', 'standard', 'robust', 'minmax']

You run this with the python command

python -m nkululeko.flags --config examples/exp_polish_flags.ini

and then would get the output

...

=== BEST CONFIGURATION ===
Best Result: 0.4666666666666666
Best Parameters:
  models: xgb
  features: praat
  balancing: none
  scale: none

To use these parameters, set in your config file:
[MODEL]
type = xgb
[FEATS]
type = ['praat']
balancing = none
scale = none

Flags experiments time: 65.32 seconds (1.09 minutes)
DONE

Sometimes you might want to check how a model performs when the predictions of a single speaker are combined. In nkululeko there are two functions supported: mode (selecting the majority value) and mean (taking the arithmetic mean).

Simply specify in the PLOT section:

[PLOT]
combine_per_speaker = mean # or mode

Only for linear classifiers like XGB, SVM, SGR and SVR you have the possibility to disregard training and development splits and do a cross validation, i.e. validate one data set in a circular manner against itself.

The basic idea is that you take part of the data and evaluate against the rest, and in the next round take another part and so forth, until all data has been evaluated. Because the speaker identity is so strong in speech, this is done usually in a speaker exclusive manner, known under the term "leave one speaker out " (LOSO).

If you have too many speakers and/or each speaker really only one sample, you might want to split your speakers into groups and do a "leave one speaker group out" strategy (LOGO).

A related approach is known under the name k fold cross validation, where k usually equals 10.
When you only have one sample per speaker, this might make more sense.
So, how would you do that with Nkululeko?
First, you would define a training and development split for your data anyway, because Nkululeko is expecting it if there is only one database. You might set that to random, it's not used anyway:

[DATA]
mydata.split_strategy = random 

Then in your config file, you specify in the MODEL section either:

[MODEL] 
logo = 10 

to assign 10 groups to your speakers and then evaluate each group against all others.
If you want to do a leave-one-speaker_out experiment (LOSO), simply assign the number for logo the number of your speakers.

If there already is a fold column in your data, this will be used, otherwise Nkululeko will randomly assign folds to speakers.

Or you do

[MODEL] 
k_fold_cross = 5 

for instance to disregard speaker information and simply evaluate 5 times a fifth of the data against the rest.
We use stratified sets, i.e. the algorithm tries to balance the class data within each set.

Often you simply start an experiment with some audio data that you got from somewhere in no special format. Often the labels are encoded in the filenames.
If so, this Python script can help to convert the audio to a Nkululeko readable format and generate a CSV (comma separated values) file.

import os
from audeer import list_file_names
from os.path import basename

# folder with the original audio files (in wav format)
root = './orig_wav/'
# output folder, empty at the beginning
out_dir = './audio/'
# name of the output file list
out_file = 'data.csv'

# get a list of wav files
list = list_file_names(root, filetype = 'wav', basenames=True, recursive=True)
# write the list header (change to your data)
with open(out_file, 'a') as the_file:
    the_file.write('file,type\n')
# for each file
for file in list:
    # get the file name without path
    fn = basename(file)
    # convert to 16kHz sampling rate and mono channel 
    os.system(f'sox {root+file} -r 16000 -c 1 {out_dir+fn}')
    # extract the annotation label from the file name (change this to your needs)
    label = fn[0]
    # lastly: add file to list 
    with open(out_file, 'a') as the_file:
        the_file.write(f'{out_dir+fn},{label}\n')

The resulting data list can then be read by Nkululeko in the config file (using randomly 30 % of the data as development set):

[DATA]
my_data = /some_path/data.csv
my_data.type = csv
my_data.split_strategy = random
my_data.testsplit = 30