Sometimes you will want to run several experiments without the need to manually start them one after the other, e.g. if you want to run them over night.
This post shows you one way how to do this.
The necessary Python files are part of the Nkululeko distribution.

You need three files:

The value parser

First i created a Python file that accepts nkululeko ini file values as targets, called parse_nkulu.py:

# imports
import sys
sys.path.append("../src")
import constants
import numpy as np
import experiment as exp
import configparser
from util import Util
import argparse
import os.path

def main():

# use the argparse package to parse arguments:
    parser = argparse.ArgumentParser(description='Call the nkululeko framework.')
    parser.add_argument('--data', help='The databases', nargs='*', \
        action='append')
    parser.add_argument('--label', nargs='*', help='The labels for the target', \
        action='append')
    parser.add_argument('--tuning_params', nargs='*', help='parameters to be tuned', \
        action='append')
    parser.add_argument('--model', default='xgb', help='The model type', required=True)
    parser.add_argument('--feat', default='os', help='The model type')
    parser.add_argument('--set', help='The opensmile set')
    parser.add_argument('--with_os', help='To add os features')
    parser.add_argument('--target', help='The target designation')

    args = parser.parse_args()

# Use a prepared config file with values that are stable across experiments:
    config_file = './exp.ini'
    util = Util()
    # test if config is there
    if not os.path.isfile(config_file):
        util.error(f'no such file {config_file}')

    config = configparser.ConfigParser()
    config.read(config_file)

# fill the config file
    if args.data is not None:
        databases = []
        for t in args.data:
            databases.append(t[0])
        print(f'got databases: {databases}')
        config['DATA']['databases'] = str(databases)
    if args.label is not None:
        labels = []
        for l in args.label:
            labels.append(l[0])
        print(f'got labels: {labels}')
        config['DATA']['labels'] = str(labels)
    if args.tuning_params is not None:
        tuning_params = []
        for tp in args.tuning_params:
            tuning_params.append(tp[0])
        config['MODEL']['tuning_params'] = str(tuning_params)
    if args.target is not None:
        config['DATA']['target'] = args.target
    if args.model is not None:
        config['MODEL']['type'] = args.model
    if args.feat is not None:
        config['FEATS']['type'] = args.feat
    if args.with_os is not None:
        config['FEATS']['with_os'] = args.with_os
    if args.set is not None:
        config['FEATS']['set'] = args.set
    name = config['EXP']['name']
    util = Util()
    util.debug(f'running {name}, Nkululeko version {constants.VERSION}')

# Now run the experiment
    # init the experiment
    expr = exp.Experiment(config)
    # load the data
    expr.load_datasets()
    # split into train and test
    expr.fill_train_and_tests()
    # extract features
    expr.extract_feats()
    # initialize a run manager
    expr.init_runmanager()
    # run the experiment
    reports = expr.run()
    result = reports[-1].result.test
    # report result
    util.debug(f'result for {expr.get_name()} is {result}')

if __name__ == "__main__":
    main()

The configuration file

A Nkululeko config file with the constant values for all experiments (to be adapted to your needs and pathes)

[EXP]
root = ./
name = exp
runs = 1
epochs = 1
[DATA]
root_folders = ../data_roots.ini
databases = ['mydata']
target = mytarget
labels = ['label1', 'label2']
[FEATS]
wav2vec.model = xxx/wav2vec2-large-robust-ft-swbd-300h
xbow.model = xxx/openXBOW/
trill.model = xxx/trill_model
mld.model = xxx/mld/src
scale = standard
[MODEL]
C_val = .001
loso = True

The script to specify and run all experiments

Lastly, you need a script to start and specify the experiments, here's an example that combines tweo classifiers and eight feature sets:

import os

classifiers = [
    {'--model': 'xgb'},
    {'--model': 'svm'},
]

features = [
    {'--feat': 'os'},
    {'--feat': 'os', 
    '--set': 'ComParE_2016',
    },
    {'--feat': 'mld'},
    {'--feat': 'mld',
    '--with_os': 'True',
    },
    {'--feat': 'xbow'},
    {'--feat': 'xbow',
    '--with_os': 'True',
    },
    {'--feat': 'trill'},
    {'--feat': 'wav2vec'},
]

for c in classifiers:
    for f in features:
        cmd = f'python parse_nkulu.py '
        for item in c:
            cmd += f'{item} {c[item]} '
        for item in f:
            cmd += f'{item} {f[item]} '
        print(cmd)
        os.system(cmd)

Sometimes you might want to check how a model performs when the predictions of a single speaker are combined. In nkululeko there are two functions supported: mode (selecting the majority value) and mean (taking the arithmetic mean).

Simply specify in the PLOT section:

[PLOT]
combine_per_speaker = mean # or mode

Only for linear classifiers like XGB, SVM, SGR and SVR you have the possibility to disregard training and development splits and do a cross validation, i.e. validate one data set in a circular manner against itself.

The basic idea is that you take part of the data and evaluate against the rest, and in the next round take another part and so forth, until all data has been evaluated. Because the speaker identity is so strong in speech, this is done usually in a speaker exclusive manner, known under the term "leave one speaker out " (LOSO).

If you have too many speakers and/or each speaker really only one sample, you might want to split your speakers into groups and do a "leave one speaker group out" strategy (LOGO).

A related approach is known under the name k fold cross validation, where k usually equals 10.
When you only have one sample per speaker, this might make more sense.
So, how would you do that with Nkululeko?
First, you would define a training and development split for your data anyway, because Nkululeko is expecting it if there is only one database. You might set that to random, it's not used anyway:

[DATA]
mydata.split_strategy = random 

Then in your config file, you specify in the MODEL section either:

[MODEL] 
logo = 10 

to assign 10 groups to your speakers and then evaluate each group against all others.
If you want to do a leave-one-speaker_out experiment (LOSO), simply assign the number for logo the number of your speakers.

If there already is a fold column in your data, this will be used, otherwise Nkululeko will randomly assign folds to speakers.

Or you do

[MODEL] 
k_fold_cross = 5 

for instance to disregard speaker information and simply evaluate 5 times a fifth of the data against the rest.
We use stratified sets, i.e. the algorithm tries to balance the class data within each set.

Often you simply start an experiment with some audio data that you got from somewhere in no special format. Often the labels are encoded in the filenames.
If so, this Python script can help to convert the audio to a Nkululeko readable format and generate a CSV (comma separated values) file.

import os
from audeer import list_file_names
from os.path import basename

# folder with the original audio files (in wav format)
root = './orig_wav/'
# output folder, empty at the beginning
out_dir = './audio/'
# name of the output file list
out_file = 'data.csv'

# get a list of wav files
list = list_file_names(root, filetype = 'wav', basenames=True, recursive=True)
# write the list header (change to your data)
with open(out_file, 'a') as the_file:
    the_file.write('file,type\n')
# for each file
for file in list:
    # get the file name without path
    fn = basename(file)
    # convert to 16kHz sampling rate and mono channel 
    os.system(f'sox {root+file} -r 16000 -c 1 {out_dir+fn}')
    # extract the annotation label from the file name (change this to your needs)
    label = fn[0]
    # lastly: add file to list 
    with open(out_file, 'a') as the_file:
        the_file.write(f'{out_dir+fn},{label}\n')

The resulting data list can then be read by Nkululeko in the config file (using randomly 30 % of the data as development set):

[DATA]
my_data = /some_path/data.csv
my_data.type = csv
my_data.split_strategy = random
my_data.testsplit = 30