Category Archives: nkululeko

Nkululeko

This is the entry post for Nkululeko: a framework to do machine learning experiments on audio data based on configuration files.

Here's an overview on the tutorials:

How to import features from outside the Nkululeko software

Since version 0.29.1 there is the possibilty to directly import acoustic features into the Nkululeko framework.

You can specify a file to be imported in the FEATS section:

[FEATS]
type = ['import']
import_file = /home/.../my_features.csv

Of course the features still can be combined with other feature sets and will be assigned to training and test splits accordingly.

The feature file must be in CSV format (comma separated values) in audformat with segmented index.
Here is an example:

file,start,end,voice segments,HNR Mean (dB),F1 Mean (Hz)
/home/.../a42_1.wav,0 days,0 days 00:00:07.815875,4.13,45,7.48,

Nkululeko: How to evaluate a test set with a given best model

Since version 0.27.0, Nululeko has a concept for a test set, despite train and dev set.

Let's recap the concept of train/dev/test splits:

  • train is used to train a supervised model
  • dev is a set to evaluate this model, i.e. know when it is a good model (that doesn't overfit)
  • test is a set to be used ONLY once: for the real use of the model. If you would use the test as a dev set, you can't be sure if you're not overfitting again (because you used the dev set to adjust the meta parameters of your model).

So, in order to evaluate a third dataset ( beneath train and dev) you set a label_data entry in the configuration [DATA] section like so:

[DATA]
...
label_data = emovo
label_result = my_label_result.csv

and then run the experiment.

Nkululeko FAQ

This post is the start of a troubleshooting /FAQ list

The number of speakers in my test/train splits seems wrong

  • Did you check if perhaps the speakers from several datasets are the same? If so, you can add a dataname.rename_speakers = True to the configuration. This will prepend the dataset name to the speakers

How to combine feature sets with Nkululeko

If you want to use combine several acoustic parameter (feature) sets with nkululeko, you might state

[FEATS]
type = ['mld', 'praat']
features = ['JitterPCA', 'meanF0Hz', 'hld_sylRate']

This would combine the

  • hld_sylRate feature from MLD
  • JitterPCA feature from Feinberg's Praat features and
  • meansF0Hz feature from Feinberg's Praat features

Of course you could omit the features entry and simply use all of them.

It's interesting to see how many emotions from Berlin Emodb can still be recognized with only these three parameters:

How to use selected features from Praat with Nkululeko

If you want to use acoustic parameters extracted by the wonderful Praat software with nkululeko, you state

[FEATS]
type=praat

in the feature section of your config file.
If you like to use only some features of all the ones that are extracted by David R. Feinberg's Praat scripts, you can look at the output and select some of them in the FEAT section, e.g.

[FEATS]
type = praat
features = ['JitterPCA', 'meanF0Hz']

it is interesting to see, how many emotions of Berlin EmoDB still get recognized with only mean F0 and Jitter as features:

image

What kind of features are there, you might ask yoursel?
Here's a list:
'duration', 'meanF0Hz', 'stdevF0Hz', 'HNR', 'localJitter',
'localabsoluteJitter', 'rapJitter', 'ppq5Jitter', 'ddpJitter',
'localShimmer', 'localdbShimmer', 'apq3Shimmer', 'apq5Shimmer',
'apq11Shimmer', 'ddaShimmer', 'f1_mean', 'f2_mean', 'f3_mean',
'f4_mean', 'f1_median', 'f2_median', 'f3_median', 'f4_median',
'JitterPCA', 'ShimmerPCA', 'pF', 'fdisp', 'avgFormant', 'mff',
'fitch_vtl', 'delta_f', 'vtl_delta_f''

How to test a trained model on a new test set with Nkululeko

Sometimes you might want to test your already trained model(s) on a new dataset, e.g. because the training took a lot of resources.
If you stored your models during the training this is possible.

[DATA]
databases = ['emodb']
....
[MODEL]
save = True

In a new config file for your experiment that uses a dufferent test set, you set

[DATA]
databases = ['emodb', 'polish']
trains = ['emodb']
tests = ['polish']
strategy = cross_data....
[MODEL]
only_test = True

In the example above, emodb has been used as the training database, and polish in a second experiment later as a test database.

How to compare several MLP layer layouts with each other

Some days ago I showed how you can run several experiments in one go.
Obviously this can be used to compare several ANN layer architectures as an alternative to the approach discussed in this (much earlier) post

There is an example configuration shipped with Nkululeko, and you simply can specify your layer specifications per experiment like this:

classifiers = [
    {'--model': 'mlp',
    '--layers': '\"{\'l1\':16,\'l2\':4}\"'},
    {'--model': 'mlp',
    '--layers': '\"{\'l1\':64,\'l2\':16}\"'},
    {'--model': 'mlp',
    '--layers': '\"{\'l1\':128,\'l2\':32}\"',
    '--learning_rate': '.0001',
    '--drop': '.3',},
    {'--model': 'xgb',
    '--epochs':1},
    {'--model': 'svm',
    '--epochs':1},
]

i.e in this example three MLP classifiers are specified with architectures:

  • (hidden) layer 1 with 16 neurons, and (hidden) layer 2 with 4 neurons
  • one layer with 64 and one with 16 neurons
  • and a third one with
    • one layer with 128 and a second one with 32 neurons,
    • learning rate of .0001 and
    • dropout probability of 30%

and, for comparison:

  • a XGB classifier
  • and a SVM classifier

both only need to be trained one epoch because there are no weights to be adapted.
The MLP classifiers are trained with the epoch number that is specified in the sceleton config file

How to run multiple experiments in one go with Nkululeko

Sometimes you will want to run several experiments without the need to manually start them one after the other, e.g. if you want to run them over night.
This post shows you one way how to do this.
The necessary Python files are part of the Nkululeko distribution.

You need three files:

The value parser

First i created a Python file that accepts nkululeko ini file values as targets, called parse_nkulu.py:

# imports
import sys
sys.path.append("../src")
import constants
import numpy as np
import experiment as exp
import configparser
from util import Util
import argparse
import os.path

def main():

# use the argparse package to parse arguments:
    parser = argparse.ArgumentParser(description='Call the nkululeko framework.')
    parser.add_argument('--data', help='The databases', nargs='*', \
        action='append')
    parser.add_argument('--label', nargs='*', help='The labels for the target', \
        action='append')
    parser.add_argument('--tuning_params', nargs='*', help='parameters to be tuned', \
        action='append')
    parser.add_argument('--model', default='xgb', help='The model type', required=True)
    parser.add_argument('--feat', default='os', help='The model type')
    parser.add_argument('--set', help='The opensmile set')
    parser.add_argument('--with_os', help='To add os features')
    parser.add_argument('--target', help='The target designation')

    args = parser.parse_args()

# Use a prepared config file with values that are stable across experiments:
    config_file = './exp.ini'
    util = Util()
    # test if config is there
    if not os.path.isfile(config_file):
        util.error(f'no such file {config_file}')

    config = configparser.ConfigParser()
    config.read(config_file)

# fill the config file
    if args.data is not None:
        databases = []
        for t in args.data:
            databases.append(t[0])
        print(f'got databases: {databases}')
        config['DATA']['databases'] = str(databases)
    if args.label is not None:
        labels = []
        for l in args.label:
            labels.append(l[0])
        print(f'got labels: {labels}')
        config['DATA']['labels'] = str(labels)
    if args.tuning_params is not None:
        tuning_params = []
        for tp in args.tuning_params:
            tuning_params.append(tp[0])
        config['MODEL']['tuning_params'] = str(tuning_params)
    if args.target is not None:
        config['DATA']['target'] = args.target
    if args.model is not None:
        config['MODEL']['type'] = args.model
    if args.feat is not None:
        config['FEATS']['type'] = args.feat
    if args.with_os is not None:
        config['FEATS']['with_os'] = args.with_os
    if args.set is not None:
        config['FEATS']['set'] = args.set
    name = config['EXP']['name']
    util = Util()
    util.debug(f'running {name}, Nkululeko version {constants.VERSION}')

# Now run the experiment
    # init the experiment
    expr = exp.Experiment(config)
    # load the data
    expr.load_datasets()
    # split into train and test
    expr.fill_train_and_tests()
    # extract features
    expr.extract_feats()
    # initialize a run manager
    expr.init_runmanager()
    # run the experiment
    reports = expr.run()
    result = reports[-1].result.test
    # report result
    util.debug(f'result for {expr.get_name()} is {result}')

if __name__ == "__main__":
    main()

The configuration file

A Nkululeko config file with the constant values for all experiments (to be adapted to your needs and pathes)

[EXP]
root = ./
name = exp
runs = 1
epochs = 1
[DATA]
root_folders = ../data_roots.ini
databases = ['mydata']
target = mytarget
labels = ['label1', 'label2']
[FEATS]
wav2vec.model = xxx/wav2vec2-large-robust-ft-swbd-300h
xbow.model = xxx/openXBOW/
trill.model = xxx/trill_model
mld.model = xxx/mld/src
scale = standard
[MODEL]
C_val = .001
loso = True

The script to specify and run all experiments

Lastly, you need a script to start and specify the experiments, here's an example that combines tweo classifiers and eight feature sets:

import os

classifiers = [
    {'--model': 'xgb'},
    {'--model': 'svm'},
]

features = [
    {'--feat': 'os'},
    {'--feat': 'os', 
    '--set': 'ComParE_2016',
    },
    {'--feat': 'mld'},
    {'--feat': 'mld',
    '--with_os': 'True',
    },
    {'--feat': 'xbow'},
    {'--feat': 'xbow',
    '--with_os': 'True',
    },
    {'--feat': 'trill'},
    {'--feat': 'wav2vec'},
]

for c in classifiers:
    for f in features:
        cmd = f'python parse_nkulu.py '
        for item in c:
            cmd += f'{item} {c[item]} '
        for item in f:
            cmd += f'{item} {f[item]} '
        print(cmd)
        os.system(cmd)