With nkululeko since version 1.7, there is the new predict module, which can be used for any us of already trained models like demoing them, using them on some test data or participating in a challenge.

nkululeko.predict

nkululeko.predict is the unified prediction module of Nkululeko. It replaces
the previous nkululeko.demo, nkululeko.feature_demo and nkululeko.testing
modules and bundles all of their functionality behind a single command-line
interface.

You can use it to predict labels for:

• one or more individual audio files (--file)
• every audio file inside a folder (--folder)
• the audio paths listed in a CSV (--list, original columns are preserved)
• a live microphone recording (--mic)
• the dataframe defined by an experiment config — pass --config without
  any of the input flags above and the module loads the databases declared
  in [DATA] (subset via EXP.sample_selection, default all)

…using one of two prediction sources:

• a feature extractor or autopredict target such as age, gender,
  emotion, mos, snr (--type feats, the default)
• the best model from a previously trained experiment
  (--type model, requires --config)

Command-line interface

python -m nkululeko.predict
    [--file AUDIO [AUDIO ...] | --folder FOLDER | --list CSV | --mic]
    [--model MODEL] [--type {feats,model}]
    [--config CONFIG.ini] [--outfile OUTFILE]
    [--language LANG] [--no_playback]

Argument	Description
--file AUDIO [AUDIO ...]	One or more audio files. A single space-separated string also works (e.g. --file "a.wav b.wav"). Writes a per-file <name>_result.txt next to each input and prints results to stdout.
--folder FOLDER	Folder to scan recursively for audio (wav, mp3, flac, ogg, m4a, au, aac). Writes a single CSV to --outfile.
--list CSV	CSV with audio paths. Existing columns and the audformat index are preserved; prediction columns are appended. Writes a single CSV to --outfile.
--mic	Record 5 seconds from the microphone in a loop and print predictions to stdout.
--model MODEL	Either an autopredict target name (age, gender, emotion, mos, snr, pesq, sdr, stoi, arousal, valence, dominance, speaker, text, textclassification, translation) or a feature-extractor name (wav2vec2-..., opensmile, audmodel, emotion2vec-..., praat, clap, spkrec, trill, agender, whisper-..., ast, hubert-..., wavlm-..., squim, mos, snr). When --type model, --model is ignored — the trained model from the experiment is used.
--type {feats,model}	feats (default): use --model as autopredict target or feature extractor. model: load the best model from the experiment defined by --config.
--config CONFIG.ini	Optional INI file. Required for --type model. With --type feats it may supply FEATS.type so that --model can be omitted. When passed alone (without --file/--folder/--list/--mic), the dataframe defined by the experiment's [DATA] section is used; EXP.sample_selection (default all) selects train / test / all.
--outfile OUTFILE	Output CSV path for --list and --folder. Default: ./prediction_result.csv.
--language LANG	ISO 639-1 code (en, de, pl, …) for the text and translation autopredict targets. For --model text it sets the Whisper source language (overrides EXP.language). For --model translation it sets the Google Translate target language (overrides PREDICT.target_language).
--no_playback	In --mic mode, suppress the playback of the recording before prediction.

The four input arguments (--file, --folder, --list, --mic) are mutually
exclusive.

Examples

Predict emotion for a couple of audio files

python -m nkululeko.predict --file test.mp3 test2.wav --model emotion

This writes test_result.txt and test2_result.txt next to each input and
also prints the predictions to stdout. With --model emotion, the
nkululeko.autopredict.ap_emotion predictor is used.

Predict SNR for every audio file in a folder

python -m nkululeko.predict --folder ./recordings --model snr --outfile snr.csv

The output CSV contains the audformat segmented index plus the new
snr_pred column.

Add prediction columns to an existing CSV, keeping all original columns

python -m nkululeko.predict \
    --list testdata.csv \
    --model mos \
    --outfile testdata_with_mos.csv

If testdata.csv is a valid audformat CSV (segmented or filewise index), the
index is preserved. Otherwise the first column is interpreted as the audio
path. Any further columns are passed through to the output.

Use the best model of a trained experiment

python -m nkululeko.predict \
    --list testdata.csv \
    --config config.ini \
    --type model

This loads the experiment specified in config.ini (which must have been
trained with MODEL.save = True) and runs its best model on each file in the
list. For classification, the output contains one column per class label with
the probability/score and a predicted column with the top-1 label. For
regression, a single predicted column is written.

Loop over microphone input using the FEATS section of a config

python -m nkululeko.predict --mic --config config.ini

Press Enter to record 5 seconds, q + Enter to quit.

Transcribe German audio with Whisper

python -m nkululeko.predict --file lecture.mp3 --model text --language de

--language de overrides EXP.language for the Whisper source language.

Predict over the dataframe defined by a config

When you only pass --config, the module loads the databases declared in the
config's [DATA] section and runs over the selection from
EXP.sample_selection (default all):

python -m nkululeko.predict \
    --config experiments/emodb/exp.ini \
    --model snr \
    --outfile emodb_snr.csv

Set EXP.sample_selection = train (or test) in the INI to restrict the
run to that subset.

Translate transcriptions to French

python -m nkululeko.predict \
    --list transcribed.csv \
    --model translation \
    --language fr \
    --outfile translated.csv

--language fr overrides PREDICT.target_language for Google Translate.

Autopredict targets

When --model NAME matches one of the autopredict targets below, the matching
nkululeko.autopredict.* predictor is used. The added column name follows the
<target>_pred convention.

Target	Predictor module	Added column
speaker	ap_sid.SIDPredictor	speaker_pred
gender	ap_gender.GenderPredictor (audEERING agender)	gender_pred
age	ap_age.AgePredictor (audEERING agender)	age_pred
emotion	ap_emotion.EmotionPredictor (emotion2vec)	emotion_pred
arousal	ap_arousal.ArousalPredictor (audEERING dim)	arousal_pred
valence	ap_valence.ValencePredictor (audEERING dim)	valence_pred
dominance	ap_dominance.DominancePredictor (audEERING dim)	dominance_pred
mos	ap_mos.MOSPredictor	mos_pred
pesq	ap_pesq.PESQPredictor (SQUIM)	pesq_pred
sdr	ap_sdr.SDRPredictor (SQUIM)	sdr_pred
stoi	ap_stoi.STOIPredictor (SQUIM)	stoi_pred
snr	ap_snr.SNRPredictor	snr_pred
text	ap_text.TextPredictor (whisper transcription)	text
textclassification	ap_textclassifier.TextClassificationPredictor	classification_winner + one column per candidate label
translation	ap_translate.TextTranslator	column named after PREDICT.target_language (default: en)

Feature extractors

If --model does not match an autopredict target, it is interpreted as a
feature-extractor name. The output columns are feat_0, feat_1, …. Examples:

python -m nkululeko.predict --file test.wav --model praat
python -m nkululeko.predict --folder ./voices --model wav2vec2-large-robust-ft-swbd-300h --outfile feats.csv
python -m nkululeko.predict --list audio.csv --model audmodel --outfile feats.csv --config has_audmodel_id.ini

Recognized prefixes / names: wav2vec2*, hubert*, wavlm*, whisper*,
ast*, emotion2vec*, opensmile/gemaps/compare, clap*, spkrec* /
xvect* / ecapa*, trill*, praat*, audmodel*, agender*, squim* /
pesq* / sdr*, mos*, snr*.

Note on overlapping names. mos and snr are both autopredict targets
and feature extractors. They resolve to the autopredict path. If you need
the raw feature extractor for these, use the lower-level extractor classes
directly.

Output formats

Mode	Where the result is written
--file	<name>_result.txt per input file (one key: value per line), plus stdout.
--folder	Single CSV at --outfile with the audformat segmented index of the discovered files and the prediction columns.
--list	Single CSV at --outfile with the original columns of the input CSV plus the prediction columns. The audformat index is preserved when the input is a valid audformat CSV.
--mic	stdout only.

With nkululeko you can extract any features from audio, provided a model.

Since version 1.7.0, this is done with the predict module

With nkululeko since version 0.94 (aleatoric) uncertainty, i.e. the confidence of the model, is explicitly visualized. You simply find a plot in the image folder after running an experiment, like so:

You see the distribution for true vs. false predictions wrt. uncertainty, i.e. in this case this worked out quite well (because less uncertain prediction are usually correct).

The approach is described in our paper Uncertainty-Based Ensemble Learning For Speech Classification

You can use this to tweak your results if you specify an uncertainty-threshold, i.e. you refuse to predict sample that are above some threshold:

[PLOT]
uncertainty_thresshold = .4

You will than get additionally a confusion plot that only takes the selected samples into account.

This might feel like cheating, but especially in critical use cases it might be better to deliver not prediction than a wrong one.

As described in this previous post, features scaling can be quite important in machine learning.

With nkululeko since version 0.97 you have a multitude if scaling methods at hand.

You simply state in the config:

[FEATS]
scale = xxx

For xxx you specify the scaling methods are

• standard: z-transformation (mean of 0 and std of 1) based on the training set
  • robust: robust scaler
• speaker: like standard but based on individual speaker sets (also for the test)
• bins: convert feature values into 0, .5 and 1 (for low, mid and high)
• minmax: rescales the data set such that all feature values are in the range [0, 1]
• maxabs: similar to MinMaxScaler except that the values are mapped across several ranges depending on whether negative OR positive values are present
• normalizer: scales each sample (row) individually to have unit norm (e.g., L2 norm)
• powertransformer: applies a power transformation to each feature to make the data more Gaussian-like in order to stabilize variance and minimize skewness
• quantiletransformer: applies a non-linear transformation such that the probability density function of each feature will be mapped to a uniform or Gaussian distribution (range [0, 1])

With nkululeko since version 0.96 you there are linguistic feature extractors, i.e. using the text of the spoḱen words as input.

Of course you can combine them with acoustic features and use any fitting model architecture with it.

[EXP]
# optional: language for linguistics
language = de

[DATA]
data = ../mydata
# the linguistic feature extractors require a column named "text"
# example, perhaps not needed!
data.col_names = {"transcription":"text"}

[FEAT]
# combine linguistic bert features with acoustic open smile features
type = ['bert', 'os']

[MODEL]
type = xgb

With nkululeko since version 0.95.9 you can use google translate to translate your data automatically.

Simply set the language (default is en) in the PREDICT section and a prediction target translation like this:

[EXP]
# optional
language = de 
[PREDICT]
targets = ['translation']
# optional
target_language = en 

and then run the module:

python -m nkululeko.predict --config my_conf.ini.

With nkululeko since version 0.95.0 you can use whisper from openAI to transcribe you data automatically.

Simply set the language (default is en) in the header and a prediction target text like this:

[EXP]
language = en
[PREDICT]
targets = ['text']

and then run the module:

python -m nkululeko.predict --config my_conf.ini.

Supervised machine learning operates as follows: during the training phase, a learning algorithm is adapted to a training dataset, producing a trained model, which is then used to make predictions on a test set during the inference phase.

One potential issue with this approach is that, for sufficiently complex models, they may simply memorise all items in the training set rather than learning a generalised distinction based on an underlying process, such as emotional expression or speaker age. This means that while the model performs well on the training data, it fails to generalise to new data—a phenomenon known as overfitting.

To mitigate this, the model's hyperparameters are optimised using a held-out evaluation set that is not used during training. One particularly important hyperparameter is the number of epochs—that is, the number of times the entire training set is processed. Typically, to prevent overfitting, training is halted when performance on the evaluation set begins to decline, a technique known as early stopping. The model that performs best on the evaluation data is then selected.

However, this approach introduces a new problem: the model may (and most likely has) now overfitted to the evaluation data. This is why a third dataset is necessary for final testing—one that has not been used at any stage of model development.

The evaluation set is often referred to as the dev set (short for development set). Consequently, Nkululeko now provides support for three distinct data splits: train, dev, and test.

Here is an example how you would do this with emoDB (the distribution has no predefined splits for train, dev and test)

[EXP]
root = ./experiments/emodb_3split/
name = results
epochs = 100
traindevtest = True
[DATA]
databases = ['emodb']
emodb = ./data/emodb/emodb
emodb.split_strategy = speaker_split
labels = ["neutral", "sadness", "happiness"]
target = emotion
[FEATS]
type = ['os']
[MODEL]
type = mlp
layers = {'l1':100, 'l2':16}
patience = 10
[PLOT]
best_model = True
epoch_progression = True

You trigger the handling of three splits with

traindevtest = True

and the rest happens in this case automatically, the results are then shown for

best model based on development sets:

best model for dev set, but evaluated on test set:

and the last model, evaluated on the dev set:

In this case, you see that the 62nd epoch performed like the 52nd for the dev set. But, this best model evaluated on the test set, drops by more than 20 % average recall, which is a more stable value for the general performance of this model (this is only a toy example with 4 speakers in the training, and 2 each for dev and test set.)

With nkululeko since version 0.93.0 the pyannote segmentation package is interfaced (as an alternative to silero)

There are two modules that you can use for this:

• SEGMENT
• PREDICT

The (huge) difference is, that the SEGMENT module looks at each file in the input data and looks for speakers per file (can be only one large file), while the PREDICT module concatenates all input data and looks for different speakers in the whole database.

In any case best run it on a GPU, as CPU will be very slow (and there is no progress bar).

Segment module

If you specify the method in [SEGMENT] section and the hf_token (needed for the pyannote model) in the [MODEL] section

[SEGMENT]
method = pyannote
segment_target = _segmented
sample_selection = all
[MODEL]
hf_token = <my hugging face token>

your resulting segmentations will have predicted speaker id attachched.. Be aware that this is really slow on CPU, so best run on GPU and declare so in the [MODEL] section:

[MODEL]
hf_token = <my hugging face token>
device=gpu # or cuda:0

As a result a new plot would appear in the image folder: the distribution of speakers that were found, e.g. like this:

Predict module

Simply select speaker as the prediction target:

[PREDICT]
targets = ["speaker"]

Generally, the PREDICT module is described here