Nkululeko: how to visualize your data distribution

If you just want to see how your data distributes on the target and speaker gender, you can do a value_counts plot with the explore module

In your config, you would specify like this:

[EXPL]
# all samples, or only test or train split?
sample_selection = all 
# activate the plot
value_counts = True

and then, run this with the explore module:

python -m nkululeko.explore --config myconfig.ini

The result looks similar to this:

Nkululeko: visualize clusters of your acoustic features

It can be very interesting to reduce the dimensionality of your acoustic or learned features to two or three dimensions and then color the single samples features with the label.

Nkululeko supports three different ways to reduce the dimensionality:

  • pca: Principal Componen Analysis
  • tsne: t-distributed stochastic neighbor embedding
    • perplexity=30, learning_rate=200
  • umap: Uniform Manifold Approximation and Projection
    • n_neighbors=10, random_state=0

To do this, you simply state your data and features as usual. The approaches you want to use can be set in the scatter field of the EXPL section:

[EXPL]
scatter = ['umap', 'tsne', 'pca']

(of course you don't have to use all) and then call the explore interface

python -m nkululeko.explore --config myconfig.ini

The images appear in the image folder of your experiment and might look like this (all from the same data):

PCA

T-SNE

UMAP

The emotion cube

There is a multitude of ways to model emotions, and some of them are collected in the EmotionML vacabularies.
Really popular with engineers and non-psychologists are two approaches:

  • discreet categories like anger, sadness, fear or joy, often associated with an intensity.
  • continuous dimensions like valence/pleasure, arousal or dominance

The emotion cube maps the emotional categories to a three dimensional space:

Nkululeko: how to augment the training set

To do data augmentation with Nkululeko, you can use the augment interface.
In the DATA section of your configuration file, you specify the name of the output list of files like so

[DATA]
augment = my_augmentations.csv

and then call the interface:

python - nkululeko.augment --config myconfig.ini

Currently, Nkululeko simply uses the augmentations that are specified as a demo in the audiomentations documentation, i.e.:

self.audioment = Compose([
    AddGaussianNoise(min_amplitude=0.001, max_amplitude=0.015, p=0.5),
    TimeStretch(min_rate=0.8, max_rate=1.25, p=0.5),
    PitchShift(min_semitones=-4, max_semitones=4, p=0.5),
    Shift(min_fraction=-0.5, max_fraction=0.5, p=0.5),
])

These manipulations are applied randomly to your training set.

You should find the augmented files in the storage folder of the result folder of your experiment and could listen to them there.

Once you augmentations have been processed, you can add them to the training in a new experiment:

[DATA]
databases = ['original data', 'augment']
augment = my_augmentations.csv
augment.type = csv
augment.absolute_path = True
augment.split_strategy = train

Supervised vs. unsupervised

Supervised vs. unsupervised

means the distinction whether your training data is annotated (or labeled) with respect to your task. An example: If you want to build a machine learner for human age estimation based on speech, you might give an algorithm a lot of examples of human speech annotated with the age of the person. This would be your training data and the approach would be supervised (by the age annotations). With unsupervised learning, you would give an algorithm simply a lot of human speech data and might ask it to cluster the data, based on differences. And might hope that the resulting clusters coincide with age.

Nkululeko exercise

-> Nkululeko: install the Berlin Emodb

This database contains examples of labels:

  • emotion and gender labels as categorical data, for classification
  • age labels as numerical data, for regression

Nkululeko: show feature importance

Since version 0.40, Nkululeko can now show the best performing X acoustic features according to some model.

There is a new section call EXPL (short for exploration), and you could state

[EXPL]
model = tree
sample_num = 15

in your config file, and then run the exploration module like this:

python -m nkululeko.explore --config my_config.ini

The resulting list will then appear in the result folder and a barplot image in the image folder.

Nkululeko: how to plot distributions of feature values

As shown in this post, with Nkululeko you can select only specific features from your features sets by specifying them in the [FEAT] section:

[FEATS]
features = ['JitterPCA', 'meanF0Hz', 'hld_sylRate']

What you can also do, is plotting them per category (only for classification), by specifying in the PLOT section if you would like that for all samples or only test or train samples:

[EXPL]
# turn it on
feature_distributions = True 
# use only training samples
sample_selection = train 
# only plot the 5 most important features 
max_feats = 5  

You would have to call nkululeko with the explore interface:

python -m nkululeko.explore --config <myConfig.ini>

The image file is in the image folder and should look similar to this:

Nkululeko: how to predict many samples

There are three ways to predict a number of samples:

  1. If you want to save the predictions of an experiment for later use, you can do so by stating in the EXP section

    [EXP]
    save_test = ./my_saved_test_predictions.csv

    The output format is CSV, comma seperated values.

  2. Alternatively, you can test an existing database against the best model you trained before, by stating the databases as tests in the DATA section:

    [DATA]
    tests = ['my_testdb']
    my_testdb = /mypath/my_testdb
    ...

    and then calling Nkululeko's test module

    python -m nkululeko.test --config mycoonfg.ini --outfile myresults.csv
  3. Run the demo module simply for a set of files:

    python -m nkululeko.demo --config mycoonfg.ini --list my_filelist.txt

How to normalize features

"Normalizing" or scaling feature values means to shift them to a common range, or distribution with same mean and standard deviation (also called z-transformation).
You would do that for several reasons:

  • Artificial neural nets can handle small numbers best, so they all should be in the range -1, 1
  • Speakers have their individual ways to speak which you are not interested in if you want to learn a general task, e.g. emotion or age. So you would speaker-normalize the values for each speaker individually. Of course this is in most applications not possible because you don't have already samples of your test speakers.
  • You might want to normalize the sexes, because woman typicall have a higher pitch. But another way out is also to use only relative values and not absolute ones.

Mind that you shouldn't use your test set for normalization as it really only should be used for test and is supposed to be unknown. That's why you should compute your normalization parameters on the training set, you can then use them to normalize/scale the test.