Within nkululeko, since version 0.53.0, you can perform automatic data checks, which means that some of your data might be filtered out if it doesn't fulfill certain requirements.
Currently two checks are implemented:
[DATA]
# check the filesize of all samples in train and test splits, in bytes
check_size = 1000
# check if the files contain speech with voice activity detection (VAD)
check_vad = TrueVAD is using silero VAD
If you just want to see how your data distributes on the target with nkululeko, you can do a value_counts plot with the explore module
In your config, you would specify like this:
[EXPL]
# all samples, or only test or train split?
sample_selection = all
# activate the plot
value_counts = [['age'], ['gender'], ['duration'], ['duration', 'age']] and then, run this with the explore module:
python -m nkululeko.explore --config myconfig.iniThe results, for a data set with target=depression, looks similar to this for all samples:
and this for the speakers (if there is a speaker annotation)
If you prefer a kernel density estimation over a histogram, you can select this with
[EXPL]
dist_type = kdewhich would result for duration to:
Nkululeko distinguishes between categorical and continuous properties, this would be the output for gender
You can show the distribution of two sample properties at once, by using a scatter plot:
In addition, this module will automatically plot the distribution of samples per speaker, per gender (if annotated):
It can be very interesting to reduce the dimensionality of your acoustic or learned features to two or three dimensions and then color the single samples features with the label.
Nkululeko supports three different ways to reduce the dimensionality:
To do this, you simply state your data and features as usual. The approaches you want to use can be set in the scatter field of the EXPL section:
[EXPL]
scatter = ['umap', 'tsne', 'pca'](of course you don't have to use all) and then call the explore interface
python -m nkululeko.explore --config myconfig.iniYou can do this for all columns in your data, not only the target value.
If you want a scatter plot for a different target, state it like this (example):
[EXPL]
scatter = ['pca']
scatter.target = ['gender', 'age', 'likability']The images appear in the image folder of your experiment and might look like this (all from the same data):
PCA
T-SNE
UMAP
There is a multitude of ways to model emotions, and some of them are collected in the EmotionML vacabularies.
Really popular with engineers and non-psychologists are two approaches:
The emotion cube maps the emotional categories to a three dimensional space:
To do data augmentation with Nkululeko, you can use the augment interface.
In the DATA section of your configuration file, you specify the method and name of the output list of file
[DATA]
# select the samples to augment: either train, test, or all
augment = train
# file name to store the augmented data (can then be added to training)
augment_result = augment.csv
or, if you want to use random-splicing:
[DATA]
# random_splice: select the samples to be random spliced: either train, test, or all
random_splice = train
# random_splice_result: file name to store the random spliced data (can then be added to training)
random_splice_result = random_spliced.csvand then call the interface:
python -m nkululeko.augment --config myconfig.iniCurrently, apart from random-splicing, Nkululeko simply uses the augmentations that are specified as a demo in the audiomentations documentation, i.e.:
self.audioment = Compose([
AddGaussianNoise(min_amplitude=0.001, max_amplitude=0.015, p=0.5),
TimeStretch(min_rate=0.8, max_rate=1.25, p=0.5),
PitchShift(min_semitones=-4, max_semitones=4, p=0.5),
Shift(min_fraction=-0.5, max_fraction=0.5, p=0.5),
])These manipulations are applied randomly to your training set.
You should find the augmented files in the storage folder of the result folder of your experiment and could listen to them there.
Once you augmentations have been processed, you can add them to the training in a new experiment:
[DATA]
databases = ['original data', 'augment']
augment = my_augmentations.csv
augment.type = csv
augment.absolute_path = True
augment.split_strategy = trainSupervised vs. unsupervised
means the distinction whether your training data is annotated (or labeled) with respect to your task. An example: If you want to build a machine learner for human age estimation based on speech, you might give an algorithm a lot of examples of human speech annotated with the age of the person. This would be your training data and the approach would be supervised (by the age annotations). With unsupervised learning, you would give an algorithm simply a lot of human speech data and might ask it to cluster the data, based on differences. And might hope that the resulting clusters coincide with age.
Nkululeko exercise
-> Nkululeko: install the Berlin Emodb
This database contains examples of labels:
Since version 0.40, Nkululeko can now show the best performing X acoustic features according to some model.
There is a new section call EXPL (short for exploration), and you could state
[EXPL]
model = tree
sample_num = 15in your config file, and then run the exploration module like this:
python -m nkululeko.explore --config my_config.iniThe resulting list will then appear in the result folder and a barplot image in the image folder.
Afterwards you could inspect single features as described here
As shown in this post, with Nkululeko you can select only specific features from your features sets by specifying them in the [FEAT] section:
[FEATS]
features = ['JitterPCA', 'meanF0Hz', 'hld_sylRate']What you can also do, is plotting them per category (only for classification), by specifying in the PLOT section if you would like that for all samples or only test or train samples:
[EXPL]
# turn it on
feature_distributions = True
# use only training samples
sample_selection = train
# only plot the 5 most important features
max_feats = 5 You would have to call nkululeko with the explore interface:
python -m nkululeko.explore --config <myConfig.ini>The image file is in the image folder and should look similar to this:
There are three ways to predict a number of samples:
If you want to save the predictions of an experiment for later use, you can do so by stating in the EXP section
[EXP]
save_test = ./my_saved_test_predictions.csvThe output format is CSV, comma seperated values.
Alternatively, you can test an existing database against the best model you trained before, by stating the databases as tests in the DATA section:
[DATA]
tests = ['my_testdb']
my_testdb = /mypath/my_testdb
...and then calling Nkululeko's test module
python -m nkululeko.test --config mycoonfg.ini --outfile myresults.csvRun the demo module simply for a set of files:
python -m nkululeko.demo --config mycoonfg.ini --list my_filelist.txtwrite your
like in the nkululeko project
When you have a new version, you can test it with
pip install .and, when happy,
python -m build will generate the dist folder
twine upload dist/* will upload to pypi server
If you want to develop your package, you can run
python setup.py develop