Nkululeko Transformer Finetuning Guide

Overview

Starting with version 0.85.0, Nkululeko enables finetuning of pretrained transformer models using Hugging Face, with optional publication to their hub.

Finetuning vs. Normal Model Training

In normal Nkululeko training, [FEATS] extracts a fixed numeric feature vector for each sample once (e.g. openSMILE, wav2vec2 embeddings), and [MODEL] (e.g. svm, xgb, mlp, cnn) fits a classifier or regressor on top of those static features — the feature extractor itself never changes. Finetuning instead treats the pretrained transformer as the model itself: its own weights are updated directly on the raw audio via backpropagation, jointly adjusting the acoustic representation and the prediction head for your specific task.

Basic Configuration

To enable finetuning, set the MODEL type and leave acoustic features empty:

[FEATS]
type = []
[MODEL]
type = finetune

The system handles acoustic processing through CNN layers in the transformer, pooling frames for utterances up to 8 seconds by default.

Model Selection

The default base model is from Facebook (wav2vec2-large-robust-ft-swbd-300h), but alternatives can be specified. Finetuning-specific settings go in their own [FINETUNE] section:

[MODEL]
type = finetune

[FINETUNE]
pretrained_model = microsoft/wavlm-base
max_duration = 10.5

Training Parameters

Configuration keys, set under [FINETUNE]:

• learning_rate: Training rate (default: 0.0001)
• batch_size: Samples per iteration (default: 8)
• device: GPU/CPU specification — a bare GPU index (e.g. 0) or cpu; autodetects if unset
• measure: Evaluation metric, regression only (default: ccc); classification always uses UAR
• loss: Loss function (default: cross for classification, 1-ccc for regression)
• max_duration: Maximum sample length in seconds (default: 8)
• push_to_hub: Boolean for publishing to Hugging Face (default: False)
• drop: Dropout applied in the classification/regression head (default: 0.1)
• balancing: Training-set balancing algorithm: ros, smote, or adasyn (default: none/disabled - false, none, or an empty value all mean disabled)
• class_weight: Weight the loss by inverse class frequency, classification only (default: False)
• freeze_layers: Number of pretrained encoder layers to freeze, counted from the input side (default: 0, meaning the whole backbone finetunes). Useful for partial finetuning on small datasets or to save GPU memory.
• num_layers: Total number of encoder layers to build the model with, truncating the pretrained architecture to a smaller one (default: empty/unset, meaning the pretrained model's full depth). Unlike freeze_layers, this actually removes layers rather than just freezing their weights — fewer parameters, faster inference, smaller checkpoint.

Nkululeko validates 0 <= freeze_layers < num_layers <= <pretrained model's layer count> (using the pretrained depth wherever num_layers is left unset), and fails fast with a clear error otherwise — e.g. a num_layers larger than the checkpoint provides, or a freeze_layers that would freeze the entire resulting backbone and leave nothing trainable.

[FINETUNE]
pretrained_model = microsoft/wavlm-large
num_layers = 6
freeze_layers = 2

freeze_layers and num_layers are only supported for the standard HuggingFace wav2vec2/WavLM/HuBERT backends; both are ignored (with a warning) for emotion2vec* pretrained models.

Output & Loss Functions

Results include the best model and Hugging Face logs compatible with TensorBoard. Loss functions default to weighted cross-entropy for classification and concordance correlation coefficient for regression tasks.