Turkish Text Classification with XLM-RoBERTa, Step by Step

In short: XLM-RoBERTa is a language model pre-trained on around a hundred languages, including Turkish, and can be adapted to a text classification task in a few hours with Hugging Face transformers. This article walks step by step through the fine-tuning I did for sentiment classification of Turkish and English financial headlines: merging data and preventing leakage, a class-weighted loss, training settings and evaluation. Result: 0.79 macro-F1 on the mixed test set (English 0.85, Turkish 0.78).

This article covers the technical side of my financial news sentiment analysis project. I told the "benchmark versus real world" story of the project in a separate article; here the focus is the fine-tuning itself.

Why XLM-RoBERTa?

There are two routes for Turkish text classification: a Turkish-only model or a multilingual one. My dataset was a mix of Turkish and English sentences, and the same service had to work in both languages. XLM-RoBERTa was a natural choice:

• One model understands both languages; no separate services needed.
• Signal learned from English data partly transfers to the Turkish side.
• xlm-roberta-base is small enough to train on a laptop (Apple Silicon, MPS).

Step 1: Merging data and preventing leakage

Training data came from three sources: the English Financial PhraseBank, a Turkish translation of PhraseBank and ITU's KAP headlines. There is an easily missed trap here: if an English sentence and its Turkish translation land in different splits, the model has effectively "seen" the test sentence during training and the score is artificially inflated.

The fix is to give each sentence a group key and split by group rather than by row:

def _bucket(group: pd.Series) -> pd.Series:
    return group.map(lambda g: int(hashlib.md5(g.encode()).hexdigest(), 16) % 10)

b = _bucket(df.group)
splits = {"train": df[b >= 2], "val": df[b == 1], "test": df[b == 0]}

Because the translation takes the same group value as the English original, the two always land in the same split. Hash-based splitting is also deterministic: the splits do not change when the data is rebuilt.

Step 2: Tokenizer and dataset

from datasets import Dataset
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("xlm-roberta-base")

def to_ds(df, tok, max_len):
    ds = Dataset.from_pandas(df[["text", "label"]].reset_index(drop=True))
    ds = ds.map(lambda b: tok(b["text"], truncation=True, max_length=max_len), batched=True)
    ds = ds.map(lambda b: {"labels": [LABEL2ID[l] for l in b["label"]]}, batched=True)
    return ds.remove_columns(["text", "label"])  # string columns break the collator

Headlines are short, so max_length=64 is enough; it cuts memory and training time considerably. The remove_columns in the last line is small but important: if text columns stay in the dataset, DataCollatorWithPadding throws an error.

Step 3: A weighted loss for class imbalance

Classes in financial data are not balanced. The first instinct is scikit-learn's "balanced" weights. On my dataset that suppressed the neutral class too much. For a gentler balance I took the square root of the weights and passed them to the loss through a custom Trainer:

counts = tr.label.value_counts().reindex(LABELS).values
cw = np.sqrt(len(tr) / (len(LABELS) * counts)).tolist()

class WeightedTrainer(Trainer):
    def __init__(self, class_weights, **kw):
        super().__init__(**kw)
        self.cw = torch.tensor(class_weights, dtype=torch.float)

    def compute_loss(self, model, inputs, return_outputs=False, **kw):
        labels = inputs.pop("labels")
        out = model(**inputs)
        loss = torch.nn.functional.cross_entropy(
            out.logits, labels, weight=self.cw.to(out.logits.device))
        return (loss, out) if return_outputs else loss

Step 4: Training settings

args = TrainingArguments(
    output_dir="models/xlm-roberta-base/ckpt",
    num_train_epochs=3, learning_rate=2e-5,
    per_device_train_batch_size=32, per_device_eval_batch_size=64,
    warmup_steps=40, weight_decay=0.01,
    eval_strategy="epoch", save_strategy="epoch",
    load_best_model_at_end=True, metric_for_best_model="macro_f1",
    save_total_limit=1, report_to="none", dataloader_pin_memory=False,
)

A few notes:

• metric_for_best_model="macro_f1": accuracy misleads on imbalanced data; the best checkpoint is chosen by macro-F1.
• 3 epochs at a 2e-5 learning rate: a safe starting point for fine-tuning pre-trained models. More usually leads to overfitting.
• dataloader_pin_memory=False: avoids needless warnings and issues on Apple Silicon (MPS).

Step 5: Evaluation — look per language

A single overall score hides which language is weaker in a bilingual model, so I also evaluated the test set per language:

Beyond the benchmark

These scores belong to the benchmark. When I tried the model on real, everyday Turkish news headlines, macro-F1 dropped to 0.53 because of domain shift between the training data and real headlines. A neutral decision threshold calibrated with cross-validation lifted it to 0.72. I described the whole process in this article. The lesson: finishing fine-tuning is half the job; the real test is real data.

Frequently asked questions

Does XLM-RoBERTa support Turkish?

Yes. XLM-RoBERTa was pre-trained on text in around a hundred languages, including Turkish, and can be fine-tuned for Turkish text classification tasks.

How much data do you need for Turkish text classification?

There is no fixed number. Pre-trained models can do well with a few thousand labelled examples; with less data, training a simple classifier on the model's embeddings can be more efficient.

Do you need a GPU to fine-tune XLM-RoBERTa?

A xlm-roberta-base model can be trained on short texts with local hardware such as Apple Silicon (MPS). For large datasets and large models, a GPU speeds things up considerably.

Which metric should you use with imbalanced classes?

Macro-F1 gives each class equal weight, making it a more reliable indicator than accuracy on imbalanced datasets.

How do you prevent leakage between training and test sets?

Marking different forms of the same content — for example a sentence and its translation — with a shared group key and splitting by group rather than by row is the most reliable way to prevent leakage.