Sentiment analysis on financial headlines: why the benchmark misleads

I trained a model that reads sentiment from financial news and scored 0.79 macro-F1 on the benchmark. A perfectly good number. Then I tried the model on real, everyday Turkish news headlines and the score dropped to 0.53.

This post is about where that gap came from and how I partly closed it.

Setup

The goal was simple: give it a headline, get a positive / negative / neutral label. I trained on two sources:

• Financial PhraseBank — English financial sentences labelled by experts.
• ITU's KAP headlines and a Turkish translation of PhraseBank — for the Turkish side.

I started with the simplest model: TF-IDF + logistic regression, 0.69 macro-F1 on the English test. Then I fine-tuned xlm-roberta-base on a mix of both languages: 0.79 on the mixed test set (English 0.85, Turkish 0.78).

There was one leak I closed while merging the data: an English PhraseBank sentence and its Turkish translation are assigned to the same group. Otherwise one sits in training, the other in testing, and the model looks as if it "knows" a sentence it has effectively memorised.

Meeting real headlines

Benchmarks are clean, balanced and consistent. A real news feed is not. I labelled 129 headlines collected from live RSS sources by hand and tested the model on them:

The confusion matrix made the problem clear: the model almost never said "neutral".

The cause is domain shift. Most of the Turkish training data comes from KAP disclosures, which describe concrete events — "net profit rose X percent", "contract signed". Most real headlines signal no direction at all: "All eyes on the central bank's rate decision", "The week ahead in markets". The model had learned to see a direction in every headline.

One threshold, a big difference

The first fix was not new data but a decision rule: if the model picks a sentiment class but is not confident enough, fall back to neutral.

def apply_margin(probs, margin):
    """Say neutral if argmax is a sentiment class with probability < margin."""
    out = []
    for p in probs:
        i = int(p.argmax())
        out.append(LABELS[NEU] if i != NEU and p[i] < margin else LABELS[i])
    return out

I chose the threshold with cross-validation on the hand-labelled headlines rather than by guesswork; the average of the values picked in two folds became the default. The result:

I am not hiding that this is a patch. The model learned the wrong thing; the threshold only softens the effect. The lasting fix is retraining on real headlines.

A subtle tweak to class weights

The classes were imbalanced, and my first instinct was scikit-learn's "balanced" weights. That crushed the neutral class and pushed the model even further toward sentiment-bearing predictions — exactly the problem I later hit in the real world. Taking the square root of those weights gave a gentler balance.

Filtering out the off-topic

The live feed had another problem: celebrity, sports and health news. The sentiment model labelled those confidently too. I put a binary "finance / off-topic" filter in front of it.

I had only 46 off-topic examples. TF-IDF could not cope with so little data (macro-F1 0.69, off-topic recall 0.43). On the same data, logistic regression on XLM-R embeddings reached 0.82. Embeddings from a pre-trained language model make learning from a handful of examples possible. Adding 294 Turkish headlines labelled by a language model pushed it to 0.84.

What I learned

A benchmark score does not measure how good your model is — it measures how much your data resembles the benchmark.

• Validate on real data, even a little. 129 hand-labelled headlines exposed in an afternoon what a benchmark of thousands of rows had hidden.
• Look at the confusion matrix. A single score says "bad"; the confusion matrix tells you why.
• Use embeddings when data is scarce. Pre-trained representations work even with a few dozen examples, where a model learning from scratch cannot.

The next step is clear: grow the set of real headlines, retrain on them, and reach the point where the threshold is no longer needed.