Text Classification with Little Data: 0.82 F1 from 46 Examples

In short: With only a few dozen labelled examples, TF-IDF usually falls short for text classification. A simple logistic regression built on sentence embeddings from a pre-trained language model can do much better with the same data. In a filter I built to remove non-financial headlines from a news feed, TF-IDF reached 0.69 macro-F1 with 46 off-topic examples, while the embedding-based model reached 0.82.

This article tells the story of a small but critical component of my financial news sentiment analysis project: the off-topic news filter.

The problem: the sentiment model labels everything

When I started pulling headlines from live news sources, I hit a problem: the feed was not only financial news. Sports, celebrity and health headlines came in too, and the sentiment model confidently called them "positive" or "negative". Counting "National team wins friendly match" as positive for the markets skewed the day's sentiment ratios.

The fix was a binary classifier running before the sentiment model: finance or off-topic?

Why TF-IDF was not enough

My first attempt was the simplest approach: TF-IDF vectorisation followed by logistic regression. But I had only 46 off-topic examples.

As expected, the result was weak: 0.69 macro-F1 in cross-validation, with off-topic recall of just 0.43 — more than half of off-topic headlines slipped through.

The reason is simple: TF-IDF only looks at word overlap. If it never saw "match" during training, it cannot tell that a headline containing "derby" is sports news. With little data, vocabulary coverage is very narrow.

The fix: pre-trained embeddings

A pre-trained language model has learned relationships between words and concepts from an enormous amount of text. It already knows "derby" and "match" belong to the same world. To use that knowledge you do not need to fine-tune the model; you only need to turn sentences into fixed-length vectors (embeddings).

I reused the XLM-RoBERTa encoder from the sentiment model and took the attention-mask-weighted average of the last layer's token representations (mean pooling):

class Embedder:
    """Mean-pooled sentence embedding, sharing the sentiment model's encoder."""

    def __init__(self, path, device=None):
        self.tok = AutoTokenizer.from_pretrained(str(path))
        self.model = AutoModel.from_pretrained(str(path)).eval()
        self.device = device or ("mps" if torch.backends.mps.is_available() else "cpu")
        self.model.to(self.device)

    @torch.no_grad()
    def __call__(self, texts, bs=32):
        out = []
        for i in range(0, len(texts), bs):
            enc = self.tok(list(texts[i:i + bs]), padding=True, truncation=True,
                           max_length=64, return_tensors="pt").to(self.device)
            h = self.model(**enc).last_hidden_state
            m = enc["attention_mask"].unsqueeze(-1).float()
            out.append(((h * m).sum(1) / m.sum(1)).cpu().numpy())
        return np.vstack(out)

Using the mask matters: if padding tokens are included in the average, embeddings for short headlines are distorted.

Then I trained an ordinary logistic regression on these vectors:

X = Embedder(MODEL_PATH)(texts)
clf = LogisticRegression(max_iter=3000, C=1.0, class_weight="balanced").fit(X, y)

Results

Same data, same classifier; the only difference is how the input is represented. It shows clearly that with little data, representation can matter more than the model.

Growing the labelled data: labelling with a language model

To improve the filter, I had a large language model label 294 Turkish headlines and added them to training; the score rose to 0.84. I was careful about one rule: the headlines I labelled by hand were kept aside for testing, and language-model labels were used for training only. Otherwise the model would have been fed from the same source it was evaluated on.

When does this approach work?

• When you have tens or a few hundred labelled examples,
• When the task depends on the general meaning of the text (topic, intent, domain detection),
• When you want a fast, explainable model.

Once you reach thousands of labelled examples, fine-tuning the whole model usually performs better. I covered that in the XLM-RoBERTa fine-tuning article.

Frequently asked questions

How do you do text classification with little data?

Turning texts into sentence embeddings with a pre-trained language model and training a simple classifier such as logistic regression on those vectors usually performs much better than TF-IDF on small data.

What is a sentence embedding?

A fixed-length numeric vector representing the meaning of a sentence. Sentences with similar meanings have vectors that are close to each other.

Why should mean pooling use the attention mask?

In batches, short texts are padded with filler tokens. If those tokens enter the average, the representation of short texts is distorted; the mask ensures only real tokens are averaged.

Is data labelled by a language model reliable?

It can be useful for growing training data, but evaluation should always be done on a human-labelled set that was not used for training.

When is TF-IDF still a good choice?

When labelled data is plentiful, classes are separated by distinctive keywords, and a very fast, lightweight model is needed, TF-IDF is a strong baseline.