Tiny Transformers for Financial Sentiment Analysis: Class Imbalance Resolution and CPU-Optimized Deployment

Abstract

This paper shows that small-scale Transformer models can successfully replace large-scale language models in financial sentiment analysis and break through essential barriers of class imbalances and CPU-alone deployability. With the 9.5M-parameter Tiny Transformer, we present a systematic enhancement pipeline: (E1) long training with better scheduling and (E2) class-weighted loss tuning. The baseline model obtains just 26.2% Macro-F1, which shows strong bias toward the dominant positive class (64.7% of samples). Our method improves performance to 61.6% Macro-F1—a 135% rel. Improvement—despite achieving just 11.5ms inference latency in one CPU core (613 samples/second rate). Ironically, we prove that large models (19M–35M params) become progressively less effective, and we achieve a better accuracy-per-parameter ratio with our compact model. Rigorous per-class analysis substantiates that minority sentiment detection significantly improves—from 0% to 43.5% for the negative class—without hurting the overall predictive stability. These results prove that if trained with domain-scholar techniques, small models can beat larger baselines at the cost of being deployable in resource-limited financial settings. The method we introduce is a reproducible roadmap for constructing real-time finance NLP systems running opportunistically without GPUs or clouds.