An LLM can explain QE in a Haiku but it can’t pick stocks

DEEPSEEK Prompt: Explain quantitative easing in a Haiku poem

DEEPSEEK response

New coin from thin air,
Banks buy bonds, rates fall to nil—
Prices rise, yet wealth? Still.

 

The success of a Large Language Model appearing to be creative and communicative has generated false hope that AI should be able to make us all rich trading stocks.  Flick through Tik Tok and you will be targeted by advertisers touting trading models. ‘For just $20 you can see for yourself…’ is the pitch.  Studies show that 80% of the trading models lost around 30% of their value over 2 weeks.  How can a model that performs so reliably well that it can explain Quantitative Easing in a Haiku poem, stuff up when deciding to buy Google versus Apple?

I think the answer to this question comes down to the process generating the data.  Financial datasets are notoriously unstable whereas language is intended to be an intermediary that gets repeated and reused.  The whole point of language is to communicate with another individual with subtle semantic variation.  Financial data is a blancmange of actions and reactions to events that are neither predictable nor understood.  It has no objective to replicate itself.  The practice of training your model on 80% of your data and testing it on the remaining 20% should deliver an Haiku poem when prompted, whereas a recent run of 15 successive upticks in Tesla’s stock price does not mean the next move will be up (or down).

AI researchers have a habit of saying ‘…here’s the internet, go fit a model…’.  Implicitly, the approach to financial datasets has been to add parameters so as to tame the data and make it fit.  This might provide scaffolding for the 80% training data but will not force the 20% test data into conformity.  So rather than let one’s LLM run wild on a dataset, an enhancement might be to prescreen the dataset to judge its relevance for the task at hand.  This could be an evaluation of the stability of the data before attempting to estimate anything.  Simple similarity tests can be used to compare the split datasets for compatibility (e.g. Difference tests such as difference in mean, difference in variance and difference in factors such as principal components) and researchers should be quick to resist the impulse to fit a model to an unfriendly dataset.  Above all, however, there should be some common sense applied to the data.  What is it about the data that makes us confident that the stock trading model will work?

Historically, research has fit models to data. Data is abundant these days, so maybe its time to ask the question “…does the data fit the model?”