Redesigning the mannequin
TimesFM is a patched decoder that tokenizes each 32 contiguous timepoints (a patch) as an enter token and applies a transformer stack on prime of the sequence of enter tokens to generate the output tokens. It then applies a shared multilayer perceptron (MLP) to translate every output token again to a time collection of 128 timepoints.
To create TimesFM-ICF (In-Context Superb-tuning), we begin with the bottom TimesFM mannequin and proceed the pre-training with new context: the forecast historical past plus all in-context examples. Step one is to verify the mannequin doesn’t confuse or conflate the forecasting historical past and the in-context examples. Think about you are giving the mannequin an inventory of numbers that signify just a few various things, possibly sun shades gross sales figures from one retailer, then umbrella gross sales figures from one other. When you simply merge all these numbers collectively, the mannequin may get confused, considering it is one steady stream of knowledge. For instance, if the primary retailer’s gross sales had been going up and the second retailer’s gross sales had been happening, the mannequin may incorrectly see it as a single up-and-down sample, moderately than two separate, easy developments.
To repair this, we put a particular, learnable “frequent separator token” — like a digital “cease signal” or a “new paragraph” image — after every set of numbers. With these separators in place, as quickly because the mannequin attends to the separator token of an instance it has seen earlier than, it will not combine it up with the information it is presently making an attempt to foretell. This theoretically permits the mannequin to study from patterns in these previous examples and apply that data to the present forecast. As an example, the mannequin might study that “all the shop gross sales are exhibiting constant, directional developments currently, so I ought to predict an upward development for my new retailer’s sunscreen gross sales.”
