TimesFM: Time Series Foundation Model for time-series forecasting
https://github.com/google-research/timesfmI've worked on language models since 2018, even then it was obvious why language was a useful and transferable task. I do not at all feel the same way about general univariate time series that could have any underlying process.
Also, the first realistic approximation of Solomonoff induction we achieve is going to be interesting because it will destroy the stock market.
"Jim Simons' Renaissance Technologies suffers $11 billion of client withdrawals in 7 months" - https://markets.businessinsider.com/news/stocks/jim-simons-r...
So you think the multi-trillion dollar stock market, consisting of thousands of global companies, has no use beyond "pulling money out of systems"? Weird.
Just to say, weirdness happens.
Yes, people arbitrage away these anomalies, and make billions doing it.
There's an extensive body of literature across numerous domains that demonstrates the benefits of Multi-Task Learning (MTL). Actually I have a whole folder of research papers on this topic, here's one of the earliest references on hand that I feel captures the idea succinctly in the context of modern ML:
“MTL improves generalization by leveraging the domain-specific information contained in the training signals of related tasks" [Caruana, 1998]
I see repetition and structure everywhere in life. To me it's not far fetched that a model trained on daily or yearly trends could leverage that information in the context of e.g. biological signals which are influenced by circadian rhythm etc.
Disclaimer: my background is in ML & bio-signals, I work with time series too much.
"The Unreasonable Effectiveness of Mathematics in the Natural Sciences" [1] hints that there might be some value here.
[1] https://en.m.wikipedia.org/wiki/The_Unreasonable_Effectivene...
But you wouldn't want this model for file upload storage usage which only increases, where you would put alerts based on max values and not patterns/periodic values.
Think about something like traffic patterns. You probably won't predict higher traffic on game days, but predicting rush hour is going to be pretty trivial.
Operator guidance is often based on heuristics - when metric A exceeds X value for Y seconds take action Z. Or rates of change if the signal is changing at a rate of more than x etc.
So in these areas there exists potential for ML solution, especially if it's capable of learning (i.e. last response overshot by X so trim next response appropriately).
It's not just that control charts are great signal detectors, but also managing processes like that takes a certain statistical literacy one gets from applying SPC faithfully for a while, and does not get from tossing ML onto it and crossing fingers.
There are clear counterexamples to your experience, most notably in maintaining plasma stability in tokamak reactors: https://www.nature.com/articles/s41586-021-04301-9
After all, the physical world (down to the subatomic level) is governed by physical laws. Ilya Sutskever from OpenAI stated that next-token prediction might be enough to learn a world model (see [1]). That would imply that a model learns a "world model" indirectly, which is even more unrealistic than learning the world model directly through pre-training on time-series data.
People on the AI-hype side of things tend to believe this, but I really fundamentally don't.
It's become a philosophical debate at this point (what does it mean to "understand" something, etc.)
There's a huge industry around time series forecasting used for all kinds of things like engineering, finance, climate science, etc. and many of the modern ones incorporate some kind of machine learning because they deal with very high dimensional data. Given the very surprising success of LLMs in non-language fields, it seems reasonable that people would work on this.
Efficiently Modeling Long Sequences with Structured State Spaces
https://www.youtube.com/watch?v=luCBXCErkCs
They made one of the best time series models and it later became one of the best language models too (Mamba).
It's difficult to use off the shelf tools when starting with math models.
https://raw.githubusercontent.com/ESWAT/john-carmack-plan-ar...
Thank you for this treasure.
The relevant bits:
> I am now allowing the client to guess at the results of the users movement until the authoritative response from the server comes through. This is a biiiig architectural change. The client now needs to know about solidity of objects, friction, gravity, etc. I am sad to see the elegent client-as-terminal setup go away, but I am practical above idealistic.
> The server is still the final word, so the client is allways repredicting it's movement based off of the last known good message from the server.
If you want to predict not just position but also orientation in a shooter game, that's basically predicting the mouse movements.
For instance, will it be able to predict dynamics for a machine with thousands of sensors?
My complaint was more illustrative than earnest.
- most OCs on this thread
2. It doesn't actually replace a USB drive. Most people I know e-mail files to themselves or host them somewhere online to be able to perform presentations, but they still carry a USB drive in case there are connectivity problems. This does not solve the connectivity issue.
3. It does not seem very "viral" or income-generating. I know this is premature at this point, but without charging users for the service, is it reasonable to expect to make money off of this?
Previous discussion: https://news.ycombinator.com/item?id=39235983
This one looks pretty easy to setup, in fairness, but some other models I've looked at have been surprisingly fiddly / locked behind an API.
Perhaps such a thing already exists somewhere?
Suppose you want to buy stocks? Would you look on a time based graph and buy according to that? Or you rather look at financial data, see earnings, profits? Wouldn't a graph that has financial performance on x-axis be more meaningful that one that has time?
What if you research real estate in a particular area? Wouldn't be square footage a better measure than time?
[5, 3, 3, 2, 2, 2, 1, …]
What is the next number? Well let’s start with the search space - what is the possible range of the next number? Assuming unsigned 32bit integers (for explanation simplicity) it’s 0-(2^32-1)
So are all of those possible outputs equally likely? The next number could be 1, or it could be 345,654,543 … are those outputs equally likely?
Even though we know nothing about this sequence, most time series don’t make enormous random jumps, so no, they are not equally likely, 1 is the more likely of the two we discussed.
Ok, so some patterns are more likely than others, let’s analyse lots and lots of time series data and see if we can build a generalised model that can be fine tuned or used as a feature extractor.
Many time series datasets have repeating patterns, momentum, symmetries, all of these can be learned. Is it perfect? No, but what model is? And things don’t have to be perfect to be useful.
There you go - that’s a pre-trained time series model in a nutshell
If not, what would be a useful model?
"Hidden Markov map matching through noise and sparseness" (2009)
https://www.microsoft.com/en-us/research/wp-content/uploads/...
And exponentially higher cost for ML models.
I also think it's a dead end to try to have foundation models for "time series" - it's a class of data! Like when people tried to have foundation models for any general graph type.
You could make foundation models for data within that type - eg. meteorological time series, or social network graphs. But for the abstract class type it seems like a dead end.
For example you can easily predict the weather with descent accuracy. Tomorrow is going to be about the same than today. From there you can work on better models.
Or predicting a failure in a factory because a vibration pattern on an industrial machine always ended up in a massive failure after a few days.
But I agree that if a model is good at predicting the stock market, it’s not going to be released.