I'm not surprised that a linear regression estimated by a method that knows it should be estimating a linear regression would beat everything else.
The traditional OLS estimator in Excel has all sorts of classical optimality properties when its assumptions (normality, linearity) are true, so no fancy neural net can outperform it even in principle (the only way to outperform it would be to have an informative prior for how you generated the data set's parameters). So if the LLM's beat, or even matched, Excel in that case, they would be thinking too narrowly.
Same with any other models where we know the form of the answer up to some unknown parameters.
If we want parameter estimates, that means we already have a functional form in mind. In that case, we get to use established statistical theory to design optimal estimators, whether by Bayesian or other methods. Black box neural magic wouldn't help (or it might help indirectly, in computationally intractable cases).
What we would want the LLM's to do, ideally, is explore the space of known/possible 'patterns' and perform well in situations where the underlying relationship exists, is not known in advance, and is known not to have a simple form we can describe. Much like they (and we!) produce text without being able to describe why they are producing that particular text, we would expect them to make those predictions without being able to explain them in terms of parameters and functions - not without a whole other layer of explainability machinery.
> In that case, we get to use established statistical theory to design optimal estimators, whether by Bayesian or other methods. Black box neural magic wouldn't help (or it might help indirectly, in computationally intractable cases).
Using 'established statistical theory to design optimal estimators' isn't trivial for most people. Black box magic might still be useful for them.
Let the LLM's magically write code for the optimal estimators, using existing theory; but that code will implement interpretable math with provable characteristics, as opposed to magic.
The traditional OLS estimator in Excel has all sorts of classical optimality properties when its assumptions (normality, linearity) are true, so no fancy neural net can outperform it even in principle (the only way to outperform it would be to have an informative prior for how you generated the data set's parameters). So if the LLM's beat, or even matched, Excel in that case, they would be thinking too narrowly.
Same with any other models where we know the form of the answer up to some unknown parameters.
If we want parameter estimates, that means we already have a functional form in mind. In that case, we get to use established statistical theory to design optimal estimators, whether by Bayesian or other methods. Black box neural magic wouldn't help (or it might help indirectly, in computationally intractable cases).
What we would want the LLM's to do, ideally, is explore the space of known/possible 'patterns' and perform well in situations where the underlying relationship exists, is not known in advance, and is known not to have a simple form we can describe. Much like they (and we!) produce text without being able to describe why they are producing that particular text, we would expect them to make those predictions without being able to explain them in terms of parameters and functions - not without a whole other layer of explainability machinery.