Summary

Dan Luu investigates the widely-circulated claim that dynamic languages are significantly more token-efficient for LLMs than static languages, finding it stems from evaluations using trivially small tasks. Running his own evals on implementing a Zstd decoder and Pandoc across many languages, he shows the claimed 2x-3x dynamic-language advantage largely disappears at higher effort levels and on realistic-scale tasks. Language popularity, not static vs. dynamic typing, shows a weak positive correlation with correctness and cost efficiency, likely because AI labs generate more RL training data for mainstream languages. He publishes these results as 'half-baked notes' partly to model a different publishing norm for exploratory empirical work.

Key Insight

The dynamic-language token efficiency advantage is an artifact of trivially small benchmark tasks and doesn't survive contact with realistic workloads, where language popularity is a better predictor of LLM performance than static vs. dynamic typing.

Spicy Quotes (click to share)

• 7

  The very strong relationships that held in the trivial evals don't generalize to this larger case.

• 4

  It turns out that if we plot language popularity vs. performance on this eval, we observe a weak to moderate positive correlation where more popular languages end up with more correct as well as cheaper solutions.

• 5

  With LLMs, a lot of the questions have gone from being effectively unanswerable to being answerable with a bit of effort and some tokens.

• 6

  LLMs massively reduce the amount of effort it takes to get a result that's strong enough to satisfy my curiosity but, AFAICT, they don't reduce the effort it takes to publish a result by much.

• 5

  There's no way to look at the score on one eval or even five or ten evals and draw a conclusion about programming in general.

Tone

analytical, empirically skeptical, self-deprecating