林彥廷：LLM的本質與未來：從第一性原理出發

1. In one sentence

The real bottleneck in LLM development is not a bigger model or a cleverer algorithm but how you design the benchmark — where you plant the flag determines which mountain everyone climbs, so learning to set up a good benchmark matters more than chasing any model's score.

2. The session

Opening: a disrupted schedule and the "mystery guest" joke

He said curator 李慕約 only told him the day before that a "mystery guest" — "a VP from Google" — would be speaking in the slot before him, meaning 紀懷新. He joked that what he had prepared was more of a "story", and that after hearing the preceding session, "the company closest to AGI, the most visionary talk", he felt he should offer something different, something fewer people are willing to say publicly — which is why he decided to share this whole set of lessons about evaluation. He also mentioned graduating from NTU in January, and that this was the day of NTU's graduation ceremony: he should have been in his doctoral gown for photos with his supervisor, and instead spent the night rewriting his slides for the next day's talk.

Motivation: a junior researcher's confession

He framed the session as "a junior researcher's confession", echoing an earlier speaker's remark that day about "it's the people at the bottom who need to learn AI". His central metaphor: once you plant a flag and build a mountain, somebody will climb to the summit — so where you plant the flag (the benchmark you define) matters more than any algorithmic cleverness. He mentioned open-source projects he has worked on (without going into detail here; he means Taiwan-LLM and others, see section seven) but said they are not really relevant to today's topic.

He was candid: training an LLM or any AI, whatever the input and output modality, involves a pipeline that is already extremely complex inside a large company, and very few researchers can overturn a whole algorithm single-handed. What most people actually do is tune the dataset — each researcher is assigned a territory (his own was mathematical reasoning, others might have image output or therapeutic dialogue) and keeps feeding their slice into the central training system, while holding to a budget (other capabilities must not drop more than 10% to 20%, say). Do your territory well and keep the rest from sliding too far, and you have passed.

A case: an AIME maths problem from zero to something you can report

He told a story he had never told publicly. Last summer OpenAI's o1 scored somewhere around 50 to 80 on AIME (the harder competition American high school students reach after AMC), while the model he was training scored zero — it could not even produce a parseable, normal answer. His boss said only "find a way to improve mathematical ability", and he laid out three options: a bigger model, a cleverer algorithm, or more data. He said that as a research intern he had no standing to alter the existing pipeline in a large company, whose compilation cycle is glacially slow, so the only lever he could touch was data.

Technically the evaluation used "answer matching": the model generates a reasoning chain and an answer, and a checking module compares it against the reference answer for equivalence. His example (quoted directly from the slide): "if 4 daps equal 7 yaps, and 5 yaps equal 3 baps, how many daps equal 42 baps", with a reference answer of 40. When the model output something like "160/4 daps", mathematically equivalent but differently written, he found that the checking module was in fact wrong — because it is rule-based, and cannot tell that two mathematical expressions written differently, such as integrals or derivatives, are actually equal.

The lesson he took: having found this kind of low-hanging fruit, simply reporting "there's a bug here" usually gets you ignored, because there is no impact in it. What actually works is exploiting the hole in reverse: generate a large volume of hard maths problems, sample the model up to nearly ten thousand or even fifty thousand times per question — the logic of the monkeys and typewriters eventually producing a good essay — keep the correct samples, discard the wrong ones, retrain the model on that newly produced high-quality data, and only then take a good-looking benchmark scorecard to your boss, presented as a comparison against other companies' baselines.

Where academic test sets fall short: benchmark maxxing

He considers mathematics one of the few domains where scoring is stable, because right and wrong are objective and no two mathematicians will argue about it; but vibe coding or building a product usually has no reference answer at all, and academic test sets do not cover that ground. He noted that Taiwan has a great many AI companies who nevertheless care greatly about academic test scores — he himself produced a benchmark about two years ago while working on Taiwan LLM, built around junior and senior high school exams, entrance exams, the comprehensive assessment and Taiwan's civil service examinations. But Google Gemini, OpenAI's models, Grok and another model whose name he said but was hard to make out, possibly "FarPig" (unverified, see section seven), can now solve most of those questions — meaning that benchmark is thoroughly obsolete. His swipe: even translated into Chinese, nobody would really ask questions like these in practice, and "benchmark maxxing" is of little practical help to anyone building AI applications, so there is no need to force your product to fit an academic benchmark. He also cautioned that when influencers say "AI scored such-and-such on this benchmark", opening that dataset usually reveals it has nothing to do with your own use case, and the score is of no reference value to your product.

Be careful what you wish for: from DAU to sycophancy

Setting academic tests aside, he said many researchers are actually doing online testing, and what surprised him most is that how you tune a model's training genuinely affects daily active users — a different matter from marketing. He mentioned that when interviewing at OpenAI, they stressed how much they care about DAU — which he found interesting: companies that in theory should be pursuing the highest intelligence and approaching AGI end up valuing active users. He cited Sam Altman saying in a Stratechery interview with Ben Thompson that users matter more than the most advanced model (the original wording of that quote was not separately verified — marked as the speaker's account, see section seven), alongside similar user behaviour metrics such as like rates and how often content gets copied.

He then pivoted on "be careful what you wish for": over-optimising these user-pleasing metrics tends to backfire. He gave two recent examples: Meta's Llama 4 was observed pushing hard on LM Arena scores (a head-to-head LLM rating platform) at the cost of quality elsewhere; and ChatGPT (GPT-4o) shipped an update that made the model excessively sycophantic and was rolled back shortly after launch, with the official post-mortem conceding that "we didn't specifically evaluate for sycophancy in deployment" (checking afterwards, that is the content of OpenAI's published "Expanding on what we missed with sycophancy" — see section seven). He added that this had been flagged two years earlier — Jerry Wei at Google DeepMind had already published a paper showing LLMs have a severe tendency toward sycophancy (checked afterwards, see section seven) — and yet two years on the industry still has not made it a routine part of benchmarking.

How to define acceptance criteria: OpenAI's preparedness framework

He brought the topic back to practice: defining "acceptance criteria" for B2B applications or client work is genuinely difficult. The usual formulations look like "95% matching the expected result" or "90% translation accuracy", which sound numerical but, with no objective standard, end up as rule by person — the manager looks at it and passes it, or does not.

He then introduced how OpenAI handles this: the official line is the "preparedness framework", presented externally as preparedness for societal impact, though he believes the real intent is to assess how ready these models are to take over software engineers' work. He listed several categories inside it: OpenAI's own software engineering interview questions; the ability to resolve external GitHub issues; the ability to resolve OpenAI's own internal issues (he mentioned Anthropic saying seventy per cent of its issues are resolved by its own Claude models, and that OpenAI has similar internal data — a year ago GPT-4o could resolve 3% of internal issues and it has now improved to 34%, a third of them; no independently checkable public sources were found for these specific percentages, so they are marked as the speaker's account, see section seven); the ability of a software engineering agent to work as a contractor; and its ability to work as a researcher. He particularly admires the "resolve our own internal issues" metric, because OpenAI certainly will not train its models on its own code (that would leak the answers), which makes this evaluation relatively clean.

He also noted that for tasks like agents — which everyone now loves to use and which are hard to judge — OpenAI's approach is to construct a rubric for every question. His example was HealthBench: for a question about an infant possibly showing muscle weakness, the rubric itemises whether the answer explains that the infant may have muscle weakness, whether it briefly describes common causes, whether it includes advice to seek immediate medical attention, and whether points should be deducted if the referral advice is buried in the middle or the end of the response. Each line is a yes-or-no item along the lines of "present, add 7 points; absent, zero", which is far easier to write than attempting to compose one perfect model answer, because you only need to define the scoring criteria and then let the AI grade against them. His slide also cited PaperBench as another rubric-based benchmark.

Building incentives: good evaluation data is itself training data

He considers this the most important slide of the session: there is a conflict of interest in engineers writing their own evaluation data. He returned to the earlier story — once you find a bug, reporting it directly has little impact, whereas using the hole to produce more data has a great deal. Writing evaluation data follows the same logic: a good evaluation set doubles as a test set and a training set, and for the company and senior management the training set usually carries more impact than the test set. So as a manager you have to create enough incentive — more monetary reward, for instance — for the people below you to be willing to put effort into producing good data, rather than dumping annotation responsibility on whichever engineer happens to own that scope; otherwise the manager simply gets fobbed off indefinitely. His example was GPQA: a dataset led by NYU, using doctoral students for data annotation and paying up to tens of thousands of NT dollars per item, against a general annotation company paying about NT$3,000 an hour (he gave no verifiable original source for either figure — marked as the speaker's account, see section seven) — the point being that high-quality, highly discriminating evaluation data should carry correspondingly high incentives.

He also stressed the importance of discriminability: experts should agree with each other on the scoring criteria, while a layperson should basically be unable to answer — because if even a layperson can answer, then the strongest AI today, or the AI three months from now, certainly can, and at that point the evaluation's power drops straight to zero.

He closed on one line: he hopes everyone can set up a really good benchmark, whether what you want to do is "papermaking" or "reading". That echoes his earlier self-deprecation — most people should read more (apply existing knowledge) rather than learn papermaking as he does (fumbling from the very bottom of the pipeline) — even though almost everything he shared today is a papermaking story.

3. Figures and cases

• Last summer's AIME maths competition: OpenAI's o1 scored roughly 50 to 80, while his model scored zero (speaker's own account, no external source).
• Rejection sampling on hard maths problems at nearly ten thousand to fifty thousand samples per question, keeping the correct ones as new training data (speaker's own account, no external source).
• The answer matching example question and its reference answer of 40 come from the MATH dataset paper (checked afterwards, see section seven).
• The benchmark he designed two years ago while working on Taiwan LLM, built on junior and senior high school exams, entrance exams, the comprehensive assessment and civil service exams, is now largely solved by Gemini, OpenAI's models and Grok (the Taiwan LLM paper's year, checked afterwards, broadly matches "two years ago" — see section seven).
• OpenAI's GPT-4o resolved 3% of its own internal issues a year ago, improving to 34% now (o3); Anthropic says seventy per cent of its issues are resolved by its own Claude models (both the speaker's account, no external source, see section seven).
• HealthBench's rubric is written as yes-or-no items — whether it mentions the infant may have muscle weakness, whether it describes common causes, whether it advises immediate medical attention (checked afterwards, see section seven).
• GPQA annotation costs: up to tens of thousands of NT dollars per item, against roughly NT$3,000 an hour for general annotation (speaker's own account, no external source; the dataset itself has external corroboration, see section seven).

4. Lines worth keeping

• "Once you've planted a flag and built a mountain, somebody will definitely come and successfully climb to the summit."
• "Be careful what you wish for."
• "You need to create a lot of incentive so the people below you get more monetary reward for collecting this good data — because that's how you stop being fobbed off by them."
• "If even a layperson can answer it, then the strongest AI today, or the AI three months from now, certainly can."
• "I hope everyone can set up a really good benchmark, whether what you want to do is papermaking or reading."

5. Tools and terms mentioned

• Taiwan-LLM (Taiwan-LLaMa): the Traditional Chinese large language model project he developed.
• AMC / AIME: American high school mathematics competitions; doing well at AMC qualifies you for the harder AIME.
• The MATH dataset / answer matching: the hard mathematical problem-solving benchmark proposed by Hendrycks et al. in 2021, and the practice of comparing a model's answer against the reference answer for equivalence.
• Rejection sampling: the data generation technique of sampling a model repeatedly on the same question and keeping only the correct results.
• LM Arena: the head-to-head LLM rating platform where users vote between two models' answers to produce a ranking.
• Sycophancy: a model's tendency to sacrifice objective correctness in order to agree with the user's views or feelings.
• DAU (daily active users): the measure of a product's activity, which he notes is also one of the online testing metrics model tuning cares about.
• Preparedness framework: OpenAI's framework for tracking model capability and risk.
• SWE-bench / GitHub issue resolution: a way of benchmarking an AI's ability to resolve real software engineering issues.
• HealthBench: OpenAI's rubric-based benchmark for the medical domain.
• PaperBench: OpenAI's benchmark for an AI's ability to reproduce academic papers.
• GPQA: the graduate-level, "Google-proof" hard question-answering dataset.
• Rubric: the itemised scoring criteria for a single question, which lets an AI act as the grader.

6. Wider observations

This session lands harder than any flashy demo, because he is describing the wall everyone building AI applications eventually runs into: you think you are working on a model or a product, and what you are actually doing all along is the political decision of which set of numbers to believe. His line about where you plant the flag determining which mountain everyone climbs is worth remembering more than any benchmark score on any slide.

The AIME story also reopens the question of whether you should report a bug the moment you find it. His answer is utilitarian but honest: simply raising your hand about a problem gets you ignored, so you first have to turn the problem into a bigger opportunity and present them together. That is the other side of the same coin as the earlier sessions on team culture and product development.

7. Sources