吳柏翰：語言模型如何學習

1. In one sentence

吳柏翰 used the metaphor of a central kitchen steaming lunchboxes to explain how APMIC uses knowledge distillation to condense a giant general-purpose model's domain knowledge into a small model an enterprise can afford, then uses test-time scaling to bring the small model's reasoning quality close to the large one's.

2. The session

Opening: on-premises models have reached the point of adoption

He set the frame immediately: on-premises models are now a good fit for enterprise adoption — open-weight model performance and reasoning have grown substantially, there is more choice in inference hardware, and the cost is becoming bearable. He immediately added a very practical caveat: on-premises, never think in terms of one very large model doing all sorts of different things. Do limited things with a limited model at the lowest possible cost, so that the most people can actually get to use it.

A driving metaphor: RAG is a passenger giving directions, fine-tuning is learning the route yourself

This was his most direct explanation of the difference between RAG (which he pronounced "rig" throughout) and fine-tuning. RAG is like having someone in the passenger seat constantly saying "a bit left, a bit right, you're about to hit that" — you only turn the wheel when someone tells you to. Fine-tuning is learning the whole route into your own head, so you need nobody beside you and your reaction is fastest. He extended the example: to build a legal AI assistant, the generic approach throws the statutes and the company's documents into RAG and looks everything up, thinks and answers afresh every time; fine-tuning trains the statutes and company knowledge directly into the model's head, so afterwards it only has to handle newly arriving documents — a noticeable difference in speed.

Why distil: compressing 2TB into 6GB

He described enterprise knowledge as a database-scale search problem: what used to mean querying datasets in the terabytes now gets compressed by fine-tuning down to 6GB, and shifts from "search" to "conversation". That, in his view, is where distillation is most tangible for an enterprise: compressing an enormous body of knowledge into a small model you can talk to directly and keep evolving.

Where "S1" came from: a paper from Fei-Fei Li's team two months earlier

This was the most interesting detail in the session. He mentioned that APMIC's fine-tuning and distillation offering is called "S1", named after a paper published by Fei-Fei Li's team not long before. On verification, the paper is s1: Simple test-time scaling, first published on 31 January 2025, with ten authors including Fei-Fei Li (alongside Niklas Muennighoff, Percy Liang and other Stanford team members). It has two central methods: first, that just one thousand carefully curated high-quality reasoning examples (the paper's s1K dataset) suffice to train a model with competition-level mathematical reasoning; second, "budget forcing", a technique that truncates or extends the model's thinking with a "Wait" so it revisits and corrects its own reasoning errors at test time, trading more test-time compute for a better answer. That spirit — a small amount of high-quality data plus thinking time at test time — maps exactly onto the "as little as a thousand labelled examples" tier in APMIC's own S1 line.

The S1 line: Lite to Ultra, four tiers for four kinds of enterprise need

Per the full product overview on the slide, S1 is tiered by data volume and budget:

He asked the room whether anyone's company had more than 10 billion tokens of data; almost no hands went up. He used that straw poll to make the point that most enterprises only need to start at S1 Lite or S1 Mini, rather than reaching straight for an Ultra-class in-house model.

Three things open-sourced: datasets, models, evaluation tooling

He announced three open-source releases on stage:

1. Traditional Chinese reasoning datasets: tw-math-reasoning-2k, tw-reasoning-instruct-50k, tw-function-call-reasoning-10k, in a format carrying Chinese input, output and the thinking process, for anyone wanting to fine-tune themselves.
2. An evaluation tool (he called it Auto Eval on stage; on verification, INSIDE's feature confirms the product name is Twinkle Eval): he described the problems with existing Traditional Chinese evaluation tooling — inference too slow, unstable answer extraction (models like to put the answer inside \boxed{}), opaque evaluation settings such as temperature and timeouts, and options that are never randomised so models simply memorise the answer. They collected these pain points and redesigned a faster tool with parallel evaluation support.
3. A 3B reasoning model combined with MCP: he described it as the world's first distilled 3B reasoning model integrating MCP (Model Context Protocol), with over 1,000 downloads, trained on 25 NVIDIA H100s.

Distillation results: 3B model scores before and after fine-tuning

The specific evaluation numbers on the slide (all provided by the speaker, with no independent third-party verification seen — marked as the speaker's own account):

• TMMLU+: from 15.49% to 42.18%, a 172% gain.
• Taiwan legal exam: from 25.68% to 31.26%, a 22% gain.
• MMLU: from 6.90% to 52.07%, a 654% gain (he described it verbally as "six times better").
• BFCL (Berkeley Function Calling Leaderboard, which evaluates a model's ability to call functions and tools; on verification, it is a public leaderboard maintained by the Gorilla team at UC Berkeley): their open 3B model scored 91 on one function-calling subtask, ahead of Llama-3.2-3B-Instruct at 84 and GPT-4o-mini at 87, and well ahead of Llama-3.1-8B-Instruct at 57.

One key warning: decide the problem before you collect the data

He stressed that the order of building a language model has completely reversed — you do not start by piling up data, you first decide clearly what problem you are solving and then work backwards to how the data should be collected. Fine-tune blindly without a clear problem definition and "all you'll receive is a stack of invoices, and no result".

He had given similar material at NVIDIA GTC Taipei the day before

He opened by asking whether anyone had attended NVIDIA GTC Taipei the previous day, admitting there would be some overlap. On verification, NVIDIA GTC Taipei 2025 ran on 21–22 May 2025 (TechNews, Zerone announcement), one to two days before this developer conference on 23 May — the dates line up.

3. Figures and cases

S1 hardware and data requirements: see the table above.

BFCL function-calling evaluation (speaker's own account):

Distillation results (speaker's own account): gains of 172%, 22% and 654% on TMMLU+, the Taiwan legal exam and MMLU respectively — see the section above.

Company scale (speaker's own account, partly filled in afterwards): APMIC was founded in 2017, so about eight years old at the time of this session. On checking, reporting indicates NVIDIA CEO Jensen Huang named APMIC six times across his 2024 and 2025 Taiwan talks, and APMIC even appeared on the backdrop of his 2025 Taipei keynote — which lines up directly with 林鉦育's "we finally got onto the backdrop" story the day before. In the same week's conference, standing on an NVIDIA backdrop had accidentally become a shared talking point in the fintech world. A further INSIDE Side Chat interview (published July 2025) fills in the context: APMIC joined NVIDIA's Inception startup programme in 2023 and has since appeared on the partner wall of Huang's COMPUTEX keynote two years running, which corroborates the six-mentions account.

4. Lines worth keeping

• "On-premises, the last thing you should think about is one very large model doing all sorts of different things."
• "All you'll receive is a stack of invoices, and no result."
• "Building a language model now basically starts from the back: decide your problem first, then decide how to collect your data."
• "Learn all your own rules into its head, and then it can just rig your colleague's new documents."
• "Think of us as a central kitchen, steaming different lunchboxes for our customers."

5. Tools and terms mentioned

• Knowledge distillation: transferring a large model's knowledge into a small one; the classic reference is Hinton et al.'s 2015 Distilling the Knowledge in a Neural Network.
• Test-time scaling: improving a model's performance by spending more compute at inference; the key paper here is s1: Simple test-time scaling.
• NVIDIA NeMo: the NVIDIA framework APMIC's fine-tuning and distillation offering is built on.
• CPT (continual pre-training) / CoT (chain of thought): the two training stages in the distillation flow on the slide.
• MCP (Model Context Protocol): the protocol APMIC's 3B model integrates, letting the model call external tools or functions.
• BFCL (Berkeley Function Calling Leaderboard): the public leaderboard for evaluating an LLM's function-calling ability.

6. Wider observations

The most notable thing here is not the impressive scores but the logic behind the name "S1" — a Taiwanese company took a paper's core idea (a small amount of high-quality data plus test-time compute) and turned it into a tiered enterprise product line in under three months, with the tier thresholds mapping almost directly onto the paper's own settings. That says knowledge distillation and test-time scaling are no longer just academic concepts but capabilities you can productise and sell by tier, fast. Set against 林鉦育's session next door about thinking through the use case before adopting AI in banking, this one addresses the other end of the problem: if an enterprise has already worked out its use case but is unsure whether to do RAG or fine-tuning, the decision framework here — can you live with looking it up every time, do you have enough labelled data — is a very practical starting point.

7. Sources