李慕約：可以知道自己可以怎麼樣馬上入門 AI Agent

1. In one sentence

AI agents will end up better at using tools than people are, so humans need to play the client well — and a good client knows how to sign work off and how to write the brief.

2. The session

The difference between an AI agent and ChatGPT

He started by defining the session's central terms: with ChatGPT you speak and it speaks back; with an AI agent you speak and it goes and does something. Doing something requires two things: the model has to be smart enough (in his view only with the reasoning model line — ChatGPT's o1, o3 — does it truly qualify), and the model has to be able to use tools. He also cautioned that although everyone calls this the year of the AI agent, the AI agents actually usable today can only handle a few tools, the most widespread being search — that is, the deep research features vendors have been rolling out.

A visual reasoning demonstration: guessing the right location from a street view

He demonstrated ChatGPT o3's visual reasoning live: pull an arbitrary street view photo off a map, hand it to the model, and ask only "where is this photo". From clues in the frame — the utility pole marked Taiwan Power Company, the ornamental palms, hill terrain found only around New Taipei and Yilan — the model inferred the border of Wulai, Pinglin and Shiding, exactly right. He made a point of showing the reasoning trace: the model decides for itself that there may be text in the frame, and proactively zooms in and converts to black and white to read a blurry number. He stressed this differs from a traditional workflow with predefined steps: the model decides autonomously whether to reach for a given tool rather than following a hard-coded sequence. He also mentioned the photo-dating game from the opening: he gave all five photos to ChatGPT o3 and scored in the top two per cent of all players, far above what he managed playing it himself.

Signing off: how do you know whether the AI agent did well

He used a bill-splitting example to raise the importance of acceptance: hand the AI a screenshot listing purchases and ask it to turn them into a table. The first test worked, so he assumed the task was easy — only to discover later that the success rate on that recognition task is only about fifty per cent. From that he distilled three questions: how many times do you need to check, can the testing be automated, and are you even capable of checking it yourself.

He used Bayes' theorem to explain how many checks are enough: if something succeeds three times in a row, the fourth has about an eighty per cent chance; ten in a row, about ninety per cent; and to reach ninety-nine per cent confidence you need roughly a hundred consecutive successes beforehand. His conclusion: a human gets impatient after about three, and can never really verify at that scale.

His answer is to let the AI set its own questions, answer them and have another AI check, forming a closed loop: give the AI a small sample of data and ask it to fabricate more test data in the same shape while keeping each item's correct answer; let the AI agent under test process these newly fabricated screenshots; then use another AI to compare the recognition results against the correct answers, estimating overall accuracy in bulk and automatically.

His third point, "sometimes there simply is no correct answer", came from a deep research trial: he asked the AI to research which drugstore products to buy in Japan, and got back a list of highly rated, good-value, high-volume items. He initially scored that answer highly himself, but the heavy user who actually raised the request and knows the field gave it sixty out of a hundred, for reasons including: what you buy should be *expensive* rather than good value (the point of going abroad is buying what you cannot get at home); it should draw on newer, more specialist sources (he mentioned a Japanese magazine called "LDK", known for being scathing, which is first to report genuinely good products) rather than simply whatever has the most online buzz; searching in Japanese gets closer to local information than searching in Chinese; and the scope was set far too wide — "drugstore products" should have been narrowed to something as specific as lipstick.

Writing the brief: how to put work out to an AI agent

He organised "how to instruct an AI agent" into four points:

• The heavy user's superpower: if you know a field well, the way you read material differs from an ordinary person's and even from the AI's, and that perspective gets amplified by the agent. Equally important is knowing the right keywords (like "LDK" above), knowing which language to search in, and knowing how narrow to make the scope. He demonstrated a small trick about language choice: searching for Tokyo ramen in Chinese mostly returns content written by Taiwanese bloggers, but specifying "search in Japanese, present the results in Chinese" switches the sources to recommendations from Japanese locals.
• The reality and the imagination of human-machine collaboration: he cautioned that judgements trivial for a human (reading a table at a glance) can be hard for an AI, and vice versa; understanding that gap is what makes collaboration go smoothly. He gave two "imagination" examples: a press release that got a brand name and his own name wrong in three places, which he fed into deep research so it could correct the spellings from context; and a travel piece he was half way through, where he left the details he could not remember as parentheses with question marks (was that year a La Niña year, how much snow fell in a given place that week), then handed the whole piece with its question marks to deep research to verify and fill in afterwards.
• Say it clearly: he recommends three techniques — ask ChatGPT to rewrite your own instruction more clearly first; instruct the AI to ask you questions before it starts work, to clear up ambiguity (his example: if you only say "find me cheap miso", the AI should ask back whether "cheap" means the lowest total price or the lowest price per unit); and discuss the requirement with the AI for a round first, then ask it to rewrite the final instruction from that discussion.
• The client's strategy: including putting the same task out to several AI agents at once, running each twice and picking the best answer (the cost of a single run is now low enough); if the yield on something is persistently poor, simply do not have the agent do it (a view he credits to Andrew [吳剛志] from the developer conference); and his own formulation, "only wait, never teach" — if the AI cannot do something well today, parking it and waiting for a model update is more efficient than spending a lot of time teaching it. He drew an analogy to the phrase that has circulated online in recent years, "in the adult world you filter, you don't educate", rewriting it as "in the AI world, you wait, you don't educate".

3. Figures and cases

4. Lines worth keeping

• "AI agents will end up better at using tools than people are; people need to be good clients, and a good client knows how to write a brief and how to sign off."
• "There aren't that many AI agents you can use today — they can only use a handful of tools."
• "In the AI world, you wait, you don't educate."

5. Tools and terms mentioned

ChatGPT o1 and o3, ChatGPT deep research, Google deep research, Grok, Perplexity, Felo Search, Bayes' theorem, LDK (the scathing Japanese review magazine), AmazingTalker, 艾立運能 (both brand and personal names from the press release proofreading example).

6. Wider observations

The most striking thing here is the "client" metaphor, translating an abstract question about AI capability straight into a commercial relationship everyone understands: acceptance is hard because confidence needs a great deal of repeated verification to build, and briefing is hard because you have to know the field before you can write the spec. That is the same logic as Wisely Chen's demonstration in the previous session of deliberately leaving mistakes to see whether the agent corrects them — one is about how you assess an AI, the other about how you give an AI room to be wrong. Verification confirms that the concluding line was indeed quoted by the press, and the headline is almost a verbatim lift of his own words — one of the few cases here of a line landing in the room and then genuinely becoming a news headline.

7. Sources