吳剛志：從Prompt到Product，掌控AI APP的正確率

1. In one sentence

吳剛志 used a real 91APP project — auto-filling product specification fields — to show how an architect uses two quantitative metrics, accuracy and coverage, to take a team through dozens of prompt improvement cycles, moving a feature from 70% accuracy (looks usable, is not) to a shippable state at 90% accuracy, 50% coverage and able to keep iterating.

2. The session

Where 91APP uses AI: the blue mindset and the red one

He opened by drawing the two quadrants of 91APP's AI use: on one side, using AI to develop products and improve process and efficiency — strengthening the development team's own output, benefiting people in the software industry; on the other, building AI product services — strengthening the product's own capability for consumers and clients, benefiting people outside software. The slide marks these as the blue "prompt" mindset and the red "product" mindset, and the contrast is sharp: in the blue setting you write your own prompt, hope the AI gives you an answer you had not thought of, and review its response yourself, repeating on the order of a hundred times, so case-by-case review is possible; in the red setting the client may not understand prompts at all and sees only the result, expects predictable behaviour, does not want to be asked to review a great deal — and that prompt may run on the order of a million times, making case-by-case checking impossible. That split, prompts internally and products externally, is the starting point for every evaluation method that follows.

The case: a small module where under 90% accuracy is no better than guessing

The case on the slide is not a fashionable conversational agent but "a not-very-large small module": using a product's description to fill in the client's specification and category options automatically. The background is that listing on a storefront requires filling in many fields the database never had, and 91APP wanted generative AI to do it in bulk on the client's behalf, raising their operational efficiency and saving labour. The slide stresses this is "a core feature of a core product", because product listing is the main feature every client uses; it also gives the scale — listing one product means picking about twenty values from roughly two hundred options.

The slide then lays out a real input and output example: the description text of a "romantic lace pocket satin A-line skirt" (a long run of colour, size, material and other information), with a system prompt asking the AI to fill the specification according to three field constraints — single choice, multiple choice, free text — and leave it blank when it cannot answer. Three practices in this stretch are marked "(wrong)" in self-deprecation: assuming the AI understands everything anyway and feeding it the raw format; having the AI generate the original import format for convenience; and a prompt that spends its effort on the *format* engineers care about instead of the *principles* the client cares about. The slide makes the key point: the genuinely important part of this prompt takes up about one sentence, and the remaining ninety per cent of the tokens are solving the engineer's own problem (making the downstream code easier), not the client's — and this prompt will be run on the order of a million times, so every single run pays extra tokens for the engineer's convenience. Checking afterwards, in INSIDE's interview about this session 吳剛志 put it even more bluntly: "A prompt inside a consumer product might be used a million times a year — do you want to spend that many tokens? And possibly get a different result each time?" — which echoes the slide's argument.

Why accuracy matters more than coverage

The slide poses a multiple choice: which result would the client prefer? A, every specification option gets an answer but accuracy is only about 70%; or B, only 50% of products get an answer at over 90% accuracy, with the other 30% having no answer but a hint telling the user what product information is missing. The conclusion is that B helps the client more — because with A, however well the UI is designed, the client still has to check every single item, so no time is saved; with B, provided accuracy is high enough, experienced and inexperienced users alike can trust it, saving 50% of the work straight away, and the 30% with hints needed data filling in regardless, so that is another half saved, another 15% — ideally saving the client over half their time. Two metrics are defined explicitly here: coverage is the number of specifications the AI answered divided by the total number of specifications; accuracy is the number the AI got right divided by the number it answered. The target: accuracy must clear the ninety per cent threshold first, and only then is it meaningful to chase coverage.

The slide then shows how these two are quantified: a demonstration table of products A, B and C, each listing five specification options (red, orange, yellow, green, blue) against an expected answer and the AI's answer, marking each cell as correct, wrong or abstained — turning abstract accuracy and coverage into a data table you can check line by line.

After the first scoring run: 52.3% accuracy, 87% coverage

The slide shows a real set of early evaluation numbers: 52.3% accuracy and 87% coverage, and then poses the question "so what do I change next", which introduces the session's second theme: how to work back from a score to a concrete improvement direction.

The improvement work splits into three steps. First, look at the distribution of the data, guess which ranges are going wrong, and adjust the prompt broadly along with threshold values on parameters such as a confidence score. Second, lay out the per-product results and go through them one by one, distilling handling principles into prompt corrections — and where you hit a special case, exclude it and handle it with a rulebase rather than forcing it into the prompt. Third, establish a collaboration mechanism between the technical expert (the architect himself) and the domain expert (the slide calls them the "big PO"): the architect and the big PO set the handling principles early on, and later hand over to the tech lead and the product owner to keep optimising.

The slide sorts the remaining specifications into three classes with different handling: handled by rules — anything with a shallow rule to follow, and anything unsuited to AI, gets excluded up front, and the more you exclude the better; prompting the client for more information (Hint) — blocks that cannot be answered for whatever reason get a clear annotation; and suggesting an answer to the client (Autofill) — blocks the AI can answer, where the goal is to drive the number of wrong answers as low as possible and shift the "wrong" bucket elsewhere. A judgement principle appears here too: for the client, a wrong answer costs more than no answer, so a hard choice (a single-choice field with several candidate answers, say) should simply be abandoned; badly designed or exceptional attributes should be handled by the rulebase; and an unreasonable reference dataset should be fixed at the dataset itself rather than contorting the prompt around it.

The slide also mentions the improvement cycle ran about twenty rounds: the architect accompanied the team for rounds one to ten, standing up the execution environment and process; then rounds eleven to thirteen together with the big PO, establishing the decision flow for judging problems and choosing improvement directions; and the remaining dozen or so rounds the team ran themselves, without the architect present.

Engineering-side acceleration: the right format, the right tool, the right report

The slide names three improvements made from an engineering angle, each with a concrete figure: improving the AI's own results by restructuring the input and output, which added five percentage points of accuracy; improving engineers' execution efficiency by cutting the edit-and-score cycle from four hours to ten minutes; and improving the product owner's and domain expert's efficiency by cutting review-and-decide time from eight hours to thirty minutes.

For the input and output restructuring the slide gives a real comparison: the input was originally one long unsegmented run of product data text, and became Markdown-segmented field markers (headings such as #ProductID, #SalePageName, #ProductDescription separating the fields) so the AI can better understand what the text means; the output went from one line of plain text description to a structured JSON-plus-schema format, where each specification field carries not only the filled value but also sure (is it confident) and reason (the reasoning), making the format both more precise and easier for code to consume. This change added five percentage points of accuracy.

The slide names the tool actually used to accelerate the cycle: Dify, for "accelerating the development and validation of AI applications", letting the team validate results immediately after each change to LLM parameters and prompts and compressing a four-hour validation into ten minutes. The slide also carries a screenshot of a reference for evaluation methodology, credited to Anthropic's documentation (the build-with-claude/develop-tests chapter); the next page links directly to an ihower blog post as a second reference for evaluation method.

For the reporting, the slide lists three outputs: a summary table (CSV) computing each evaluation's metric values with Excel-like formulas so the product owner can read the result quickly; a per-item scoring report (Markdown) where each cell maps to correct answer, AI answer and annotation, linking back to the original product information and specification definition so the product owner can get to the bottom of anything; and a data flow diagram drawn in Mermaid, counting where the whole batch finally landed. The numbers on that flow diagram are very specific: 393 items entered effective evaluation, with 68 excluded; of the 393, the AI answered 231, abstained on 150 and errored on 12; of the 231 answered, 192 were confident answers (trusted) and 39 were not; and of the 192 confident answers, 167 were finally confirmed correct and 25 wrong. That flow diagram lays out every raw count behind the accuracy and coverage figures for reconciliation.

Conclusion: for the client this is an assistive feature, not a place to show off AI

The slide closes by stressing that not every AI agent has to be a conversational application; done right, a "behind the scenes" feature like this helps clients just as much. What the client actually expects has to be converted into quantitative metrics: efficiency equals the reduction in data the client has to enter by hand plus the reduction in data the client has to check. The slide writes the final goal as a set of numbers: management expects over 90% accuracy, over 90% coverage, and 90% of the client's time saved. Checking afterwards, INSIDE's interview also writes the project's final outcome as over 90% accuracy, over 90% coverage and roughly 90% time saved, matching the slide's target figures; in the interview 吳剛志 also reached a similar conclusion — "this is an assistive feature, and you can't approach it with a mindset of showing off AI" — essentially the same sentence as the title of that slide.

The slide ends with four internal lessons, each with a footnote: Scope (define what you handle) — your prompt will run over a million times, and handling a well-defined problem in code beats handing it to the AI; Evaluate (control correctness) — convert the goal into a quantifiable metric, build an evaluation mechanism, and separately develop your ability to evaluate multiple-choice-type and open-question-type outputs; Optimize the improvement cycle (make the loop faster) — how fast you go from evaluation to improvement directly caps how much you can optimise, and where necessary putting development effort into building the evaluation loop matters more than building the feature itself; Prompt engineering (learn to use LLMs properly) — engineers should master the LLM API properly, with JSON mode, function calling and prompt engineering all necessary skills.

3. Figures and cases

• Scale of the case: listing one product means picking about 20 specification values from roughly 200 options.
• Initial baseline: 52.3% accuracy, 87% coverage.
• The multiple-choice experiment: option A answers every item at about 70% accuracy; option B answers only 50% of items at over 90% accuracy, with hints on the other 30%.
• Engineering improvements: restructuring input and output raised accuracy by 5 percentage points; engineers' scoring-run validation went from 4 hours to 10 minutes; the product owner's review-and-decide time went from 8 hours to 30 minutes.
• Scale of the improvement cycle: the architect led rounds 1–10, worked with the big PO on rounds 11–13, and the team completed the remaining dozen or so alone.
• Raw counts in the data flow: 393 items entered evaluation, 68 excluded; the AI answered 231, abstained on 150, errored on 12; of those answered, 192 were confident and 39 were not; of the 192 confident, 167 were correct and 25 wrong.
• Target thresholds: accuracy must reach 90% before coverage is considered meaningful; the eventual expectation is over 90% accuracy, over 90% coverage and 90% of the client's time saved (the speaker's stated target, no external source).
• Prompt execution scale: internal use repeats on the order of a hundred times and can be reviewed case by case; productised, it runs over a million times and case-by-case review is impossible.

4. Lines worth keeping

All of these are slogans or key lines written directly on the slides:

• "Only features that output precisely, whose accuracy can be quantified, and that generative AI genuinely strengthens, get to ship."
• "For the client this is an assistive feature, not a place to show off AI."
• "Efficiency = less data the client has to enter by hand + less data the client has to check."
• "Your prompt will be run 1,000,000+ times."
• "Convert the goal into a quantifiable metric, and build an evaluation mechanism."

Checking afterwards, INSIDE's interview about this session quotes two further lines from him, listed here with the link and marked separately from the slide slogans above:

• "A prompt inside a consumer product might be used a million times a year — do you want to spend that many tokens? And possibly get a different result each time?"
• "This is an assistive feature, and you can't approach it with a mindset of showing off AI."

(INSIDE interview: how 91APP's 吳剛志 builds high-accuracy AI features)

5. Tools and terms mentioned

• Dify: named on the slide as the platform used to accelerate AI application development and validation; officially positioned as a visual agent and RAG workflow development platform, available as cloud, self-hosted enterprise and open source. Checked afterwards: Dify website.
• Accuracy / coverage: the two evaluation metrics defined on the slide. Coverage = specifications the AI answered ÷ total specifications; accuracy = specifications the AI got right ÷ specifications the AI answered.
• Rulebase: the mechanism on the slide for handling special cases and badly designed attributes, on the principle that special cases should not be forced into the prompt.
• JSON mode / function calling: listed on the slide as LLM API techniques the architect believes engineers must master.
• Mermaid: the syntax used on the slide to draw the data flow statistics diagram.
• Checked afterwards: one of the evaluation methodology references cited on the slide is the "define success criteria and build evaluations" chapter of Anthropic's documentation, covering ways of defining success criteria such as task fidelity and consistency, along with evaluation methods including exact match, cosine similarity, ROUGE-L, Likert scales and binary classification. Anthropic documentation.
• Checked afterwards: the other reference cited on the slide is a blog post by ihower (張文鈿, the community handle of the fourth speaker on the same day), "eval-driven development: an answer to the non-determinism of generative AI software", whose core idea is to build the evaluation mechanism before developing the LLM application, paired with LLMOps to keep collecting live data and iterating. The original ihower post.

6. Wider observations

The core value of this session is in the concrete numbers of sections 2 and 3: 52.3% to 90%, 87% to 50%, four hours to ten minutes, eight hours to thirty minutes — each pair mapping to a specific improvement action (add rules, change the format, change the tool). Compared with the many talks that offer vision and no figures, this one is close to laying the whole eval-driven development manual open in front of you. Another session at the following day's main conference (李昆謀's) uses exactly the same definitions of accuracy and coverage and mentions the same product-specification auto-fill case, so read together the two give you the product-side and architecture-side views of the same 91APP project, filling in each other's gaps.

What is a shame is that the slide never says how large the evaluation dataset behind that 91% accuracy was (hundreds of products or thousands), and it never makes clear exactly what role Dify played in this project — prompt version management, batch evaluation runs, or both. The slide gives only the "four hours to ten minutes" result, leaving the detail of how it connects up blank. The blue-and-red contrast, though, is the best frame of the whole session: it separates "using AI internally" from "an AI product built for clients" — two things routinely conflated — by asking whether you can review the output and how many times it repeats. Worth writing down and reusing on other project evaluations.

Incidentally, 91APP effectively had three speakers across the two days: 吳剛志 in this session, 陳俊毅 in the same day's seventh slot (written up separately), and 李昆謀 at the main conference the following day. Checking afterwards, 李昆謀's own write-up mentions that "two colleagues from 91APP also spoke at the developer conference", which matches 吳剛志 (Andrew) and 陳俊毅 (Levi) (happylee.blog). As for whether 91APP was a formal sponsor of this conference, no official 2025 named sponsor list could be found to corroborate it, so no ruling is made on that here.

7. Sources