Direct answer: do not begin an AI equipment-design pilot with a promised productivity percentage. Begin with a frozen work package and measure how many declared engineering obligations are correctly closed, how much review/rework is required, which failures are caught and whether the native artifact survives the customer’s normal release process.
Define the unit of work
Hours alone are a weak denominator if two projects contain different numbers of parts, connections or exceptions. The pilot should count engineering obligations: required P&ID edges, approved components, interface assignments, placement constraints, routing tasks, mates and validation gates.
Each obligation receives a status such as auto-pass, pass with engineer review, fail-closed or cannot verify. This lets a manager distinguish throughput from correctness and makes the remaining manual work visible.
Freeze a baseline
Select comparable historical or parallel work under the same engineering rules. Record input quality, component-library completeness, number of obligations, engineer experience, review stages and change requests. Without these controls, a faster result may simply reflect an easier drawing or cleaner data.
Five KPI groups
1. Obligation closure
- share of in-scope obligations correctly realized;
- share requiring engineer correction;
- share blocked because evidence or rules were insufficient;
- false-pass count—an obligation marked complete that later fails review.
2. Cycle time
- elapsed time from frozen input to first reviewable assembly;
- active engineer time spent preparing inputs, reviewing evidence and correcting output;
- time to incorporate an approved requirement change.
3. Rework and review load
- parts, poses, mates or routes manually changed after generation;
- review comments by severity and root cause;
- number of review iterations before release;
- downstream defects attributable to an accepted AI result.
4. Evidence quality
- obligations with complete source-to-artifact trace;
- reproducibility under the same approved input and software versions;
- quality of failure explanations and missing-evidence requests;
- ability to recreate the verdict from stored facts.
5. Governance
- correct enforcement of role, approval and release boundaries;
- customer data retained only in the approved environment;
- versioned rules and model libraries;
- auditability of human overrides.
Why fail-closed events are not automatically bad
A refusal can prevent expensive rework if the underlying input is genuinely ambiguous. The metric should classify refusals: correct safety boundary, missing master data, unsupported case, solver limitation or avoidable system defect. Driving the refusal rate to zero can encourage unsafe guessing.
Calculate productivity only after correctness
Once the customer accepts the output quality, compare engineer hours and calendar time per equivalent obligation. Report the sample, scope, input quality and confidence limits. Separate preparation, generation, review and downstream correction. Do not combine a controlled fixture with a production project in one headline percentage.
A defensible management conclusion
A pilot succeeds when it produces a reviewable native assembly, closes a meaningful set of obligations with an acceptable false-pass rate, reduces total engineering effort after review/rework and leaves an evidence record the customer can audit. If any of those conditions is missing, the correct conclusion is “not yet proven for this scope.”
References
- NIST AI RMF 1.0 — trustworthiness and risk management across the AI lifecycle.
- W3C PROV-DM — provenance entities, activities, agents and derivations.
- SOLIDWORKS API 2026 — assembly operations available through IAssemblyDoc.
Limit: MST does not publish a universal productivity percentage. Efficiency must be measured against the customer’s own frozen baseline and accepted engineering result.
MST Engineering AI. “How to Measure an AI Equipment-Design Pilot Without Inventing an Efficiency Claim.” MST Engineering AI. Updated 2026-08-08. https://mst-us.ai/measure-ai-equipment-design-pilot-outcomes/
CUSTOMERS · PARTNERS · INVESTORS
Talk to the engineering team.
Bring a non-confidential P&ID question, pilot boundary or technical diligence request. Protected engineering material is exchanged only after identity, NDA and scope review.
Operated by MOORE SOLUTION TECHNOLOGY PTE. LTD. · Singapore UEN 202315239E