Explainability is an engineering requirement, not a compliance afterthought

The retrofit problem

There is a specific and expensive mistake being repeated across enterprise AI programmes right now. A model is built to optimise an outcome. It performs well. It goes into production. Some months later a regulator, an auditor or a customer asks why a particular decision was made, and the organisation discovers that it cannot answer.

At that point there are two options, both bad. Rebuild the system with interpretability designed in, having already paid for the first version. Or bolt on a post hoc explanation layer that approximates the model's reasoning without actually reproducing it, which satisfies a checklist and satisfies nothing else.

The cost differential between designing for explainability at the outset and retrofitting it under pressure is not marginal. It is usually the difference between a design constraint and a rewrite.

What explainability actually requires

The term is used loosely enough to be nearly meaningless, so it is worth being specific. In an enterprise deployment it generally means four separable things.

• Feature attribution. Which inputs moved this particular decision, and by how much.
• Counterfactual reasoning. What would have needed to differ for the outcome to change. This is the form most useful to the person affected, because it is actionable.
• Provenance. Which data, which model version, which point in time. Without this, reconstruction is impossible regardless of how interpretable the model architecture is.
• Population level behaviour. How the model performs across different groups, which is where disparate impact becomes visible or stays hidden.

The third of these is the one most often missed, and it is the one that is genuinely impossible to add later. If you did not log which model version and which input state produced a decision at the time, that information does not exist anywhere and no amount of subsequent engineering will recover it.

A model you cannot audit is not a model you own. It is a dependency you are hoping nobody asks about.

The regional dimension

Regulatory frameworks governing automated decision making are tightening across ASEAN at different speeds and in different directions. Some jurisdictions are moving on data residency first, others on sector specific rules for financial services, others on general purpose AI provisions modelled loosely on European approaches.

For an organisation operating across several of these markets, this creates a practical architecture question. A system built to explain itself is largely portable across divergent frameworks, because most of them are asking a version of the same question. A system built as an opaque optimiser needs separate remediation in each jurisdiction that moves, and it will need it on the regulator's timetable rather than the engineering team's.

That is a commercial argument rather than an ethical one, and it is worth making explicitly, because the ethical argument alone has a poor track record of surviving budget review.

Where this connects to the longer argument

The position Ivan Teh has argued consistently in public forums is that technology should be shaped around human requirements rather than the reverse. Explainability is that argument applied to machine learning specifically. A system whose reasoning cannot be reconstructed has, in a precise sense, made the human subordinate: the organisation is acting on an output it cannot interrogate.

There is also a straightforward adoption case. Internal users do not act on recommendations they do not understand and cannot defend. A credit officer who cannot explain a decline to a customer will override the model. A clinician who cannot see the reasoning will ignore the flag. The interpretability layer is not only a governance mechanism. It is frequently the difference between a deployed system and a used one.

The uncomfortable trade off

It should be said plainly that this is not free. In some problem classes the most accurate available model is also the least interpretable, and constraining architecture for explainability costs measurable performance.

The honest position is that this trade off is real and that it should be made deliberately rather than by default. For a recommendation engine suggesting products, opacity may be entirely acceptable. For a system influencing whether someone receives a loan or a diagnosis, a few points of accuracy is a poor exchange for the inability to explain a decision to the person it was made about, or to the regulator who eventually asks.