The Model Did It - Understanding the What, but not the Why

The growing use of AI/ML models in software introduces a peculiar challenge in patent litigation: the model may reveal what it does, without necessarily revealing why it does it.

In conventional software, an expert can often trace source code, follow conditional logic, and identify the instructions responsible for a particular outcome. Machine learning is different. A trained model derives its behavior from training data and learned parameters. Its outputs may be identifiable and reproducible, but the underlying decision-making process may not translate into a sequence of human-readable instructions.

This creates a problem during patent source code review. An expert may identify where a model is invoked, examine its inputs and outputs, and understand how the application uses its results. Yet the actual functionality may reside within the trained model, rather than in the surrounding source code. Reading the code alone may therefore reveal what the system does, but not explain how or why the model arrives at a particular result.

So where does the missing explanation come from?

Source code review should be supplemented with targeted discovery directed at the model itself and the people responsible for its development and deployment.

1. Interrogatories: Establish the model's provenance and purpose

Seek identification of:

• Every AI/ML model used in the accused functionality, including model name, version, architecture, and whether it was developed internally or obtained from a third party.

• The training datasets, their sources, and any preprocessing, labeling, or filtering performed.

• The model's intended purpose, training objectives, loss functions, and optimization criteria.

• Any fine-tuning, retraining, or customization performed by the defendant.

• The inputs supplied to the model, the outputs it generates, and how those outputs are consumed by the accused application.

2. Requests for production: Obtain the technical artifacts

Request, where relevant and proportionate:

• Model cards, architecture diagrams, technical specifications, and API documentation.

• Training scripts, configuration files, hyperparameters, and model evaluation reports.

• Model weights or checkpoints and version histories.

• Relevant datasets or representative samples, together with data dictionaries and labeling guidelines.

• Inference logs, input/output traces, and model monitoring records.

• Internal engineering documents, design reviews, and communications discussing model selection, training, limitations, and behavior.

3. Depositions: Question the people who understand the model

Identify and depose the engineers, data scientists, and product architects responsible for the relevant model. Questions should include:

• Why was this particular model or architecture selected?

• What patterns or features was the model trained to recognize?

• What happens internally between receiving an input and generating an output?

• Which inputs have the greatest influence on the output?

• How does the model behave when inputs are changed or removed?

• What happens when the model encounters unfamiliar or unexpected inputs?

• How are confidence scores, thresholds, and model outputs used by the application?

• What changes were made between model versions, and why?

The objective is to obtain an explanation of the model's design and operation that may not be apparent from reading its source code.

4. Public technical documentation: Establish the scientific context

Where the defendant uses an established third-party model, examine the model provider's technical papers, model cards, architecture documentation, patents, API specifications, and published research. These sources may explain the underlying architecture and known behavior. However, they should be treated as technical context, not proof that the defendant's particular implementation operates identically.

5. Controlled testing: Observe what the model actually does

Where access permits, conduct controlled experiments using representative inputs. Change one input at a time, introduce edge cases, compare outputs across model versions, and examine the effects of removing or modifying particular features. This can help establish functional behavior, identify dependencies, and test whether the model performs operations relevant to specific patent limitations.

Ultimately, the explanation of an AI system may not reside in its source code. It may be distributed across the model, its training process, its documentation, and the knowledge of the people who developed and deployed it.

For patent litigators and technical experts, this suggests a broader approach: source code review should establish the implementation; discovery should establish its design and provenance; and technical testing should help establish its actual behavior.

In the age of machine learning, understanding the accused technology may require looking well beyond the code.