Data Leakage and Model Memorisation

Modern large language models and deep neural networks have a well-documented tendency to memorise specific examples from their training data. This is called model memorisation, and it creates a direct path from your model to your training data — even without any adversarial activity.

What Is Model Memorisation?

When a model is trained, it is supposed to learn general patterns, not specific examples. But in practice, especially with large models and small datasets, models often retain verbatim fragments of training data that can be extracted by querying the model with the right prompts.

Assessing Your Exposure

• ◆What data did you fine-tune your LLMs on? Does it include PII, commercially sensitive content, or privileged information?
• ◆Have you tested whether your deployed models can be prompted to reproduce training content?
• ◆Do your AI acceptable use policies address what data employees may use to fine-tune models?
• ◆If you use a third-party AI provider, can you verify what your data is trained on?

Minimising Memorisation Risk

• ◆Don't fine-tune on sensitive data unless necessary — Pre-trained general models with context injection (RAG) are often a safer alternative
• ◆Differential privacy during fine-tuning — Provides mathematical guarantees that individual records cannot be extracted
• ◆Regular red-team testing — Explicitly attempt to extract training data using prompt injection and other techniques before deployment
• ◆Data minimisation — Train on the minimum data needed; data you don't train on cannot be memorised