← Fintech GlossaryAI & LLM

Knowledge Distillation

Knowledge distillation is a model compression technique where a smaller, simpler model (the student) is trained to replicate the behavior of a larger, more complex model (the teacher). The student learns from the teacher's output probabilities, logits, or intermediate representations rather than from the raw training data alone. This process transfers the teacher's knowledge — including its nuanced understanding of patterns, edge cases, and decision boundaries — into a more compact and efficient model. Distillation can reduce model size by 40-90% while retaining 95-99% of the teacher's performance, making it one of the most effective techniques for deploying AI in resource-constrained environments. There are several variants of distillation: response-based distillation where the student mimics the teacher's final outputs, feature-based distillation where the student learns from the teacher's intermediate layer representations, and relation-based distillation where the student learns the relationships between different data points as captured by the teacher. The technique was first popularized by Geoffrey Hinton and his colleagues in their 2015 paper 'Distilling the Knowledge in a Neural Network,' and has since become a foundational approach in the field of model optimization. In practice, distillation involves training the student model on a combination of the original training data and the teacher's soft probability distributions, which contain richer information than hard labels. The temperature parameter controls how much the soft labels emphasize fine-grained differences between classes, with higher temperatures producing softer distributions that reveal more about the teacher's internal reasoning. Distillation is particularly valuable for deploying large language models in production environments where latency, memory, and computational cost are critical constraints. Financial institutions use distillation to create smaller, faster versions of powerful LLMs that can run on local infrastructure without sacrificing accuracy on domain-specific tasks. The technique also enables ensemble compression, where the knowledge from multiple specialized models can be combined into a single student model, providing a practical way to deploy diverse AI capabilities within a single inference pipeline. Additional variants include self-distillation where the model serves as its own teacher, online distillation where teacher and student are trained simultaneously, and progressive distillation where the student gradually increases in complexity during training. The choice of distillation approach depends on the specific requirements of the deployment scenario, including latency targets, hardware constraints, and accuracy thresholds.

In Financial Services

In financial services, knowledge distillation enables institutions to deploy advanced AI capabilities on local infrastructure while maintaining data sovereignty and regulatory compliance. Banks and asset managers often need to process sensitive financial data on-premises due to data residency requirements, but running large state-of-the-art models locally is computationally prohibitive. Distillation solves this by creating compact models that retain the essential knowledge of frontier models but require a fraction of the compute resources. For example, a distilled version of a 70-billion-parameter model can run on a single GPU server rather than a cluster, reducing inference costs by 80-90% while maintaining comparable accuracy on financial tasks like document classification, entity extraction, and sentiment analysis. Distillation also enables real-time applications where latency is critical, such as high-frequency trading signal processing, real-time fraud detection scoring, and live customer interaction systems. The technique is particularly valuable for financial institutions operating in markets with limited cloud access or strict data localization laws, as it allows them to run sophisticated AI models entirely within their own infrastructure. Additionally, distillation supports model risk management by creating simpler, more interpretable models that are easier to validate and audit compared to their larger counterparts. The distilled model's decision boundaries are often more transparent, making it easier to explain to regulators why certain predictions were made, which is a key requirement under SR 11-7 and other model risk management guidelines. Distillation also facilitates the deployment of AI across multiple business units without multiplying infrastructure costs, as a single distilled model can be replicated across different departments while a larger model would require dedicated GPU clusters for each deployment. Financial institutions are increasingly using distillation to create domain-specific models that excel at particular financial tasks while remaining small enough for widespread deployment. This approach is particularly effective for use cases like trade surveillance, where models must process high volumes of transactions in real-time, and for compliance monitoring, where models need to scan large volumes of communications efficiently.

Real-World Example

A large European bank deploys a distilled version of a frontier LLM for its customer-facing document analysis platform. The original model, a 175-billion-parameter transformer, requires 8 A100 GPUs for inference and costs $0.03 per query. After distillation, the 7-billion-parameter student model runs on a single CPU server, costs $0.001 per query, and achieves 97% of the teacher's accuracy on financial document understanding tasks. The bank processes 2 million queries per month, reducing inference costs from $60,000 to $2,000 per month while maintaining regulatory compliance by keeping all data processing on-premises. The distilled model also reduces latency from 2.5 seconds to 150 milliseconds, enabling real-time document analysis during customer calls. The bank's model risk team validates the distilled model against the original using a test suite of 10,000 financial documents, confirming that the performance gap is within acceptable thresholds for all use cases. The success of this deployment leads the bank to distill models for three additional use cases: trade surveillance, compliance monitoring, and credit risk assessment, each achieving similar cost reductions while maintaining performance within acceptable ranges. The bank ultimately deploys five distilled models across its organization, achieving a 90% reduction in total AI infrastructure costs while expanding AI capabilities to departments that previously lacked access to such technology.

Why It Matters for Finance

Knowledge distillation is a critical enabler for financial AI adoption because it bridges the gap between the capabilities of frontier AI models and the practical constraints of financial institutions. Most banks and asset managers cannot run 100-billion-parameter models on-premises due to infrastructure costs, power requirements, and cooling constraints. Distillation makes it feasible to deploy AI capabilities that would otherwise be accessible only through cloud APIs, which may violate data residency requirements. As financial institutions increasingly adopt AI for mission-critical applications, the ability to run efficient, distilled models on local infrastructure becomes a competitive advantage. Institutions that master distillation can deploy AI more broadly across their organizations, achieve lower latency for real-time applications, and maintain full control over their data. The technique also supports sustainability goals by reducing the energy consumption of AI inference, which is increasingly important for financial institutions with ESG commitments. Furthermore, distillation enables the democratization of AI within financial organizations, allowing smaller teams and business units to benefit from advanced AI capabilities without requiring dedicated GPU infrastructure or specialized ML engineering support. The ongoing trend toward smaller, more efficient models suggests that distillation will play an increasingly important role in financial AI strategy, enabling institutions to do more with less while maintaining the highest standards of accuracy and compliance.

Explore in Finatune

Phi-4 (Microsoft)Ollama

Frequently Asked Questions

What is model distillation in AI?

Model distillation is a technique where a smaller student model is trained to replicate the behavior of a larger teacher model. The student learns from the teacher's probability distributions rather than just raw training data, enabling it to achieve comparable performance with significantly fewer parameters. This reduces model size by 40-90% while retaining 95-99% of the teacher's accuracy.

How does distillation help financial institutions deploy smaller AI models?

Distillation enables financial institutions to run sophisticated AI models on local infrastructure rather than cloud APIs, reducing inference costs by 80-90% and latency by 10-20x. This is critical for meeting data residency requirements, regulatory compliance, and real-time processing needs. Banks can deploy distilled models on single CPU servers instead of GPU clusters.

Which distilled models are best for financial services?

Microsoft's Phi-4 family offers distilled models with strong performance on financial tasks, while open-source models distilled from Llama and Mistral provide flexibility for customization. The best choice depends on your specific use case, infrastructure constraints, and performance requirements. Financial institutions should evaluate distilled models against a benchmark suite of domain-specific tasks.

← Previous Term: In-Context Learning

Next Term: Knowledge Graph →

View All Fintech Terms →