| Description |
1 online resource (viii, 232 pages) : illustrations (chiefly color) |
| Series |
Studies in computational intelligence ; volume 1100 |
|
Studies in computational intelligence ; v. 1100.
|
| Contents |
Intro -- Preface -- Contents -- Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillation -- 1 Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillation -- 1.1 Response-Based Knowledge Distillation -- 1.2 Feature-Based Knowledge Distillation -- 1.3 Relation-Based Knowledge Distillation -- 2 Distillation Schemes -- 2.1 Offline Knowledge Distillation -- 2.2 Online Knowledge Distillation -- 2.3 Self-knowledge Distillation -- 2.4 Comprehensive Comparison -- 3 Distillation Algorithms -- 3.1 Multi-teacher Distillation |
|
3.2 Cross-Modal Distillation -- 3.3 Attention-Based Distillation -- 3.4 Data-Free Distillation -- 3.5 Adversarial Distillation -- 4 Conclusion -- References -- A Geometric Perspective on Feature-Based Distillation -- 1 Introduction -- 2 Prior Art on Feature-Based Knowledge Distillation -- 2.1 Definitions -- 2.2 Related Work -- 3 Geometric Considerations on FKD -- 3.1 Local Manifolds and FKD -- 3.2 Manifold-Manifold Distance Functions -- 3.3 Interpretation of Graph Reordering as a Tool Measuring Similarity -- 4 Formulating Geometric FKD Loss Functions -- 4.1 Neighboring Pattern Loss |
|
4.2 Affinity Contrast Loss -- 5 Experimental Verification -- 5.1 Materials and Methods -- 5.2 Knowledge Distillation from Large Teacher to Small Student Models -- 5.3 Comparison with Vanilla Knowledge Distillation -- 5.4 Knowledge Distillation Between Large Models -- 5.5 Effects of Neighborhood -- 6 Case Study: Geometric FKD in Data-Free Knowledge Transfer Between Architectures. An Application in Offline Signature Verification -- 6.1 Problem Formulation -- 6.2 Experimental Setup -- 6.3 Results -- 7 Discussion -- 8 Conclusions -- References -- Knowledge Distillation Across Vision and Language |
|
1 Introduction -- 2 Vision Language Learning and Contrastive Distillation -- 2.1 Vision and Language Representation Learning -- 2.2 Contrastive Learning and Knowledge Distillation -- 2.3 Contrastive Distillation for Self-supervised Learning -- 3 Contrastive Distillation for Vision Language Representation Learning -- 3.1 DistillVLM -- 3.2 Attention Distribution Distillation -- 3.3 Hidden Representation Distillation -- 3.4 Classification Distillation -- 4 Experiments -- 4.1 Datasets -- 4.2 Implementation Details Visual Representation -- 4.3 VL Pre-training and Distillation |
|
4.4 Transferring to Downstream Tasks -- 4.5 Experimental Results -- 4.6 Distillation over Different Losses -- 4.7 Different Distillation Strategies -- 4.8 Is VL Distillation Data Efficient? -- 4.9 Results for Captioning -- 5 VL Distillation on Unified One-Stage Architecture -- 5.1 One-Stage VL Architecture -- 5.2 VL Distillation on One-Stage Architecture -- 6 Conclusion and Future Works -- References -- Knowledge Distillation in Granular Fuzzy Models by Solving Fuzzy Relation Equations -- 1 Introduction -- 2 Related Works -- 2.1 Knowledge Granularity in Transfer Learning |
| Summary |
The book provides a timely coverage of the paradigm of knowledge distillationan efficient way of model compression. Knowledge distillation is positioned in a general setting of transfer learning, which effectively learns a lightweight student model from a large teacher model. The book covers a variety of training schemes, teacherstudent architectures, and distillation algorithms. The book covers a wealth of topics including recent developments in vision and language learning, relational architectures, multi-task learning, and representative applications to image processing, computer vision, edge intelligence, and autonomous systems. The book is of relevance to a broad audience including researchers and practitioners active in the area of machine learning and pursuing fundamental and applied research in the area of advanced learning paradigms |
| Notes |
Includes index |
|
Online resource; title from PDF title page (SpringerLink, viewed June 22, 2023) |
| Subject |
Artificial intelligence.
|
|
Information technology -- Technological innovations
|
|
Knowledge management -- Data processing
|
|
artificial intelligence.
|
|
Artificial intelligence
|
|
Information technology -- Technological innovations
|
|
Knowledge management -- Data processing
|
| Form |
Electronic book
|
| Author |
Pedrycz, Witold, 1953- editor.
|
|
Chen, Shyi-Ming, editor.
|
| ISBN |
9783031320958 |
|
3031320956 |
|
