Membership and Memorization in LLM Knowledge Distillation

Pytorch implementation for the paper:
Membership and Memorization in LLM Knowledge Distillation
Ziqi Zhang, Ali Shahin Shamsabadi, Hanxiao Lu, Yifeng Cai, Hamed Haddadi

Scripts

The scripts for membership inference on distilled models are in scripts/model_family_6k. The scripts of memorization is in mia.

Environment Configuration

Please see README_distiLLM.md.

Acknowledgement

The code framework is based on DistiLLM and MiniLLM

Name		Name	Last commit message	Last commit date
configs		configs
data_utils		data_utils
distillm		distillm
imgs		imgs
mia		mia
minillm		minillm
scripts/model_family_6k		scripts/model_family_6k
tools		tools
transformers		transformers
.gitattributes		.gitattributes
.gitignore		.gitignore
README.md		README.md
README_distiLLM.md		README_distiLLM.md
arguments.py		arguments.py
evaluate.py		evaluate.py
evaluate_main.py		evaluate_main.py
evaluate_memorization.py		evaluate_memorization.py
evaluate_memorization_w_teacher.py		evaluate_memorization_w_teacher.py
evaluate_mi.py		evaluate_mi.py
evaluate_mi_blimp.py		evaluate_mi_blimp.py
evaluate_mi_main.py		evaluate_mi_main.py
evaluate_mi_mope.py		evaluate_mi_mope.py
finetune.py		finetune.py
generate.py		generate.py
install.sh		install.sh
mi_utils.py		mi_utils.py
rouge_metric.py		rouge_metric.py
train_minillm.py		train_minillm.py
utils.py		utils.py

Navigation Menu

Repository files navigation

Membership and Memorization in LLM Knowledge Distillation

Scripts

Environment Configuration

Acknowledgement

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

Membership and Memorization in LLM Knowledge Distillation

Scripts

Environment Configuration

Acknowledgement

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages