lora訓練之偷師
2023-11-08 06:00:58
自stable diffusion開源之後AIGC繪畫方向客製化化百花齊放百家爭鳴。
而c站 https://civitai.com/
也聚集了全球愛好者的各種微調訓練模型分享。
其中以lora為首，應用最廣泛。
而這些模型是怎麼訓練出來的，細節到底是什麼樣的，沒有多少人能說得清楚。
而就博主知道的，
使用最廣泛的訓練程式碼是這份 https://github.com/bmaltais/kohya_ss
它流行的原因主要是在其做了個GUI介面，便於很多不懂程式碼的人也能使用。
這裡博主沒打算講解kohya_ss下具體訓練方法和其引數。
想知道具體訓練引數和細節，可以移步其wiki
https://github.com/bmaltais/kohya_ss/wiki/LoRA-training-parameters
還是不懂，也可以問各種gpt或者直接翻譯成中文閱讀即可。
現在主流最廣泛使用的模型儲存格式為 huggingface的safetensors
https://github.com/huggingface/safetensors
kohya_ss裡預設會把訓練引數填到safetensors的metadata裡。
說到這裡，lora訓練之偷師 答案呼之欲出。
假設我們從c站上把safetensors模型下載下來，
然後讀出來它的metadata資訊,
那不就能知道這個模型具體的訓練引數嗎？
而在博主實操讀出來好幾個主流模型的引數之後，很感慨:
c站上的模型真的很demo，訓練的資料集大多都不大，解析度也不高，有的也就百來張圖。
訓練引數也可以大概評估出來，每個模型作者的用心程度，
當然也可以作為評價模型的一種標準，
例如：看過了多少張圖，什麼樣的解析度訓練的，訓練了多久之類的。
也就對這個微調的模型能有個大概認知。
多看幾個模型之後，
新手也可以快速上手訓練自己的lora，客製化自己的AI女友之類的。
你說你想直接抄他們的引數訓練，當然也是可以的，偷師不可恥很光榮。
為了方便讀取模型的模型的訓練資訊，博主寫了個小工具。
開源地址為:
https://github.com/cpuimage/st_metadata_parser
使用也非常簡單:
python st_metadata_parser.py input_file.safetensors
輸出的引數具體含義可以參考：
https://github.com/bmaltais/kohya_ss/wiki/LoRA-training-parameters
附一份C站上某個lora模型檔案的訓練引數供一覽：
[('ss_cache_latents', 'True'),
('ss_caption_dropout_every_n_epochs', '0'),
('ss_caption_dropout_rate', '0.0'),
('ss_caption_tag_dropout_rate', '0.0'),
('ss_clip_skip', '2'),
('ss_dataset_dirs', '{"rapunzel": {"n_repeats": 5, "img_count": 40}}'),
('ss_datasets',
'[{"is_dreambooth": true, "batch_size_per_device": 2, "num_train_images": '
'200, "num_reg_images": 0, "resolution": [512, 512], "enable_bucket": true, '
'"min_bucket_reso": 256, "max_bucket_reso": 1024, "tag_frequency": '
'{"rapunzel": {"rapunzel 1girl": 40, "solo": 39, "long hair": 37, "smile": '
'26, "blonde hair": 38, "dress": 21, "very long hair": 18, "green eyes": 30, '
'"crossed arms": 3, "corset": 5, "purple dress": 12, "breasts": 13, "open '
'mouth": 7, "medium breasts": 5, "nipples": 7, "swimsuit": 2, "bikini": 1, '
'"tongue": 2, "striped": 1, "lips": 7, "micro bikini": 1, "freckles": 5, '
'"looking at viewer": 10, "long sleeves": 7, "cleavage": 3, "multiple boys": '
'1, "puffy sleeves": 7, "2boys": 1, "full body": 4, "absurdly long hair": 2, '
'"grin": 2, "hand on hip": 3, "juliet sleeves": 1, "blush": 3, "large '
'breasts": 4, "hair ornament": 5, "standing": 2, "flower": 7, "cowboy shot": '
'1, "teeth": 5, "indoors": 1, "hair flower": 5, "one-piece swimsuit": 1, '
'"covered navel": 1, "makeup": 2, "cameltoe": 1, "highleg": 1, "lamp": 1, '
'"braid": 4, "blurry": 2, "arm up": 1, "depth of field": 1, "blurry '
'background": 2, "looking up": 3, "bokeh": 1, "upper body": 4, "short '
'sleeves": 3, "closed eyes": 1, "small breasts": 1, "lying": 3, "signature": '
'1, "on back": 1, "arms up": 2, "eyelashes": 1, "from above": 1, "happy": 1, '
'"grass": 1, "cross-laced clothes": 1, "hair spread out": 1, "sitting": 4, '
'"outdoors": 2, "barefoot": 7, "water": 1, "tree": 2, "pink dress": 1, '
'"holding flower": 1, "lily pad": 1, "simple background": 5, "white '
'background": 3, "frog": 4, "black background": 2, "child": 1, "star print": '
'1, "lantern": 1, "blue eyes": 2, "navel": 5, "underwear": 2, "panties": 2, '
'"thigh gap": 1, "bound": 2, "bdsm": 2, "bondage": 2, "gag": 1, "gagged": 1, '
'"improvised gag": 1, "cloth gag": 1, "nude": 5, "tongue out": 1, "saliva": '
'1, "arms behind back": 1, "seiza": 1, "bound arms": 1, "on head": 1, '
'"animal on head": 1, "holding hair": 1, "tiara": 1, "crown": 1, "princess": '
'1, "hands on own head": 1, "mirror": 2, "frying pan": 2, "nature": 1, '
'"parted lips": 1, "blue background": 1, "shirt": 1, "choker": 2, "huge '
'breasts": 1, "colored skin": 1, "bursting breasts": 1, "thighhighs": 2, '
'"jewelry": 1, "thighs": 1, "earrings": 1, "pussy": 3, "spread legs": 2, '
'"cum": 1, "high heels": 1, "kneeling": 1, "uncensored": 3, "cum in pussy": '
'1, "phone": 1, "cellphone": 1, "smartphone": 1, "after sex": 1, "holding '
'phone": 1, "cumdrip": 1, "brown thighhighs": 1, "gloves": 1, "multicolored '
'hair": 1, "white gloves": 1, "topless": 1, "pink panties": 1, "fishnet '
'thighhighs": 1, "ass": 1, "feet": 1, "soles": 1, "on stomach": 1, "the '
'pose": 1, "nail polish": 1, "tan": 1, "personification": 1, "tanlines": 1, '
'"detached sleeves": 1, "armpits": 1, "pillow": 1}}, "bucket_info": '
'{"buckets": {"0": {"resolution": [320, 704], "count": 30}, "1": '
'{"resolution": [384, 640], "count": 95}, "2": {"resolution": [448, 576], '
'"count": 50}, "3": {"resolution": [512, 512], "count": 20}, "4": '
'{"resolution": [576, 448], "count": 5}}, "mean_img_ar_error": '
'0.042025975009129204}, "subsets": [{"img_count": 40, "num_repeats": 5, '
'"color_aug": false, "flip_aug": false, "random_crop": false, '
'"shuffle_caption": true, "keep_tokens": 1, "image_dir": "rapunzel", '
'"class_tokens": null, "is_reg": false}]}]'),
('ss_epoch', '10'),
('ss_face_crop_aug_range', 'None'),
('ss_full_fp16', 'False'),
('ss_gradient_accumulation_steps', '1'),
('ss_gradient_checkpointing', 'False'),
('ss_learning_rate', '0.0005'),
('ss_lowram', 'True'),
('ss_lr_scheduler', 'cosine_with_restarts'),
('ss_lr_warmup_steps', '50'),
('ss_max_grad_norm', '1.0'),
('ss_max_token_length', '225'),
('ss_max_train_steps', '1010'),
('ss_min_snr_gamma', '5.0'),
('ss_mixed_precision', 'fp16'),
('ss_multires_noise_discount', '0.3'),
('ss_multires_noise_iterations', 'None'),
('ss_network_alpha', '16'),
('ss_network_dim', '32'),
('ss_network_module', 'networks.lora'),
('ss_new_sd_model_hash',
'b0c91a8a93ab63b8b513046c2e2181a4ce78b83367ee7e26354ba9e018ea102f'),
('ss_noise_offset', 'None'),
('ss_num_batches_per_epoch', '101'),
('ss_num_epochs', '10'),
('ss_num_reg_images', '0'),
('ss_num_train_images', '200'),
('ss_optimizer', 'bitsandbytes.optim.adamw.AdamW8bit'),
('ss_output_name', 'rapunzel'),
('ss_prior_loss_weight', '1.0'),
('ss_sd_model_hash', '66f58dbb'),
('ss_sd_model_name', 'fp16-pruned.ckpt'),
('ss_sd_scripts_commit_hash', 'e6ad3cbc66130fdc3bf9ecd1e0272969b1d613f7'),
('ss_seed', '42'),
('ss_session_id', '1637709685'),
('ss_steps', '1010'),
('ss_tag_frequency',
'{"rapunzel": {"rapunzel 1girl": 40, "solo": 39, "long hair": 37, "smile": '
'26, "blonde hair": 38, "dress": 21, "very long hair": 18, "green eyes": 30, '
'"crossed arms": 3, "corset": 5, "purple dress": 12, "breasts": 13, "open '
'mouth": 7, "medium breasts": 5, "nipples": 7, "swimsuit": 2, "bikini": 1, '
'"tongue": 2, "striped": 1, "lips": 7, "micro bikini": 1, "freckles": 5, '
'"looking at viewer": 10, "long sleeves": 7, "cleavage": 3, "multiple boys": '
'1, "puffy sleeves": 7, "2boys": 1, "full body": 4, "absurdly long hair": 2, '
'"grin": 2, "hand on hip": 3, "juliet sleeves": 1, "blush": 3, "large '
'breasts": 4, "hair ornament": 5, "standing": 2, "flower": 7, "cowboy shot": '
'1, "teeth": 5, "indoors": 1, "hair flower": 5, "one-piece swimsuit": 1, '
'"covered navel": 1, "makeup": 2, "cameltoe": 1, "highleg": 1, "lamp": 1, '
'"braid": 4, "blurry": 2, "arm up": 1, "depth of field": 1, "blurry '
'background": 2, "looking up": 3, "bokeh": 1, "upper body": 4, "short '
'sleeves": 3, "closed eyes": 1, "small breasts": 1, "lying": 3, "signature": '
'1, "on back": 1, "arms up": 2, "eyelashes": 1, "from above": 1, "happy": 1, '
'"grass": 1, "cross-laced clothes": 1, "hair spread out": 1, "sitting": 4, '
'"outdoors": 2, "barefoot": 7, "water": 1, "tree": 2, "pink dress": 1, '
'"holding flower": 1, "lily pad": 1, "simple background": 5, "white '
'background": 3, "frog": 4, "black background": 2, "child": 1, "star print": '
'1, "lantern": 1, "blue eyes": 2, "navel": 5, "underwear": 2, "panties": 2, '
'"thigh gap": 1, "bound": 2, "bdsm": 2, "bondage": 2, "gag": 1, "gagged": 1, '
'"improvised gag": 1, "cloth gag": 1, "nude": 5, "tongue out": 1, "saliva": '
'1, "arms behind back": 1, "seiza": 1, "bound arms": 1, "on head": 1, '
'"animal on head": 1, "holding hair": 1, "tiara": 1, "crown": 1, "princess": '
'1, "hands on own head": 1, "mirror": 2, "frying pan": 2, "nature": 1, '
'"parted lips": 1, "blue background": 1, "shirt": 1, "choker": 2, "huge '
'breasts": 1, "colored skin": 1, "bursting breasts": 1, "thighhighs": 2, '
'"jewelry": 1, "thighs": 1, "earrings": 1, "pussy": 3, "spread legs": 2, '
'"cum": 1, "high heels": 1, "kneeling": 1, "uncensored": 3, "cum in pussy": '
'1, "phone": 1, "cellphone": 1, "smartphone": 1, "after sex": 1, "holding '
'phone": 1, "cumdrip": 1, "brown thighhighs": 1, "gloves": 1, "multicolored '
'hair": 1, "white gloves": 1, "topless": 1, "pink panties": 1, "fishnet '
'thighhighs": 1, "ass": 1, "feet": 1, "soles": 1, "on stomach": 1, "the '
'pose": 1, "nail polish": 1, "tan": 1, "personification": 1, "tanlines": 1, '
'"detached sleeves": 1, "armpits": 1, "pillow": 1}}'),
('ss_text_encoder_lr', '0.0001'),
('ss_training_comment', 'None'),
('ss_training_finished_at', '1694906008.5775223'),
('ss_training_started_at', '1694905352.3001792'),
('ss_unet_lr', '0.0005'),
('ss_v2', 'False'),
('sshs_legacy_hash', 'fd5d0c6f'),
('sshs_model_hash',
'0f9a9c96e707b34f1591ed9d24e788769cb72e2040f380aa7c31b6a49ca783b2')]
有資料就比較直觀，不然這篇博文，就顯得乾巴巴的。
也算為業界做了一點點貢獻。
提一個額外的話題，
由於diffusers和stable-diffusion-webui的實現十分臃腫，
並且屎山越來越大，非常不便於研發以及學習。
為了遠離那堆玩意，博主化身屎殼郎，花了不少精力對stable diffusion進行了整理和重寫。
開源了sd 1.5和 sdxl 1.0 兩個倉庫，主要以keras/TensorFlow為主，相容pytroch的模型權重。
專案地址為:
sd1.5:
https://github.com/cpuimage/minSDTF
sdxl 1.0：
https://github.com/cpuimage/minSDXLTF
目前有空會持續更新，目標是把webui的主要功能以比較簡潔優雅的方式實現，路漫漫其修遠兮。
如果有小夥伴有能力也想加入，歡迎聯絡博主，一起搖擺。
也有業內好友期望博主寫一份keras/tensorflow版本的lora訓練框架，只能說有時間的話，會寫的。
最後嘮叨幾句，kohya_ss訓練框架的作者，其程式碼好幾個地方邏輯寫錯了，小夥伴們要注意了。
最後的最後，
有stable diffusion相關的訓練或者開發客製化，不論是需求還是各種問題都可以通過以下方式聯絡博主，
微信: Dbgmonks
QQ: 200759103
注: 博主提供有償服務, 不註明來意者一律拒絕。