Starting the GUI... this might take some time...
12:17:19-892211 WARNING  Skipping requirements verification.
12:17:19-907836 INFO     headless: False
12:17:19-907836 INFO     Using shell=True when running external commands...
* Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
12:17:41-446816 INFO     Start training Dreambooth...
12:17:41-447825 INFO     Validating lr scheduler arguments...
12:17:41-448817 INFO     Validating optimizer arguments...
12:17:41-449816 INFO     Validating C:/Users/PC/Desktop/AI/Projects/SwarmUI/Models/diffusion_models\model existence and
                         writability... SUCCESS
12:17:41-450533 INFO     Validating C:/Users/PC/Desktop/AI/Projects/Kohya_FLUX_DreamBooth_v17/flux1-dev.safetensors existence...
                         SUCCESS
12:17:41-450533 INFO     Validating C:/Users/PC/Desktop/AI/Projects/SwarmUI/Models/diffusion_models\img existence... SUCCESS
12:17:41-451580 INFO     Folder 1_my$1$1$1face man: 1 repeats found
12:17:41-452580 INFO     Folder 1_my$1$1$1face man: 115 images found
12:17:41-453580 INFO     Folder 1_my$1$1$1face man: 115 * 1 = 115 steps
12:17:41-454136 INFO     Regularization factor: 1
12:17:41-454136 INFO     Total steps: 115
12:17:41-455160 INFO     Train batch size: 2
12:17:41-456199 INFO     Gradient accumulation steps: 1
12:17:41-456199 INFO     Epoch: 40
12:17:41-457199 INFO     max_train_steps (115 / 2 / 1 * 40 * 1) = 2300
12:17:41-457199 INFO     lr_warmup_steps = 0
12:17:41-459200 INFO     Saving training config to
                         C:/Users/PC/Desktop/AI/Projects/SwarmUI/Models/diffusion_models\model\my_20250303-121741.json...
12:17:41-460056 INFO     Executing command:
                         C:\Users\PC\Desktop\AI\Projects\Kohya_FLUX_DreamBooth_v17\kohya_ss\venv\Scripts\accelerate.EXE launch
                         --dynamo_backend no --dynamo_mode default --gpu_ids 0 --mixed_precision bf16 --num_processes 1
                         --num_machines 1 --num_cpu_threads_per_process 20
                         C:/Users/PC/Desktop/AI/Projects/Kohya_FLUX_DreamBooth_v17/kohya_ss/sd-scripts/flux_train.py --config_file
                         C:/Users/PC/Desktop/AI/Projects/SwarmUI/Models/diffusion_models\model/config_dreambooth-20250303-121741.toml
C:\Users\PC\Desktop\AI\Projects\Kohya_FLUX_DreamBooth_v17\kohya_ss\venv\lib\site-packages\diffusers\utils\outputs.py:63: FutureWarning: `torch.utils._pytree._register_pytree_node` is deprecated. Please use `torch.utils._pytree.register_pytree_node` instead.
  torch.utils._pytree._register_pytree_node(
C:\Users\PC\Desktop\AI\Projects\Kohya_FLUX_DreamBooth_v17\kohya_ss\venv\lib\site-packages\diffusers\utils\outputs.py:63: FutureWarning: `torch.utils._pytree._register_pytree_node` is deprecated. Please use `torch.utils._pytree.register_pytree_node` instead.
  torch.utils._pytree._register_pytree_node(
2025-03-03 12:17:47 INFO     Loading settings from                                                                 train_util.py:4625
                             C:/Users/PC/Desktop/AI/Projects/SwarmUI/Models/diffusion_models\model/config_dreamboo
                             th-20250303-121741.toml...
2025-03-03 12:17:47 INFO     Using DreamBooth method.                                                               flux_train.py:115
                    INFO     prepare images.                                                                       train_util.py:2053
                    INFO     get image size from name of cache files                                               train_util.py:1944
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████| 115/115 [00:00<?, ?it/s]
                    INFO     set image size from cache files: 115/115                                              train_util.py:1974
                    INFO     found directory                                                                       train_util.py:2000
                             C:\Users\PC\Desktop\AI\Projects\SwarmUI\Models\diffusion_models\img\1_my$1$1$1face
                             man contains 115 image files
read caption: 100%|███████████████████████████████████████████████████████████████████████████████████████| 115/115 [00:00<?, ?it/s]
                    INFO     115 train images with repeats.                                                        train_util.py:2096
                    INFO     0 reg images with repeats.                                                            train_util.py:2100
                    WARNING  no regularization images / 正則化画像が見つかりませんでした                           train_util.py:2105
                    INFO     [Dataset 0]                                                                           config_util.py:575
                               batch_size: 2
                               resolution: (1024, 1024)
                               enable_bucket: False

                               [Subset 0 of Dataset 0]
                                 image_dir:
                             "C:\Users\PC\Desktop\AI\Projects\SwarmUI\Models\diffusion_models\img\1_my$1$1$1face
                             man"
                                 image_count: 115
                                 num_repeats: 1
                                 shuffle_caption: False
                                 keep_tokens: 0
                                 caption_dropout_rate: 0
                                 caption_dropout_every_n_epochs: 0
                                 caption_tag_dropout_rate: 0.0
                                 caption_prefix: None
                                 caption_suffix: None
                                 color_aug: False
                                 flip_aug: False
                                 face_crop_aug_range: None
                                 random_crop: False
                                 token_warmup_min: 1,
                                 token_warmup_step: 0,
                                 alpha_mask: False
                                 custom_attributes: {}
                                 is_reg: False
                                 class_tokens: my$1$1$1face man
                                 caption_extension: .txt


                    INFO     [Prepare dataset 0]                                                                   config_util.py:587
                    INFO     loading image sizes.                                                                   train_util.py:972
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████| 115/115 [00:00<?, ?it/s]
                    INFO     prepare dataset                                                                        train_util.py:997
                    INFO     Checking the state dict: Diffusers or BFL, dev or schnell                               flux_utils.py:43
                    INFO     prepare accelerator                                                                    flux_train.py:186
accelerator device: cuda
                    INFO     Building AutoEncoder                                                                   flux_utils.py:144
                    INFO     Loading state dict from                                                                flux_utils.py:149
                             C:/Users/PC/Desktop/AI/Projects/Kohya_FLUX_DreamBooth_v17/ae.safetensors
                    INFO     Loaded AE: <All keys matched successfully>                                             flux_utils.py:152
                    INFO     [Dataset 0]                                                                           train_util.py:2589
                    INFO     caching latents with caching strategy.                                                train_util.py:1097
                    INFO     caching latents...                                                                    train_util.py:1146
100%|███████████████████████████████████████████████████████████████████████████████████████████| 115/115 [00:00<00:00, 3679.97it/s]
C:\Users\PC\Desktop\AI\Projects\Kohya_FLUX_DreamBooth_v17\kohya_ss\venv\lib\site-packages\transformers\tokenization_utils_base.py:1601: FutureWarning: `clean_up_tokenization_spaces` was not set. It will be set to `True` by default. This behavior will be depracted in transformers v4.45, and will be then set to `False` by default. For more details check this issue: https://github.com/huggingface/transformers/issues/31884
  warnings.warn(
You are using the default legacy behaviour of the <class 'transformers.models.t5.tokenization_t5.T5Tokenizer'>. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565
2025-03-03 12:17:48 INFO     Building CLIP-L                                                                        flux_utils.py:179
                    INFO     Loading state dict from                                                                flux_utils.py:275
                             C:/Users/PC/Desktop/AI/Projects/Kohya_FLUX_DreamBooth_v17/clip_l.safetensors
                    INFO     Loaded CLIP-L: <All keys matched successfully>                                         flux_utils.py:278
                    INFO     Loading state dict from                                                                flux_utils.py:330
                             C:/Users/PC/Desktop/AI/Projects/Kohya_FLUX_DreamBooth_v17/t5xxl_fp16.safetensors
2025-03-03 12:17:55 INFO     Loaded T5xxl: <All keys matched successfully>                                          flux_utils.py:333
2025-03-03 12:17:56 INFO     [Dataset 0]                                                                           train_util.py:2611
                    INFO     caching Text Encoder outputs with caching strategy.                                   train_util.py:1280
                    INFO     checking cache validity...                                                            train_util.py:1291
100%|███████████████████████████████████████████████████████████████████████████████████████████| 115/115 [00:00<00:00, 7361.35it/s]
                    INFO     no Text Encoder outputs to cache                                                      train_util.py:1318
                    INFO     cache Text Encoder outputs for sample prompt:                                          flux_train.py:249
                             C:/Users/PC/Desktop/AI/Projects/SwarmUI/Models/diffusion_models\model\sample/prompt.tx
                             t
2025-03-03 12:17:57 INFO     Checking the state dict: Diffusers or BFL, dev or schnell                               flux_utils.py:43
                    INFO     Building Flux model dev from BFL checkpoint                                            flux_utils.py:101
                    INFO     Loading state dict from                                                                flux_utils.py:118
                             C:/Users/PC/Desktop/AI/Projects/Kohya_FLUX_DreamBooth_v17/flux1-dev.safetensors
                    INFO     Loaded Flux: <All keys matched successfully>                                           flux_utils.py:137
FLUX: Gradient checkpointing enabled. CPU offload: False
                    INFO     enable block swap: blocks_to_swap=10                                                   flux_train.py:304
FLUX: Block swap enabled. Swapping 10 blocks, double blocks: 5, single blocks: 10.
number of trainable parameters: 11901408320
prepare optimizer, data loader etc.
                    INFO     use Adafactor optimizer | {'scale_parameter': False, 'relative_step': False,          train_util.py:4937
                             'warmup_init': False, 'weight_decay': 0.01}
                    WARNING  because max_grad_norm is set, clip_grad_norm is enabled. consider set to 0 /          train_util.py:4965
                             max_grad_normが設定されているためclip_grad_normが有効になります。0に設定して無効にし
                             たほうがいいかもしれません
                    WARNING  constant_with_warmup will be good /                                                   train_util.py:4969
                             スケジューラはconstant_with_warmupが良いかもしれません
enable full bf16 training.
running training / 学習開始
  num examples / サンプル数: 115
  num batches per epoch / 1epochのバッチ数: 58
  num epochs / epoch数: 40
  batch size per device / バッチサイズ: 2
  gradient accumulation steps / 勾配を合計するステップ数 = 1
  total optimization steps / 学習ステップ数: 2300
steps:   0%|                                                                                               | 0/2300 [00:00<?, ?it/s]
epoch 1/40
2025-03-03 12:18:07 INFO     epoch is incremented. current_epoch: 0, epoch: 1                                       train_util.py:764
C:\Users\PC\Desktop\AI\Projects\Kohya_FLUX_DreamBooth_v17\kohya_ss\venv\lib\site-packages\torch\autograd\graph.py:825: UserWarning: cuDNN SDPA backward got grad_output.strides() != output.strides(), attempting to materialize a grad_output with matching strides... (Triggered internally at C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cudnn\MHA.cpp:676.)
  return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
steps:   3%|█▋                                                                  | 58/2300 [09:32<6:08:56,  9.87s/it, avr_loss=0.476]
epoch 2/40
2025-03-03 12:27:30 INFO     epoch is incremented. current_epoch: 1, epoch: 2                                       train_util.py:764
steps:   5%|███▍                                                               | 116/2300 [19:58<6:15:58, 10.33s/it, avr_loss=0.454]
epoch 3/40
2025-03-03 12:37:55 INFO     epoch is incremented. current_epoch: 2, epoch: 3                                       train_util.py:764
steps:   8%|█████                                                              | 174/2300 [30:01<6:06:56, 10.36s/it, avr_loss=0.457]
epoch 4/40
2025-03-03 12:47:59 INFO     epoch is incremented. current_epoch: 3, epoch: 4                                       train_util.py:764
steps:  10%|██████▊                                                            | 232/2300 [38:40<5:44:41, 10.00s/it, avr_loss=0.441]
epoch 5/40
2025-03-03 12:56:37 INFO     epoch is incremented. current_epoch: 4, epoch: 5                                       train_util.py:764
steps:  13%|████████▍                                                          | 290/2300 [47:16<5:27:37,  9.78s/it, avr_loss=0.431]
epoch 6/40
2025-03-03 13:05:13 INFO     epoch is incremented. current_epoch: 5, epoch: 6                                       train_util.py:764
steps:  15%|██████████▏                                                        | 348/2300 [55:54<5:13:34,  9.64s/it, avr_loss=0.454]
epoch 7/40
2025-03-03 13:13:51 INFO     epoch is incremented. current_epoch: 6, epoch: 7                                       train_util.py:764
steps:  18%|███████████▍                                                     | 406/2300 [1:04:35<5:01:19,  9.55s/it, avr_loss=0.468]
epoch 8/40
2025-03-03 13:22:32 INFO     epoch is incremented. current_epoch: 7, epoch: 8                                       train_util.py:764
steps:  20%|█████████████                                                    | 464/2300 [1:13:51<4:52:13,  9.55s/it, avr_loss=0.451]
epoch 9/40
2025-03-03 13:31:48 INFO     epoch is incremented. current_epoch: 8, epoch: 9                                       train_util.py:764
steps:  23%|██████████████▊                                                  | 522/2300 [1:22:59<4:42:39,  9.54s/it, avr_loss=0.455]
epoch 10/40
2025-03-03 13:40:56 INFO     epoch is incremented. current_epoch: 9, epoch: 10                                      train_util.py:764
steps:  25%|████████████████▍                                                | 580/2300 [1:32:07<4:33:11,  9.53s/it, avr_loss=0.431]
epoch 11/40
2025-03-03 13:50:04 INFO     epoch is incremented. current_epoch: 10, epoch: 11                                     train_util.py:764
steps:  28%|██████████████████                                               | 638/2300 [1:41:16<4:23:49,  9.52s/it, avr_loss=0.447]
epoch 12/40
2025-03-03 13:59:13 INFO     epoch is incremented. current_epoch: 11, epoch: 12                                     train_util.py:764
steps:  30%|███████████████████▋                                             | 696/2300 [1:50:28<4:14:36,  9.52s/it, avr_loss=0.454]
epoch 13/40
2025-03-03 14:08:25 INFO     epoch is incremented. current_epoch: 12, epoch: 13                                     train_util.py:764
steps:  33%|█████████████████████▎                                           | 754/2300 [1:59:41<4:05:24,  9.52s/it, avr_loss=0.445]
epoch 14/40
2025-03-03 14:17:38 INFO     epoch is incremented. current_epoch: 13, epoch: 14                                     train_util.py:764
steps:  35%|██████████████████████▉                                          | 812/2300 [2:08:52<3:56:09,  9.52s/it, avr_loss=0.455]
epoch 15/40
2025-03-03 14:26:49 INFO     epoch is incremented. current_epoch: 14, epoch: 15                                     train_util.py:764
steps:  38%|████████████████████████▌                                        | 870/2300 [2:18:03<3:46:55,  9.52s/it, avr_loss=0.457]
epoch 16/40
2025-03-03 14:36:00 INFO     epoch is incremented. current_epoch: 15, epoch: 16                                     train_util.py:764
steps:  40%|██████████████████████████▏                                      | 928/2300 [2:27:16<3:37:44,  9.52s/it, avr_loss=0.435]
epoch 17/40
2025-03-03 14:45:14 INFO     epoch is incremented. current_epoch: 16, epoch: 17                                     train_util.py:764
steps:  43%|███████████████████████████▊                                     | 986/2300 [2:36:30<3:28:34,  9.52s/it, avr_loss=0.454]
epoch 18/40
2025-03-03 14:54:28 INFO     epoch is incremented. current_epoch: 17, epoch: 18                                     train_util.py:764
steps:  45%|█████████████████████████████                                   | 1044/2300 [2:45:42<3:19:20,  9.52s/it, avr_loss=0.441]
epoch 19/40
2025-03-03 15:03:39 INFO     epoch is incremented. current_epoch: 18, epoch: 19                                     train_util.py:764
steps:  48%|██████████████████████████████▋                                 | 1102/2300 [2:54:52<3:10:06,  9.52s/it, avr_loss=0.437]
epoch 20/40
2025-03-03 15:12:50 INFO     epoch is incremented. current_epoch: 19, epoch: 20                                     train_util.py:764
steps:  50%|████████████████████████████████▎                               | 1160/2300 [3:04:06<3:00:56,  9.52s/it, avr_loss=0.444]2025-03-03 15:22:04 INFO                                                                                           train_util.py:5763
                    INFO     saving checkpoint:                                                                    train_util.py:5764
                             C:/Users/PC/Desktop/AI/Projects/SwarmUI/Models/diffusion_models\model\my-000020.safet
                             ensors
Using memory efficient save file: C:/Users/PC/Desktop/AI/Projects/SwarmUI/Models/diffusion_models\model\my-000020.safetensors

epoch 21/40
2025-03-03 15:22:58 INFO     epoch is incremented. current_epoch: 20, epoch: 21                                     train_util.py:764
steps:  53%|█████████████████████████████████▉                              | 1218/2300 [3:14:10<2:52:29,  9.57s/it, avr_loss=0.445]
epoch 22/40
2025-03-03 15:32:08 INFO     epoch is incremented. current_epoch: 21, epoch: 22                                     train_util.py:764
steps:  55%|███████████████████████████████████▌                            | 1276/2300 [3:23:23<2:43:13,  9.56s/it, avr_loss=0.453]
epoch 23/40
2025-03-03 15:41:20 INFO     epoch is incremented. current_epoch: 22, epoch: 23                                     train_util.py:764
steps:  58%|█████████████████████████████████████                           | 1334/2300 [3:32:32<2:33:54,  9.56s/it, avr_loss=0.462]
epoch 24/40
2025-03-03 15:50:30 INFO     epoch is incremented. current_epoch: 23, epoch: 24                                     train_util.py:764
steps:  61%|██████████████████████████████████████▋                         | 1392/2300 [3:41:44<2:24:38,  9.56s/it, avr_loss=0.445]
epoch 25/40
2025-03-03 15:59:41 INFO     epoch is incremented. current_epoch: 24, epoch: 25                                     train_util.py:764
steps:  63%|████████████████████████████████████████▎                       | 1450/2300 [3:50:52<2:15:20,  9.55s/it, avr_loss=0.428]
epoch 26/40
2025-03-03 16:08:50 INFO     epoch is incremented. current_epoch: 25, epoch: 26                                     train_util.py:764
steps:  66%|█████████████████████████████████████████▉                      | 1508/2300 [4:00:04<2:06:05,  9.55s/it, avr_loss=0.453]
epoch 27/40
2025-03-03 16:18:02 INFO     epoch is incremented. current_epoch: 26, epoch: 27                                     train_util.py:764
steps:  68%|███████████████████████████████████████████▌                    | 1566/2300 [4:09:14<1:56:49,  9.55s/it, avr_loss=0.436]
epoch 28/40
2025-03-03 16:27:12 INFO     epoch is incremented. current_epoch: 27, epoch: 28                                     train_util.py:764
steps:  71%|█████████████████████████████████████████████▉                   | 1624/2300 [4:18:25<1:47:34,  9.55s/it, avr_loss=0.45]
epoch 29/40
2025-03-03 16:36:22 INFO     epoch is incremented. current_epoch: 28, epoch: 29                                     train_util.py:764
steps:  73%|██████████████████████████████████████████████▊                 | 1682/2300 [4:27:37<1:38:19,  9.55s/it, avr_loss=0.459]
epoch 30/40
2025-03-03 16:45:34 INFO     epoch is incremented. current_epoch: 29, epoch: 30                                     train_util.py:764
steps:  76%|████████████████████████████████████████████████▍               | 1740/2300 [4:36:50<1:29:05,  9.55s/it, avr_loss=0.424]
epoch 31/40
2025-03-03 16:54:47 INFO     epoch is incremented. current_epoch: 30, epoch: 31                                     train_util.py:764
steps:  78%|██████████████████████████████████████████████████              | 1798/2300 [4:46:00<1:19:51,  9.54s/it, avr_loss=0.437]
epoch 32/40
2025-03-03 17:03:58 INFO     epoch is incremented. current_epoch: 31, epoch: 32                                     train_util.py:764
steps:  81%|███████████████████████████████████████████████████▋            | 1856/2300 [4:55:18<1:10:38,  9.55s/it, avr_loss=0.453]
epoch 33/40
2025-03-03 17:13:16 INFO     epoch is incremented. current_epoch: 32, epoch: 33                                     train_util.py:764
steps:  83%|█████████████████████████████████████████████████████▎          | 1914/2300 [5:04:27<1:01:24,  9.54s/it, avr_loss=0.435]
epoch 34/40
2025-03-03 17:22:24 INFO     epoch is incremented. current_epoch: 33, epoch: 34                                     train_util.py:764
steps:  86%|████████████████████████████████████████████████████████▌         | 1972/2300 [5:13:36<52:09,  9.54s/it, avr_loss=0.436]
epoch 35/40
2025-03-03 17:31:33 INFO     epoch is incremented. current_epoch: 34, epoch: 35                                     train_util.py:764
steps:  88%|██████████████████████████████████████████████████████████▎       | 2030/2300 [5:22:46<42:55,  9.54s/it, avr_loss=0.444]
epoch 36/40
2025-03-03 17:40:43 INFO     epoch is incremented. current_epoch: 35, epoch: 36                                     train_util.py:764
steps:  91%|███████████████████████████████████████████████████████████▉      | 2088/2300 [5:31:56<33:42,  9.54s/it, avr_loss=0.443]
epoch 37/40
2025-03-03 17:49:53 INFO     epoch is incremented. current_epoch: 36, epoch: 37                                     train_util.py:764
steps:  93%|█████████████████████████████████████████████████████████████▌    | 2146/2300 [5:41:10<24:29,  9.54s/it, avr_loss=0.447]
epoch 38/40
2025-03-03 17:59:08 INFO     epoch is incremented. current_epoch: 37, epoch: 38                                     train_util.py:764
steps:  96%|███████████████████████████████████████████████████████████████▏  | 2204/2300 [5:50:24<15:15,  9.54s/it, avr_loss=0.432]
epoch 39/40
2025-03-03 18:08:22 INFO     epoch is incremented. current_epoch: 38, epoch: 39                                     train_util.py:764
steps:  98%|████████████████████████████████████████████████████████████████▉ | 2262/2300 [5:59:31<06:02,  9.54s/it, avr_loss=0.451]
epoch 40/40
2025-03-03 18:17:29 INFO     epoch is incremented. current_epoch: 39, epoch: 40                                     train_util.py:764
steps: 100%|██████████████████████████████████████████████████████████████████| 2300/2300 [6:05:31<00:00,  9.54s/it, avr_loss=0.455]2025-03-03 18:23:28 INFO     save trained model as StableDiffusion checkpoint to                                   train_util.py:5921
                             C:/Users/PC/Desktop/AI/Projects/SwarmUI/Models/diffusion_models\model\my.safetensors
Using memory efficient save file: C:/Users/PC/Desktop/AI/Projects/SwarmUI/Models/diffusion_models\model\my.safetensors
2025-03-03 18:23:48 INFO     model saved.                                                                           flux_train.py:782
steps: 100%|██████████████████████████████████████████████████████████████████| 2300/2300 [6:05:50<00:00,  9.54s/it, avr_loss=0.455]
18:23:50-451134 INFO     Training has ended.