home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/transformers/utils/generic.py:482: FutureWarning: `torch.utils._pytree._register_pytree_node` is deprecated. Please use `torch.utils._pytree.register_pytree_node` instead. _torch_pytree._register_pytree_node( /home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/diffusers/utils/outputs.py:63: FutureWarning: `torch.utils._pytree._register_pytree_node` is deprecated. Please use `torch.utils._pytree.register_pytree_node` instead. torch.utils._pytree._register_pytree_node( /home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/transformers/utils/generic.py:482: FutureWarning: `torch.utils._pytree._register_pytree_node` is deprecated. Please use `torch.utils._pytree.register_pytree_node` instead. _torch_pytree._register_pytree_node( /home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/transformers/utils/generic.py:339: FutureWarning: `torch.utils._pytree._register_pytree_node` is deprecated. Please use `torch.utils._pytree.register_pytree_node` instead. _torch_pytree._register_pytree_node( 2025-03-03 04:04:45.314093: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered 2025-03-03 04:04:45.314172: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered 2025-03-03 04:04:45.316655: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered 2025-03-03 04:04:45.328085: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations. To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags. 2025-03-03 04:04:46.707616: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT Traceback (most recent call last): File "/home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/diffusers/utils/import_utils.py", line 710, in _get_module return importlib.import_module("." + module_name, self.__name__) File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1050, in _gcd_import File "", line 1027, in _find_and_load File "", line 1006, in _find_and_load_unlocked File "", line 688, in _load_unlocked File "", line 883, in exec_module File "", line 241, in _call_with_frames_removed File "/home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/diffusers/loaders/ip_adapter.py", line 34, in from ..models.attention_processor import ( File "/home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/diffusers/models/attention_processor.py", line 32, in import xformers.ops File "/home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/xformers/ops/__init__.py", line 8, in from .fmha import ( File "/home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/xformers/ops/fmha/__init__.py", line 10, in from . import ( File "/home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/xformers/ops/fmha/triton_splitk.py", line 110, in from ._triton.splitk_kernels import _fwd_kernel_splitK, _splitK_reduce File "/home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/xformers/ops/fmha/_triton/splitk_kernels.py", line 632, in _fwd_kernel_splitK_autotune[num_groups] = autotune_kernel( File "/home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/xformers/ops/fmha/_triton/splitk_kernels.py", line 614, in autotune_kernel kernel = triton.autotune( TypeError: autotune() got an unexpected keyword argument 'use_cuda_graph' The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/diffusers/utils/import_utils.py", line 710, in _get_module return importlib.import_module("." + module_name, self.__name__) File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1050, in _gcd_import File "", line 1027, in _find_and_load File "", line 1006, in _find_and_load_unlocked File "", line 688, in _load_unlocked File "", line 883, in exec_module File "", line 241, in _call_with_frames_removed File "/home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py", line 24, in from ...loaders import FromSingleFileMixin, IPAdapterMixin, LoraLoaderMixin, TextualInversionLoaderMixin File "", line 1075, in _handle_fromlist File "/home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/diffusers/utils/import_utils.py", line 700, in __getattr__ module = self._get_module(self._class_to_module[name]) File "/home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/diffusers/utils/import_utils.py", line 712, in _get_module raise RuntimeError( RuntimeError: Failed to import diffusers.loaders.ip_adapter because of the following error (look up to see its traceback): autotune() got an unexpected keyword argument 'use_cuda_graph' The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/Ubuntu/apps/kohya_ss/sd-scripts/flux_train.py", line 33, in from library import deepspeed_utils, flux_train_utils, flux_utils, strategy_base, strategy_flux File "/home/Ubuntu/apps/kohya_ss/sd-scripts/library/flux_train_utils.py", line 17, in from library import flux_models, flux_utils, strategy_base, train_util File "/home/Ubuntu/apps/kohya_ss/sd-scripts/library/train_util.py", line 58, in from diffusers import ( File "", line 1075, in _handle_fromlist File "/home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/diffusers/utils/import_utils.py", line 701, in __getattr__ value = getattr(module, name) File "/home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/diffusers/utils/import_utils.py", line 701, in __getattr__ value = getattr(module, name) File "/home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/diffusers/utils/import_utils.py", line 700, in __getattr__ module = self._get_module(self._class_to_module[name]) File "/home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/diffusers/utils/import_utils.py", line 712, in _get_module raise RuntimeError( RuntimeError: Failed to import diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion because of the following error (look up to see its traceback): Failed to import diffusers.loaders.ip_adapter because of the following error (look up to see its traceback): autotune() got an unexpected keyword argument 'use_cuda_graph' Traceback (most recent call last): File "/home/Ubuntu/apps/kohya_ss/venv/bin/accelerate", line 8, in sys.exit(main()) File "/home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 48, in main args.func(args) File "/home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 1106, in launch_command simple_launcher(args) File "/home/Ubuntu/apps/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 704, in simple_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command '['/home/Ubuntu/apps/kohya_ss/venv/bin/python3.10', '/home/Ubuntu/apps/kohya_ss/sd-scripts/flux_train.py', '--config_file', '/home/Ubuntu/Downloads/Output/config_dreambooth-20250303-040433.toml']' returned non-zero exit status 1. 04:04:51-917647 INFO Training has ended.