2025-06-23T01:23:25.544763334Z ========== 2025-06-23T01:23:25.544780039Z == CUDA == 2025-06-23T01:23:25.544853039Z ========== 2025-06-23T01:23:25.558765052Z CUDA Version 12.8.1 2025-06-23T01:23:25.558769669Z Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. 2025-06-23T01:23:25.558773945Z This container image and its contents are governed by the NVIDIA Deep Learning Container License. 2025-06-23T01:23:25.558776489Z By pulling and using the container, you accept the terms and conditions of this license: 2025-06-23T01:23:25.558778572Z https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license 2025-06-23T01:23:25.558782168Z A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience. 2025-06-23T01:23:25.564874714Z worker-comfyui: Starting ComfyUI 2025-06-23T01:23:25.565104971Z worker-comfyui: Starting RunPod Handler 2025-06-23T01:23:25.621035499Z Adding extra search path checkpoints /runpod-volume/models/checkpoints 2025-06-23T01:23:25.621063742Z Adding extra search path clip /runpod-volume/models/clip 2025-06-23T01:23:25.621066436Z Adding extra search path clip_vision /runpod-volume/models/clip_vision 2025-06-23T01:23:25.621072065Z Adding extra search path configs /runpod-volume/models/configs 2025-06-23T01:23:25.621074068Z Adding extra search path controlnet /runpod-volume/models/controlnet 2025-06-23T01:23:25.621105064Z Adding extra search path embeddings /runpod-volume/models/embeddings 2025-06-23T01:23:25.621111604Z Adding extra search path loras /runpod-volume/models/loras 2025-06-23T01:23:25.621143883Z Adding extra search path upscale_models /runpod-volume/models/upscale_models 2025-06-23T01:23:25.621160297Z Adding extra search path vae /runpod-volume/models/vae 2025-06-23T01:23:25.621166647Z Adding extra search path unet /runpod-volume/models/unet 2025-06-23T01:23:25.621185445Z Adding extra search path diffusion_models /runpod-volume/models/diffusion_models 2025-06-23T01:23:25.621198004Z Adding extra search path text_encoders /runpod-volume/models/text_encoders 2025-06-23T01:23:26.284870280Z [START] Security scan 2025-06-23T01:23:26.284909630Z [DONE] Security scan 2025-06-23T01:23:26.284915278Z ## ComfyUI-Manager: installing dependencies done. 2025-06-23T01:23:26.284920125Z ** ComfyUI startup time: 2025-06-23 01:23:26.284 2025-06-23T01:23:26.284924953Z ** Platform: Linux 2025-06-23T01:23:26.284929509Z ** Python version: 3.12.3 (main, Feb 4 2025, 14:48:35) [GCC 13.3.0] 2025-06-23T01:23:26.284936600Z ** Python executable: /opt/venv/bin/python 2025-06-23T01:23:26.284941127Z ** ComfyUI Path: /comfyui 2025-06-23T01:23:26.284983451Z ** ComfyUI Base Folder Path: /comfyui 2025-06-23T01:23:26.285017942Z ** User directory: /comfyui/user 2025-06-23T01:23:26.285062910Z ** ComfyUI-Manager config path: /comfyui/user/default/ComfyUI-Manager/config.ini 2025-06-23T01:23:26.285138484Z ** Log path: /comfyui/user/comfyui.log 2025-06-23T01:23:26.834850231Z Prestartup times for custom nodes: 2025-06-23T01:23:26.834856610Z 0.0 seconds: /comfyui/custom_nodes/rgthree-comfy 2025-06-23T01:23:26.834893426Z 0.0 seconds: /comfyui/custom_nodes/ComfyUI-Easy-Use 2025-06-23T01:23:26.834899325Z 1.2 seconds: /comfyui/custom_nodes/ComfyUI-Manager 2025-06-23T01:23:27.375278515Z worker-comfyui - Starting handler... 2025-06-23T01:23:27.375313328Z --- Starting Serverless Worker | Version 1.7.10 --- 2025-06-23T01:23:27.854874809Z Checkpoint files will always be loaded safely. 2025-06-23T01:23:28.149346942Z Traceback (most recent call last): 2025-06-23T01:23:28.149391349Z File "/comfyui/main.py", line 130, in 2025-06-23T01:23:28.149441004Z import execution 2025-06-23T01:23:28.149496678Z File "/comfyui/execution.py", line 13, in 2025-06-23T01:23:28.149606753Z import nodes 2025-06-23T01:23:28.149676749Z File "/comfyui/nodes.py", line 22, in 2025-06-23T01:23:28.149772012Z import comfy.diffusers_load 2025-06-23T01:23:28.149889489Z File "/comfyui/comfy/diffusers_load.py", line 3, in 2025-06-23T01:23:28.149969049Z import comfy.sd 2025-06-23T01:23:28.150021188Z File "/comfyui/comfy/sd.py", line 7, in 2025-06-23T01:23:28.150139716Z from comfy import model_management 2025-06-23T01:23:28.150202791Z File "/comfyui/comfy/model_management.py", line 221, in 2025-06-23T01:23:28.150334900Z total_vram = get_total_memory(get_torch_device()) / (1024 * 1024) 2025-06-23T01:23:28.151454657Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-06-23T01:23:28.151499885Z File "/comfyui/comfy/model_management.py", line 206, in get_total_memory 2025-06-23T01:23:28.151633736Z _, mem_total_cuda = torch.cuda.mem_get_info(dev) 2025-06-23T01:23:28.152749737Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-06-23T01:23:28.152783919Z File "/opt/venv/lib/python3.12/site-packages/torch/cuda/memory.py", line 841, in mem_get_info 2025-06-23T01:23:28.152991441Z return torch.cuda.cudart().cudaMemGetInfo(device) 2025-06-23T01:23:28.154076145Z ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-06-23T01:23:28.154205470Z RuntimeError: CUDA error: CUDA-capable device(s) is/are busy or unavailable 2025-06-23T01:23:28.154210317Z CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. 2025-06-23T01:23:28.154215445Z For debugging consider passing CUDA_LAUNCH_BLOCKING=1 2025-06-23T01:23:28.154219982Z Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions. 2025-06-23T01:23:28.623067557Z {"requestId": null, "message": "Jobs in queue: 1", "level": "INFO"} 2025-06-23T01:23:28.623146916Z {"requestId": null, "message": "Jobs in progress: 1", "level": "INFO"} 2025-06-23T01:23:28.623153246Z {"requestId": "0462b632-56d1-46f5-bcad-fae3a2cb82d0-e2", "message": "Started.", "level": "INFO"} 2025-06-23T01:23:28.623158623Z worker-comfyui - Checking API server at http://127.0.0.1:8188/... 2025-06-23T01:23:54.423690636Z worker-comfyui - Failed to connect to server at http://127.0.0.1:8188/ after 500 attempts. 2025-06-23T01:23:55.046770084Z {"requestId": "0462b632-56d1-46f5-bcad-fae3a2cb82d0-e2", "message": "Finished.", "level": "INFO"}