2026-03-24T18:17:01.765792568Z ==========
2026-03-24T18:17:01.765797782Z == CUDA ==
2026-03-24T18:17:01.765800136Z ==========
2026-03-24T18:17:01.771845027Z CUDA Version 12.3.2
2026-03-24T18:17:01.773759934Z Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2026-03-24T18:17:01.775399622Z This container image and its contents are governed by the NVIDIA Deep Learning Container License.
2026-03-24T18:17:01.775402125Z By pulling and using the container, you accept the terms and conditions of this license:
2026-03-24T18:17:01.775404142Z https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license
2026-03-24T18:17:01.775407933Z A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.
2026-03-24T18:17:04.031333051Z --- Starting Serverless Worker |  Version 1.7.9 ---
2026-03-24T18:17:04.223027667Z {"requestId": null, "message": "Jobs in queue: 1", "level": "INFO"}
2026-03-24T18:17:04.223081551Z {"requestId": null, "message": "Jobs in progress: 1", "level": "INFO"}
2026-03-24T18:17:04.223087827Z {"requestId": "7be2dd5c-0506-44dd-9b1f-0b5b82be64b7-e2", "message": "Started.", "level": "INFO"}
2026-03-24T18:17:09.414707001Z Loading model: large-v3...
2026-03-24T18:17:09.850547007Z Error loading model large-v3: CUDA failed with error unknown error
2026-03-24T18:17:09.852111768Z {"requestId": "7be2dd5c-0506-44dd-9b1f-0b5b82be64b7-e2", "message": "Captured Handler Exception", "level": "ERROR"}
2026-03-24T18:17:09.852160919Z {"requestId": null, "message": "{\n    \"error_type\": \"<class 'ValueError'>\",\n    \"error_message\": \"Failed to load model large-v3: CUDA failed with error unknown error\",\n    \"error_traceback\": \"Traceback (most recent call last):\\n  File \\\"/predict.py\\\", line 99, in predict\\n    loaded_model = WhisperModel(\\n  File \\\"/usr/local/lib/python3.10/dist-packages/faster_whisper/transcribe.py\\\", line 634, in __init__\\n    self.model = ctranslate2.models.Whisper(\\nRuntimeError: CUDA failed with error unknown error\\n\\nThe above exception was the direct cause of the following exception:\\n\\nTraceback (most recent call last):\\n  File \\\"/usr/local/lib/python3.10/dist-packages/runpod/serverless/modules/rp_job.py\\\", line 182, in run_job\\n    handler_return = handler(job)\\n  File \\\"/usr/local/lib/python3.10/dist-packages/runpod/serverless/utils/rp_debugger.py\\\", line 168, in __call__\\n    result = self.function(*args, **kwargs)\\n  File \\\"/rp_handler.py\\\", line 72, in run_whisper_job\\n    whisper_results = MODEL.predict(\\n  File \\\"/predict.py\\\", line 109, in predict\\n    raise ValueError(f\\\"Failed to load model {model_name}: {e}\\\") from e\\nValueError: Failed to load model large-v3: CUDA failed with error unknown error\\n\",\n    \"hostname\": \"k25lu2qfzb8x8d-64411f21\",\n    \"worker_id\": \"k25lu2qfzb8x8d\",\n    \"runpod_version\": \"1.7.9\"\n}", "level": "ERROR"}
2026-03-24T18:17:10.071468263Z {"requestId": "7be2dd5c-0506-44dd-9b1f-0b5b82be64b7-e2", "message": "Finished.", "level": "INFO"}
2026-03-24T18:23:54.741636756Z connectionpool.py   :868  2026-03-24 18:23:54,741 Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='api.runpod.ai', port=443): Read timed out. (read timeout=8)")': /v2/kq8ruf9ce1qlwk/ping/k25lu2qfzb8x8d?gpu=NVIDIA+GeForce+RTX+5090&runpod_version=1.7.9