Falling back to ['CPUExecutionProvider'] and retrying.
****************************************
[10/01/25 23:56:50] INFO     Downloading visual model 'ViT-B-32__openai'. This  
                             may take a while.                                  

Fetching 11 files:   0%|          | 0/11 [00:00<?, ?it/s]
Fetching 11 files:   9%|▉         | 1/11 [00:00<00:03,  3.00it/s]
Fetching 11 files:  45%|████▌     | 5/11 [00:41<00:53,  8.87s/it]
Fetching 11 files: 100%|██████████| 11/11 [00:41<00:00,  3.80s/it]
[10/01/25 23:57:33] INFO     Loading visual model 'ViT-B-32__openai' to memory  
[10/01/25 23:57:45] INFO     Setting execution providers to                     
                             ['CUDAExecutionProvider', 'CPUExecutionProvider'], 
                             in descending order of preference                  
2025-10-01 23:57:48.088136348 [E:onnxruntime:Default, cuda_call.cc:118 CudaCall] CUDA failure 100: no CUDA-capable device is detected ; GPU=-1 ; hostname=f47cf558b37d ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/cuda_execution_provider_info.cc ; line=66 ; expr=cudaGetDeviceCount(&num_devices); 
*************** EP Error ***************
EP Error /onnxruntime_src/onnxruntime/core/providers/cuda/cuda_execution_provider_info.cc:59 static onnxruntime::CUDAExecutionProviderInfo onnxruntime::CUDAExecutionProviderInfo::FromProviderOptions(const onnxruntime::ProviderOptions&) [ONNXRuntimeError] : 1 : FAIL : provider_options_utils.h:151 Parse Failed to parse provider option "device_id": CUDA failure 100: no CUDA-capable device is detected ; GPU=-1 ; hostname=f47cf558b37d ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/cuda_execution_provider_info.cc ; line=66 ; expr=cudaGetDeviceCount(&num_devices); 
 when using ['CUDAExecutionProvider', 'CPUExecutionProvider']
Falling back to ['CPUExecutionProvider'] and retrying.
****************************************
[10/01/25 23:57:50] INFO     Loading recognition model 'buffalo_l' to memory    
[10/01/25 23:57:50] INFO     Setting execution providers to                     
                             ['CUDAExecutionProvider', 'CPUExecutionProvider'], 
                             in descending order of preference                  
2025-10-01 23:57:51.054065989 [E:onnxruntime:Default, cuda_call.cc:118 CudaCall] CUDA failure 100: no CUDA-capable device is detected ; GPU=-1 ; hostname=f47cf558b37d ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/cuda_execution_provider_info.cc ; line=66 ; expr=cudaGetDeviceCount(&num_devices); 
*************** EP Error ***************
EP Error /onnxruntime_src/onnxruntime/core/providers/cuda/cuda_execution_provider_info.cc:59 static onnxruntime::CUDAExecutionProviderInfo onnxruntime::CUDAExecutionProviderInfo::FromProviderOptions(const onnxruntime::ProviderOptions&) [ONNXRuntimeError] : 1 : FAIL : provider_options_utils.h:151 Parse Failed to parse provider option "device_id": CUDA failure 100: no CUDA-capable device is detected ; GPU=-1 ; hostname=f47cf558b37d ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/cuda_execution_provider_info.cc ; line=66 ; expr=cudaGetDeviceCount(&num_devices); 
 when using ['CUDAExecutionProvider', 'CPUExecutionProvider']
Falling back to ['CPUExecutionProvider'] and retrying.
****************************************
[10/01/25 23:58:05] INFO     Setting execution providers to                     
                             ['CUDAExecutionProvider', 'CPUExecutionProvider'], 
                             in descending order of preference                  
*************** EP Error ***************
EP Error /onnxruntime_src/onnxruntime/core/providers/cuda/cuda_execution_provider_info.cc:59 static onnxruntime::CUDAExecutionProviderInfo onnxruntime::CUDAExecutionProviderInfo::FromProviderOptions(const onnxruntime::ProviderOptions&) [ONNXRuntimeError] : 1 : FAIL : provider_options_utils.h:151 Parse Failed to parse provider option "device_id": CUDA failure 100: no CUDA-capable device is detected ; GPU=-1 ; hostname=f47cf558b37d ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/cuda_execution_provider_info.cc ; line=66 ; expr=cudaGetDeviceCount(&num_devices); 
 when using ['CUDAExecutionProvider', 'CPUExecutionProvider']
Falling back to ['CPUExecutionProvider'] and retrying.
****************************************
2025-10-01 23:58:05.721270399 [E:onnxruntime:Default, cuda_call.cc:118 CudaCall] CUDA failure 100: no CUDA-capable device is detected ; GPU=-1 ; hostname=f47cf558b37d ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/cuda_execution_provider_info.cc ; line=66 ; expr=cudaGetDeviceCount(&num_devices); 
[10/02/25 00:57:30] INFO     Shutting down due to inactivity.                   
[10/02/25 00:57:30] INFO     Shutting down                                      
[10/02/25 00:57:30] INFO     Waiting for application shutdown.                  
[10/02/25 00:57:32] INFO     Application shutdown complete.                     
[10/02/25 00:57:32] INFO     Finished server process [8]                        
[10/02/25 00:57:33] ERROR    Worker (pid:8) was sent SIGINT!                    
[10/02/25 00:57:33] INFO     Booting worker with pid: 1175                      
[10/02/25 00:57:39] INFO     Started server process [1175]                      
[10/02/25 00:57:39] INFO     Waiting for application startup.                   
[10/02/25 00:57:39] INFO     Created in-memory cache with unloading after 300s  
                             of inactivity.                                     
[10/02/25 00:57:39] INFO     Initialized request thread pool with 4 threads.    
[10/02/25 00:57:39] INFO     Application startup complete.                      
[10/02/25 14:53:56] INFO     Starting gunicorn 23.0.0                           
[10/02/25 14:53:56] INFO     Listening at: http://[::]:3003 (7)                 
[10/02/25 14:53:56] INFO     Using worker: immich_ml.config.CustomUvicornWorker 
[10/02/25 14:53:56] INFO     Booting worker with pid: 8                         
[10/02/25 14:54:00] INFO     Started server process [8]                         
[10/02/25 14:54:00] INFO     Waiting for application startup.                   
[10/02/25 14:54:00] INFO     Created in-memory cache with unloading after 300s  
                             of inactivity.                                     
[10/02/25 14:54:00] INFO     Initialized request thread pool with 4 threads.    
[10/02/25 14:54:00] INFO     Application startup complete.                      
[10/02/25 14:54:30] INFO     Loading visual model 'ViT-B-32__openai' to memory  
[10/02/25 14:54:30] INFO     Setting execution providers to                     
                             ['CUDAExecutionProvider', 'CPUExecutionProvider'], 
                             in descending order of preference                  
2025-10-02 14:54:32.845955651 [E:onnxruntime:Default, cuda_call.cc:118 CudaCall] CUDA failure 100: no CUDA-capable device is detected ; GPU=-1 ; hostname=f47cf558b37d ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/cuda_execution_provider_info.cc ; line=66 ; expr=cudaGetDeviceCount(&num_devices); 
*************** EP Error ***************
EP Error /onnxruntime_src/onnxruntime/core/providers/cuda/cuda_execution_provider_info.cc:59 static onnxruntime::CUDAExecutionProviderInfo onnxruntime::CUDAExecutionProviderInfo::FromProviderOptions(const onnxruntime::ProviderOptions&) [ONNXRuntimeError] : 1 : FAIL : provider_options_utils.h:151 Parse Failed to parse provider option "device_id": CUDA failure 100: no CUDA-capable device is detected ; GPU=-1 ; hostname=f47cf558b37d ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/cuda_execution_provider_info.cc ; line=66 ; expr=cudaGetDeviceCount(&num_devices); 
 when using ['CUDAExecutionProvider', 'CPUExecutionProvider']
Falling back to ['CPUExecutionProvider'] and retrying.
****************************************
[10/02/25 14:54:35] INFO     Loading detection model 'buffalo_l' to memory      
[10/02/25 14:54:35] INFO     Setting execution providers to                     
                             ['CUDAExecutionProvider', 'CPUExecutionProvider'], 
                             in descending order of preference                  
2025-10-02 14:54:35.198689364 [E:onnxruntime:Default, cuda_call.cc:118 CudaCall] CUDA failure 100: no CUDA-capable device is detected ; GPU=-1 ; hostname=f47cf558b37d ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/cuda_execution_provider_info.cc ; line=66 ; expr=cudaGetDeviceCount(&num_devices); 
*************** EP Error ***************
EP Error /onnxruntime_src/onnxruntime/core/providers/cuda/cuda_execution_provider_info.cc:59 static onnxruntime::CUDAExecutionProviderInfo onnxruntime::CUDAExecutionProviderInfo::FromProviderOptions(const onnxruntime::ProviderOptions&) [ONNXRuntimeError] : 1 : FAIL : provider_options_utils.h:151 Parse Failed to parse provider option "device_id": CUDA failure 100: no CUDA-capable device is detected ; GPU=-1 ; hostname=f47cf558b37d ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/cuda_execution_provider_info.cc ; line=66 ; expr=cudaGetDeviceCount(&num_devices); 
 when using ['CUDAExecutionProvider', 'CPUExecutionProvider']
Falling back to ['CPUExecutionProvider'] and retrying.
****************************************
[10/02/25 14:54:36] INFO     Loading recognition model 'buffalo_l' to memory    
[10/02/25 14:54:36] INFO     Setting execution providers to                     
                             ['CUDAExecutionProvider', 'CPUExecutionProvider'], 
                             in descending order of preference                  
2025-10-02 14:54:36.851118234 [E:onnxruntime:Default, cuda_call.cc:118 CudaCall] CUDA failure 100: no CUDA-capable device is detected ; GPU=-1 ; hostname=f47cf558b37d ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/cuda_execution_provider_info.cc ; line=66 ; expr=cudaGetDeviceCount(&num_devices); 
*************** EP Error ***************
EP Error /onnxruntime_src/onnxruntime/core/providers/cuda/cuda_execution_provider_info.cc:59 static onnxruntime::CUDAExecutionProviderInfo onnxruntime::CUDAExecutionProviderInfo::FromProviderOptions(const onnxruntime::ProviderOptions&) [ONNXRuntimeError] : 1 : FAIL : provider_options_utils.h:151 Parse Failed to parse provider option "device_id": CUDA failure 100: no CUDA-capable device is detected ; GPU=-1 ; hostname=f47cf558b37d ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/cuda_execution_provider_info.cc ; line=66 ; expr=cudaGetDeviceCount(&num_devices); 
 when using ['CUDAExecutionProvider', 'CPUExecutionProvider']
Falling back to ['CPUExecutionProvider'] and retrying.
****************************************
[10/02/25 14:59:40] INFO     Shutting down due to inactivity.                   
[10/02/25 14:59:40] INFO     Shutting down                                      
[10/02/25 14:59:40] INFO     Waiting for application shutdown.                  
[10/02/25 14:59:40] INFO     Application shutdown complete.                     
[10/02/25 14:59:40] INFO     Finished server process [8]                        
[10/02/25 14:59:41] ERROR    Worker (pid:8) was sent SIGINT!                    
[10/02/25 14:59:41] INFO     Booting worker with pid: 102                       
[10/02/25 14:59:45] INFO     Started server process [102]                       
[10/02/25 14:59:45] INFO     Waiting for application startup.                   
[10/02/25 14:59:45] INFO     Created in-memory cache with unloading after 300s  
                             of inactivity.                                     
[10/02/25 14:59:45] INFO     Initialized request thread pool with 4 threads.    
[10/02/25 14:59:45] INFO     Application startup complete.