immich_machine_learning  | [10/31/25 02:07:01] INFO     Starting gunicorn 23.0.0                           
immich_machine_learning  | [10/31/25 02:07:01] INFO     Listening at: http://[::]:3003 (8)                 
immich_machine_learning  | [10/31/25 02:07:01] INFO     Using worker: immich_ml.config.CustomUvicornWorker 
immich_machine_learning  | [10/31/25 02:07:01] INFO     Booting worker with pid: 9                         
immich_machine_learning  | [10/31/25 02:07:02] INFO     generated new fontManager                          
immich_machine_learning  | [10/31/25 02:07:03] INFO     Started server process [9]                         
immich_machine_learning  | [10/31/25 02:07:03] INFO     Waiting for application startup.                   
immich_machine_learning  | [10/31/25 02:07:03] INFO     Created in-memory cache with unloading after 300s  
immich_machine_learning  |                              of inactivity.                                     
immich_machine_learning  | [10/31/25 02:07:03] INFO     Initialized request thread pool with 16 threads.   
immich_machine_learning  | [10/31/25 02:07:03] INFO     Application startup complete.                      
immich_machine_learning  | [10/31/25 02:07:04] INFO     Loading detection model 'PP-OCRv5_mobile' to memory
immich_machine_learning  | [10/31/25 02:07:04] INFO     Setting execution providers to                     
immich_machine_learning  |                              ['CUDAExecutionProvider', 'CPUExecutionProvider'], 
immich_machine_learning  |                              in descending order of preference                  
immich_machine_learning  | [10/31/25 02:07:04] INFO     Using engine_name: onnxruntime                     
immich_machine_learning  | [10/31/25 02:07:08] INFO     Loading recognition model 'PP-OCRv5_mobile' to     
immich_machine_learning  |                              memory                                             
immich_machine_learning  | [10/31/25 02:07:08] INFO     Setting execution providers to                     
immich_machine_learning  |                              ['CUDAExecutionProvider', 'CPUExecutionProvider'], 
immich_machine_learning  |                              in descending order of preference                  
immich_machine_learning  | *************** EP Error ***************
immich_machine_learning  | EP Error /onnxruntime_src/onnxruntime/core/providers/cuda/cuda_call.cc:123 std::conditional_t<THRW, void, onnxruntime::common::Status> onnxruntime::CudaCall(ERRTYPE, const char*, const char*, ERRTYPE, const char*, const char*, int) [with ERRTYPE = cudaError; bool THRW = true; std::conditional_t<THRW, void, common::Status> = void] /onnxruntime_src/onnxruntime/core/providers/cuda/cuda_call.cc:116 std::conditional_t<THRW, void, onnxruntime::common::Status> onnxruntime::CudaCall(ERRTYPE, const char*, const char*, ERRTYPE, const char*, const char*, int) [with ERRTYPE = cudaError; bool THRW = true; std::conditional_t<THRW, void, common::Status> = void] CUDA failure 900: operation not permitted when stream is capturing ; GPU=0 ; hostname=487e143366f2 ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/cuda_execution_provider.cc ; line=283 ; expr=cudaDeviceSynchronize(); 
immich_machine_learning  | 
immich_machine_learning  |  when using ['CUDAExecutionProvider', 'CPUExecutionProvider']
immich_machine_learning  | Falling back to ['CPUExecutionProvider'] and retrying.
immich_machine_learning  | ****************************************
immich_machine_learning  | [10/31/25 02:07:09] INFO     Using engine_name: onnxruntime                     
immich_machine_learning  | [10/31/25 02:07:09] INFO     Initiating download:                               
immich_machine_learning  |                              https://www.modelscope.cn/models/RapidAI/RapidOCR/r
immich_machine_learning  |                              esolve/v3.4.0/resources/fonts/FZYTK.TTF            
immich_machine_learning  | [10/31/25 02:07:10] INFO     Download size: 3.09MB                              
immich_machine_learning  | [10/31/25 02:07:10] WARNING  File exists but is invalid, redownloading:         
immich_machine_learning  |                              /opt/venv/lib/python3.11/site-packages/rapidocr/mod
immich_machine_learning  |                              els/FZYTK.TTF                                      
immich_machine_learning  | [10/31/25 02:07:10] INFO     Initiating download:                               
immich_machine_learning  |                              https://www.modelscope.cn/models/RapidAI/RapidOCR/r
immich_machine_learning  |                              esolve/v3.4.0/resources/fonts/FZYTK.TTF            
immich_machine_learning  | [10/31/25 02:07:10] INFO     Successfully saved to:                             
immich_machine_learning  |                              /opt/venv/lib/python3.11/site-packages/rapidocr/mod
immich_machine_learning  |                              els/FZYTK.TTF                                      
immich_machine_learning  | 2025-10-31 02:07:11.045622014 [E:onnxruntime:, sequential_executor.cc:516 ExecuteKernel] Non-zero status code returned while running Conv node. Name:'Conv.0' Status Message: CUDA error cudaErrorStreamCaptureUnsupported:operation not permitted when stream is capturing
immich_machine_learning  | [10/31/25 02:07:11] ERROR    Exception in ASGI application