2025-10-26 23:08:20.763 | [10/27/25 06:08:20] INFO     Starting gunicorn 23.0.0                           
2025-10-26 23:08:20.764 | [10/27/25 06:08:20] INFO     Listening at: http://[::]:3003 (8)                 
2025-10-26 23:08:20.764 | [10/27/25 06:08:20] INFO     Using worker: immich_ml.config.CustomUvicornWorker 
2025-10-26 23:08:20.767 | [10/27/25 06:08:20] INFO     Booting worker with pid: 9                         
2025-10-26 23:08:21.756 | [10/27/25 06:08:21] INFO     Started server process [9]                         
2025-10-26 23:08:21.757 | [10/27/25 06:08:21] INFO     Waiting for application startup.                   
2025-10-26 23:08:21.757 | [10/27/25 06:08:21] INFO     Created in-memory cache with unloading after 300s  
2025-10-26 23:08:21.757 |                              of inactivity.                                     
2025-10-26 23:08:21.757 | [10/27/25 06:08:21] INFO     Initialized request thread pool with 32 threads.   
2025-10-26 23:08:21.758 | [10/27/25 06:08:21] INFO     Application startup complete.                      
2025-10-26 23:12:36.311 | [10/27/25 06:12:36] INFO     Downloading visual model 'ViT-B-32__openai'. This  
2025-10-26 23:12:36.311 |                              may take a while.                                  
2025-10-26 23:12:44.886 | 
2025-10-26 23:12:44.887 | [10/27/25 06:12:44] INFO     Loading visual model 'ViT-B-32__openai' to memory  
2025-10-26 23:12:44.888 | [10/27/25 06:12:44] INFO     Setting execution providers to                     
2025-10-26 23:12:44.888 |                              ['CUDAExecutionProvider', 'CPUExecutionProvider'], 
2025-10-26 23:12:44.888 |                              in descending order of preference                  
2025-10-26 23:12:45.523 | [10/27/25 06:12:45] INFO     Downloading detection model 'buffalo_l'. This may  
2025-10-26 23:12:45.523 |                              take a while.                                      
2025-10-26 23:12:50.599 | [10/27/25 06:12:50] INFO     Loading detection model 'buffalo_l' to memory      
2025-10-26 23:12:50.599 | 
2025-10-26 23:12:50.600 | [10/27/25 06:12:50] INFO     Setting execution providers to                     
2025-10-26 23:12:50.600 |                              ['CUDAExecutionProvider', 'CPUExecutionProvider'], 
2025-10-26 23:12:50.600 |                              in descending order of preference                  
2025-10-26 23:12:51.844 | [10/27/25 06:12:51] INFO     Loading recognition model 'buffalo_l' to memory    
2025-10-26 23:12:51.845 | [10/27/25 06:12:51] INFO     Setting execution providers to                     
2025-10-26 23:12:51.845 |                              ['CUDAExecutionProvider', 'CPUExecutionProvider'], 
2025-10-26 23:12:51.845 |                              in descending order of preference                  
2025-10-26 23:12:53.149 | [10/27/25 06:12:53] INFO     Setting execution providers to                     
2025-10-26 23:12:53.149 |                              ['CUDAExecutionProvider', 'CPUExecutionProvider'], 
2025-10-26 23:12:53.149 |                              in descending order of preference                  
2025-10-26 23:31:51.717 | [10/27/25 06:31:51] INFO     Shutting down due to inactivity.                   
2025-10-26 23:31:51.771 | [10/27/25 06:31:51] INFO     Shutting down                                      
2025-10-26 23:31:51.871 | [10/27/25 06:31:51] INFO     Waiting for application shutdown.                  
2025-10-26 23:31:51.907 | [10/27/25 06:31:51] INFO     Application shutdown complete.                     
2025-10-26 23:31:51.907 | [10/27/25 06:31:51] INFO     Finished server process [9]                        
2025-10-26 23:31:51.997 | [10/27/25 06:31:51] ERROR    Worker (pid:9) was sent SIGINT!                    
2025-10-26 23:31:52.000 | [10/27/25 06:31:51] INFO     Booting worker with pid: 585                       
2025-10-26 23:31:52.942 | [10/27/25 06:31:52] INFO     Started server process [585]                       
2025-10-26 23:31:52.942 | [10/27/25 06:31:52] INFO     Waiting for application startup.                   
2025-10-26 23:31:52.943 | [10/27/25 06:31:52] INFO     Created in-memory cache with unloading after 300s  
2025-10-26 23:31:52.943 |                              of inactivity.                                     
2025-10-26 23:31:52.944 | [10/27/25 06:31:52] INFO     Initialized request thread pool with 32 threads.   
2025-10-26 23:31:52.944 | [10/27/25 06:31:52] INFO     Application startup complete.                      
2025-10-26 23:34:04.506 | [10/27/25 06:34:04] INFO     Loading detection model 'buffalo_l' to memory      
2025-10-26 23:34:04.507 | [10/27/25 06:34:04] INFO     Setting execution providers to                     
2025-10-26 23:34:04.507 |                              ['CUDAExecutionProvider', 'CPUExecutionProvider'], 
2025-10-26 23:34:04.507 |                              in descending order of preference                  
2025-10-26 23:34:04.760 | [10/27/25 06:34:04] INFO     Loading visual model 'ViT-B-32__openai' to memory  
2025-10-26 23:34:04.766 | [10/27/25 06:34:04] INFO     Setting execution providers to                     
2025-10-26 23:34:04.766 |                              ['CUDAExecutionProvider', 'CPUExecutionProvider'], 
2025-10-26 23:34:04.766 |                              in descending order of preference                  
2025-10-26 23:34:06.526 | [10/27/25 06:34:06] INFO     Loading recognition model 'buffalo_l' to memory    
2025-10-26 23:34:06.527 | [10/27/25 06:34:06] INFO     Setting execution providers to                     
2025-10-26 23:34:06.527 |                              ['CUDAExecutionProvider', 'CPUExecutionProvider'], 
2025-10-26 23:34:06.527 |                              in descending order of preference                  
2025-10-27 00:11:22.888 | [10/27/25 07:11:22] INFO     Shutting down due to inactivity.                   
2025-10-27 00:11:22.894 | [10/27/25 07:11:22] INFO     Shutting down                                      
2025-10-27 00:11:22.995 | [10/27/25 07:11:22] INFO     Waiting for application shutdown.                  
2025-10-27 00:11:23.022 | [10/27/25 07:11:23] INFO     Application shutdown complete.                     
2025-10-27 00:11:23.023 | [10/27/25 07:11:23] INFO     Finished server process [585]                      
2025-10-27 00:11:23.091 | [10/27/25 07:11:23] ERROR    Worker (pid:585) was sent SIGINT!                  
2025-10-27 00:11:23.094 | [10/27/25 07:11:23] INFO     Booting worker with pid: 1311                      
2025-10-27 00:11:24.035 | [10/27/25 07:11:24] INFO     Started server process [1311]                      
2025-10-27 00:11:24.035 | [10/27/25 07:11:24] INFO     Waiting for application startup.                   
2025-10-27 00:11:24.036 | [10/27/25 07:11:24] INFO     Created in-memory cache with unloading after 300s  
2025-10-27 00:11:24.036 |                              of inactivity.                                     
2025-10-27 00:11:24.036 | [10/27/25 07:11:24] INFO     Initialized request thread pool with 32 threads.   
2025-10-27 00:11:24.037 | [10/27/25 07:11:24] INFO     Application startup complete.                      
2025-10-27 03:41:49.958 | [10/27/25 10:41:49] INFO     Loading visual model 'ViT-B-32__openai' to memory  
2025-10-27 03:41:49.982 | [10/27/25 10:41:49] INFO     Setting execution providers to                     
2025-10-27 03:41:49.982 |                              ['CUDAExecutionProvider', 'CPUExecutionProvider'], 
2025-10-27 03:41:49.982 |                              in descending order of preference                  
2025-10-27 03:41:50.994 | [10/27/25 10:41:50] INFO     Loading detection model 'buffalo_l' to memory      
2025-10-27 03:41:50.996 | [10/27/25 10:41:50] INFO     Setting execution providers to                     
2025-10-27 03:41:50.996 |                              ['CUDAExecutionProvider', 'CPUExecutionProvider'], 
2025-10-27 03:41:50.996 |                              in descending order of preference                  
2025-10-27 03:41:52.710 | [10/27/25 10:41:52] INFO     Loading recognition model 'buffalo_l' to memory    
2025-10-27 03:41:52.712 | [10/27/25 10:41:52] INFO     Setting execution providers to                     
2025-10-27 03:41:52.712 |                              ['CUDAExecutionProvider', 'CPUExecutionProvider'], 
2025-10-27 03:41:52.712 |                              in descending order of preference                  
2025-10-27 03:59:23.515 | [10/27/25 10:59:23] INFO     Shutting down due to inactivity.                   
2025-10-27 03:59:23.585 | [10/27/25 10:59:23] INFO     Shutting down                                      
2025-10-27 03:59:23.694 | [10/27/25 10:59:23] INFO     Waiting for application shutdown.                  
2025-10-27 03:59:25.273 | [10/27/25 10:59:25] INFO     Application shutdown complete.                     
2025-10-27 03:59:25.274 | [10/27/25 10:59:25] INFO     Finished server process [1311]                     
2025-10-27 03:59:25.426 | [10/27/25 10:59:25] ERROR    Worker (pid:1311) was sent SIGINT!                 
2025-10-27 03:59:25.454 | [10/27/25 10:59:25] INFO     Booting worker with pid: 4648                      
2025-10-27 03:59:27.059 | [10/27/25 10:59:27] INFO     Started server process [4648]                      
2025-10-27 03:59:27.059 | [10/27/25 10:59:27] INFO     Waiting for application startup.                   
2025-10-27 03:59:27.060 | [10/27/25 10:59:27] INFO     Created in-memory cache with unloading after 300s  
2025-10-27 03:59:27.060 |                              of inactivity.                                     
2025-10-27 03:59:27.060 | [10/27/25 10:59:27] INFO     Initialized request thread pool with 32 threads.   
2025-10-27 03:59:27.061 | [10/27/25 10:59:27] INFO     Application startup complete.                      
2025-10-27 06:06:41.165 | [10/27/25 13:06:41] INFO     Loading textual model 'ViT-B-32__openai' to memory 
2025-10-27 06:06:41.174 | [10/27/25 13:06:41] INFO     Setting execution providers to                     
2025-10-27 06:06:41.174 |                              ['CUDAExecutionProvider', 'CPUExecutionProvider'], 
2025-10-27 06:06:41.174 |                              in descending order of preference                  
2025-10-27 06:06:41.901 | 2025-10-27 13:06:41.900429010 [W:onnxruntime:, transformer_memcpy.cc:74 ApplyImpl] 2 Memcpy nodes are added to the graph main_graph for CUDAExecutionProvider. It might have negative impact on performance (including unable to run CUDA graph). Set session_options.log_severity_level=1 to see the detail logs before this message.
2025-10-27 06:11:46.761 | [10/27/25 13:11:46] INFO     Shutting down due to inactivity.                   
2025-10-27 06:11:46.773 | [10/27/25 13:11:46] INFO     Shutting down                                      
2025-10-27 06:11:46.885 | [10/27/25 13:11:46] INFO     Waiting for application shutdown.                  
2025-10-27 06:11:49.157 | [10/27/25 13:11:49] INFO     Application shutdown complete.                     
2025-10-27 06:11:49.157 | [10/27/25 13:11:49] INFO     Finished server process [4648]                     
2025-10-27 06:11:49.292 | [10/27/25 13:11:49] ERROR    Worker (pid:4648) was sent SIGINT!                 
2025-10-27 06:11:49.329 | [10/27/25 13:11:49] INFO     Booting worker with pid: 6603                      
2025-10-27 06:11:51.077 | [10/27/25 13:11:51] INFO     Started server process [6603]                      
2025-10-27 06:11:51.077 | [10/27/25 13:11:51] INFO     Waiting for application startup.                   
2025-10-27 06:11:51.078 | [10/27/25 13:11:51] INFO     Created in-memory cache with unloading after 300s  
2025-10-27 06:11:51.078 |                              of inactivity.                                     
2025-10-27 06:11:51.079 | [10/27/25 13:11:51] INFO     Initialized request thread pool with 32 threads.   
2025-10-27 06:11:51.079 | [10/27/25 13:11:51] INFO     Application startup complete.                      
2025-10-27 18:59:58.435 | Fetching 11 files:   0%|          | 0/11 [00:00<?, ?it/s]
2025-10-27 18:59:58.435 | Fetching 11 files:   9%|▉         | 1/11 [00:00<00:01,  8.20it/s]
2025-10-27 18:59:58.435 | Fetching 11 files:  45%|████▌     | 5/11 [00:08<00:10,  1.79s/it]
2025-10-27 18:59:58.435 | Fetching 11 files: 100%|██████████| 11/11 [00:08<00:00,  1.30it/s]
2025-10-27 18:59:58.435 | Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
2025-10-27 18:59:58.435 | Fetching 4 files:  25%|██▌       | 1/4 [00:00<00:00,  9.80it/s]
2025-10-27 18:59:58.435 | Fetching 4 files:  75%|███████▌  | 3/4 [00:01<00:00,  1.56it/s]
2025-10-27 18:59:58.435 | Fetching 4 files: 100%|██████████| 4/4 [00:04<00:00,  1.53s/it]
2025-10-27 18:59:58.435 | Fetching 4 files: 100%|██████████| 4/4 [00:04<00:00,  1.25s/it]
2025-10-27 19:00:33.995 | [10/28/25 02:00:33] INFO     Loading textual model 'ViT-B-32__openai' to memory 
2025-10-27 19:00:34.006 | [10/28/25 02:00:34] INFO     Setting execution providers to                     
2025-10-27 19:00:34.006 |                              ['CUDAExecutionProvider', 'CPUExecutionProvider'], 
2025-10-27 19:00:34.006 |                              in descending order of preference                  
2025-10-27 19:00:34.763 | 2025-10-28 02:00:34.763048352 [W:onnxruntime:, transformer_memcpy.cc:74 ApplyImpl] 2 Memcpy nodes are added to the graph main_graph for CUDAExecutionProvider. It might have negative impact on performance (including unable to run CUDA graph). Set session_options.log_severity_level=1 to see the detail logs before this message.
2025-10-27 19:05:42.633 | [10/28/25 02:05:42] INFO     Shutting down due to inactivity.                   
2025-10-27 19:05:42.698 | [10/28/25 02:05:42] INFO     Shutting down                                      
2025-10-27 19:05:42.811 | [10/28/25 02:05:42] INFO     Waiting for application shutdown.                  
2025-10-27 19:05:45.311 | [10/28/25 02:05:45] INFO     Application shutdown complete.                     
2025-10-27 19:05:45.312 | [10/28/25 02:05:45] INFO     Finished server process [6603]                     
2025-10-27 19:05:45.450 | [10/28/25 02:05:45] ERROR    Worker (pid:6603) was sent SIGINT!                 
2025-10-27 19:05:45.488 | [10/28/25 02:05:45] INFO     Booting worker with pid: 17134                     
2025-10-27 19:05:47.416 | [10/28/25 02:05:47] INFO     Started server process [17134]                     
2025-10-27 19:05:47.417 | [10/28/25 02:05:47] INFO     Waiting for application startup.                   
2025-10-27 19:05:47.417 | [10/28/25 02:05:47] INFO     Created in-memory cache with unloading after 300s  
2025-10-27 19:05:47.417 |                              of inactivity.                                     
2025-10-27 19:05:47.418 | [10/28/25 02:05:47] INFO     Initialized request thread pool with 32 threads.   
2025-10-27 19:05:47.418 | [10/28/25 02:05:47] INFO     Application startup complete.