Starting microservices worker [Nest] 716 - 11/12/2025, 9:59:26 AM LOG [Microservices:WebsocketRepository] Initialized websocket server [Nest] 716 - 11/12/2025, 9:59:26 AM WARN [Microservices:DatabaseService] DEPRECATION WARNING: The pgvecto.rs extension is deprecated and support for it will be removed very soon. See https://docs.immich.app/install/upgrading#migrating-to-vectorchord in order to switch to the VectorChord extension instead. [Nest] 716 - 11/12/2025, 9:59:26 AM LOG [Microservices:StorageService] Verifying system mount folder checks, current state: {"mountChecks":{"thumbs":true,"upload":true,"backups":true,"library":true,"profile":true,"encoded-video":true}} [Nest] 716 - 11/12/2025, 9:59:26 AM LOG [Microservices:StorageService] Successfully verified system mount folder checks [Nest] 716 - 11/12/2025, 9:59:26 AM LOG [Microservices:MetadataService] Bootstrapping metadata service [Nest] 716 - 11/12/2025, 9:59:26 AM LOG [Microservices:MetadataService] Initializing metadata service [Nest] 716 - 11/12/2025, 9:59:26 AM LOG [Microservices:MapRepository] Initializing metadata repository [Nest] 716 - 11/12/2025, 9:59:26 AM LOG [Microservices:MetadataService] Initialized local reverse geocoder [Nest] 716 - 11/12/2025, 9:59:26 AM LOG [Microservices:ServerService] Feature Flags: { "smartSearch": true, "facialRecognition": true, "duplicateDetection": true, "map": true, "reverseGeocoding": true, "importFaces": false, "sidecar": true, "search": true, "trash": true, "oauth": false, "oauthAutoLaunch": false, "ocr": true, "passwordLogin": true, "configFile": false, "email": false } [Nest] 716 - 11/12/2025, 9:59:26 AM LOG [Microservices:SystemConfigService] LogLevel=log (set via system config) [Nest] 716 - 11/12/2025, 9:59:26 AM LOG [Microservices:MachineLearningRepository] Machine learning server became unhealthy (http://127.0.0.1:3003). [Nest] 716 - 11/12/2025, 9:59:26 AM LOG [Microservices:MachineLearningRepository] Machine learning server became unhealthy (http://192.168.2.10:3003). [Nest] 716 - 11/12/2025, 9:59:26 AM LOG [Microservices:LibraryService] Starting to watch library 860e26e6-dbbb-4e50-b21f-c079c8044b90 with import path(s) /libraries/Xavier [Nest] 716 - 11/12/2025, 9:59:26 AM WARN [Microservices:MachineLearningRepository] Machine learning request to "http://127.0.0.1:3003" failed: fetch failed [Nest] 716 - 11/12/2025, 9:59:26 AM WARN [Microservices:MachineLearningRepository] Machine learning request to "http://127.0.0.1:3003" failed: fetch failed [Nest] 716 - 11/12/2025, 9:59:26 AM WARN [Microservices:MachineLearningRepository] Machine learning request to "http://192.168.2.10:3003" failed: fetch failed [Nest] 716 - 11/12/2025, 9:59:26 AM WARN [Microservices:MachineLearningRepository] Machine learning request to "http://192.168.2.10:3003" failed: fetch failed [11/12/25 09:59:26] INFO Starting gunicorn 23.0.0 [11/12/25 09:59:26] INFO Listening at: http://0.0.0.0:3003 (767) [11/12/25 09:59:26] INFO Using worker: immich_ml.config.CustomUvicornWorker [11/12/25 09:59:26] INFO Booting worker with pid: 768 [Nest] 716 - 11/12/2025, 9:59:27 AM ERROR [Microservices:{"source":"upload","id":"21f0ca9e-c75a-4974-8d5a-7bd4bae82e58"}] Unable to run job handler (AssetDetectFaces): Error: Machine learning request '{"facial-recognition":{"detection":{"modelName":"buffalo_s","options":{"minScore":0.7}},"recognition":{"modelName":"buffalo_s"}}}' failed for all URLs Error: Machine learning request '{"facial-recognition":{"detection":{"modelName":"buffalo_s","options":{"minScore":0.7}},"recognition":{"modelName":"buffalo_s"}}}' failed for all URLs at MachineLearningRepository.predict (/app/immich/server/dist/repositories/machine-learning.repository.js:117:15) at process.processTicksAndRejections (node:internal/process/task_queues:105:5) at async MachineLearningRepository.detectFaces (/app/immich/server/dist/repositories/machine-learning.repository.js:126:26) at async PersonService.handleDetectFaces (/app/immich/server/dist/services/person.service.js:243:52) at async JobService.onJobRun (/app/immich/server/dist/services/job.service.js:199:30) at async EventRepository.onEvent (/app/immich/server/dist/repositories/event.repository.js:91:13) at async /app/immich/server/node_modules/.pnpm/bullmq@5.61.2/node_modules/bullmq/dist/cjs/classes/worker.js:528:32 [Nest] 716 - 11/12/2025, 9:59:27 AM ERROR [Microservices:{"source":"upload","id":"8c6e828c-7cb1-4f5e-b9b3-601c7823b14f"}] Unable to run job handler (SmartSearch): Error: Machine learning request '{"clip":{"visual":{"modelName":"ViT-B-32__openai"}}}' failed for all URLs Error: Machine learning request '{"clip":{"visual":{"modelName":"ViT-B-32__openai"}}}' failed for all URLs at MachineLearningRepository.predict (/app/immich/server/dist/repositories/machine-learning.repository.js:117:15) at process.processTicksAndRejections (node:internal/process/task_queues:105:5) at async MachineLearningRepository.encodeImage (/app/immich/server/dist/repositories/machine-learning.repository.js:135:26) at async SmartInfoService.handleEncodeClip (/app/immich/server/dist/services/smart-info.service.js:91:27) at async JobService.onJobRun (/app/immich/server/dist/services/job.service.js:199:30) at async EventRepository.onEvent (/app/immich/server/dist/repositories/event.repository.js:91:13) at async /app/immich/server/node_modules/.pnpm/bullmq@5.61.2/node_modules/bullmq/dist/cjs/classes/worker.js:528:32 [11/12/25 09:59:37] INFO generated new fontManager [11/12/25 09:59:37] INFO Started server process [768] [11/12/25 09:59:37] INFO Waiting for application startup. [11/12/25 09:59:37] INFO Created in-memory cache with unloading after 300s of inactivity. [11/12/25 09:59:37] INFO Initialized request thread pool with 12 threads. [11/12/25 09:59:37] INFO Application startup complete. [11/12/25 09:59:38] INFO Loading visual model 'ViT-B-32__openai' to memory [11/12/25 09:59:38] INFO Setting execution providers to ['CPUExecutionProvider'], in descending order of preference [Nest] 716 - 11/12/2025, 9:59:41 AM LOG [Microservices:NestFactory] Starting Nest application... [Nest] 716 - 11/12/2025, 9:59:41 AM LOG [Microservices:InstanceLoader] BullModule dependencies initialized [Nest] 716 - 11/12/2025, 9:59:41 AM LOG [Microservices:InstanceLoader] ClsModule dependencies initialized [Nest] 716 - 11/12/2025, 9:59:41 AM LOG [Microservices:InstanceLoader] ClsCommonModule dependencies initialized [Nest] 716 - 11/12/2025, 9:59:41 AM LOG [Microservices:InstanceLoader] OpenTelemetryModule dependencies initialized [Nest] 716 - 11/12/2025, 9:59:41 AM LOG [Microservices:InstanceLoader] KyselyModule$1 dependencies initialized [Nest] 716 - 11/12/2025, 9:59:41 AM LOG [Microservices:InstanceLoader] KyselyCoreModule$1 dependencies initialized [Nest] 716 - 11/12/2025, 9:59:41 AM LOG [Microservices:InstanceLoader] DiscoveryModule dependencies initialized [Nest] 716 - 11/12/2025, 9:59:41 AM LOG [Microservices:InstanceLoader] OpenTelemetryCoreModule dependencies initialized [Nest] 716 - 11/12/2025, 9:59:41 AM LOG [Microservices:InstanceLoader] ClsRootModule dependencies initialized [Nest] 716 - 11/12/2025, 9:59:41 AM LOG [Microservices:InstanceLoader] BullModule dependencies initialized [Nest] 716 - 11/12/2025, 9:59:41 AM LOG [Microservices:InstanceLoader] BullModule dependencies initialized [Nest] 716 - 11/12/2025, 9:59:41 AM LOG [Microservices:InstanceLoader] MicroservicesModule dependencies initialized [Nest] 716 - 11/12/2025, 9:59:41 AM LOG [Microservices:NestApplication] Nest application successfully started [Nest] 716 - 11/12/2025, 9:59:41 AM LOG [Microservices:Bootstrap] Immich Microservices is running [v2.2.3] [production] [11/12/25 09:59:42] INFO Loading detection model 'buffalo_s' to memory [11/12/25 09:59:42] INFO Setting execution providers to ['CPUExecutionProvider'], in descending order of preference [Nest] 716 - 11/12/2025, 9:59:42 AM LOG [Microservices:MachineLearningRepository] Machine learning server became healthy (http://127.0.0.1:3003). [11/12/25 09:59:42] INFO Loading recognition model 'buffalo_s' to memory [11/12/25 09:59:42] INFO Setting execution providers to ['CPUExecutionProvider'], in descending order of preference 2025-11-12 09:59:43.593238442 [W:onnxruntime:, execution_frame.cc:876 VerifyOutputSizes] Expected shape from model of {1,512} does not match actual shape of {4,512} for output 516 2025-11-12 09:59:43.597051776 [W:onnxruntime:, execution_frame.cc:876 VerifyOutputSizes] Expected shape from model of {1,512} does not match actual shape of {3,512} for output 516 2025-11-12 09:59:43.612899077 [W:onnxruntime:, execution_frame.cc:876 VerifyOutputSizes] Expected shape from model of {1,512} does not match actual shape of {5,512} for output 516 [Nest] 696 - 11/12/2025, 9:59:52 AM LOG [Api:MachineLearningRepository] Machine learning server became healthy (http://127.0.0.1:3003).