[09/03/25 07:14:06] INFO Starting gunicorn 23.0.0 [09/03/25 07:14:06] INFO Listening at: http://[::]:3003 (9) [09/03/25 07:14:06] INFO Using worker: immich_ml.config.CustomUvicornWorker [09/03/25 07:14:06] INFO Booting worker with pid: 10 [09/03/25 07:14:08] INFO Started server process [10] [09/03/25 07:14:08] INFO Waiting for application startup. [09/03/25 07:14:08] INFO Created in-memory cache with unloading after 300s of inactivity. [09/03/25 07:14:08] INFO Initialized request thread pool with 4 threads. [09/03/25 07:14:08] INFO Application startup complete. [09/03/25 07:17:02] INFO Downloading detection model 'buffalo_l'. This may take a while. [09/03/25 07:17:02] WARNING Failed to load detection model 'buffalo_l'. Clearing cache. [09/03/25 07:17:02] WARNING Attempted to clear cache for model 'buffalo_l', but cache directory does not exist [09/03/25 07:17:02] INFO Downloading detection model 'buffalo_l'. This may take a while. [09/03/25 07:17:02] ERROR Exception in ASGI application ╭─────── Traceback (most recent call last) ───────╮ │ /opt/venv/lib/python3.11/site-packages/urllib3/ │ │ connection.py:203 in _new_conn │ │ │ │ 200 │ │ :return: New socket connection. │ │ 201 │ │ """ │ │ 202 │ │ try: │ │ ❱ 203 │ │ │ sock = connection.create_conn │ │ 204 │ │ │ │ (self._dns_host, self.por │ │ 205 │ │ │ │ self.timeout, │ │ 206 │ │ │ │ source_address=self.sourc │ │ │ │ /opt/venv/lib/python3.11/site-packages/urllib3/ │ │ util/connection.py:85 in create_connection │ │ │ │ 82 │ │ │ 83 │ if err is not None: │ │ 84 │ │ try: │ │ ❱ 85 │ │ │ raise err │ │ 86 │ │ finally: │ │ 87 │ │ │ # Break explicitly a referenc │ │ 88 │ │ │ err = None │ │ │ │ /opt/venv/lib/python3.11/site-packages/urllib3/ │ │ util/connection.py:73 in create_connection │ │ │ │ 70 │ │ │ │ sock.settimeout(timeout) │ │ 71 │ │ │ if source_address: │ │ 72 │ │ │ │ sock.bind(source_address) │ │ ❱ 73 │ │ │ sock.connect(sa) │ │ 74 │ │ │ # Break explicitly a referenc │ │ 75 │ │ │ err = None │ │ 76 │ │ │ return sock │ ╰─────────────────────────────────────────────────╯ ConnectionRefusedError: [Errno 111] Connection refused The above exception was the direct cause of the following exception: ╭─────── Traceback (most recent call last) ───────╮ │ /opt/venv/lib/python3.11/site-packages/urllib3/ │ │ connectionpool.py:790 in urlopen │ │ │ │ 787 │ │ │ response_conn = conn if not │ │ 788 │ │ │ │ │ 789 │ │ │ # Make the request on the HT │ │ ❱ 790 │ │ │ response = self._make_reques │ │ 791 │ │ │ │ conn, │ │ 792 │ │ │ │ method, │ │ 793 │ │ │ │ url, │ │ │ │ /opt/venv/lib/python3.11/site-packages/urllib3/ │ │ connectionpool.py:491 in _make_request │ │ │ │ 488 │ │ │ │ new_e, (OSError, NewConn │ │ 489 │ │ │ ) and (conn and conn.proxy a │ │ 490 │ │ │ │ new_e = _wrap_proxy_erro │ │ ❱ 491 │ │ │ raise new_e │ │ 492 │ │ │ │ 493 │ │ # conn.request() calls http.clie │ │ 494 │ │ # urllib3.request. It also calls │ │ │ │ /opt/venv/lib/python3.11/site-packages/urllib3/ │ │ connectionpool.py:467 in _make_request │ │ │ │ 464 │ │ try: │ │ 465 │ │ │ # Trigger any extra validati │ │ 466 │ │ │ try: │ │ ❱ 467 │ │ │ │ self._validate_conn(conn │ │ 468 │ │ │ except (SocketTimeout, BaseS │ │ 469 │ │ │ │ self._raise_timeout(err= │ │ 470 │ │ │ │ raise │ │ │ │ /opt/venv/lib/python3.11/site-packages/urllib3/ │ │ connectionpool.py:1096 in _validate_conn │ │ │ │ 1093 │ │ │ │ 1094 │ │ # Force connect early to allow u │ │ 1095 │ │ if conn.is_closed: │ │ ❱ 1096 │ │ │ conn.connect() │ │ 1097 │ │ │ │ 1098 │ │ if not conn.is_verified: │ │ 1099 │ │ │ warnings.warn( │ │ │ │ /opt/venv/lib/python3.11/site-packages/urllib3/ │ │ connection.py:611 in connect │ │ │ │ 608 │ │ │ 609 │ def connect(self) -> None: │ │ 610 │ │ sock: socket.socket | ssl.SSLSock │ │ ❱ 611 │ │ self.sock = sock = self._new_conn │ │ 612 │ │ server_hostname: str = self.host │ │ 613 │ │ tls_in_tls = False │ │ 614 │ │ │ │ /opt/venv/lib/python3.11/site-packages/urllib3/ │ │ connection.py:218 in _new_conn │ │ │ │ 215 │ │ │ ) from e │ │ 216 │ │ │ │ 217 │ │ except OSError as e: │ │ ❱ 218 │ │ │ raise NewConnectionError( │ │ 219 │ │ │ │ self, f"Failed to establi │ │ 220 │ │ │ ) from e │ │ 221 │ ╰─────────────────────────────────────────────────╯ NewConnectionError: : Failed to establish a new connection: [Errno 111] Connection refused The above exception was the direct cause of the following exception: ╭─────── Traceback (most recent call last) ───────╮ │ /opt/venv/lib/python3.11/site-packages/requests │ │ /adapters.py:667 in send │ │ │ │ 664 │ │ │ timeout = TimeoutSauce(connec │ │ 665 │ │ │ │ 666 │ │ try: │ │ ❱ 667 │ │ │ resp = conn.urlopen( │ │ 668 │ │ │ │ method=request.method, │ │ 669 │ │ │ │ url=url, │ │ 670 │ │ │ │ body=request.body, │ │ │ │ /opt/venv/lib/python3.11/site-packages/urllib3/ │ │ connectionpool.py:844 in urlopen │ │ │ │ 841 │ │ │ elif isinstance(new_e, (OSEr │ │ 842 │ │ │ │ new_e = ProtocolError("C │ │ 843 │ │ │ │ │ ❱ 844 │ │ │ retries = retries.increment( │ │ 845 │ │ │ │ method, url, error=new_e │ │ 846 │ │ │ ) │ │ 847 │ │ │ retries.sleep() │ │ │ │ /opt/venv/lib/python3.11/site-packages/urllib3/ │ │ util/retry.py:515 in increment │ │ │ │ 512 │ │ │ │ 513 │ │ if new_retry.is_exhausted(): │ │ 514 │ │ │ reason = error or ResponseErr │ │ ❱ 515 │ │ │ raise MaxRetryError(_pool, ur │ │ ignore[arg-type] │ │ 516 │ │ │ │ 517 │ │ log.debug("Incremented Retry for │ │ 518 │ ╰─────────────────────────────────────────────────╯ MaxRetryError: HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /api/models/immich-app/buffalo_l/revision/main (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused')) During handling of the above exception, another exception occurred: ╭─────── Traceback (most recent call last) ───────╮ │ /opt/venv/lib/python3.11/site-packages/huggingf │ │ ace_hub/_snapshot_download.py:165 in │ │ snapshot_download │ │ │ │ 162 │ │ # try/except logic to handle diff │ │ 163 │ │ try: │ │ 164 │ │ │ # if we have internet connect │ │ ❱ 165 │ │ │ repo_info = api.repo_info(rep │ │ revision=revision) │ │ 166 │ │ except (requests.exceptions.SSLEr │ │ 167 │ │ │ # Actually raise for those su │ │ 168 │ │ │ raise │ │ │ │ /opt/venv/lib/python3.11/site-packages/huggingf │ │ ace_hub/utils/_validators.py:114 in _inner_fn │ │ │ │ 111 │ │ if check_use_auth_token: │ │ 112 │ │ │ kwargs = smoothly_deprecate_u │ │ has_token=has_token, kwargs=kwargs) │ │ 113 │ │ │ │ ❱ 114 │ │ return fn(*args, **kwargs) │ │ 115 │ │ │ 116 │ return _inner_fn # type: ignore │ │ 117 │ │ │ │ /opt/venv/lib/python3.11/site-packages/huggingf │ │ ace_hub/hf_api.py:2853 in repo_info │ │ │ │ 2850 │ │ │ method = self.space_info # │ │ 2851 │ │ else: │ │ 2852 │ │ │ raise ValueError("Unsupport │ │ ❱ 2853 │ │ return method( │ │ 2854 │ │ │ repo_id, │ │ 2855 │ │ │ revision=revision, │ │ 2856 │ │ │ token=token, │ │ │ │ /opt/venv/lib/python3.11/site-packages/huggingf │ │ ace_hub/utils/_validators.py:114 in _inner_fn │ │ │ │ 111 │ │ if check_use_auth_token: │ │ 112 │ │ │ kwargs = smoothly_deprecate_u │ │ has_token=has_token, kwargs=kwargs) │ │ 113 │ │ │ │ ❱ 114 │ │ return fn(*args, **kwargs) │ │ 115 │ │ │ 116 │ return _inner_fn # type: ignore │ │ 117 │ │ │ │ /opt/venv/lib/python3.11/site-packages/huggingf │ │ ace_hub/hf_api.py:2637 in model_info │ │ │ │ 2634 │ │ │ params["blobs"] = True │ │ 2635 │ │ if expand: │ │ 2636 │ │ │ params["expand"] = expand │ │ ❱ 2637 │ │ r = get_session().get(path, hea │ │ 2638 │ │ hf_raise_for_status(r) │ │ 2639 │ │ data = r.json() │ │ 2640 │ │ return ModelInfo(**data) │ │ │ │ /opt/venv/lib/python3.11/site-packages/requests │ │ /sessions.py:602 in get │ │ │ │ 599 │ │ """ │ │ 600 │ │ │ │ 601 │ │ kwargs.setdefault("allow_redirect │ │ ❱ 602 │ │ return self.request("GET", url, * │ │ 603 │ │ │ 604 │ def options(self, url, **kwargs): │ │ 605 │ │ r"""Sends a OPTIONS request. Retu │ │ │ │ /opt/venv/lib/python3.11/site-packages/requests │ │ /sessions.py:589 in request │ │ │ │ 586 │ │ │ "allow_redirects": allow_redi │ │ 587 │ │ } │ │ 588 │ │ send_kwargs.update(settings) │ │ ❱ 589 │ │ resp = self.send(prep, **send_kwa │ │ 590 │ │ │ │ 591 │ │ return resp │ │ 592 │ │ │ │ /opt/venv/lib/python3.11/site-packages/requests │ │ /sessions.py:703 in send │ │ │ │ 700 │ │ start = preferred_clock() │ │ 701 │ │ │ │ 702 │ │ # Send the request │ │ ❱ 703 │ │ r = adapter.send(request, **kwarg │ │ 704 │ │ │ │ 705 │ │ # Total elapsed time of the reque │ │ 706 │ │ elapsed = preferred_clock() - sta │ │ │ │ /opt/venv/lib/python3.11/site-packages/huggingf │ │ ace_hub/utils/_http.py:96 in send │ │ │ │ 93 │ │ if constants.HF_DEBUG: │ │ 94 │ │ │ logger.debug(f"Send: {_curlif │ │ 95 │ │ try: │ │ ❱ 96 │ │ │ return super().send(request, │ │ 97 │ │ except requests.RequestException │ │ 98 │ │ │ request_id = request.headers. │ │ 99 │ │ │ if request_id is not None: │ │ │ │ /opt/venv/lib/python3.11/site-packages/requests │ │ /adapters.py:700 in send │ │ │ │ 697 │ │ │ │ # This branch is for urll │ │ 698 │ │ │ │ raise SSLError(e, request │ │ 699 │ │ │ │ │ ❱ 700 │ │ │ raise ConnectionError(e, requ │ │ 701 │ │ │ │ 702 │ │ except ClosedPoolError as e: │ │ 703 │ │ │ raise ConnectionError(e, requ │ ╰─────────────────────────────────────────────────╯ ConnectionError: (MaxRetryError("HTTPSConnectionPool(host='huggingfa ce.co', port=443): Max retries exceeded with url: /api/models/immich-app/buffalo_l/revision/main (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused'))"), '(Request ID: 242d5e46-38f9-4304-af1d-f4645a96d3cf)') The above exception was the direct cause of the following exception: ╭─────── Traceback (most recent call last) ───────╮ │ /usr/src/immich_ml/main.py:177 in predict │ │ │ │ 174 │ │ inputs = text │ │ 175 │ else: │ │ 176 │ │ raise HTTPException(400, "Either │ │ ❱ 177 │ response = await run_inference(inputs │ │ 178 │ return ORJSONResponse(response) │ │ 179 │ │ 180 │ │ │ │ /usr/src/immich_ml/main.py:200 in run_inference │ │ │ │ 197 │ │ response[entry["task"]] = output │ │ 198 │ │ │ 199 │ without_deps, with_deps = entries │ │ ❱ 200 │ await asyncio.gather(*[_run_inference │ │ 201 │ if with_deps: │ │ 202 │ │ await asyncio.gather(*[_run_infer │ │ 203 │ if isinstance(payload, Image): │ │ │ │ /usr/src/immich_ml/main.py:194 in │ │ _run_inference │ │ │ │ 191 │ │ │ except KeyError: │ │ 192 │ │ │ │ message = f"Task {entry[' │ │ output of {dep}" │ │ 193 │ │ │ │ raise HTTPException(400, │ │ ❱ 194 │ │ model = await load(model) │ │ 195 │ │ output = await run(model.predict, │ │ 196 │ │ outputs[model.identity] = output │ │ 197 │ │ response[entry["task"]] = output │ │ │ │ /usr/src/immich_ml/main.py:242 in load │ │ │ │ 239 │ except (OSError, InvalidProtobuf, Bad │ │ 240 │ │ log.warning(f"Failed to load {mod │ │ '{model.model_name}'. Clearing cache.") │ │ 241 │ │ model.clear_cache() │ │ ❱ 242 │ │ return await run(_load, model) │ │ 243 │ │ 244 │ │ 245 async def idle_shutdown_task() -> None: │ │ │ │ /usr/local/lib/python3.11/concurrent/futures/th │ │ read.py:58 in run │ │ │ │ /usr/src/immich_ml/main.py:228 in _load │ │ │ │ 225 │ │ │ │ model.load() │ │ 226 │ │ │ except FileNotFoundError as e │ │ 227 │ │ │ │ if model.model_format == │ │ ❱ 228 │ │ │ │ │ raise e │ │ 229 │ │ │ │ log.warning( │ │ 230 │ │ │ │ │ f"{model.model_format │ │ '{model.model_name}' does not support it. │ │ 231 │ │ │ │ │ exc_info=e, │ │ │ │ /usr/src/immich_ml/main.py:225 in _load │ │ │ │ 222 │ │ │ raise HTTPException(500, f"Fa │ │ 223 │ │ with lock: │ │ 224 │ │ │ try: │ │ ❱ 225 │ │ │ │ model.load() │ │ 226 │ │ │ except FileNotFoundError as e │ │ 227 │ │ │ │ if model.model_format == │ │ 228 │ │ │ │ │ raise e │ │ │ │ /usr/src/immich_ml/models/base.py:51 in load │ │ │ │ 48 │ │ │ return │ │ 49 │ │ self.load_attempts += 1 │ │ 50 │ │ │ │ ❱ 51 │ │ self.download() │ │ 52 │ │ attempt = f"Attempt #{self.load_a │ │ else "Loading" │ │ 53 │ │ log.info(f"{attempt} {self.model_ │ │ '{self.model_name}' to memory") │ │ 54 │ │ self.session = self._load() │ │ │ │ /usr/src/immich_ml/models/base.py:44 in │ │ download │ │ │ │ 41 │ │ │ log.info( │ │ 42 │ │ │ │ f"Downloading {self.model │ │ '{self.model_name}'. This may take a whil │ │ 43 │ │ │ ) │ │ ❱ 44 │ │ │ self._download() │ │ 45 │ │ │ 46 │ def load(self) -> None: │ │ 47 │ │ if self.loaded: │ │ │ │ /usr/src/immich_ml/models/base.py:76 in │ │ _download │ │ │ │ 73 │ │ │ ModelFormat.RKNN: ["*.armnn"] │ │ 74 │ │ } │ │ 75 │ │ │ │ ❱ 76 │ │ snapshot_download( │ │ 77 │ │ │ f"immich-app/{clean_name(self │ │ 78 │ │ │ cache_dir=self.cache_dir, │ │ 79 │ │ │ local_dir=self.cache_dir, │ │ │ │ /opt/venv/lib/python3.11/site-packages/huggingf │ │ ace_hub/utils/_validators.py:114 in _inner_fn │ │ │ │ 111 │ │ if check_use_auth_token: │ │ 112 │ │ │ kwargs = smoothly_deprecate_u │ │ has_token=has_token, kwargs=kwargs) │ │ 113 │ │ │ │ ❱ 114 │ │ return fn(*args, **kwargs) │ │ 115 │ │ │ 116 │ return _inner_fn # type: ignore │ │ 117 │ │ │ │ /opt/venv/lib/python3.11/site-packages/huggingf │ │ ace_hub/_snapshot_download.py:248 in │ │ snapshot_download │ │ │ │ 245 │ │ │ raise api_call_error │ │ 246 │ │ else: │ │ 247 │ │ │ # Otherwise: most likely a co │ │ the user │ │ ❱ 248 │ │ │ raise LocalEntryNotFoundError │ │ 249 │ │ │ │ "An error happened while │ │ cannot find the appropriate" │ │ 250 │ │ │ │ " snapshot folder for the │ │ check your internet connection" │ │ 251 │ │ │ │ " and try again." │ ╰─────────────────────────────────────────────────╯ LocalEntryNotFoundError: An error happened while trying to locate the files on the Hub and we cannot find the appropriate snapshot folder for the specified revision on the local disk. Please check your internet connection and try again.