2025-11-05 22:50:48.462719171 [E:onnxruntime:, sequential_executor.cc:572 ExecuteKernel] Non-zero status code returned while running OpenVINO-EP-subgraph_3 node. Name:'OpenVINOExecutionProvider_OpenVINO-EP-subgraph_3_0' Status Message: /onnxruntime/onnxruntime/core/providers/openvino/ov_interface.cc:24 auto onnxruntime::openvino_ep::OvExceptionBoundary(Func&&, std::format_string<_Args ...>&&, Args&& ...) [with Func = OVInferRequest::Infer()::<lambda()>; Args = {}; std::format_string<_Args ...> = std::basic_format_string<char>] [OpenVINO-EP] In Error Couldn't start Inference: Exception from src/inference/src/cpp/infer_request.cpp:223:
Exception from src/plugins/intel_gpu/src/runtime/ocl/ocl_memory.cpp:576:
[GPU] clWaitForEvents, error code: -58 CL_INVALID_EVENT



[11/05/25 22:50:48] ERROR    Exception in ASGI application

                             ╭─────── Traceback (most recent call last) ───────╮
                             │ /usr/src/immich_ml/main.py:177 in predict       │
                             │                                                 │
                             │   174 │   │   inputs = text                     │
                             │   175 │   else:                                 │
                             │   176 │   │   raise HTTPException(400, "Either  │
                             │ ❱ 177 │   response = await run_inference(inputs │
                             │   178 │   return ORJSONResponse(response)       │
                             │   179                                           │
                             │   180                                           │
                             │                                                 │
                             │ /usr/src/immich_ml/main.py:202 in run_inference │
                             │                                                 │
                             │   199 │   │   response[entry["task"]] = output  │
                             │   200 │                                         │
                             │   201 │   without_deps, with_deps = entries     │
                             │ ❱ 202 │   await asyncio.gather(*[_run_inference │
                             │   203 │   if with_deps:                         │
                             │   204 │   │   await asyncio.gather(*[_run_infer │
                             │   205 │   if isinstance(payload, Image):        │
                             │                                                 │
                             │ /usr/src/immich_ml/main.py:197 in               │
                             │ _run_inference                                  │
                             │                                                 │
                             │   194 │   │   │   │   message = f"Task {entry[' │
                             │       output of {dep}"                          │
                             │   195 │   │   │   │   raise HTTPException(400,  │
                             │   196 │   │   model = await load(model)         │
                             │ ❱ 197 │   │   output = await run(model.predict, │
                             │   198 │   │   outputs[model.identity] = output  │
                             │   199 │   │   response[entry["task"]] = output  │
                             │   200                                           │
                             │                                                 │
                             │ /usr/src/immich_ml/main.py:215 in run           │
                             │                                                 │
                             │   212 │   if thread_pool is None:               │
                             │   213 │   │   return func(*args, **kwargs)      │
                             │   214 │   partial_func = partial(func, *args, * │
                             │ ❱ 215 │   return await asyncio.get_running_loop │
                             │   216                                           │
                             │   217                                           │
                             │   218 async def load(model: InferenceModel) ->  │
                             │                                                 │
                             │ /usr/local/lib/python3.13/concurrent/futures/th │
                             │ read.py:59 in run                               │
                             │                                                 │
                             │ /usr/src/immich_ml/models/base.py:60 in predict │
                             │                                                 │
                             │    57 │   │   self.load()                       │
                             │    58 │   │   if model_kwargs:                  │
                             │    59 │   │   │   self.configure(**model_kwargs │
                             │ ❱  60 │   │   return self._predict(*inputs)     │
                             │    61 │                                         │
                             │    62 │   @abstractmethod                       │
                             │    63 │   def _predict(self, *inputs: Any, **mo │
                             │                                                 │
                             │ /usr/src/immich_ml/models/ocr/detection.py:70   │
                             │ in _predict                                     │
                             │                                                 │
                             │    67 │   │   w, h = inputs.size                │
                             │    68 │   │   if w < 32 or h < 32:              │
                             │    69 │   │   │   return self._empty            │
                             │ ❱  70 │   │   out = self.session.run(None, {"x" │
                             │    71 │   │   boxes, scores = self.postprocess( │
                             │    72 │   │   if len(boxes) == 0:               │
                             │    73 │   │   │   return self._empty            │
                             │                                                 │
                             │ /usr/src/immich_ml/sessions/ort.py:52 in run    │
                             │                                                 │
                             │    49 │   │   input_feed: dict[str, NDArray[np. │
                             │    50 │   │   run_options: Any = None,          │
                             │    51 │   ) -> list[NDArray[np.float32]]:       │
                             │ ❱  52 │   │   outputs: list[NDArray[np.float32] │
                             │       run_options)                              │
                             │    53 │   │   return outputs                    │
                             │    54 │                                         │
                             │    55 │   @property                             │
                             │                                                 │
                             │ /opt/venv/lib/python3.13/site-packages/onnxrunt │
                             │ ime/capi/onnxruntime_inference_collection.py:27 │
                             │ 5 in run                                        │
                             │                                                 │
                             │    272 │   │   if not output_names:             │
                             │    273 │   │   │   output_names = [output.name  │
                             │    274 │   │   try:                             │
                             │ ❱  275 │   │   │   return self._sess.run(output │
                             │    276 │   │   except C.EPFail as err:          │
                             │    277 │   │   │   if self._enable_fallback:    │
                             │    278 │   │   │   │   print(f"EP Error: {err!s │
                             ╰─────────────────────────────────────────────────╯
                             Fail: [ONNXRuntimeError] : 1 : FAIL : Non-zero
                             status code returned while running
                             OpenVINO-EP-subgraph_3 node.
                             Name:'OpenVINOExecutionProvider_OpenVINO-EP-subgrap
                             h_3_0' Status Message:
                             /onnxruntime/onnxruntime/core/providers/openvino/ov
                             _interface.cc:24 auto
                             onnxruntime::openvino_ep::OvExceptionBoundary(Func&
                             &, std::format_string<_Args ...>&&, Args&& ...)
                             [with Func = OVInferRequest::Infer()::<lambda()>;
                             Args = {}; std::format_string<_Args ...> =
                             std::basic_format_string<char>] [OpenVINO-EP] In
                             Error Couldn't start Inference: Exception from
                             src/inference/src/cpp/infer_request.cpp:223:
                             Exception from
                             src/plugins/intel_gpu/src/runtime/ocl/ocl_memory.cp
                             p:576:
                             [GPU] clWaitForEvents, error code: -58
                             CL_INVALID_EVENT