MIOpen Error: /long_pathname_so_that_rpms_can_package_the_debug_info/src/MLOpen/src/ocl/convolutionocl.cpp:869: Buffers cannot be NULL
2025-12-12 08:46:02.241325270 [E:onnxruntime:Default, rocm_call.cc:119 RocmCall] MIOPEN failure 3: miopenStatusBadParm ; GPU=0 ; hostname=10d6bb49b9d7 ; file=/code/onnxruntime/onnxruntime/core/providers/rocm/nn/conv_transpose.cc ; line=133 ; expr=miopenFindConvolutionBackwardDataAlgorithm( GetMiopenHandle(context), s_.x_tensor, x_data, s_.w_desc, w_data, s_.conv_desc, s_.y_tensor, y_data, 1, &algo_count, &perf, algo_search_workspace.get(), AlgoSearchWorkspaceSize, false); 
2025-12-12 08:46:02.241334688 [E:onnxruntime:, sequential_executor.cc:516 ExecuteKernel] Non-zero status code returned while running ConvTranspose node. Name:'ConvTranspose.0' Status Message: MIOPEN failure 3: miopenStatusBadParm ; GPU=0 ; hostname=10d6bb49b9d7 ; file=/code/onnxruntime/onnxruntime/core/providers/rocm/nn/conv_transpose.cc ; line=133 ; expr=miopenFindConvolutionBackwardDataAlgorithm( GetMiopenHandle(context), s_.x_tensor, x_data, s_.w_desc, w_data, s_.conv_desc, s_.y_tensor, y_data, 1, &algo_count, &perf, algo_search_workspace.get(), AlgoSearchWorkspaceSize, false); 
[12/12/25 08:46:02] ERROR    Exception in ASGI application                      
                                                                                
                             ╭─────── Traceback (most recent call last) ───────╮
                             │ /usr/src/immich_ml/main.py:177 in predict       │
                             │                                                 │
                             │   174 │   │   inputs = text                     │
                             │   175 │   else:                                 │
                             │   176 │   │   raise HTTPException(400, "Either  │
                             │ ❱ 177 │   response = await run_inference(inputs │
                             │   178 │   return ORJSONResponse(response)       │
                             │   179                                           │
                             │   180                                           │
                             │                                                 │
                             │ /usr/src/immich_ml/main.py:202 in run_inference │
                             │                                                 │
                             │   199 │   │   response[entry["task"]] = output  │
                             │   200 │                                         │
                             │   201 │   without_deps, with_deps = entries     │
                             │ ❱ 202 │   await asyncio.gather(*[_run_inference │
                             │   203 │   if with_deps:                         │
                             │   204 │   │   await asyncio.gather(*[_run_infer │
                             │   205 │   if isinstance(payload, Image):        │
                             │                                                 │
                             │ /usr/src/immich_ml/main.py:197 in               │
                             │ _run_inference                                  │
                             │                                                 │
                             │   194 │   │   │   │   message = f"Task {entry[' │
                             │       output of {dep}"                          │
                             │   195 │   │   │   │   raise HTTPException(400,  │
                             │   196 │   │   model = await load(model)         │
                             │ ❱ 197 │   │   output = await run(model.predict, │
                             │   198 │   │   outputs[model.identity] = output  │
                             │   199 │   │   response[entry["task"]] = output  │
                             │   200                                           │
                             │                                                 │
                             │ /usr/src/immich_ml/main.py:215 in run           │
                             │                                                 │
                             │   212 │   if thread_pool is None:               │
                             │   213 │   │   return func(*args, **kwargs)      │
                             │   214 │   partial_func = partial(func, *args, * │
                             │ ❱ 215 │   return await asyncio.get_running_loop │
                             │   216                                           │
                             │   217                                           │
                             │   218 async def load(model: InferenceModel) ->  │
                             │                                                 │
                             │ /usr/lib/python3.10/concurrent/futures/thread.p │
                             │ y:58 in run                                     │
                             │                                                 │
                             │ /usr/src/immich_ml/models/base.py:60 in predict │
                             │                                                 │
                             │    57 │   │   self.load()                       │
                             │    58 │   │   if model_kwargs:                  │
                             │    59 │   │   │   self.configure(**model_kwargs │
                             │ ❱  60 │   │   return self._predict(*inputs)     │
                             │    61 │                                         │
                             │    62 │   @abstractmethod                       │
                             │    63 │   def _predict(self, *inputs: Any, **mo │
                             │                                                 │
                             │ /usr/src/immich_ml/models/ocr/detection.py:70   │
                             │ in _predict                                     │
                             │                                                 │
                             │    67 │   │   w, h = inputs.size                │
                             │    68 │   │   if w < 32 or h < 32:              │
                             │    69 │   │   │   return self._empty            │
                             │ ❱  70 │   │   out = self.session.run(None, {"x" │
                             │    71 │   │   boxes, scores = self.postprocess( │
                             │    72 │   │   if len(boxes) == 0:               │
                             │    73 │   │   │   return self._empty            │
                             │                                                 │
                             │ /usr/src/immich_ml/sessions/ort.py:51 in run    │
                             │                                                 │
                             │    48 │   │   input_feed: dict[str, NDArray[np. │
                             │    49 │   │   run_options: Any = None,          │
                             │    50 │   ) -> list[NDArray[np.float32]]:       │
                             │ ❱  51 │   │   outputs: list[NDArray[np.float32] │
                             │       run_options)                              │
                             │    52 │   │   return outputs                    │
                             │    53 │                                         │
                             │    54 │   @property                             │
                             │                                                 │
                             │ /opt/venv/lib/python3.10/site-packages/onnxrunt │
                             │ ime/capi/onnxruntime_inference_collection.py:26 │
                             │ 6 in run                                        │
                             │                                                 │
                             │    263 │   │   if not output_names:             │
                             │    264 │   │   │   output_names = [output.name  │
                             │    265 │   │   try:                             │
                             │ ❱  266 │   │   │   return self._sess.run(output │
                             │    267 │   │   except C.EPFail as err:          │
                             │    268 │   │   │   if self._enable_fallback:    │
                             │    269 │   │   │   │   print(f"EP Error: {err!s │
                             ╰─────────────────────────────────────────────────╯
                             Fail: [ONNXRuntimeError] : 1 : FAIL : Non-zero     
                             status code returned while running ConvTranspose   
                             node. Name:'ConvTranspose.0' Status Message: MIOPEN
                             failure 3: miopenStatusBadParm ; GPU=0 ;           
                             hostname=10d6bb49b9d7 ;                            
                             file=/code/onnxruntime/onnxruntime/core/providers/r
                             ocm/nn/conv_transpose.cc ; line=133 ;              
                             expr=miopenFindConvolutionBackwardDataAlgorithm(   
                             GetMiopenHandle(context), s_.x_tensor, x_data,     
                             s_.w_desc, w_data, s_.conv_desc, s_.y_tensor,      
                             y_data, 1, &algo_count, &perf,                     
                             algo_search_workspace.get(),                       
                             AlgoSearchWorkspaceSize, false);