2025-09-25T11:37:51.943409Z INFO windmill-worker/src/worker.rs:1751: fetched job 019980aa-5a09-710e-ee9a-a629259d4e4e (root job: none, scheduled for: 2025-09-25 11:37:51.881483 UTC) workspace_id=testsr2 job_id=019980aa-5a09-710e-ee9a-a629259d4e4e root_id=none 2025-09-25T11:37:51.967687Z INFO job: windmill-worker/src/handle_child.rs:692: job 019980aa-5a09-710e-ee9a-a629259d4e4e on wk-default-8f999a223e30-hpXEe in testsr2 worker memory snapshot 33900kB/25596kB job_id=019980aa-5a09-710e-ee9a-a629259d4e4e workspace_id=testsr2 worker=wk-default-8f999a223e30-hpXEe hostname=8f999a223e30 tag=duckdb language="duckdb" language="duckdb" otel.name="job" otel.name="job" script_path="u/rupprecht/felicitous_script" script_path="u/rupprecht/felicitous_script" 2025-09-25T11:37:51.973174Z INFO job: windmill-worker/src/handle_child.rs:704: job 019980aa-5a09-710e-ee9a-a629259d4e4e on wk-default-8f999a223e30-hpXEe in testsr2 still running. mem: 0kB, peak mem: 0kB job_id=019980aa-5a09-710e-ee9a-a629259d4e4e workspace_id=testsr2 worker=wk-default-8f999a223e30-hpXEe hostname=8f999a223e30 tag=duckdb language="duckdb" language="duckdb" otel.name="job" otel.name="job" script_path="u/rupprecht/felicitous_script" script_path="u/rupprecht/felicitous_script" 2025-09-25T11:37:51.974663Z INFO job: windmill-common/src/variables.rs:95: crypt for workspace testsr2 with key oSSQU72hR7b349AwkQ4RXKIoocowkvSwyG2wKe7TxLlNoLwI3vJ6OahkCFvW0pCu expired, refetching job_id=019980aa-5a09-710e-ee9a-a629259d4e4e workspace_id=testsr2 worker=wk-default-8f999a223e30-hpXEe hostname=8f999a223e30 tag=duckdb language="duckdb" language="duckdb" otel.name="job" otel.name="job" script_path="u/rupprecht/felicitous_script" script_path="u/rupprecht/felicitous_script" 2025-09-25T11:37:52.465117Z INFO job: windmill-worker/src/handle_child.rs:704: job 019980aa-5a09-710e-ee9a-a629259d4e4e on wk-default-8f999a223e30-hpXEe in testsr2 still running. mem: 0kB, peak mem: 0kB job_id=019980aa-5a09-710e-ee9a-a629259d4e4e workspace_id=testsr2 worker=wk-default-8f999a223e30-hpXEe hostname=8f999a223e30 tag=duckdb language="duckdb" language="duckdb" otel.name="job" otel.name="job" script_path="u/rupprecht/felicitous_script" script_path="u/rupprecht/felicitous_script" 2025-09-25T11:37:53.062926Z INFO job: windmill-worker/src/handle_child.rs:704: job 019980aa-5a09-710e-ee9a-a629259d4e4e on wk-default-8f999a223e30-hpXEe in testsr2 still running. mem: 0kB, peak mem: 0kB job_id=019980aa-5a09-710e-ee9a-a629259d4e4e workspace_id=testsr2 worker=wk-default-8f999a223e30-hpXEe hostname=8f999a223e30 tag=duckdb language="duckdb" language="duckdb" otel.name="job" otel.name="job" script_path="u/rupprecht/felicitous_script" script_path="u/rupprecht/felicitous_script" 2025-09-25T11:37:53.465488Z INFO job: windmill-worker/src/handle_child.rs:704: job 019980aa-5a09-710e-ee9a-a629259d4e4e on wk-default-8f999a223e30-hpXEe in testsr2 still running. mem: 0kB, peak mem: 0kB job_id=019980aa-5a09-710e-ee9a-a629259d4e4e workspace_id=testsr2 worker=wk-default-8f999a223e30-hpXEe hostname=8f999a223e30 tag=duckdb language="duckdb" language="duckdb" otel.name="job" otel.name="job" script_path="u/rupprecht/felicitous_script" script_path="u/rupprecht/felicitous_script" 2025-09-25T11:37:54.062866Z INFO job: windmill-worker/src/handle_child.rs:704: job 019980aa-5a09-710e-ee9a-a629259d4e4e on wk-default-8f999a223e30-hpXEe in testsr2 still running. mem: 0kB, peak mem: 0kB job_id=019980aa-5a09-710e-ee9a-a629259d4e4e workspace_id=testsr2 worker=wk-default-8f999a223e30-hpXEe hostname=8f999a223e30 tag=duckdb language="duckdb" language="duckdb" otel.name="job" otel.name="job" script_path="u/rupprecht/felicitous_script" script_path="u/rupprecht/felicitous_script" 2025-09-25T11:37:54.347319Z INFO request: windmill-audit/src/audit_ee.rs:139: kind="audit" operation="variables.decrypt_secret" action_kind=Execute resource="u/rupprecht/meticulous_s3" parameters=null workspace_id="testsr2" username="backend" email="backend" method=HEAD uri=/api/w/testsr2/s3_proxy/_default_/datalake/main/states2/ducklake-019980aa-6391-7047-a139-e01c8fcdb657.parquet traceId="1b5da58a-a0c1-4a57-873f-7c84b5265d1e" 2025-09-25T11:37:54.465671Z INFO job: windmill-worker/src/handle_child.rs:704: job 019980aa-5a09-710e-ee9a-a629259d4e4e on wk-default-8f999a223e30-hpXEe in testsr2 still running. mem: 0kB, peak mem: 0kB job_id=019980aa-5a09-710e-ee9a-a629259d4e4e workspace_id=testsr2 worker=wk-default-8f999a223e30-hpXEe hostname=8f999a223e30 tag=duckdb language="duckdb" language="duckdb" otel.name="job" otel.name="job" script_path="u/rupprecht/felicitous_script" script_path="u/rupprecht/felicitous_script" 2025-09-25T11:37:54.486979Z WARN request: windmill-common/src/error.rs:245: Not found: datalake/main/states2/ducklake-019980aa-6391-7047-a139-e01c8fcdb657.parquet: Error performing GET https://183ffc10b4ad1e976c87438989a4e138.eu.r2.cloudflarestorage.com/pva-dev-sr/datalake/main/states2/ducklake-019980aa-6391-7047-a139-e01c8fcdb657.parquet in 116.893643ms - Server returned non-2xx status code: 404 Not Found: NoSuchKeyThe specified key does not exist. method=HEAD uri=/api/w/testsr2/s3_proxy/_default_/datalake/main/states2/ducklake-019980aa-6391-7047-a139-e01c8fcdb657.parquet traceId="1b5da58a-a0c1-4a57-873f-7c84b5265d1e" 2025-09-25T11:37:54.487833Z WARN request: windmill-api/src/tracing_init.rs:37: response latency=161 status=404 method=HEAD uri=/api/w/testsr2/s3_proxy/_default_/datalake/main/states2/ducklake-019980aa-6391-7047-a139-e01c8fcdb657.parquet traceId="1b5da58a-a0c1-4a57-873f-7c84b5265d1e" 2025-09-25T11:37:54.497179Z INFO request: windmill-audit/src/audit_ee.rs:139: kind="audit" operation="variables.decrypt_secret" action_kind=Execute resource="u/rupprecht/meticulous_s3" parameters=null workspace_id="testsr2" username="backend" email="backend" method=POST uri=/api/w/testsr2/s3_proxy/_default_/datalake/main/states2/ducklake-019980aa-6391-7047-a139-e01c8fcdb657.parquet?uploads= traceId="3e1280e5-ce3e-402d-ad0a-fee09ce98a8e" 2025-09-25T11:37:54.567428Z ERROR request: windmill-api/src/tracing_init.rs:39: response latency=77 status=403 method=POST uri=/api/w/testsr2/s3_proxy/_default_/datalake/main/states2/ducklake-019980aa-6391-7047-a139-e01c8fcdb657.parquet?uploads= traceId="3e1280e5-ce3e-402d-ad0a-fee09ce98a8e" 2025-09-25T11:37:54.598800Z INFO windmill-worker/src/worker_utils.rs:84: ping update, memory: container=48MB, windmill=25MB worker=wk-default-8f999a223e30-hpXEe hostname=8f999a223e30 2025-09-25T11:37:54.600005Z ERROR job_postprocessing: windmill-queue/src/jobs.rs:723: job 019980aa-5a09-710e-ee9a-a629259d4e4e in testsr2 did not succeed: {"error":{"message":"execution error:\n\"HTTP Error: Unable to connect to URL http://localhost:36069/api/w/testsr2/s3_proxy/_default_/datalake/main/states2/ducklake-019980aa-6391-7047-a139-e01c8fcdb657.parquet?uploads=: Forbidden (HTTP code 403)\"","name":"ExecutionErr"}} job_id=019980aa-5a09-710e-ee9a-a629259d4e4e workspace_id=testsr2 worker=wk-default-8f999a223e30-hpXEe tag=duckdb success=false language="duckdb" language="duckdb" otel.name="job postprocessing" otel.name="job postprocessing" script_path="u/rupprecht/felicitous_script" script_path="u/rupprecht/felicitous_script" error.message="execution error:\n\"HTTP Error: Unable to connect to URL http://localhost:36069/api/w/testsr2/s3_proxy/_default_/datalake/main/states2/ducklake-019980aa-6391-7047-a139-e01c8fcdb657.parquet?uploads=: Forbidden (HTTP code 403)\"" error.message="execution error:\n\"HTTP Error: Unable to connect to URL http://localhost:36069/api/w/testsr2/s3_proxy/_default_/datalake/main/states2/ducklake-019980aa-6391-7047-a139-e01c8fcdb657.parquet?uploads=: Forbidden (HTTP code 403)\"" error.name="ExecutionErr" error.name="ExecutionErr" 2025-09-25T11:37:54.609789Z INFO job_postprocessing: windmill-queue/src/jobs.rs:1121: inserted completed job: 019980aa-5a09-710e-ee9a-a629259d4e4e (success: false) job_id=019980aa-5a09-710e-ee9a-a629259d4e4e root_job="" path="u/rupprecht/felicitous_script" job_kind=Preview started_at="2025-09-25 11:37:51.939126 UTC" duration=2662 permissioned_as="u/rupprecht" email="rupprecht@fastec.de" created_by="rupprecht" is_flow_step=false language=Some(DuckDb) scheduled_for=2025-09-25T11:37:51.881483Z workspace_id="testsr2" success=false job_id=019980aa-5a09-710e-ee9a-a629259d4e4e workspace_id=testsr2 worker=wk-default-8f999a223e30-hpXEe tag=duckdb success=false language="duckdb" language="duckdb" otel.name="job postprocessing" otel.name="job postprocessing" script_path="u/rupprecht/felicitous_script" script_path="u/rupprecht/felicitous_script" error.message="execution error:\n\"HTTP Error: Unable to connect to URL http://localhost:36069/api/w/testsr2/s3_proxy/_default_/datalake/main/states2/ducklake-019980aa-6391-7047-a139-e01c8fcdb657.parquet?uploads=: Forbidden (HTTP code 403)\"" error.message="execution error:\n\"HTTP Error: Unable to connect to URL http://localhost:36069/api/w/testsr2/s3_proxy/_default_/datalake/main/states2/ducklake-019980aa-6391-7047-a139-e01c8fcdb657.parquet?uploads=: Forbidden (HTTP code 403)\"" error.name="ExecutionErr" error.name="ExecutionErr" 2025-09-25T11:38:00.613944Z INFO windmill-worker/src/worker_utils.rs:84: ping update, memory: container=47MB, windmill=25MB worker=wk-default-8f999a223e30-hpXEe hostname=8f999a223e30