INFO 07-27 11:44:10 [metrics.py:417] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 45.2 tokens/s, Running: 7 reqs, Swapped: 0 reqs, Pending: 5 reqs, GPU KV cache usage: 94.8%, CPU KV cache usage: 0.0%.
INFO 07-27 11:44:15 [metrics.py:417] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 50.6 tokens/s, Running: 7 reqs, Swapped: 0 reqs, Pending: 5 reqs, GPU KV cache usage: 94.9%, CPU KV cache usage: 0.0%.
INFO 07-27 11:44:21 [metrics.py:417] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 46.2 tokens/s, Running: 7 reqs, Swapped: 0 reqs, Pending: 5 reqs, GPU KV cache usage: 95.1%, CPU KV cache usage: 0.0%.
INFO 07-27 11:44:26 [metrics.py:417] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 43.7 tokens/s, Running: 7 reqs, Swapped: 0 reqs, Pending: 5 reqs, GPU KV cache usage: 95.2%, CPU KV cache usage: 0.0%.
INFO 07-27 11:44:31 [metrics.py:417] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 49.4 tokens/s, Running: 7 reqs, Swapped: 0 reqs, Pending: 5 reqs, GPU KV cache usage: 95.4%, CPU KV cache usage: 0.0%.
INFO 07-27 11:44:32 [async_llm_engine.py:178] Finished request chatcmpl-f87ca61f29ea4d16b89e95b2a31e36d8.
{"requestId": "sync-d8f92f13-43ed-4838-8d7a-8123e31231a0-e2", "message": "Finished running generator.", "level": "INFO"}
{"requestId": "sync-d8f92f13-43ed-4838-8d7a-8123e31231a0-e2", "message": "Failed to return job results. | 400, message='Bad Request', url='https://api.runpod.ai/v2/q7znxcqlshj1oe/job-done/yftdi7jfiivfn3/sync-d8f92f13-43ed-4838-8d7a-8123e31231a0-e2?gpu=NVIDIA+A100+80GB+PCIe&isStream=true'", "level": "ERROR"}
{"requestId": "sync-d8f92f13-43ed-4838-8d7a-8123e31231a0-e2", "message": "Finished.", "level": "INFO"}
{"requestId": null, "message": "Jobs in progress: 11", "level": "INFO"}
INFO 07-27 11:44:39 [metrics.py:417] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 12.1 tokens/s, Running: 8 reqs, Swapped: 0 reqs, Pending: 3 reqs, GPU KV cache usage: 93.4%, CPU KV cache usage: 0.0%.
INFO 07-27 11:44:45 [metrics.py:417] Avg prompt throughput: 5732.2 tokens/s, Avg generation throughput: 57.9 tokens/s, Running: 8 reqs, Swapped: 0 reqs, Pending: 3 reqs, GPU KV cache usage: 93.6%, CPU KV cache usage: 0.0%.
INFO 07-27 11:44:50 [metrics.py:417] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 50.1 tokens/s, Running: 8 reqs, Swapped: 0 reqs, Pending: 3 reqs, GPU KV cache usage: 93.8%, CPU KV cache usage: 0.0%.
INFO 07-27 11:44:55 [metrics.py:417] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 51.7 tokens/s, Running: 8 reqs, Swapped: 0 reqs, Pending: 3 reqs, GPU KV cache usage: 93.9%, CPU KV cache usage: 0.0%.
INFO 07-27 11:45:00 [metrics.py:417] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 49.4 tokens/s, Running: 8 reqs, Swapped: 0 reqs, Pending: 3 reqs, GPU KV cache usage: 94.1%, CPU KV cache usage: 0.0%.