2025-06-12T15:18:41.718561091Z INFO 06-12 15:18:41 [__init__.py:243] Automatically detected platform cuda.
2025-06-12T15:18:44.783595731Z INFO 06-12 15:18:44 [__init__.py:31] Available plugins for group vllm.general_plugins:
2025-06-12T15:18:44.783627483Z INFO 06-12 15:18:44 [__init__.py:33] - lora_filesystem_resolver -> vllm.plugins.lora_resolvers.filesystem_resolver:register_filesystem_resolver
2025-06-12T15:18:44.783632686Z INFO 06-12 15:18:44 [__init__.py:36] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
2025-06-12T15:18:44.784140833Z engine.py           :27   2025-06-12 15:18:44,783 Engine args: AsyncEngineArgs(model='meta-llama/Llama-3.1-8B-Instruct', served_model_name=None, tokenizer=None, hf_config_path=None, task='auto', skip_tokenizer_init=False, enable_prompt_embeds=False, tokenizer_mode='auto', trust_remote_code=False, allowed_local_media_path='', download_dir=None, load_format='auto', config_format='auto', dtype='auto', kv_cache_dtype='auto', seed=0, max_model_len=None, cuda_graph_sizes=[512], distributed_executor_backend=None, pipeline_parallel_size=1, tensor_parallel_size=1, data_parallel_size=1, data_parallel_size_local=None, data_parallel_address=None, data_parallel_rpc_port=None, enable_expert_parallel=False, max_parallel_loading_workers=None, block_size=16, enable_prefix_caching=False, prefix_caching_hash_algo='builtin', disable_sliding_window=False, disable_cascade_attn=False, use_v2_block_manager='true', swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.95, max_num_batched_tokens=None, max_num_partial_prefills=1, max_long_partial_prefills=1, long_prefill_token_threshold=0, max_num_seqs=256, max_logprobs=20, disable_log_stats=False, revision=None, code_revision=None, rope_scaling={}, rope_theta=None, hf_token=None, hf_overrides={}, tokenizer_revision=None, quantization=None, enforce_eager=False, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config={}, limit_mm_per_prompt={}, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=False, enable_lora_bias=False, max_loras=1, max_lora_rank=16, fully_sharded_loras=False, max_cpu_loras=None, lora_dtype='auto', lora_extra_vocab_size=256, long_lora_scaling_factors=None, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, ray_workers_use_nsight=False, num_gpu_blocks_override=None, num_lookahead_slots=0, model_loader_extra_config={}, ignore_patterns=None, preemption_mode=None, scheduler_delay_factor=0.0, enable_chunked_prefill=None, disable_chunked_mm_input=False, guided_decoding_backend='outlines', guided_decoding_disable_fallback=False, guided_decoding_disable_any_whitespace=False, guided_decoding_disable_additional_properties=False, logits_processor_pattern=None, speculative_config=None, qlora_adapter_name_or_path=None, show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, scheduling_policy='fcfs', scheduler_cls='vllm.core.scheduler.Scheduler', override_neuron_config={}, override_pooler_config=None, compilation_config=None, worker_cls='auto', worker_extension_cls='', kv_transfer_config=None, kv_events_config=None, generation_config='auto', enable_sleep_mode=False, override_generation_config={}, model_impl='auto', calculate_kv_scales=False, additional_config=None, enable_reasoning=None, reasoning_parser='', use_tqdm_on_load=True, pt_load_map_location='cpu', disable_log_requests=False)
2025-06-12T15:18:57.895630392Z tokenizer_name_or_path: meta-llama/Llama-3.1-8B-Instruct, tokenizer_revision: None, trust_remote_code: False
2025-06-12T15:18:57.895671966Z INFO 06-12 15:18:57 [config.py:793] This model supports multiple tasks: {'generate', 'embed', 'classify', 'reward', 'score'}. Defaulting to 'generate'.
2025-06-12T15:18:57.895845028Z WARNING 06-12 15:18:57 [arg_utils.py:1583] --guided-decoding-backend=outlines is not supported by the V1 Engine. Falling back to V0.
2025-06-12T15:18:57.895896295Z WARNING 06-12 15:18:57 [arg_utils.py:1420] Chunked prefill is enabled by default for models with max_model_len > 32K. Chunked prefill might not work with some features or models. If you encounter any issues, please disable by launching with --enable-chunked-prefill=False.
2025-06-12T15:18:57.896422636Z INFO 06-12 15:18:57 [config.py:2118] Chunked prefill is enabled with max_num_batched_tokens=2048.
2025-06-12T15:18:57.899404060Z INFO 06-12 15:18:57 [llm_engine.py:230] Initializing a V0 LLM engine (v0.9.0.1) with config: model='meta-llama/Llama-3.1-8B-Instruct', speculative_config=None, tokenizer='meta-llama/Llama-3.1-8B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=131072, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(backend='outlines', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=meta-llama/Llama-3.1-8B-Instruct, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=False, chunked_prefill_enabled=True, use_async_output_proc=True, pooler_config=None, compilation_config={"compile_sizes": [], "inductor_compile_config": {"enable_auto_functionalized_v2": false}, "cudagraph_capture_sizes": [256, 248, 240, 232, 224, 216, 208, 200, 192, 184, 176, 168, 160, 152, 144, 136, 128, 120, 112, 104, 96, 88, 80, 72, 64, 56, 48, 40, 32, 24, 16, 8, 4, 2, 1], "max_capture_size": 256}, use_cached_outputs=False,
2025-06-12T15:18:59.194045264Z INFO 06-12 15:18:59 [cuda.py:292] Using Flash Attention backend.
2025-06-12T15:18:59.775721159Z INFO 06-12 15:18:59 [parallel_state.py:1064] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
2025-06-12T15:18:59.778065862Z INFO 06-12 15:18:59 [model_runner.py:1170] Starting to load model meta-llama/Llama-3.1-8B-Instruct...
2025-06-12T15:19:00.246102244Z INFO 06-12 15:19:00 [weight_utils.py:291] Using model weights format ['*.safetensors']
2025-06-12T15:19:17.266335310Z INFO 06-12 15:19:17 [weight_utils.py:307] Time spent downloading weights for meta-llama/Llama-3.1-8B-Instruct: 17.019623 seconds
2025-06-12T15:19:17.566182854Z 
Loading safetensors checkpoint shards:   0% Completed | 0/4 [00:00<?, ?it/s]
2025-06-12T15:19:17.809209565Z 
Loading safetensors checkpoint shards:  25% Completed | 1/4 [00:00<00:00,  4.12it/s]
2025-06-12T15:19:18.750937067Z 
Loading safetensors checkpoint shards:  50% Completed | 2/4 [00:01<00:01,  1.53it/s]
2025-06-12T15:19:19.741671653Z 
Loading safetensors checkpoint shards:  75% Completed | 3/4 [00:02<00:00,  1.24it/s]
2025-06-12T15:19:20.833907021Z 
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:03<00:00,  1.09it/s]
2025-06-12T15:19:20.834205509Z 
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:03<00:00,  1.22it/s]
2025-06-12T15:19:21.035059305Z INFO 06-12 15:19:21 [default_loader.py:280] Loading weights took 3.47 seconds
2025-06-12T15:19:21.259510061Z INFO 06-12 15:19:21 [model_runner.py:1202] Model loading took 14.9889 GiB and 21.261471 seconds
2025-06-12T15:19:22.337949834Z INFO 06-12 15:19:22 [worker.py:291] Memory profiling takes 0.88 seconds
2025-06-12T15:19:22.337997542Z INFO 06-12 15:19:22 [worker.py:291] the current vLLM instance can use total_gpu_memory (44.45GiB) x gpu_memory_utilization (0.95) = 42.23GiB
2025-06-12T15:19:22.338000167Z INFO 06-12 15:19:22 [worker.py:291] model weights take 14.99GiB; non_torch_memory takes 0.06GiB; PyTorch activation peak memory takes 1.19GiB; the rest of the memory reserved for KV Cache is 25.99GiB.
2025-06-12T15:19:22.525740778Z INFO 06-12 15:19:22 [executor_base.py:112] # cuda blocks: 13307, # CPU blocks: 2048
2025-06-12T15:19:22.526020587Z INFO 06-12 15:19:22 [executor_base.py:117] Maximum concurrency for 131072 tokens per request: 1.62x
2025-06-12T15:19:28.805352062Z INFO 06-12 15:19:28 [model_runner.py:1512] Capturing cudagraphs for decoding. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI. If out-of-memory error occurs during cudagraph capture, consider decreasing `gpu_memory_utilization` or switching to eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
2025-06-12T15:19:52.119173635Z 
Capturing CUDA graph shapes:   0%|          | 0/35 [00:00<?, ?it/s]
Capturing CUDA graph shapes:   3%|▎         | 1/35 [00:00<00:31,  1.09it/s]
Capturing CUDA graph shapes:   6%|▌         | 2/35 [00:01<00:31,  1.06it/s]
Capturing CUDA graph shapes:   9%|▊         | 3/35 [00:03<00:34,  1.08s/it]
Capturing CUDA graph shapes:  11%|█▏        | 4/35 [00:04<00:32,  1.05s/it]
Capturing CUDA graph shapes:  14%|█▍        | 5/35 [00:04<00:28,  1.04it/s]
Capturing CUDA graph shapes:  17%|█▋        | 6/35 [00:05<00:24,  1.21it/s]
Capturing CUDA graph shapes:  20%|██        | 7/35 [00:06<00:22,  1.27it/s]
Capturing CUDA graph shapes:  23%|██▎       | 8/35 [00:07<00:24,  1.10it/s]
Capturing CUDA graph shapes:  26%|██▌       | 9/35 [00:08<00:24,  1.05it/s]
Capturing CUDA graph shapes:  29%|██▊       | 10/35 [00:09<00:24,  1.01it/s]
Capturing CUDA graph shapes:  31%|███▏      | 11/35 [00:10<00:24,  1.00s/it]
Capturing CUDA graph shapes:  34%|███▍      | 12/35 [00:11<00:20,  1.14it/s]
Capturing CUDA graph shapes:  37%|███▋      | 13/35 [00:11<00:17,  1.27it/s]
Capturing CUDA graph shapes:  40%|████      | 14/35 [00:12<00:14,  1.41it/s]
Capturing CUDA graph shapes:  43%|████▎     | 15/35 [00:12<00:14,  1.43it/s]
Capturing CUDA graph shapes:  46%|████▌     | 16/35 [00:13<00:12,  1.52it/s]
Capturing CUDA graph shapes:  49%|████▊     | 17/35 [00:14<00:11,  1.53it/s]
Capturing CUDA graph shapes:  51%|█████▏    | 18/35 [00:14<00:10,  1.62it/s]
Capturing CUDA graph shapes:  54%|█████▍    | 19/35 [00:15<00:09,  1.69it/s]
Capturing CUDA graph shapes:  57%|█████▋    | 20/35 [00:15<00:08,  1.76it/s]
Capturing CUDA graph shapes:  60%|██████    | 21/35 [00:16<00:07,  1.75it/s]
Capturing CUDA graph shapes:  63%|██████▎   | 22/35 [00:16<00:07,  1.71it/s]
Capturing CUDA graph shapes:  66%|██████▌   | 23/35 [00:17<00:06,  1.76it/s]
Capturing CUDA graph shapes:  69%|██████▊   | 24/35 [00:17<00:06,  1.81it/s]
Capturing CUDA graph shapes:  71%|███████▏  | 25/35 [00:18<00:05,  1.87it/s]
Capturing CUDA graph shapes:  74%|███████▍  | 26/35 [00:18<00:04,  1.93it/s]
Capturing CUDA graph shapes:  77%|███████▋  | 27/35 [00:19<00:04,  1.94it/s]
Capturing CUDA graph shapes:  80%|████████  | 28/35 [00:19<00:03,  1.95it/s]
Capturing CUDA graph shapes:  83%|████████▎ | 29/35 [00:20<00:03,  1.95it/s]
Capturing CUDA graph shapes:  86%|████████▌ | 30/35 [00:20<00:02,  1.99it/s]
Capturing CUDA graph shapes:  89%|████████▊ | 31/35 [00:21<00:01,  2.02it/s]
Capturing CUDA graph shapes:  91%|█████████▏| 32/35 [00:21<00:01,  2.03it/s]
Capturing CUDA graph shapes:  94%|█████████▍| 33/35 [00:22<00:00,  2.04it/s]
Capturing CUDA graph shapes:  97%|█████████▋| 34/35 [00:22<00:00,  2.06it/s]
Capturing CUDA graph shapes: 100%|██████████| 35/35 [00:23<00:00,  2.08it/s]
Capturing CUDA graph shapes: 100%|██████████| 35/35 [00:23<00:00,  1.50it/s]
2025-06-12T15:19:52.119730326Z INFO 06-12 15:19:52 [model_runner.py:1670] Graph capturing finished in 23 secs, took 0.26 GiB
2025-06-12T15:19:52.120455112Z INFO 06-12 15:19:52 [llm_engine.py:428] init engine (profile, create kv cache, warmup model) took 30.86 seconds
2025-06-12T15:19:52.763740054Z engine.py           :113  2025-06-12 15:19:52,763 Initialized vLLM engine in 64.21s
2025-06-12T15:19:52.764243358Z engine.py           :27   2025-06-12 15:19:52,764 Engine args: AsyncEngineArgs(model='meta-llama/Llama-3.1-8B-Instruct', served_model_name=None, tokenizer=None, hf_config_path=None, task='auto', skip_tokenizer_init=False, enable_prompt_embeds=False, tokenizer_mode='auto', trust_remote_code=False, allowed_local_media_path='', download_dir=None, load_format='auto', config_format='auto', dtype='auto', kv_cache_dtype='auto', seed=0, max_model_len=None, cuda_graph_sizes=[512], distributed_executor_backend=None, pipeline_parallel_size=1, tensor_parallel_size=1, data_parallel_size=1, data_parallel_size_local=None, data_parallel_address=None, data_parallel_rpc_port=None, enable_expert_parallel=False, max_parallel_loading_workers=None, block_size=16, enable_prefix_caching=False, prefix_caching_hash_algo='builtin', disable_sliding_window=False, disable_cascade_attn=False, use_v2_block_manager='true', swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.95, max_num_batched_tokens=None, max_num_partial_prefills=1, max_long_partial_prefills=1, long_prefill_token_threshold=0, max_num_seqs=256, max_logprobs=20, disable_log_stats=False, revision=None, code_revision=None, rope_scaling={}, rope_theta=None, hf_token=None, hf_overrides={}, tokenizer_revision=None, quantization=None, enforce_eager=False, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config={}, limit_mm_per_prompt={}, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=False, enable_lora_bias=False, max_loras=1, max_lora_rank=16, fully_sharded_loras=False, max_cpu_loras=None, lora_dtype='auto', lora_extra_vocab_size=256, long_lora_scaling_factors=None, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, ray_workers_use_nsight=False, num_gpu_blocks_override=None, num_lookahead_slots=0, model_loader_extra_config={}, ignore_patterns=None, preemption_mode=None, scheduler_delay_factor=0.0, enable_chunked_prefill=None, disable_chunked_mm_input=False, guided_decoding_backend='outlines', guided_decoding_disable_fallback=False, guided_decoding_disable_any_whitespace=False, guided_decoding_disable_additional_properties=False, logits_processor_pattern=None, speculative_config=None, qlora_adapter_name_or_path=None, show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, scheduling_policy='fcfs', scheduler_cls='vllm.core.scheduler.Scheduler', override_neuron_config={}, override_pooler_config=None, compilation_config=None, worker_cls='auto', worker_extension_cls='', kv_transfer_config=None, kv_events_config=None, generation_config='auto', enable_sleep_mode=False, override_generation_config={}, model_impl='auto', calculate_kv_scales=False, additional_config=None, enable_reasoning=None, reasoning_parser='', use_tqdm_on_load=True, pt_load_map_location='cpu', disable_log_requests=False)
2025-06-12T15:19:53.809649308Z tokenizer_name_or_path: meta-llama/Llama-3.1-8B-Instruct, tokenizer_revision: None, trust_remote_code: False
2025-06-12T15:19:53.809676254Z WARNING 06-12 15:19:53 [config.py:1339] Default sampling parameters have been overridden by the model's Hugging Face generation config recommended from the model creator. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
2025-06-12T15:19:53.809825748Z INFO 06-12 15:19:53 [serving_chat.py:117] Using default chat sampling params from model: {'temperature': 0.6, 'top_p': 0.9}
2025-06-12T15:19:53.969191310Z INFO 06-12 15:19:53 [serving_completion.py:65] Using default completion sampling params from model: {'temperature': 0.6, 'top_p': 0.9}
2025-06-12T15:19:54.489795008Z --- Starting Serverless Worker |  Version 1.7.10 ---
2025-06-12T15:19:54.489829357Z {"requestId": null, "message": "Jobs in queue: 1", "level": "INFO"}
2025-06-12T15:19:54.489846289Z {"requestId": null, "message": "Jobs in progress: 1", "level": "INFO"}
2025-06-12T15:19:54.529302514Z {"requestId": "sync-729bf8b0-8486-446e-97cc-977ec42540c4-e1", "message": "Finished running generator.", "level": "INFO"}
2025-06-12T15:19:54.559501673Z {"requestId": "sync-729bf8b0-8486-446e-97cc-977ec42540c4-e1", "message": "Finished.", "level": "INFO"}
2025-06-12T15:23:19.671415557Z {"requestId": null, "message": "Jobs in queue: 1", "level": "INFO"}
2025-06-12T15:23:19.671679806Z {"requestId": null, "message": "Jobs in progress: 1", "level": "INFO"}
2025-06-12T15:23:19.707083155Z INFO 06-12 15:23:19 [chat_utils.py:419] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
2025-06-12T15:23:19.739886202Z INFO 06-12 15:23:19 [async_llm_engine.py:211] Added request chatcmpl-78127a2b22f340b68bb58de49b0f8e07.
2025-06-12T15:23:29.727334013Z INFO 06-12 15:23:29 [metrics.py:486] Avg prompt throughput: 0.3 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
2025-06-12T15:23:30.544485164Z INFO 06-12 15:23:30 [async_llm_engine.py:179] Finished request chatcmpl-78127a2b22f340b68bb58de49b0f8e07.
2025-06-12T15:23:30.597721261Z {"requestId": "sync-37c4cfef-46fb-4649-9e2d-350e231fa154-e2", "message": "Finished running generator.", "level": "INFO"}
2025-06-12T15:23:30.631747086Z {"requestId": "sync-37c4cfef-46fb-4649-9e2d-350e231fa154-e2", "message": "Finished.", "level": "INFO"}
2025-06-12T15:28:03.659432661Z {"requestId": null, "message": "Jobs in queue: 1", "level": "INFO"}
2025-06-12T15:28:03.659473882Z {"requestId": null, "message": "Jobs in progress: 1", "level": "INFO"}
2025-06-12T15:28:03.789284070Z INFO 06-12 15:28:03 [async_llm_engine.py:211] Added request chatcmpl-e465907c0ca34357a6dac2cede8f1c80.
2025-06-12T15:28:08.013748143Z INFO 06-12 15:28:08 [metrics.py:486] Avg prompt throughput: 7.4 tokens/s, Avg generation throughput: 0.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 6.9%, CPU KV cache usage: 0.0%.
2025-06-12T15:28:13.039347405Z INFO 06-12 15:28:13 [metrics.py:486] Avg prompt throughput: 2501.0 tokens/s, Avg generation throughput: 11.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 6.9%, CPU KV cache usage: 0.0%.
2025-06-12T15:28:18.050944495Z INFO 06-12 15:28:18 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 6.9%, CPU KV cache usage: 0.0%.
2025-06-12T15:28:23.059204096Z INFO 06-12 15:28:23 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 7.0%, CPU KV cache usage: 0.0%.
2025-06-12T15:28:28.069428247Z INFO 06-12 15:28:28 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 7.0%, CPU KV cache usage: 0.0%.
2025-06-12T15:28:33.084310719Z INFO 06-12 15:28:33 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 7.1%, CPU KV cache usage: 0.0%.
2025-06-12T15:28:38.088085732Z INFO 06-12 15:28:38 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 7.1%, CPU KV cache usage: 0.0%.
2025-06-12T15:28:43.128539371Z INFO 06-12 15:28:43 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 7.2%, CPU KV cache usage: 0.0%.
2025-06-12T15:28:48.178918618Z INFO 06-12 15:28:48 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 7.2%, CPU KV cache usage: 0.0%.
2025-06-12T15:28:53.184733983Z INFO 06-12 15:28:53 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 7.3%, CPU KV cache usage: 0.0%.
2025-06-12T15:28:58.217957643Z INFO 06-12 15:28:58 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 16.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 7.3%, CPU KV cache usage: 0.0%.
2025-06-12T15:29:03.233578866Z INFO 06-12 15:29:03 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 14.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 7.4%, CPU KV cache usage: 0.0%.
2025-06-12T15:29:08.252212703Z INFO 06-12 15:29:08 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 7.4%, CPU KV cache usage: 0.0%.
2025-06-12T15:29:13.297578381Z INFO 06-12 15:29:13 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 7.5%, CPU KV cache usage: 0.0%.
2025-06-12T15:29:18.308249711Z INFO 06-12 15:29:18 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 7.5%, CPU KV cache usage: 0.0%.
2025-06-12T15:29:23.320929237Z INFO 06-12 15:29:23 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 7.6%, CPU KV cache usage: 0.0%.
2025-06-12T15:29:28.342868572Z INFO 06-12 15:29:28 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 7.6%, CPU KV cache usage: 0.0%.
2025-06-12T15:29:33.378698867Z INFO 06-12 15:29:33 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 7.7%, CPU KV cache usage: 0.0%.
2025-06-12T15:29:38.423462098Z INFO 06-12 15:29:38 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 7.7%, CPU KV cache usage: 0.0%.
2025-06-12T15:29:43.462694330Z INFO 06-12 15:29:43 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 7.8%, CPU KV cache usage: 0.0%.
2025-06-12T15:29:48.466233453Z INFO 06-12 15:29:48 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 7.8%, CPU KV cache usage: 0.0%.
2025-06-12T15:29:53.524245609Z INFO 06-12 15:29:53 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 13.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 7.8%, CPU KV cache usage: 0.0%.
2025-06-12T15:29:58.568689481Z INFO 06-12 15:29:58 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 17.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 7.9%, CPU KV cache usage: 0.0%.
2025-06-12T15:30:03.581475268Z INFO 06-12 15:30:03 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 7.9%, CPU KV cache usage: 0.0%.
2025-06-12T15:30:08.583844939Z INFO 06-12 15:30:08 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 8.0%, CPU KV cache usage: 0.0%.
2025-06-12T15:30:13.633704943Z INFO 06-12 15:30:13 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 8.0%, CPU KV cache usage: 0.0%.
2025-06-12T15:30:18.666467183Z INFO 06-12 15:30:18 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 8.1%, CPU KV cache usage: 0.0%.
2025-06-12T15:30:23.707939463Z INFO 06-12 15:30:23 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 8.1%, CPU KV cache usage: 0.0%.
2025-06-12T15:30:28.731359241Z INFO 06-12 15:30:28 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 8.2%, CPU KV cache usage: 0.0%.
2025-06-12T15:30:33.738091600Z INFO 06-12 15:30:33 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 8.2%, CPU KV cache usage: 0.0%.
2025-06-12T15:30:38.795131479Z INFO 06-12 15:30:38 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 8.3%, CPU KV cache usage: 0.0%.
2025-06-12T15:30:43.899050351Z INFO 06-12 15:30:43 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 8.3%, CPU KV cache usage: 0.0%.
2025-06-12T15:30:48.935475592Z INFO 06-12 15:30:48 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 8.4%, CPU KV cache usage: 0.0%.
2025-06-12T15:30:53.955022365Z INFO 06-12 15:30:53 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 8.4%, CPU KV cache usage: 0.0%.
2025-06-12T15:30:58.989482258Z INFO 06-12 15:30:58 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 8.5%, CPU KV cache usage: 0.0%.
2025-06-12T15:31:04.023515721Z INFO 06-12 15:31:04 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 8.5%, CPU KV cache usage: 0.0%.
2025-06-12T15:31:09.029920184Z INFO 06-12 15:31:09 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 8.6%, CPU KV cache usage: 0.0%.
2025-06-12T15:31:14.039693731Z INFO 06-12 15:31:14 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 8.6%, CPU KV cache usage: 0.0%.
2025-06-12T15:31:19.081403720Z INFO 06-12 15:31:19 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 8.6%, CPU KV cache usage: 0.0%.
2025-06-12T15:31:24.121924516Z INFO 06-12 15:31:24 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 8.7%, CPU KV cache usage: 0.0%.
2025-06-12T15:31:29.145202777Z INFO 06-12 15:31:29 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 13.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 8.7%, CPU KV cache usage: 0.0%.
2025-06-12T15:31:34.151402893Z INFO 06-12 15:31:34 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 18.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 8.8%, CPU KV cache usage: 0.0%.
2025-06-12T15:31:39.172835054Z INFO 06-12 15:31:39 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 8.8%, CPU KV cache usage: 0.0%.
2025-06-12T15:31:44.202054844Z INFO 06-12 15:31:44 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 8.9%, CPU KV cache usage: 0.0%.
2025-06-12T15:31:49.222098551Z INFO 06-12 15:31:49 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 8.9%, CPU KV cache usage: 0.0%.
2025-06-12T15:31:54.265237383Z INFO 06-12 15:31:54 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.0%, CPU KV cache usage: 0.0%.
2025-06-12T15:31:59.300772556Z INFO 06-12 15:31:59 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.0%, CPU KV cache usage: 0.0%.
2025-06-12T15:32:04.324748961Z INFO 06-12 15:32:04 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.1%, CPU KV cache usage: 0.0%.
2025-06-12T15:32:09.327061469Z INFO 06-12 15:32:09 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.1%, CPU KV cache usage: 0.0%.
2025-06-12T15:32:14.419407800Z INFO 06-12 15:32:14 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 17.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.2%, CPU KV cache usage: 0.0%.
2025-06-12T15:32:19.466923440Z INFO 06-12 15:32:19 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 13.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.2%, CPU KV cache usage: 0.0%.
2025-06-12T15:32:24.516073349Z INFO 06-12 15:32:24 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.2%, CPU KV cache usage: 0.0%.
2025-06-12T15:32:29.560161308Z INFO 06-12 15:32:29 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.3%, CPU KV cache usage: 0.0%.
2025-06-12T15:32:34.593530206Z INFO 06-12 15:32:34 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.3%, CPU KV cache usage: 0.0%.
2025-06-12T15:32:39.618600273Z INFO 06-12 15:32:39 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.4%, CPU KV cache usage: 0.0%.
2025-06-12T15:32:44.625906738Z INFO 06-12 15:32:44 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.4%, CPU KV cache usage: 0.0%.
2025-06-12T15:32:49.668731816Z INFO 06-12 15:32:49 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.5%, CPU KV cache usage: 0.0%.
2025-06-12T15:32:54.711226136Z INFO 06-12 15:32:54 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.5%, CPU KV cache usage: 0.0%.
2025-06-12T15:32:59.736540378Z INFO 06-12 15:32:59 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.6%, CPU KV cache usage: 0.0%.
2025-06-12T15:33:04.832721956Z INFO 06-12 15:33:04 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.6%, CPU KV cache usage: 0.0%.
2025-06-12T15:33:09.840045545Z INFO 06-12 15:33:09 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.6%, CPU KV cache usage: 0.0%.
2025-06-12T15:33:14.841166315Z INFO 06-12 15:33:14 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.7%, CPU KV cache usage: 0.0%.
2025-06-12T15:33:19.848573631Z INFO 06-12 15:33:19 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.7%, CPU KV cache usage: 0.0%.
2025-06-12T15:33:24.859074037Z INFO 06-12 15:33:24 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.8%, CPU KV cache usage: 0.0%.
2025-06-12T15:33:29.876413932Z INFO 06-12 15:33:29 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.8%, CPU KV cache usage: 0.0%.
2025-06-12T15:33:33.176862646Z {"requestId": null, "message": "Jobs in queue: 1", "level": "INFO"}
2025-06-12T15:33:34.884501050Z INFO 06-12 15:33:34 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.9%, CPU KV cache usage: 0.0%.
2025-06-12T15:33:39.901751500Z INFO 06-12 15:33:39 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 9.9%, CPU KV cache usage: 0.0%.
2025-06-12T15:33:44.908279166Z INFO 06-12 15:33:44 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 10.0%, CPU KV cache usage: 0.0%.
2025-06-12T15:33:49.960714101Z INFO 06-12 15:33:49 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 15.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 10.0%, CPU KV cache usage: 0.0%.
2025-06-12T15:33:54.993424205Z INFO 06-12 15:33:54 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 15.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 10.1%, CPU KV cache usage: 0.0%.
2025-06-12T15:34:00.025285977Z INFO 06-12 15:34:00 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 10.1%, CPU KV cache usage: 0.0%.
2025-06-12T15:34:05.031716024Z INFO 06-12 15:34:05 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 10.2%, CPU KV cache usage: 0.0%.
2025-06-12T15:34:10.032744473Z INFO 06-12 15:34:10 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 10.2%, CPU KV cache usage: 0.0%.
2025-06-12T15:34:15.042731136Z INFO 06-12 15:34:15 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 10.2%, CPU KV cache usage: 0.0%.
2025-06-12T15:34:20.045861412Z INFO 06-12 15:34:20 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 10.3%, CPU KV cache usage: 0.0%.
2025-06-12T15:34:25.067796134Z INFO 06-12 15:34:25 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 10.3%, CPU KV cache usage: 0.0%.
2025-06-12T15:34:30.081562756Z INFO 06-12 15:34:30 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 10.4%, CPU KV cache usage: 0.0%.
2025-06-12T15:34:35.152502636Z INFO 06-12 15:34:35 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 18.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 10.4%, CPU KV cache usage: 0.0%.
2025-06-12T15:34:40.153416273Z INFO 06-12 15:34:40 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 11.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 10.5%, CPU KV cache usage: 0.0%.
2025-06-12T15:34:45.190994656Z INFO 06-12 15:34:45 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 10.5%, CPU KV cache usage: 0.0%.
2025-06-12T15:34:50.229643159Z INFO 06-12 15:34:50 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 10.6%, CPU KV cache usage: 0.0%.
2025-06-12T15:34:55.242988083Z INFO 06-12 15:34:55 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 10.6%, CPU KV cache usage: 0.0%.
2025-06-12T15:35:00.247561740Z INFO 06-12 15:35:00 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 10.6%, CPU KV cache usage: 0.0%.
2025-06-12T15:35:05.290316589Z INFO 06-12 15:35:05 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 10.7%, CPU KV cache usage: 0.0%.
2025-06-12T15:35:10.308057997Z INFO 06-12 15:35:10 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 10.7%, CPU KV cache usage: 0.0%.
2025-06-12T15:35:15.318160341Z INFO 06-12 15:35:15 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 10.8%, CPU KV cache usage: 0.0%.
2025-06-12T15:35:20.349045261Z INFO 06-12 15:35:20 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 17.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 10.8%, CPU KV cache usage: 0.0%.
2025-06-12T15:35:25.360581363Z INFO 06-12 15:35:25 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 12.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 10.9%, CPU KV cache usage: 0.0%.
2025-06-12T15:35:30.365773950Z INFO 06-12 15:35:30 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 10.9%, CPU KV cache usage: 0.0%.
2025-06-12T15:35:35.415761835Z INFO 06-12 15:35:35 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 11.0%, CPU KV cache usage: 0.0%.
2025-06-12T15:35:40.458755790Z INFO 06-12 15:35:40 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 11.0%, CPU KV cache usage: 0.0%.
2025-06-12T15:35:45.473313568Z INFO 06-12 15:35:45 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 11.1%, CPU KV cache usage: 0.0%.
2025-06-12T15:35:50.494994150Z INFO 06-12 15:35:50 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 11.1%, CPU KV cache usage: 0.0%.
2025-06-12T15:35:55.495716732Z INFO 06-12 15:35:55 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 11.2%, CPU KV cache usage: 0.0%.
2025-06-12T15:36:00.512498865Z INFO 06-12 15:36:00 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 11.2%, CPU KV cache usage: 0.0%.
2025-06-12T15:36:05.579467762Z INFO 06-12 15:36:05 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 14.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 11.2%, CPU KV cache usage: 0.0%.
2025-06-12T15:36:10.597415817Z INFO 06-12 15:36:10 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 16.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 11.3%, CPU KV cache usage: 0.0%.
2025-06-12T15:36:15.617585772Z INFO 06-12 15:36:15 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 11.3%, CPU KV cache usage: 0.0%.
2025-06-12T15:36:20.634030526Z INFO 06-12 15:36:20 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 11.4%, CPU KV cache usage: 0.0%.
2025-06-12T15:36:25.640722109Z INFO 06-12 15:36:25 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 11.4%, CPU KV cache usage: 0.0%.
2025-06-12T15:36:30.645838442Z INFO 06-12 15:36:30 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 11.5%, CPU KV cache usage: 0.0%.
2025-06-12T15:36:35.650523481Z INFO 06-12 15:36:35 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 22.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 11.5%, CPU KV cache usage: 0.0%.
2025-06-12T15:36:40.689128046Z INFO 06-12 15:36:40 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 11.6%, CPU KV cache usage: 0.0%.
2025-06-12T15:36:45.725469940Z INFO 06-12 15:36:45 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 11.6%, CPU KV cache usage: 0.0%.
2025-06-12T15:36:50.726481356Z INFO 06-12 15:36:50 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 13.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 11.6%, CPU KV cache usage: 0.0%.
2025-06-12T15:36:55.734686357Z INFO 06-12 15:36:55 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 17.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 11.7%, CPU KV cache usage: 0.0%.
2025-06-12T15:37:00.760852643Z INFO 06-12 15:37:00 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 11.7%, CPU KV cache usage: 0.0%.
2025-06-12T15:37:05.785501067Z INFO 06-12 15:37:05 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 11.8%, CPU KV cache usage: 0.0%.
2025-06-12T15:37:10.820367509Z INFO 06-12 15:37:10 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 11.8%, CPU KV cache usage: 0.0%.
2025-06-12T15:37:15.830403555Z INFO 06-12 15:37:15 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 11.9%, CPU KV cache usage: 0.0%.
2025-06-12T15:37:20.872297785Z INFO 06-12 15:37:20 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 11.9%, CPU KV cache usage: 0.0%.
2025-06-12T15:37:25.888451976Z INFO 06-12 15:37:25 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 12.0%, CPU KV cache usage: 0.0%.
2025-06-12T15:37:30.904309839Z INFO 06-12 15:37:30 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 12.0%, CPU KV cache usage: 0.0%.
2025-06-12T15:37:35.962454620Z INFO 06-12 15:37:35 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 12.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 12.0%, CPU KV cache usage: 0.0%.
2025-06-12T15:37:40.982038813Z INFO 06-12 15:37:40 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 16.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 12.1%, CPU KV cache usage: 0.0%.
2025-06-12T15:37:46.011223768Z INFO 06-12 15:37:46 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 12.1%, CPU KV cache usage: 0.0%.
2025-06-12T15:37:51.045130279Z INFO 06-12 15:37:51 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 12.2%, CPU KV cache usage: 0.0%.
2025-06-12T15:37:56.080178729Z INFO 06-12 15:37:56 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 12.2%, CPU KV cache usage: 0.0%.
2025-06-12T15:38:01.098759140Z INFO 06-12 15:38:01 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 12.3%, CPU KV cache usage: 0.0%.
2025-06-12T15:38:06.115915900Z INFO 06-12 15:38:06 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 12.3%, CPU KV cache usage: 0.0%.
2025-06-12T15:38:11.160894398Z INFO 06-12 15:38:11 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 12.4%, CPU KV cache usage: 0.0%.
2025-06-12T15:38:16.179743288Z INFO 06-12 15:38:16 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 12.4%, CPU KV cache usage: 0.0%.
2025-06-12T15:38:21.253266115Z INFO 06-12 15:38:21 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 11.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 12.5%, CPU KV cache usage: 0.0%.
2025-06-12T15:38:26.296333776Z INFO 06-12 15:38:26 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 18.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 12.5%, CPU KV cache usage: 0.0%.
2025-06-12T15:38:31.338914742Z INFO 06-12 15:38:31 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 12.5%, CPU KV cache usage: 0.0%.
2025-06-12T15:38:36.378326461Z INFO 06-12 15:38:36 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 12.6%, CPU KV cache usage: 0.0%.
2025-06-12T15:38:41.430950024Z INFO 06-12 15:38:41 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 12.6%, CPU KV cache usage: 0.0%.
2025-06-12T15:38:46.442522939Z INFO 06-12 15:38:46 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 12.7%, CPU KV cache usage: 0.0%.
2025-06-12T15:38:51.487119301Z INFO 06-12 15:38:51 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 12.7%, CPU KV cache usage: 0.0%.
2025-06-12T15:38:56.535882728Z INFO 06-12 15:38:56 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 12.8%, CPU KV cache usage: 0.0%.
2025-06-12T15:39:01.581252036Z INFO 06-12 15:39:01 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 12.8%, CPU KV cache usage: 0.0%.
2025-06-12T15:39:04.260791823Z {"requestId": null, "message": "Jobs in queue: 2", "level": "INFO"}
2025-06-12T15:39:06.673078936Z INFO 06-12 15:39:06 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 11.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 12.9%, CPU KV cache usage: 0.0%.
2025-06-12T15:39:11.704279090Z INFO 06-12 15:39:11 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 18.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 12.9%, CPU KV cache usage: 0.0%.
2025-06-12T15:39:16.731348324Z INFO 06-12 15:39:16 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 13.0%, CPU KV cache usage: 0.0%.
2025-06-12T15:39:21.739556863Z INFO 06-12 15:39:21 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 13.0%, CPU KV cache usage: 0.0%.
2025-06-12T15:39:26.778704866Z INFO 06-12 15:39:26 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 13.0%, CPU KV cache usage: 0.0%.
2025-06-12T15:39:31.782435531Z INFO 06-12 15:39:31 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 13.1%, CPU KV cache usage: 0.0%.
2025-06-12T15:39:36.808732442Z INFO 06-12 15:39:36 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 13.1%, CPU KV cache usage: 0.0%.
2025-06-12T15:39:41.843276138Z INFO 06-12 15:39:41 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 13.2%, CPU KV cache usage: 0.0%.
2025-06-12T15:39:46.888346679Z INFO 06-12 15:39:46 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 13.2%, CPU KV cache usage: 0.0%.
2025-06-12T15:39:51.973875780Z INFO 06-12 15:39:51 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 16.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 13.3%, CPU KV cache usage: 0.0%.
2025-06-12T15:39:56.978892705Z INFO 06-12 15:39:56 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 13.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 13.3%, CPU KV cache usage: 0.0%.
2025-06-12T15:40:02.015557901Z INFO 06-12 15:40:02 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 13.4%, CPU KV cache usage: 0.0%.
2025-06-12T15:40:07.055827181Z INFO 06-12 15:40:07 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 13.4%, CPU KV cache usage: 0.0%.
2025-06-12T15:40:12.082147508Z INFO 06-12 15:40:12 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 13.5%, CPU KV cache usage: 0.0%.
2025-06-12T15:40:17.085129657Z INFO 06-12 15:40:17 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 13.5%, CPU KV cache usage: 0.0%.
2025-06-12T15:40:22.141159777Z INFO 06-12 15:40:22 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 13.6%, CPU KV cache usage: 0.0%.
2025-06-12T15:40:27.199511575Z INFO 06-12 15:40:27 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 13.6%, CPU KV cache usage: 0.0%.
2025-06-12T15:40:32.238164179Z INFO 06-12 15:40:32 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 13.6%, CPU KV cache usage: 0.0%.
2025-06-12T15:40:37.335547279Z INFO 06-12 15:40:37 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 13.7%, CPU KV cache usage: 0.0%.
2025-06-12T15:40:42.376402097Z INFO 06-12 15:40:42 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 13.7%, CPU KV cache usage: 0.0%.
2025-06-12T15:40:47.404469597Z INFO 06-12 15:40:47 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 13.8%, CPU KV cache usage: 0.0%.
2025-06-12T15:40:52.408224158Z INFO 06-12 15:40:52 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 13.8%, CPU KV cache usage: 0.0%.
2025-06-12T15:40:57.411288934Z INFO 06-12 15:40:57 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 13.9%, CPU KV cache usage: 0.0%.
2025-06-12T15:41:02.436955995Z INFO 06-12 15:41:02 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 13.9%, CPU KV cache usage: 0.0%.
2025-06-12T15:41:07.447381727Z INFO 06-12 15:41:07 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.0%, CPU KV cache usage: 0.0%.
2025-06-12T15:41:12.496675955Z INFO 06-12 15:41:12 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.0%, CPU KV cache usage: 0.0%.
2025-06-12T15:41:17.513649088Z INFO 06-12 15:41:17 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.1%, CPU KV cache usage: 0.0%.
2025-06-12T15:41:22.539926214Z INFO 06-12 15:41:22 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.1%, CPU KV cache usage: 0.0%.
2025-06-12T15:41:27.597719405Z INFO 06-12 15:41:27 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 11.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.1%, CPU KV cache usage: 0.0%.
2025-06-12T15:41:32.636999109Z INFO 06-12 15:41:32 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 18.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.2%, CPU KV cache usage: 0.0%.
2025-06-12T15:41:37.643068783Z INFO 06-12 15:41:37 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.2%, CPU KV cache usage: 0.0%.
2025-06-12T15:41:42.693389994Z INFO 06-12 15:41:42 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.3%, CPU KV cache usage: 0.0%.
2025-06-12T15:41:47.733334476Z INFO 06-12 15:41:47 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.3%, CPU KV cache usage: 0.0%.
2025-06-12T15:41:52.761923645Z INFO 06-12 15:41:52 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.4%, CPU KV cache usage: 0.0%.
2025-06-12T15:41:57.789065277Z INFO 06-12 15:41:57 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.4%, CPU KV cache usage: 0.0%.
2025-06-12T15:42:02.815325032Z INFO 06-12 15:42:02 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.5%, CPU KV cache usage: 0.0%.
2025-06-12T15:42:07.843523308Z INFO 06-12 15:42:07 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.5%, CPU KV cache usage: 0.0%.
2025-06-12T15:42:12.861734461Z INFO 06-12 15:42:12 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 16.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.5%, CPU KV cache usage: 0.0%.
2025-06-12T15:42:17.898901831Z INFO 06-12 15:42:17 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 13.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.6%, CPU KV cache usage: 0.0%.
2025-06-12T15:42:22.927168006Z INFO 06-12 15:42:22 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.6%, CPU KV cache usage: 0.0%.
2025-06-12T15:42:27.969887471Z INFO 06-12 15:42:27 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.7%, CPU KV cache usage: 0.0%.
2025-06-12T15:42:32.970709060Z INFO 06-12 15:42:32 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.7%, CPU KV cache usage: 0.0%.
2025-06-12T15:42:38.004583869Z INFO 06-12 15:42:38 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 9.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.7%, CPU KV cache usage: 0.0%.
2025-06-12T15:42:43.018950871Z INFO 06-12 15:42:43 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.8%, CPU KV cache usage: 0.0%.
2025-06-12T15:42:48.043468256Z INFO 06-12 15:42:48 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.8%, CPU KV cache usage: 0.0%.
2025-06-12T15:42:53.049680880Z INFO 06-12 15:42:53 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.9%, CPU KV cache usage: 0.0%.
2025-06-12T15:42:58.054808561Z INFO 06-12 15:42:58 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 14.9%, CPU KV cache usage: 0.0%.
2025-06-12T15:43:03.069522179Z INFO 06-12 15:43:03 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.0%, CPU KV cache usage: 0.0%.
2025-06-12T15:43:08.124515944Z INFO 06-12 15:43:08 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.0%, CPU KV cache usage: 0.0%.
2025-06-12T15:43:13.144329930Z INFO 06-12 15:43:13 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.1%, CPU KV cache usage: 0.0%.
2025-06-12T15:43:18.198604419Z INFO 06-12 15:43:18 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.1%, CPU KV cache usage: 0.0%.
2025-06-12T15:43:23.236136255Z INFO 06-12 15:43:23 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.2%, CPU KV cache usage: 0.0%.
2025-06-12T15:43:28.291711007Z INFO 06-12 15:43:28 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.2%, CPU KV cache usage: 0.0%.
2025-06-12T15:43:33.336520098Z INFO 06-12 15:43:33 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.3%, CPU KV cache usage: 0.0%.
2025-06-12T15:43:38.354011766Z INFO 06-12 15:43:38 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.3%, CPU KV cache usage: 0.0%.
2025-06-12T15:43:43.383578453Z INFO 06-12 15:43:43 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.4%, CPU KV cache usage: 0.0%.
2025-06-12T15:43:48.384596753Z INFO 06-12 15:43:48 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.4%, CPU KV cache usage: 0.0%.
2025-06-12T15:43:53.411243776Z INFO 06-12 15:43:53 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.5%, CPU KV cache usage: 0.0%.
2025-06-12T15:43:58.417484947Z INFO 06-12 15:43:58 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.5%, CPU KV cache usage: 0.0%.
2025-06-12T15:44:03.451440189Z INFO 06-12 15:44:03 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.5%, CPU KV cache usage: 0.0%.
2025-06-12T15:44:08.488738958Z INFO 06-12 15:44:08 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.6%, CPU KV cache usage: 0.0%.
2025-06-12T15:44:13.529682022Z INFO 06-12 15:44:13 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 17.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.6%, CPU KV cache usage: 0.0%.
2025-06-12T15:44:18.559511304Z INFO 06-12 15:44:18 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.7%, CPU KV cache usage: 0.0%.
2025-06-12T15:44:23.563494152Z INFO 06-12 15:44:23 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.7%, CPU KV cache usage: 0.0%.
2025-06-12T15:44:28.576261222Z INFO 06-12 15:44:28 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.8%, CPU KV cache usage: 0.0%.
2025-06-12T15:44:33.601202210Z INFO 06-12 15:44:33 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.8%, CPU KV cache usage: 0.0%.
2025-06-12T15:44:38.641022471Z INFO 06-12 15:44:38 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.9%, CPU KV cache usage: 0.0%.
2025-06-12T15:44:43.641824424Z INFO 06-12 15:44:43 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 15.9%, CPU KV cache usage: 0.0%.
2025-06-12T15:44:48.668669170Z INFO 06-12 15:44:48 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 17.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 16.0%, CPU KV cache usage: 0.0%.
2025-06-12T15:44:53.677528048Z INFO 06-12 15:44:53 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 16.0%, CPU KV cache usage: 0.0%.
2025-06-12T15:44:58.690364749Z INFO 06-12 15:44:58 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 16.1%, CPU KV cache usage: 0.0%.
2025-06-12T15:45:03.712085829Z INFO 06-12 15:45:03 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 16.1%, CPU KV cache usage: 0.0%.
2025-06-12T15:45:08.759347308Z INFO 06-12 15:45:08 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 16.1%, CPU KV cache usage: 0.0%.
2025-06-12T15:45:13.779736420Z INFO 06-12 15:45:13 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 16.2%, CPU KV cache usage: 0.0%.
2025-06-12T15:45:18.783735497Z INFO 06-12 15:45:18 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 16.2%, CPU KV cache usage: 0.0%.
2025-06-12T15:45:23.790935581Z INFO 06-12 15:45:23 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 17.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 16.3%, CPU KV cache usage: 0.0%.
2025-06-12T15:45:28.811103568Z INFO 06-12 15:45:28 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 16.3%, CPU KV cache usage: 0.0%.
2025-06-12T15:45:33.817960901Z INFO 06-12 15:45:33 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 16.4%, CPU KV cache usage: 0.0%.
2025-06-12T15:45:38.827035332Z INFO 06-12 15:45:38 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 16.4%, CPU KV cache usage: 0.0%.
2025-06-12T15:45:43.859410636Z INFO 06-12 15:45:43 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 16.5%, CPU KV cache usage: 0.0%.
2025-06-12T15:45:48.892387822Z INFO 06-12 15:45:48 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 16.5%, CPU KV cache usage: 0.0%.
2025-06-12T15:45:53.901389038Z INFO 06-12 15:45:53 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 16.6%, CPU KV cache usage: 0.0%.
2025-06-12T15:45:58.924455407Z INFO 06-12 15:45:58 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 18.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 16.6%, CPU KV cache usage: 0.0%.
2025-06-12T15:46:03.950849338Z INFO 06-12 15:46:03 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 16.7%, CPU KV cache usage: 0.0%.
2025-06-12T15:46:08.978064642Z INFO 06-12 15:46:08 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 16.7%, CPU KV cache usage: 0.0%.
2025-06-12T15:46:13.998506226Z INFO 06-12 15:46:13 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 16.8%, CPU KV cache usage: 0.0%.
2025-06-12T15:46:19.041891583Z INFO 06-12 15:46:19 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 16.8%, CPU KV cache usage: 0.0%.
2025-06-12T15:46:24.082477555Z INFO 06-12 15:46:24 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 16.8%, CPU KV cache usage: 0.0%.
2025-06-12T15:46:29.113193086Z INFO 06-12 15:46:29 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 16.9%, CPU KV cache usage: 0.0%.
2025-06-12T15:46:34.123430153Z INFO 06-12 15:46:34 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 18.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 16.9%, CPU KV cache usage: 0.0%.
2025-06-12T15:46:39.129757475Z INFO 06-12 15:46:39 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 17.0%, CPU KV cache usage: 0.0%.
2025-06-12T15:46:44.163359704Z INFO 06-12 15:46:44 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 17.0%, CPU KV cache usage: 0.0%.
2025-06-12T15:46:49.185244963Z INFO 06-12 15:46:49 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 17.1%, CPU KV cache usage: 0.0%.
2025-06-12T15:46:54.200438249Z INFO 06-12 15:46:54 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 17.1%, CPU KV cache usage: 0.0%.
2025-06-12T15:46:59.247751762Z INFO 06-12 15:46:59 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 17.2%, CPU KV cache usage: 0.0%.
2025-06-12T15:47:04.277983800Z INFO 06-12 15:47:04 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 17.2%, CPU KV cache usage: 0.0%.
2025-06-12T15:47:09.303248396Z INFO 06-12 15:47:09 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 17.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 17.3%, CPU KV cache usage: 0.0%.
2025-06-12T15:47:14.353599793Z INFO 06-12 15:47:14 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 17.3%, CPU KV cache usage: 0.0%.
2025-06-12T15:47:19.377545971Z INFO 06-12 15:47:19 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 17.4%, CPU KV cache usage: 0.0%.
2025-06-12T15:47:24.420511060Z INFO 06-12 15:47:24 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 17.4%, CPU KV cache usage: 0.0%.
2025-06-12T15:47:29.471761556Z INFO 06-12 15:47:29 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 17.4%, CPU KV cache usage: 0.0%.
2025-06-12T15:47:34.519281458Z INFO 06-12 15:47:34 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 17.5%, CPU KV cache usage: 0.0%.
2025-06-12T15:47:39.523020389Z INFO 06-12 15:47:39 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 17.5%, CPU KV cache usage: 0.0%.
2025-06-12T15:47:44.571272332Z INFO 06-12 15:47:44 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 17.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 17.6%, CPU KV cache usage: 0.0%.
2025-06-12T15:47:49.579354090Z INFO 06-12 15:47:49 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 17.6%, CPU KV cache usage: 0.0%.
2025-06-12T15:47:54.601800996Z INFO 06-12 15:47:54 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 17.7%, CPU KV cache usage: 0.0%.
2025-06-12T15:47:59.624386847Z INFO 06-12 15:47:59 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 17.7%, CPU KV cache usage: 0.0%.
2025-06-12T15:48:04.651375318Z INFO 06-12 15:48:04 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 17.8%, CPU KV cache usage: 0.0%.
2025-06-12T15:48:09.692637338Z INFO 06-12 15:48:09 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 17.8%, CPU KV cache usage: 0.0%.
2025-06-12T15:48:14.694506501Z INFO 06-12 15:48:14 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 17.9%, CPU KV cache usage: 0.0%.
2025-06-12T15:48:19.756589092Z INFO 06-12 15:48:19 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 17.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 17.9%, CPU KV cache usage: 0.0%.
2025-06-12T15:48:24.779616332Z INFO 06-12 15:48:24 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 18.0%, CPU KV cache usage: 0.0%.
2025-06-12T15:48:29.803907429Z INFO 06-12 15:48:29 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 18.0%, CPU KV cache usage: 0.0%.
2025-06-12T15:48:34.843536173Z INFO 06-12 15:48:34 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 18.1%, CPU KV cache usage: 0.0%.
2025-06-12T15:48:39.889771947Z INFO 06-12 15:48:39 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 18.1%, CPU KV cache usage: 0.0%.
2025-06-12T15:48:44.913866125Z INFO 06-12 15:48:44 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 18.1%, CPU KV cache usage: 0.0%.
2025-06-12T15:48:49.915383666Z INFO 06-12 15:48:49 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 18.2%, CPU KV cache usage: 0.0%.
2025-06-12T15:48:54.947256806Z INFO 06-12 15:48:54 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 17.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 18.2%, CPU KV cache usage: 0.0%.
2025-06-12T15:49:00.002341122Z INFO 06-12 15:49:00 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 18.3%, CPU KV cache usage: 0.0%.
2025-06-12T15:49:05.020162087Z INFO 06-12 15:49:05 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 18.3%, CPU KV cache usage: 0.0%.
2025-06-12T15:49:10.043164377Z INFO 06-12 15:49:10 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 18.4%, CPU KV cache usage: 0.0%.
2025-06-12T15:49:15.057950713Z INFO 06-12 15:49:15 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 18.4%, CPU KV cache usage: 0.0%.
2025-06-12T15:49:20.062724625Z INFO 06-12 15:49:20 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 18.5%, CPU KV cache usage: 0.0%.
2025-06-12T15:49:25.109950485Z INFO 06-12 15:49:25 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 18.5%, CPU KV cache usage: 0.0%.
2025-06-12T15:49:30.129739243Z INFO 06-12 15:49:30 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 17.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 18.6%, CPU KV cache usage: 0.0%.
2025-06-12T15:49:35.137852777Z INFO 06-12 15:49:35 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 18.6%, CPU KV cache usage: 0.0%.
2025-06-12T15:49:40.157176881Z INFO 06-12 15:49:40 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 18.6%, CPU KV cache usage: 0.0%.
2025-06-12T15:49:45.176746893Z INFO 06-12 15:49:45 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 18.7%, CPU KV cache usage: 0.0%.
2025-06-12T15:49:50.194256902Z INFO 06-12 15:49:50 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 18.7%, CPU KV cache usage: 0.0%.
2025-06-12T15:49:55.220148201Z INFO 06-12 15:49:55 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 18.8%, CPU KV cache usage: 0.0%.
2025-06-12T15:50:00.262614946Z INFO 06-12 15:50:00 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 18.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 18.8%, CPU KV cache usage: 0.0%.
2025-06-12T15:50:05.297024757Z INFO 06-12 15:50:05 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 17.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 18.9%, CPU KV cache usage: 0.0%.
2025-06-12T15:50:10.306740245Z INFO 06-12 15:50:10 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 18.9%, CPU KV cache usage: 0.0%.
2025-06-12T15:50:15.327190669Z INFO 06-12 15:50:15 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 19.0%, CPU KV cache usage: 0.0%.
2025-06-12T15:50:20.344128111Z INFO 06-12 15:50:20 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 19.0%, CPU KV cache usage: 0.0%.
2025-06-12T15:50:25.362894475Z INFO 06-12 15:50:25 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 19.1%, CPU KV cache usage: 0.0%.
2025-06-12T15:50:30.392130510Z INFO 06-12 15:50:30 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 19.1%, CPU KV cache usage: 0.0%.
2025-06-12T15:50:35.416109722Z INFO 06-12 15:50:35 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 19.2%, CPU KV cache usage: 0.0%.
2025-06-12T15:50:40.435780465Z INFO 06-12 15:50:40 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 19.2%, CPU KV cache usage: 0.0%.
2025-06-12T15:50:45.441124720Z INFO 06-12 15:50:45 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 19.2%, CPU KV cache usage: 0.0%.
2025-06-12T15:50:50.470949545Z INFO 06-12 15:50:50 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 19.3%, CPU KV cache usage: 0.0%.
2025-06-12T15:50:55.478374445Z INFO 06-12 15:50:55 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 19.3%, CPU KV cache usage: 0.0%.
2025-06-12T15:51:00.497239959Z INFO 06-12 15:51:00 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 19.4%, CPU KV cache usage: 0.0%.
2025-06-12T15:51:05.515265545Z INFO 06-12 15:51:05 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 19.4%, CPU KV cache usage: 0.0%.
2025-06-12T15:51:10.554144763Z INFO 06-12 15:51:10 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 19.5%, CPU KV cache usage: 0.0%.
2025-06-12T15:51:15.576307572Z INFO 06-12 15:51:15 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 19.5%, CPU KV cache usage: 0.0%.
2025-06-12T15:51:20.592529766Z INFO 06-12 15:51:20 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 19.6%, CPU KV cache usage: 0.0%.
2025-06-12T15:51:25.606249152Z INFO 06-12 15:51:25 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 19.6%, CPU KV cache usage: 0.0%.
2025-06-12T15:51:30.623048960Z INFO 06-12 15:51:30 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 19.7%, CPU KV cache usage: 0.0%.
2025-06-12T15:51:35.629007359Z INFO 06-12 15:51:35 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 19.7%, CPU KV cache usage: 0.0%.
2025-06-12T15:51:40.669560205Z INFO 06-12 15:51:40 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 19.8%, CPU KV cache usage: 0.0%.
2025-06-12T15:51:45.708017619Z INFO 06-12 15:51:45 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 19.8%, CPU KV cache usage: 0.0%.
2025-06-12T15:51:50.726470089Z INFO 06-12 15:51:50 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 19.9%, CPU KV cache usage: 0.0%.
2025-06-12T15:51:55.742553577Z INFO 06-12 15:51:55 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 19.9%, CPU KV cache usage: 0.0%.
2025-06-12T15:52:00.777302291Z INFO 06-12 15:52:00 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 20.0%, CPU KV cache usage: 0.0%.
2025-06-12T15:52:05.820183427Z INFO 06-12 15:52:05 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 20.0%, CPU KV cache usage: 0.0%.
2025-06-12T15:52:10.851655041Z INFO 06-12 15:52:10 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 20.1%, CPU KV cache usage: 0.0%.
2025-06-12T15:52:15.854626682Z INFO 06-12 15:52:15 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 20.1%, CPU KV cache usage: 0.0%.
2025-06-12T15:52:20.864628997Z INFO 06-12 15:52:20 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 20.2%, CPU KV cache usage: 0.0%.
2025-06-12T15:52:25.882743474Z INFO 06-12 15:52:25 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 20.2%, CPU KV cache usage: 0.0%.
2025-06-12T15:52:30.895410920Z INFO 06-12 15:52:30 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 20.2%, CPU KV cache usage: 0.0%.
2025-06-12T15:52:35.897825737Z INFO 06-12 15:52:35 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 20.3%, CPU KV cache usage: 0.0%.
2025-06-12T15:52:40.928159995Z INFO 06-12 15:52:40 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 20.3%, CPU KV cache usage: 0.0%.
2025-06-12T15:52:45.945662931Z INFO 06-12 15:52:45 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 20.4%, CPU KV cache usage: 0.0%.
2025-06-12T15:52:50.993973282Z INFO 06-12 15:52:50 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 20.4%, CPU KV cache usage: 0.0%.
2025-06-12T15:52:55.996630879Z INFO 06-12 15:52:55 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 20.5%, CPU KV cache usage: 0.0%.
2025-06-12T15:53:00.997186604Z INFO 06-12 15:53:00 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 20.5%, CPU KV cache usage: 0.0%.
2025-06-12T15:53:06.034727098Z INFO 06-12 15:53:06 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 20.6%, CPU KV cache usage: 0.0%.
2025-06-12T15:53:11.074983328Z INFO 06-12 15:53:11 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 20.6%, CPU KV cache usage: 0.0%.
2025-06-12T15:53:16.099126903Z INFO 06-12 15:53:16 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 20.7%, CPU KV cache usage: 0.0%.
2025-06-12T15:53:21.120827154Z INFO 06-12 15:53:21 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 20.7%, CPU KV cache usage: 0.0%.
2025-06-12T15:53:26.137812462Z INFO 06-12 15:53:26 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 20.8%, CPU KV cache usage: 0.0%.
2025-06-12T15:53:31.162619638Z INFO 06-12 15:53:31 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 20.8%, CPU KV cache usage: 0.0%.
2025-06-12T15:53:36.181662939Z INFO 06-12 15:53:36 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 20.9%, CPU KV cache usage: 0.0%.
2025-06-12T15:53:41.224642388Z INFO 06-12 15:53:41 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 20.9%, CPU KV cache usage: 0.0%.
2025-06-12T15:53:46.270056315Z INFO 06-12 15:53:46 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 21.0%, CPU KV cache usage: 0.0%.
2025-06-12T15:53:51.278346153Z INFO 06-12 15:53:51 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 21.0%, CPU KV cache usage: 0.0%.
2025-06-12T15:53:56.295871542Z INFO 06-12 15:53:56 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 21.0%, CPU KV cache usage: 0.0%.
2025-06-12T15:54:01.303681393Z INFO 06-12 15:54:01 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 21.1%, CPU KV cache usage: 0.0%.
2025-06-12T15:54:06.326559097Z INFO 06-12 15:54:06 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 21.1%, CPU KV cache usage: 0.0%.
2025-06-12T15:54:11.367596084Z INFO 06-12 15:54:11 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 18.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 21.2%, CPU KV cache usage: 0.0%.
2025-06-12T15:54:16.405878269Z INFO 06-12 15:54:16 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 21.2%, CPU KV cache usage: 0.0%.
2025-06-12T15:54:21.421028335Z INFO 06-12 15:54:21 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 15.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 21.3%, CPU KV cache usage: 0.0%.
2025-06-12T15:54:26.464443443Z INFO 06-12 15:54:26 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 12.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 21.3%, CPU KV cache usage: 0.0%.
2025-06-12T15:54:31.483766691Z INFO 06-12 15:54:31 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 18.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 21.3%, CPU KV cache usage: 0.0%.
2025-06-12T15:54:36.530800773Z INFO 06-12 15:54:36 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 18.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 21.4%, CPU KV cache usage: 0.0%.
2025-06-12T15:54:41.531661592Z INFO 06-12 15:54:41 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 18.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 21.4%, CPU KV cache usage: 0.0%.
2025-06-12T15:54:46.595184038Z INFO 06-12 15:54:46 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 11.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 21.5%, CPU KV cache usage: 0.0%.
2025-06-12T15:54:51.669635007Z INFO 06-12 15:54:51 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 18.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 21.5%, CPU KV cache usage: 0.0%.
2025-06-12T15:54:56.700069046Z INFO 06-12 15:54:56 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 18.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 21.5%, CPU KV cache usage: 0.0%.
2025-06-12T15:55:01.740570045Z INFO 06-12 15:55:01 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 15.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 21.6%, CPU KV cache usage: 0.0%.
2025-06-12T15:55:06.781163716Z INFO 06-12 15:55:06 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 21.6%, CPU KV cache usage: 0.0%.
2025-06-12T15:55:11.806950421Z INFO 06-12 15:55:11 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 21.7%, CPU KV cache usage: 0.0%.
2025-06-12T15:55:16.825301435Z INFO 06-12 15:55:16 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 21.7%, CPU KV cache usage: 0.0%.
2025-06-12T15:55:21.865986286Z INFO 06-12 15:55:21 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 17.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 21.8%, CPU KV cache usage: 0.0%.
2025-06-12T15:55:26.884896629Z INFO 06-12 15:55:26 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 21.8%, CPU KV cache usage: 0.0%.
2025-06-12T15:55:31.893354428Z INFO 06-12 15:55:31 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 21.9%, CPU KV cache usage: 0.0%.
2025-06-12T15:55:36.897524433Z INFO 06-12 15:55:36 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 21.9%, CPU KV cache usage: 0.0%.
2025-06-12T15:55:41.933059123Z INFO 06-12 15:55:41 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 22.0%, CPU KV cache usage: 0.0%.
2025-06-12T15:55:46.968478417Z INFO 06-12 15:55:46 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 22.0%, CPU KV cache usage: 0.0%.
2025-06-12T15:55:51.992709048Z INFO 06-12 15:55:51 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 22.1%, CPU KV cache usage: 0.0%.
2025-06-12T15:55:57.014531763Z INFO 06-12 15:55:57 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 22.1%, CPU KV cache usage: 0.0%.
2025-06-12T15:56:02.050338940Z INFO 06-12 15:56:02 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 18.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 22.1%, CPU KV cache usage: 0.0%.
2025-06-12T15:56:07.090346016Z INFO 06-12 15:56:07 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 22.2%, CPU KV cache usage: 0.0%.
2025-06-12T15:56:12.095303670Z INFO 06-12 15:56:12 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 22.2%, CPU KV cache usage: 0.0%.
2025-06-12T15:56:17.099325520Z INFO 06-12 15:56:17 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 22.3%, CPU KV cache usage: 0.0%.
2025-06-12T15:56:22.147579903Z INFO 06-12 15:56:22 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 22.3%, CPU KV cache usage: 0.0%.
2025-06-12T15:56:27.151647483Z INFO 06-12 15:56:27 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 22.4%, CPU KV cache usage: 0.0%.
2025-06-12T15:56:32.161647792Z INFO 06-12 15:56:32 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 22.4%, CPU KV cache usage: 0.0%.
2025-06-12T15:56:37.168186975Z INFO 06-12 15:56:37 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 17.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 22.5%, CPU KV cache usage: 0.0%.
2025-06-12T15:56:42.203286390Z INFO 06-12 15:56:42 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 22.5%, CPU KV cache usage: 0.0%.
2025-06-12T15:56:47.236013727Z INFO 06-12 15:56:47 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 22.6%, CPU KV cache usage: 0.0%.
2025-06-12T15:56:52.257800386Z INFO 06-12 15:56:52 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 22.6%, CPU KV cache usage: 0.0%.
2025-06-12T15:56:57.286101053Z INFO 06-12 15:56:57 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 22.7%, CPU KV cache usage: 0.0%.
2025-06-12T15:57:02.313075577Z INFO 06-12 15:57:02 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 22.7%, CPU KV cache usage: 0.0%.
2025-06-12T15:57:07.315432817Z INFO 06-12 15:57:07 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 22.8%, CPU KV cache usage: 0.0%.
2025-06-12T15:57:12.363272882Z INFO 06-12 15:57:12 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 18.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 22.8%, CPU KV cache usage: 0.0%.
2025-06-12T15:57:17.394191892Z INFO 06-12 15:57:17 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 22.9%, CPU KV cache usage: 0.0%.
2025-06-12T15:57:22.439620522Z INFO 06-12 15:57:22 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 21.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 22.9%, CPU KV cache usage: 0.0%.
2025-06-12T15:57:27.444701218Z INFO 06-12 15:57:27 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 23.0%, CPU KV cache usage: 0.0%.
2025-06-12T15:57:32.455134636Z INFO 06-12 15:57:32 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 23.0%, CPU KV cache usage: 0.0%.
2025-06-12T15:57:37.468866207Z INFO 06-12 15:57:37 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 23.0%, CPU KV cache usage: 0.0%.
2025-06-12T15:57:42.516712448Z INFO 06-12 15:57:42 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 23.1%, CPU KV cache usage: 0.0%.
2025-06-12T15:57:47.540829240Z INFO 06-12 15:57:47 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 14.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 23.1%, CPU KV cache usage: 0.0%.
2025-06-12T15:57:52.544655872Z INFO 06-12 15:57:52 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 23.2%, CPU KV cache usage: 0.0%.
2025-06-12T15:57:57.570789783Z INFO 06-12 15:57:57 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 18.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 23.2%, CPU KV cache usage: 0.0%.
2025-06-12T15:58:02.615166676Z INFO 06-12 15:58:02 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 23.3%, CPU KV cache usage: 0.0%.
2025-06-12T15:58:07.643894641Z INFO 06-12 15:58:07 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 23.3%, CPU KV cache usage: 0.0%.
2025-06-12T15:58:12.659011114Z INFO 06-12 15:58:12 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 23.4%, CPU KV cache usage: 0.0%.
2025-06-12T15:58:17.664799884Z INFO 06-12 15:58:17 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 23.4%, CPU KV cache usage: 0.0%.
2025-06-12T15:58:22.707551113Z INFO 06-12 15:58:22 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 17.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 23.4%, CPU KV cache usage: 0.0%.
2025-06-12T15:58:27.744610549Z INFO 06-12 15:58:27 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 23.5%, CPU KV cache usage: 0.0%.
2025-06-12T15:58:32.795695283Z INFO 06-12 15:58:32 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 23.5%, CPU KV cache usage: 0.0%.
2025-06-12T15:58:37.804335133Z INFO 06-12 15:58:37 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 23.6%, CPU KV cache usage: 0.0%.
2025-06-12T15:58:42.848401719Z INFO 06-12 15:58:42 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 23.6%, CPU KV cache usage: 0.0%.
2025-06-12T15:58:47.855497744Z INFO 06-12 15:58:47 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 23.7%, CPU KV cache usage: 0.0%.
2025-06-12T15:58:52.861154131Z INFO 06-12 15:58:52 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 23.7%, CPU KV cache usage: 0.0%.
2025-06-12T15:58:57.931059249Z INFO 06-12 15:58:57 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 14.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 23.8%, CPU KV cache usage: 0.0%.
2025-06-12T15:59:03.051340345Z INFO 06-12 15:59:03 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 13.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 23.8%, CPU KV cache usage: 0.0%.
2025-06-12T15:59:08.067431094Z INFO 06-12 15:59:08 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 23.8%, CPU KV cache usage: 0.0%.
2025-06-12T15:59:13.090721681Z INFO 06-12 15:59:13 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 23.9%, CPU KV cache usage: 0.0%.
2025-06-12T15:59:18.133236460Z INFO 06-12 15:59:18 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 23.9%, CPU KV cache usage: 0.0%.
2025-06-12T15:59:23.146858161Z INFO 06-12 15:59:23 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 23.9%, CPU KV cache usage: 0.0%.
2025-06-12T15:59:28.147309812Z INFO 06-12 15:59:28 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 24.0%, CPU KV cache usage: 0.0%.
2025-06-12T15:59:33.190138545Z INFO 06-12 15:59:33 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 24.0%, CPU KV cache usage: 0.0%.
2025-06-12T15:59:38.201517924Z INFO 06-12 15:59:38 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 24.1%, CPU KV cache usage: 0.0%.
2025-06-12T15:59:43.232294955Z INFO 06-12 15:59:43 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 16.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 24.1%, CPU KV cache usage: 0.0%.
2025-06-12T15:59:48.275594311Z INFO 06-12 15:59:48 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 13.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 24.2%, CPU KV cache usage: 0.0%.
2025-06-12T15:59:53.364908954Z INFO 06-12 15:59:53 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 11.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 24.2%, CPU KV cache usage: 0.0%.
2025-06-12T15:59:58.365498307Z INFO 06-12 15:59:58 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 17.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 24.2%, CPU KV cache usage: 0.0%.
2025-06-12T16:00:03.398904083Z INFO 06-12 16:00:03 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 24.3%, CPU KV cache usage: 0.0%.
2025-06-12T16:00:08.446949847Z INFO 06-12 16:00:08 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 24.3%, CPU KV cache usage: 0.0%.
2025-06-12T16:00:13.449550505Z INFO 06-12 16:00:13 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 24.4%, CPU KV cache usage: 0.0%.
2025-06-12T16:00:18.474865544Z INFO 06-12 16:00:18 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 24.4%, CPU KV cache usage: 0.0%.
2025-06-12T16:00:23.497810976Z INFO 06-12 16:00:23 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 24.5%, CPU KV cache usage: 0.0%.
2025-06-12T16:00:28.523580305Z INFO 06-12 16:00:28 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 24.5%, CPU KV cache usage: 0.0%.
2025-06-12T16:00:33.568866148Z INFO 06-12 16:00:33 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 9.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 24.5%, CPU KV cache usage: 0.0%.
2025-06-12T16:00:38.611621288Z INFO 06-12 16:00:38 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 24.6%, CPU KV cache usage: 0.0%.
2025-06-12T16:00:43.645897348Z INFO 06-12 16:00:43 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 24.6%, CPU KV cache usage: 0.0%.
2025-06-12T16:00:48.688884155Z INFO 06-12 16:00:48 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 24.7%, CPU KV cache usage: 0.0%.
2025-06-12T16:00:53.725137528Z INFO 06-12 16:00:53 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 24.7%, CPU KV cache usage: 0.0%.
2025-06-12T16:00:58.754102668Z INFO 06-12 16:00:58 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 24.8%, CPU KV cache usage: 0.0%.
2025-06-12T16:01:03.783673239Z INFO 06-12 16:01:03 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 24.8%, CPU KV cache usage: 0.0%.
2025-06-12T16:01:08.930259109Z INFO 06-12 16:01:08 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 16.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 24.8%, CPU KV cache usage: 0.0%.
2025-06-12T16:01:13.962909925Z INFO 06-12 16:01:13 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 12.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 24.9%, CPU KV cache usage: 0.0%.
2025-06-12T16:01:18.964301748Z INFO 06-12 16:01:18 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 24.9%, CPU KV cache usage: 0.0%.
2025-06-12T16:01:23.999366326Z INFO 06-12 16:01:23 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 25.0%, CPU KV cache usage: 0.0%.
2025-06-12T16:01:29.027501762Z INFO 06-12 16:01:29 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 25.0%, CPU KV cache usage: 0.0%.
2025-06-12T16:01:34.053422767Z INFO 06-12 16:01:34 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 25.1%, CPU KV cache usage: 0.0%.
2025-06-12T16:01:39.078926197Z INFO 06-12 16:01:39 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 25.1%, CPU KV cache usage: 0.0%.
2025-06-12T16:01:44.118083397Z INFO 06-12 16:01:44 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 25.2%, CPU KV cache usage: 0.0%.
2025-06-12T16:01:49.131387944Z INFO 06-12 16:01:49 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 18.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 25.2%, CPU KV cache usage: 0.0%.
2025-06-12T16:01:54.176162874Z INFO 06-12 16:01:54 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 25.2%, CPU KV cache usage: 0.0%.
2025-06-12T16:01:59.211922003Z INFO 06-12 16:01:59 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 25.3%, CPU KV cache usage: 0.0%.
2025-06-12T16:02:04.245429282Z INFO 06-12 16:02:04 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 25.3%, CPU KV cache usage: 0.0%.
2025-06-12T16:02:09.255376128Z INFO 06-12 16:02:09 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 25.4%, CPU KV cache usage: 0.0%.
2025-06-12T16:02:14.263844441Z INFO 06-12 16:02:14 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 25.4%, CPU KV cache usage: 0.0%.
2025-06-12T16:02:19.286423084Z INFO 06-12 16:02:19 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 25.5%, CPU KV cache usage: 0.0%.
2025-06-12T16:02:24.336639547Z INFO 06-12 16:02:24 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 25.5%, CPU KV cache usage: 0.0%.
2025-06-12T16:02:29.402311957Z INFO 06-12 16:02:29 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 15.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 25.5%, CPU KV cache usage: 0.0%.
2025-06-12T16:02:34.408049840Z INFO 06-12 16:02:34 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 12.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 25.6%, CPU KV cache usage: 0.0%.
2025-06-12T16:02:39.445030379Z INFO 06-12 16:02:39 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 25.6%, CPU KV cache usage: 0.0%.
2025-06-12T16:02:44.466717229Z INFO 06-12 16:02:44 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 25.7%, CPU KV cache usage: 0.0%.
2025-06-12T16:02:49.522810114Z INFO 06-12 16:02:49 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 25.7%, CPU KV cache usage: 0.0%.
2025-06-12T16:02:54.539710685Z INFO 06-12 16:02:54 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 25.7%, CPU KV cache usage: 0.0%.
2025-06-12T16:02:59.545163957Z INFO 06-12 16:02:59 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 25.8%, CPU KV cache usage: 0.0%.
2025-06-12T16:03:04.548065025Z INFO 06-12 16:03:04 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 25.8%, CPU KV cache usage: 0.0%.
2025-06-12T16:03:09.595272628Z INFO 06-12 16:03:09 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 12.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 25.9%, CPU KV cache usage: 0.0%.
2025-06-12T16:03:14.614403951Z INFO 06-12 16:03:14 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 16.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 25.9%, CPU KV cache usage: 0.0%.
2025-06-12T16:03:19.625417920Z INFO 06-12 16:03:19 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 26.0%, CPU KV cache usage: 0.0%.
2025-06-12T16:03:24.667716812Z INFO 06-12 16:03:24 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 18.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 26.0%, CPU KV cache usage: 0.0%.
2025-06-12T16:03:29.676241644Z INFO 06-12 16:03:29 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 26.0%, CPU KV cache usage: 0.0%.
2025-06-12T16:03:34.719944898Z INFO 06-12 16:03:34 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 17.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 26.1%, CPU KV cache usage: 0.0%.
2025-06-12T16:03:39.741133444Z INFO 06-12 16:03:39 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 16.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 26.1%, CPU KV cache usage: 0.0%.
2025-06-12T16:03:44.789142241Z INFO 06-12 16:03:44 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 18.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 26.2%, CPU KV cache usage: 0.0%.
2025-06-12T16:03:49.828709023Z INFO 06-12 16:03:49 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 14.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 26.2%, CPU KV cache usage: 0.0%.
2025-06-12T16:03:54.840904119Z INFO 06-12 16:03:54 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 11.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 26.2%, CPU KV cache usage: 0.0%.
2025-06-12T16:03:59.864481858Z INFO 06-12 16:03:59 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 17.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 26.3%, CPU KV cache usage: 0.0%.
2025-06-12T16:04:04.905241243Z INFO 06-12 16:04:04 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 17.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 26.3%, CPU KV cache usage: 0.0%.
2025-06-12T16:04:09.926095555Z INFO 06-12 16:04:09 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 17.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 26.4%, CPU KV cache usage: 0.0%.
2025-06-12T16:04:14.932143843Z INFO 06-12 16:04:14 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 16.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 26.4%, CPU KV cache usage: 0.0%.
2025-06-12T16:04:19.966124738Z INFO 06-12 16:04:19 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 17.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 26.4%, CPU KV cache usage: 0.0%.
2025-06-12T16:04:24.995598726Z INFO 06-12 16:04:24 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 17.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 26.5%, CPU KV cache usage: 0.0%.
2025-06-12T16:04:30.027608796Z INFO 06-12 16:04:30 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 18.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 26.5%, CPU KV cache usage: 0.0%.
2025-06-12T16:04:35.059586245Z INFO 06-12 16:04:35 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 16.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 26.6%, CPU KV cache usage: 0.0%.
2025-06-12T16:04:40.136203155Z INFO 06-12 16:04:40 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 15.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 26.6%, CPU KV cache usage: 0.0%.
2025-06-12T16:04:45.198752949Z INFO 06-12 16:04:45 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 15.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 26.6%, CPU KV cache usage: 0.0%.
2025-06-12T16:04:50.231366588Z INFO 06-12 16:04:50 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 14.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 26.7%, CPU KV cache usage: 0.0%.
2025-06-12T16:04:55.273870506Z INFO 06-12 16:04:55 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 26.7%, CPU KV cache usage: 0.0%.
2025-06-12T16:05:00.446400345Z INFO 06-12 16:05:00 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 13.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 26.7%, CPU KV cache usage: 0.0%.
2025-06-12T16:05:05.481033144Z INFO 06-12 16:05:05 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 9.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 26.7%, CPU KV cache usage: 0.0%.
2025-06-12T16:05:10.501007066Z INFO 06-12 16:05:10 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 16.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 26.8%, CPU KV cache usage: 0.0%.
2025-06-12T16:05:15.515241450Z INFO 06-12 16:05:15 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 16.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 26.8%, CPU KV cache usage: 0.0%.
2025-06-12T16:05:20.540404703Z INFO 06-12 16:05:20 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 26.8%, CPU KV cache usage: 0.0%.
2025-06-12T16:05:25.552074330Z INFO 06-12 16:05:25 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 16.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 26.9%, CPU KV cache usage: 0.0%.
2025-06-12T16:05:30.570830144Z INFO 06-12 16:05:30 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 17.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 26.9%, CPU KV cache usage: 0.0%.
2025-06-12T16:05:35.583278728Z INFO 06-12 16:05:35 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 16.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 27.0%, CPU KV cache usage: 0.0%.
2025-06-12T16:05:40.595087199Z INFO 06-12 16:05:40 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 16.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 27.0%, CPU KV cache usage: 0.0%.
2025-06-12T16:05:45.617706391Z INFO 06-12 16:05:45 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 13.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 27.0%, CPU KV cache usage: 0.0%.
2025-06-12T16:05:50.651900808Z INFO 06-12 16:05:50 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 14.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 27.1%, CPU KV cache usage: 0.0%.
2025-06-12T16:05:55.673276636Z INFO 06-12 16:05:55 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 27.1%, CPU KV cache usage: 0.0%.
2025-06-12T16:06:00.686230559Z INFO 06-12 16:06:00 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 27.2%, CPU KV cache usage: 0.0%.
2025-06-12T16:06:05.778589861Z INFO 06-12 16:06:05 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 12.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 27.2%, CPU KV cache usage: 0.0%.
2025-06-12T16:06:10.817895372Z INFO 06-12 16:06:10 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 16.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 27.2%, CPU KV cache usage: 0.0%.
2025-06-12T16:06:15.840511085Z INFO 06-12 16:06:15 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 27.3%, CPU KV cache usage: 0.0%.
2025-06-12T16:06:20.868842799Z INFO 06-12 16:06:20 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 27.3%, CPU KV cache usage: 0.0%.
2025-06-12T16:06:25.895473474Z INFO 06-12 16:06:25 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 14.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 27.4%, CPU KV cache usage: 0.0%.
2025-06-12T16:06:30.897338442Z INFO 06-12 16:06:30 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 27.4%, CPU KV cache usage: 0.0%.
2025-06-12T16:06:35.901938921Z INFO 06-12 16:06:35 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 27.4%, CPU KV cache usage: 0.0%.
2025-06-12T16:06:40.935593323Z INFO 06-12 16:06:40 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 27.5%, CPU KV cache usage: 0.0%.
2025-06-12T16:06:45.979734063Z INFO 06-12 16:06:45 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 27.5%, CPU KV cache usage: 0.0%.
2025-06-12T16:06:51.019658047Z INFO 06-12 16:06:51 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 27.6%, CPU KV cache usage: 0.0%.
2025-06-12T16:06:56.048813566Z INFO 06-12 16:06:56 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 27.6%, CPU KV cache usage: 0.0%.
2025-06-12T16:07:01.098051452Z INFO 06-12 16:07:01 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 18.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 27.7%, CPU KV cache usage: 0.0%.
2025-06-12T16:07:06.108154855Z INFO 06-12 16:07:06 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 16.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 27.7%, CPU KV cache usage: 0.0%.
2025-06-12T16:07:11.146351254Z INFO 06-12 16:07:11 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 27.8%, CPU KV cache usage: 0.0%.
2025-06-12T16:07:16.165548786Z INFO 06-12 16:07:16 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 27.8%, CPU KV cache usage: 0.0%.
2025-06-12T16:07:21.178889262Z INFO 06-12 16:07:21 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 27.9%, CPU KV cache usage: 0.0%.
2025-06-12T16:07:26.226225409Z INFO 06-12 16:07:26 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 27.9%, CPU KV cache usage: 0.0%.
2025-06-12T16:07:31.237767381Z INFO 06-12 16:07:31 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 28.0%, CPU KV cache usage: 0.0%.
2025-06-12T16:07:36.242494865Z INFO 06-12 16:07:36 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 28.0%, CPU KV cache usage: 0.0%.
2025-06-12T16:07:41.271408160Z INFO 06-12 16:07:41 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 13.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 28.0%, CPU KV cache usage: 0.0%.
2025-06-12T16:07:46.277024397Z INFO 06-12 16:07:46 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 28.1%, CPU KV cache usage: 0.0%.
2025-06-12T16:07:51.300706893Z INFO 06-12 16:07:51 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 28.1%, CPU KV cache usage: 0.0%.
2025-06-12T16:07:56.322085717Z INFO 06-12 16:07:56 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 9.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 28.1%, CPU KV cache usage: 0.0%.
2025-06-12T16:08:01.323578657Z INFO 06-12 16:08:01 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 18.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 28.2%, CPU KV cache usage: 0.0%.
2025-06-12T16:08:06.371141106Z INFO 06-12 16:08:06 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 28.2%, CPU KV cache usage: 0.0%.
2025-06-12T16:08:11.407726004Z INFO 06-12 16:08:11 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 28.3%, CPU KV cache usage: 0.0%.
2025-06-12T16:08:16.431584391Z INFO 06-12 16:08:16 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 13.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 28.3%, CPU KV cache usage: 0.0%.
2025-06-12T16:08:21.455680836Z INFO 06-12 16:08:21 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 28.4%, CPU KV cache usage: 0.0%.
2025-06-12T16:08:26.491504652Z INFO 06-12 16:08:26 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 28.4%, CPU KV cache usage: 0.0%.
2025-06-12T16:08:31.527154457Z INFO 06-12 16:08:31 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 28.5%, CPU KV cache usage: 0.0%.
2025-06-12T16:08:36.588764257Z INFO 06-12 16:08:36 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 28.5%, CPU KV cache usage: 0.0%.
2025-06-12T16:08:41.638112913Z INFO 06-12 16:08:41 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 28.5%, CPU KV cache usage: 0.0%.
2025-06-12T16:08:46.658643066Z INFO 06-12 16:08:46 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 28.6%, CPU KV cache usage: 0.0%.
2025-06-12T16:08:51.743941550Z INFO 06-12 16:08:51 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 15.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 28.6%, CPU KV cache usage: 0.0%.
2025-06-12T16:08:56.744619088Z INFO 06-12 16:08:56 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 16.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 28.6%, CPU KV cache usage: 0.0%.
2025-06-12T16:09:01.749547762Z INFO 06-12 16:09:01 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 28.7%, CPU KV cache usage: 0.0%.
2025-06-12T16:09:06.775297830Z INFO 06-12 16:09:06 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 28.7%, CPU KV cache usage: 0.0%.
2025-06-12T16:09:11.815751953Z INFO 06-12 16:09:11 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 18.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 28.8%, CPU KV cache usage: 0.0%.
2025-06-12T16:09:16.825476101Z INFO 06-12 16:09:16 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 28.8%, CPU KV cache usage: 0.0%.
2025-06-12T16:09:21.830943089Z INFO 06-12 16:09:21 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 28.9%, CPU KV cache usage: 0.0%.
2025-06-12T16:09:26.881629229Z INFO 06-12 16:09:26 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 28.9%, CPU KV cache usage: 0.0%.
2025-06-12T16:09:31.884741532Z INFO 06-12 16:09:31 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 14.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 29.0%, CPU KV cache usage: 0.0%.
2025-06-12T16:09:36.916207398Z INFO 06-12 16:09:36 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 12.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 29.0%, CPU KV cache usage: 0.0%.
2025-06-12T16:09:41.944179612Z INFO 06-12 16:09:41 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 17.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 29.0%, CPU KV cache usage: 0.0%.
2025-06-12T16:09:46.951932570Z INFO 06-12 16:09:46 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 20.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 29.1%, CPU KV cache usage: 0.0%.
2025-06-12T16:09:51.995869891Z INFO 06-12 16:09:51 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 29.1%, CPU KV cache usage: 0.0%.
2025-06-12T16:09:57.023763521Z INFO 06-12 16:09:57 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 29.2%, CPU KV cache usage: 0.0%.
2025-06-12T16:10:02.055063786Z INFO 06-12 16:10:02 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 29.2%, CPU KV cache usage: 0.0%.
2025-06-12T16:10:07.175566298Z INFO 06-12 16:10:07 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 29.2%, CPU KV cache usage: 0.0%.
2025-06-12T16:10:12.198539645Z INFO 06-12 16:10:12 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 14.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 29.3%, CPU KV cache usage: 0.0%.
2025-06-12T16:10:17.221421256Z INFO 06-12 16:10:17 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 29.3%, CPU KV cache usage: 0.0%.
2025-06-12T16:10:22.256829896Z INFO 06-12 16:10:22 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 29.4%, CPU KV cache usage: 0.0%.
2025-06-12T16:10:27.272109183Z INFO 06-12 16:10:27 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 29.4%, CPU KV cache usage: 0.0%.
2025-06-12T16:10:32.302938384Z INFO 06-12 16:10:32 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 16.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 29.5%, CPU KV cache usage: 0.0%.
2025-06-12T16:10:37.310929871Z INFO 06-12 16:10:37 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 13.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 29.5%, CPU KV cache usage: 0.0%.
2025-06-12T16:10:42.333490136Z INFO 06-12 16:10:42 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 29.5%, CPU KV cache usage: 0.0%.
2025-06-12T16:10:47.381444344Z INFO 06-12 16:10:47 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 29.6%, CPU KV cache usage: 0.0%.
2025-06-12T16:10:52.395684253Z INFO 06-12 16:10:52 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 29.6%, CPU KV cache usage: 0.0%.
2025-06-12T16:10:57.410347130Z INFO 06-12 16:10:57 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 29.7%, CPU KV cache usage: 0.0%.
2025-06-12T16:11:02.450603022Z INFO 06-12 16:11:02 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 29.7%, CPU KV cache usage: 0.0%.
2025-06-12T16:11:07.464473337Z INFO 06-12 16:11:07 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 17.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 29.8%, CPU KV cache usage: 0.0%.
2025-06-12T16:11:12.534616608Z INFO 06-12 16:11:12 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 15.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 29.8%, CPU KV cache usage: 0.0%.
2025-06-12T16:11:17.541171750Z INFO 06-12 16:11:17 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 9.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 29.8%, CPU KV cache usage: 0.0%.
2025-06-12T16:11:22.583328199Z INFO 06-12 16:11:22 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 29.9%, CPU KV cache usage: 0.0%.
2025-06-12T16:11:27.609239849Z INFO 06-12 16:11:27 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 29.9%, CPU KV cache usage: 0.0%.
2025-06-12T16:11:32.626189817Z INFO 06-12 16:11:32 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 12.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 29.9%, CPU KV cache usage: 0.0%.
2025-06-12T16:11:37.672188739Z INFO 06-12 16:11:37 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 12.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 30.0%, CPU KV cache usage: 0.0%.
2025-06-12T16:11:42.699870096Z INFO 06-12 16:11:42 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 30.0%, CPU KV cache usage: 0.0%.
2025-06-12T16:11:47.699961428Z INFO 06-12 16:11:47 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 30.1%, CPU KV cache usage: 0.0%.
2025-06-12T16:11:52.723328244Z INFO 06-12 16:11:52 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 18.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 30.1%, CPU KV cache usage: 0.0%.
2025-06-12T16:11:57.742582843Z INFO 06-12 16:11:57 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 30.1%, CPU KV cache usage: 0.0%.
2025-06-12T16:12:02.744183979Z INFO 06-12 16:12:02 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 30.2%, CPU KV cache usage: 0.0%.
2025-06-12T16:12:07.815519913Z INFO 06-12 16:12:07 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 17.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 30.2%, CPU KV cache usage: 0.0%.
2025-06-12T16:12:12.921970108Z INFO 06-12 16:12:12 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 12.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 30.3%, CPU KV cache usage: 0.0%.
2025-06-12T16:12:17.961744696Z INFO 06-12 16:12:17 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 15.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 30.3%, CPU KV cache usage: 0.0%.
2025-06-12T16:12:22.994833949Z INFO 06-12 16:12:22 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 30.3%, CPU KV cache usage: 0.0%.
2025-06-12T16:12:28.000603947Z INFO 06-12 16:12:28 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 30.4%, CPU KV cache usage: 0.0%.
2025-06-12T16:12:33.028246828Z INFO 06-12 16:12:33 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 30.4%, CPU KV cache usage: 0.0%.
2025-06-12T16:12:38.029661303Z INFO 06-12 16:12:38 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 30.5%, CPU KV cache usage: 0.0%.
2025-06-12T16:12:43.074289434Z INFO 06-12 16:12:43 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 30.5%, CPU KV cache usage: 0.0%.
2025-06-12T16:12:48.137875482Z INFO 06-12 16:12:48 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 18.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 30.6%, CPU KV cache usage: 0.0%.
2025-06-12T16:12:53.140071364Z INFO 06-12 16:12:53 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 15.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 30.6%, CPU KV cache usage: 0.0%.
2025-06-12T16:12:58.179852818Z INFO 06-12 16:12:58 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 30.7%, CPU KV cache usage: 0.0%.
2025-06-12T16:13:03.206272577Z INFO 06-12 16:13:03 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 30.7%, CPU KV cache usage: 0.0%.
2025-06-12T16:13:08.222990099Z INFO 06-12 16:13:08 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 30.7%, CPU KV cache usage: 0.0%.
2025-06-12T16:13:13.253039031Z INFO 06-12 16:13:13 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 30.8%, CPU KV cache usage: 0.0%.
2025-06-12T16:13:18.266075708Z INFO 06-12 16:13:18 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 30.8%, CPU KV cache usage: 0.0%.
2025-06-12T16:13:23.367834788Z INFO 06-12 16:13:23 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 17.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 30.9%, CPU KV cache usage: 0.0%.
2025-06-12T16:13:28.375763361Z INFO 06-12 16:13:28 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 16.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 30.9%, CPU KV cache usage: 0.0%.
2025-06-12T16:13:33.376546956Z INFO 06-12 16:13:33 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 18.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 31.0%, CPU KV cache usage: 0.0%.
2025-06-12T16:13:38.424702742Z INFO 06-12 16:13:38 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 31.0%, CPU KV cache usage: 0.0%.
2025-06-12T16:13:43.460631978Z INFO 06-12 16:13:43 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 18.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 31.0%, CPU KV cache usage: 0.0%.
2025-06-12T16:13:48.464467557Z INFO 06-12 16:13:48 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 15.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 31.1%, CPU KV cache usage: 0.0%.
2025-06-12T16:13:53.489252285Z INFO 06-12 16:13:53 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 31.1%, CPU KV cache usage: 0.0%.
2025-06-12T16:13:58.533460706Z INFO 06-12 16:13:58 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 31.2%, CPU KV cache usage: 0.0%.
2025-06-12T16:14:03.559260637Z INFO 06-12 16:14:03 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 17.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 31.2%, CPU KV cache usage: 0.0%.
2025-06-12T16:14:08.591444813Z INFO 06-12 16:14:08 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 16.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 31.3%, CPU KV cache usage: 0.0%.
2025-06-12T16:14:13.615103448Z INFO 06-12 16:14:13 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 13.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 31.3%, CPU KV cache usage: 0.0%.
2025-06-12T16:14:18.683643344Z INFO 06-12 16:14:18 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 12.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 31.3%, CPU KV cache usage: 0.0%.
2025-06-12T16:14:23.787647106Z INFO 06-12 16:14:23 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 13.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 31.4%, CPU KV cache usage: 0.0%.
2025-06-12T16:14:28.827352772Z INFO 06-12 16:14:28 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 31.4%, CPU KV cache usage: 0.0%.
2025-06-12T16:14:33.830162499Z INFO 06-12 16:14:33 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 12.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 31.4%, CPU KV cache usage: 0.0%.
2025-06-12T16:14:38.849587954Z INFO 06-12 16:14:38 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 31.4%, CPU KV cache usage: 0.0%.
2025-06-12T16:14:43.953522743Z INFO 06-12 16:14:43 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 8.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 31.4%, CPU KV cache usage: 0.0%.
2025-06-12T16:14:49.007641134Z INFO 06-12 16:14:49 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 10.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 31.5%, CPU KV cache usage: 0.0%.
2025-06-12T16:14:54.131008584Z INFO 06-12 16:14:54 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 7.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 31.5%, CPU KV cache usage: 0.0%.
2025-06-12T16:14:59.174270055Z INFO 06-12 16:14:59 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 14.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 31.5%, CPU KV cache usage: 0.0%.
2025-06-12T16:15:04.179153907Z INFO 06-12 16:15:04 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 31.6%, CPU KV cache usage: 0.0%.
2025-06-12T16:15:09.224228488Z INFO 06-12 16:15:09 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 31.6%, CPU KV cache usage: 0.0%.
2025-06-12T16:15:14.227813327Z INFO 06-12 16:15:14 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 31.7%, CPU KV cache usage: 0.0%.
2025-06-12T16:15:19.240255896Z INFO 06-12 16:15:19 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 18.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 31.7%, CPU KV cache usage: 0.0%.
2025-06-12T16:15:24.258217068Z INFO 06-12 16:15:24 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 18.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 31.7%, CPU KV cache usage: 0.0%.
2025-06-12T16:15:29.283664460Z INFO 06-12 16:15:29 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 18.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 31.8%, CPU KV cache usage: 0.0%.
2025-06-12T16:15:34.376528440Z INFO 06-12 16:15:34 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 12.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 31.8%, CPU KV cache usage: 0.0%.
2025-06-12T16:15:39.433613088Z INFO 06-12 16:15:39 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 9.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 31.8%, CPU KV cache usage: 0.0%.
2025-06-12T16:15:44.464382583Z INFO 06-12 16:15:44 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 15.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 31.9%, CPU KV cache usage: 0.0%.
2025-06-12T16:15:49.496184255Z INFO 06-12 16:15:49 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 18.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 31.9%, CPU KV cache usage: 0.0%.
2025-06-12T16:15:54.520575344Z INFO 06-12 16:15:54 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 32.0%, CPU KV cache usage: 0.0%.
2025-06-12T16:15:59.535594865Z INFO 06-12 16:15:59 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 18.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 32.0%, CPU KV cache usage: 0.0%.
2025-06-12T16:16:04.588234525Z INFO 06-12 16:16:04 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 32.1%, CPU KV cache usage: 0.0%.
2025-06-12T16:16:09.594075232Z INFO 06-12 16:16:09 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 32.1%, CPU KV cache usage: 0.0%.
2025-06-12T16:16:14.633778041Z INFO 06-12 16:16:14 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 17.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 32.1%, CPU KV cache usage: 0.0%.
2025-06-12T16:16:19.669456787Z INFO 06-12 16:16:19 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 17.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 32.2%, CPU KV cache usage: 0.0%.
2025-06-12T16:16:24.717527255Z INFO 06-12 16:16:24 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 32.2%, CPU KV cache usage: 0.0%.
2025-06-12T16:16:29.761487744Z INFO 06-12 16:16:29 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 12.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 32.3%, CPU KV cache usage: 0.0%.
2025-06-12T16:16:34.763352551Z INFO 06-12 16:16:34 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 12.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 32.3%, CPU KV cache usage: 0.0%.
2025-06-12T16:16:39.930129217Z INFO 06-12 16:16:39 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 9.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 32.3%, CPU KV cache usage: 0.0%.
2025-06-12T16:16:44.936045397Z INFO 06-12 16:16:44 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 14.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 32.3%, CPU KV cache usage: 0.0%.
2025-06-12T16:16:49.984963377Z INFO 06-12 16:16:49 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 16.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 32.4%, CPU KV cache usage: 0.0%.
2025-06-12T16:16:55.015391403Z INFO 06-12 16:16:55 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 15.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 32.4%, CPU KV cache usage: 0.0%.
2025-06-12T16:17:00.057339538Z INFO 06-12 16:17:00 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 15.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 32.5%, CPU KV cache usage: 0.0%.
2025-06-12T16:17:05.121542690Z INFO 06-12 16:17:05 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 17.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 32.5%, CPU KV cache usage: 0.0%.
2025-06-12T16:17:10.141224410Z INFO 06-12 16:17:10 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 12.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 32.5%, CPU KV cache usage: 0.0%.
2025-06-12T16:17:15.175450530Z INFO 06-12 16:17:15 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 13.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 32.6%, CPU KV cache usage: 0.0%.
2025-06-12T16:17:20.184734563Z INFO 06-12 16:17:20 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 15.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 32.6%, CPU KV cache usage: 0.0%.
2025-06-12T16:17:25.235179045Z INFO 06-12 16:17:25 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 18.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 32.6%, CPU KV cache usage: 0.0%.
2025-06-12T16:17:30.305805167Z INFO 06-12 16:17:30 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 17.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 32.7%, CPU KV cache usage: 0.0%.
2025-06-12T16:17:35.346546582Z INFO 06-12 16:17:35 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 18.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 32.7%, CPU KV cache usage: 0.0%.
2025-06-12T16:17:40.392290596Z INFO 06-12 16:17:40 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 32.8%, CPU KV cache usage: 0.0%.
2025-06-12T16:17:45.396648047Z INFO 06-12 16:17:45 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 32.8%, CPU KV cache usage: 0.0%.
2025-06-12T16:17:50.397123784Z INFO 06-12 16:17:50 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 32.9%, CPU KV cache usage: 0.0%.
2025-06-12T16:17:55.436030650Z INFO 06-12 16:17:55 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 32.9%, CPU KV cache usage: 0.0%.
2025-06-12T16:18:00.475519511Z INFO 06-12 16:18:00 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 33.0%, CPU KV cache usage: 0.0%.
2025-06-12T16:18:05.485310394Z INFO 06-12 16:18:05 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 33.0%, CPU KV cache usage: 0.0%.
2025-06-12T16:18:10.516115782Z INFO 06-12 16:18:10 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 13.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 33.0%, CPU KV cache usage: 0.0%.
2025-06-12T16:18:15.532072382Z INFO 06-12 16:18:15 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 16.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 33.1%, CPU KV cache usage: 0.0%.
2025-06-12T16:18:20.595889645Z INFO 06-12 16:18:20 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 17.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 33.1%, CPU KV cache usage: 0.0%.
2025-06-12T16:18:25.608937077Z INFO 06-12 16:18:25 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 17.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 33.2%, CPU KV cache usage: 0.0%.
2025-06-12T16:18:30.614109500Z INFO 06-12 16:18:30 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 18.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 33.2%, CPU KV cache usage: 0.0%.
2025-06-12T16:18:35.654760335Z INFO 06-12 16:18:35 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 17.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 33.2%, CPU KV cache usage: 0.0%.
2025-06-12T16:18:40.656051641Z INFO 06-12 16:18:40 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 33.3%, CPU KV cache usage: 0.0%.
2025-06-12T16:18:45.695472044Z INFO 06-12 16:18:45 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 33.3%, CPU KV cache usage: 0.0%.
2025-06-12T16:18:50.705153426Z INFO 06-12 16:18:50 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 33.4%, CPU KV cache usage: 0.0%.
2025-06-12T16:18:55.723548168Z INFO 06-12 16:18:55 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 33.4%, CPU KV cache usage: 0.0%.
2025-06-12T16:19:00.736894586Z INFO 06-12 16:19:00 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 33.5%, CPU KV cache usage: 0.0%.
2025-06-12T16:19:05.765299665Z INFO 06-12 16:19:05 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 33.5%, CPU KV cache usage: 0.0%.
2025-06-12T16:19:10.872380410Z INFO 06-12 16:19:10 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 16.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 33.6%, CPU KV cache usage: 0.0%.
2025-06-12T16:19:15.909297136Z INFO 06-12 16:19:15 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 12.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 33.6%, CPU KV cache usage: 0.0%.
2025-06-12T16:19:20.919173835Z INFO 06-12 16:19:20 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 13.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 33.6%, CPU KV cache usage: 0.0%.
2025-06-12T16:19:25.976467545Z INFO 06-12 16:19:25 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 16.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 33.7%, CPU KV cache usage: 0.0%.
2025-06-12T16:19:30.997634236Z INFO 06-12 16:19:30 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 33.7%, CPU KV cache usage: 0.0%.
2025-06-12T16:19:36.003586020Z INFO 06-12 16:19:36 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 33.7%, CPU KV cache usage: 0.0%.
2025-06-12T16:19:41.014928633Z INFO 06-12 16:19:41 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 33.8%, CPU KV cache usage: 0.0%.
2025-06-12T16:19:46.042720128Z INFO 06-12 16:19:46 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 33.8%, CPU KV cache usage: 0.0%.
2025-06-12T16:19:51.084954496Z INFO 06-12 16:19:51 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 33.9%, CPU KV cache usage: 0.0%.
2025-06-12T16:19:56.107539002Z INFO 06-12 16:19:56 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 33.9%, CPU KV cache usage: 0.0%.
2025-06-12T16:20:01.119698960Z INFO 06-12 16:20:01 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 15.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 34.0%, CPU KV cache usage: 0.0%.
2025-06-12T16:20:06.188182965Z INFO 06-12 16:20:06 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 12.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 34.0%, CPU KV cache usage: 0.0%.
2025-06-12T16:20:11.235275226Z INFO 06-12 16:20:11 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 12.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 34.0%, CPU KV cache usage: 0.0%.
2025-06-12T16:20:16.271686060Z INFO 06-12 16:20:16 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 16.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 34.1%, CPU KV cache usage: 0.0%.
2025-06-12T16:20:21.301499236Z INFO 06-12 16:20:21 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 34.1%, CPU KV cache usage: 0.0%.
2025-06-12T16:20:26.333668635Z INFO 06-12 16:20:26 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 34.2%, CPU KV cache usage: 0.0%.
2025-06-12T16:20:31.340315562Z INFO 06-12 16:20:31 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 34.2%, CPU KV cache usage: 0.0%.
2025-06-12T16:20:36.373982912Z INFO 06-12 16:20:36 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 34.2%, CPU KV cache usage: 0.0%.
2025-06-12T16:20:41.387685540Z INFO 06-12 16:20:41 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 34.3%, CPU KV cache usage: 0.0%.
2025-06-12T16:20:46.407225819Z INFO 06-12 16:20:46 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 34.3%, CPU KV cache usage: 0.0%.
2025-06-12T16:20:51.418639573Z INFO 06-12 16:20:51 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 13.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 34.4%, CPU KV cache usage: 0.0%.
2025-06-12T16:20:56.464813900Z INFO 06-12 16:20:56 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 16.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 34.4%, CPU KV cache usage: 0.0%.
2025-06-12T16:21:01.497250672Z INFO 06-12 16:21:01 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 17.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 34.4%, CPU KV cache usage: 0.0%.
2025-06-12T16:21:06.538684257Z INFO 06-12 16:21:06 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 16.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 34.5%, CPU KV cache usage: 0.0%.
2025-06-12T16:21:11.570669894Z INFO 06-12 16:21:11 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 18.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 34.5%, CPU KV cache usage: 0.0%.
2025-06-12T16:21:16.613438344Z INFO 06-12 16:21:16 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 17.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 34.6%, CPU KV cache usage: 0.0%.
2025-06-12T16:21:21.629945381Z INFO 06-12 16:21:21 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 34.6%, CPU KV cache usage: 0.0%.
2025-06-12T16:21:26.677110956Z INFO 06-12 16:21:26 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 18.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 34.7%, CPU KV cache usage: 0.0%.
2025-06-12T16:21:31.698322432Z INFO 06-12 16:21:31 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 18.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 34.7%, CPU KV cache usage: 0.0%.
2025-06-12T16:21:36.719883935Z INFO 06-12 16:21:36 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 34.8%, CPU KV cache usage: 0.0%.
2025-06-12T16:21:41.765526154Z INFO 06-12 16:21:41 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 34.8%, CPU KV cache usage: 0.0%.
2025-06-12T16:21:46.826386278Z INFO 06-12 16:21:46 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 34.8%, CPU KV cache usage: 0.0%.
2025-06-12T16:21:51.878082262Z INFO 06-12 16:21:51 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 14.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 34.9%, CPU KV cache usage: 0.0%.
2025-06-12T16:21:56.912073011Z INFO 06-12 16:21:56 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 16.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 34.9%, CPU KV cache usage: 0.0%.
2025-06-12T16:22:01.940621933Z INFO 06-12 16:22:01 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 17.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 35.0%, CPU KV cache usage: 0.0%.
2025-06-12T16:22:06.967241296Z INFO 06-12 16:22:06 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 15.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 35.0%, CPU KV cache usage: 0.0%.
2025-06-12T16:22:12.005919964Z INFO 06-12 16:22:12 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 13.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 35.0%, CPU KV cache usage: 0.0%.
2025-06-12T16:22:17.026890043Z INFO 06-12 16:22:17 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 35.1%, CPU KV cache usage: 0.0%.
2025-06-12T16:22:22.034266820Z INFO 06-12 16:22:22 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 35.1%, CPU KV cache usage: 0.0%.
2025-06-12T16:22:27.042819822Z INFO 06-12 16:22:27 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 35.2%, CPU KV cache usage: 0.0%.
2025-06-12T16:22:32.093214562Z INFO 06-12 16:22:32 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 35.2%, CPU KV cache usage: 0.0%.
2025-06-12T16:22:37.102756658Z INFO 06-12 16:22:37 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 35.3%, CPU KV cache usage: 0.0%.
2025-06-12T16:22:42.106727672Z INFO 06-12 16:22:42 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 35.3%, CPU KV cache usage: 0.0%.
2025-06-12T16:22:47.181149703Z INFO 06-12 16:22:47 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 17.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 35.3%, CPU KV cache usage: 0.0%.
2025-06-12T16:22:52.222856647Z INFO 06-12 16:22:52 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 15.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 35.4%, CPU KV cache usage: 0.0%.
2025-06-12T16:22:57.253111586Z INFO 06-12 16:22:57 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 17.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 35.4%, CPU KV cache usage: 0.0%.
2025-06-12T16:23:02.273451136Z INFO 06-12 16:23:02 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 17.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 35.5%, CPU KV cache usage: 0.0%.
2025-06-12T16:23:07.343257577Z INFO 06-12 16:23:07 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 13.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 35.5%, CPU KV cache usage: 0.0%.
2025-06-12T16:23:12.376045705Z INFO 06-12 16:23:12 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 16.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 35.5%, CPU KV cache usage: 0.0%.
2025-06-12T16:23:17.401191455Z INFO 06-12 16:23:17 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 18.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 35.6%, CPU KV cache usage: 0.0%.
2025-06-12T16:23:22.414493533Z INFO 06-12 16:23:22 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 35.6%, CPU KV cache usage: 0.0%.
2025-06-12T16:23:27.473058657Z INFO 06-12 16:23:27 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 18.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 35.7%, CPU KV cache usage: 0.0%.
2025-06-12T16:23:32.501941788Z INFO 06-12 16:23:32 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 35.7%, CPU KV cache usage: 0.0%.
2025-06-12T16:23:37.526728336Z INFO 06-12 16:23:37 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 35.8%, CPU KV cache usage: 0.0%.
2025-06-12T16:23:42.536656006Z INFO 06-12 16:23:42 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 35.8%, CPU KV cache usage: 0.0%.
2025-06-12T16:23:47.579013797Z INFO 06-12 16:23:47 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 12.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 35.8%, CPU KV cache usage: 0.0%.
2025-06-12T16:23:52.616727694Z INFO 06-12 16:23:52 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 13.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 35.9%, CPU KV cache usage: 0.0%.
2025-06-12T16:23:57.652854665Z INFO 06-12 16:23:57 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 13.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 35.9%, CPU KV cache usage: 0.0%.
2025-06-12T16:24:02.663110525Z INFO 06-12 16:24:02 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 16.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 35.9%, CPU KV cache usage: 0.0%.
2025-06-12T16:24:07.668891127Z INFO 06-12 16:24:07 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 17.6 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.0%, CPU KV cache usage: 0.0%.
2025-06-12T16:24:12.698749629Z INFO 06-12 16:24:12 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 18.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.0%, CPU KV cache usage: 0.0%.
2025-06-12T16:24:17.724375857Z INFO 06-12 16:24:17 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 18.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.1%, CPU KV cache usage: 0.0%.
2025-06-12T16:24:22.745753098Z INFO 06-12 16:24:22 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 18.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.1%, CPU KV cache usage: 0.0%.
2025-06-12T16:24:27.757728775Z INFO 06-12 16:24:27 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 18.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.2%, CPU KV cache usage: 0.0%.
2025-06-12T16:24:32.803355894Z INFO 06-12 16:24:32 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.2%, CPU KV cache usage: 0.0%.
2025-06-12T16:24:37.855291787Z INFO 06-12 16:24:37 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.3%, CPU KV cache usage: 0.0%.
2025-06-12T16:24:42.883463951Z INFO 06-12 16:24:42 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.3%, CPU KV cache usage: 0.0%.
2025-06-12T16:24:47.914075230Z INFO 06-12 16:24:47 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.3%, CPU KV cache usage: 0.0%.
2025-06-12T16:24:52.942054979Z INFO 06-12 16:24:52 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.4%, CPU KV cache usage: 0.0%.
2025-06-12T16:24:57.950764164Z INFO 06-12 16:24:57 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.0 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.4%, CPU KV cache usage: 0.0%.
2025-06-12T16:25:02.996702072Z INFO 06-12 16:25:02 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 18.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.5%, CPU KV cache usage: 0.0%.
2025-06-12T16:25:08.034402401Z INFO 06-12 16:25:08 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.5%, CPU KV cache usage: 0.0%.
2025-06-12T16:25:13.073392073Z INFO 06-12 16:25:13 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 18.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.6%, CPU KV cache usage: 0.0%.
2025-06-12T16:25:18.078705850Z INFO 06-12 16:25:18 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.6%, CPU KV cache usage: 0.0%.
2025-06-12T16:25:23.123421463Z INFO 06-12 16:25:23 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.2 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.7%, CPU KV cache usage: 0.0%.
2025-06-12T16:25:28.137997337Z INFO 06-12 16:25:28 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.7%, CPU KV cache usage: 0.0%.
2025-06-12T16:25:33.151229966Z INFO 06-12 16:25:33 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.7%, CPU KV cache usage: 0.0%.
2025-06-12T16:25:38.172305523Z INFO 06-12 16:25:38 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.8%, CPU KV cache usage: 0.0%.
2025-06-12T16:25:43.201828031Z INFO 06-12 16:25:43 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.3 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.8%, CPU KV cache usage: 0.0%.
2025-06-12T16:25:48.240445968Z INFO 06-12 16:25:48 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.4 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.9%, CPU KV cache usage: 0.0%.
2025-06-12T16:25:53.268843934Z INFO 06-12 16:25:53 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.5 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 36.9%, CPU KV cache usage: 0.0%.
2025-06-12T16:25:58.305353349Z INFO 06-12 16:25:58 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 18.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.0%, CPU KV cache usage: 0.0%.
2025-06-12T16:26:03.332318531Z INFO 06-12 16:26:03 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 18.7 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.0%, CPU KV cache usage: 0.0%.
2025-06-12T16:26:08.376807588Z INFO 06-12 16:26:08 [metrics.py:486] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 18.8 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 37.1%, CPU KV cache usage: 0.0%.