2025-05-14T03:07:41.819621278Z INFO 05-14 03:07:41 [__init__.py:239] Automatically detected platform cuda.
2025-05-14T03:07:44.493530219Z engine.py           :27   2025-05-14 03:07:44,493 Engine args: AsyncEngineArgs(model='carlfm01/qwen3-4b', served_model_name=None, tokenizer='carlfm01/qwen3-4b', hf_config_path=None, task='auto', skip_tokenizer_init=False, tokenizer_mode='auto', trust_remote_code=False, allowed_local_media_path='', download_dir=None, load_format='auto', config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', kv_cache_dtype='auto', seed=0, max_model_len=None, distributed_executor_backend=None, pipeline_parallel_size=1, tensor_parallel_size=1, data_parallel_size=1, enable_expert_parallel=False, max_parallel_loading_workers=None, block_size=16, enable_prefix_caching=False, prefix_caching_hash_algo='builtin', disable_sliding_window=False, disable_cascade_attn=False, use_v2_block_manager='true', swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.95, max_num_batched_tokens=None, max_num_partial_prefills=1, max_long_partial_prefills=1, long_prefill_token_threshold=0, max_num_seqs=256, max_logprobs=20, disable_log_stats=False, revision=None, code_revision=None, rope_scaling=None, rope_theta=None, hf_token=None, hf_overrides=None, tokenizer_revision=None, quantization=None, enforce_eager=False, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config={}, limit_mm_per_prompt={}, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=False, enable_lora_bias=False, max_loras=1, max_lora_rank=16, fully_sharded_loras=False, max_cpu_loras=None, lora_dtype='auto', lora_extra_vocab_size=256, long_lora_scaling_factors=None, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, ray_workers_use_nsight=False, num_gpu_blocks_override=None, num_lookahead_slots=0, model_loader_extra_config={}, ignore_patterns=None, preemption_mode=None, scheduler_delay_factor=0.0, enable_chunked_prefill=None, disable_chunked_mm_input=False, guided_decoding_backend='outlines', logits_processor_pattern=None, speculative_config=None, qlora_adapter_name_or_path=None, show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, scheduling_policy='fcfs', scheduler_cls='vllm.core.scheduler.Scheduler', override_neuron_config=None, override_pooler_config=None, compilation_config=None, worker_cls='auto', worker_extension_cls='', kv_transfer_config=None, generation_config='auto', override_generation_config=None, enable_sleep_mode=False, model_impl='auto', calculate_kv_scales=False, additional_config=None, enable_reasoning=None, reasoning_parser=None, use_tqdm_on_load=True, disable_log_requests=False)
2025-05-14T03:07:55.151751929Z tokenizer_name_or_path: carlfm01/qwen3-4b, tokenizer_revision: None, trust_remote_code: False
2025-05-14T03:07:55.151770977Z INFO 05-14 03:07:55 [config.py:717] This model supports multiple tasks: {'embed', 'classify', 'generate', 'score', 'reward'}. Defaulting to 'generate'.
2025-05-14T03:07:55.151906088Z WARNING 05-14 03:07:55 [arg_utils.py:1658] --guided-decoding-backend=outlines is not supported by the V1 Engine. Falling back to V0.
2025-05-14T03:07:55.151982998Z WARNING 05-14 03:07:55 [arg_utils.py:1525] Chunked prefill is enabled by default for models with max_model_len > 32K. Chunked prefill might not work with some features or models. If you encounter any issues, please disable by launching with --enable-chunked-prefill=False.
2025-05-14T03:07:55.152655640Z INFO 05-14 03:07:55 [config.py:2003] Chunked prefill is enabled with max_num_batched_tokens=2048.
2025-05-14T03:07:55.154452064Z INFO 05-14 03:07:55 [llm_engine.py:240] Initializing a V0 LLM engine (v0.8.5) with config: model='carlfm01/qwen3-4b', speculative_config=None, tokenizer='carlfm01/qwen3-4b', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=40960, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines', reasoning_backend=None), observability_config=ObservabilityConfig(show_hidden_metrics=False, otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=carlfm01/qwen3-4b, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=False, chunked_prefill_enabled=True, use_async_output_proc=True, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"splitting_ops":[],"compile_sizes":[],"cudagraph_capture_sizes":[256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":256}, use_cached_outputs=False,
2025-05-14T03:07:56.057245692Z INFO 05-14 03:07:56 [cuda.py:292] Using Flash Attention backend.
2025-05-14T03:07:56.434403790Z INFO 05-14 03:07:56 [parallel_state.py:1004] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0
2025-05-14T03:07:56.436007519Z INFO 05-14 03:07:56 [model_runner.py:1108] Starting to load model carlfm01/qwen3-4b...
2025-05-14T03:07:56.864472489Z INFO 05-14 03:07:56 [weight_utils.py:265] Using model weights format ['*.safetensors']
2025-05-14T03:08:04.527559357Z INFO 05-14 03:08:04 [weight_utils.py:281] Time spent downloading weights for carlfm01/qwen3-4b: 7.662188 seconds
2025-05-14T03:08:04.792867280Z 
Loading safetensors checkpoint shards:   0% Completed | 0/2 [00:00<?, ?it/s]
2025-05-14T03:08:05.261927244Z 
Loading safetensors checkpoint shards:  50% Completed | 1/2 [00:00<00:00,  2.13it/s]
2025-05-14T03:08:06.015907691Z 
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:01<00:00,  1.57it/s]
2025-05-14T03:08:06.015987714Z 
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:01<00:00,  1.64it/s]
2025-05-14T03:08:06.142751223Z INFO 05-14 03:08:06 [loader.py:458] Loading weights took 1.35 seconds
2025-05-14T03:08:06.352211520Z INFO 05-14 03:08:06 [model_runner.py:1140] Model loading took 7.5552 GiB and 9.711935 seconds
2025-05-14T03:08:07.166444223Z INFO 05-14 03:08:07 [worker.py:287] Memory profiling takes 0.66 seconds
2025-05-14T03:08:07.166470520Z INFO 05-14 03:08:07 [worker.py:287] the current vLLM instance can use total_gpu_memory (23.67GiB) x gpu_memory_utilization (0.95) = 22.49GiB
2025-05-14T03:08:07.166473889Z INFO 05-14 03:08:07 [worker.py:287] model weights take 7.56GiB; non_torch_memory takes 0.05GiB; PyTorch activation peak memory takes 1.40GiB; the rest of the memory reserved for KV Cache is 13.49GiB.
2025-05-14T03:08:07.330921101Z INFO 05-14 03:08:07 [executor_base.py:112] # cuda blocks: 6139, # CPU blocks: 1820
2025-05-14T03:08:07.330969751Z INFO 05-14 03:08:07 [executor_base.py:117] Maximum concurrency for 40960 tokens per request: 2.40x
2025-05-14T03:08:08.925943936Z INFO 05-14 03:08:08 [model_runner.py:1450] Capturing cudagraphs for decoding. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI. If out-of-memory error occurs during cudagraph capture, consider decreasing `gpu_memory_utilization` or switching to eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
2025-05-14T03:08:23.067249228Z 
Capturing CUDA graph shapes:   0%|          | 0/35 [00:00<?, ?it/s]
Capturing CUDA graph shapes:   3%|▎         | 1/35 [00:00<00:14,  2.37it/s]
Capturing CUDA graph shapes:   6%|▌         | 2/35 [00:00<00:13,  2.43it/s]
Capturing CUDA graph shapes:   9%|▊         | 3/35 [00:01<00:13,  2.44it/s]
Capturing CUDA graph shapes:  11%|█▏        | 4/35 [00:01<00:12,  2.45it/s]
Capturing CUDA graph shapes:  14%|█▍        | 5/35 [00:02<00:12,  2.46it/s]
Capturing CUDA graph shapes:  17%|█▋        | 6/35 [00:02<00:11,  2.47it/s]
Capturing CUDA graph shapes:  20%|██        | 7/35 [00:02<00:11,  2.46it/s]
Capturing CUDA graph shapes:  23%|██▎       | 8/35 [00:03<00:10,  2.46it/s]
Capturing CUDA graph shapes:  26%|██▌       | 9/35 [00:03<00:10,  2.47it/s]
Capturing CUDA graph shapes:  29%|██▊       | 10/35 [00:04<00:10,  2.49it/s]
Capturing CUDA graph shapes:  31%|███▏      | 11/35 [00:04<00:09,  2.52it/s]
Capturing CUDA graph shapes:  34%|███▍      | 12/35 [00:04<00:09,  2.52it/s]
Capturing CUDA graph shapes:  37%|███▋      | 13/35 [00:05<00:08,  2.50it/s]
Capturing CUDA graph shapes:  40%|████      | 14/35 [00:05<00:08,  2.46it/s]
Capturing CUDA graph shapes:  43%|████▎     | 15/35 [00:06<00:08,  2.47it/s]
Capturing CUDA graph shapes:  46%|████▌     | 16/35 [00:06<00:07,  2.39it/s]
Capturing CUDA graph shapes:  49%|████▊     | 17/35 [00:06<00:07,  2.35it/s]
Capturing CUDA graph shapes:  51%|█████▏    | 18/35 [00:07<00:07,  2.41it/s]
Capturing CUDA graph shapes:  54%|█████▍    | 19/35 [00:07<00:06,  2.47it/s]
Capturing CUDA graph shapes:  57%|█████▋    | 20/35 [00:08<00:06,  2.47it/s]
Capturing CUDA graph shapes:  60%|██████    | 21/35 [00:08<00:05,  2.49it/s]
Capturing CUDA graph shapes:  63%|██████▎   | 22/35 [00:08<00:05,  2.38it/s]
Capturing CUDA graph shapes:  66%|██████▌   | 23/35 [00:09<00:04,  2.41it/s]
Capturing CUDA graph shapes:  69%|██████▊   | 24/35 [00:09<00:04,  2.43it/s]
Capturing CUDA graph shapes:  71%|███████▏  | 25/35 [00:10<00:04,  2.46it/s]
Capturing CUDA graph shapes:  74%|███████▍  | 26/35 [00:10<00:03,  2.50it/s]
Capturing CUDA graph shapes:  77%|███████▋  | 27/35 [00:11<00:03,  2.46it/s]
Capturing CUDA graph shapes:  80%|████████  | 28/35 [00:11<00:02,  2.50it/s]
Capturing CUDA graph shapes:  83%|████████▎ | 29/35 [00:11<00:02,  2.53it/s]
Capturing CUDA graph shapes:  86%|████████▌ | 30/35 [00:12<00:01,  2.52it/s]
Capturing CUDA graph shapes:  89%|████████▊ | 31/35 [00:12<00:01,  2.55it/s]
Capturing CUDA graph shapes:  91%|█████████▏| 32/35 [00:12<00:01,  2.53it/s]
Capturing CUDA graph shapes:  94%|█████████▍| 33/35 [00:13<00:00,  2.54it/s]
Capturing CUDA graph shapes:  97%|█████████▋| 34/35 [00:13<00:00,  2.55it/s]
Capturing CUDA graph shapes: 100%|██████████| 35/35 [00:14<00:00,  2.53it/s]
Capturing CUDA graph shapes: 100%|██████████| 35/35 [00:14<00:00,  2.48it/s]
2025-05-14T03:08:23.067453507Z INFO 05-14 03:08:23 [model_runner.py:1592] Graph capturing finished in 14 secs, took 0.43 GiB
2025-05-14T03:08:23.067893911Z INFO 05-14 03:08:23 [llm_engine.py:437] init engine (profile, create kv cache, warmup model) took 16.72 seconds
2025-05-14T03:08:23.133689515Z engine.py           :113  2025-05-14 03:08:23,133 Initialized vLLM engine in 35.56s
2025-05-14T03:08:23.133985867Z engine.py           :27   2025-05-14 03:08:23,133 Engine args: AsyncEngineArgs(model='carlfm01/qwen3-4b', served_model_name=None, tokenizer='carlfm01/qwen3-4b', hf_config_path=None, task='auto', skip_tokenizer_init=False, tokenizer_mode='auto', trust_remote_code=False, allowed_local_media_path='', download_dir=None, load_format='auto', config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', kv_cache_dtype='auto', seed=0, max_model_len=None, distributed_executor_backend=None, pipeline_parallel_size=1, tensor_parallel_size=1, data_parallel_size=1, enable_expert_parallel=False, max_parallel_loading_workers=None, block_size=16, enable_prefix_caching=False, prefix_caching_hash_algo='builtin', disable_sliding_window=False, disable_cascade_attn=False, use_v2_block_manager='true', swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.95, max_num_batched_tokens=None, max_num_partial_prefills=1, max_long_partial_prefills=1, long_prefill_token_threshold=0, max_num_seqs=256, max_logprobs=20, disable_log_stats=False, revision=None, code_revision=None, rope_scaling=None, rope_theta=None, hf_token=None, hf_overrides=None, tokenizer_revision=None, quantization=None, enforce_eager=False, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config={}, limit_mm_per_prompt={}, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=False, enable_lora_bias=False, max_loras=1, max_lora_rank=16, fully_sharded_loras=False, max_cpu_loras=None, lora_dtype='auto', lora_extra_vocab_size=256, long_lora_scaling_factors=None, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, ray_workers_use_nsight=False, num_gpu_blocks_override=None, num_lookahead_slots=0, model_loader_extra_config={}, ignore_patterns=None, preemption_mode=None, scheduler_delay_factor=0.0, enable_chunked_prefill=None, disable_chunked_mm_input=False, guided_decoding_backend='outlines', logits_processor_pattern=None, speculative_config=None, qlora_adapter_name_or_path=None, show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, scheduling_policy='fcfs', scheduler_cls='vllm.core.scheduler.Scheduler', override_neuron_config=None, override_pooler_config=None, compilation_config=None, worker_cls='auto', worker_extension_cls='', kv_transfer_config=None, generation_config='auto', override_generation_config=None, enable_sleep_mode=False, model_impl='auto', calculate_kv_scales=False, additional_config=None, enable_reasoning=None, reasoning_parser=None, use_tqdm_on_load=True, disable_log_requests=False)
2025-05-14T03:08:23.720906225Z tokenizer_name_or_path: carlfm01/qwen3-4b, tokenizer_revision: None, trust_remote_code: False
2025-05-14T03:08:23.720950711Z WARNING 05-14 03:08:23 [config.py:1239] Default sampling parameters have been overridden by the model's Hugging Face generation config recommended from the model creator. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
2025-05-14T03:08:23.720957628Z INFO 05-14 03:08:23 [serving_chat.py:118] Using default chat sampling params from model: {'repetition_penalty': 1.1, 'temperature': 0.7, 'top_k': 20, 'top_p': 0.8, 'min_p': 0.0}
2025-05-14T03:08:23.848544981Z INFO 05-14 03:08:23 [serving_completion.py:61] Using default completion sampling params from model: {'repetition_penalty': 1.1, 'temperature': 0.7, 'top_k': 20, 'top_p': 0.8, 'min_p': 0.0}
2025-05-14T03:08:24.281906859Z --- Starting Serverless Worker |  Version 1.7.9 ---
2025-05-14T03:08:24.281936141Z {"requestId": null, "message": "Jobs in queue: 1", "level": "INFO"}
2025-05-14T03:08:24.281938956Z {"requestId": null, "message": "Jobs in progress: 1", "level": "INFO"}
2025-05-14T03:08:24.282236494Z INFO 05-14 03:08:24 [async_llm_engine.py:211] Added request cd1064a8f9234a53b6e7dfa6051a5285.
2025-05-14T03:08:25.938522518Z INFO 05-14 03:08:25 [async_llm_engine.py:179] Finished request cd1064a8f9234a53b6e7dfa6051a5285.
2025-05-14T03:08:26.100491405Z {"requestId": "299b3c55-bdef-48eb-85f5-b202ef03eb50-u1", "message": "Finished running generator.", "level": "INFO"}
2025-05-14T03:08:26.343805275Z {"requestId": "299b3c55-bdef-48eb-85f5-b202ef03eb50-u1", "message": "Finished.", "level": "INFO"}