Please See Attached Txt File.
Couldn't fit all info in prompt due to too many Char
#Lama 4 Scout Quantized Help
digital crown
digital crown
here is my question that is not in a attachment:
greetings everyone, If anyone could help me with this error I am getting, I would greatly appreciate it!
I am following the tutorial for using the quantized version of llama 4 from the Unsloth Documentationn
Found Here: https://docs.unsloth.ai/basics/tutorial-how-to-run-and-fine-tune-llama-4
I am attempting to run Llama-4-scout Version - Q2_K_XL Size - 42.2GB
Here is my following hardware that is relevant:
CPU = Threadripper 7970x (64 threads)
GPU/s = 2 * 3090TI Founders Edition
Memory = 128GB DDR5 ECC @ 7000
Here is my prompt (Note: this is very similar to the tutorial other than I have changed my CPU's threads to match my cpu, and I am using the Q2_K_XL version VS the IQ2_XXS used in the Tutorial.)
./llama.cpp/llama-cli \
--model unsloth/Llama-4-Scout-17B-16E-Instruct-GGUF/Llama-4-Scout-17B-16E-Instruct-UD-Q2_K_XL.gguf \ --threads 64 \ --ctx-size 16384 \ --n-gpu-layers 99 \ --seed 3407 \ --prio 3 \ --temp 0.6 \ --min-p 0.01 \ --top-p 0.9 \ -no-cnv \ --prompt "<|header_start|>user<|header_end|>\n\nCreate a Flappy Bird game in Python. You must include these things:\n1. You must use pygame.\n2. The background color should be randomly chosen and is a light shade. Start with a light blue color.\n3. Pressing SPACE multiple times will accelerate the bird.\n4. The bird's shape should be randomly chosen as a square, circle or triangle. The color should be randomly chosen as a dark color.\n5. Place on the bottom some land colored as dark brown or yellow chosen randomly.\n6. Make a score shown on the top right side. Increment if you pass pipes and don't hit them.\n7. Make randomly spaced pipes with enough space. Color them randomly as dark green or light brown or a dark gray shade.\n8. When you lose, show the best score. Make the text inside the screen. Pressing q or Esc will quit the game. Restarting is pressing SPACE again.\nThe final game should be inside a markdown section in Python. Check your code for errors and fix them before the final markdown section.<|eot|><|header_start|>assistant<|header_end|>\n\n"
Here are My Terminal Results:
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 2 CUDA devices:
Device 0: NVIDIA GeForce RTX 3090 Ti, compute capability 8.6, VMM: yes
Device 1: NVIDIA GeForce RTX 3090 Ti, compute capability 8.6, VMM: yes
build: 5149 (2f74c354) with cc (Ubuntu 11.4.0-2ubuntu1~20.04) 11.4.0 for x86_64-linux-gnu
main: llama backend init
main: load the model and apply lora adapter, if any
llama_model_load_from_file_impl: using device CUDA0 (NVIDIA GeForce RTX 3090 Ti) - 21757 MiB free
llama_model_load_from_file_impl: using device CUDA1 (NVIDIA GeForce RTX 3090 Ti) - 21757 MiB free
llama_model_loader: loaded meta data with 48 key-value pairs and 628 tensors from unsloth/Llama-4-Scout-17B-16E-Instruct-GGUF/Llama-4-Scout-17B-16E-Instruct-UD-Q2_K_XL.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = llama4
llama_model_loader: - kv 1: general.type str = model
llama_model_loader: - kv 2: general.name str = Llama-4-Scout-17B-16E-Instruct
llama_model_loader: - kv 3: general.finetune str = 16E-Instruct
llama_model_loader: - kv 4: general.basename str = Llama-4-Scout-17B-16E-Instruct
llama_model_loader: - kv 5: general.quantized_by str = Unsloth
llama_model_loader: - kv 6: general.size_label str = 17B
llama_model_loader: - kv 7: general.license str = other
llama_model_loader: - kv 8: general.license.name str = llama4
llama_model_loader: - kv 9: general.repo_url str = https://huggingface.co/unsloth
llama_model_loader: - kv 10: general.base_model.count u32 = 1
llama_model_loader: - kv 11: general.base_model.0.name str = Llama 4 Scout 17B 16E
llama_model_loader: - kv 12: general.base_model.0.organization str = Meta Llama
llama_model_loader: - kv 13: general.base_model.0.repo_url str = https://huggingface.co/meta-llama/Lla...
llama_model_loader: - kv 14: general.tags arr[str,5] = ["facebook", "meta", "pytorch", "llam...
llama_model_loader: - kv 15: general.languages arr[str,12] = ["ar", "de", "en", "es", "fr", "hi", ...
llama_model_loader: - kv 16: llama4.block_count u32 = 48
llama_model_loader: - kv 17: llama4.context_length u32 = 10485760
llama_model_loader: - kv 18: llama4.embedding_length u32 = 5120
llama_model_loader: - kv 19: llama4.feed_forward_length u32 = 16384
llama_model_loader: - kv 20: llama4.attention.head_count u32 = 40
llama_model_loader: - kv 21: llama4.attention.head_count_kv u32 = 8
llama_model_loader: - kv 22: llama4.rope.freq_base f32 = 500000.000000
llama_model_loader: - kv 23: llama4.attention.layer_norm_rms_epsilon f32 = 0.000010
llama_model_loader: - kv 24: llama4.expert_count u32 = 16
llama_model_loader: - kv 25: llama4.expert_used_count u32 = 1
llama_model_loader: - kv 26: llama4.attention.key_length u32 = 128
llama_model_loader: - kv 27: llama4.attention.value_length u32 = 128
llama_model_loader: - kv 28: llama4.vocab_size u32 = 202048
llama_model_loader: - kv 29: llama4.rope.dimension_count u32 = 128
llama_model_loader: - kv 30: llama4.interleave_moe_layer_step u32 = 1
llama_model_loader: - kv 31: llama4.expert_feed_forward_length u32 = 8192
llama_model_loader: - kv 32: tokenizer.ggml.model str = gpt2
llama_model_loader: - kv 33: tokenizer.ggml.pre str = llama4
llama_model_loader: - kv 34: tokenizer.ggml.tokens arr[str,202048] = ["À", "Á", "õ", "ö", "÷", "ø", ...
llama_model_loader: - kv 35: tokenizer.ggml.token_type arr[i32,202048] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv 36: tokenizer.ggml.merges arr[str,439802] = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "...
llama_model_loader: - kv 37: tokenizer.ggml.bos_token_id u32 = 200000
llama_model_loader: - kv 38: tokenizer.ggml.eos_token_id u32 = 200008
llama_model_loader: - kv 39: tokenizer.ggml.padding_token_id u32 = 200018
llama_model_loader: - kv 40: tokenizer.chat_template str = {{- bos_token }}\n{%- if custom_tools ...
llama_model_loader: - kv 41: tokenizer.ggml.add_bos_token bool = true
llama_model_loader: - kv 42: general.quantization_version u32 = 2
llama_model_loader: - kv 43: general.file_type u32 = 10
llama_model_loader: - kv 44: quantize.imatrix.file str = Llama-4-Scout-17B-16E-Instruct-GGUF/i...
llama_model_loader: - kv 45: quantize.imatrix.dataset str = unsloth_calibration_Llama-4-Scout-17B...
llama_model_loader: - kv 46: quantize.imatrix.entries_count i32 = 528
llama_model_loader: - kv 47: quantize.imatrix.chunks_count i32 = 524
llama_model_loader: - type f32: 146 tensors
llama_model_loader: - type q2_K: 96 tensors
llama_model_loader: - type q3_K: 42 tensors
llama_model_loader: - type q4_K: 295 tensors
llama_model_loader: - type q5_K: 1 tensors
llama_model_loader: - type q6_K: 48 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type = Q2_K - Medium
print_info: file size = 39.53 GiB (3.15 BPW)
load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
load: special tokens cache size = 1135
load: token to piece cache size = 1.3873 MB
print_info: arch = llama4
print_info: vocab_only = 0
print_info: n_ctx_train = 10485760
print_info: n_embd = 5120
print_info: n_layer = 48
print_info: n_head = 40
print_info: n_head_kv = 8
print_info: n_rot = 128
print_info: n_swa = 1
print_info: n_swa_pattern = 4
print_info: n_embd_head_k = 128
print_info: n_embd_head_v = 128
print_info: n_gqa = 5
print_info: n_embd_k_gqa = 1024
print_info: n_embd_v_gqa = 1024
print_info: f_norm_eps = 0.0e+00
print_info: f_norm_rms_eps = 1.0e-05
print_info: f_clamp_kqv = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale = 0.0e+00
print_info: f_attn_scale = 0.0e+00
print_info: n_ff = 16384
print_info: n_expert = 16
print_info: n_expert_used = 1
print_info: causal attn = 1
print_info: pooling type = 0
print_info: rope type = 0
print_info: rope scaling = linear
print_info: freq_base_train = 500000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn = 10485760
print_info: rope_finetuned = unknown
print_info: ssm_d_conv = 0
print_info: ssm_d_inner = 0
print_info: ssm_d_state = 0
print_info: ssm_dt_rank = 0
print_info: ssm_dt_b_c_rms = 0
print_info: model type = 17Bx16E (Scout)
print_info: model params = 107.77 B
print_info: general.name = Llama-4-Scout-17B-16E-Instruct
print_info: vocab type = BPE
print_info: n_vocab = 202048
print_info: n_merges = 439802
print_info: BOS token = 200000 '<|begin_of_text|>'
print_info: EOS token = 200008 '<|eot|>'
print_info: PAD token = 200018 '<|finetune_right_pad|>'
print_info: LF token = 198 'Ċ'
print_info: FIM PRE token = 200002 '<|fim_prefix|>'
print_info: FIM SUF token = 200004 '<|fim_suffix|>'
print_info: FIM MID token = 200003 '<|fim_middle|>'
print_info: EOG token = 200008 '<|eot|>'
print_info: max token length = 192
load_tensors: loading model tensors, this can take a while... (mmap = true)
load_tensors: offloading 48 repeating layers to GPU
load_tensors: offloading output layer to GPU
load_tensors: offloaded 49/49 layers to GPU
load_tensors: CPU_Mapped model buffer size = 554.94 MiB
load_tensors: CUDA0 model buffer size = 20682.85 MiB
load_tensors: CUDA1 model buffer size = 19237.30 MiB
...............................................................
llama_context: constructing llama_context
llama_context: n_seq_max = 1
llama_context: n_ctx = 16384
llama_context: n_ctx_per_seq = 16384
llama_context: n_batch = 2048
llama_context: n_ubatch = 512
llama_context: causal_attn = 1
llama_context: flash_attn = 0
llama_context: freq_base = 500000.0
llama_context: freq_scale = 1
llama_context: n_ctx_per_seq (16384) < n_ctx_train (10485760) -- the full capacity of the model will not be utilized
llama_context: CUDA_Host output buffer size = 0.77 MiB
init: kv_size = 16384, offload = 1, type_k = 'f16', type_v = 'f16', n_layer = 48, can_shift = 1
ggml_backend_cuda_buffer_type_alloc_buffer: allocating 1600.00 MiB on device 0: cudaMalloc failed: out of memory
alloc_tensor_range: failed to allocate CUDA0 buffer of size 1677721600
init: failed to allocate buffer for kv cache
llama_init_from_model: failed to initialize the context: failed to initialize self-attention cache
common_init_from_params: failed to create context with model 'unsloth/Llama-4-Scout-17B-16E-Instruct-GGUF/Llama-4-Scout-17B-16E-Instruct-UD-Q2_K_XL.gguf'
main: error: unable to load model
My Confusion:
The Error indicates that I do not have enough memory to properly load this Model. However, This model is not even using all of the memory between my 2 GPU's.
Is there something else I need to add to my prompt to properly allocate all of the necessary memory?
Thanks!
valid bolt
its possible the ctx is too big, try ctx-size 4096