u/AgreeableExternal238

Has anyone tried running local LLMs on AIKosh cloud computing notebooks?

I’m testing Qwen2.5-Coder-3B-Instruct, but the model loading consistently gets stuck around 56%.

I’m using the regular notebook environment (CPU, not GPU), so I’m wondering if this is a RAM limitation or compatibility issue.

Thinking about switching to:

GGUF

Q4_K_M quantization

llama.cpp backend

Has anyone successfully run 3B–7B models on AIKosh notebooks?

Would love recommendations for:

best quantization format

working backend

practical model size limits for the AIKosh notebook environment