Using llama-cpp-python

[knc6]

Hi,

Thanks for creating this wonderful package!
The save_to_gguf currently fails because llama.ccp installation seems to be broken.
Could something like llama-cpp-python be used instead?

[danielhanchen]

Is this via Colab or Kaggle or local machines?

[knc6]

I am using a local machine. Actually, I was able to save the model using save_pretrained_gguf . It produced a .Q8_0.gguf file. Could you tell me how to load it for inference on CPU/GPU?

[erwe324]

@knc6 Can you please please elaborate on the steps you took to save the model after getting a broken llama.cpp error?
I am constantly facing this error on my local machine but never on Colab.

As for the gguf file you can use a lot of different software available. Simplest one would be to use llama.cpp directly. Most of he other CUI/GUI also use it in the backend.

[danielhanchen]

@erwe324 Oh no :( Still GGUF issues? :(

[erwe324]

@danielhanchen
Please Note that in my case this behaviour is from the last 2 months. However,
There is some mystery.
No matter what I do on my local machine I AM unable to run llama.cpp to create gguf.

However, the same code runs without any issues or modification on Google Collab. I am pre occupied but soon I will try to identify the root cause for this. Most probably this is due to some dependency for llama cpp

[knc6]

@erwe324 Same issue, the script works on colab but not on local machine. Perhaps a conda-package/environment, nix, docker container for unsloth would be useful.

[danielhanchen]

Yes a Docker env would be useful. How about if you do a manual one?

git clone --recursive https://github.com/ggerganov/llama.cpp
make clean -C llama.cpp
make all -j -C llama.cpp
pip install gguf protobuf

python llama.cpp/convert-hf-to-gguf.py FOLDER --outfile OUTPUT --outtype f16