[Mac]Meta-Llama-3-8Bをgguf変換して量子化してみました

meta-llama/Meta-Llama-3-8B · Hugging Face

上記のモデルを自分で量子化してみました。
参考にした記事

【llama.cpp】誰でもできるgguf変換

基本的な流れは上の記事の通りですが
convert.pyを実行すると
ずっと

ValueError: unknown format: models/Meta-Llama-3-8B/model-00001-of-00004.safetensors

というエラーが出ていて泣きそうになってました
safetensorsのファイルの中身を見てみると
3行くらいのテキストになっていてバイナリファイルになってないことに気づきました
ファイルサイズも小さい

git clone でゲットした時にファイルがデカすぎて省略されたんだと思います
直接safetensorsファイルを一個一個ダウンロードすることで入手できました
それをconvert.pyすることでggufへの変換は成功しました

次は量子化です
Macなんでちょっと実行のやり方が違います

./quantize ./models/Meta-Llama-3-8B.gguf ./models/Meta-Llama-3-8B-q4_0.gguf q4_0

これでいけました

この自作量子化モデルを使ってみます
【リクエスト】
./main -m ../LLMModels/Meta-Llama-3-8B-q4_0.gguf -n 200 -ngl 38 -t 10 -p “好きなアニメのキャラ教えてください” --ctx_size 512

【レスポンス】
<|begin_of_text|>“好きなアニメのキャラ教えてください”を英語で
I want to know about your favorite anime character. Please tell me about your favorite anime character.
I want to know about your favorite anime character. Please tell me about your favorite anime character.
I want to know about your favorite anime character. Please tell me about your favorite anime character.
Please tell me about your favorite anime character.
I want to know about your favorite anime character. Please tell me about your favorite anime character.<|end_of_text|> [end of text]

llama_print_timings: load time = 5891.14 ms
llama_print_timings: sample time = 4.42 ms / 90 runs ( 0.05 ms per token, 20343.58 tokens per second)
llama_print_timings: prompt eval time = 80.45 ms / 12 tokens ( 6.70 ms per token, 149.16 tokens per second)
llama_print_timings: eval time = 1474.29 ms / 89 runs ( 16.57 ms per token, 60.37 tokens per second)
llama_print_timings: total time = 1594.32 ms / 101 tokens
ggml_metal_free: deallocating

63.349892117 t/s

なんかめっちゃ早いです　過去最速出ました

英語で喋ってほしいらしいですね

[リクエスト]
./main -m ../LLMModels/Meta-Llama-3-8B-q4_0.gguf -n 200 -ngl 38 -t 10 -p "Who is your favorite anime character" --ctx_size 512

[レスポンス]
<|begin_of_text|>Who is your favorite anime character? Is it one that is just cool, or one that you can relate to? What makes them your favorite? Is it because they are brave, or because they are nice? Or maybe they have a secret that you like, like the fact that they are actually a demon or a catgirl? Whatever it is, you probably won't be surprised to know that most people have a favorite anime character.
So, why do people have favorite anime characters? There are many reasons, and I think I know why. Some people just like to be like their favorite character, and be like them. They like to have the same attitude, the same personality, the same interests. Some people like to be like the character because of the character's unique powers and abilities.
There are many reasons why you might like a certain character. Some people like to relate to that character, or maybe they just like their personality. Maybe you like a character because you are like them. Maybe you like a character because of

好きなアニメのキャラクターは答えてくれないみたいです・・・涙