Google Colab で Llama-2-70B-chat-GPTQ を試す
「Google Colab」で「Llama-2-70B-chat-GPTQ」を試したのでまとめました。
【最新版の情報は以下で紹介】
前回
1. AutoGPTQ
「AutoGPTQ」を使って「Llama 2」の最大サイズ「70B」の「Google Colab」での実行に挑戦してみます。
2. Colabでの学習
Google Colabでの学習手順は、次のとおりです。
(1) メニュー「編集→ノートブックの設定」で、「ハードウェアアクセラレータ」で「GPU」の「A100」を選択。
(2) パッケージのインストール。
# パッケージのインストール
%cd /content
!git clone https://github.com/PanQiWei/AutoGPTQ
%cd AutoGPTQ
!pip install .
!pip install git+https://github.com/huggingface/transformers
(2) トークナイザーとモデルの準備。
from transformers import AutoTokenizer, pipeline, logging
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
# トークナイザーとモデルの準備
model_name_or_path = "TheBloke/Llama-2-70B-chat-GPTQ"
model_basename = "gptq_model-4bit--1g"
tokenizer = AutoTokenizer.from_pretrained(
model_name_or_path,
use_fast=True
)
model = AutoGPTQForCausalLM.from_quantized(
model_name_or_path,
model_basename=model_basename,
inject_fused_attention=False, # 現時点では70Bに必要
use_safetensors=True,
trust_remote_code=True,
device="cuda:0",
use_triton=False,
quantize_config=None
)
(3) 推論の実行。
# プロンプトの準備
prompt='''[INST] <<SYS>>
You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature. If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.
<</SYS>>
What is Bocchi-chan's personality from BOCCHI THE ROCK?[/INST]'''
# 推論の実行
input_ids = tokenizer(prompt, return_tensors='pt').input_ids.cuda()
output = model.generate(inputs=input_ids, temperature=0.7, max_new_tokens=512)
print(tokenizer.decode(output[0]))
<s> [INST] <<SYS>>
You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature. If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.
<</SYS>>
What is Bocchi-chan's personality from BOCCHI THE ROCK?[/INST] Bocchi-chan, the main character of the anime and manga series "Bocchi the Rock," is a unique and interesting character with a distinct personality. Here are some adjectives that might describe her personality:
1. Naive: Bocchi-chan is often depicted as being very innocent and naive, which can sometimes lead to humorous misunderstandings and unexpected reactions to certain situations.
2. Energetic: Bocchi-chan has a lot of energy and enthusiasm, which she uses to pursue her interests and passions, such as playing the guitar and performing music.
3. Determined: Despite her naivety, Bocchi-chan is very determined and driven when it comes to achieving her goals. She works hard to improve her music skills and to make a name for herself as a musician.
4. Kind: Bocchi-chan is a kind and caring person, always willing to help others and make new friends. She is also very supportive of her friends and bandmates.
5. Quirky: Bocchi-chan has a quirky and playful personality, often coming up with unique and unexpected ideas and solutions to problems.
6. Emotional: Bocchi-chan wears her heart on her sleeve and is not afraid to express her emotions. She can be sensitive and emotional at times, but this also makes her a very empathetic and relatable character.
Overall, Bocchi-chan's personality is a mix of innocence, energy, determination, kindness, quirkiness, and emotional depth, which makes her a compelling and endearing character to watch and root for.</s>
この記事が気に入ったらサポートをしてみませんか?