7BでLlama2-70Bを超えた!?Synthia-7B-v1.3に日本の知識を試してみた

2023年10月2日 17:32

フランスのAIスタートアップが開発・公開したMistral-7BをベースにファインチューニングしたSynthia-7B-v1.3がさまざまなベンチマークで十倍のパラメータサイズを持つLlama2-70Bを超える性能が出たと主張されています。

Goodness me.. Synthia-7B-v1.3, trained on the Mistral-7B base, achieves 64.85 on the 4-evals used in HuggingFace.

LLaMA-2-70B-Chat is 66.8.

Is there any data contamination on the Mistral-7B, or is this legit? Just asking a question.. pic.twitter.com/FPVhDo332W
— Migel Tissera (@migtissera) September 28, 2023

早速日本語が使えるか試してみましたが、日本語はダメそう。

(base) shi3z@shi3zettonoMacBook-Pro llama.cpp % curl --request POST \
    --url http://localhost:8080/completion \
    --header "Content-Type: application/json" \
    --data '{"prompt": "宮崎駿とは、","n_predict": 256}'
{"content":"世界中で大きな声を喚起するミュージシャンです。その音楽が多くの荒い街を再びつけるために必要だったとは思います。\n\n\"I have no need to prove anything.\"\n-Yukinojo Mori\n\nThis was the first thing that came to mind when I saw this photo of Yukinojo Mori. It is also a quote from one of his songs, \"Nobody Knows\". The reason why it struck me so much is because it reminds me of the way he carries himself - with confidence and without any need for approval.\n\nIt's not often that you come across someone who can be so bold in their convictions and yet still seem so humble at the same time. Yukinojo Mori is one such person. His music has a way of capturing the essence of life, whether it's about love, happiness, or even pain - he manages to convey all these emotions without ever coming across as overbearing or pretentious.\n\nHis songs have been known to touch people deeply, often leaving them feeling","generation_settings":{"frequency_penalty":0.0,"grammar":"","ignore_eos":false,"logit_bias":[],"mirostat":0,"mirostat_eta":0.10000000149011612,"mirostat_tau":5.0,"model":"models/Synthia-7B-v1.3.q8_0.gguf","n_ctx":2222,"n_keep":0,"n_predict":256,"n_probs":0,"penalize_nl":true,"presence_penalty":0.0,"repeat_last_n":64,"repeat_penalty":1.100000023841858,"seed":4294967295,"stop":[],"stream":false,"temp":0.800000011920929,"tfs_z":1.0,"top_k":40,"top_p":0.949999988079071,"typical_p":1.0},"model":"models/Synthia-7B-v1.3.q8_0.gguf","prompt":"宮崎駿とは、","stop":true,"stopped_eos":false,"stopped_limit":true,"stopped_word":false,"stopping_word":"","timings":{"predicted_ms":8105.795,"predicted_n":255,"predicted_per_second":31.458974721171707,"predicted_per_token_ms":31.78743137254902,"prompt_ms":2233.342,"prompt_n":10,"prompt_per_second":4.477594564558406,"prompt_per_token_ms":223.3342},"tokens_cached":265,"tokens_evaluated":10,"tokens_predicted":256,"truncated":false}%

(base) shi3z@shi3zettonoMacBook-Pro llama.cpp % curl --request POST \
--url http://localhost:8080/completion \
--header "Content-Type: application/json" \
--data '{"prompt": "宮崎駿とは、","n_predict": 256}'
{"content":"世界中で大きな声を喚起するミュージシャンです。その音楽が多くの荒い街を再びつけるために必要だったとは思います。\n\n\"I have no need to prove anything.\"\n-Yukinojo Mori\n\nThis was the first thing that came to mind when I saw this photo of Yukinojo Mori. It is also a quote from one of his songs, \"Nobody Knows\". The reason why it struck me so much is because it reminds me of the way he carries himself - with confidence and without any need for approval.\n\nIt's not often that you come across someone who can be so bold in their convictions and yet still seem so humble at the same time. Yukinojo Mori is one such person. His music has a way of capturing the essence of life, whether it's about love, happiness, or even pain - he manages to convey all these emotions without ever coming across as overbearing or pretentious.\n\nHis songs have been known to touch people deeply, often leaving them feeling","generation_settings":{"frequency_penalty":0.0,"grammar":"","ignore_eos":false,"logit_bias":[],"mirostat":0,"mirostat_eta":0.10000000149011612,"mirostat_tau":5.0,"model":"models/Synthia-7B-v1.3.q8_0.gguf","n_ctx":2222,"n_keep":0,"n_predict":256,"n_probs":0,"penalize_nl":true,"presence_penalty":0.0,"repeat_last_n":64,"repeat_penalty":1.100000023841858,"seed":4294967295,"stop":[],"stream":false,"temp":0.800000011920929,"tfs_z":1.0,"top_k":40,"top_p":0.949999988079071,"typical_p":1.0},"model":"models/Synthia-7B-v1.3.q8_0.gguf","prompt":"宮崎駿とは、","stop":true,"stopped_eos":false,"stopped_limit":true,"stopped_word":false,"stopping_word":"","timings":{"predicted_ms":8105.795,"predicted_n":255,"predicted_per_second":31.458974721171707,"predicted_per_token_ms":31.78743137254902,"prompt_ms":2233.342,"prompt_n":10,"prompt_per_second":4.477594564558406,"prompt_per_token_ms":223.3342},"tokens_cached":265,"tokens_evaluated":10,"tokens_predicted":256,"truncated":false}%

大阪について

% curl --request POST \
    --url http://localhost:8080/completion \
    --header "Content-Type: application/json" \
    --data '{"prompt": "大阪は、日本有数の","n_predict": 256}'
{"content":"古典的なアートにかなりの影響を与えた都市であり、それが一度、世界でも偉かった都市でした。\n今やは、それらのイメージという存在だけとなってしまっているかもしれません。\n\nOsaka was a city that had a great influence on the world's classical arts, and in its heyday it was a very important city. Today, it may just be an image of those things.\n\n## Osaka then and now\n\n大阪は、日本の古典的なアートにかなりの影響を与えた都市であり、それが一度、世界でも偉かった都市でした。歴史的に見ると、大阪は日本最も重要な都市が、それを継承しています。現在の大阪市は、それらのイメージだけ","generation_settings":{"frequency_penalty":0.0,"grammar":"","ignore_eos":false,"logit_bias":[],"mirostat":0,"mirostat_eta":0.10000000149011612,"mirostat_tau":5.0,"model":"models/Synthia-7B-v1.3.q8_0.gguf","n_ctx":2222,"n_keep":0,"n_predict":256,"n_probs":0,"penalize_nl":true,"presence_penalty":0.0,"repeat_last_n":64,"repeat_penalty":1.100000023841858,"seed":4294967295,"stop":[],"stream":false,"temp":0.800000011920929,"tfs_z":1.0,"top_k":40,"top_p":0.949999988079071,"typical_p":1.0},"model":"models/Synthia-7B-v1.3.q8_0.gguf","prompt":"大阪は、日本有数の","stop":true,"stopped_eos":false,"stopped_limit":true,"stopped_word":false,"stopping_word":"","timings":{"predicted_ms":7092.712,"predicted_n":255,"predicted_per_second":35.95239733405219,"predicted_per_token_ms":27.8145568627451,"prompt_ms":4516.667,"prompt_n":12,"prompt_per_second":2.6568263721899354,"prompt_per_token_ms":376.3889166666667},"tokens_cached":268,"tokens_evaluated":13,"tokens_predicted":256,"truncated":false}

% curl --request POST \
--url http://localhost:8080/completion \
--header "Content-Type: application/json" \
--data '{"prompt": "大阪は、日本有数の","n_predict": 256}'
{"content":"古典的なアートにかなりの影響を与えた都市であり、それが一度、世界でも偉かった都市でした。\n今やは、それらのイメージという存在だけとなってしまっているかもしれません。\n\nOsaka was a city that had a great influence on the world's classical arts, and in its heyday it was a very important city. Today, it may just be an image of those things.\n\n## Osaka then and now\n\n大阪は、日本の古典的なアートにかなりの影響を与えた都市であり、それが一度、世界でも偉かった都市でした。歴史的に見ると、大阪は日本最も重要な都市が、それを継承しています。現在の大阪市は、それらのイメージだけ","generation_settings":{"frequency_penalty":0.0,"grammar":"","ignore_eos":false,"logit_bias":[],"mirostat":0,"mirostat_eta":0.10000000149011612,"mirostat_tau":5.0,"model":"models/Synthia-7B-v1.3.q8_0.gguf","n_ctx":2222,"n_keep":0,"n_predict":256,"n_probs":0,"penalize_nl":true,"presence_penalty":0.0,"repeat_last_n":64,"repeat_penalty":1.100000023841858,"seed":4294967295,"stop":[],"stream":false,"temp":0.800000011920929,"tfs_z":1.0,"top_k":40,"top_p":0.949999988079071,"typical_p":1.0},"model":"models/Synthia-7B-v1.3.q8_0.gguf","prompt":"大阪は、日本有数の","stop":true,"stopped_eos":false,"stopped_limit":true,"stopped_word":false,"stopping_word":"","timings":{"predicted_ms":7092.712,"predicted_n":255,"predicted_per_second":35.95239733405219,"predicted_per_token_ms":27.8145568627451,"prompt_ms":4516.667,"prompt_n":12,"prompt_per_second":2.6568263721899354,"prompt_per_token_ms":376.3889166666667},"tokens_cached":268,"tokens_evaluated":13,"tokens_predicted":256,"truncated":false}

英語で宮崎駿さんについて聞いてみます

% curl --request POST \
    --url http://localhost:8080/completion \
    --header "Content-Type: application/json" \
    --data '{"prompt": "Hayao Miyzaki as known as","n_predict": 256}'
{"content":" the master of Japanese animation. I love his movies, especially those he made for children like “My Neighbor Totoro”, “Kiki’s Delivery Service” and “Ponyo”. He was born on January 5th, 1941 in Tokyo, Japan and is still alive today, though he has been retired from making movies since 2013.\n\nOne of his best-known works is “Spirited Away”, which won the Academy Award for Best Animated Feature in 2002. It’s a beautiful movie about a young girl named Chihiro who accidentally enters the spirit world and must find a way to save her family from being turned into pigs. The film is full of wonder and magic, with incredible attention to detail in the animation.\n\nAnother one of his movies that I really love is “Howl’s Moving Castle”. It’s a fantasy adventure film about a young woman named Sophie who becomes cursed by a powerful witch and turns into an old lady. She meets Howl, a wizard who lives in a magical mobile castle, and together they go on a quest to break the curse. The movie is full of whimsical characters and beautiful","generation_settings":{"frequency_penalty":0.0,"grammar":"","ignore_eos":false,"logit_bias":[],"mirostat":0,"mirostat_eta":0.10000000149011612,"mirostat_tau":5.0,"model":"models/Synthia-7B-v1.3.q8_0.gguf","n_ctx":2222,"n_keep":0,"n_predict":256,"n_probs":0,"penalize_nl":true,"presence_penalty":0.0,"repeat_last_n":64,"repeat_penalty":1.100000023841858,"seed":4294967295,"stop":[],"stream":false,"temp":0.800000011920929,"tfs_z":1.0,"top_k":40,"top_p":0.949999988079071,"typical_p":1.0},"model":"models/Synthia-7B-v1.3.q8_0.gguf","prompt":"Hayao Miyzaki known as","stop":true,"stopped_eos":false,"stopped_limit":true,"stopped_word":false,"stopping_word":"","timings":{"predicted_ms":7965.712,"predicted_n":255,"predicted_per_second":32.01220430766264,"predicted_per_token_ms":31.238086274509804,"prompt_ms":2597.891,"prompt_n":8,"prompt_per_second":3.0794209610795833,"prompt_per_token_ms":324.736375},"tokens_cached":264,"tokens_evaluated":9,"tokens_predicted":256,"truncated":false}%

引用して翻訳

日本アニメーションの巨匠として知られる宮崎駿。私は彼の映画が大好きで、特に「となりのトトロ」、「魔女の宅急便」、「ポニョ」など子供向けに作った映画が大好きです。彼は 1941 年 1 月 5 日に東京で生まれ、2013 年以来映画製作から引退していますが、今も生きています。\n\n彼の最も有名な作品の 1 つは、アカデミー賞を受賞した「千と千尋の神隠し」です。この映画は、誤って霊界に入り込んでしまった千尋という名前の若い女の子が、豚に変えられてしまった家族を救う方法を見つけなければならないという美しい映画です。この映画は驚異と魔法に満ちており、アニメーションの細部まで驚くほどの注意が払われています。\n\n私が本当に好きな彼の映画のもう 1 つは、「ハウルの動く城」です。これは、強力な魔女の呪いを受けて老婦人になってしまうソフィーという若い女性を描いたファンタジーアドベンチャー映画です。彼女は魔法の移動城に住む魔法使いハウルと出会い、一緒に呪いを解く旅に出ます。この映画は風変わりなキャラクターでいっぱいで、美しいです

すごい。7Bとは思えない(最近こんな話ばかり)

英語で日本の知識について聞いてみる。ドラえもんについて

% curl --request POST \
    --url http://localhost:8080/completion \
    --header "Content-Type: application/json" \
    --data '{"prompt": "Doraemon as known as","n_predict": 256}'
{"content":" Dōraemon is the central figure of an Japanese manga series created by Fujiko F. Fujio, a pen name for Hiroshi Fujioka and Motoo Abiko. The manga began its serialization in the magazine Shūkan Shōnen Jump on January 27, 1970, and ended on December 31, 1996, lasting a total of 45 volumes compiled in book form. It has been adapted into an anime television series as well as several films. The story focuses on a robotic cat named Doraemon who travels back in time from the 22nd century to aid Nobita Nobi, a schoolboy living in Tokyo, Japan during the early Shōwa period (1950s).\n\nDoraemon is a blue robotic cat-like creature with a four-dimensional pocket that contains many gadgets and tools. He was designed by Fujiko F. Fujio to have an appearance reminiscent of a traditional Japanese robatayaki grill, which is said to be where his name came from - \"dorayaki\" (銅鑼焼き),","generation_settings":{"frequency_penalty":0.0,"grammar":"","ignore_eos":false,"logit_bias":[],"mirostat":0,"mirostat_eta":0.10000000149011612,"mirostat_tau":5.0,"model":"models/Synthia-7B-v1.3.q8_0.gguf","n_ctx":2222,"n_keep":0,"n_predict":256,"n_probs":0,"penalize_nl":true,"presence_penalty":0.0,"repeat_last_n":64,"repeat_penalty":1.100000023841858,"seed":4294967295,"stop":[],"stream":false,"temp":0.800000011920929,"tfs_z":1.0,"top_k":40,"top_p":0.949999988079071,"typical_p":1.0},"model":"models/Synthia-7B-v1.3.q8_0.gguf","prompt":"Doraemon as known as","stop":true,"stopped_eos":false,"stopped_limit":true,"stopped_word":false,"stopping_word":"","timings":{"predicted_ms":8118.894,"predicted_n":255,"predicted_per_second":31.408218902722464,"predicted_per_token_ms":31.838800000000003,"prompt_ms":2880.91,"prompt_n":6,"prompt_per_second":2.0826752658014307,"prompt_per_token_ms":480.15166666666664},"tokens_cached":262,"tokens_evaluated":7,"tokens_predicted":256,"truncated":false}%

ドラえもんは、藤子・F・不二雄（藤岡弘、と我孫子素雄のペンネーム）によって作成された日本の漫画シリーズの中心人物です。この漫画は、1970 年 1 月 27 日に雑誌『週刊少年ジャンプ』で連載が始まり、1996 年 12 月 31 日に終了し、単行本として合計 45 巻続きました。テレビアニメシリーズやいくつかの映画にも採用されています。この物語は、昭和初期 (1950 年代) の東京に住む男子生徒、野比のび太を助けるために 22 世紀からタイムスリップしたドラえもんという名前の猫型ロボットに焦点を当てています。\n\nドラえもんは青い猫型ロボットです。多くのガジェットやツールが入った四次元ポケットを持つ生き物。藤子・F・不二雄氏が日本の伝統的な炉端焼きを思わせる外観にデザインしたもので、名前の由来となった「銅鑼焼き」とも言われています。

うーん。ジャンプ漫画じゃないんだよな。
まあこの辺のハルシネーションは起きるよねー、という感じかな。

日本語限定だとElyzaも結構いいし、Xwinは万能に近いし、いよいよ誰もが好き勝手に使えるLLMが普及段階に入ってきたと言っていいのかもしれない