見出し画像

PaperspaceでLLaMA2を試してみた

alex123bg

StableDiffusionをPaperspace上で動かして遊んでいます。いろいろ試しているうちに、画像生成だけでなく、ChatGPTのようなテキスト生成もしてみたくなりました。
そこで、Paperspaceのブログの解説記事で、試しに動かしてみようと思います。ブログはこちらです。

チュートリアルに書いてあること

はじめの説明文を和訳、要約をBingGPTにしてもらいました。
『自然言語処理（NLP）は、人間が使う言語をコンピュータで分析する技術、機械学習の一分野です。最近では、事前学習された生成モデルであるGPT（Generative Pretrained Transformer）が多くのウェブサイトやインターフェースで利用されています。その中でも、完全にオープンソースで公開されたGPTモデルがAIコミュニティで注目を集めています。LLaMAというプロジェクトは、オープンソースのデータセットと学習方法で高性能なGPTモデルを作成し、複雑な入力に対応できるように改良したり、派生プロジェクトを生み出したりしています。
　この記事では、LLaMAの最新版であるLLaMA 2について紹介します。LLaMA 2は、データセットの拡大、人間の好みに応じてチャットモードを調整、パラメータ数の増加などの改良が施されています。新しい機能や更新点について詳しく説明した後、PaperspaceのノートブックでOogaboogaというテキスト生成のWebUIを使ってモデルを実行する方法を示します。
　無料のGPUを使ってノートブックを起動するには、ページの上部とデモの前にあるRun on Gradientのリンクをクリックしてください。』

途中の説明は飛ばして、デモに進みます。

デモ

説明文をGoogle翻訳します
『ここで、Gradient Notebook にジャンプして、独自のプロジェクトで LLaMA 2 を開始する方法を見てみましょう。 これを実行するために必要なのは Gradient アカウントだけなので、無料の GPU サービスにアクセスできます。 このようにして、必要に応じて、A100 GPU で 70B モデルを使用するようにスケールアップすることもできます。 Gradio ベースの Oogabooga Text Generation Web UI で実行される GPTQ バージョンを使用してモデルを実行します。 このデモでは、ノートブックをセットアップし、モデルをダウンロードし、推論を実行する方法を示します。』

デモプロジェクトの起動リンク

クリックしてみます。そのままクリックすると同じ画面で遷移するので、別タブで開きます。

初期画面

セットアップ

『まずは環境を整えることから始めます。 WebUI リポジトリをルート ディレクトリとしてノートブックを起動しました。 まず、llama.ipynb ノートブック ファイルを開いてみましょう。 これには、Web UI でモデルを実行するために必要なものがすべて含まれています。 まず、提供されているrequirements.txtファイルを使用して要件をインストールします。 いくつかの追加パッケージも更新する必要があります。 以下のセルを実行するとセットアップが完了します。』
とのことなので、Files→llama.ipynb を開いてから、START MACHINE してみます。

llama.ipynb

START MACHINE クリックすると、いまはマシンの空きがない様子。

マシンの空きが無い

パラメータを Free-P5000 に変えたところ、マシンは起動して、エラーが出ています。

https://console.paperspace.com/t33haawew8/notebook/rf4n6i02eqobzy8?machine=Free-P5000&ref=blog.paperspace.com&_gl=1*l4fy2v*_gcl_au*MTU1NDUxNTIxMS4xNzAxOTQ5Mzgx&file=%2FREADME.md

RELOADしたり、もう一度、 llama.ipynb を開いたりしました。

仮想マシン起動後

Setup: を Run します。

SetupをRun
!pip install -r requirements.txt
2
!pip install -U datasets transformers tokenizers pydantic auto_gptq gradio
3
​
Collecting git+https://github.com/huggingface/peft (from -r requirements.txt (line 17))
  Cloning https://github.com/huggingface/peft to /tmp/pip-req-build-onvz_s47
  Running command git clone --filter=blob:none --quiet https://github.com/huggingface/peft /tmp/pip-req-build-onvz_s47
  Resolved https://github.com/huggingface/peft to commit 799420aef18146cffe98db7eeb3e4b34dc566110
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Ignoring llama-cpp-python: markers 'platform_system == "Windows"' don't match your environment
Collecting accelerate
  Downloading accelerate-0.25.0-py3-none-any.whl (265 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 265.7/265.7 kB 22.0 MB/s eta 0:00:00
Requirement already satisfied: colorama in /usr/local/lib/python3.9/dist-packages (from -r requirements.txt (line 2)) (0.4.3)
Requirement already satisfied: datasets in /usr/local/lib/python3.9/dist-packages (from -r requirements.txt (line 3)) (2.4.0)
Collecting flexgen==0.1.7
  Downloading flexgen-0.1.7-py3-none-any.whl (50 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50.8/50.8 kB 9.2 MB/s eta 0:00:00
Collecting gradio_client==0.1.4
  Downloading gradio_client-0.1.4-py3-none-any.whl (286 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 286.7/286.7 kB 30.0 MB/s eta 0:00:00
Collecting gradio
  Downloading gradio-4.12.0-py3-none-any.whl (16.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 16.6/16.6 MB 53.3 MB/s eta 0:00:0000:0100:01
Requirement already satisfied: markdown in /usr/local/lib/python3.9/dist-packages (from -r requirements.txt (line 7)) (3.4.1)
Requirement already satisfied: numpy in /usr/local/lib/python3.9/dist-packages (from -r requirements.txt (line 8)) (1.23.4)
Requirement already satisfied: pandas in /usr/local/lib/python3.9/dist-packages (from -r requirements.txt (line 9)) (1.5.0)
Collecting Pillow>=9.5.0
  Downloading Pillow-10.1.0-cp39-cp39-manylinux_2_28_x86_64.whl (3.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.6/3.6 MB 64.6 MB/s eta 0:00:00:00:01
Requirement already satisfied: pyyaml in /usr/local/lib/python3.9/dist-packages (from -r requirements.txt (line 11)) (5.4.1)
Requirement already satisfied: requests in /usr/local/lib/python3.9/dist-packages (from -r requirements.txt (line 12)) (2.28.2)
Collecting rwkv==0.7.3
  Downloading rwkv-0.7.3-py3-none-any.whl (16 kB)
Collecting safetensors==0.3.1
  Downloading safetensors-0.3.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 55.4 MB/s eta 0:00:00
Requirement already satisfied: sentencepiece in /usr/local/lib/python3.9/dist-packages (from -r requirements.txt (line 15)) (0.1.97)
Requirement already satisfied: tqdm in /usr/local/lib/python3.9/dist-packages (from -r requirements.txt (line 16)) (4.64.1)
Collecting transformers==4.29.1
  Downloading transformers-4.29.1-py3-none-any.whl (7.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.1/7.1 MB 69.3 MB/s eta 0:00:00:00:0100:01
Collecting bitsandbytes==0.38.1
  Downloading bitsandbytes-0.38.1-py3-none-any.whl (104.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 104.3/104.3 MB 13.2 MB/s eta 0:00:0000:0100:01
Collecting llama-cpp-python==0.1.50
  Downloading llama_cpp_python-0.1.50.tar.gz (1.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 54.7 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: torch>=1.12 in /usr/local/lib/python3.9/dist-packages (from flexgen==0.1.7->-r requirements.txt (line 4)) (1.12.1+cu116)
Collecting pulp
  Downloading PuLP-2.7.0-py3-none-any.whl (14.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 14.3/14.3 MB 55.6 MB/s eta 0:00:0000:010:01
Requirement already satisfied: attrs in /usr/local/lib/python3.9/dist-packages (from flexgen==0.1.7->-r requirements.txt (line 4)) (18.2.0)
Requirement already satisfied: fsspec in /usr/local/lib/python3.9/dist-packages (from gradio_client==0.1.4->-r requirements.txt (line 5)) (2023.1.0)
Requirement already satisfied: typing-extensions in /usr/local/lib/python3.9/dist-packages (from gradio_client==0.1.4->-r requirements.txt (line 5)) (4.4.0)
Collecting httpx
  Downloading httpx-0.26.0-py3-none-any.whl (75 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 75.9/75.9 kB 12.9 MB/s eta 0:00:00
Collecting websockets
  Downloading websockets-12.0-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (130 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 130.0/130.0 kB 23.3 MB/s eta 0:00:00
Collecting huggingface-hub>=0.13.0
  Downloading huggingface_hub-0.20.1-py3-none-any.whl (330 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 330.1/330.1 kB 46.6 MB/s eta 0:00:00
Requirement already satisfied: packaging in /usr/local/lib/python3.9/dist-packages (from gradio_client==0.1.4->-r requirements.txt (line 5)) (23.0)
Collecting tokenizers>=0.13.2
  Downloading tokenizers-0.15.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.8/3.8 MB 65.7 MB/s eta 0:00:00ta 0:00:01
Requirement already satisfied: filelock in /usr/local/lib/python3.9/dist-packages (from transformers==4.29.1->-r requirements.txt (line 18)) (3.9.0)
  Downloading tokenizers-0.13.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.8/7.8 MB 63.5 MB/s eta 0:00:0000:0100:01
Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.9/dist-packages (from transformers==4.29.1->-r requirements.txt (line 18)) (2022.10.31)
Collecting typing-extensions
  Downloading typing_extensions-4.9.0-py3-none-any.whl (32 kB)
Requirement already satisfied: psutil in /usr/local/lib/python3.9/dist-packages (from accelerate->-r requirements.txt (line 1)) (5.9.4)
Requirement already satisfied: pyarrow>=6.0.0 in /usr/local/lib/python3.9/dist-packages (from datasets->-r requirements.txt (line 3)) (10.0.1)
Requirement already satisfied: aiohttp in /usr/local/lib/python3.9/dist-packages (from datasets->-r requirements.txt (line 3)) (3.8.3)
Requirement already satisfied: responses<0.19 in /usr/local/lib/python3.9/dist-packages (from datasets->-r requirements.txt (line 3)) (0.18.0)
Requirement already satisfied: dill<0.3.6 in /usr/local/lib/python3.9/dist-packages (from datasets->-r requirements.txt (line 3)) (0.3.5.1)
Requirement already satisfied: xxhash in /usr/local/lib/python3.9/dist-packages (from datasets->-r requirements.txt (line 3)) (3.2.0)
Requirement already satisfied: multiprocess in /usr/local/lib/python3.9/dist-packages (from datasets->-r requirements.txt (line 3)) (0.70.13)
Collecting aiofiles<24.0,>=22.0
  Downloading aiofiles-23.2.1-py3-none-any.whl (15 kB)
Collecting orjson~=3.0
  Downloading orjson-3.9.10-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (138 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 138.6/138.6 kB 20.4 MB/s eta 0:00:00
Requirement already satisfied: importlib-resources<7.0,>=1.3 in /usr/local/lib/python3.9/dist-packages (from gradio->-r requirements.txt (line 6)) (5.10.2)
Collecting pydantic>=2.0
  Downloading pydantic-2.5.3-py3-none-any.whl (381 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 381.9/381.9 kB 49.2 MB/s eta 0:00:00
Collecting uvicorn>=0.14.0
  Downloading uvicorn-0.25.0-py3-none-any.whl (60 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 60.3/60.3 kB 13.9 MB/s eta 0:00:00
Collecting altair<6.0,>=4.2.0
  Downloading altair-5.2.0-py3-none-any.whl (996 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 996.9/996.9 kB 63.8 MB/s eta 0:00:00
Collecting gradio
  Downloading gradio-4.11.0-py3-none-any.whl (16.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 16.6/16.6 MB 61.8 MB/s eta 0:00:0000:0100:01
  Downloading gradio-4.10.0-py3-none-any.whl (16.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 16.6/16.6 MB 49.6 MB/s eta 0:00:0000:0100:01
  Downloading gradio-4.9.1-py3-none-any.whl (16.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 16.6/16.6 MB 47.5 MB/s eta 0:00:0000:0100:01
  Downloading gradio-4.9.0-py3-none-any.whl (16.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 16.6/16.6 MB 50.2 MB/s eta 0:00:0000:0100:01
  Downloading gradio-4.8.0-py3-none-any.whl (16.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 16.5/16.5 MB 46.0 MB/s eta 0:00:0000:0100:01
  Downloading gradio-4.7.1-py3-none-any.whl (16.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 16.5/16.5 MB 53.8 MB/s eta 0:00:0000:0100:01
  Downloading gradio-4.5.0-py3-none-any.whl (16.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 16.5/16.5 MB 55.7 MB/s eta 0:00:0000:0100:01
  Downloading gradio-4.4.1-py3-none-any.whl (15.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 15.4/15.4 MB 50.7 MB/s eta 0:00:0000:0100:01
  Downloading gradio-4.4.0-py3-none-any.whl (15.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 15.4/15.4 MB 60.9 MB/s eta 0:00:0000:0100:01
  Downloading gradio-4.3.0-py3-none-any.whl (15.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 15.4/15.4 MB 52.9 MB/s eta 0:00:0000:0100:01
  Downloading gradio-4.2.0-py3-none-any.whl (15.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 15.4/15.4 MB 28.3 MB/s eta 0:00:0000:0100:01
  Downloading gradio-4.1.2-py3-none-any.whl (15.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 15.4/15.4 MB 50.4 MB/s eta 0:00:0000:0100:01
  Downloading gradio-4.1.1-py3-none-any.whl (15.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 15.4/15.4 MB 60.4 MB/s eta 0:00:0000:0100:01
  Downloading gradio-4.1.0-py3-none-any.whl (15.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 15.4/15.4 MB 51.9 MB/s eta 0:00:0000:0100:01
  Downloading gradio-4.0.2-py3-none-any.whl (25.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 25.3/25.3 MB 43.1 MB/s eta 0:00:0000:0100:01
  Downloading gradio-4.0.1-py3-none-any.whl (25.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 25.3/25.3 MB 42.1 MB/s eta 0:00:0000:0100:01
  Downloading gradio-4.0.0-py3-none-any.whl (25.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 25.3/25.3 MB 44.6 MB/s eta 0:00:0000:0100:01
  Downloading gradio-3.50.2-py3-none-any.whl (20.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 20.3/20.3 MB 47.1 MB/s eta 0:00:0000:0100:01
Collecting websockets
  Downloading websockets-11.0.3-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (129 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 129.7/129.7 kB 20.5 MB/s eta 0:00:00
Requirement already satisfied: pydantic!=1.8,!=1.8.1,!=2.0.0,!=2.0.1,<3.0.0,>=1.7.4 in /usr/local/lib/python3.9/dist-packages (from gradio->-r requirements.txt (line 6)) (1.9.2)
Collecting ffmpy
  Downloading ffmpy-0.3.1.tar.gz (5.5 kB)
  Preparing metadata (setup.py) ... done
Requirement already satisfied: matplotlib~=3.0 in /usr/local/lib/python3.9/dist-packages (from gradio->-r requirements.txt (line 6)) (3.6.1)
Collecting fastapi
  Downloading fastapi-0.105.0-py3-none-any.whl (93 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 93.1/93.1 kB 20.2 MB/s eta 0:00:00
Collecting pydub
  Downloading pydub-0.25.1-py2.py3-none-any.whl (32 kB)
Requirement already satisfied: markupsafe~=2.0 in /usr/local/lib/python3.9/dist-packages (from gradio->-r requirements.txt (line 6)) (2.1.2)
Collecting semantic-version~=2.0
  Downloading semantic_version-2.10.0-py2.py3-none-any.whl (15 kB)
Collecting gradio
  Downloading gradio-3.50.1-py3-none-any.whl (20.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 20.3/20.3 MB 49.6 MB/s eta 0:00:0000:0100:01
  Downloading gradio-3.50.0-py3-none-any.whl (20.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 20.3/20.3 MB 58.4 MB/s eta 0:00:0000:0100:01
  Downloading gradio-3.49.0-py3-none-any.whl (20.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 20.3/20.3 MB 51.7 MB/s eta 0:00:0000:0100:01
  Downloading gradio-3.48.0-py3-none-any.whl (20.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 20.3/20.3 MB 56.0 MB/s eta 0:00:0000:0100:01
  Downloading gradio-3.47.1-py3-none-any.whl (20.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 20.3/20.3 MB 53.1 MB/s eta 0:00:0000:0100:01
  Downloading gradio-3.47.0-py3-none-any.whl (20.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 20.3/20.3 MB 56.9 MB/s eta 0:00:0000:0100:01
  Downloading gradio-3.46.1-py3-none-any.whl (20.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 20.2/20.2 MB 56.8 MB/s eta 0:00:0000:0100:01
  Downloading gradio-3.46.0-py3-none-any.whl (20.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 20.2/20.2 MB 52.5 MB/s eta 0:00:0000:0100:01
  Downloading gradio-3.45.2-py3-none-any.whl (20.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 20.2/20.2 MB 56.6 MB/s eta 0:00:0000:0100:01
  Downloading gradio-3.45.1-py3-none-any.whl (20.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 20.2/20.2 MB 47.6 MB/s eta 0:00:0000:0100:01
  Downloading gradio-3.45.0-py3-none-any.whl (20.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 20.2/20.2 MB 54.3 MB/s eta 0:00:0000:0100:01
  Downloading gradio-3.44.4-py3-none-any.whl (20.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 20.2/20.2 MB 51.7 MB/s eta 0:00:0000:0100:01
  Downloading gradio-3.44.3-py3-none-any.whl (20.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 20.2/20.2 MB 51.9 MB/s eta 0:00:0000:0100:01
  Downloading gradio-3.44.2-py3-none-any.whl (20.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 20.2/20.2 MB 48.0 MB/s eta 0:00:0000:0100:01
  Downloading gradio-3.44.1-py3-none-any.whl (20.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 20.2/20.2 MB 50.2 MB/s eta 0:00:0000:0100:01
  Downloading gradio-3.44.0-py3-none-any.whl (20.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 20.2/20.2 MB 48.9 MB/s eta 0:00:0000:0100:01
  Downloading gradio-3.43.2-py3-none-any.whl (20.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 20.1/20.1 MB 53.1 MB/s eta 0:00:0000:0100:01
  Downloading gradio-3.43.1-py3-none-any.whl (20.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 20.1/20.1 MB 56.6 MB/s eta 0:00:0000:0100:01
  Downloading gradio-3.43.0-py3-none-any.whl (20.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 20.1/20.1 MB 44.1 MB/s eta 0:00:0000:0100:01
  Downloading gradio-3.42.0-py3-none-any.whl (20.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 20.1/20.1 MB 54.8 MB/s eta 0:00:0000:0100:01
  Downloading gradio-3.41.2-py3-none-any.whl (20.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 20.1/20.1 MB 53.8 MB/s eta 0:00:0000:0100:01
  Downloading gradio-3.41.1-py3-none-any.whl (20.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 20.1/20.1 MB 59.0 MB/s eta 0:00:0000:0100:01
  Downloading gradio-3.41.0-py3-none-any.whl (20.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 20.1/20.1 MB 55.9 MB/s eta 0:00:0000:0100:01
  Downloading gradio-3.40.1-py3-none-any.whl (20.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 20.0/20.0 MB 62.3 MB/s eta 0:00:0000:0100:01
Collecting mdit-py-plugins<=0.3.3
  Downloading mdit_py_plugins-0.3.3-py3-none-any.whl (50 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50.5/50.5 kB 10.4 MB/s eta 0:00:00
Collecting gradio
  Downloading gradio-3.40.0-py3-none-any.whl (20.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 20.0/20.0 MB 39.9 MB/s eta 0:00:0000:0100:01
  Downloading gradio-3.39.0-py3-none-any.whl (19.9 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 19.9/19.9 MB 55.3 MB/s eta 0:00:0000:0100:01
  Downloading gradio-3.38.0-py3-none-any.whl (19.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 19.8/19.8 MB 57.7 MB/s eta 0:00:0000:0100:01
  Downloading gradio-3.37.0-py3-none-any.whl (19.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 19.8/19.8 MB 56.3 MB/s eta 0:00:0000:0100:01
  Downloading gradio-3.36.1-py3-none-any.whl (19.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 19.8/19.8 MB 51.1 MB/s eta 0:00:0000:0100:01
Requirement already satisfied: jinja2 in /usr/local/lib/python3.9/dist-packages (from gradio->-r requirements.txt (line 6)) (3.1.2)
  Downloading gradio-3.36.0-py3-none-any.whl (19.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 19.8/19.8 MB 51.1 MB/s eta 0:00:0000:0100:01
  Downloading gradio-3.35.2-py3-none-any.whl (19.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 19.7/19.7 MB 56.2 MB/s eta 0:00:0000:0100:01
  Downloading gradio-3.35.1-py3-none-any.whl (19.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 19.7/19.7 MB 8.7 MB/s eta 0:00:0000:0100:01m
  Downloading gradio-3.35.0-py3-none-any.whl (19.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 19.7/19.7 MB 57.7 MB/s eta 0:00:0000:0100:01
  Downloading gradio-3.34.0-py3-none-any.whl (20.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 20.0/20.0 MB 54.1 MB/s eta 0:00:0000:0100:01
  Downloading gradio-3.33.1-py3-none-any.whl (20.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 20.0/20.0 MB 50.0 MB/s eta 0:00:0000:0100:01
  Downloading gradio-3.33.0-py3-none-any.whl (20.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 20.0/20.0 MB 36.4 MB/s eta 0:00:0000:0100:01
  Downloading gradio-3.32.0-py3-none-any.whl (19.9 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 19.9/19.9 MB 57.2 MB/s eta 0:00:0000:0100:01
  Downloading gradio-3.31.0-py3-none-any.whl (17.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17.4/17.4 MB 56.3 MB/s eta 0:00:0000:0100:01
  Downloading gradio-3.30.0-py3-none-any.whl (17.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17.3/17.3 MB 51.0 MB/s eta 0:00:0000:0100:01
  Downloading gradio-3.29.0-py3-none-any.whl (17.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17.3/17.3 MB 56.9 MB/s eta 0:00:0000:0100:01
  Downloading gradio-3.28.3-py3-none-any.whl (17.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17.3/17.3 MB 58.9 MB/s eta 0:00:0000:0100:01
Requirement already satisfied: markdown-it-py[linkify]>=2.0.0 in /usr/local/lib/python3.9/dist-packages (from gradio->-r requirements.txt (line 6)) (2.1.0)
Requirement already satisfied: pygments>=2.12.0 in /usr/local/lib/python3.9/dist-packages (from gradio->-r requirements.txt (line 6)) (2.14.0)
Collecting python-multipart
  Downloading python_multipart-0.0.6-py3-none-any.whl (45 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 45.7/45.7 kB 10.0 MB/s eta 0:00:00
Requirement already satisfied: importlib-metadata>=4.4 in /usr/local/lib/python3.9/dist-packages (from markdown->-r requirements.txt (line 7)) (6.0.0)
Requirement already satisfied: python-dateutil>=2.8.1 in /usr/local/lib/python3.9/dist-packages (from pandas->-r requirements.txt (line 9)) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.9/dist-packages (from pandas->-r requirements.txt (line 9)) (2022.7.1)
Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3/dist-packages (from requests->-r requirements.txt (line 12)) (2.8)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.9/dist-packages (from requests->-r requirements.txt (line 12)) (2.1.1)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.9/dist-packages (from requests->-r requirements.txt (line 12)) (1.26.14)
Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests->-r requirements.txt (line 12)) (2019.11.28)
Collecting torch>=1.12
  Downloading torch-2.1.2-cp39-cp39-manylinux1_x86_64.whl (670.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 670.2/670.2 MB 2.6 MB/s eta 0:00:0000:0100:01
Requirement already satisfied: toolz in /usr/local/lib/python3.9/dist-packages (from altair<6.0,>=4.2.0->gradio->-r requirements.txt (line 6)) (0.12.0)
Requirement already satisfied: jsonschema>=3.0 in /usr/local/lib/python3.9/dist-packages (from altair<6.0,>=4.2.0->gradio->-r requirements.txt (line 6)) (4.17.3)
Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /usr/local/lib/python3.9/dist-packages (from aiohttp->datasets->-r requirements.txt (line 3)) (4.0.2)
Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.9/dist-packages (from aiohttp->datasets->-r requirements.txt (line 3)) (1.3.3)
Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.9/dist-packages (from aiohttp->datasets->-r requirements.txt (line 3)) (1.3.1)
Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.9/dist-packages (from aiohttp->datasets->-r requirements.txt (line 3)) (1.8.2)
Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.9/dist-packages (from aiohttp->datasets->-r requirements.txt (line 3)) (6.0.4)
Collecting huggingface-hub>=0.13.0
  Downloading huggingface_hub-0.20.0-py3-none-any.whl (329 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 329.1/329.1 kB 52.1 MB/s eta 0:00:00
  Downloading huggingface_hub-0.19.4-py3-none-any.whl (311 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 311.7/311.7 kB 45.2 MB/s eta 0:00:00
  Downloading huggingface_hub-0.19.3-py3-none-any.whl (311 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 311.2/311.2 kB 29.6 MB/s eta 0:00:00
  Downloading huggingface_hub-0.19.2-py3-none-any.whl (311 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 311.2/311.2 kB 42.7 MB/s eta 0:00:00
  Downloading huggingface_hub-0.19.1-py3-none-any.whl (311 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 311.1/311.1 kB 31.4 MB/s eta 0:00:00
  Downloading huggingface_hub-0.19.0-py3-none-any.whl (311 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 311.2/311.2 kB 40.6 MB/s eta 0:00:00
  Downloading huggingface_hub-0.18.0-py3-none-any.whl (301 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 302.0/302.0 kB 38.7 MB/s eta 0:00:00
  Downloading huggingface_hub-0.17.3-py3-none-any.whl (295 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 295.0/295.0 kB 44.8 MB/s eta 0:00:00
Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.9/dist-packages (from importlib-metadata>=4.4->markdown->-r requirements.txt (line 7)) (3.11.0)
Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.9/dist-packages (from markdown-it-py[linkify]>=2.0.0->gradio->-r requirements.txt (line 6)) (0.1.2)
Collecting linkify-it-py~=1.0
  Downloading linkify_it_py-1.0.3-py3-none-any.whl (19 kB)
Requirement already satisfied: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil>=2.8.1->pandas->-r requirements.txt (line 9)) (1.14.0)
Collecting nvidia-cudnn-cu12==8.9.2.26
  Downloading nvidia_cudnn_cu12-8.9.2.26-py3-none-manylinux1_x86_64.whl (731.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 731.7/731.7 MB 2.3 MB/s eta 0:00:0000:0100:01
Collecting nvidia-cuda-nvrtc-cu12==12.1.105
  Downloading nvidia_cuda_nvrtc_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (23.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 23.7/23.7 MB 47.3 MB/s eta 0:00:0000:0100:01
Collecting nvidia-cufft-cu12==11.0.2.54
  Downloading nvidia_cufft_cu12-11.0.2.54-py3-none-manylinux1_x86_64.whl (121.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 121.6/121.6 MB 10.7 MB/s eta 0:00:0000:0100:01
Collecting nvidia-cusolver-cu12==11.4.5.107
  Downloading nvidia_cusolver_cu12-11.4.5.107-py3-none-manylinux1_x86_64.whl (124.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 124.2/124.2 MB 10.9 MB/s eta 0:00:0000:0100:01
Collecting nvidia-cuda-runtime-cu12==12.1.105
  Downloading nvidia_cuda_runtime_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (823 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 823.6/823.6 kB 49.9 MB/s eta 0:00:00
Collecting nvidia-cusparse-cu12==12.1.0.106
  Downloading nvidia_cusparse_cu12-12.1.0.106-py3-none-manylinux1_x86_64.whl (196.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 196.0/196.0 MB 7.8 MB/s eta 0:00:0000:0100:01
Collecting sympy
  Downloading sympy-1.12-py3-none-any.whl (5.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.7/5.7 MB 66.4 MB/s eta 0:00:00:00:010:01
Collecting nvidia-cuda-cupti-cu12==12.1.105
  Downloading nvidia_cuda_cupti_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (14.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 14.1/14.1 MB 62.7 MB/s eta 0:00:0000:010:01
Collecting nvidia-cublas-cu12==12.1.3.1
  Downloading nvidia_cublas_cu12-12.1.3.1-py3-none-manylinux1_x86_64.whl (410.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 410.6/410.6 MB 4.2 MB/s eta 0:00:0000:0100:01
Collecting nvidia-nccl-cu12==2.18.1
  Downloading nvidia_nccl_cu12-2.18.1-py3-none-manylinux1_x86_64.whl (209.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 209.8/209.8 MB 5.4 MB/s eta 0:00:0000:0100:01
Collecting nvidia-curand-cu12==10.3.2.106
  Downloading nvidia_curand_cu12-10.3.2.106-py3-none-manylinux1_x86_64.whl (56.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56.5/56.5 MB 22.7 MB/s eta 0:00:0000:0100:01
Collecting nvidia-nvtx-cu12==12.1.105
  Downloading nvidia_nvtx_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (99 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 99.1/99.1 kB 19.6 MB/s eta 0:00:00
Collecting triton==2.1.0
  Downloading triton-2.1.0-0-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (89.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 89.3/89.3 MB 15.9 MB/s eta 0:00:0000:0100:01
Requirement already satisfied: networkx in /usr/local/lib/python3.9/dist-packages (from torch>=1.12->flexgen==0.1.7->-r requirements.txt (line 4)) (3.0)
Collecting nvidia-nvjitlink-cu12
  Downloading nvidia_nvjitlink_cu12-12.3.101-py3-none-manylinux1_x86_64.whl (20.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 20.5/20.5 MB 53.8 MB/s eta 0:00:0000:0100:01
Collecting anyio<4.0.0,>=3.7.1
  Downloading anyio-3.7.1-py3-none-any.whl (80 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 80.9/80.9 kB 20.2 MB/s eta 0:00:00
Collecting starlette<0.28.0,>=0.27.0
  Downloading starlette-0.27.0-py3-none-any.whl (66 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 67.0/67.0 kB 12.4 MB/s eta 0:00:00
Requirement already satisfied: sniffio in /usr/local/lib/python3.9/dist-packages (from httpx->gradio_client==0.1.4->-r requirements.txt (line 5)) (1.3.0)
Collecting httpcore==1.*
  Downloading httpcore-1.0.2-py3-none-any.whl (76 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 76.9/76.9 kB 18.6 MB/s eta 0:00:00
Collecting h11<0.15,>=0.13
  Downloading h11-0.14.0-py3-none-any.whl (58 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 58.3/58.3 kB 10.3 MB/s eta 0:00:00
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.9/dist-packages (from matplotlib~=3.0->gradio->-r requirements.txt (line 6)) (1.4.4)
Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.9/dist-packages (from matplotlib~=3.0->gradio->-r requirements.txt (line 6)) (1.0.7)
Requirement already satisfied: pyparsing>=2.2.1 in /usr/local/lib/python3.9/dist-packages (from matplotlib~=3.0->gradio->-r requirements.txt (line 6)) (3.0.9)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.9/dist-packages (from matplotlib~=3.0->gradio->-r requirements.txt (line 6)) (0.11.0)
Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.9/dist-packages (from matplotlib~=3.0->gradio->-r requirements.txt (line 6)) (4.38.0)
Requirement already satisfied: click>=7.0 in /usr/local/lib/python3.9/dist-packages (from uvicorn>=0.14.0->gradio->-r requirements.txt (line 6)) (8.1.3)
Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.9/dist-packages (from anyio<4.0.0,>=3.7.1->fastapi->gradio->-r requirements.txt (line 6)) (1.1.0)
Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /usr/local/lib/python3.9/dist-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio->-r requirements.txt (line 6)) (0.19.3)
Collecting uc-micro-py
  Downloading uc_micro_py-1.0.2-py3-none-any.whl (6.2 kB)
Collecting mpmath>=0.19
  Downloading mpmath-1.3.0-py3-none-any.whl (536 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 536.2/536.2 kB 54.4 MB/s eta 0:00:00
Building wheels for collected packages: llama-cpp-python, peft, ffmpy
  Building wheel for llama-cpp-python (pyproject.toml) ... done
  Created wheel for llama-cpp-python: filename=llama_cpp_python-0.1.50-cp39-cp39-linux_x86_64.whl size=204024 sha256=511158addaff65c6e4975d77188ba86e545abc5e2be2c09ad537969d55fdd89b
  Stored in directory: /root/.cache/pip/wheels/5f/32/00/68c22a9aef38784afb1998077e3c501a15f209cb262ee971d0
  Building wheel for peft (pyproject.toml) ... done
  Created wheel for peft: filename=peft-0.7.2.dev0-py3-none-any.whl size=169329 sha256=7a409f6f7f5bb4d9e6b61af2ddb7e1b9a60d7b8f1b0106fa8b6929c9e4b11790
  Stored in directory: /tmp/pip-ephem-wheel-cache-lvdbgll_/wheels/45/06/33/0048c03714539b315d99beef9ea6b6dad0fd5750105e221583
  Building wheel for ffmpy (setup.py) ... done
  Created wheel for ffmpy: filename=ffmpy-0.3.1-py3-none-any.whl size=5579 sha256=55defb8e92cfb1924ef6b5cf02f60d5384e193a249495c9f72ca7cfe16e0e3b2
  Stored in directory: /root/.cache/pip/wheels/3c/79/71/1b3a0bc0ca224fc8af5087101cf28adc0054ea6521c4b758ec
Successfully built llama-cpp-python peft ffmpy
Installing collected packages: tokenizers, safetensors, pydub, pulp, mpmath, ffmpy, bitsandbytes, websockets, uc-micro-py, typing-extensions, triton, sympy, semantic-version, rwkv, python-multipart, Pillow, orjson, nvidia-nvtx-cu12, nvidia-nvjitlink-cu12, nvidia-nccl-cu12, nvidia-curand-cu12, nvidia-cufft-cu12, nvidia-cuda-runtime-cu12, nvidia-cuda-nvrtc-cu12, nvidia-cuda-cupti-cu12, nvidia-cublas-cu12, h11, anyio, aiofiles, uvicorn, starlette, nvidia-cusparse-cu12, nvidia-cudnn-cu12, mdit-py-plugins, llama-cpp-python, linkify-it-py, huggingface-hub, httpcore, transformers, nvidia-cusolver-cu12, httpx, fastapi, altair, torch, gradio_client, gradio, flexgen, accelerate, peft
  Attempting uninstall: tokenizers
    Found existing installation: tokenizers 0.12.1
    Uninstalling tokenizers-0.12.1:
      Successfully uninstalled tokenizers-0.12.1
  Attempting uninstall: typing-extensions
    Found existing installation: typing_extensions 4.4.0
    Uninstalling typing_extensions-4.4.0:
      Successfully uninstalled typing_extensions-4.4.0
  Attempting uninstall: Pillow
    Found existing installation: Pillow 9.2.0
    Uninstalling Pillow-9.2.0:
      Successfully uninstalled Pillow-9.2.0
  Attempting uninstall: anyio
    Found existing installation: anyio 3.6.2
    Uninstalling anyio-3.6.2:
      Successfully uninstalled anyio-3.6.2
  Attempting uninstall: huggingface-hub
    Found existing installation: huggingface-hub 0.12.0
    Uninstalling huggingface-hub-0.12.0:
      Successfully uninstalled huggingface-hub-0.12.0
  Attempting uninstall: transformers
    Found existing installation: transformers 4.21.3
    Uninstalling transformers-4.21.3:
      Successfully uninstalled transformers-4.21.3
  Attempting uninstall: torch
    Found existing installation: torch 1.12.1+cu116
    Uninstalling torch-1.12.1+cu116:
      Successfully uninstalled torch-1.12.1+cu116
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
torchvision 0.13.1+cu116 requires torch==1.12.1, but you have torch 2.1.2 which is incompatible.
torchaudio 0.12.1+cu116 requires torch==1.12.1, but you have torch 2.1.2 which is incompatible.
Successfully installed Pillow-10.1.0 accelerate-0.25.0 aiofiles-23.2.1 altair-5.2.0 anyio-3.7.1 bitsandbytes-0.38.1 fastapi-0.105.0 ffmpy-0.3.1 flexgen-0.1.7 gradio-3.28.3 gradio_client-0.1.4 h11-0.14.0 httpcore-1.0.2 httpx-0.26.0 huggingface-hub-0.17.3 linkify-it-py-1.0.3 llama-cpp-python-0.1.50 mdit-py-plugins-0.3.3 mpmath-1.3.0 nvidia-cublas-cu12-12.1.3.1 nvidia-cuda-cupti-cu12-12.1.105 nvidia-cuda-nvrtc-cu12-12.1.105 nvidia-cuda-runtime-cu12-12.1.105 nvidia-cudnn-cu12-8.9.2.26 nvidia-cufft-cu12-11.0.2.54 nvidia-curand-cu12-10.3.2.106 nvidia-cusolver-cu12-11.4.5.107 nvidia-cusparse-cu12-12.1.0.106 nvidia-nccl-cu12-2.18.1 nvidia-nvjitlink-cu12-12.3.101 nvidia-nvtx-cu12-12.1.105 orjson-3.9.10 peft-0.7.2.dev0 pulp-2.7.0 pydub-0.25.1 python-multipart-0.0.6 rwkv-0.7.3 safetensors-0.3.1 semantic-version-2.10.0 starlette-0.27.0 sympy-1.12 tokenizers-0.13.3 torch-2.1.2 transformers-4.29.1 triton-2.1.0 typing-extensions-4.9.0 uc-micro-py-1.0.2 uvicorn-0.25.0 websockets-12.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Requirement already satisfied: datasets in /usr/local/lib/python3.9/dist-packages (2.4.0)
Collecting datasets
  Downloading datasets-2.16.0-py3-none-any.whl (507 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 507.1/507.1 kB 34.1 MB/s eta 0:00:00
Requirement already satisfied: transformers in /usr/local/lib/python3.9/dist-packages (4.29.1)
Collecting transformers
  Downloading transformers-4.36.2-py3-none-any.whl (8.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.2/8.2 MB 58.5 MB/s eta 0:00:0000:0100:01
Requirement already satisfied: tokenizers in /usr/local/lib/python3.9/dist-packages (0.13.3)
Collecting tokenizers
  Using cached tokenizers-0.15.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.8 MB)
Requirement already satisfied: pydantic in /usr/local/lib/python3.9/dist-packages (1.9.2)
Collecting pydantic
  Using cached pydantic-2.5.3-py3-none-any.whl (381 kB)
Collecting auto_gptq
  Downloading auto_gptq-0.6.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.8/4.8 MB 61.5 MB/s eta 0:00:0000:01:00:01
Requirement already satisfied: gradio in /usr/local/lib/python3.9/dist-packages (3.28.3)
Collecting gradio
  Using cached gradio-4.12.0-py3-none-any.whl (16.6 MB)
Requirement already satisfied: multiprocess in /usr/local/lib/python3.9/dist-packages (from datasets) (0.70.13)
Requirement already satisfied: filelock in /usr/local/lib/python3.9/dist-packages (from datasets) (3.9.0)
Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.9/dist-packages (from datasets) (5.4.1)
Requirement already satisfied: aiohttp in /usr/local/lib/python3.9/dist-packages (from datasets) (3.8.3)
Requirement already satisfied: xxhash in /usr/local/lib/python3.9/dist-packages (from datasets) (3.2.0)
Requirement already satisfied: dill<0.3.8,>=0.3.0 in /usr/local/lib/python3.9/dist-packages (from datasets) (0.3.5.1)
Requirement already satisfied: pandas in /usr/local/lib/python3.9/dist-packages (from datasets) (1.5.0)
Requirement already satisfied: tqdm>=4.62.1 in /usr/local/lib/python3.9/dist-packages (from datasets) (4.64.1)
Requirement already satisfied: requests>=2.19.0 in /usr/local/lib/python3.9/dist-packages (from datasets) (2.28.2)
Requirement already satisfied: fsspec[http]<=2023.10.0,>=2023.1.0 in /usr/local/lib/python3.9/dist-packages (from datasets) (2023.1.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.9/dist-packages (from datasets) (23.0)
Collecting pyarrow-hotfix
  Downloading pyarrow_hotfix-0.6-py3-none-any.whl (7.9 kB)
Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.9/dist-packages (from datasets) (1.23.4)
Requirement already satisfied: pyarrow>=8.0.0 in /usr/local/lib/python3.9/dist-packages (from datasets) (10.0.1)
Collecting huggingface-hub>=0.19.4
  Using cached huggingface_hub-0.20.1-py3-none-any.whl (330 kB)
Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.9/dist-packages (from transformers) (2022.10.31)
Requirement already satisfied: safetensors>=0.3.1 in /usr/local/lib/python3.9/dist-packages (from transformers) (0.3.1)
Requirement already satisfied: typing-extensions>=4.6.1 in /usr/local/lib/python3.9/dist-packages (from pydantic) (4.9.0)
Collecting pydantic-core==2.14.6
  Downloading pydantic_core-2.14.6-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 69.7 MB/s eta 0:00:00
Collecting annotated-types>=0.4.0
  Downloading annotated_types-0.6.0-py3-none-any.whl (12 kB)
Collecting gekko
  Downloading gekko-1.0.6-py3-none-any.whl (12.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.2/12.2 MB 61.3 MB/s eta 0:00:0000:0100:01
Requirement already satisfied: accelerate>=0.22.0 in /usr/local/lib/python3.9/dist-packages (from auto_gptq) (0.25.0)
Requirement already satisfied: torch>=1.13.0 in /usr/local/lib/python3.9/dist-packages (from auto_gptq) (2.1.2)
Requirement already satisfied: peft>=0.5.0 in /usr/local/lib/python3.9/dist-packages (from auto_gptq) (0.7.2.dev0)
Collecting rouge
  Downloading rouge-1.0.1-py3-none-any.whl (13 kB)
Requirement already satisfied: sentencepiece in /usr/local/lib/python3.9/dist-packages (from auto_gptq) (0.1.97)
Collecting tomlkit==0.12.0
  Downloading tomlkit-0.12.0-py3-none-any.whl (37 kB)
Requirement already satisfied: pydub in /usr/local/lib/python3.9/dist-packages (from gradio) (0.25.1)
Requirement already satisfied: orjson~=3.0 in /usr/local/lib/python3.9/dist-packages (from gradio) (3.9.10)
Requirement already satisfied: matplotlib~=3.0 in /usr/local/lib/python3.9/dist-packages (from gradio) (3.6.1)
Requirement already satisfied: ffmpy in /usr/local/lib/python3.9/dist-packages (from gradio) (0.3.1)
Requirement already satisfied: pillow<11.0,>=8.0 in /usr/local/lib/python3.9/dist-packages (from gradio) (10.1.0)
Collecting typer[all]<1.0,>=0.9
  Downloading typer-0.9.0-py3-none-any.whl (45 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 45.9/45.9 kB 9.7 MB/s eta 0:00:00
Collecting gradio-client==0.8.0
  Downloading gradio_client-0.8.0-py3-none-any.whl (305 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 305.1/305.1 kB 44.7 MB/s eta 0:00:00
Requirement already satisfied: semantic-version~=2.0 in /usr/local/lib/python3.9/dist-packages (from gradio) (2.10.0)
Requirement already satisfied: uvicorn>=0.14.0 in /usr/local/lib/python3.9/dist-packages (from gradio) (0.25.0)
Requirement already satisfied: altair<6.0,>=4.2.0 in /usr/local/lib/python3.9/dist-packages (from gradio) (5.2.0)
Requirement already satisfied: python-multipart in /usr/local/lib/python3.9/dist-packages (from gradio) (0.0.6)
Requirement already satisfied: httpx in /usr/local/lib/python3.9/dist-packages (from gradio) (0.26.0)
Requirement already satisfied: fastapi in /usr/local/lib/python3.9/dist-packages (from gradio) (0.105.0)
Requirement already satisfied: jinja2<4.0 in /usr/local/lib/python3.9/dist-packages (from gradio) (3.1.2)
Requirement already satisfied: aiofiles<24.0,>=22.0 in /usr/local/lib/python3.9/dist-packages (from gradio) (23.2.1)
Requirement already satisfied: importlib-resources<7.0,>=1.3 in /usr/local/lib/python3.9/dist-packages (from gradio) (5.10.2)
Requirement already satisfied: markupsafe~=2.0 in /usr/local/lib/python3.9/dist-packages (from gradio) (2.1.2)
Collecting websockets<12.0,>=10.0
  Using cached websockets-11.0.3-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (129 kB)
Requirement already satisfied: psutil in /usr/local/lib/python3.9/dist-packages (from accelerate>=0.22.0->auto_gptq) (5.9.4)
Requirement already satisfied: jsonschema>=3.0 in /usr/local/lib/python3.9/dist-packages (from altair<6.0,>=4.2.0->gradio) (4.17.3)
Requirement already satisfied: toolz in /usr/local/lib/python3.9/dist-packages (from altair<6.0,>=4.2.0->gradio) (0.12.0)
Requirement already satisfied: charset-normalizer<3.0,>=2.0 in /usr/local/lib/python3.9/dist-packages (from aiohttp->datasets) (2.1.1)
Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.9/dist-packages (from aiohttp->datasets) (18.2.0)
Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.9/dist-packages (from aiohttp->datasets) (1.3.3)
Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.9/dist-packages (from aiohttp->datasets) (1.3.1)
Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.9/dist-packages (from aiohttp->datasets) (1.8.2)
Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /usr/local/lib/python3.9/dist-packages (from aiohttp->datasets) (4.0.2)
Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.9/dist-packages (from aiohttp->datasets) (6.0.4)
Collecting huggingface-hub>=0.19.4
  Using cached huggingface_hub-0.20.0-py3-none-any.whl (329 kB)
  Using cached huggingface_hub-0.19.4-py3-none-any.whl (311 kB)
INFO: pip is looking at multiple versions of aiohttp to determine which version is compatible with other requirements. This could take a while.
Collecting aiohttp
  Downloading aiohttp-3.9.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 55.9 MB/s eta 0:00:00
INFO: pip is looking at multiple versions of fsspec[http] to determine which version is compatible with other requirements. This could take a while.
Collecting fsspec[http]<=2023.10.0,>=2023.1.0
  Downloading fsspec-2023.10.0-py3-none-any.whl (166 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 166.4/166.4 kB 33.9 MB/s eta 0:00:00
Requirement already satisfied: zipp>=3.1.0 in /usr/local/lib/python3.9/dist-packages (from importlib-resources<7.0,>=1.3->gradio) (3.11.0)
Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.9/dist-packages (from matplotlib~=3.0->gradio) (4.38.0)
Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.9/dist-packages (from matplotlib~=3.0->gradio) (1.0.7)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.9/dist-packages (from matplotlib~=3.0->gradio) (0.11.0)
Requirement already satisfied: pyparsing>=2.2.1 in /usr/local/lib/python3.9/dist-packages (from matplotlib~=3.0->gradio) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.9/dist-packages (from matplotlib~=3.0->gradio) (2.8.2)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.9/dist-packages (from matplotlib~=3.0->gradio) (1.4.4)
Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.9/dist-packages (from pandas->datasets) (2022.7.1)
Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests>=2.19.0->datasets) (2019.11.28)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.9/dist-packages (from requests>=2.19.0->datasets) (1.26.14)
Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3/dist-packages (from requests>=2.19.0->datasets) (2.8)
Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in /usr/local/lib/python3.9/dist-packages (from torch>=1.13.0->auto_gptq) (12.1.105)
Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in /usr/local/lib/python3.9/dist-packages (from torch>=1.13.0->auto_gptq) (11.0.2.54)
Requirement already satisfied: networkx in /usr/local/lib/python3.9/dist-packages (from torch>=1.13.0->auto_gptq) (3.0)
Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in /usr/local/lib/python3.9/dist-packages (from torch>=1.13.0->auto_gptq) (10.3.2.106)
Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in /usr/local/lib/python3.9/dist-packages (from torch>=1.13.0->auto_gptq) (12.1.105)
Requirement already satisfied: nvidia-cudnn-cu12==8.9.2.26 in /usr/local/lib/python3.9/dist-packages (from torch>=1.13.0->auto_gptq) (8.9.2.26)
Requirement already satisfied: sympy in /usr/local/lib/python3.9/dist-packages (from torch>=1.13.0->auto_gptq) (1.12)
Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in /usr/local/lib/python3.9/dist-packages (from torch>=1.13.0->auto_gptq) (12.1.105)
Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in /usr/local/lib/python3.9/dist-packages (from torch>=1.13.0->auto_gptq) (12.1.105)
Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in /usr/local/lib/python3.9/dist-packages (from torch>=1.13.0->auto_gptq) (12.1.0.106)
Requirement already satisfied: triton==2.1.0 in /usr/local/lib/python3.9/dist-packages (from torch>=1.13.0->auto_gptq) (2.1.0)
Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in /usr/local/lib/python3.9/dist-packages (from torch>=1.13.0->auto_gptq) (11.4.5.107)
Requirement already satisfied: nvidia-nccl-cu12==2.18.1 in /usr/local/lib/python3.9/dist-packages (from torch>=1.13.0->auto_gptq) (2.18.1)
Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in /usr/local/lib/python3.9/dist-packages (from torch>=1.13.0->auto_gptq) (12.1.3.1)
Requirement already satisfied: nvidia-nvjitlink-cu12 in /usr/local/lib/python3.9/dist-packages (from nvidia-cusolver-cu12==11.4.5.107->torch>=1.13.0->auto_gptq) (12.3.101)
Requirement already satisfied: click<9.0.0,>=7.1.1 in /usr/local/lib/python3.9/dist-packages (from typer[all]<1.0,>=0.9->gradio) (8.1.3)
Requirement already satisfied: colorama<0.5.0,>=0.4.3 in /usr/local/lib/python3.9/dist-packages (from typer[all]<1.0,>=0.9->gradio) (0.4.3)
Requirement already satisfied: rich<14.0.0,>=10.11.0 in /usr/local/lib/python3.9/dist-packages (from typer[all]<1.0,>=0.9->gradio) (13.2.0)
Requirement already satisfied: shellingham<2.0.0,>=1.3.0 in /usr/local/lib/python3.9/dist-packages (from typer[all]<1.0,>=0.9->gradio) (1.5.0.post1)
Requirement already satisfied: h11>=0.8 in /usr/local/lib/python3.9/dist-packages (from uvicorn>=0.14.0->gradio) (0.14.0)
Requirement already satisfied: anyio<4.0.0,>=3.7.1 in /usr/local/lib/python3.9/dist-packages (from fastapi->gradio) (3.7.1)
Requirement already satisfied: starlette<0.28.0,>=0.27.0 in /usr/local/lib/python3.9/dist-packages (from fastapi->gradio) (0.27.0)
Requirement already satisfied: httpcore==1.* in /usr/local/lib/python3.9/dist-packages (from httpx->gradio) (1.0.2)
Requirement already satisfied: sniffio in /usr/local/lib/python3.9/dist-packages (from httpx->gradio) (1.3.0)
Requirement already satisfied: six in /usr/lib/python3/dist-packages (from rouge->auto_gptq) (1.14.0)
Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.9/dist-packages (from anyio<4.0.0,>=3.7.1->fastapi->gradio) (1.1.0)
Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /usr/local/lib/python3.9/dist-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio) (0.19.3)
Requirement already satisfied: markdown-it-py<3.0.0,>=2.1.0 in /usr/local/lib/python3.9/dist-packages (from rich<14.0.0,>=10.11.0->typer[all]<1.0,>=0.9->gradio) (2.1.0)
Requirement already satisfied: pygments<3.0.0,>=2.6.0 in /usr/local/lib/python3.9/dist-packages (from rich<14.0.0,>=10.11.0->typer[all]<1.0,>=0.9->gradio) (2.14.0)
Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.9/dist-packages (from sympy->torch>=1.13.0->auto_gptq) (1.3.0)
Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.9/dist-packages (from markdown-it-py<3.0.0,>=2.1.0->rich<14.0.0,>=10.11.0->typer[all]<1.0,>=0.9->gradio) (0.1.2)
Installing collected packages: websockets, typer, tomlkit, rouge, pydantic-core, pyarrow-hotfix, gekko, fsspec, annotated-types, pydantic, huggingface-hub, tokenizers, gradio-client, transformers, gradio, datasets, auto_gptq
  Attempting uninstall: websockets
    Found existing installation: websockets 12.0
    Uninstalling websockets-12.0:
      Successfully uninstalled websockets-12.0
  Attempting uninstall: typer
    Found existing installation: typer 0.4.2
    Uninstalling typer-0.4.2:
      Successfully uninstalled typer-0.4.2
  Attempting uninstall: fsspec
    Found existing installation: fsspec 2023.1.0
    Uninstalling fsspec-2023.1.0:
      Successfully uninstalled fsspec-2023.1.0
  Attempting uninstall: pydantic
    Found existing installation: pydantic 1.9.2
    Uninstalling pydantic-1.9.2:
      Successfully uninstalled pydantic-1.9.2
  Attempting uninstall: huggingface-hub
    Found existing installation: huggingface-hub 0.17.3
    Uninstalling huggingface-hub-0.17.3:
      Successfully uninstalled huggingface-hub-0.17.3
  Attempting uninstall: tokenizers
    Found existing installation: tokenizers 0.13.3
    Uninstalling tokenizers-0.13.3:
      Successfully uninstalled tokenizers-0.13.3
  Attempting uninstall: gradio-client
    Found existing installation: gradio_client 0.1.4
    Uninstalling gradio_client-0.1.4:
      Successfully uninstalled gradio_client-0.1.4
  Attempting uninstall: transformers
    Found existing installation: transformers 4.29.1
    Uninstalling transformers-4.29.1:
      Successfully uninstalled transformers-4.29.1
  Attempting uninstall: gradio
    Found existing installation: gradio 3.28.3
    Uninstalling gradio-3.28.3:
      Successfully uninstalled gradio-3.28.3
  Attempting uninstall: datasets
    Found existing installation: datasets 2.4.0
    Uninstalling datasets-2.4.0:
      Successfully uninstalled datasets-2.4.0
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
thinc 8.1.7 requires pydantic!=1.8,!=1.8.1,<1.11.0,>=1.7.4, but you have pydantic 2.5.3 which is incompatible.
spacy 3.4.1 requires pydantic!=1.8,!=1.8.1,<1.10.0,>=1.7.4, but you have pydantic 2.5.3 which is incompatible.
spacy 3.4.1 requires typer<0.5.0,>=0.3.0, but you have typer 0.9.0 which is incompatible.
confection 0.0.4 requires pydantic!=1.8,!=1.8.1,<1.11.0,>=1.7.4, but you have pydantic 2.5.3 which is incompatible.
Successfully installed annotated-types-0.6.0 auto_gptq-0.6.0 datasets-2.16.0 fsspec-2023.10.0 gekko-1.0.6 gradio-4.12.0 gradio-client-0.8.0 huggingface-hub-0.20.1 pyarrow-hotfix-0.6 pydantic-2.5.3 pydantic-core-2.14.6 rouge-1.0.1 tokenizers-0.15.0 tomlkit-0.12.0 transformers-4.36.2 typer-0.9.0 websockets-11.0.3
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

モデルのダウンロード

『Oogabooga テキスト生成 Web UI は、GPT モデルを使用した推論とトレーニングの実行を非常に簡単にするように設計されており、特に HuggingFace 形式のモデルで動作します。 これらの大きなファイルへのアクセスを容易にするために、HuggingFace モデルを簡単にダウンロードできるモデル ダウンロード スクリプトが提供されています。 2 番目のコード セルのコードを実行して、Web UI を実行する LLaMA 2 の 7B バージョンをダウンロードします。 モデルの GPTQ 最適化バージョンをダウンロードします。これにより、量子化を使用してモデルを実行するコストが大幅に削減されます。』

モデルのダウンロード
1
!python download-model.py TheBloke/Llama-2-7B-GPTQ
Downloading the model to models/TheBloke_Llama-2-7B-GPTQ
100%|██████████████████████████████████████████████████| 23.1k /23.1k  19.7MiB/s
100%|██████████████████████████████████████████████████| 4.77k /4.77k  13.2MiB/s
100%|██████████████████████████████████████████████████| 784   /784    1.81MiB/s
100%|███████████████████████████████████████████████████| 132   /132    309kiB/s
100%|██████████████████████████████████████████████████| 3.90G /3.90G  56.2MiB/s
100%|███████████████████████████████████████████████████| 188   /188    628kiB/s
100%|███████████████████████████████████████████████████| 411   /411    953kiB/s
100%|██████████████████████████████████████████████████| 1.84M /1.84M  72.0MiB/s
100%|██████████████████████████████████████████████████| 500k  /500k   18.6MiB/s
100%|██████████████████████████████████████████████████| 727   /727    1.70MiB/s

アプリケーションの実行


実行してみます。

実行コード

!python server.py --share --model TheBloke_Llama-2-13B-chat-GPTQ --load-in-8bit --bf16 --auto-devices 
2023-12-24 14:50:33 WARNING:The gradio "share link" feature uses a proprietary executable to create a reverse tunnel. Use it with care.
2023-12-24 14:50:35 INFO:Loading TheBloke_Llama-2-13B-chat-GPTQ...
2023-12-24 14:50:35 ERROR:The path to the model does not exist. Exiting.
Traceback (most recent call last):
  File "/notebooks/server.py", line 1027, in <module>
    create_interface()
  File "/notebooks/server.py", line 504, in create_interface
    create_file_saving_menus()
  File "/notebooks/server.py", line 379, in create_file_saving_menus
    with gr.Box(visible=False, elem_classes='file-saver') as shared.gradio['file_saver']:
AttributeError: module 'gradio' has no attribute 'Box'

エラーになりました。よく見ると、ダウンロードしたは Llama-2-7B-GPTQ なのに、上のスクリプトは Llama-2-13B-chat-GPTQ で異なっています。
Llama-2-7B-GPTQ　に変えてみます。

!python server.py --share --model TheBloke_Llama-2-7B-GPTQ --load-in-8bit --bf16 --auto-devices 
2023-12-24 15:13:59 WARNING:The gradio "share link" feature uses a proprietary executable to create a reverse tunnel. Use it with care.
2023-12-24 15:14:00 INFO:Loading TheBloke_Llama-2-7B-GPTQ...
2023-12-24 15:14:01 WARNING:Auto-assiging --gpu-memory 15 for your GPU to try to prevent out-of-memory errors. You can manually set other values.
2023-12-24 15:14:01 INFO:The AutoGPTQ params are: {'model_basename': 'model', 'device': 'cuda:0', 'use_triton': False, 'inject_fused_attention': True, 'inject_fused_mlp': True, 'use_safetensors': True, 'trust_remote_code': False, 'max_memory': {0: '15GiB', 'cpu': '99GiB'}, 'quantize_config': None, 'use_cuda_fp16': True}
2023-12-24 15:14:08 WARNING:Skipping module injection for FusedLlamaMLPForQuantizedModel as currently not supported with use_triton=False.
2023-12-24 15:15:18 INFO:Loaded the model in 77.57 seconds.

Traceback (most recent call last):
  File "/notebooks/server.py", line 1027, in <module>
    create_interface()
  File "/notebooks/server.py", line 504, in create_interface
    create_file_saving_menus()
  File "/notebooks/server.py", line 379, in create_file_saving_menus
    with gr.Box(visible=False, elem_classes='file-saver') as shared.gradio['file_saver']:
AttributeError: module 'gradio' has no attribute 'Box'

違うエラーが出ました。

注意書きがありますが、関連が良くわかりません。

サポートに問い合わせて解決


以下の問い合わせをしてみます。

サポート問い合わせフォーム
問い合わせ完了画面

サポートからの回答は、『Please make sure to install the latest version of Gradio and then retry the process.』との。
色々調べると、4.0.0 から attribute Box は削除されたとのことで、
その前のバージョン、3.50.2 を入れてみます。


!pip install gradio==3.50.2
Collecting gradio==3.50.2
  Using cached gradio-3.50.2-py3-none-any.whl (20.3 MB)
Requirement already satisfied: pillow<11.0,>=8.0 in /usr/local/lib/python3.9/dist-packages (from gradio==3.50.2) (10.1.0)
Requirement already satisfied: uvicorn>=0.14.0 in /usr/local/lib/python3.9/dist-packages (from gradio==3.50.2) (0.25.0)
Requirement already satisfied: packaging in /usr/local/lib/python3.9/dist-packages (from gradio==3.50.2) (23.0)
Requirement already satisfied: python-multipart in /usr/local/lib/python3.9/dist-packages (from gradio==3.50.2) (0.0.6)
Requirement already satisfied: semantic-version~=2.0 in /usr/local/lib/python3.9/dist-packages (from gradio==3.50.2) (2.10.0)
Collecting gradio-client==0.6.1
  Downloading gradio_client-0.6.1-py3-none-any.whl (299 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 299.2/299.2 kB 15.9 MB/s eta 0:00:00
Requirement already satisfied: importlib-resources<7.0,>=1.3 in /usr/local/lib/python3.9/dist-packages (from gradio==3.50.2) (5.10.2)
Requirement already satisfied: websockets<12.0,>=10.0 in /usr/local/lib/python3.9/dist-packages (from gradio==3.50.2) (11.0.3)
Requirement already satisfied: requests~=2.0 in /usr/local/lib/python3.9/dist-packages (from gradio==3.50.2) (2.28.2)
Requirement already satisfied: markupsafe~=2.0 in /usr/local/lib/python3.9/dist-packages (from gradio==3.50.2) (2.1.2)
Requirement already satisfied: aiofiles<24.0,>=22.0 in /usr/local/lib/python3.9/dist-packages (from gradio==3.50.2) (23.2.1)
Requirement already satisfied: pydantic!=1.8,!=1.8.1,!=2.0.0,!=2.0.1,<3.0.0,>=1.7.4 in /usr/local/lib/python3.9/dist-packages (from gradio==3.50.2) (2.5.3)
Requirement already satisfied: jinja2<4.0 in /usr/local/lib/python3.9/dist-packages (from gradio==3.50.2) (3.1.2)
Requirement already satisfied: matplotlib~=3.0 in /usr/local/lib/python3.9/dist-packages (from gradio==3.50.2) (3.6.1)
Requirement already satisfied: typing-extensions~=4.0 in /usr/local/lib/python3.9/dist-packages (from gradio==3.50.2) (4.9.0)
Requirement already satisfied: altair<6.0,>=4.2.0 in /usr/local/lib/python3.9/dist-packages (from gradio==3.50.2) (5.2.0)
Requirement already satisfied: huggingface-hub>=0.14.0 in /usr/local/lib/python3.9/dist-packages (from gradio==3.50.2) (0.20.1)
Requirement already satisfied: ffmpy in /usr/local/lib/python3.9/dist-packages (from gradio==3.50.2) (0.3.1)
Requirement already satisfied: fastapi in /usr/local/lib/python3.9/dist-packages (from gradio==3.50.2) (0.105.0)
Requirement already satisfied: numpy~=1.0 in /usr/local/lib/python3.9/dist-packages (from gradio==3.50.2) (1.23.4)
Requirement already satisfied: orjson~=3.0 in /usr/local/lib/python3.9/dist-packages (from gradio==3.50.2) (3.9.10)
Requirement already satisfied: pandas<3.0,>=1.0 in /usr/local/lib/python3.9/dist-packages (from gradio==3.50.2) (1.5.0)
Requirement already satisfied: pyyaml<7.0,>=5.0 in /usr/local/lib/python3.9/dist-packages (from gradio==3.50.2) (5.4.1)
Requirement already satisfied: pydub in /usr/local/lib/python3.9/dist-packages (from gradio==3.50.2) (0.25.1)
Requirement already satisfied: httpx in /usr/local/lib/python3.9/dist-packages (from gradio==3.50.2) (0.26.0)
Requirement already satisfied: fsspec in /usr/local/lib/python3.9/dist-packages (from gradio-client==0.6.1->gradio==3.50.2) (2023.10.0)
Requirement already satisfied: jsonschema>=3.0 in /usr/local/lib/python3.9/dist-packages (from altair<6.0,>=4.2.0->gradio==3.50.2) (4.17.3)
Requirement already satisfied: toolz in /usr/local/lib/python3.9/dist-packages (from altair<6.0,>=4.2.0->gradio==3.50.2) (0.12.0)
Requirement already satisfied: filelock in /usr/local/lib/python3.9/dist-packages (from huggingface-hub>=0.14.0->gradio==3.50.2) (3.9.0)
Requirement already satisfied: tqdm>=4.42.1 in /usr/local/lib/python3.9/dist-packages (from huggingface-hub>=0.14.0->gradio==3.50.2) (4.64.1)
Requirement already satisfied: zipp>=3.1.0 in /usr/local/lib/python3.9/dist-packages (from importlib-resources<7.0,>=1.3->gradio==3.50.2) (3.11.0)
Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.9/dist-packages (from matplotlib~=3.0->gradio==3.50.2) (4.38.0)
Requirement already satisfied: pyparsing>=2.2.1 in /usr/local/lib/python3.9/dist-packages (from matplotlib~=3.0->gradio==3.50.2) (3.0.9)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.9/dist-packages (from matplotlib~=3.0->gradio==3.50.2) (1.4.4)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.9/dist-packages (from matplotlib~=3.0->gradio==3.50.2) (0.11.0)
Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.9/dist-packages (from matplotlib~=3.0->gradio==3.50.2) (1.0.7)
Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.9/dist-packages (from matplotlib~=3.0->gradio==3.50.2) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.9/dist-packages (from pandas<3.0,>=1.0->gradio==3.50.2) (2022.7.1)
Requirement already satisfied: annotated-types>=0.4.0 in /usr/local/lib/python3.9/dist-packages (from pydantic!=1.8,!=1.8.1,!=2.0.0,!=2.0.1,<3.0.0,>=1.7.4->gradio==3.50.2) (0.6.0)
Requirement already satisfied: pydantic-core==2.14.6 in /usr/local/lib/python3.9/dist-packages (from pydantic!=1.8,!=1.8.1,!=2.0.0,!=2.0.1,<3.0.0,>=1.7.4->gradio==3.50.2) (2.14.6)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.9/dist-packages (from requests~=2.0->gradio==3.50.2) (2.1.1)
Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests~=2.0->gradio==3.50.2) (2019.11.28)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.9/dist-packages (from requests~=2.0->gradio==3.50.2) (1.26.14)
Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3/dist-packages (from requests~=2.0->gradio==3.50.2) (2.8)
Requirement already satisfied: h11>=0.8 in /usr/local/lib/python3.9/dist-packages (from uvicorn>=0.14.0->gradio==3.50.2) (0.14.0)
Requirement already satisfied: click>=7.0 in /usr/local/lib/python3.9/dist-packages (from uvicorn>=0.14.0->gradio==3.50.2) (8.1.3)
Requirement already satisfied: starlette<0.28.0,>=0.27.0 in /usr/local/lib/python3.9/dist-packages (from fastapi->gradio==3.50.2) (0.27.0)
Requirement already satisfied: anyio<4.0.0,>=3.7.1 in /usr/local/lib/python3.9/dist-packages (from fastapi->gradio==3.50.2) (3.7.1)
Requirement already satisfied: httpcore==1.* in /usr/local/lib/python3.9/dist-packages (from httpx->gradio==3.50.2) (1.0.2)
Requirement already satisfied: sniffio in /usr/local/lib/python3.9/dist-packages (from httpx->gradio==3.50.2) (1.3.0)
Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.9/dist-packages (from anyio<4.0.0,>=3.7.1->fastapi->gradio==3.50.2) (1.1.0)
Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /usr/local/lib/python3.9/dist-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio==3.50.2) (0.19.3)
Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.9/dist-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio==3.50.2) (18.2.0)
Requirement already satisfied: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil>=2.7->matplotlib~=3.0->gradio==3.50.2) (1.14.0)
Installing collected packages: gradio-client, gradio
  Attempting uninstall: gradio-client
    Found existing installation: gradio_client 0.7.3
    Uninstalling gradio_client-0.7.3:
      Successfully uninstalled gradio_client-0.7.3
  Attempting uninstall: gradio
    Found existing installation: gradio 4.11.0
    Uninstalling gradio-4.11.0:
      Successfully uninstalled gradio-4.11.0
Successfully installed gradio-3.50.2 gradio-client-0.6.1
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

その結果、起動はできました。

!python server.py --share --model TheBloke_Llama-2-7B-GPTQ --load-in-8bit --bf16 --auto-devices 
2023-12-25 12:20:49 WARNING:The gradio "share link" feature uses a proprietary executable to create a reverse tunnel. Use it with care.
2023-12-25 12:20:50 INFO:Loading TheBloke_Llama-2-7B-GPTQ...
2023-12-25 12:20:51 WARNING:Auto-assiging --gpu-memory 15 for your GPU to try to prevent out-of-memory errors. You can manually set other values.
2023-12-25 12:20:51 INFO:The AutoGPTQ params are: {'model_basename': 'model', 'device': 'cuda:0', 'use_triton': False, 'inject_fused_attention': True, 'inject_fused_mlp': True, 'use_safetensors': True, 'trust_remote_code': False, 'max_memory': {0: '15GiB', 'cpu': '99GiB'}, 'quantize_config': None, 'use_cuda_fp16': True}
2023-12-25 12:21:00 WARNING:Skipping module injection for FusedLlamaMLPForQuantizedModel as currently not supported with use_triton=False.
2023-12-25 12:22:07 INFO:Loaded the model in 76.74 seconds.

/usr/local/lib/python3.9/dist-packages/gradio/components/dropdown.py:231: UserWarning: The value passed into gr.Dropdown() is not in the list of choices. Please update the list of choices to include: None or set allow_custom_value=True.
  warnings.warn(
Running on local URL:  http://127.0.0.1:7860
Running on public URL: https://bbfde97664de8310b3.gradio.live

This share link expires in 72 hours. For free permanent hosting and GPU upgrades, run `gradio deploy` from Terminal to deploy to Spaces (https://huggingface.co/spaces)

https://bbfde97664de8310b3.gradio.live

起動後の画面
入力例「Question: How Long Is A Fortnight?」

　テンプレートがいろいろ用意されています。（左下のプルダウン）

テンプレート

一応、入力に対して出力されています。
テンプレートがいろいろあるので、試した結果を随時、アップしていきます。

この記事が気に入ったらサポートをしてみませんか？