LLM text generation notebook for Google Colab¶
This notebook uses https://github.com/utensil/text-generation-webui/tree/dev to train LoRa.
Run all the cells and a public gradio URL will appear at the bottom in around 5 minutes.
Parameters¶
- save_logs_to_google_drive: saves your chat logs, characters, and softprompts to Google Drive automatically, so that they will persist across sessions.
- text_streaming: streams the text output in real time instead of waiting for the full response to be completed.
- cai_chat: makes the interface look like Character.AI. Otherwise, it looks like a standard WhatsApp-like chat.
- load_in_8bit: loads the model with 8-bit precision, reducing the GPU memory usage by half. This allows you to use the full 2048 prompt length without running out of memory, at a small accuracy and speed cost.
- activate_silero_text_to_speech: responses will be audios instead of text. There are 118 voices available (
en_0toen_117), which can be set in the "Extensions" tab of the interface. You can find samples here: Silero samples. - activate_sending_pictures: adds a menu for sending pictures to the bot, which are automatically captioned using BLIP.
- activate_character_bias: an extension that adds an user-defined, hidden string at the beginning of the bot's reply with the goal of biasing the rest of the response.
- chat_language: if different than English, activates automatic translation using Google Translate, allowing you to communicate with the bot in a different language.
Credits¶
Based on https://colab.research.google.com/github/oobabooga/AI-Notebooks/blob/main/Colab-TextGen-GPU.ipynb from https://github.com/oobabooga/text-generation-webui/wiki/Running-on-Colab.
In [ ]:
#@title 1. Keep this tab alive to prevent Colab from disconnecting you { display-mode: "form" }
#@markdown Press play on the music player that will appear below:
%%html
<audio src="https://oobabooga.github.io/silence.m4a" controls>
In [ ]:
%cd /content
!git clone https://github.com/utensil/text-generation-webui
!git checkout dev
!mkdir text-generation-webui/logs
%cd text-generation-webui
!pip install -r requirements.txt
/content fatal: destination path 'text-generation-webui' already exists and is not an empty directory. fatal: not a git repository (or any of the parent directories): .git mkdir: cannot create directory ‘text-generation-webui/logs’: File exists /content/text-generation-webui Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/ Collecting git+https://github.com/huggingface/transformers (from -r requirements.txt (line 14)) Cloning https://github.com/huggingface/transformers to /tmp/pip-req-build-jtas1923 Running command git clone --filter=blob:none --quiet https://github.com/huggingface/transformers /tmp/pip-req-build-jtas1923 Resolved https://github.com/huggingface/transformers to commit cd73b9a8c140fb74cd93187f5c3d380cfc308023 Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... done Requirement already satisfied: accelerate==0.18.0 in /usr/local/lib/python3.9/dist-packages (from -r requirements.txt (line 1)) (0.18.0) Requirement already satisfied: bitsandbytes==0.37.2 in /usr/local/lib/python3.9/dist-packages (from -r requirements.txt (line 2)) (0.37.2) Requirement already satisfied: flexgen==0.1.7 in /usr/local/lib/python3.9/dist-packages (from -r requirements.txt (line 3)) (0.1.7) Requirement already satisfied: gradio==3.23.0 in /usr/local/lib/python3.9/dist-packages (from -r requirements.txt (line 4)) (3.23.0) Requirement already satisfied: markdown in /usr/local/lib/python3.9/dist-packages (from -r requirements.txt (line 5)) (3.4.3) Requirement already satisfied: numpy in /usr/local/lib/python3.9/dist-packages (from -r requirements.txt (line 6)) (1.22.4) Requirement already satisfied: peft==0.2.0 in /usr/local/lib/python3.9/dist-packages (from -r requirements.txt (line 7)) (0.2.0) Requirement already satisfied: requests in /usr/local/lib/python3.9/dist-packages (from -r requirements.txt (line 8)) (2.27.1) Requirement already satisfied: rwkv==0.7.1 in /usr/local/lib/python3.9/dist-packages (from -r requirements.txt (line 9)) (0.7.1) Requirement already satisfied: safetensors==0.3.0 in /usr/local/lib/python3.9/dist-packages (from -r requirements.txt (line 10)) (0.3.0) Requirement already satisfied: sentencepiece in /usr/local/lib/python3.9/dist-packages (from -r requirements.txt (line 11)) (0.1.97) Requirement already satisfied: tqdm in /usr/local/lib/python3.9/dist-packages (from -r requirements.txt (line 12)) (4.65.0) Requirement already satisfied: datasets in /usr/local/lib/python3.9/dist-packages (from -r requirements.txt (line 13)) (2.11.0) Requirement already satisfied: psutil in /usr/local/lib/python3.9/dist-packages (from accelerate==0.18.0->-r requirements.txt (line 1)) (5.9.4) Requirement already satisfied: pyyaml in /usr/local/lib/python3.9/dist-packages (from accelerate==0.18.0->-r requirements.txt (line 1)) (6.0) Requirement already satisfied: torch>=1.4.0 in /usr/local/lib/python3.9/dist-packages (from accelerate==0.18.0->-r requirements.txt (line 1)) (1.13.1+cu116) Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.9/dist-packages (from accelerate==0.18.0->-r requirements.txt (line 1)) (23.0) Requirement already satisfied: pulp in /usr/local/lib/python3.9/dist-packages (from flexgen==0.1.7->-r requirements.txt (line 3)) (2.7.0) Requirement already satisfied: attrs in /usr/local/lib/python3.9/dist-packages (from flexgen==0.1.7->-r requirements.txt (line 3)) (22.2.0) Requirement already satisfied: httpx in /usr/local/lib/python3.9/dist-packages (from gradio==3.23.0->-r requirements.txt (line 4)) (0.23.3) Requirement already satisfied: pandas in /usr/local/lib/python3.9/dist-packages (from gradio==3.23.0->-r requirements.txt (line 4)) (1.4.4) Requirement already satisfied: matplotlib in /usr/local/lib/python3.9/dist-packages (from gradio==3.23.0->-r requirements.txt (line 4)) (3.7.1) Requirement already satisfied: orjson in /usr/local/lib/python3.9/dist-packages (from gradio==3.23.0->-r requirements.txt (line 4)) (3.8.9) Requirement already satisfied: semantic-version in /usr/local/lib/python3.9/dist-packages (from gradio==3.23.0->-r requirements.txt (line 4)) (2.10.0) Requirement already satisfied: aiofiles in /usr/local/lib/python3.9/dist-packages (from gradio==3.23.0->-r requirements.txt (line 4)) (23.1.0) Requirement already satisfied: markupsafe in /usr/local/lib/python3.9/dist-packages (from gradio==3.23.0->-r requirements.txt (line 4)) (2.1.2) Requirement already satisfied: mdit-py-plugins<=0.3.3 in /usr/local/lib/python3.9/dist-packages (from gradio==3.23.0->-r requirements.txt (line 4)) (0.3.3) Requirement already satisfied: typing-extensions in /usr/local/lib/python3.9/dist-packages (from gradio==3.23.0->-r requirements.txt (line 4)) (4.5.0) Requirement already satisfied: jinja2 in /usr/local/lib/python3.9/dist-packages (from gradio==3.23.0->-r requirements.txt (line 4)) (3.1.2) Requirement already satisfied: python-multipart in /usr/local/lib/python3.9/dist-packages (from gradio==3.23.0->-r requirements.txt (line 4)) (0.0.6) Requirement already satisfied: aiohttp in /usr/local/lib/python3.9/dist-packages (from gradio==3.23.0->-r requirements.txt (line 4)) (3.8.4) Requirement already satisfied: altair>=4.2.0 in /usr/local/lib/python3.9/dist-packages (from gradio==3.23.0->-r requirements.txt (line 4)) (4.2.2) Requirement already satisfied: pillow in /usr/local/lib/python3.9/dist-packages (from gradio==3.23.0->-r requirements.txt (line 4)) (8.4.0) Requirement already satisfied: markdown-it-py[linkify]>=2.0.0 in /usr/local/lib/python3.9/dist-packages (from gradio==3.23.0->-r requirements.txt (line 4)) (2.2.0) Requirement already satisfied: pydantic in /usr/local/lib/python3.9/dist-packages (from gradio==3.23.0->-r requirements.txt (line 4)) (1.10.7) Requirement already satisfied: uvicorn in /usr/local/lib/python3.9/dist-packages (from gradio==3.23.0->-r requirements.txt (line 4)) (0.21.1) Requirement already satisfied: huggingface-hub in /usr/local/lib/python3.9/dist-packages (from gradio==3.23.0->-r requirements.txt (line 4)) (0.13.3) Requirement already satisfied: fastapi in /usr/local/lib/python3.9/dist-packages (from gradio==3.23.0->-r requirements.txt (line 4)) (0.95.0) Requirement already satisfied: ffmpy in /usr/local/lib/python3.9/dist-packages (from gradio==3.23.0->-r requirements.txt (line 4)) (0.3.0) Requirement already satisfied: pydub in /usr/local/lib/python3.9/dist-packages (from gradio==3.23.0->-r requirements.txt (line 4)) (0.25.1) Requirement already satisfied: websockets>=10.0 in /usr/local/lib/python3.9/dist-packages (from gradio==3.23.0->-r requirements.txt (line 4)) (10.4) Requirement already satisfied: fsspec in /usr/local/lib/python3.9/dist-packages (from gradio==3.23.0->-r requirements.txt (line 4)) (2023.3.0) Requirement already satisfied: tokenizers>=0.13.2 in /usr/local/lib/python3.9/dist-packages (from rwkv==0.7.1->-r requirements.txt (line 9)) (0.13.2) Requirement already satisfied: importlib-metadata>=4.4 in /usr/local/lib/python3.9/dist-packages (from markdown->-r requirements.txt (line 5)) (6.1.0) Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.9/dist-packages (from requests->-r requirements.txt (line 8)) (2022.12.7) Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.9/dist-packages (from requests->-r requirements.txt (line 8)) (3.4) Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.9/dist-packages (from requests->-r requirements.txt (line 8)) (1.26.15) Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.9/dist-packages (from requests->-r requirements.txt (line 8)) (2.0.12) Requirement already satisfied: multiprocess in /usr/local/lib/python3.9/dist-packages (from datasets->-r requirements.txt (line 13)) (0.70.14) Requirement already satisfied: dill<0.3.7,>=0.3.0 in /usr/local/lib/python3.9/dist-packages (from datasets->-r requirements.txt (line 13)) (0.3.6) Requirement already satisfied: responses<0.19 in /usr/local/lib/python3.9/dist-packages (from datasets->-r requirements.txt (line 13)) (0.18.0) Requirement already satisfied: pyarrow>=8.0.0 in /usr/local/lib/python3.9/dist-packages (from datasets->-r requirements.txt (line 13)) (9.0.0) Requirement already satisfied: xxhash in /usr/local/lib/python3.9/dist-packages (from datasets->-r requirements.txt (line 13)) (3.2.0) Requirement already satisfied: filelock in /usr/local/lib/python3.9/dist-packages (from transformers==4.28.0.dev0->-r requirements.txt (line 14)) (3.10.7) Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.9/dist-packages (from transformers==4.28.0.dev0->-r requirements.txt (line 14)) (2022.10.31) Requirement already satisfied: jsonschema>=3.0 in /usr/local/lib/python3.9/dist-packages (from altair>=4.2.0->gradio==3.23.0->-r requirements.txt (line 4)) (4.3.3) Requirement already satisfied: entrypoints in /usr/local/lib/python3.9/dist-packages (from altair>=4.2.0->gradio==3.23.0->-r requirements.txt (line 4)) (0.4) Requirement already satisfied: toolz in /usr/local/lib/python3.9/dist-packages (from altair>=4.2.0->gradio==3.23.0->-r requirements.txt (line 4)) (0.12.0) Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.9/dist-packages (from aiohttp->gradio==3.23.0->-r requirements.txt (line 4)) (1.3.1) Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.9/dist-packages (from aiohttp->gradio==3.23.0->-r requirements.txt (line 4)) (6.0.4) Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.9/dist-packages (from aiohttp->gradio==3.23.0->-r requirements.txt (line 4)) (1.8.2) Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /usr/local/lib/python3.9/dist-packages (from aiohttp->gradio==3.23.0->-r requirements.txt (line 4)) (4.0.2) Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.9/dist-packages (from aiohttp->gradio==3.23.0->-r requirements.txt (line 4)) (1.3.3) Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.9/dist-packages (from importlib-metadata>=4.4->markdown->-r requirements.txt (line 5)) (3.15.0) Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.9/dist-packages (from markdown-it-py[linkify]>=2.0.0->gradio==3.23.0->-r requirements.txt (line 4)) (0.1.2) Requirement already satisfied: linkify-it-py<3,>=1 in /usr/local/lib/python3.9/dist-packages (from markdown-it-py[linkify]>=2.0.0->gradio==3.23.0->-r requirements.txt (line 4)) (2.0.0) Requirement already satisfied: python-dateutil>=2.8.1 in /usr/local/lib/python3.9/dist-packages (from pandas->gradio==3.23.0->-r requirements.txt (line 4)) (2.8.2) Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.9/dist-packages (from pandas->gradio==3.23.0->-r requirements.txt (line 4)) (2022.7.1) Requirement already satisfied: starlette<0.27.0,>=0.26.1 in /usr/local/lib/python3.9/dist-packages (from fastapi->gradio==3.23.0->-r requirements.txt (line 4)) (0.26.1) Requirement already satisfied: sniffio in /usr/local/lib/python3.9/dist-packages (from httpx->gradio==3.23.0->-r requirements.txt (line 4)) (1.3.0) Requirement already satisfied: rfc3986[idna2008]<2,>=1.3 in /usr/local/lib/python3.9/dist-packages (from httpx->gradio==3.23.0->-r requirements.txt (line 4)) (1.5.0) Requirement already satisfied: httpcore<0.17.0,>=0.15.0 in /usr/local/lib/python3.9/dist-packages (from httpx->gradio==3.23.0->-r requirements.txt (line 4)) (0.16.3) Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.9/dist-packages (from matplotlib->gradio==3.23.0->-r requirements.txt (line 4)) (1.4.4) Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.9/dist-packages (from matplotlib->gradio==3.23.0->-r requirements.txt (line 4)) (1.0.7) Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.9/dist-packages (from matplotlib->gradio==3.23.0->-r requirements.txt (line 4)) (4.39.2) Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.9/dist-packages (from matplotlib->gradio==3.23.0->-r requirements.txt (line 4)) (3.0.9) Requirement already satisfied: importlib-resources>=3.2.0 in /usr/local/lib/python3.9/dist-packages (from matplotlib->gradio==3.23.0->-r requirements.txt (line 4)) (5.12.0) Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.9/dist-packages (from matplotlib->gradio==3.23.0->-r requirements.txt (line 4)) (0.11.0) Requirement already satisfied: h11>=0.8 in /usr/local/lib/python3.9/dist-packages (from uvicorn->gradio==3.23.0->-r requirements.txt (line 4)) (0.14.0) Requirement already satisfied: click>=7.0 in /usr/local/lib/python3.9/dist-packages (from uvicorn->gradio==3.23.0->-r requirements.txt (line 4)) (8.1.3) Requirement already satisfied: anyio<5.0,>=3.0 in /usr/local/lib/python3.9/dist-packages (from httpcore<0.17.0,>=0.15.0->httpx->gradio==3.23.0->-r requirements.txt (line 4)) (3.6.2) Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /usr/local/lib/python3.9/dist-packages (from jsonschema>=3.0->altair>=4.2.0->gradio==3.23.0->-r requirements.txt (line 4)) (0.19.3) Requirement already satisfied: uc-micro-py in /usr/local/lib/python3.9/dist-packages (from linkify-it-py<3,>=1->markdown-it-py[linkify]>=2.0.0->gradio==3.23.0->-r requirements.txt (line 4)) (1.0.1) Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.9/dist-packages (from python-dateutil>=2.8.1->pandas->gradio==3.23.0->-r requirements.txt (line 4)) (1.16.0)
In [ ]:
!ls
api-example.py extensions prompts api-example-stream.py LICENSE README.md characters logs requirements.txt convert-to-flexgen.py loras server.py convert-to-safetensors.py models settings-template.json css modules softprompts download-model.py presets training
In [ ]:
!python download-model.py decapoda-research/llama-7b-hf
Downloading the model to models/llama-7b-hf Downloading file 1 of 40... 100% 8.31k/8.31k [00:00<00:00, 4.68MiB/s] Downloading file 2 of 40... 100% 427/427 [00:00<00:00, 422kiB/s] Downloading file 3 of 40... 100% 124/124 [00:00<00:00, 120kiB/s] Downloading file 4 of 40... 100% 405M/405M [00:04<00:00, 93.7MiB/s] Downloading file 5 of 40... 100% 405M/405M [00:05<00:00, 69.4MiB/s] Downloading file 6 of 40... 100% 405M/405M [00:04<00:00, 86.2MiB/s] Downloading file 7 of 40... 100% 405M/405M [00:04<00:00, 95.0MiB/s] Downloading file 8 of 40... 100% 405M/405M [00:06<00:00, 65.4MiB/s] Downloading file 9 of 40... 100% 405M/405M [00:04<00:00, 95.9MiB/s] Downloading file 10 of 40... 100% 405M/405M [00:04<00:00, 94.3MiB/s] Downloading file 11 of 40... 100% 405M/405M [00:05<00:00, 77.3MiB/s] Downloading file 12 of 40... 100% 405M/405M [00:04<00:00, 95.5MiB/s] Downloading file 13 of 40... 100% 405M/405M [00:06<00:00, 63.3MiB/s] Downloading file 14 of 40... 100% 405M/405M [00:04<00:00, 95.3MiB/s] Downloading file 15 of 40... 100% 405M/405M [00:04<00:00, 88.1MiB/s] Downloading file 16 of 40... 100% 405M/405M [00:04<00:00, 82.2MiB/s] Downloading file 17 of 40... 100% 405M/405M [00:04<00:00, 96.3MiB/s] Downloading file 18 of 40... 100% 405M/405M [00:05<00:00, 68.1MiB/s] Downloading file 19 of 40... 100% 405M/405M [00:04<00:00, 96.8MiB/s] Downloading file 20 of 40... 100% 405M/405M [00:04<00:00, 83.8MiB/s] Downloading file 21 of 40... 100% 405M/405M [00:04<00:00, 85.5MiB/s] Downloading file 22 of 40... 100% 405M/405M [00:04<00:00, 94.8MiB/s] Downloading file 23 of 40... 100% 405M/405M [00:06<00:00, 66.4MiB/s] Downloading file 24 of 40... 100% 405M/405M [00:04<00:00, 94.6MiB/s] Downloading file 25 of 40... 100% 405M/405M [00:05<00:00, 77.4MiB/s] Downloading file 26 of 40... 100% 405M/405M [00:04<00:00, 89.8MiB/s] Downloading file 27 of 40... 100% 405M/405M [00:04<00:00, 93.3MiB/s] Downloading file 28 of 40... 100% 405M/405M [00:05<00:00, 68.5MiB/s] Downloading file 29 of 40... 100% 405M/405M [00:04<00:00, 95.2MiB/s] Downloading file 30 of 40... 100% 405M/405M [00:05<00:00, 75.3MiB/s] Downloading file 31 of 40... 100% 405M/405M [00:04<00:00, 96.6MiB/s] Downloading file 32 of 40... 100% 405M/405M [00:04<00:00, 96.2MiB/s] Downloading file 33 of 40... 100% 405M/405M [00:05<00:00, 71.0MiB/s] Downloading file 34 of 40... 100% 405M/405M [00:04<00:00, 94.4MiB/s] Downloading file 35 of 40... 100% 405M/405M [00:05<00:00, 71.8MiB/s] Downloading file 36 of 40... 100% 524M/524M [00:05<00:00, 96.9MiB/s] Downloading file 37 of 40... 100% 25.5k/25.5k [00:00<00:00, 110kiB/s] Downloading file 38 of 40... 100% 2.00/2.00 [00:00<00:00, 1.81kiB/s] Downloading file 39 of 40... 100% 500k/500k [00:00<00:00, 44.3MiB/s] Downloading file 40 of 40... 100% 141/141 [00:00<00:00, 130kiB/s]
In [ ]:
!ls -1 models/**/tokenizer_config.json|xargs sed -i -e 's/LLaMATokenizer/LlamaTokenizer/g'
In [ ]:
%cd training/datasets/
!wget https://raw.githubusercontent.com/tloen/alpaca-lora/main/alpaca_data_cleaned.json
/content/text-generation-webui/training/datasets --2023-03-30 08:02:51-- https://raw.githubusercontent.com/tloen/alpaca-lora/main/alpaca_data_cleaned.json Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ... Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 22680910 (22M) [text/plain] Saving to: ‘alpaca_data_cleaned.json.2’ alpaca_data_cleaned 100%[===================>] 21.63M --.-KB/s in 0.1s 2023-03-30 08:02:53 (217 MB/s) - ‘alpaca_data_cleaned.json.2’ saved [22680910/22680910]
In [ ]:
from google.colab import drive
drive.mount('/content/drive/', force_remount=True)
Mounted at /content/drive/
In [ ]:
!cp /content/drive/MyDrive/Colab\ Notebooks/3b.txt /content/text-generation-webui/training/datasets/
In [ ]:
!ls /content/text-generation-webui/training/datasets
In [ ]:
!head /content/text-generation-webui/training/datasets/3b.txt -n 300 | tee /content/text-generation-webui/training/datasets/3b-300.txt
In [ ]:
!python download-model.py tloen/alpaca-lora-7b
Downloading the model to loras/alpaca-lora-7b Downloading file 1 of 3... 100% 791/791 [00:00<00:00, 571kiB/s] Downloading file 2 of 3... 100% 399/399 [00:00<00:00, 344kiB/s] Downloading file 3 of 3... 100% 67.2M/67.2M [00:01<00:00, 64.6MiB/s]
In [ ]:
!python download-model.py silk-road/luotuo-lora-7b-1.0
Downloading the model to loras/luotuo-lora-7b-1.0 Downloading file 1 of 3... 100% 28.0/28.0 [00:00<00:00, 20.2kiB/s] Downloading file 2 of 3... 100% 370/370 [00:00<00:00, 256kiB/s] Downloading file 3 of 3... 100% 16.8M/16.8M [00:02<00:00, 6.31MiB/s]
In [ ]:
%cd /content/text-generation-webui
!python server.py --model llama-7b-hf --lora luotuo-lora-7b-1.0 --cai-chat --load-in-8bit --share
/content/text-generation-webui
===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================
/usr/local/lib/python3.9/dist-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: /usr/lib64-nvidia did not contain libcudart.so as expected! Searching further paths...
warn(msg)
/usr/local/lib/python3.9/dist-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/sys/fs/cgroup/memory.events /var/colab/cgroup/jupyter-children/memory.events')}
warn(msg)
/usr/local/lib/python3.9/dist-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('--listen_host=172.28.0.12 --target_host=172.28.0.12 --tunnel_background_save_url=https'), PosixPath('//colab.research.google.com/tun/m/cc48301118ce562b961b3c22d803539adc1e0c19/gpu-t4-s-3iqjdg781wur --tunnel_background_save_delay=10s --tunnel_periodic_background_save_frequency=30m0s --enable_output_coalescing=true --output_coalescing_required=true')}
warn(msg)
/usr/local/lib/python3.9/dist-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/env/python')}
warn(msg)
/usr/local/lib/python3.9/dist-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('module'), PosixPath('//ipykernel.pylab.backend_inline')}
warn(msg)
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching /usr/local/cuda/lib64...
CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so
CUDA SETUP: Highest compute capability among GPUs detected: 7.5
CUDA SETUP: Detected CUDA version 118
CUDA SETUP: Loading binary /usr/local/lib/python3.9/dist-packages/bitsandbytes/libbitsandbytes_cuda118.so...
2023-03-30 09:30:40.518307: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib64-nvidia
2023-03-30 09:30:40.518454: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib64-nvidia
2023-03-30 09:30:40.518476: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
Loading llama-7b-hf...
Loading checkpoint shards: 100% 33/33 [01:01<00:00, 1.87s/it]
Loaded the model in 65.67 seconds.
Adding the LoRA luotuo-lora-7b-1.0 to the model...
Loading the extension "gallery"... Ok.
/usr/local/lib/python3.9/dist-packages/gradio/deprecation.py:40: UserWarning: The 'type' parameter has been deprecated. Use the Number component instead.
warnings.warn(value)
Running on local URL: http://127.0.0.1:7860
Running on public URL: https://0e48ac4781a197d41f.gradio.live
This share link expires in 72 hours. For free permanent hosting and GPU upgrades (NEW!), check out Spaces: https://huggingface.co/spaces
/usr/local/lib/python3.9/dist-packages/transformers/generation/utils.py:1211: UserWarning: You have modified the pretrained model configuration to control generation. This is a deprecated strategy to control generation and will be removed soon, in a future version. Please use a generation configuration file (see https://huggingface.co/docs/transformers/main_classes/text_generation)
warnings.warn(
Output generated in 7.34 seconds (3.13 tokens/s, 23 tokens, context 38)
Output generated in 9.28 seconds (4.42 tokens/s, 41 tokens, context 75)
Output generated in 18.57 seconds (4.52 tokens/s, 84 tokens, context 135)
Output generated in 19.83 seconds (4.79 tokens/s, 95 tokens, context 237)
Output generated in 28.13 seconds (4.48 tokens/s, 126 tokens, context 347)
Output generated in 45.16 seconds (4.41 tokens/s, 199 tokens, context 688)
In [ ]:
%cd /content/text-generation-webui
!python server.py --model llama-7b-hf --lora test --cai-chat --load-in-8bit --share
/content/text-generation-webui
===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================
/usr/local/lib/python3.9/dist-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: /usr/lib64-nvidia did not contain libcudart.so as expected! Searching further paths...
warn(msg)
/usr/local/lib/python3.9/dist-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/sys/fs/cgroup/memory.events /var/colab/cgroup/jupyter-children/memory.events')}
warn(msg)
/usr/local/lib/python3.9/dist-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('--listen_host=172.28.0.12 --target_host=172.28.0.12 --tunnel_background_save_url=https'), PosixPath('//colab.research.google.com/tun/m/cc48301118ce562b961b3c22d803539adc1e0c19/gpu-t4-s-3iqjdg781wur --tunnel_background_save_delay=10s --tunnel_periodic_background_save_frequency=30m0s --enable_output_coalescing=true --output_coalescing_required=true')}
warn(msg)
/usr/local/lib/python3.9/dist-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/env/python')}
warn(msg)
/usr/local/lib/python3.9/dist-packages/bitsandbytes/cuda_setup/main.py:136: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//ipykernel.pylab.backend_inline'), PosixPath('module')}
warn(msg)
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching /usr/local/cuda/lib64...
CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so
CUDA SETUP: Highest compute capability among GPUs detected: 7.5
CUDA SETUP: Detected CUDA version 118
CUDA SETUP: Loading binary /usr/local/lib/python3.9/dist-packages/bitsandbytes/libbitsandbytes_cuda118.so...
2023-03-30 08:53:38.332948: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib64-nvidia
2023-03-30 08:53:38.333322: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib64-nvidia
2023-03-30 08:53:38.333346: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
Loading llama-7b-hf...
Loading checkpoint shards: 100% 33/33 [01:02<00:00, 1.88s/it]
Loaded the model in 64.95 seconds.
Adding the LoRA test to the model...
/usr/local/lib/python3.9/dist-packages/gradio/deprecation.py:40: UserWarning: The 'type' parameter has been deprecated. Use the Number component instead.
warnings.warn(value)
Running on local URL: http://127.0.0.1:7860
Running on public URL: https://d149ad415811cdf4fe.gradio.live
This share link expires in 72 hours. For free permanent hosting and GPU upgrades (NEW!), check out Spaces: https://huggingface.co/spaces
Closing server running on port: 7860
/usr/local/lib/python3.9/dist-packages/gradio/deprecation.py:40: UserWarning: The 'type' parameter has been deprecated. Use the Number component instead.
warnings.warn(value)
Running on local URL: http://127.0.0.1:7860
Running on public URL: https://737335d9ab1314cadb.gradio.live
This share link expires in 72 hours. For free permanent hosting and GPU upgrades (NEW!), check out Spaces: https://huggingface.co/spaces
/usr/local/lib/python3.9/dist-packages/transformers/generation/utils.py:1211: UserWarning: You have modified the pretrained model configuration to control generation. This is a deprecated strategy to control generation and will be removed soon, in a future version. Please use a generation configuration file (see https://huggingface.co/docs/transformers/main_classes/text_generation)
warnings.warn(
Output generated in 31.28 seconds (4.32 tokens/s, 135 tokens, context 38)
Output generated in 9.50 seconds (3.69 tokens/s, 35 tokens, context 192)
Output generated in 4.68 seconds (4.70 tokens/s, 22 tokens, context 252)
Output generated in 6.77 seconds (4.73 tokens/s, 32 tokens, context 297)
Output generated in 7.18 seconds (4.74 tokens/s, 34 tokens, context 374)
Output generated in 48.05 seconds (4.14 tokens/s, 199 tokens, context 427)
Error in sys.excepthook:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/dist-packages/rich/traceback.py", line 97, in excepthook
def excepthook(
KeyboardInterrupt
Original exception was:
Traceback (most recent call last):
File "/content/text-generation-webui/server.py", line 521, in <module>
time.sleep(0.5)
KeyboardInterrupt
Killing tunnel 127.0.0.1:7860 <> https://737335d9ab1314cadb.gradio.live
Killing tunnel 127.0.0.1:7860 <> https://d149ad415811cdf4fe.gradio.live
^C
In [ ]:
!ls loras
place-your-loras-here.txt test
In [ ]: