based on https://colab.research.google.com/drive/1wfrKqyPkz5BGD1Gkij_cvbUeweIDdRav?usp=sharing
on Colbab, it would OOM around 85% of the merge.
In [1]:
# download code git repository
# 下载代码仓库
!git clone -b dev https://github.com/utensil/Alpaca-CoT.git code
Cloning into 'code'... remote: Enumerating objects: 759, done. remote: Counting objects: 100% (100/100), done. remote: Compressing objects: 100% (56/56), done. remote: Total 759 (delta 56), reused 74 (delta 44), pack-reused 659 Receiving objects: 100% (759/759), 127.13 MiB | 57.27 MiB/s, done. Resolving deltas: 100% (405/405), done.
In [2]:
# install basic python package
# 安装必要的python库
!python -m pip install bitsandbytes
!python -m pip install datasets
!python -m pip install git+https://github.com/huggingface/transformers.git
!python -m pip install peft
!python -m pip install sentencepiece
!python -m pip install gradio
Requirement already satisfied: bitsandbytes in /usr/local/lib/python3.10/dist-packages (0.38.1) WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv [notice] A new release of pip is available: 23.0.1 -> 23.1 [notice] To update, run: python -m pip install --upgrade pip Requirement already satisfied: datasets in /usr/local/lib/python3.10/dist-packages (2.11.0) Requirement already satisfied: requests>=2.19.0 in /usr/local/lib/python3.10/dist-packages (from datasets) (2.28.2) Requirement already satisfied: pyarrow>=8.0.0 in /usr/local/lib/python3.10/dist-packages (from datasets) (11.0.0) Requirement already satisfied: tqdm>=4.62.1 in /usr/local/lib/python3.10/dist-packages (from datasets) (4.65.0) Requirement already satisfied: huggingface-hub<1.0.0,>=0.11.0 in /usr/local/lib/python3.10/dist-packages (from datasets) (0.13.3) Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from datasets) (6.0) Requirement already satisfied: fsspec[http]>=2021.11.1 in /usr/local/lib/python3.10/dist-packages (from datasets) (2023.3.0) Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from datasets) (1.5.3) Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (from datasets) (3.8.4) Requirement already satisfied: dill<0.3.7,>=0.3.0 in /usr/local/lib/python3.10/dist-packages (from datasets) (0.3.6) Requirement already satisfied: responses<0.19 in /usr/local/lib/python3.10/dist-packages (from datasets) (0.18.0) Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from datasets) (1.24.2) Requirement already satisfied: multiprocess in /usr/local/lib/python3.10/dist-packages (from datasets) (0.70.14) Requirement already satisfied: xxhash in /usr/local/lib/python3.10/dist-packages (from datasets) (3.2.0) Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from datasets) (23.0) Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (4.0.2) Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (1.8.2) Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (1.3.3) Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (22.2.0) Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (1.3.1) Requirement already satisfied: charset-normalizer<4.0,>=2.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (3.0.1) Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (6.0.4) Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0.0,>=0.11.0->datasets) (4.5.0) Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0.0,>=0.11.0->datasets) (3.10.4) Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3/dist-packages (from requests>=2.19.0->datasets) (2.8) Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests>=2.19.0->datasets) (2019.11.28) Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests>=2.19.0->datasets) (1.26.15) Requirement already satisfied: python-dateutil>=2.8.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets) (2.8.2) Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets) (2022.7.1) Requirement already satisfied: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil>=2.8.1->pandas->datasets) (1.14.0) WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv [notice] A new release of pip is available: 23.0.1 -> 23.1 [notice] To update, run: python -m pip install --upgrade pip Collecting git+https://github.com/huggingface/transformers.git Cloning https://github.com/huggingface/transformers.git to /tmp/pip-req-build-4n5ulz4x Running command git clone --filter=blob:none --quiet https://github.com/huggingface/transformers.git /tmp/pip-req-build-4n5ulz4x Resolved https://github.com/huggingface/transformers.git to commit dacd34568d1a27b91f84610eab526640ed8f94e0 Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... done Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.10/dist-packages (from transformers==4.29.0.dev0) (4.65.0) Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from transformers==4.29.0.dev0) (6.0) Requirement already satisfied: huggingface-hub<1.0,>=0.11.0 in /usr/local/lib/python3.10/dist-packages (from transformers==4.29.0.dev0) (0.13.3) Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers==4.29.0.dev0) (3.10.4) Requirement already satisfied: tokenizers!=0.11.3,<0.14,>=0.11.1 in /usr/local/lib/python3.10/dist-packages (from transformers==4.29.0.dev0) (0.13.2) Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers==4.29.0.dev0) (2023.3.23) Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from transformers==4.29.0.dev0) (2.28.2) Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from transformers==4.29.0.dev0) (1.24.2) Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from transformers==4.29.0.dev0) (23.0) Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.11.0->transformers==4.29.0.dev0) (4.5.0) Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.29.0.dev0) (1.26.15) Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.29.0.dev0) (3.0.1) Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests->transformers==4.29.0.dev0) (2019.11.28) Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3/dist-packages (from requests->transformers==4.29.0.dev0) (2.8) Building wheels for collected packages: transformers Building wheel for transformers (pyproject.toml) ... done Created wheel for transformers: filename=transformers-4.29.0.dev0-py3-none-any.whl size=6928059 sha256=f8a12c7142e912792ebd17593fd1b3a9901a4a6c098f3c95ca23087792ed8067 Stored in directory: /tmp/pip-ephem-wheel-cache-y6y9fuff/wheels/e7/9c/5b/e1a9c8007c343041e61cc484433d512ea9274272e3fcbe7c16 Successfully built transformers Installing collected packages: transformers Attempting uninstall: transformers Found existing installation: transformers 4.28.1 Uninstalling transformers-4.28.1: Successfully uninstalled transformers-4.28.1 Successfully installed transformers-4.29.0.dev0 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv [notice] A new release of pip is available: 23.0.1 -> 23.1 [notice] To update, run: python -m pip install --upgrade pip Requirement already satisfied: peft in /usr/local/lib/python3.10/dist-packages (0.3.0.dev0) Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from peft) (23.0) Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from peft) (5.9.4) Requirement already satisfied: transformers in /usr/local/lib/python3.10/dist-packages (from peft) (4.29.0.dev0) Requirement already satisfied: pyyaml in /usr/local/lib/python3.10/dist-packages (from peft) (6.0) Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from peft) (1.24.2) Requirement already satisfied: accelerate in /usr/local/lib/python3.10/dist-packages (from peft) (0.18.0) Requirement already satisfied: torch>=1.13.0 in /usr/local/lib/python3.10/dist-packages (from peft) (1.13.1+cu116) Requirement already satisfied: typing-extensions in /usr/local/lib/python3.10/dist-packages (from torch>=1.13.0->peft) (4.5.0) Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers->peft) (3.10.4) Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from transformers->peft) (2.28.2) Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers->peft) (2023.3.23) Requirement already satisfied: huggingface-hub<1.0,>=0.11.0 in /usr/local/lib/python3.10/dist-packages (from transformers->peft) (0.13.3) Requirement already satisfied: tokenizers!=0.11.3,<0.14,>=0.11.1 in /usr/local/lib/python3.10/dist-packages (from transformers->peft) (0.13.2) Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.10/dist-packages (from transformers->peft) (4.65.0) Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->transformers->peft) (3.0.1) Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->transformers->peft) (1.26.15) Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests->transformers->peft) (2019.11.28) Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3/dist-packages (from requests->transformers->peft) (2.8) WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv [notice] A new release of pip is available: 23.0.1 -> 23.1 [notice] To update, run: python -m pip install --upgrade pip Requirement already satisfied: sentencepiece in /usr/local/lib/python3.10/dist-packages (0.1.97) WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv [notice] A new release of pip is available: 23.0.1 -> 23.1 [notice] To update, run: python -m pip install --upgrade pip Requirement already satisfied: gradio in /usr/local/lib/python3.10/dist-packages (3.25.0) Requirement already satisfied: aiofiles in /usr/local/lib/python3.10/dist-packages (from gradio) (22.1.0) Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from gradio) (1.5.3) Requirement already satisfied: fastapi in /usr/local/lib/python3.10/dist-packages (from gradio) (0.95.0) Requirement already satisfied: pillow in /usr/local/lib/python3.10/dist-packages (from gradio) (9.5.0) Requirement already satisfied: pydantic in /usr/local/lib/python3.10/dist-packages (from gradio) (1.10.7) Requirement already satisfied: typing-extensions in /usr/local/lib/python3.10/dist-packages (from gradio) (4.5.0) Requirement already satisfied: markupsafe in /usr/local/lib/python3.10/dist-packages (from gradio) (2.1.2) Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (from gradio) (3.8.4) Requirement already satisfied: huggingface-hub>=0.13.0 in /usr/local/lib/python3.10/dist-packages (from gradio) (0.13.3) Requirement already satisfied: websockets>=10.0 in /usr/local/lib/python3.10/dist-packages (from gradio) (10.4) Requirement already satisfied: pydub in /usr/local/lib/python3.10/dist-packages (from gradio) (0.25.1) Requirement already satisfied: orjson in /usr/local/lib/python3.10/dist-packages (from gradio) (3.8.8) Requirement already satisfied: semantic-version in /usr/local/lib/python3.10/dist-packages (from gradio) (2.10.0) Requirement already satisfied: uvicorn in /usr/local/lib/python3.10/dist-packages (from gradio) (0.21.1) Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from gradio) (3.1.2) Requirement already satisfied: markdown-it-py[linkify]>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from gradio) (2.2.0) Requirement already satisfied: altair>=4.2.0 in /usr/local/lib/python3.10/dist-packages (from gradio) (4.2.2) Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from gradio) (2.28.2) Requirement already satisfied: ffmpy in /usr/local/lib/python3.10/dist-packages (from gradio) (0.3.0) Requirement already satisfied: matplotlib in /usr/local/lib/python3.10/dist-packages (from gradio) (3.7.1) Requirement already satisfied: gradio-client>=0.0.8 in /usr/local/lib/python3.10/dist-packages (from gradio) (0.1.3) Requirement already satisfied: httpx in /usr/local/lib/python3.10/dist-packages (from gradio) (0.23.3) Requirement already satisfied: pyyaml in /usr/local/lib/python3.10/dist-packages (from gradio) (6.0) Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from gradio) (1.24.2) Requirement already satisfied: python-multipart in /usr/local/lib/python3.10/dist-packages (from gradio) (0.0.6) Requirement already satisfied: mdit-py-plugins<=0.3.3 in /usr/local/lib/python3.10/dist-packages (from gradio) (0.3.3) Requirement already satisfied: entrypoints in /usr/local/lib/python3.10/dist-packages (from altair>=4.2.0->gradio) (0.4) Requirement already satisfied: jsonschema>=3.0 in /usr/local/lib/python3.10/dist-packages (from altair>=4.2.0->gradio) (4.17.3) Requirement already satisfied: toolz in /usr/local/lib/python3.10/dist-packages (from altair>=4.2.0->gradio) (0.12.0) Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from gradio-client>=0.0.8->gradio) (2023.3.0) Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from gradio-client>=0.0.8->gradio) (23.0) Requirement already satisfied: tqdm>=4.42.1 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.13.0->gradio) (4.65.0) Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from huggingface-hub>=0.13.0->gradio) (3.10.4) Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.10/dist-packages (from markdown-it-py[linkify]>=2.0.0->gradio) (0.1.2) Requirement already satisfied: linkify-it-py<3,>=1 in /usr/local/lib/python3.10/dist-packages (from markdown-it-py[linkify]>=2.0.0->gradio) (2.0.0) Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->gradio) (2022.7.1) Requirement already satisfied: python-dateutil>=2.8.1 in /usr/local/lib/python3.10/dist-packages (from pandas->gradio) (2.8.2) Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /usr/local/lib/python3.10/dist-packages (from aiohttp->gradio) (4.0.2) Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp->gradio) (1.3.1) Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->gradio) (22.2.0) Requirement already satisfied: charset-normalizer<4.0,>=2.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->gradio) (3.0.1) Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->gradio) (1.8.2) Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp->gradio) (1.3.3) Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp->gradio) (6.0.4) Requirement already satisfied: starlette<0.27.0,>=0.26.1 in /usr/local/lib/python3.10/dist-packages (from fastapi->gradio) (0.26.1) Requirement already satisfied: rfc3986[idna2008]<2,>=1.3 in /usr/local/lib/python3.10/dist-packages (from httpx->gradio) (1.5.0) Requirement already satisfied: certifi in /usr/lib/python3/dist-packages (from httpx->gradio) (2019.11.28) Requirement already satisfied: httpcore<0.17.0,>=0.15.0 in /usr/local/lib/python3.10/dist-packages (from httpx->gradio) (0.16.3) Requirement already satisfied: sniffio in /usr/local/lib/python3.10/dist-packages (from httpx->gradio) (1.3.0) Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/dist-packages (from matplotlib->gradio) (0.11.0) Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->gradio) (1.0.7) Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->gradio) (1.4.4) Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->gradio) (3.0.9) Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib->gradio) (4.39.2) Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3/dist-packages (from requests->gradio) (2.8) Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->gradio) (1.26.15) Requirement already satisfied: click>=7.0 in /usr/local/lib/python3.10/dist-packages (from uvicorn->gradio) (8.1.3) Requirement already satisfied: h11>=0.8 in /usr/local/lib/python3.10/dist-packages (from uvicorn->gradio) (0.14.0) Requirement already satisfied: anyio<5.0,>=3.0 in /usr/local/lib/python3.10/dist-packages (from httpcore<0.17.0,>=0.15.0->httpx->gradio) (3.6.2) Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /usr/local/lib/python3.10/dist-packages (from jsonschema>=3.0->altair>=4.2.0->gradio) (0.19.3) Requirement already satisfied: uc-micro-py in /usr/local/lib/python3.10/dist-packages (from linkify-it-py<3,>=1->markdown-it-py[linkify]>=2.0.0->gradio) (1.0.1) Requirement already satisfied: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil>=2.8.1->pandas->gradio) (1.14.0) WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv [notice] A new release of pip is available: 23.0.1 -> 23.1 [notice] To update, run: python -m pip install --upgrade pip
In [4]:
# download model git repository
# 拉取模型仓库(checkpoint)
!GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/QingyiSi/Alpaca-CoT model
Cloning into 'model'... remote: Enumerating objects: 44, done. remote: Counting objects: 100% (10/10), done. remote: Compressing objects: 100% (10/10), done. remote: Total 44 (delta 2), reused 0 (delta 0), pack-reused 34 Unpacking objects: 100% (44/44), 5.44 KiB | 619.00 KiB/s, done.
In [6]:
# pull specify model and move it
# 从仓库中拉取指定的模型checkpoint，你可以在 https://huggingface.co/QingyiSi/Alpaca-CoT 查看可选的checkpoint
!cd /workspace/model && git config core.sparsecheckout true && git config lfs.fetchinclude saved-alpaca7b
!cd /workspace/model && git lfs pull
!du -lh /workspace/model/ -d 1
17M /workspace/model/.git100% (1/1), 17 MB | 0 B/s 8.0K /workspace/model/saved-alpaca-belle-cot13b 16K /workspace/model/saved-alpaca-belle-cot7b 16K /workspace/model/saved-alpaca-belle7b 16K /workspace/model/saved-alpaca-cot13b 16K /workspace/model/saved-alpaca-cot7b 17M /workspace/model/saved-alpaca7b 8.0K /workspace/model/saved-belle1.5m7b 16K /workspace/model/saved-belle7b 16K /workspace/model/saved-bloom7b_instinwild_cn-belle1.5m 16K /workspace/model/saved-cot7b 8.0K /workspace/model/saved_llama7b_codealpaca 33M /workspace/model/
In [7]:
!mkdir /workspace/merged
In [10]:
# parameter merging (optional)
# 合并模型参数，可选
!cd /workspace/code && python merge.py --model_type llama --size 7b --lora_dir "/workspace/model/saved-alpaca7b" --merged_dir "/workspace/merged"
===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please run
python -m bitsandbytes
and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================
bin /usr/local/lib/python3.10/dist-packages/bitsandbytes/libbitsandbytes_cuda116.so
/usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/main.py:145: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/usr/local/nvidia/lib64'), PosixPath('/usr/local/nvidia/lib')}
warn(msg)
/usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/main.py:145: UserWarning: /usr/local/nvidia/lib:/usr/local/nvidia/lib64 did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths...
warn(msg)
/usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/main.py:145: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('module'), PosixPath('//matplotlib_inline.backend_inline')}
warn(msg)
/usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/main.py:145: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('PygmalionAI/pygmalion-6b')}
warn(msg)
/usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/main.py:145: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('noninteractiveSHELL=/bin/bash')}
warn(msg)
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
/usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/main.py:145: UserWarning: Found duplicate ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] files: {PosixPath('/usr/local/cuda/lib64/libcudart.so.11.0'), PosixPath('/usr/local/cuda/lib64/libcudart.so')}.. We'll flip a coin and try one of these, in order to fail forward.
Either way, this might cause trouble in the future:
If you get `CUDA error: invalid device function` errors, the above might be the cause and the solution is to make sure only one ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] in the paths that we search based on your env.
warn(msg)
CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so.11.0
CUDA SETUP: Highest compute capability among GPUs detected: 8.6
CUDA SETUP: Detected CUDA version 116
CUDA SETUP: Loading binary /usr/local/lib/python3.10/dist-packages/bitsandbytes/libbitsandbytes_cuda116.so...
Loading checkpoint shards: 100%|████████████████| 33/33 [00:06<00:00, 5.10it/s]
Traceback (most recent call last):
File "/workspace/code/merge.py", line 34, in <module>
assert not torch.allclose(first_weight_old, first_weight)
AssertionError
In [13]:
!python -m pip install --force-reinstall peft
Collecting peft
Using cached peft-0.2.0-py3-none-any.whl (40 kB)
Collecting transformers
Using cached transformers-4.28.1-py3-none-any.whl (7.0 MB)
Collecting accelerate
Using cached accelerate-0.18.0-py3-none-any.whl (215 kB)
Collecting torch>=1.13.0
Downloading torch-2.0.0-cp310-cp310-manylinux1_x86_64.whl (619.9 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 619.9/619.9 MB 7.4 MB/s eta 0:00:0000:0100:01
Collecting pyyaml
Using cached PyYAML-6.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (682 kB)
Collecting packaging>=20.0
Using cached packaging-23.1-py3-none-any.whl (48 kB)
Collecting psutil
Downloading psutil-5.9.5-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (282 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 282.1/282.1 kB 13.8 MB/s eta 0:00:00
Collecting numpy>=1.17
Using cached numpy-1.24.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.3 MB)
Collecting nvidia-cuda-cupti-cu11==11.7.101
Downloading nvidia_cuda_cupti_cu11-11.7.101-py3-none-manylinux1_x86_64.whl (11.8 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.8/11.8 MB 30.7 MB/s eta 0:00:0000:0100:01
Collecting nvidia-cublas-cu11==11.10.3.66
Downloading nvidia_cublas_cu11-11.10.3.66-py3-none-manylinux1_x86_64.whl (317.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 317.1/317.1 MB 13.6 MB/s eta 0:00:0000:0100:01
Collecting triton==2.0.0
Downloading triton-2.0.0-1-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (63.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 63.3/63.3 MB 25.7 MB/s eta 0:00:0000:0100:01
Collecting nvidia-nccl-cu11==2.14.3
Downloading nvidia_nccl_cu11-2.14.3-py3-none-manylinux1_x86_64.whl (177.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 177.1/177.1 MB 19.0 MB/s eta 0:00:0000:0100:01
Collecting sympy
Downloading sympy-1.11.1-py3-none-any.whl (6.5 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.5/6.5 MB 31.3 MB/s eta 0:00:00a 0:00:01
Collecting nvidia-cuda-runtime-cu11==11.7.99
Downloading nvidia_cuda_runtime_cu11-11.7.99-py3-none-manylinux1_x86_64.whl (849 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 849.3/849.3 kB 34.4 MB/s eta 0:00:00
Collecting nvidia-cudnn-cu11==8.5.0.96
Downloading nvidia_cudnn_cu11-8.5.0.96-2-py3-none-manylinux1_x86_64.whl (557.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 557.1/557.1 MB 9.8 MB/s eta 0:00:0000:0100:01
Collecting nvidia-cusparse-cu11==11.7.4.91
Downloading nvidia_cusparse_cu11-11.7.4.91-py3-none-manylinux1_x86_64.whl (173.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 173.2/173.2 MB 28.8 MB/s eta 0:00:0000:0100:01
Collecting nvidia-nvtx-cu11==11.7.91
Downloading nvidia_nvtx_cu11-11.7.91-py3-none-manylinux1_x86_64.whl (98 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 98.6/98.6 kB 39.7 MB/s eta 0:00:00
Collecting typing-extensions
Using cached typing_extensions-4.5.0-py3-none-any.whl (27 kB)
Collecting nvidia-cuda-nvrtc-cu11==11.7.99
Downloading nvidia_cuda_nvrtc_cu11-11.7.99-2-py3-none-manylinux1_x86_64.whl (21.0 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 21.0/21.0 MB 81.1 MB/s eta 0:00:0000:0100:01
Collecting nvidia-curand-cu11==10.2.10.91
Downloading nvidia_curand_cu11-10.2.10.91-py3-none-manylinux1_x86_64.whl (54.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 54.6/54.6 MB 57.9 MB/s eta 0:00:0000:0100:01
Collecting nvidia-cufft-cu11==10.9.0.58
Downloading nvidia_cufft_cu11-10.9.0.58-py3-none-manylinux1_x86_64.whl (168.4 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 168.4/168.4 MB 27.5 MB/s eta 0:00:0000:0100:01
Collecting nvidia-cusolver-cu11==11.4.0.1
Downloading nvidia_cusolver_cu11-11.4.0.1-2-py3-none-manylinux1_x86_64.whl (102.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 102.6/102.6 MB 43.3 MB/s eta 0:00:0000:0100:01
Collecting networkx
Downloading networkx-3.1-py3-none-any.whl (2.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 97.5 MB/s eta 0:00:00
Collecting filelock
Downloading filelock-3.11.0-py3-none-any.whl (10.0 kB)
Collecting jinja2
Using cached Jinja2-3.1.2-py3-none-any.whl (133 kB)
Collecting wheel
Using cached wheel-0.40.0-py3-none-any.whl (64 kB)
Collecting setuptools
Using cached setuptools-67.6.1-py3-none-any.whl (1.1 MB)
Collecting lit
Downloading lit-16.0.1.tar.gz (137 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 137.9/137.9 kB 33.9 MB/s eta 0:00:00
Preparing metadata (setup.py) ... done
Collecting cmake
Using cached cmake-3.26.3-py2.py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (24.0 MB)
Collecting regex!=2019.12.17
Using cached regex-2023.3.23-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (769 kB)
Collecting huggingface-hub<1.0,>=0.11.0
Downloading huggingface_hub-0.13.4-py3-none-any.whl (200 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 200.1/200.1 kB 54.6 MB/s eta 0:00:00
Collecting requests
Using cached requests-2.28.2-py3-none-any.whl (62 kB)
Collecting tokenizers!=0.11.3,<0.14,>=0.11.1
Downloading tokenizers-0.13.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.8 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.8/7.8 MB 95.7 MB/s eta 0:00:00:00:0100:01
Collecting tqdm>=4.27
Using cached tqdm-4.65.0-py3-none-any.whl (77 kB)
Collecting MarkupSafe>=2.0
Using cached MarkupSafe-2.1.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)
Collecting charset-normalizer<4,>=2
Downloading charset_normalizer-3.1.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (199 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 199.3/199.3 kB 55.2 MB/s eta 0:00:00
Collecting certifi>=2017.4.17
Downloading certifi-2022.12.7-py3-none-any.whl (155 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 155.3/155.3 kB 47.6 MB/s eta 0:00:00
Collecting urllib3<1.27,>=1.21.1
Using cached urllib3-1.26.15-py2.py3-none-any.whl (140 kB)
Collecting idna<4,>=2.5
Downloading idna-3.4-py3-none-any.whl (61 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.5/61.5 kB 17.5 MB/s eta 0:00:00
Collecting mpmath>=0.19
Downloading mpmath-1.3.0-py3-none-any.whl (536 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 536.2/536.2 kB 85.8 MB/s eta 0:00:00
Building wheels for collected packages: lit
Building wheel for lit (setup.py) ... done
Created wheel for lit: filename=lit-16.0.1-py3-none-any.whl size=88174 sha256=cc1135c6e57cebc7cf25c064c9ae27408542f841062baa5af84d1ffd616e4a33
Stored in directory: /root/.cache/pip/wheels/33/c2/b7/b91592eb5167b4293827b97fb02d52686773dded13f7fb1054
Successfully built lit
Installing collected packages: tokenizers, mpmath, lit, cmake, wheel, urllib3, typing-extensions, tqdm, sympy, setuptools, regex, pyyaml, psutil, packaging, nvidia-nccl-cu11, nvidia-cufft-cu11, nvidia-cuda-nvrtc-cu11, numpy, networkx, MarkupSafe, idna, filelock, charset-normalizer, certifi, requests, nvidia-nvtx-cu11, nvidia-cusparse-cu11, nvidia-curand-cu11, nvidia-cuda-runtime-cu11, nvidia-cuda-cupti-cu11, nvidia-cublas-cu11, jinja2, nvidia-cusolver-cu11, nvidia-cudnn-cu11, huggingface-hub, transformers, triton, torch, accelerate, peft
Attempting uninstall: tokenizers
Found existing installation: tokenizers 0.13.2
Uninstalling tokenizers-0.13.2:
Successfully uninstalled tokenizers-0.13.2
Attempting uninstall: wheel
Found existing installation: wheel 0.38.4
Uninstalling wheel-0.38.4:
Successfully uninstalled wheel-0.38.4
Attempting uninstall: urllib3
Found existing installation: urllib3 1.26.15
Uninstalling urllib3-1.26.15:
Successfully uninstalled urllib3-1.26.15
Attempting uninstall: typing-extensions
Found existing installation: typing_extensions 4.5.0
Uninstalling typing_extensions-4.5.0:
Successfully uninstalled typing_extensions-4.5.0
Attempting uninstall: tqdm
Found existing installation: tqdm 4.65.0
Uninstalling tqdm-4.65.0:
Successfully uninstalled tqdm-4.65.0
Attempting uninstall: setuptools
Found existing installation: setuptools 67.3.2
Uninstalling setuptools-67.3.2:
Successfully uninstalled setuptools-67.3.2
Attempting uninstall: regex
Found existing installation: regex 2023.3.23
Uninstalling regex-2023.3.23:
Successfully uninstalled regex-2023.3.23
Attempting uninstall: pyyaml
Found existing installation: PyYAML 6.0
Uninstalling PyYAML-6.0:
Successfully uninstalled PyYAML-6.0
Attempting uninstall: psutil
Found existing installation: psutil 5.9.4
Uninstalling psutil-5.9.4:
Successfully uninstalled psutil-5.9.4
Attempting uninstall: packaging
Found existing installation: packaging 23.0
Uninstalling packaging-23.0:
Successfully uninstalled packaging-23.0
Attempting uninstall: numpy
Found existing installation: numpy 1.24.2
Uninstalling numpy-1.24.2:
Successfully uninstalled numpy-1.24.2
Attempting uninstall: MarkupSafe
Found existing installation: MarkupSafe 2.1.2
Uninstalling MarkupSafe-2.1.2:
Successfully uninstalled MarkupSafe-2.1.2
Attempting uninstall: idna
Found existing installation: idna 2.8
Uninstalling idna-2.8:
Successfully uninstalled idna-2.8
Attempting uninstall: filelock
Found existing installation: filelock 3.10.4
Uninstalling filelock-3.10.4:
Successfully uninstalled filelock-3.10.4
Attempting uninstall: charset-normalizer
Found existing installation: charset-normalizer 3.0.1
Uninstalling charset-normalizer-3.0.1:
Successfully uninstalled charset-normalizer-3.0.1
Attempting uninstall: certifi
Found existing installation: certifi 2019.11.28
Uninstalling certifi-2019.11.28:
Successfully uninstalled certifi-2019.11.28
Attempting uninstall: requests
Found existing installation: requests 2.28.2
Uninstalling requests-2.28.2:
Successfully uninstalled requests-2.28.2
Attempting uninstall: jinja2
Found existing installation: Jinja2 3.1.2
Uninstalling Jinja2-3.1.2:
Successfully uninstalled Jinja2-3.1.2
Attempting uninstall: huggingface-hub
Found existing installation: huggingface-hub 0.13.3
Uninstalling huggingface-hub-0.13.3:
Successfully uninstalled huggingface-hub-0.13.3
Attempting uninstall: transformers
Found existing installation: transformers 4.29.0.dev0
Uninstalling transformers-4.29.0.dev0:
Successfully uninstalled transformers-4.29.0.dev0
Attempting uninstall: torch
Found existing installation: torch 1.13.1+cu116
Uninstalling torch-1.13.1+cu116:
Successfully uninstalled torch-1.13.1+cu116
Attempting uninstall: accelerate
Found existing installation: accelerate 0.18.0
Uninstalling accelerate-0.18.0:
Successfully uninstalled accelerate-0.18.0
Attempting uninstall: peft
Found existing installation: peft 0.3.0.dev0
Uninstalling peft-0.3.0.dev0:
Successfully uninstalled peft-0.3.0.dev0
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
torchvision 0.14.1+cu116 requires torch==1.13.1, but you have torch 2.0.0 which is incompatible.
torchaudio 0.13.1+cu116 requires torch==1.13.1, but you have torch 2.0.0 which is incompatible.
Successfully installed MarkupSafe-2.1.2 accelerate-0.18.0 certifi-2022.12.7 charset-normalizer-3.1.0 cmake-3.26.3 filelock-3.11.0 huggingface-hub-0.13.4 idna-3.4 jinja2-3.1.2 lit-16.0.1 mpmath-1.3.0 networkx-3.1 numpy-1.24.2 nvidia-cublas-cu11-11.10.3.66 nvidia-cuda-cupti-cu11-11.7.101 nvidia-cuda-nvrtc-cu11-11.7.99 nvidia-cuda-runtime-cu11-11.7.99 nvidia-cudnn-cu11-8.5.0.96 nvidia-cufft-cu11-10.9.0.58 nvidia-curand-cu11-10.2.10.91 nvidia-cusolver-cu11-11.4.0.1 nvidia-cusparse-cu11-11.7.4.91 nvidia-nccl-cu11-2.14.3 nvidia-nvtx-cu11-11.7.91 packaging-23.1 peft-0.2.0 psutil-5.9.5 pyyaml-6.0 regex-2023.3.23 requests-2.28.2 setuptools-67.6.1 sympy-1.11.1 tokenizers-0.13.3 torch-2.0.0 tqdm-4.65.0 transformers-4.28.1 triton-2.0.0 typing-extensions-4.5.0 urllib3-1.26.15 wheel-0.40.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
[notice] A new release of pip is available: 23.0.1 -> 23.1
[notice] To update, run: python -m pip install --upgrade pip
In [26]:
!cd /workspace/code && python merge.py --model_type llama --size 7b --lora_dir "/workspace/model/saved-alpaca7b" --merged_dir "/workspace/merged"
===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please run
python -m bitsandbytes
and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================
bin /usr/local/lib/python3.10/dist-packages/bitsandbytes/libbitsandbytes_cuda116.so
/usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/main.py:145: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/usr/local/nvidia/lib64'), PosixPath('/usr/local/nvidia/lib')}
warn(msg)
/usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/main.py:145: UserWarning: /usr/local/nvidia/lib:/usr/local/nvidia/lib64 did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths...
warn(msg)
/usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/main.py:145: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//matplotlib_inline.backend_inline'), PosixPath('module')}
warn(msg)
/usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/main.py:145: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('PygmalionAI/pygmalion-6b')}
warn(msg)
/usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/main.py:145: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('noninteractiveSHELL=/bin/bash')}
warn(msg)
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
/usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/main.py:145: UserWarning: Found duplicate ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] files: {PosixPath('/usr/local/cuda/lib64/libcudart.so'), PosixPath('/usr/local/cuda/lib64/libcudart.so.11.0')}.. We'll flip a coin and try one of these, in order to fail forward.
Either way, this might cause trouble in the future:
If you get `CUDA error: invalid device function` errors, the above might be the cause and the solution is to make sure only one ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] in the paths that we search based on your env.
warn(msg)
CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so
CUDA SETUP: Highest compute capability among GPUs detected: 8.6
CUDA SETUP: Detected CUDA version 116
CUDA SETUP: Loading binary /usr/local/lib/python3.10/dist-packages/bitsandbytes/libbitsandbytes_cuda116.so...
Loading checkpoint shards: 100%|████████████████| 33/33 [00:07<00:00, 4.50it/s]
In [29]:
!ls merged
config.json pytorch_model-00020-of-00039.bin generation_config.json pytorch_model-00021-of-00039.bin pytorch_model-00001-of-00039.bin pytorch_model-00022-of-00039.bin pytorch_model-00002-of-00039.bin pytorch_model-00023-of-00039.bin pytorch_model-00003-of-00039.bin pytorch_model-00024-of-00039.bin pytorch_model-00004-of-00039.bin pytorch_model-00025-of-00039.bin pytorch_model-00005-of-00039.bin pytorch_model-00026-of-00039.bin pytorch_model-00006-of-00039.bin pytorch_model-00027-of-00039.bin pytorch_model-00007-of-00039.bin pytorch_model-00028-of-00039.bin pytorch_model-00008-of-00039.bin pytorch_model-00029-of-00039.bin pytorch_model-00009-of-00039.bin pytorch_model-00030-of-00039.bin pytorch_model-00010-of-00039.bin pytorch_model-00031-of-00039.bin pytorch_model-00011-of-00039.bin pytorch_model-00032-of-00039.bin pytorch_model-00012-of-00039.bin pytorch_model-00033-of-00039.bin pytorch_model-00013-of-00039.bin pytorch_model-00034-of-00039.bin pytorch_model-00014-of-00039.bin pytorch_model-00035-of-00039.bin pytorch_model-00015-of-00039.bin pytorch_model-00036-of-00039.bin pytorch_model-00016-of-00039.bin pytorch_model-00037-of-00039.bin pytorch_model-00017-of-00039.bin pytorch_model-00038-of-00039.bin pytorch_model-00018-of-00039.bin pytorch_model-00039-of-00039.bin pytorch_model-00019-of-00039.bin pytorch_model.bin.index.json
In [ ]: