Prepare dataset for Finetuning falcon-40b¶
- Axolotl+QLoRA
- minotaur datasets
- deepspeed ZeRO 3 8xGPU
In [ ]:
!git clone https://github.com/OpenAccess-AI-Collective/axolotl.git
Cloning into 'axolotl'... remote: Enumerating objects: 2985, done. remote: Counting objects: 100% (1179/1179), done. remote: Compressing objects: 100% (294/294), done. remote: Total 2985 (delta 976), reused 978 (delta 849), pack-reused 1806 Receiving objects: 100% (2985/2985), 1.39 MiB | 7.16 MiB/s, done. Resolving deltas: 100% (1866/1866), done.
In [ ]:
%cd /workspace/axolotl
/workspace/axolotl
In [ ]:
!pip install -e .
!pip install -U git+https://github.com/huggingface/peft.git
Obtaining file:///workspace/axolotl
Preparing metadata (setup.py) ... done
Collecting transformers@ git+https://github.com/huggingface/transformers.git (from axolotl==0.1)
Cloning https://github.com/huggingface/transformers.git to /tmp/pip-install-6qt0ze0g/transformers_622ada43523a4cc595d97ee326b89ec2
Running command git clone --filter=blob:none --quiet https://github.com/huggingface/transformers.git /tmp/pip-install-6qt0ze0g/transformers_622ada43523a4cc595d97ee326b89ec2
Resolved https://github.com/huggingface/transformers.git to commit dadc9fb4275f4b7c2984d16d6d9a7880ec76d872
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: bitsandbytes>=0.39.0 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages/bitsandbytes-0.39.0-py3.9.egg (from axolotl==0.1) (0.39.0)
Requirement already satisfied: accelerate in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from axolotl==0.1) (0.21.0.dev0)
Requirement already satisfied: addict in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from axolotl==0.1) (2.4.0)
Requirement already satisfied: fire in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from axolotl==0.1) (0.5.0)
Requirement already satisfied: PyYAML==6.0 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from axolotl==0.1) (6.0)
Requirement already satisfied: datasets in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from axolotl==0.1) (2.12.0)
Requirement already satisfied: sentencepiece in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from axolotl==0.1) (0.1.99)
Requirement already satisfied: wandb in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from axolotl==0.1) (0.15.3)
Requirement already satisfied: einops in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from axolotl==0.1) (0.6.1)
Requirement already satisfied: xformers in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from axolotl==0.1) (0.0.20)
Requirement already satisfied: bert-score==0.3.13 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from axolotl==0.1) (0.3.13)
Requirement already satisfied: evaluate==0.4.0 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from axolotl==0.1) (0.4.0)
Requirement already satisfied: rouge-score==0.1.2 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from axolotl==0.1) (0.1.2)
Requirement already satisfied: scipy in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from axolotl==0.1) (1.10.1)
Requirement already satisfied: scikit-learn==1.2.2 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from axolotl==0.1) (1.2.2)
Requirement already satisfied: torch>=1.0.0 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from bert-score==0.3.13->axolotl==0.1) (2.0.1)
Requirement already satisfied: pandas>=1.0.1 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from bert-score==0.3.13->axolotl==0.1) (2.0.2)
Requirement already satisfied: numpy in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from bert-score==0.3.13->axolotl==0.1) (1.24.3)
Requirement already satisfied: requests in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from bert-score==0.3.13->axolotl==0.1) (2.31.0)
Requirement already satisfied: tqdm>=4.31.1 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from bert-score==0.3.13->axolotl==0.1) (4.65.0)
Requirement already satisfied: matplotlib in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from bert-score==0.3.13->axolotl==0.1) (3.7.1)
Requirement already satisfied: packaging>=20.9 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from bert-score==0.3.13->axolotl==0.1) (23.1)
Requirement already satisfied: dill in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from evaluate==0.4.0->axolotl==0.1) (0.3.6)
Requirement already satisfied: xxhash in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from evaluate==0.4.0->axolotl==0.1) (3.2.0)
Requirement already satisfied: multiprocess in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from evaluate==0.4.0->axolotl==0.1) (0.70.14)
Requirement already satisfied: fsspec[http]>=2021.05.0 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from evaluate==0.4.0->axolotl==0.1) (2023.5.0)
Requirement already satisfied: huggingface-hub>=0.7.0 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from evaluate==0.4.0->axolotl==0.1) (0.14.1)
Requirement already satisfied: responses<0.19 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from evaluate==0.4.0->axolotl==0.1) (0.18.0)
Requirement already satisfied: absl-py in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from rouge-score==0.1.2->axolotl==0.1) (1.4.0)
Requirement already satisfied: nltk in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from rouge-score==0.1.2->axolotl==0.1) (3.8.1)
Requirement already satisfied: six>=1.14.0 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from rouge-score==0.1.2->axolotl==0.1) (1.16.0)
Requirement already satisfied: joblib>=1.1.1 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from scikit-learn==1.2.2->axolotl==0.1) (1.2.0)
Requirement already satisfied: threadpoolctl>=2.0.0 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from scikit-learn==1.2.2->axolotl==0.1) (3.1.0)
Requirement already satisfied: psutil in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from accelerate->axolotl==0.1) (5.9.5)
Requirement already satisfied: pyarrow>=8.0.0 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from datasets->axolotl==0.1) (12.0.0)
Requirement already satisfied: aiohttp in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from datasets->axolotl==0.1) (3.8.4)
Requirement already satisfied: filelock in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from transformers@ git+https://github.com/huggingface/transformers.git->axolotl==0.1) (3.12.0)
Requirement already satisfied: regex!=2019.12.17 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from transformers@ git+https://github.com/huggingface/transformers.git->axolotl==0.1) (2023.5.5)
Requirement already satisfied: tokenizers!=0.11.3,<0.14,>=0.11.1 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from transformers@ git+https://github.com/huggingface/transformers.git->axolotl==0.1) (0.13.3)
Requirement already satisfied: safetensors>=0.3.1 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from transformers@ git+https://github.com/huggingface/transformers.git->axolotl==0.1) (0.3.1)
Requirement already satisfied: termcolor in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from fire->axolotl==0.1) (2.3.0)
Requirement already satisfied: Click!=8.0.0,>=7.0 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from wandb->axolotl==0.1) (8.1.3)
Requirement already satisfied: GitPython!=3.1.29,>=1.0.0 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from wandb->axolotl==0.1) (3.1.31)
Requirement already satisfied: sentry-sdk>=1.0.0 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from wandb->axolotl==0.1) (1.24.0)
Requirement already satisfied: docker-pycreds>=0.4.0 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from wandb->axolotl==0.1) (0.4.0)
Requirement already satisfied: pathtools in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from wandb->axolotl==0.1) (0.1.2)
Requirement already satisfied: setproctitle in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from wandb->axolotl==0.1) (1.3.2)
Requirement already satisfied: setuptools in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from wandb->axolotl==0.1) (67.8.0)
Requirement already satisfied: appdirs>=1.4.3 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from wandb->axolotl==0.1) (1.4.4)
Requirement already satisfied: typing-extensions in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from wandb->axolotl==0.1) (4.6.2)
Requirement already satisfied: protobuf!=4.21.0,<5,>=3.15.0 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from wandb->axolotl==0.1) (4.23.2)
Requirement already satisfied: pyre-extensions==0.0.29 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from xformers->axolotl==0.1) (0.0.29)
Requirement already satisfied: typing-inspect in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from pyre-extensions==0.0.29->xformers->axolotl==0.1) (0.9.0)
Requirement already satisfied: sympy in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from torch>=1.0.0->bert-score==0.3.13->axolotl==0.1) (1.12)
Requirement already satisfied: networkx in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from torch>=1.0.0->bert-score==0.3.13->axolotl==0.1) (3.1)
Requirement already satisfied: jinja2 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from torch>=1.0.0->bert-score==0.3.13->axolotl==0.1) (3.1.2)
Requirement already satisfied: nvidia-cuda-nvrtc-cu11==11.7.99 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from torch>=1.0.0->bert-score==0.3.13->axolotl==0.1) (11.7.99)
Requirement already satisfied: nvidia-cuda-runtime-cu11==11.7.99 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from torch>=1.0.0->bert-score==0.3.13->axolotl==0.1) (11.7.99)
Requirement already satisfied: nvidia-cuda-cupti-cu11==11.7.101 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from torch>=1.0.0->bert-score==0.3.13->axolotl==0.1) (11.7.101)
Requirement already satisfied: nvidia-cudnn-cu11==8.5.0.96 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from torch>=1.0.0->bert-score==0.3.13->axolotl==0.1) (8.5.0.96)
Requirement already satisfied: nvidia-cublas-cu11==11.10.3.66 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from torch>=1.0.0->bert-score==0.3.13->axolotl==0.1) (11.10.3.66)
Requirement already satisfied: nvidia-cufft-cu11==10.9.0.58 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from torch>=1.0.0->bert-score==0.3.13->axolotl==0.1) (10.9.0.58)
Requirement already satisfied: nvidia-curand-cu11==10.2.10.91 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from torch>=1.0.0->bert-score==0.3.13->axolotl==0.1) (10.2.10.91)
Requirement already satisfied: nvidia-cusolver-cu11==11.4.0.1 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from torch>=1.0.0->bert-score==0.3.13->axolotl==0.1) (11.4.0.1)
Requirement already satisfied: nvidia-cusparse-cu11==11.7.4.91 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from torch>=1.0.0->bert-score==0.3.13->axolotl==0.1) (11.7.4.91)
Requirement already satisfied: nvidia-nccl-cu11==2.14.3 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from torch>=1.0.0->bert-score==0.3.13->axolotl==0.1) (2.14.3)
Requirement already satisfied: nvidia-nvtx-cu11==11.7.91 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from torch>=1.0.0->bert-score==0.3.13->axolotl==0.1) (11.7.91)
Requirement already satisfied: triton==2.0.0 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from torch>=1.0.0->bert-score==0.3.13->axolotl==0.1) (2.0.0)
Requirement already satisfied: wheel in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from nvidia-cublas-cu11==11.10.3.66->torch>=1.0.0->bert-score==0.3.13->axolotl==0.1) (0.40.0)
Requirement already satisfied: cmake in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from triton==2.0.0->torch>=1.0.0->bert-score==0.3.13->axolotl==0.1) (3.26.3)
Requirement already satisfied: lit in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from triton==2.0.0->torch>=1.0.0->bert-score==0.3.13->axolotl==0.1) (16.0.5)
Requirement already satisfied: attrs>=17.3.0 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from aiohttp->datasets->axolotl==0.1) (23.1.0)
Requirement already satisfied: charset-normalizer<4.0,>=2.0 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from aiohttp->datasets->axolotl==0.1) (3.1.0)
Requirement already satisfied: multidict<7.0,>=4.5 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from aiohttp->datasets->axolotl==0.1) (6.0.4)
Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from aiohttp->datasets->axolotl==0.1) (4.0.2)
Requirement already satisfied: yarl<2.0,>=1.0 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from aiohttp->datasets->axolotl==0.1) (1.9.2)
Requirement already satisfied: frozenlist>=1.1.1 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from aiohttp->datasets->axolotl==0.1) (1.3.3)
Requirement already satisfied: aiosignal>=1.1.2 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from aiohttp->datasets->axolotl==0.1) (1.3.1)
Requirement already satisfied: gitdb<5,>=4.0.1 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from GitPython!=3.1.29,>=1.0.0->wandb->axolotl==0.1) (4.0.10)
Requirement already satisfied: python-dateutil>=2.8.2 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from pandas>=1.0.1->bert-score==0.3.13->axolotl==0.1) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from pandas>=1.0.1->bert-score==0.3.13->axolotl==0.1) (2023.3)
Requirement already satisfied: tzdata>=2022.1 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from pandas>=1.0.1->bert-score==0.3.13->axolotl==0.1) (2023.3)
Requirement already satisfied: idna<4,>=2.5 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from requests->bert-score==0.3.13->axolotl==0.1) (3.4)
Requirement already satisfied: urllib3<3,>=1.21.1 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from requests->bert-score==0.3.13->axolotl==0.1) (1.26.16)
Requirement already satisfied: certifi>=2017.4.17 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from requests->bert-score==0.3.13->axolotl==0.1) (2023.5.7)
Requirement already satisfied: contourpy>=1.0.1 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from matplotlib->bert-score==0.3.13->axolotl==0.1) (1.0.7)
Requirement already satisfied: cycler>=0.10 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from matplotlib->bert-score==0.3.13->axolotl==0.1) (0.11.0)
Requirement already satisfied: fonttools>=4.22.0 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from matplotlib->bert-score==0.3.13->axolotl==0.1) (4.39.4)
Requirement already satisfied: kiwisolver>=1.0.1 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from matplotlib->bert-score==0.3.13->axolotl==0.1) (1.4.4)
Requirement already satisfied: pillow>=6.2.0 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from matplotlib->bert-score==0.3.13->axolotl==0.1) (9.5.0)
Requirement already satisfied: pyparsing>=2.3.1 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from matplotlib->bert-score==0.3.13->axolotl==0.1) (3.0.9)
Requirement already satisfied: importlib-resources>=3.2.0 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from matplotlib->bert-score==0.3.13->axolotl==0.1) (5.12.0)
Requirement already satisfied: smmap<6,>=3.0.1 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from gitdb<5,>=4.0.1->GitPython!=3.1.29,>=1.0.0->wandb->axolotl==0.1) (5.0.0)
Requirement already satisfied: zipp>=3.1.0 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from importlib-resources>=3.2.0->matplotlib->bert-score==0.3.13->axolotl==0.1) (3.15.0)
Requirement already satisfied: MarkupSafe>=2.0 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from jinja2->torch>=1.0.0->bert-score==0.3.13->axolotl==0.1) (2.1.2)
Requirement already satisfied: mpmath>=0.19 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from sympy->torch>=1.0.0->bert-score==0.3.13->axolotl==0.1) (1.3.0)
Requirement already satisfied: mypy-extensions>=0.3.0 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from typing-inspect->pyre-extensions==0.0.29->xformers->axolotl==0.1) (1.0.0)
Installing collected packages: axolotl
Attempting uninstall: axolotl
Found existing installation: axolotl 0.1
Uninstalling axolotl-0.1:
Successfully uninstalled axolotl-0.1
Running setup.py develop for axolotl
Successfully installed axolotl-0.1
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Collecting git+https://github.com/huggingface/peft.git
Cloning https://github.com/huggingface/peft.git to /tmp/pip-req-build-nykue8ep
Running command git clone --filter=blob:none --quiet https://github.com/huggingface/peft.git /tmp/pip-req-build-nykue8ep
Resolved https://github.com/huggingface/peft.git to commit 189a6b8e357ecda05ccde13999e4c35759596a67
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: numpy>=1.17 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from peft==0.4.0.dev0) (1.24.3)
Requirement already satisfied: packaging>=20.0 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from peft==0.4.0.dev0) (23.1)
Requirement already satisfied: psutil in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from peft==0.4.0.dev0) (5.9.5)
Requirement already satisfied: pyyaml in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from peft==0.4.0.dev0) (6.0)
Requirement already satisfied: torch>=1.13.0 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from peft==0.4.0.dev0) (2.0.1)
Requirement already satisfied: transformers in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from peft==0.4.0.dev0) (4.31.0.dev0)
Requirement already satisfied: accelerate in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from peft==0.4.0.dev0) (0.21.0.dev0)
Requirement already satisfied: safetensors in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from peft==0.4.0.dev0) (0.3.1)
Requirement already satisfied: filelock in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from torch>=1.13.0->peft==0.4.0.dev0) (3.12.0)
Requirement already satisfied: typing-extensions in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from torch>=1.13.0->peft==0.4.0.dev0) (4.6.2)
Requirement already satisfied: sympy in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from torch>=1.13.0->peft==0.4.0.dev0) (1.12)
Requirement already satisfied: networkx in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from torch>=1.13.0->peft==0.4.0.dev0) (3.1)
Requirement already satisfied: jinja2 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from torch>=1.13.0->peft==0.4.0.dev0) (3.1.2)
Requirement already satisfied: nvidia-cuda-nvrtc-cu11==11.7.99 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from torch>=1.13.0->peft==0.4.0.dev0) (11.7.99)
Requirement already satisfied: nvidia-cuda-runtime-cu11==11.7.99 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from torch>=1.13.0->peft==0.4.0.dev0) (11.7.99)
Requirement already satisfied: nvidia-cuda-cupti-cu11==11.7.101 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from torch>=1.13.0->peft==0.4.0.dev0) (11.7.101)
Requirement already satisfied: nvidia-cudnn-cu11==8.5.0.96 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from torch>=1.13.0->peft==0.4.0.dev0) (8.5.0.96)
Requirement already satisfied: nvidia-cublas-cu11==11.10.3.66 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from torch>=1.13.0->peft==0.4.0.dev0) (11.10.3.66)
Requirement already satisfied: nvidia-cufft-cu11==10.9.0.58 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from torch>=1.13.0->peft==0.4.0.dev0) (10.9.0.58)
Requirement already satisfied: nvidia-curand-cu11==10.2.10.91 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from torch>=1.13.0->peft==0.4.0.dev0) (10.2.10.91)
Requirement already satisfied: nvidia-cusolver-cu11==11.4.0.1 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from torch>=1.13.0->peft==0.4.0.dev0) (11.4.0.1)
Requirement already satisfied: nvidia-cusparse-cu11==11.7.4.91 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from torch>=1.13.0->peft==0.4.0.dev0) (11.7.4.91)
Requirement already satisfied: nvidia-nccl-cu11==2.14.3 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from torch>=1.13.0->peft==0.4.0.dev0) (2.14.3)
Requirement already satisfied: nvidia-nvtx-cu11==11.7.91 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from torch>=1.13.0->peft==0.4.0.dev0) (11.7.91)
Requirement already satisfied: triton==2.0.0 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from torch>=1.13.0->peft==0.4.0.dev0) (2.0.0)
Requirement already satisfied: setuptools in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from nvidia-cublas-cu11==11.10.3.66->torch>=1.13.0->peft==0.4.0.dev0) (67.8.0)
Requirement already satisfied: wheel in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from nvidia-cublas-cu11==11.10.3.66->torch>=1.13.0->peft==0.4.0.dev0) (0.40.0)
Requirement already satisfied: cmake in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from triton==2.0.0->torch>=1.13.0->peft==0.4.0.dev0) (3.26.3)
Requirement already satisfied: lit in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from triton==2.0.0->torch>=1.13.0->peft==0.4.0.dev0) (16.0.5)
Requirement already satisfied: huggingface-hub<1.0,>=0.14.1 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from transformers->peft==0.4.0.dev0) (0.14.1)
Requirement already satisfied: regex!=2019.12.17 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from transformers->peft==0.4.0.dev0) (2023.5.5)
Requirement already satisfied: requests in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from transformers->peft==0.4.0.dev0) (2.31.0)
Requirement already satisfied: tokenizers!=0.11.3,<0.14,>=0.11.1 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from transformers->peft==0.4.0.dev0) (0.13.3)
Requirement already satisfied: tqdm>=4.27 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from transformers->peft==0.4.0.dev0) (4.65.0)
Requirement already satisfied: fsspec in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from huggingface-hub<1.0,>=0.14.1->transformers->peft==0.4.0.dev0) (2023.5.0)
Requirement already satisfied: MarkupSafe>=2.0 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from jinja2->torch>=1.13.0->peft==0.4.0.dev0) (2.1.2)
Requirement already satisfied: charset-normalizer<4,>=2 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from requests->transformers->peft==0.4.0.dev0) (3.1.0)
Requirement already satisfied: idna<4,>=2.5 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from requests->transformers->peft==0.4.0.dev0) (3.4)
Requirement already satisfied: urllib3<3,>=1.21.1 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from requests->transformers->peft==0.4.0.dev0) (1.26.16)
Requirement already satisfied: certifi>=2017.4.17 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from requests->transformers->peft==0.4.0.dev0) (2023.5.7)
Requirement already satisfied: mpmath>=0.19 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from sympy->torch>=1.13.0->peft==0.4.0.dev0) (1.3.0)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
In [ ]:
from huggingface_hub import notebook_login
notebook_login()
VBox(children=(HTML(value='<center> <img\nsrc=https://huggingface.co/front/assets/huggingface_logo-noborder.sv…
gsm8k (on Colab)¶
In [ ]:
%%writefile examples/falcon/config-40b-qlora.yml
# 1b: tiiuae/falcon-rw-1b
# 7b: tiiuae/falcon-7b
# 40b: tiiuae/falcon-40b
base_model: tiiuae/falcon-40b
base_model_config: tiiuae/falcon-40b
# required by falcon custom model code: https://huggingface.co/tiiuae/falcon-7b/tree/main
trust_remote_code: true
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer
load_in_8bit: false
# enable 4bit for QLoRA
load_in_4bit: true
gptq: false
strict: false
push_dataset_to_hub: utensil
hf_use_auth_token: true
datasets:
- path: QingyiSi/Alpaca-CoT
data_files:
- Chain-of-Thought/formatted_cot_data/gsm8k_train.json
type: "alpaca:chat"
dataset_prepared_path: last_run_prepared
val_set_size: 0.01
# enable QLoRA
adapter: qlora
lora_model_dir:
sequence_len: 2048
max_packed_sequence_len:
# hyperparameters from QLoRA paper Appendix B.2
# "We find hyperparameters to be largely robust across datasets"
lora_r: 64
lora_alpha: 16
# 0.1 for models up to 13B
# 0.05 for 33B and 65B models
lora_dropout: 0.05
# add LoRA modules on all linear layers of the base model
lora_target_modules:
lora_target_linear: true
lora_fan_in_fan_out:
wandb_project: falcon-qlora
wandb_watch:
wandb_run_id:
wandb_log_model:
output_dir: /workspace/llm-playground/axolotl-trained/falcon-qlora-40b-gsm8k/
# QLoRA paper Table 9
# - 16 for 7b & 13b
# - 32 for 33b, 64 for 64b
# Max size tested on A6000
# - 7b: 40
# - 40b: 4
# decrease if OOM, increase for max VRAM utilization
micro_batch_size: 1
gradient_accumulation_steps: 2
num_epochs: 3
# Optimizer for QLoRA
optimizer: paged_adamw_32bit
torchdistx_path:
lr_scheduler: cosine
# QLoRA paper Table 9
# - 2e-4 for 7b & 13b
# - 1e-4 for 33b & 64b
learning_rate: 0.0002
train_on_inputs: false
group_by_length: false
bf16: true
fp16: false
tf32: true
gradient_checkpointing: true
# stop training after this many evaluation losses have increased in a row
# https://huggingface.co/transformers/v4.2.2/_modules/transformers/trainer_callback.html#EarlyStoppingCallback
early_stopping_patience: 3
resume_from_checkpoint:
auto_resume_from_checkpoints: true
local_rank:
logging_steps: 1
xformers_attention: true
flash_attention:
gptq_groupsize:
gptq_model_v1:
warmup_steps: 10
eval_steps: 5
save_steps: 10
debug:
deepspeed:
weight_decay: 0.01
fsdp:
fsdp_config:
special_tokens:
pad_token: "<|endoftext|>"
bos_token: ">>ABSTRACT<<"
eos_token: "<|endoftext|>"
Overwriting examples/falcon/config-40b-qlora.yml
In [ ]:
!python scripts/finetune.py examples/falcon/config-40b-qlora.yml -prepare_ds_only
===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please run
python -m bitsandbytes
and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================
bin /usr/local/lib/python3.10/dist-packages/bitsandbytes/libbitsandbytes_cuda118.so
/usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: /usr/lib64-nvidia did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths...
warn(msg)
/usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/sys/fs/cgroup/memory.events /var/colab/cgroup/jupyter-children/memory.events')}
warn(msg)
/usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('http'), PosixPath('//172.28.0.1'), PosixPath('8013')}
warn(msg)
/usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('--logtostderr --listen_host=172.28.0.12 --target_host=172.28.0.12 --tunnel_background_save_url=https'), PosixPath('//colab.research.google.com/tun/m/cc48301118ce562b961b3c22d803539adc1e0c19/gpu-t4-s-3qu49qpehlmmz --tunnel_background_save_delay=10s --tunnel_periodic_background_save_frequency=30m0s --enable_output_coalescing=true --output_coalescing_required=true')}
warn(msg)
/usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/env/python')}
warn(msg)
/usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//ipykernel.pylab.backend_inline'), PosixPath('module')}
warn(msg)
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
/usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: Found duplicate ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] files: {PosixPath('/usr/local/cuda/lib64/libcudart.so.11.0'), PosixPath('/usr/local/cuda/lib64/libcudart.so')}.. We'll flip a coin and try one of these, in order to fail forward.
Either way, this might cause trouble in the future:
If you get `CUDA error: invalid device function` errors, the above might be the cause and the solution is to make sure only one ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] in the paths that we search based on your env.
warn(msg)
CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so.11.0
CUDA SETUP: Highest compute capability among GPUs detected: 7.5
CUDA SETUP: Detected CUDA version 118
CUDA SETUP: Loading binary /usr/local/lib/python3.10/dist-packages/bitsandbytes/libbitsandbytes_cuda118.so...
2023-06-12 06:45:03.959717: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
WARNING:root:`trust_remote_code` is set to true. Please make sure that you reviewed the remote code/model.
INFO:root:loading tokenizer... tiiuae/falcon-40b
Using bos_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using unk_token, but it is not set yet.
INFO:root:Unable to find prepared dataset in last_run_prepared/31a4e867d804a957707db033c9abcd13
INFO:root:Loading raw datasets...
INFO:root:No seed provided, using default seed of 42
WARNING:datasets.builder:Found cached dataset json (/root/.cache/huggingface/datasets/QingyiSi___json/QingyiSi--Alpaca-CoT-2953efcfeb19f105/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4)
100% 1/1 [00:00<00:00, 748.85it/s]
INFO:root:tokenizing, merging, and shuffling master dataset
INFO:root:Saving merged prepared dataset to disk... last_run_prepared/31a4e867d804a957707db033c9abcd13
INFO:root:Saving merged prepared dataset with push_to_hub... utensil/31a4e867d804a957707db033c9abcd13
Pushing dataset shards to the dataset hub: 0% 0/1 [00:00<?, ?it/s]
Creating parquet from Arrow format: 0% 0/8 [00:00<?, ?ba/s]
Creating parquet from Arrow format: 38% 3/8 [00:00<00:00, 21.31ba/s]
Creating parquet from Arrow format: 100% 8/8 [00:00<00:00, 22.66ba/s]
Upload 1 LFS files: 0% 0/1 [00:00<?, ?it/s]
Upload 1 LFS files: 100% 1/1 [00:00<00:00, 1.87it/s]
Pushing dataset shards to the dataset hub: 100% 1/1 [00:01<00:00, 1.25s/it]
INFO:root:Finished preparing dataset. Exiting...
In [ ]:
!rm -rf /content/axolotl/last_run_prepared
minotaur (deprecated)¶
Datasets from https://huggingface.co/openaccess-ai-collective/minotaur-13b/blob/main/configs/minotaur.yml
Plus this fix
In [ ]:
%%writefile examples/falcon/config-40b-qlora.yml
# 1b: tiiuae/falcon-rw-1b
# 7b: tiiuae/falcon-7b
# 40b: tiiuae/falcon-40b
base_model: tiiuae/falcon-40b
base_model_config: tiiuae/falcon-40b
# required by falcon custom model code: https://huggingface.co/tiiuae/falcon-7b/tree/main
trust_remote_code: true
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer
load_in_8bit: false
# enable 4bit for QLoRA
load_in_4bit: true
gptq: false
strict: false
push_dataset_to_hub: utensil
hf_use_auth_token: true
datasets:
- path: winglian/evals
data_files:
- hf/ARC-Challenge.jsonl
- hf/ARC-Easy.jsonl
- hf/riddle_sense.jsonl
type: explainchoice:chat
- path: winglian/evals
data_files:
- hf/gsm8k.jsonl
- hf/winogrande.jsonl
type: alpaca_chat.load_qa
- path: winglian/evals
data_files:
- custom/n_task.jsonl
- custom/misconceptions.jsonl
- custom/context_insensitivity.jsonl
type: alpaca_chat
- path: camel-ai/math
type: alpaca_chat.load_camel_ai
- path: camel-ai/biology
type: alpaca_chat.load_camel_ai
- path: camel-ai/physics
type: alpaca_chat.load_camel_ai
- path: camel-ai/chemistry
type: alpaca_chat.load_camel_ai
- path: winglian/evals
data_files:
- custom/in_context_qa.jsonl
type: context_qa
- path: winglian/evals
data_files:
- custom/in_context_qa.jsonl
type: context_qa.load_404
- path: winglian/evals
data_files:
- custom/jokes_explained_500up.jsonl
type: sharegpt_jokes
- path: winglian/evals
data_files:
- custom/classify-self-chat.sharegpt.jsonl
- custom/coding-self-chat.sharegpt.jsonl
- custom/prose-gpt4.sharegpt.jsonl
- custom/prose-rewrite-gpt4.sharegpt.jsonl
type: sharegpt_simple.load_role
- path: winglian/evals
data_files:
- openai/tldr.jsonl
type: summarizetldr:chat
- path: winglian/evals
data_files:
- hellaswag/hellaswag.jsonl
type: explainchoice:chat
- path: metaeval/ScienceQA_text_only
type: concisechoice:chat
- path: teknium/GPT4-LLM-Cleaned
type: alpaca_chat
- path: teknium/GPTeacher-General-Instruct
data_files: gpt4-instruct-similarity-0.6-dataset.json
type: gpteacher:chat
- path: QingyiSi/Alpaca-CoT
data_files:
- Chain-of-Thought/formatted_cot_data/aqua_train.json
- Chain-of-Thought/formatted_cot_data/creak_train.json
- Chain-of-Thought/formatted_cot_data/ecqa_train.json
- Chain-of-Thought/formatted_cot_data/esnli_train.json
- Chain-of-Thought/formatted_cot_data/qasc_train.json
- Chain-of-Thought/formatted_cot_data/qed_train.json
- Chain-of-Thought/formatted_cot_data/sensemaking_train.json
- Chain-of-Thought/formatted_cot_data/strategyqa_train.json
- GPTeacher/Roleplay/formatted_roleplay-similarity_0.6-instruct-dataset.json
type: alpaca_chat
- path: ehartford/WizardLM_alpaca_evol_instruct_70k_unfiltered
type: alpaca_chat
- path: ehartford/wizard_vicuna_70k_unfiltered
type: sharegpt:chat
dataset_prepared_path: last_run_prepared
val_set_size: 0.01
# enable QLoRA
adapter: qlora
lora_model_dir:
sequence_len: 2048
max_packed_sequence_len:
# hyperparameters from QLoRA paper Appendix B.2
# "We find hyperparameters to be largely robust across datasets"
lora_r: 64
lora_alpha: 16
# 0.1 for models up to 13B
# 0.05 for 33B and 65B models
lora_dropout: 0.05
# add LoRA modules on all linear layers of the base model
lora_target_modules:
lora_target_linear: true
lora_fan_in_fan_out:
wandb_project: falcon-qlora
wandb_watch:
wandb_run_id:
wandb_log_model:
output_dir: /workspace/llm-playground/axolotl-trained/falcon-qlora-40b-gsm8k/
# QLoRA paper Table 9
# - 16 for 7b & 13b
# - 32 for 33b, 64 for 64b
# Max size tested on A6000
# - 7b: 40
# - 40b: 4
# decrease if OOM, increase for max VRAM utilization
micro_batch_size: 1
gradient_accumulation_steps: 2
num_epochs: 3
# Optimizer for QLoRA
optimizer: paged_adamw_32bit
torchdistx_path:
lr_scheduler: cosine
# QLoRA paper Table 9
# - 2e-4 for 7b & 13b
# - 1e-4 for 33b & 64b
learning_rate: 0.0002
train_on_inputs: false
group_by_length: false
bf16: true
fp16: false
tf32: true
gradient_checkpointing: true
# stop training after this many evaluation losses have increased in a row
# https://huggingface.co/transformers/v4.2.2/_modules/transformers/trainer_callback.html#EarlyStoppingCallback
early_stopping_patience: 3
resume_from_checkpoint:
auto_resume_from_checkpoints: true
local_rank:
logging_steps: 1
xformers_attention: true
flash_attention:
gptq_groupsize:
gptq_model_v1:
warmup_steps: 10
eval_steps: 5
save_steps: 10
debug:
deepspeed:
weight_decay: 0.01
fsdp:
fsdp_config:
special_tokens:
pad_token: "<|endoftext|>"
bos_token: ">>ABSTRACT<<"
eos_token: "<|endoftext|>"
Writing examples/falcon/config-40b-qlora.yml
In [ ]:
!python scripts/finetune.py examples/falcon/config-40b-qlora.yml --prepare_ds_only
===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please run
python -m bitsandbytes
and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================
bin /root/miniconda3/envs/py3.9/lib/python3.9/site-packages/bitsandbytes-0.39.0-py3.9.egg/bitsandbytes/libbitsandbytes_cuda118.so
/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/bitsandbytes-0.39.0-py3.9.egg/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/usr/local/nvidia/lib'), PosixPath('/usr/local/nvidia/lib64')}
warn(msg)
/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/bitsandbytes-0.39.0-py3.9.egg/bitsandbytes/cuda_setup/main.py:149: UserWarning: /usr/local/nvidia/lib:/usr/local/nvidia/lib64 did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths...
warn(msg)
/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/bitsandbytes-0.39.0-py3.9.egg/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//matplotlib_inline.backend_inline'), PosixPath('module')}
warn(msg)
/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/bitsandbytes-0.39.0-py3.9.egg/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDvnE4umHheXhWsDJbbukYvvyc47/mC4z8syS93btA72T90WDrQagOy5O+DrhdXOvr5i/JwsTlAImy57eLRrtRFOrQq73jyi7Dzo0tvrAiNLVgX2q2dFLoplRyXDXiVYLPmPieMWQOeUCLeSb8FC5zzllcocZwjMXpxScDerZqnlAR0ccpSkGyKIod4ZMkn/29A/C5kHEb/wT8cOAq+MWJ/2okZZgbiR0AMV4DynAkrtcx9JnJnTs9chiMyH+dyCS42Ai24sHWJBkQo6TfxXkyKo9GOpu3Y2WLgrHyaot9Lk5mA1mujyIWdlReD2nvjeCQKjl3KW3xZ73m4nD97MydWSWoJfEWlr+VZvk8EWsZk3CYLZCIBLdod6xXJJ0DD0pvTIq11c8VB7XkgVjapuU/sC8M6HFzHW/NBeE+xX/txPkZkIGqrnxeQ0AtBXdN9ukyNGhGzTkPYJNliiYpY0dCvVuz/BJ2FawFTQGnD1EHOenUCRajREFGCbKoYZqi40j8= utensil@Utensils-MacBook-Pro.local')}
warn(msg)
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/bitsandbytes-0.39.0-py3.9.egg/bitsandbytes/cuda_setup/main.py:149: UserWarning: Found duplicate ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] files: {PosixPath('/usr/local/cuda/lib64/libcudart.so'), PosixPath('/usr/local/cuda/lib64/libcudart.so.11.0')}.. We'll flip a coin and try one of these, in order to fail forward.
Either way, this might cause trouble in the future:
If you get `CUDA error: invalid device function` errors, the above might be the cause and the solution is to make sure only one ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] in the paths that we search based on your env.
warn(msg)
CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so
CUDA SETUP: Highest compute capability among GPUs detected: 8.6
CUDA SETUP: Detected CUDA version 118
CUDA SETUP: Loading binary /root/miniconda3/envs/py3.9/lib/python3.9/site-packages/bitsandbytes-0.39.0-py3.9.egg/bitsandbytes/libbitsandbytes_cuda118.so...
Setting ds_accelerator to cuda (auto detect)
WARNING:root:`trust_remote_code` is set to true. Please make sure that you reviewed the remote code/model.
INFO:root:loading tokenizer... tiiuae/falcon-40b
Downloading (…)okenizer_config.json: 100%|█████| 175/175 [00:00<00:00, 53.7kB/s]
Downloading (…)/main/tokenizer.json: 100%|█| 2.73M/2.73M [00:00<00:00, 20.5MB/s]
Downloading (…)cial_tokens_map.json: 100%|██████| 281/281 [00:00<00:00, 218kB/s]
Using bos_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using unk_token, but it is not set yet.
INFO:root:Unable to find prepared dataset in last_run_prepared/9a63aa2c07ace8350a0e8b32ab913f2a
INFO:root:Loading raw datasets...
INFO:root:No seed provided, using default seed of 42
Downloading readme: 100%|██████████████████████| 259/259 [00:00<00:00, 2.28MB/s]
Downloading and preparing dataset json/winglian--evals to /root/.cache/huggingface/datasets/winglian___json/winglian--evals-967fbffa8fbc6a56/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4...
Downloading data files: 0%| | 0/1 [00:00<?, ?it/s]
Downloading data: 0%| | 0.00/2.05M [00:00<?, ?B/s]
Downloading data: 4%|▊ | 75.8k/2.05M [00:00<00:02, 732kB/s]
Downloading data: 7%|█▌ | 150k/2.05M [00:00<00:02, 700kB/s]
Downloading data: 26%|█████▍ | 526k/2.05M [00:00<00:00, 1.70MB/s]
Downloading data: 100%|████████████████████| 2.05M/2.05M [00:00<00:00, 4.20MB/s]
Downloading data: 0%| | 0.00/3.22M [00:00<?, ?B/s]
Downloading data: 27%|█████▋ | 882k/3.22M [00:00<00:00, 7.59MB/s]
Downloading data: 100%|████████████████████| 3.22M/3.22M [00:00<00:00, 14.2MB/s]
Downloading data: 0%| | 0.00/3.94M [00:00<?, ?B/s]
Downloading data: 2%|▍ | 89.1k/3.94M [00:00<00:07, 495kB/s]
Downloading data: 8%|█▋ | 307k/3.94M [00:00<00:03, 1.18MB/s]
Downloading data: 28%|█████▌ | 1.10M/3.94M [00:00<00:00, 3.03MB/s]
Downloading data: 100%|████████████████████| 3.94M/3.94M [00:00<00:00, 6.60MB/s]
Downloading data files: 100%|█████████████████████| 1/1 [00:02<00:00, 2.90s/it]
Extracting data files: 100%|█████████████████████| 1/1 [00:00<00:00, 451.15it/s]
Dataset json downloaded and prepared to /root/.cache/huggingface/datasets/winglian___json/winglian--evals-967fbffa8fbc6a56/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4. Subsequent calls will reuse this data.
100%|████████████████████████████████████████████| 1/1 [00:00<00:00, 372.86it/s]
Downloading and preparing dataset json/winglian--evals to /root/.cache/huggingface/datasets/winglian___json/winglian--evals-2d9807bf90eb06df/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4...
Downloading data files: 0%| | 0/1 [00:00<?, ?it/s]
Downloading data: 0%| | 0.00/7.79M [00:00<?, ?B/s]
Downloading data: 1%|▏ | 75.8k/7.79M [00:00<00:10, 729kB/s]
Downloading data: 3%|▋ | 239k/7.79M [00:00<00:08, 900kB/s]
Downloading data: 12%|██▌ | 943k/7.79M [00:00<00:02, 3.15MB/s]
Downloading data: 28%|█████▋ | 2.19M/7.79M [00:00<00:00, 6.37MB/s]
Downloading data: 39%|███████▉ | 3.07M/7.79M [00:00<00:00, 7.16MB/s]
Downloading data: 100%|████████████████████| 7.79M/7.79M [00:00<00:00, 10.5MB/s]
Downloading data: 0%| | 0.00/5.89M [00:00<?, ?B/s]
Downloading data: 4%|▊ | 216k/5.89M [00:00<00:02, 2.00MB/s]
Downloading data: 28%|█████▌ | 1.65M/5.89M [00:00<00:00, 6.56MB/s]
Downloading data: 100%|████████████████████| 5.89M/5.89M [00:00<00:00, 12.6MB/s]
Downloading data files: 100%|█████████████████████| 1/1 [00:04<00:00, 4.27s/it]
Extracting data files: 100%|█████████████████████| 1/1 [00:00<00:00, 645.28it/s]
Dataset json downloaded and prepared to /root/.cache/huggingface/datasets/winglian___json/winglian--evals-2d9807bf90eb06df/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4. Subsequent calls will reuse this data.
100%|████████████████████████████████████████████| 1/1 [00:00<00:00, 471.27it/s]
Downloading and preparing dataset json/winglian--evals to /root/.cache/huggingface/datasets/winglian___json/winglian--evals-5354680ac81acd67/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4...
Downloading data files: 0%| | 0/1 [00:00<?, ?it/s]
Downloading data: 100%|██████████████████████| 172k/172k [00:00<00:00, 4.32MB/s]
Downloading data: 0%| | 0.00/395k [00:00<?, ?B/s]
Downloading data: 6%|█▎ | 24.6k/395k [00:00<00:01, 227kB/s]
Downloading data: 100%|██████████████████████| 395k/395k [00:00<00:00, 1.35MB/s]
Downloading data: 100%|████████████████████| 79.9k/79.9k [00:00<00:00, 5.18MB/s]
Downloading data files: 100%|█████████████████████| 1/1 [00:01<00:00, 1.80s/it]
Extracting data files: 100%|█████████████████████| 1/1 [00:00<00:00, 708.38it/s]
Dataset json downloaded and prepared to /root/.cache/huggingface/datasets/winglian___json/winglian--evals-5354680ac81acd67/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4. Subsequent calls will reuse this data.
100%|████████████████████████████████████████████| 1/1 [00:00<00:00, 827.44it/s]
Downloading readme: 100%|██████████████████| 2.19k/2.19k [00:00<00:00, 9.19MB/s]
Downloading and preparing dataset json/camel-ai--math to /root/.cache/huggingface/datasets/camel-ai___json/camel-ai--math-e65613a5e6088d3f/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4...
Downloading data files: 0%| | 0/1 [00:00<?, ?it/s]
Downloading data: 0%| | 0.00/39.8M [00:00<?, ?B/s]
Downloading data: 3%|▌ | 1.13M/39.8M [00:00<00:03, 11.3MB/s]
Downloading data: 13%|██▌ | 4.99M/39.8M [00:00<00:01, 27.2MB/s]
Downloading data: 27%|█████▍ | 10.7M/39.8M [00:00<00:00, 38.0MB/s]
Downloading data: 36%|███████▎ | 14.4M/39.8M [00:00<00:00, 33.5MB/s]
Downloading data: 49%|█████████▋ | 19.4M/39.8M [00:00<00:00, 38.6MB/s]
Downloading data: 63%|████████████▌ | 24.9M/39.8M [00:00<00:00, 42.5MB/s]
Downloading data: 79%|███████████████▊ | 31.3M/39.8M [00:00<00:00, 49.2MB/s]
Downloading data: 100%|████████████████████| 39.8M/39.8M [00:00<00:00, 40.2MB/s]
Downloading data files: 100%|█████████████████████| 1/1 [00:01<00:00, 1.50s/it]
Extracting data files: 100%|██████████████████████| 1/1 [00:06<00:00, 6.37s/it]
Dataset json downloaded and prepared to /root/.cache/huggingface/datasets/camel-ai___json/camel-ai--math-e65613a5e6088d3f/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4. Subsequent calls will reuse this data.
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 1.61it/s]
Downloading readme: 100%|██████████████████| 2.12k/2.12k [00:00<00:00, 9.70MB/s]
Downloading and preparing dataset json/camel-ai--biology to /root/.cache/huggingface/datasets/camel-ai___json/camel-ai--biology-a0dba0b3efd07bc4/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4...
Downloading data files: 0%| | 0/1 [00:00<?, ?it/s]
Downloading data: 0%| | 0.00/27.4M [00:00<?, ?B/s]
Downloading data: 8%|█▌ | 2.14M/27.4M [00:00<00:01, 21.3MB/s]
Downloading data: 31%|██████▏ | 8.46M/27.4M [00:00<00:00, 45.9MB/s]
Downloading data: 64%|████████████▊ | 17.6M/27.4M [00:00<00:00, 66.7MB/s]
Downloading data: 100%|████████████████████| 27.4M/27.4M [00:00<00:00, 67.3MB/s]
Downloading data files: 100%|█████████████████████| 1/1 [00:00<00:00, 1.00it/s]
Extracting data files: 100%|██████████████████████| 1/1 [00:02<00:00, 2.48s/it]
Dataset json downloaded and prepared to /root/.cache/huggingface/datasets/camel-ai___json/camel-ai--biology-a0dba0b3efd07bc4/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4. Subsequent calls will reuse this data.
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 3.02it/s]
Downloading readme: 100%|██████████████████| 2.12k/2.12k [00:00<00:00, 19.3MB/s]
Downloading and preparing dataset json/camel-ai--physics to /root/.cache/huggingface/datasets/camel-ai___json/camel-ai--physics-c2d5e46897f076a2/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4...
Downloading data files: 0%| | 0/1 [00:00<?, ?it/s]
Downloading data: 0%| | 0.00/23.5M [00:00<?, ?B/s]
Downloading data: 7%|█▍ | 1.77M/23.5M [00:00<00:01, 17.7MB/s]
Downloading data: 36%|███████ | 8.37M/23.5M [00:00<00:00, 44.7MB/s]
Downloading data: 100%|████████████████████| 23.5M/23.5M [00:00<00:00, 60.4MB/s]
Downloading data files: 100%|█████████████████████| 1/1 [00:01<00:00, 1.03s/it]
Extracting data files: 100%|██████████████████████| 1/1 [00:02<00:00, 2.52s/it]
Dataset json downloaded and prepared to /root/.cache/huggingface/datasets/camel-ai___json/camel-ai--physics-c2d5e46897f076a2/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4. Subsequent calls will reuse this data.
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 3.11it/s]
Downloading readme: 100%|██████████████████| 2.14k/2.14k [00:00<00:00, 8.95MB/s]
Downloading and preparing dataset json/camel-ai--chemistry to /root/.cache/huggingface/datasets/camel-ai___json/camel-ai--chemistry-8a34ec93529b52ff/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4...
Downloading data files: 0%| | 0/1 [00:00<?, ?it/s]
Downloading data: 0%| | 0.00/21.8M [00:00<?, ?B/s]
Downloading data: 9%|█▊ | 2.00M/21.8M [00:00<00:01, 19.5MB/s]
Downloading data: 41%|████████▏ | 8.96M/21.8M [00:00<00:00, 48.7MB/s]
Downloading data: 100%|████████████████████| 21.8M/21.8M [00:00<00:00, 60.6MB/s]
Downloading data files: 100%|█████████████████████| 1/1 [00:00<00:00, 1.17it/s]
Extracting data files: 100%|██████████████████████| 1/1 [00:02<00:00, 2.48s/it]
Dataset json downloaded and prepared to /root/.cache/huggingface/datasets/camel-ai___json/camel-ai--chemistry-8a34ec93529b52ff/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4. Subsequent calls will reuse this data.
100%|█████████████████████████████████████████████| 1/1 [00:00<00:00, 5.20it/s]
Downloading and preparing dataset json/winglian--evals to /root/.cache/huggingface/datasets/winglian___json/winglian--evals-81bfad14acee858f/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4...
Downloading data files: 0%| | 0/1 [00:00<?, ?it/s]
Downloading data: 0%| | 0.00/1.84M [00:00<?, ?B/s]
Downloading data: 1%|▎ | 25.6k/1.84M [00:00<00:07, 251kB/s]
Downloading data: 5%|█ | 98.3k/1.84M [00:00<00:03, 520kB/s]
Downloading data: 13%|██▉ | 245k/1.84M [00:00<00:01, 933kB/s]
Downloading data: 29%|█████▉ | 525k/1.84M [00:00<00:00, 1.64MB/s]
Downloading data: 100%|████████████████████| 1.84M/1.84M [00:00<00:00, 3.46MB/s]
Downloading data files: 100%|█████████████████████| 1/1 [00:01<00:00, 1.65s/it]
Extracting data files: 100%|████████████████████| 1/1 [00:00<00:00, 1064.27it/s]
Dataset json downloaded and prepared to /root/.cache/huggingface/datasets/winglian___json/winglian--evals-81bfad14acee858f/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4. Subsequent calls will reuse this data.
100%|███████████████████████████████████████████| 1/1 [00:00<00:00, 1077.67it/s]
WARNING:datasets.builder:Found cached dataset json (/root/.cache/huggingface/datasets/winglian___json/winglian--evals-81bfad14acee858f/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4)
100%|████████████████████████████████████████████| 1/1 [00:00<00:00, 627.61it/s]
Downloading and preparing dataset json/winglian--evals to /root/.cache/huggingface/datasets/winglian___json/winglian--evals-c3b1db8d176877c7/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4...
Downloading data files: 0%| | 0/1 [00:00<?, ?it/s]
Downloading data: 0%| | 0.00/1.58M [00:00<?, ?B/s]
Downloading data: 2%|▎ | 26.6k/1.58M [00:00<00:05, 261kB/s]
Downloading data: 7%|█▍ | 105k/1.58M [00:00<00:02, 558kB/s]
Downloading data: 15%|███▎ | 236k/1.58M [00:00<00:01, 881kB/s]
Downloading data: 33%|██████▉ | 527k/1.58M [00:00<00:00, 1.64MB/s]
Downloading data: 100%|████████████████████| 1.58M/1.58M [00:00<00:00, 3.00MB/s]
Downloading data files: 100%|█████████████████████| 1/1 [00:01<00:00, 1.13s/it]
Extracting data files: 100%|████████████████████| 1/1 [00:00<00:00, 1009.95it/s]
Dataset json downloaded and prepared to /root/.cache/huggingface/datasets/winglian___json/winglian--evals-c3b1db8d176877c7/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4. Subsequent calls will reuse this data.
100%|████████████████████████████████████████████| 1/1 [00:00<00:00, 474.84it/s]
Downloading and preparing dataset json/winglian--evals to /root/.cache/huggingface/datasets/winglian___json/winglian--evals-f3d8ab7303b30dba/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4...
Downloading data files: 0%| | 0/1 [00:00<?, ?it/s]
Downloading data: 0%| | 0.00/626k [00:00<?, ?B/s]
Downloading data: 4%|▉ | 26.6k/626k [00:00<00:02, 261kB/s]
Downloading data: 17%|███▉ | 105k/626k [00:00<00:00, 558kB/s]
Downloading data: 39%|████████▉ | 242k/626k [00:00<00:00, 895kB/s]
Downloading data: 100%|██████████████████████| 626k/626k [00:00<00:00, 1.49MB/s]
Downloading data: 0%| | 0.00/7.87M [00:00<?, ?B/s]
Downloading data: 0%| | 32.8k/7.87M [00:00<00:24, 319kB/s]
Downloading data: 1%|▎ | 98.3k/7.87M [00:00<00:15, 504kB/s]
Downloading data: 3%|▋ | 246k/7.87M [00:00<00:08, 924kB/s]
Downloading data: 7%|█▍ | 524k/7.87M [00:00<00:04, 1.62MB/s]
Downloading data: 14%|██▊ | 1.10M/7.87M [00:00<00:02, 3.03MB/s]
Downloading data: 28%|█████▋ | 2.23M/7.87M [00:00<00:00, 5.70MB/s]
Downloading data: 100%|████████████████████| 7.87M/7.87M [00:00<00:00, 10.2MB/s]
Downloading data: 0%| | 0.00/1.12M [00:00<?, ?B/s]
Downloading data: 2%|▍ | 25.6k/1.12M [00:00<00:04, 250kB/s]
Downloading data: 9%|██ | 105k/1.12M [00:00<00:01, 562kB/s]
Downloading data: 21%|████▋ | 239k/1.12M [00:00<00:00, 894kB/s]
Downloading data: 48%|█████████▉ | 534k/1.12M [00:00<00:00, 1.67MB/s]
Downloading data: 100%|████████████████████| 1.12M/1.12M [00:00<00:00, 2.17MB/s]
Downloading data: 0%| | 0.00/1.13M [00:00<?, ?B/s]
Downloading data: 3%|▋ | 33.8k/1.13M [00:00<00:03, 329kB/s]
Downloading data: 8%|█▋ | 89.1k/1.13M [00:00<00:02, 450kB/s]
Downloading data: 21%|████▌ | 233k/1.13M [00:00<00:01, 883kB/s]
Downloading data: 46%|█████████▋ | 520k/1.13M [00:00<00:00, 1.63MB/s]
Downloading data: 100%|████████████████████| 1.13M/1.13M [00:00<00:00, 2.18MB/s]
Downloading data files: 100%|█████████████████████| 1/1 [00:05<00:00, 5.83s/it]
Extracting data files: 100%|█████████████████████| 1/1 [00:00<00:00, 385.68it/s]
Dataset json downloaded and prepared to /root/.cache/huggingface/datasets/winglian___json/winglian--evals-f3d8ab7303b30dba/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4. Subsequent calls will reuse this data.
100%|████████████████████████████████████████████| 1/1 [00:00<00:00, 710.18it/s]
Downloading and preparing dataset json/winglian--evals to /root/.cache/huggingface/datasets/winglian___json/winglian--evals-c89bb0dad6bf8005/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4...
Downloading data files: 0%| | 0/1 [00:00<?, ?it/s]
Downloading data: 0%| | 0.00/2.23M [00:00<?, ?B/s]
Downloading data: 1%|▎ | 26.6k/2.23M [00:00<00:08, 260kB/s]
Downloading data: 5%|█ | 105k/2.23M [00:00<00:03, 557kB/s]
Downloading data: 11%|██▍ | 247k/2.23M [00:00<00:02, 930kB/s]
Downloading data: 23%|████▉ | 524k/2.23M [00:00<00:01, 1.62MB/s]
Downloading data: 100%|████████████████████| 2.23M/2.23M [00:00<00:00, 4.14MB/s]
Downloading data files: 100%|█████████████████████| 1/1 [00:01<00:00, 1.59s/it]
Extracting data files: 100%|█████████████████████| 1/1 [00:00<00:00, 983.65it/s]
Dataset json downloaded and prepared to /root/.cache/huggingface/datasets/winglian___json/winglian--evals-c89bb0dad6bf8005/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4. Subsequent calls will reuse this data.
100%|████████████████████████████████████████████| 1/1 [00:00<00:00, 547.27it/s]
Downloading and preparing dataset json/winglian--evals to /root/.cache/huggingface/datasets/winglian___json/winglian--evals-99f605981858093b/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4...
Downloading data files: 0%| | 0/1 [00:00<?, ?it/s]
Downloading data: 0%| | 0.00/73.1M [00:00<?, ?B/s]
Downloading data: 0%| | 25.6k/73.1M [00:00<04:53, 249kB/s]
Downloading data: 0%| | 105k/73.1M [00:00<02:10, 560kB/s]
Downloading data: 0%| | 239k/73.1M [00:00<01:21, 892kB/s]
Downloading data: 1%|▏ | 530k/73.1M [00:00<00:43, 1.65MB/s]
Downloading data: 2%|▎ | 1.10M/73.1M [00:00<00:23, 3.06MB/s]
Downloading data: 3%|▌ | 2.23M/73.1M [00:00<00:12, 5.74MB/s]
Downloading data: 6%|█▏ | 4.48M/73.1M [00:00<00:06, 10.9MB/s]
Downloading data: 12%|██▍ | 9.00M/73.1M [00:00<00:03, 21.3MB/s]
Downloading data: 19%|███▉ | 14.2M/73.1M [00:00<00:01, 30.4MB/s]
Downloading data: 26%|█████▏ | 18.8M/73.1M [00:01<00:01, 33.1MB/s]
Downloading data: 33%|██████▌ | 24.2M/73.1M [00:01<00:01, 39.1MB/s]
Downloading data: 40%|████████ | 29.3M/73.1M [00:01<00:01, 42.5MB/s]
Downloading data: 47%|█████████▍ | 34.3M/73.1M [00:01<00:00, 44.8MB/s]
Downloading data: 54%|██████████▋ | 39.2M/73.1M [00:01<00:00, 46.2MB/s]
Downloading data: 61%|████████████ | 44.3M/73.1M [00:01<00:00, 47.2MB/s]
Downloading data: 68%|█████████████▌ | 49.4M/73.1M [00:01<00:00, 47.9MB/s]
Downloading data: 75%|██████████████▉ | 54.5M/73.1M [00:01<00:00, 48.5MB/s]
Downloading data: 82%|████████████████▎ | 59.6M/73.1M [00:01<00:00, 48.7MB/s]
Downloading data: 88%|█████████████████▋ | 64.5M/73.1M [00:01<00:00, 46.9MB/s]
Downloading data: 100%|████████████████████| 73.1M/73.1M [00:02<00:00, 34.5MB/s]
Downloading data files: 100%|█████████████████████| 1/1 [00:05<00:00, 5.55s/it]
Extracting data files: 100%|████████████████████| 1/1 [00:00<00:00, 1658.48it/s]
Dataset json downloaded and prepared to /root/.cache/huggingface/datasets/winglian___json/winglian--evals-99f605981858093b/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4. Subsequent calls will reuse this data.
100%|████████████████████████████████████████████| 1/1 [00:00<00:00, 437.45it/s]
Downloading readme: 100%|██████████████████| 1.23k/1.23k [00:00<00:00, 4.81MB/s]
Downloading and preparing dataset None/None to /root/.cache/huggingface/datasets/metaeval___parquet/metaeval--ScienceQA_text_only-00e2b9b849dc5307/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec...
Downloading data files: 0%| | 0/3 [00:00<?, ?it/s]
Downloading data: 100%|████████████████████| 1.73M/1.73M [00:00<00:00, 30.4MB/s]
Downloading data files: 33%|███████ | 1/3 [00:00<00:01, 1.27it/s]
Downloading data: 100%|██████████████████████| 576k/576k [00:00<00:00, 17.0MB/s]
Downloading data files: 67%|██████████████ | 2/3 [00:01<00:00, 1.38it/s]
Downloading data: 100%|██████████████████████| 619k/619k [00:00<00:00, 18.4MB/s]
Downloading data files: 100%|█████████████████████| 3/3 [00:02<00:00, 1.23it/s]
Extracting data files: 100%|████████████████████| 3/3 [00:00<00:00, 2335.36it/s]
Dataset parquet downloaded and prepared to /root/.cache/huggingface/datasets/metaeval___parquet/metaeval--ScienceQA_text_only-00e2b9b849dc5307/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec. Subsequent calls will reuse this data.
100%|████████████████████████████████████████████| 3/3 [00:00<00:00, 429.79it/s]
Downloading readme: 100%|██████████████████████| 501/501 [00:00<00:00, 2.13MB/s]
Downloading and preparing dataset json/teknium--GPT4-LLM-Cleaned to /root/.cache/huggingface/datasets/teknium___json/teknium--GPT4-LLM-Cleaned-a71aa8ae1ac3982d/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4...
Downloading data files: 0%| | 0/1 [00:00<?, ?it/s]
Downloading data: 0%| | 0.00/36.0M [00:00<?, ?B/s]
Downloading data: 12%|██▎ | 4.21M/36.0M [00:00<00:00, 42.1MB/s]
Downloading data: 23%|████▋ | 8.42M/36.0M [00:00<00:00, 32.4MB/s]
Downloading data: 33%|██████▌ | 11.8M/36.0M [00:00<00:00, 31.0MB/s]
Downloading data: 42%|████████▎ | 15.0M/36.0M [00:00<00:00, 30.2MB/s]
Downloading data: 58%|███████████▌ | 20.8M/36.0M [00:00<00:00, 39.4MB/s]
Downloading data: 72%|██████████████▍ | 26.1M/36.0M [00:00<00:00, 43.5MB/s]
Downloading data: 85%|████████████████▉ | 30.5M/36.0M [00:00<00:00, 35.5MB/s]
Downloading data: 100%|████████████████████| 36.0M/36.0M [00:00<00:00, 36.8MB/s]
Downloading data: 0%| | 0.00/4.91M [00:00<?, ?B/s]
Downloading data: 1%| | 26.6k/4.91M [00:00<00:18, 259kB/s]
Downloading data: 2%|▍ | 92.2k/4.91M [00:00<00:10, 479kB/s]
Downloading data: 5%|█ | 233k/4.91M [00:00<00:05, 889kB/s]
Downloading data: 11%|██▎ | 528k/4.91M [00:00<00:02, 1.67MB/s]
Downloading data: 22%|████▍ | 1.10M/4.91M [00:00<00:01, 3.06MB/s]
Downloading data: 45%|█████████ | 2.23M/4.91M [00:00<00:00, 5.73MB/s]
Downloading data: 100%|████████████████████| 4.91M/4.91M [00:00<00:00, 6.74MB/s]
Downloading data files: 100%|█████████████████████| 1/1 [00:03<00:00, 3.21s/it]
Extracting data files: 100%|█████████████████████| 1/1 [00:00<00:00, 901.03it/s]
Dataset json downloaded and prepared to /root/.cache/huggingface/datasets/teknium___json/teknium--GPT4-LLM-Cleaned-a71aa8ae1ac3982d/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4. Subsequent calls will reuse this data.
100%|████████████████████████████████████████████| 1/1 [00:00<00:00, 519.29it/s]
Downloading readme: 100%|██████████████████████| 458/458 [00:00<00:00, 2.44MB/s]
Downloading and preparing dataset json/teknium--GPTeacher-General-Instruct to /root/.cache/huggingface/datasets/teknium___json/teknium--GPTeacher-General-Instruct-ff38d362dfbec0b7/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4...
Downloading data files: 0%| | 0/1 [00:00<?, ?it/s]
Downloading data: 0%| | 0.00/12.1M [00:00<?, ?B/s]
Downloading data: 21%|████▏ | 2.55M/12.1M [00:00<00:00, 25.3MB/s]
Downloading data: 42%|████████▍ | 5.09M/12.1M [00:00<00:00, 25.0MB/s]
Downloading data: 63%|████████████▋ | 7.64M/12.1M [00:00<00:00, 25.3MB/s]
Downloading data: 100%|████████████████████| 12.1M/12.1M [00:00<00:00, 25.9MB/s]
Downloading data files: 100%|█████████████████████| 1/1 [00:01<00:00, 1.18s/it]
Extracting data files: 100%|████████████████████| 1/1 [00:00<00:00, 1500.11it/s]
Dataset json downloaded and prepared to /root/.cache/huggingface/datasets/teknium___json/teknium--GPTeacher-General-Instruct-ff38d362dfbec0b7/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4. Subsequent calls will reuse this data.
100%|████████████████████████████████████████████| 1/1 [00:00<00:00, 783.10it/s]
Downloading readme: 100%|██████████████████| 8.26k/8.26k [00:00<00:00, 34.5MB/s]
Downloading and preparing dataset json/QingyiSi--Alpaca-CoT to /root/.cache/huggingface/datasets/QingyiSi___json/QingyiSi--Alpaca-CoT-72053b9662210036/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4...
Downloading data files: 0%| | 0/1 [00:00<?, ?it/s]
Downloading data: 100%|████████████████████| 1.20M/1.20M [00:00<00:00, 25.2MB/s]
Downloading data: 100%|████████████████████| 2.28M/2.28M [00:00<00:00, 32.4MB/s]
Downloading data: 100%|████████████████████| 2.81M/2.81M [00:00<00:00, 35.6MB/s]
Downloading data: 0%| | 0.00/15.0M [00:00<?, ?B/s]
Downloading data: 23%|████▌ | 3.41M/15.0M [00:00<00:00, 33.9MB/s]
Downloading data: 100%|████████████████████| 15.0M/15.0M [00:00<00:00, 47.2MB/s]
Downloading data: 100%|██████████████████████| 548k/548k [00:00<00:00, 16.9MB/s]
Downloading data: 0%| | 0.00/5.12M [00:00<?, ?B/s]
Downloading data: 100%|████████████████████| 5.12M/5.12M [00:00<00:00, 42.1MB/s]
Downloading data: 100%|████████████████████| 2.25M/2.25M [00:00<00:00, 29.6MB/s]
Downloading data: 100%|██████████████████████| 719k/719k [00:00<00:00, 18.7MB/s]
Downloading data: 0%| | 0.00/2.58M [00:00<?, ?B/s]
Downloading data: 100%|████████████████████| 2.58M/2.58M [00:00<00:00, 25.1MB/s]
Downloading data files: 100%|█████████████████████| 1/1 [00:07<00:00, 7.12s/it]
Extracting data files: 100%|█████████████████████| 1/1 [00:00<00:00, 175.38it/s]
Dataset json downloaded and prepared to /root/.cache/huggingface/datasets/QingyiSi___json/QingyiSi--Alpaca-CoT-72053b9662210036/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4. Subsequent calls will reuse this data.
100%|████████████████████████████████████████████| 1/1 [00:00<00:00, 472.23it/s]
Downloading readme: 100%|██████████████████████| 387/387 [00:00<00:00, 2.34MB/s]
Downloading and preparing dataset json/ehartford--WizardLM_alpaca_evol_instruct_70k_unfiltered to /root/.cache/huggingface/datasets/ehartford___json/ehartford--WizardLM_alpaca_evol_instruct_70k_unfiltered-d64f16a81bd92db8/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4...
Downloading data files: 0%| | 0/1 [00:00<?, ?it/s]
Downloading data: 0%| | 0.00/99.6M [00:00<?, ?B/s]
Downloading data: 5%|█ | 4.99M/99.6M [00:00<00:01, 49.9MB/s]
Downloading data: 12%|██▎ | 11.8M/99.6M [00:00<00:01, 60.5MB/s]
Downloading data: 18%|███▌ | 17.8M/99.6M [00:00<00:01, 45.6MB/s]
Downloading data: 23%|████▌ | 22.7M/99.6M [00:00<00:01, 42.8MB/s]
Downloading data: 30%|██████ | 29.9M/99.6M [00:00<00:01, 51.6MB/s]
Downloading data: 36%|███████ | 35.4M/99.6M [00:00<00:01, 51.9MB/s]
Downloading data: 41%|████████▏ | 40.8M/99.6M [00:00<00:01, 52.4MB/s]
Downloading data: 46%|█████████▎ | 46.2M/99.6M [00:00<00:01, 48.0MB/s]
Downloading data: 52%|██████████▍ | 52.0M/99.6M [00:01<00:00, 50.8MB/s]
Downloading data: 57%|███████████▍ | 57.2M/99.6M [00:01<00:00, 50.8MB/s]
Downloading data: 63%|████████████▌ | 62.4M/99.6M [00:01<00:00, 45.7MB/s]
Downloading data: 68%|█████████████▌ | 67.6M/99.6M [00:01<00:00, 47.6MB/s]
Downloading data: 73%|██████████████▌ | 72.5M/99.6M [00:01<00:00, 46.2MB/s]
Downloading data: 78%|███████████████▌ | 77.2M/99.6M [00:01<00:00, 44.1MB/s]
Downloading data: 84%|████████████████▊ | 83.9M/99.6M [00:01<00:00, 50.2MB/s]
Downloading data: 90%|█████████████████▉ | 89.4M/99.6M [00:01<00:00, 51.5MB/s]
Downloading data: 100%|████████████████████| 99.6M/99.6M [00:02<00:00, 49.4MB/s]
Downloading data files: 100%|█████████████████████| 1/1 [00:02<00:00, 2.91s/it]
Extracting data files: 100%|████████████████████| 1/1 [00:00<00:00, 1485.24it/s]
Dataset json downloaded and prepared to /root/.cache/huggingface/datasets/ehartford___json/ehartford--WizardLM_alpaca_evol_instruct_70k_unfiltered-d64f16a81bd92db8/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4. Subsequent calls will reuse this data.
100%|████████████████████████████████████████████| 1/1 [00:00<00:00, 455.36it/s]
Downloading readme: 100%|██████████████████████| 348/348 [00:00<00:00, 2.38MB/s]
Downloading and preparing dataset json/ehartford--wizard_vicuna_70k_unfiltered to /root/.cache/huggingface/datasets/ehartford___json/ehartford--wizard_vicuna_70k_unfiltered-71f549fba4cffce0/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4...
Downloading data files: 0%| | 0/1 [00:00<?, ?it/s]
Downloading data: 0%| | 0.00/152M [00:00<?, ?B/s]
Downloading data: 3%|▌ | 4.21M/152M [00:00<00:03, 42.1MB/s]
Downloading data: 6%|█▏ | 8.43M/152M [00:00<00:03, 36.0MB/s]
Downloading data: 10%|██ | 14.5M/152M [00:00<00:02, 46.1MB/s]
Downloading data: 13%|██▋ | 19.6M/152M [00:00<00:02, 47.9MB/s]
Downloading data: 16%|███▍ | 24.4M/152M [00:00<00:03, 40.7MB/s]
Downloading data: 20%|████ | 29.6M/152M [00:00<00:02, 43.8MB/s]
Downloading data: 23%|████▊ | 34.4M/152M [00:00<00:02, 45.0MB/s]
Downloading data: 26%|█████▍ | 39.0M/152M [00:00<00:02, 43.2MB/s]
Downloading data: 29%|██████ | 43.4M/152M [00:01<00:02, 36.8MB/s]
Downloading data: 32%|██████▋ | 47.8M/152M [00:01<00:02, 38.6MB/s]
Downloading data: 34%|███████▏ | 51.9M/152M [00:01<00:02, 36.5MB/s]
Downloading data: 38%|████████ | 58.3M/152M [00:01<00:02, 43.8MB/s]
Downloading data: 41%|████████▋ | 62.9M/152M [00:01<00:02, 42.3MB/s]
Downloading data: 44%|█████████▎ | 67.2M/152M [00:01<00:02, 40.7MB/s]
Downloading data: 47%|█████████▉ | 71.4M/152M [00:01<00:01, 40.1MB/s]
Downloading data: 51%|██████████▊ | 77.7M/152M [00:01<00:01, 46.5MB/s]
Downloading data: 55%|███████████▌ | 83.3M/152M [00:01<00:01, 49.1MB/s]
Downloading data: 59%|████████████▎ | 89.0M/152M [00:02<00:01, 51.4MB/s]
Downloading data: 62%|█████████████ | 94.3M/152M [00:02<00:01, 50.5MB/s]
Downloading data: 66%|█████████████▊ | 99.4M/152M [00:02<00:01, 49.7MB/s]
Downloading data: 70%|███████████████▎ | 106M/152M [00:02<00:00, 54.0MB/s]
Downloading data: 75%|████████████████▍ | 113M/152M [00:02<00:00, 59.3MB/s]
Downloading data: 79%|█████████████████▎ | 119M/152M [00:02<00:00, 57.8MB/s]
Downloading data: 82%|██████████████████ | 125M/152M [00:02<00:00, 54.1MB/s]
Downloading data: 86%|██████████████████▉ | 130M/152M [00:02<00:00, 52.0MB/s]
Downloading data: 89%|███████████████████▋ | 136M/152M [00:02<00:00, 48.2MB/s]
Downloading data: 93%|████████████████████▍ | 140M/152M [00:03<00:00, 42.7MB/s]
Downloading data: 100%|██████████████████████| 152M/152M [00:03<00:00, 46.1MB/s]
Downloading data files: 100%|█████████████████████| 1/1 [00:03<00:00, 3.95s/it]
Extracting data files: 100%|████████████████████| 1/1 [00:00<00:00, 1921.35it/s]
Dataset json downloaded and prepared to /root/.cache/huggingface/datasets/ehartford___json/ehartford--wizard_vicuna_70k_unfiltered-71f549fba4cffce0/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4. Subsequent calls will reuse this data.
100%|████████████████████████████████████████████| 1/1 [00:00<00:00, 437.54it/s]
INFO:root:tokenizing, merging, and shuffling master dataset
WARNING:root:role with empty message: ASSISTANT
WARNING:root:role with empty message: USER
WARNING:root:role with empty message: USER
WARNING:root:role with empty message: ASSISTANT
WARNING:root:role with empty message: ASSISTANT
INFO:root:Saving merged prepared dataset to disk... last_run_prepared/9a63aa2c07ace8350a0e8b32ab913f2a
INFO:root:Saving merged prepared dataset with push_to_hub... utensil/9a63aa2c07ace8350a0e8b32ab913f2a
Pushing dataset shards to the dataset hub: 0%| | 0/5 [00:00<?, ?it/s]
Creating parquet from Arrow format: 0%| | 0/85 [00:00<?, ?ba/s]
Creating parquet from Arrow format: 2%|▏ | 2/85 [00:00<00:06, 12.59ba/s]
Creating parquet from Arrow format: 5%|▍ | 4/85 [00:00<00:06, 13.14ba/s]
Creating parquet from Arrow format: 7%|▌ | 6/85 [00:00<00:05, 13.34ba/s]
Creating parquet from Arrow format: 9%|▊ | 8/85 [00:00<00:05, 13.59ba/s]
Creating parquet from Arrow format: 12%|▊ | 10/85 [00:00<00:05, 13.60ba/s]
Creating parquet from Arrow format: 14%|▉ | 12/85 [00:00<00:05, 13.75ba/s]
Creating parquet from Arrow format: 16%|█▏ | 14/85 [00:01<00:05, 12.91ba/s]
Creating parquet from Arrow format: 19%|█▎ | 16/85 [00:01<00:05, 13.20ba/s]
Creating parquet from Arrow format: 21%|█▍ | 18/85 [00:01<00:05, 13.28ba/s]
Creating parquet from Arrow format: 24%|█▋ | 20/85 [00:01<00:04, 13.06ba/s]
Creating parquet from Arrow format: 26%|█▊ | 22/85 [00:01<00:04, 13.28ba/s]
Creating parquet from Arrow format: 28%|█▉ | 24/85 [00:01<00:04, 13.44ba/s]
Creating parquet from Arrow format: 31%|██▏ | 26/85 [00:01<00:04, 13.69ba/s]
Creating parquet from Arrow format: 33%|██▎ | 28/85 [00:02<00:04, 13.84ba/s]
Creating parquet from Arrow format: 35%|██▍ | 30/85 [00:02<00:03, 14.00ba/s]
Creating parquet from Arrow format: 38%|██▋ | 32/85 [00:02<00:03, 14.06ba/s]
Creating parquet from Arrow format: 40%|██▊ | 34/85 [00:02<00:03, 14.16ba/s]
Creating parquet from Arrow format: 42%|██▉ | 36/85 [00:02<00:03, 14.33ba/s]
Creating parquet from Arrow format: 45%|███▏ | 38/85 [00:02<00:03, 14.40ba/s]
Creating parquet from Arrow format: 47%|███▎ | 40/85 [00:02<00:03, 14.44ba/s]
Creating parquet from Arrow format: 49%|███▍ | 42/85 [00:03<00:02, 14.37ba/s]
Creating parquet from Arrow format: 52%|███▌ | 44/85 [00:03<00:02, 14.43ba/s]
Creating parquet from Arrow format: 54%|███▊ | 46/85 [00:03<00:02, 14.11ba/s]
Creating parquet from Arrow format: 56%|███▉ | 48/85 [00:03<00:02, 13.87ba/s]
Creating parquet from Arrow format: 59%|████ | 50/85 [00:03<00:02, 13.43ba/s]
Creating parquet from Arrow format: 61%|████▎ | 52/85 [00:03<00:02, 13.70ba/s]
Creating parquet from Arrow format: 64%|████▍ | 54/85 [00:03<00:02, 13.96ba/s]
Creating parquet from Arrow format: 66%|████▌ | 56/85 [00:04<00:02, 14.12ba/s]
Creating parquet from Arrow format: 68%|████▊ | 58/85 [00:04<00:01, 14.25ba/s]
Creating parquet from Arrow format: 71%|████▉ | 60/85 [00:04<00:01, 14.32ba/s]
Creating parquet from Arrow format: 73%|█████ | 62/85 [00:04<00:01, 14.31ba/s]
Creating parquet from Arrow format: 75%|█████▎ | 64/85 [00:04<00:01, 14.30ba/s]
Creating parquet from Arrow format: 78%|█████▍ | 66/85 [00:04<00:01, 14.38ba/s]
Creating parquet from Arrow format: 80%|█████▌ | 68/85 [00:04<00:01, 14.43ba/s]
Creating parquet from Arrow format: 82%|█████▊ | 70/85 [00:05<00:01, 14.43ba/s]
Creating parquet from Arrow format: 85%|█████▉ | 72/85 [00:05<00:00, 14.45ba/s]
Creating parquet from Arrow format: 87%|██████ | 74/85 [00:05<00:00, 14.33ba/s]
Creating parquet from Arrow format: 89%|██████▎| 76/85 [00:05<00:00, 14.30ba/s]
Creating parquet from Arrow format: 92%|██████▍| 78/85 [00:05<00:00, 14.37ba/s]
Creating parquet from Arrow format: 94%|██████▌| 80/85 [00:05<00:00, 14.33ba/s]
Creating parquet from Arrow format: 96%|██████▊| 82/85 [00:05<00:00, 14.33ba/s]
Creating parquet from Arrow format: 100%|███████| 85/85 [00:06<00:00, 14.11ba/s]
Upload 1 LFS files: 0%| | 0/1 [00:00<?, ?it/s]
Upload 1 LFS files: 100%|█████████████████████████| 1/1 [00:10<00:00, 10.18s/it]
Pushing dataset shards to the dataset hub: 20%|▍ | 1/5 [00:17<01:08, 17.22s/it]
Creating parquet from Arrow format: 0%| | 0/85 [00:00<?, ?ba/s]
Creating parquet from Arrow format: 2%|▏ | 2/85 [00:00<00:05, 14.70ba/s]
Creating parquet from Arrow format: 5%|▍ | 4/85 [00:00<00:05, 14.54ba/s]
Creating parquet from Arrow format: 7%|▌ | 6/85 [00:00<00:05, 14.41ba/s]
Creating parquet from Arrow format: 9%|▊ | 8/85 [00:00<00:05, 14.48ba/s]
Creating parquet from Arrow format: 12%|▊ | 10/85 [00:00<00:05, 14.39ba/s]
Creating parquet from Arrow format: 14%|▉ | 12/85 [00:00<00:05, 14.39ba/s]
Creating parquet from Arrow format: 16%|█▏ | 14/85 [00:00<00:04, 14.47ba/s]
Creating parquet from Arrow format: 19%|█▎ | 16/85 [00:01<00:04, 14.51ba/s]
Creating parquet from Arrow format: 21%|█▍ | 18/85 [00:01<00:04, 14.52ba/s]
Creating parquet from Arrow format: 24%|█▋ | 20/85 [00:01<00:04, 14.27ba/s]
Creating parquet from Arrow format: 26%|█▊ | 22/85 [00:01<00:04, 14.04ba/s]
Creating parquet from Arrow format: 28%|█▉ | 24/85 [00:01<00:04, 14.29ba/s]
Creating parquet from Arrow format: 31%|██▏ | 26/85 [00:01<00:04, 14.37ba/s]
Creating parquet from Arrow format: 33%|██▎ | 28/85 [00:01<00:03, 14.46ba/s]
Creating parquet from Arrow format: 35%|██▍ | 30/85 [00:02<00:03, 14.48ba/s]
Creating parquet from Arrow format: 38%|██▋ | 32/85 [00:02<00:03, 14.53ba/s]
Creating parquet from Arrow format: 40%|██▊ | 34/85 [00:02<00:03, 14.57ba/s]
Creating parquet from Arrow format: 42%|██▉ | 36/85 [00:02<00:03, 14.54ba/s]
Creating parquet from Arrow format: 45%|███▏ | 38/85 [00:02<00:03, 14.60ba/s]
Creating parquet from Arrow format: 47%|███▎ | 40/85 [00:02<00:03, 14.58ba/s]
Creating parquet from Arrow format: 49%|███▍ | 42/85 [00:02<00:02, 14.52ba/s]
Creating parquet from Arrow format: 52%|███▌ | 44/85 [00:03<00:02, 14.49ba/s]
Creating parquet from Arrow format: 54%|███▊ | 46/85 [00:03<00:02, 14.54ba/s]
Creating parquet from Arrow format: 56%|███▉ | 48/85 [00:03<00:02, 14.58ba/s]
Creating parquet from Arrow format: 59%|████ | 50/85 [00:03<00:02, 14.54ba/s]
Creating parquet from Arrow format: 61%|████▎ | 52/85 [00:03<00:02, 14.44ba/s]
Creating parquet from Arrow format: 64%|████▍ | 54/85 [00:03<00:02, 14.51ba/s]
Creating parquet from Arrow format: 66%|████▌ | 56/85 [00:03<00:01, 14.58ba/s]
Creating parquet from Arrow format: 68%|████▊ | 58/85 [00:04<00:01, 14.66ba/s]
Creating parquet from Arrow format: 71%|████▉ | 60/85 [00:04<00:01, 14.64ba/s]
Creating parquet from Arrow format: 73%|█████ | 62/85 [00:04<00:01, 14.63ba/s]
Creating parquet from Arrow format: 75%|█████▎ | 64/85 [00:04<00:01, 14.66ba/s]
Creating parquet from Arrow format: 78%|█████▍ | 66/85 [00:04<00:01, 14.69ba/s]
Creating parquet from Arrow format: 80%|█████▌ | 68/85 [00:04<00:01, 14.60ba/s]
Creating parquet from Arrow format: 82%|█████▊ | 70/85 [00:04<00:01, 14.63ba/s]
Creating parquet from Arrow format: 85%|█████▉ | 72/85 [00:04<00:00, 14.60ba/s]
Creating parquet from Arrow format: 87%|██████ | 74/85 [00:05<00:00, 14.65ba/s]
Creating parquet from Arrow format: 89%|██████▎| 76/85 [00:05<00:00, 14.59ba/s]
Creating parquet from Arrow format: 92%|██████▍| 78/85 [00:05<00:00, 14.59ba/s]
Creating parquet from Arrow format: 94%|██████▌| 80/85 [00:05<00:00, 14.70ba/s]
Creating parquet from Arrow format: 96%|██████▊| 82/85 [00:05<00:00, 14.11ba/s]
Creating parquet from Arrow format: 100%|███████| 85/85 [00:05<00:00, 14.50ba/s]
Upload 1 LFS files: 0%| | 0/1 [00:00<?, ?it/s]
Upload 1 LFS files: 100%|█████████████████████████| 1/1 [00:08<00:00, 8.21s/it]
Pushing dataset shards to the dataset hub: 40%|▊ | 2/5 [00:31<00:47, 15.74s/it]
Creating parquet from Arrow format: 0%| | 0/85 [00:00<?, ?ba/s]
Creating parquet from Arrow format: 2%|▏ | 2/85 [00:00<00:06, 13.70ba/s]
Creating parquet from Arrow format: 5%|▍ | 4/85 [00:00<00:05, 13.92ba/s]
Creating parquet from Arrow format: 7%|▌ | 6/85 [00:00<00:05, 13.98ba/s]
Creating parquet from Arrow format: 9%|▊ | 8/85 [00:00<00:05, 13.83ba/s]
Creating parquet from Arrow format: 12%|▊ | 10/85 [00:00<00:05, 13.83ba/s]
Creating parquet from Arrow format: 14%|▉ | 12/85 [00:00<00:05, 13.96ba/s]
Creating parquet from Arrow format: 16%|█▏ | 14/85 [00:00<00:05, 14.16ba/s]
Creating parquet from Arrow format: 19%|█▎ | 16/85 [00:01<00:04, 14.18ba/s]
Creating parquet from Arrow format: 21%|█▍ | 18/85 [00:01<00:04, 14.07ba/s]
Creating parquet from Arrow format: 24%|█▋ | 20/85 [00:01<00:04, 13.84ba/s]
Creating parquet from Arrow format: 26%|█▊ | 22/85 [00:01<00:04, 14.06ba/s]
Creating parquet from Arrow format: 28%|█▉ | 24/85 [00:01<00:04, 14.22ba/s]
Creating parquet from Arrow format: 31%|██▏ | 26/85 [00:01<00:04, 14.30ba/s]
Creating parquet from Arrow format: 33%|██▎ | 28/85 [00:01<00:03, 14.39ba/s]
Creating parquet from Arrow format: 35%|██▍ | 30/85 [00:02<00:03, 14.49ba/s]
Creating parquet from Arrow format: 38%|██▋ | 32/85 [00:02<00:03, 14.44ba/s]
Creating parquet from Arrow format: 40%|██▊ | 34/85 [00:02<00:03, 14.46ba/s]
Creating parquet from Arrow format: 42%|██▉ | 36/85 [00:02<00:03, 14.38ba/s]
Creating parquet from Arrow format: 45%|███▏ | 38/85 [00:02<00:03, 14.43ba/s]
Creating parquet from Arrow format: 47%|███▎ | 40/85 [00:02<00:03, 14.34ba/s]
Creating parquet from Arrow format: 49%|███▍ | 42/85 [00:02<00:03, 14.29ba/s]
Creating parquet from Arrow format: 52%|███▌ | 44/85 [00:03<00:02, 14.34ba/s]
Creating parquet from Arrow format: 54%|███▊ | 46/85 [00:03<00:02, 14.43ba/s]
Creating parquet from Arrow format: 56%|███▉ | 48/85 [00:03<00:02, 14.40ba/s]
Creating parquet from Arrow format: 59%|████ | 50/85 [00:03<00:02, 14.40ba/s]
Creating parquet from Arrow format: 61%|████▎ | 52/85 [00:03<00:02, 14.40ba/s]
Creating parquet from Arrow format: 64%|████▍ | 54/85 [00:03<00:02, 14.36ba/s]
Creating parquet from Arrow format: 66%|████▌ | 56/85 [00:03<00:02, 14.44ba/s]
Creating parquet from Arrow format: 68%|████▊ | 58/85 [00:04<00:01, 14.61ba/s]
Creating parquet from Arrow format: 71%|████▉ | 60/85 [00:04<00:01, 14.67ba/s]
Creating parquet from Arrow format: 73%|█████ | 62/85 [00:04<00:01, 14.62ba/s]
Creating parquet from Arrow format: 75%|█████▎ | 64/85 [00:04<00:01, 14.45ba/s]
Creating parquet from Arrow format: 78%|█████▍ | 66/85 [00:04<00:01, 14.48ba/s]
Creating parquet from Arrow format: 80%|█████▌ | 68/85 [00:04<00:01, 14.54ba/s]
Creating parquet from Arrow format: 82%|█████▊ | 70/85 [00:04<00:01, 14.54ba/s]
Creating parquet from Arrow format: 85%|█████▉ | 72/85 [00:05<00:00, 14.61ba/s]
Creating parquet from Arrow format: 87%|██████ | 74/85 [00:05<00:00, 14.53ba/s]
Creating parquet from Arrow format: 89%|██████▎| 76/85 [00:05<00:00, 14.58ba/s]
Creating parquet from Arrow format: 92%|██████▍| 78/85 [00:05<00:00, 14.48ba/s]
Creating parquet from Arrow format: 94%|██████▌| 80/85 [00:05<00:00, 14.40ba/s]
Creating parquet from Arrow format: 96%|██████▊| 82/85 [00:05<00:00, 14.40ba/s]
Creating parquet from Arrow format: 100%|███████| 85/85 [00:05<00:00, 14.47ba/s]
Upload 1 LFS files: 0%| | 0/1 [00:00<?, ?it/s]
Upload 1 LFS files: 100%|█████████████████████████| 1/1 [00:06<00:00, 7.00s/it]
Pushing dataset shards to the dataset hub: 60%|█▏| 3/5 [00:45<00:29, 14.74s/it]
Creating parquet from Arrow format: 0%| | 0/85 [00:00<?, ?ba/s]
Creating parquet from Arrow format: 2%|▏ | 2/85 [00:00<00:06, 12.27ba/s]
Creating parquet from Arrow format: 5%|▍ | 4/85 [00:00<00:05, 13.56ba/s]
Creating parquet from Arrow format: 7%|▌ | 6/85 [00:00<00:05, 13.81ba/s]
Creating parquet from Arrow format: 9%|▊ | 8/85 [00:00<00:05, 13.88ba/s]
Creating parquet from Arrow format: 12%|▊ | 10/85 [00:00<00:05, 13.69ba/s]
Creating parquet from Arrow format: 14%|▉ | 12/85 [00:00<00:05, 13.94ba/s]
Creating parquet from Arrow format: 16%|█▏ | 14/85 [00:01<00:05, 14.18ba/s]
Creating parquet from Arrow format: 19%|█▎ | 16/85 [00:01<00:04, 14.16ba/s]
Creating parquet from Arrow format: 21%|█▍ | 18/85 [00:01<00:04, 14.11ba/s]
Creating parquet from Arrow format: 24%|█▋ | 20/85 [00:01<00:04, 13.89ba/s]
Creating parquet from Arrow format: 26%|█▊ | 22/85 [00:01<00:04, 13.64ba/s]
Creating parquet from Arrow format: 28%|█▉ | 24/85 [00:01<00:04, 13.88ba/s]
Creating parquet from Arrow format: 31%|██▏ | 26/85 [00:01<00:04, 14.07ba/s]
Creating parquet from Arrow format: 33%|██▎ | 28/85 [00:02<00:04, 14.22ba/s]
Creating parquet from Arrow format: 35%|██▍ | 30/85 [00:02<00:03, 14.27ba/s]
Creating parquet from Arrow format: 38%|██▋ | 32/85 [00:02<00:03, 14.27ba/s]
Creating parquet from Arrow format: 40%|██▊ | 34/85 [00:02<00:03, 14.36ba/s]
Creating parquet from Arrow format: 42%|██▉ | 36/85 [00:02<00:03, 14.32ba/s]
Creating parquet from Arrow format: 45%|███▏ | 38/85 [00:02<00:03, 14.37ba/s]
Creating parquet from Arrow format: 47%|███▎ | 40/85 [00:02<00:03, 14.44ba/s]
Creating parquet from Arrow format: 49%|███▍ | 42/85 [00:02<00:02, 14.58ba/s]
Creating parquet from Arrow format: 52%|███▌ | 44/85 [00:03<00:02, 14.61ba/s]
Creating parquet from Arrow format: 54%|███▊ | 46/85 [00:03<00:02, 14.25ba/s]
Creating parquet from Arrow format: 56%|███▉ | 48/85 [00:03<00:02, 14.24ba/s]
Creating parquet from Arrow format: 59%|████ | 50/85 [00:03<00:02, 14.25ba/s]
Creating parquet from Arrow format: 61%|████▎ | 52/85 [00:03<00:02, 14.27ba/s]
Creating parquet from Arrow format: 64%|████▍ | 54/85 [00:03<00:02, 14.20ba/s]
Creating parquet from Arrow format: 66%|████▌ | 56/85 [00:03<00:02, 14.23ba/s]
Creating parquet from Arrow format: 68%|████▊ | 58/85 [00:04<00:01, 14.36ba/s]
Creating parquet from Arrow format: 71%|████▉ | 60/85 [00:04<00:01, 14.48ba/s]
Creating parquet from Arrow format: 73%|█████ | 62/85 [00:04<00:01, 14.47ba/s]
Creating parquet from Arrow format: 75%|█████▎ | 64/85 [00:04<00:01, 14.45ba/s]
Creating parquet from Arrow format: 78%|█████▍ | 66/85 [00:04<00:01, 14.46ba/s]
Creating parquet from Arrow format: 80%|█████▌ | 68/85 [00:04<00:01, 14.36ba/s]
Creating parquet from Arrow format: 82%|█████▊ | 70/85 [00:04<00:01, 14.40ba/s]
Creating parquet from Arrow format: 85%|█████▉ | 72/85 [00:05<00:00, 14.46ba/s]
Creating parquet from Arrow format: 87%|██████ | 74/85 [00:05<00:00, 14.54ba/s]
Creating parquet from Arrow format: 89%|██████▎| 76/85 [00:05<00:00, 14.60ba/s]
Creating parquet from Arrow format: 92%|██████▍| 78/85 [00:05<00:00, 14.55ba/s]
Creating parquet from Arrow format: 94%|██████▌| 80/85 [00:05<00:00, 14.45ba/s]
Creating parquet from Arrow format: 96%|██████▊| 82/85 [00:05<00:00, 14.43ba/s]
Creating parquet from Arrow format: 100%|███████| 85/85 [00:05<00:00, 14.37ba/s]
Upload 1 LFS files: 0%| | 0/1 [00:00<?, ?it/s]
Upload 1 LFS files: 100%|█████████████████████████| 1/1 [00:11<00:00, 11.10s/it]
Pushing dataset shards to the dataset hub: 80%|█▌| 4/5 [01:03<00:15, 15.90s/it]
Creating parquet from Arrow format: 0%| | 0/85 [00:00<?, ?ba/s]
Creating parquet from Arrow format: 2%|▏ | 2/85 [00:00<00:06, 12.39ba/s]
Creating parquet from Arrow format: 5%|▍ | 4/85 [00:00<00:05, 13.60ba/s]
Creating parquet from Arrow format: 7%|▌ | 6/85 [00:00<00:05, 13.99ba/s]
Creating parquet from Arrow format: 9%|▊ | 8/85 [00:00<00:05, 14.07ba/s]
Creating parquet from Arrow format: 12%|▊ | 10/85 [00:00<00:05, 14.21ba/s]
Creating parquet from Arrow format: 14%|▉ | 12/85 [00:00<00:05, 14.13ba/s]
Creating parquet from Arrow format: 16%|█▏ | 14/85 [00:00<00:04, 14.20ba/s]
Creating parquet from Arrow format: 19%|█▎ | 16/85 [00:01<00:04, 14.38ba/s]
Creating parquet from Arrow format: 21%|█▍ | 18/85 [00:01<00:04, 14.46ba/s]
Creating parquet from Arrow format: 24%|█▋ | 20/85 [00:01<00:04, 14.16ba/s]
Creating parquet from Arrow format: 26%|█▊ | 22/85 [00:01<00:04, 14.16ba/s]
Creating parquet from Arrow format: 28%|█▉ | 24/85 [00:01<00:04, 14.21ba/s]
Creating parquet from Arrow format: 31%|██▏ | 26/85 [00:01<00:04, 14.28ba/s]
Creating parquet from Arrow format: 33%|██▎ | 28/85 [00:01<00:04, 13.99ba/s]
Creating parquet from Arrow format: 35%|██▍ | 30/85 [00:02<00:03, 14.04ba/s]
Creating parquet from Arrow format: 38%|██▋ | 32/85 [00:02<00:03, 14.12ba/s]
Creating parquet from Arrow format: 40%|██▊ | 34/85 [00:02<00:03, 14.20ba/s]
Creating parquet from Arrow format: 42%|██▉ | 36/85 [00:02<00:03, 14.16ba/s]
Creating parquet from Arrow format: 45%|███▏ | 38/85 [00:02<00:03, 14.30ba/s]
Creating parquet from Arrow format: 47%|███▎ | 40/85 [00:02<00:03, 14.33ba/s]
Creating parquet from Arrow format: 49%|███▍ | 42/85 [00:02<00:02, 14.39ba/s]
Creating parquet from Arrow format: 52%|███▌ | 44/85 [00:03<00:02, 14.34ba/s]
Creating parquet from Arrow format: 54%|███▊ | 46/85 [00:03<00:02, 14.39ba/s]
Creating parquet from Arrow format: 56%|███▉ | 48/85 [00:03<00:02, 14.42ba/s]
Creating parquet from Arrow format: 59%|████ | 50/85 [00:03<00:02, 14.42ba/s]
Creating parquet from Arrow format: 61%|████▎ | 52/85 [00:03<00:02, 14.47ba/s]
Creating parquet from Arrow format: 64%|████▍ | 54/85 [00:03<00:02, 14.48ba/s]
Creating parquet from Arrow format: 66%|████▌ | 56/85 [00:03<00:02, 14.48ba/s]
Creating parquet from Arrow format: 68%|████▊ | 58/85 [00:04<00:01, 14.49ba/s]
Creating parquet from Arrow format: 71%|████▉ | 60/85 [00:04<00:01, 14.41ba/s]
Creating parquet from Arrow format: 73%|█████ | 62/85 [00:04<00:01, 14.47ba/s]
Creating parquet from Arrow format: 75%|█████▎ | 64/85 [00:04<00:01, 14.48ba/s]
Creating parquet from Arrow format: 78%|█████▍ | 66/85 [00:04<00:01, 14.48ba/s]
Creating parquet from Arrow format: 80%|█████▌ | 68/85 [00:04<00:01, 14.55ba/s]
Creating parquet from Arrow format: 82%|█████▊ | 70/85 [00:04<00:01, 14.57ba/s]
Creating parquet from Arrow format: 85%|█████▉ | 72/85 [00:05<00:00, 14.46ba/s]
Creating parquet from Arrow format: 87%|██████ | 74/85 [00:05<00:00, 14.21ba/s]
Creating parquet from Arrow format: 89%|██████▎| 76/85 [00:05<00:00, 14.13ba/s]
Creating parquet from Arrow format: 92%|██████▍| 78/85 [00:05<00:00, 14.22ba/s]
Creating parquet from Arrow format: 94%|██████▌| 80/85 [00:05<00:00, 14.32ba/s]
Creating parquet from Arrow format: 96%|██████▊| 82/85 [00:05<00:00, 14.38ba/s]
Creating parquet from Arrow format: 100%|███████| 85/85 [00:05<00:00, 14.41ba/s]
Upload 1 LFS files: 0%| | 0/1 [00:00<?, ?it/s]
Upload 1 LFS files: 100%|█████████████████████████| 1/1 [00:07<00:00, 7.22s/it]
Pushing dataset shards to the dataset hub: 100%|██| 5/5 [01:17<00:00, 15.40s/it]
INFO:root:Finished preparing dataset. Exiting...
In [ ]:
# For axolotl push_dataset_to_hub
import os
from huggingface_hub import notebook_login, login
# Colab:
# notebook_login()
# RunPod:
login(os.environ.get("HUGGINGFACE_TOKEN"), add_to_git_credential=True)
Token is valid (permission: write). Your token has been saved in your configured git credential helpers (store). Your token has been saved to /root/.cache/huggingface/token Login successful
In [ ]:
!pip install git+https://github.com/cphyc/jupyter-notify.git
%load_ext jupyternotify
%autonotify -a 30 -o
Collecting git+https://github.com/cphyc/jupyter-notify.git
Cloning https://github.com/cphyc/jupyter-notify.git to /tmp/pip-req-build-h4fnet0c
Running command git clone --filter=blob:none --quiet https://github.com/cphyc/jupyter-notify.git /tmp/pip-req-build-h4fnet0c
Resolved https://github.com/cphyc/jupyter-notify.git to commit 8cff958cbd3f00f7e4eb59b457f9f915e2ddff37
Preparing metadata (setup.py) ... done
Requirement already satisfied: ipython in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from jupyternotify==0.1.15) (8.14.0)
Collecting jupyter (from jupyternotify==0.1.15)
Downloading jupyter-1.0.0-py2.py3-none-any.whl (2.7 kB)
Requirement already satisfied: backcall in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from ipython->jupyternotify==0.1.15) (0.2.0)
Requirement already satisfied: decorator in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from ipython->jupyternotify==0.1.15) (5.1.1)
Requirement already satisfied: jedi>=0.16 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from ipython->jupyternotify==0.1.15) (0.18.2)
Requirement already satisfied: matplotlib-inline in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from ipython->jupyternotify==0.1.15) (0.1.6)
Requirement already satisfied: pickleshare in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from ipython->jupyternotify==0.1.15) (0.7.5)
Requirement already satisfied: prompt-toolkit!=3.0.37,<3.1.0,>=3.0.30 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from ipython->jupyternotify==0.1.15) (3.0.38)
Requirement already satisfied: pygments>=2.4.0 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from ipython->jupyternotify==0.1.15) (2.15.1)
Requirement already satisfied: stack-data in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from ipython->jupyternotify==0.1.15) (0.6.2)
Requirement already satisfied: traitlets>=5 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from ipython->jupyternotify==0.1.15) (5.9.0)
Requirement already satisfied: typing-extensions in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from ipython->jupyternotify==0.1.15) (4.6.3)
Requirement already satisfied: pexpect>4.3 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from ipython->jupyternotify==0.1.15) (4.8.0)
Requirement already satisfied: notebook in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from jupyter->jupyternotify==0.1.15) (6.5.4)
Collecting qtconsole (from jupyter->jupyternotify==0.1.15)
Downloading qtconsole-5.4.3-py3-none-any.whl (121 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 121.9/121.9 kB 10.0 MB/s eta 0:00:00
Collecting jupyter-console (from jupyter->jupyternotify==0.1.15)
Downloading jupyter_console-6.6.3-py3-none-any.whl (24 kB)
Requirement already satisfied: nbconvert in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from jupyter->jupyternotify==0.1.15) (7.5.0)
Requirement already satisfied: ipykernel in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from jupyter->jupyternotify==0.1.15) (6.23.2)
Requirement already satisfied: ipywidgets in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from jupyter->jupyternotify==0.1.15) (8.0.6)
Requirement already satisfied: parso<0.9.0,>=0.8.0 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from jedi>=0.16->ipython->jupyternotify==0.1.15) (0.8.3)
Requirement already satisfied: ptyprocess>=0.5 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from pexpect>4.3->ipython->jupyternotify==0.1.15) (0.7.0)
Requirement already satisfied: wcwidth in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from prompt-toolkit!=3.0.37,<3.1.0,>=3.0.30->ipython->jupyternotify==0.1.15) (0.2.6)
Requirement already satisfied: comm>=0.1.1 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from ipykernel->jupyter->jupyternotify==0.1.15) (0.1.3)
Requirement already satisfied: debugpy>=1.6.5 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from ipykernel->jupyter->jupyternotify==0.1.15) (1.6.7)
Requirement already satisfied: jupyter-client>=6.1.12 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from ipykernel->jupyter->jupyternotify==0.1.15) (8.2.0)
Requirement already satisfied: jupyter-core!=5.0.*,>=4.12 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from ipykernel->jupyter->jupyternotify==0.1.15) (5.3.1)
Requirement already satisfied: nest-asyncio in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from ipykernel->jupyter->jupyternotify==0.1.15) (1.5.6)
Requirement already satisfied: packaging in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from ipykernel->jupyter->jupyternotify==0.1.15) (23.1)
Requirement already satisfied: psutil in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from ipykernel->jupyter->jupyternotify==0.1.15) (5.9.5)
Requirement already satisfied: pyzmq>=20 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from ipykernel->jupyter->jupyternotify==0.1.15) (25.1.0)
Requirement already satisfied: tornado>=6.1 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from ipykernel->jupyter->jupyternotify==0.1.15) (6.3.2)
Requirement already satisfied: widgetsnbextension~=4.0.7 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from ipywidgets->jupyter->jupyternotify==0.1.15) (4.0.7)
Requirement already satisfied: jupyterlab-widgets~=3.0.7 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from ipywidgets->jupyter->jupyternotify==0.1.15) (3.0.7)
Requirement already satisfied: beautifulsoup4 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from nbconvert->jupyter->jupyternotify==0.1.15) (4.12.2)
Requirement already satisfied: bleach!=5.0.0 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from nbconvert->jupyter->jupyternotify==0.1.15) (6.0.0)
Requirement already satisfied: defusedxml in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from nbconvert->jupyter->jupyternotify==0.1.15) (0.7.1)
Requirement already satisfied: importlib-metadata>=3.6 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from nbconvert->jupyter->jupyternotify==0.1.15) (6.6.0)
Requirement already satisfied: jinja2>=3.0 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from nbconvert->jupyter->jupyternotify==0.1.15) (3.1.2)
Requirement already satisfied: jupyterlab-pygments in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from nbconvert->jupyter->jupyternotify==0.1.15) (0.2.2)
Requirement already satisfied: markupsafe>=2.0 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from nbconvert->jupyter->jupyternotify==0.1.15) (2.1.3)
Requirement already satisfied: mistune<3,>=2.0.3 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from nbconvert->jupyter->jupyternotify==0.1.15) (2.0.5)
Requirement already satisfied: nbclient>=0.5.0 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from nbconvert->jupyter->jupyternotify==0.1.15) (0.8.0)
Requirement already satisfied: nbformat>=5.7 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from nbconvert->jupyter->jupyternotify==0.1.15) (5.9.0)
Requirement already satisfied: pandocfilters>=1.4.1 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from nbconvert->jupyter->jupyternotify==0.1.15) (1.5.0)
Requirement already satisfied: tinycss2 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from nbconvert->jupyter->jupyternotify==0.1.15) (1.2.1)
Requirement already satisfied: argon2-cffi in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from notebook->jupyter->jupyternotify==0.1.15) (21.3.0)
Requirement already satisfied: ipython-genutils in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from notebook->jupyter->jupyternotify==0.1.15) (0.2.0)
Requirement already satisfied: Send2Trash>=1.8.0 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from notebook->jupyter->jupyternotify==0.1.15) (1.8.2)
Requirement already satisfied: terminado>=0.8.3 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from notebook->jupyter->jupyternotify==0.1.15) (0.17.1)
Requirement already satisfied: prometheus-client in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from notebook->jupyter->jupyternotify==0.1.15) (0.17.0)
Requirement already satisfied: nbclassic>=0.4.7 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from notebook->jupyter->jupyternotify==0.1.15) (1.0.0)
Collecting qtpy>=2.0.1 (from qtconsole->jupyter->jupyternotify==0.1.15)
Downloading QtPy-2.3.1-py3-none-any.whl (84 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 84.9/84.9 kB 36.4 MB/s eta 0:00:00
Requirement already satisfied: executing>=1.2.0 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from stack-data->ipython->jupyternotify==0.1.15) (1.2.0)
Requirement already satisfied: asttokens>=2.1.0 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from stack-data->ipython->jupyternotify==0.1.15) (2.2.1)
Requirement already satisfied: pure-eval in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from stack-data->ipython->jupyternotify==0.1.15) (0.2.2)
Requirement already satisfied: six in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from asttokens>=2.1.0->stack-data->ipython->jupyternotify==0.1.15) (1.16.0)
Requirement already satisfied: webencodings in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from bleach!=5.0.0->nbconvert->jupyter->jupyternotify==0.1.15) (0.5.1)
Requirement already satisfied: zipp>=0.5 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from importlib-metadata>=3.6->nbconvert->jupyter->jupyternotify==0.1.15) (3.15.0)
Requirement already satisfied: python-dateutil>=2.8.2 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from jupyter-client>=6.1.12->ipykernel->jupyter->jupyternotify==0.1.15) (2.8.2)
Requirement already satisfied: platformdirs>=2.5 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from jupyter-core!=5.0.*,>=4.12->ipykernel->jupyter->jupyternotify==0.1.15) (3.6.0)
Requirement already satisfied: jupyter-server>=1.8 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from nbclassic>=0.4.7->notebook->jupyter->jupyternotify==0.1.15) (2.6.0)
Requirement already satisfied: notebook-shim>=0.2.3 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from nbclassic>=0.4.7->notebook->jupyter->jupyternotify==0.1.15) (0.2.3)
Requirement already satisfied: fastjsonschema in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from nbformat>=5.7->nbconvert->jupyter->jupyternotify==0.1.15) (2.17.1)
Requirement already satisfied: jsonschema>=2.6 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from nbformat>=5.7->nbconvert->jupyter->jupyternotify==0.1.15) (4.17.3)
Requirement already satisfied: argon2-cffi-bindings in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from argon2-cffi->notebook->jupyter->jupyternotify==0.1.15) (21.2.0)
Requirement already satisfied: soupsieve>1.2 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from beautifulsoup4->nbconvert->jupyter->jupyternotify==0.1.15) (2.4.1)
Requirement already satisfied: attrs>=17.4.0 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from jsonschema>=2.6->nbformat>=5.7->nbconvert->jupyter->jupyternotify==0.1.15) (23.1.0)
Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from jsonschema>=2.6->nbformat>=5.7->nbconvert->jupyter->jupyternotify==0.1.15) (0.19.3)
Requirement already satisfied: anyio>=3.1.0 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from jupyter-server>=1.8->nbclassic>=0.4.7->notebook->jupyter->jupyternotify==0.1.15) (3.7.0)
Requirement already satisfied: jupyter-events>=0.6.0 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from jupyter-server>=1.8->nbclassic>=0.4.7->notebook->jupyter->jupyternotify==0.1.15) (0.6.3)
Requirement already satisfied: jupyter-server-terminals in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from jupyter-server>=1.8->nbclassic>=0.4.7->notebook->jupyter->jupyternotify==0.1.15) (0.4.4)
Requirement already satisfied: overrides in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from jupyter-server>=1.8->nbclassic>=0.4.7->notebook->jupyter->jupyternotify==0.1.15) (7.3.1)
Requirement already satisfied: websocket-client in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from jupyter-server>=1.8->nbclassic>=0.4.7->notebook->jupyter->jupyternotify==0.1.15) (1.6.0)
Requirement already satisfied: cffi>=1.0.1 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from argon2-cffi-bindings->argon2-cffi->notebook->jupyter->jupyternotify==0.1.15) (1.15.1)
Requirement already satisfied: idna>=2.8 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from anyio>=3.1.0->jupyter-server>=1.8->nbclassic>=0.4.7->notebook->jupyter->jupyternotify==0.1.15) (3.4)
Requirement already satisfied: sniffio>=1.1 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from anyio>=3.1.0->jupyter-server>=1.8->nbclassic>=0.4.7->notebook->jupyter->jupyternotify==0.1.15) (1.3.0)
Requirement already satisfied: exceptiongroup in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from anyio>=3.1.0->jupyter-server>=1.8->nbclassic>=0.4.7->notebook->jupyter->jupyternotify==0.1.15) (1.1.1)
Requirement already satisfied: pycparser in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from cffi>=1.0.1->argon2-cffi-bindings->argon2-cffi->notebook->jupyter->jupyternotify==0.1.15) (2.21)
Requirement already satisfied: python-json-logger>=2.0.4 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from jupyter-events>=0.6.0->jupyter-server>=1.8->nbclassic>=0.4.7->notebook->jupyter->jupyternotify==0.1.15) (2.0.7)
Requirement already satisfied: pyyaml>=5.3 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from jupyter-events>=0.6.0->jupyter-server>=1.8->nbclassic>=0.4.7->notebook->jupyter->jupyternotify==0.1.15) (6.0)
Requirement already satisfied: rfc3339-validator in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from jupyter-events>=0.6.0->jupyter-server>=1.8->nbclassic>=0.4.7->notebook->jupyter->jupyternotify==0.1.15) (0.1.4)
Requirement already satisfied: rfc3986-validator>=0.1.1 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from jupyter-events>=0.6.0->jupyter-server>=1.8->nbclassic>=0.4.7->notebook->jupyter->jupyternotify==0.1.15) (0.1.1)
Requirement already satisfied: fqdn in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from jsonschema>=2.6->nbformat>=5.7->nbconvert->jupyter->jupyternotify==0.1.15) (1.5.1)
Requirement already satisfied: isoduration in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from jsonschema>=2.6->nbformat>=5.7->nbconvert->jupyter->jupyternotify==0.1.15) (20.11.0)
Requirement already satisfied: jsonpointer>1.13 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from jsonschema>=2.6->nbformat>=5.7->nbconvert->jupyter->jupyternotify==0.1.15) (2.4)
Requirement already satisfied: uri-template in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from jsonschema>=2.6->nbformat>=5.7->nbconvert->jupyter->jupyternotify==0.1.15) (1.2.0)
Requirement already satisfied: webcolors>=1.11 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from jsonschema>=2.6->nbformat>=5.7->nbconvert->jupyter->jupyternotify==0.1.15) (1.13)
Requirement already satisfied: arrow>=0.15.0 in /root/miniconda3/envs/py3.9/lib/python3.9/site-packages (from isoduration->jsonschema>=2.6->nbformat>=5.7->nbconvert->jupyter->jupyternotify==0.1.15) (1.2.3)
Building wheels for collected packages: jupyternotify
Building wheel for jupyternotify (setup.py) ... done
Created wheel for jupyternotify: filename=jupyternotify-0.1.15-py3-none-any.whl size=9253 sha256=63a599580a2a8a14cf887dead0c12f907b7e800d7a66eb5410b87cb08093bb7b
Stored in directory: /tmp/pip-ephem-wheel-cache-34r2ggd_/wheels/ac/9c/45/690acc0737588c27d965beb8db28ae0946c87dfd8186ba8d49
Successfully built jupyternotify
Installing collected packages: qtpy, qtconsole, jupyter-console, jupyter, jupyternotify
Successfully installed jupyter-1.0.0 jupyter-console-6.6.3 jupyternotify-0.1.15 qtconsole-5.4.3 qtpy-2.3.1
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
In [ ]:
%%writefile examples/falcon/config-40b-qlora.yml
# 1b: tiiuae/falcon-rw-1b
# 7b: tiiuae/falcon-7b
# 40b: tiiuae/falcon-40b
base_model: tiiuae/falcon-40b
base_model_config: tiiuae/falcon-40b
# required by falcon custom model code: https://huggingface.co/tiiuae/falcon-7b/tree/main
trust_remote_code: true
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer
load_in_8bit: false
# enable 4bit for QLoRA
load_in_4bit: true
gptq: false
strict: false
push_dataset_to_hub: utensil
hf_use_auth_token: true
datasets:
- path: winglian/evals
data_files:
- hf/ARC-Challenge.jsonl
- hf/ARC-Easy.jsonl
- hf/riddle_sense.jsonl
- hf/piqa.jsonl
type: explainchoice:chat
- path: winglian/evals
data_files:
- hf/gsm8k.jsonl
- hf/winogrande.jsonl
type: alpaca_chat.load_qa
- path: winglian/evals
data_files:
- custom/n_task.jsonl
- custom/misconceptions.jsonl
- custom/context_insensitivity.jsonl
type: alpaca_chat
- path: camel-ai/math
type: alpaca_chat.load_camel_ai
- path: camel-ai/biology
type: alpaca_chat.load_camel_ai
- path: camel-ai/physics
type: alpaca_chat.load_camel_ai
- path: camel-ai/chemistry
type: alpaca_chat.load_camel_ai
- path: winglian/evals
data_files:
- custom/in_context_qa.jsonl
type: context_qa
- path: winglian/evals
data_files:
- custom/in_context_qa.jsonl
type: context_qa.load_404
- path: winglian/evals
data_files:
- custom/jokes_explained_500up.jsonl
type: sharegpt_jokes
- path: winglian/evals
data_files:
- custom/classify-self-chat.sharegpt.jsonl
- custom/coding-self-chat.sharegpt.jsonl
- custom/prose-gpt4.sharegpt.jsonl
- custom/prose-rewrite-gpt4.sharegpt.jsonl
type: sharegpt_simple.load_role
- path: winglian/evals
data_files:
- openai/tldr.jsonl
type: summarizetldr:chat
- path: winglian/evals
data_files:
- hellaswag/hellaswag.jsonl
type: explainchoice:chat
- path: metaeval/ScienceQA_text_only
type: concisechoice:chat
- path: teknium/GPT4-LLM-Cleaned
type: alpaca_chat
- path: teknium/GPTeacher-General-Instruct
data_files: gpt4-instruct-similarity-0.6-dataset.json
type: gpteacher:chat
- path: QingyiSi/Alpaca-CoT
data_files:
- Chain-of-Thought/formatted_cot_data/aqua_train.json
- Chain-of-Thought/formatted_cot_data/creak_train.json
- Chain-of-Thought/formatted_cot_data/ecqa_train.json
- Chain-of-Thought/formatted_cot_data/esnli_train.json
- Chain-of-Thought/formatted_cot_data/qasc_train.json
- Chain-of-Thought/formatted_cot_data/qed_train.json
- Chain-of-Thought/formatted_cot_data/sensemaking_train.json
- Chain-of-Thought/formatted_cot_data/strategyqa_train.json
- GPTeacher/Roleplay/formatted_roleplay-similarity_0.6-instruct-dataset.json
type: alpaca_chat
- path: ehartford/WizardLM_alpaca_evol_instruct_70k_unfiltered
type: alpaca_chat
- path: ehartford/wizard_vicuna_70k_unfiltered
type: sharegpt:chat
dataset_prepared_path: last_run_prepared
val_set_size: 0.01
# enable QLoRA
adapter: qlora
lora_model_dir:
sequence_len: 2048
max_packed_sequence_len: 2048
# hyperparameters from QLoRA paper Appendix B.2
# "We find hyperparameters to be largely robust across datasets"
lora_r: 64
lora_alpha: 16
# 0.1 for models up to 13B
# 0.05 for 33B and 65B models
lora_dropout: 0.05
# add LoRA modules on all linear layers of the base model
lora_target_modules:
lora_target_linear: true
lora_fan_in_fan_out:
wandb_project: falcon-qlora
wandb_watch:
wandb_run_id:
wandb_log_model:
output_dir: /content/axolotl-trained/falcon-qlora-40b-minotaur/
# QLoRA paper Table 9
# - 16 for 7b & 13b
# - 32 for 33b, 64 for 64b
# Max size tested on A6000
# - 7b: 40
# - 40b: 4
# decrease if OOM, increase for max VRAM utilization
micro_batch_size: 4
gradient_accumulation_steps: 1
num_epochs: 3
# Optimizer for QLoRA
optimizer: adamw_bnb_8bit
torchdistx_path:
lr_scheduler: cosine
# QLoRA paper Table 9
# - 2e-4 for 7b & 13b
# - 1e-4 for 33b & 64b
learning_rate: 0.0002
train_on_inputs: false
group_by_length: false
bf16: true
fp16: false
tf32: true
gradient_checkpointing: true
# stop training after this many evaluation losses have increased in a row
# https://huggingface.co/transformers/v4.2.2/_modules/transformers/trainer_callback.html#EarlyStoppingCallback
# early_stopping_patience: 3
resume_from_checkpoint:
auto_resume_from_checkpoints: true
local_rank:
logging_steps: 1
xformers_attention: true
flash_attention:
gptq_groupsize:
gptq_model_v1:
warmup_steps: 10
eval_steps: 5
save_steps: 10
debug:
deepspeed:
weight_decay: 0.01
adam_beta1:
adam_beta2: 0.999
adam_epsilon:
# Gradient clipping max norm
max_grad_norm: 0.3
fsdp:
fsdp_config:
special_tokens:
pad_token: "<|endoftext|>"
bos_token: ">>ABSTRACT<<"
eos_token: "<|endoftext|>"
Overwriting examples/falcon/config-40b-qlora.yml
In [ ]:
!python scripts/finetune.py examples/falcon/config-40b-qlora.yml --prepare_ds_only
Setting ds_accelerator to cuda (auto detect)
===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please run
python -m bitsandbytes
and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================
bin /root/miniconda3/envs/py3.9/lib/python3.9/site-packages/bitsandbytes-0.39.0-py3.9.egg/bitsandbytes/libbitsandbytes_cuda118.so
/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/bitsandbytes-0.39.0-py3.9.egg/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/usr/local/nvidia/lib'), PosixPath('/usr/local/nvidia/lib64')}
warn(msg)
/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/bitsandbytes-0.39.0-py3.9.egg/bitsandbytes/cuda_setup/main.py:149: UserWarning: /usr/local/nvidia/lib:/usr/local/nvidia/lib64 did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths...
warn(msg)
/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/bitsandbytes-0.39.0-py3.9.egg/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('module'), PosixPath('//matplotlib_inline.backend_inline')}
warn(msg)
/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/bitsandbytes-0.39.0-py3.9.egg/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDvnE4umHheXhWsDJbbukYvvyc47/mC4z8syS93btA72T90WDrQagOy5O+DrhdXOvr5i/JwsTlAImy57eLRrtRFOrQq73jyi7Dzo0tvrAiNLVgX2q2dFLoplRyXDXiVYLPmPieMWQOeUCLeSb8FC5zzllcocZwjMXpxScDerZqnlAR0ccpSkGyKIod4ZMkn/29A/C5kHEb/wT8cOAq+MWJ/2okZZgbiR0AMV4DynAkrtcx9JnJnTs9chiMyH+dyCS42Ai24sHWJBkQo6TfxXkyKo9GOpu3Y2WLgrHyaot9Lk5mA1mujyIWdlReD2nvjeCQKjl3KW3xZ73m4nD97MydWSWoJfEWlr+VZvk8EWsZk3CYLZCIBLdod6xXJJ0DD0pvTIq11c8VB7XkgVjapuU/sC8M6HFzHW/NBeE+xX/txPkZkIGqrnxeQ0AtBXdN9ukyNGhGzTkPYJNliiYpY0dCvVuz/BJ2FawFTQGnD1EHOenUCRajREFGCbKoYZqi40j8= utensil@Utensils-MacBook-Pro.local')}
warn(msg)
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/bitsandbytes-0.39.0-py3.9.egg/bitsandbytes/cuda_setup/main.py:149: UserWarning: Found duplicate ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] files: {PosixPath('/usr/local/cuda/lib64/libcudart.so.11.0'), PosixPath('/usr/local/cuda/lib64/libcudart.so')}.. We'll flip a coin and try one of these, in order to fail forward.
Either way, this might cause trouble in the future:
If you get `CUDA error: invalid device function` errors, the above might be the cause and the solution is to make sure only one ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] in the paths that we search based on your env.
warn(msg)
CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so.11.0
CUDA SETUP: Highest compute capability among GPUs detected: 8.6
CUDA SETUP: Detected CUDA version 118
CUDA SETUP: Loading binary /root/miniconda3/envs/py3.9/lib/python3.9/site-packages/bitsandbytes-0.39.0-py3.9.egg/bitsandbytes/libbitsandbytes_cuda118.so...
WARNING:root:`trust_remote_code` is set to true. Please make sure that you reviewed the remote code/model.
INFO:root:loading tokenizer... tiiuae/falcon-40b
Using bos_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using unk_token, but it is not set yet.
INFO:root:Checking for packed prepared dataset from hub... utensil/216aa4e2be3974d0a1d25d8c19db01a4
INFO:root:Loading prepared packed dataset from disk at last_run_prepared/216aa4e2be3974d0a1d25d8c19db01a4...
INFO:root:Prepared packed dataset loaded from disk...
INFO:root:Saving packed prepared dataset with push_to_hub... utensil/216aa4e2be3974d0a1d25d8c19db01a4
Pushing dataset shards to the dataset hub: 0%| | 0/5 [00:00<?, ?it/s]
Creating parquet from Arrow format: 0%| | 0/20 [00:00<?, ?ba/s]
Creating parquet from Arrow format: 5%|▍ | 1/20 [00:00<00:01, 9.61ba/s]
Creating parquet from Arrow format: 15%|█▏ | 3/20 [00:00<00:01, 10.08ba/s]
Creating parquet from Arrow format: 25%|██ | 5/20 [00:00<00:01, 10.47ba/s]
Creating parquet from Arrow format: 35%|██▊ | 7/20 [00:00<00:01, 10.65ba/s]
Creating parquet from Arrow format: 45%|███▌ | 9/20 [00:00<00:01, 10.69ba/s]
Creating parquet from Arrow format: 55%|███▊ | 11/20 [00:01<00:00, 10.83ba/s]
Creating parquet from Arrow format: 65%|████▌ | 13/20 [00:01<00:00, 11.03ba/s]
Creating parquet from Arrow format: 75%|█████▎ | 15/20 [00:01<00:00, 10.96ba/s]
Creating parquet from Arrow format: 85%|█████▉ | 17/20 [00:01<00:00, 11.05ba/s]
Creating parquet from Arrow format: 100%|███████| 20/20 [00:01<00:00, 11.39ba/s]
Upload 1 LFS files: 0%| | 0/1 [00:00<?, ?it/s]
Upload 1 LFS files: 100%|█████████████████████████| 1/1 [00:06<00:00, 6.37s/it]
^C
Pushing dataset shards to the dataset hub: 0%| | 0/5 [02:15<?, ?it/s]
Traceback (most recent call last):
File "/workspace/axolotl/scripts/finetune.py", line 352, in <module>
fire.Fire(train)
File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/fire/core.py", line 141, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/fire/core.py", line 475, in _Fire
component, remaining_args = _CallAndUpdateTrace(
File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "/workspace/axolotl/scripts/finetune.py", line 222, in train
train_dataset, eval_dataset = load_prepare_datasets(
File "/workspace/axolotl/src/axolotl/utils/data.py", line 338, in load_prepare_datasets
dataset.push_to_hub(
File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 5396, in push_to_hub
repo_id, split, uploaded_size, dataset_nbytes, repo_files, deleted_size = self._push_parquet_shards_to_hub(
File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 5291, in _push_parquet_shards_to_hub
_retry(
File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/datasets/utils/file_utils.py", line 281, in _retry
return func(*func_args, **func_kwargs)
File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
return fn(*args, **kwargs)
File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/huggingface_hub/hf_api.py", line 826, in _inner
return fn(self, *args, **kwargs)
File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/huggingface_hub/hf_api.py", line 3200, in upload_file
operation = CommitOperationAdd(
File "<string>", line 5, in __init__
File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/huggingface_hub/_commit_api.py", line 129, in __post_init__
self.upload_info = UploadInfo.from_bytes(self.path_or_fileobj)
File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/huggingface_hub/lfs.py", line 81, in from_bytes
sha = sha256(data).digest()
KeyboardInterrupt
In [ ]:
!python scripts/finetune.py examples/falcon/config-40b-qlora.yml --prepare_ds_only
Setting ds_accelerator to cuda (auto detect)
===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please run
python -m bitsandbytes
and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================
bin /root/miniconda3/envs/py3.9/lib/python3.9/site-packages/bitsandbytes-0.39.0-py3.9.egg/bitsandbytes/libbitsandbytes_cuda118.so
/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/bitsandbytes-0.39.0-py3.9.egg/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/usr/local/nvidia/lib64'), PosixPath('/usr/local/nvidia/lib')}
warn(msg)
/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/bitsandbytes-0.39.0-py3.9.egg/bitsandbytes/cuda_setup/main.py:149: UserWarning: /usr/local/nvidia/lib:/usr/local/nvidia/lib64 did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths...
warn(msg)
/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/bitsandbytes-0.39.0-py3.9.egg/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('module'), PosixPath('//matplotlib_inline.backend_inline')}
warn(msg)
/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/bitsandbytes-0.39.0-py3.9.egg/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDvnE4umHheXhWsDJbbukYvvyc47/mC4z8syS93btA72T90WDrQagOy5O+DrhdXOvr5i/JwsTlAImy57eLRrtRFOrQq73jyi7Dzo0tvrAiNLVgX2q2dFLoplRyXDXiVYLPmPieMWQOeUCLeSb8FC5zzllcocZwjMXpxScDerZqnlAR0ccpSkGyKIod4ZMkn/29A/C5kHEb/wT8cOAq+MWJ/2okZZgbiR0AMV4DynAkrtcx9JnJnTs9chiMyH+dyCS42Ai24sHWJBkQo6TfxXkyKo9GOpu3Y2WLgrHyaot9Lk5mA1mujyIWdlReD2nvjeCQKjl3KW3xZ73m4nD97MydWSWoJfEWlr+VZvk8EWsZk3CYLZCIBLdod6xXJJ0DD0pvTIq11c8VB7XkgVjapuU/sC8M6HFzHW/NBeE+xX/txPkZkIGqrnxeQ0AtBXdN9ukyNGhGzTkPYJNliiYpY0dCvVuz/BJ2FawFTQGnD1EHOenUCRajREFGCbKoYZqi40j8= utensil@Utensils-MacBook-Pro.local')}
warn(msg)
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/bitsandbytes-0.39.0-py3.9.egg/bitsandbytes/cuda_setup/main.py:149: UserWarning: Found duplicate ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] files: {PosixPath('/usr/local/cuda/lib64/libcudart.so.11.0'), PosixPath('/usr/local/cuda/lib64/libcudart.so')}.. We'll flip a coin and try one of these, in order to fail forward.
Either way, this might cause trouble in the future:
If you get `CUDA error: invalid device function` errors, the above might be the cause and the solution is to make sure only one ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] in the paths that we search based on your env.
warn(msg)
CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so.11.0
CUDA SETUP: Highest compute capability among GPUs detected: 8.6
CUDA SETUP: Detected CUDA version 118
CUDA SETUP: Loading binary /root/miniconda3/envs/py3.9/lib/python3.9/site-packages/bitsandbytes-0.39.0-py3.9.egg/bitsandbytes/libbitsandbytes_cuda118.so...
WARNING:root:`trust_remote_code` is set to true. Please make sure that you reviewed the remote code/model.
INFO:root:loading tokenizer... tiiuae/falcon-40b
Using bos_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using unk_token, but it is not set yet.
INFO:root:Checking for packed prepared dataset from hub... utensil/216aa4e2be3974d0a1d25d8c19db01a4
INFO:root:Loading prepared packed dataset from disk at last_run_prepared/216aa4e2be3974d0a1d25d8c19db01a4...
INFO:root:Prepared packed dataset loaded from disk...
INFO:root:Saving packed prepared dataset with push_to_hub... utensil/216aa4e2be3974d0a1d25d8c19db01a4
Pushing dataset shards to the dataset hub: 0%| | 0/5 [00:00<?, ?it/s]
Creating parquet from Arrow format: 0%| | 0/20 [00:00<?, ?ba/s]
Creating parquet from Arrow format: 5%|▍ | 1/20 [00:00<00:01, 9.54ba/s]
Creating parquet from Arrow format: 15%|█▏ | 3/20 [00:00<00:01, 10.13ba/s]
Creating parquet from Arrow format: 25%|██ | 5/20 [00:00<00:01, 10.44ba/s]
Creating parquet from Arrow format: 35%|██▊ | 7/20 [00:00<00:01, 10.70ba/s]
Creating parquet from Arrow format: 45%|███▌ | 9/20 [00:00<00:01, 10.74ba/s]
Creating parquet from Arrow format: 55%|███▊ | 11/20 [00:01<00:00, 10.91ba/s]
Creating parquet from Arrow format: 65%|████▌ | 13/20 [00:01<00:00, 11.00ba/s]
Creating parquet from Arrow format: 75%|█████▎ | 15/20 [00:01<00:00, 11.00ba/s]
Creating parquet from Arrow format: 85%|█████▉ | 17/20 [00:01<00:00, 11.04ba/s]
Creating parquet from Arrow format: 100%|███████| 20/20 [00:01<00:00, 11.41ba/s]
Upload 1 LFS files: 0%| | 0/1 [00:00<?, ?it/s]
Upload 1 LFS files: 100%|█████████████████████████| 1/1 [00:05<00:00, 5.74s/it]
^C
Pushing dataset shards to the dataset hub: 0%| | 0/5 [06:15<?, ?it/s]
Traceback (most recent call last):
File "/workspace/axolotl/scripts/finetune.py", line 352, in <module>
fire.Fire(train)
File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/fire/core.py", line 141, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/fire/core.py", line 475, in _Fire
component, remaining_args = _CallAndUpdateTrace(
File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "/workspace/axolotl/scripts/finetune.py", line 222, in train
train_dataset, eval_dataset = load_prepare_datasets(
File "/workspace/axolotl/src/axolotl/utils/data.py", line 338, in load_prepare_datasets
dataset.push_to_hub(
File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 5396, in push_to_hub
repo_id, split, uploaded_size, dataset_nbytes, repo_files, deleted_size = self._push_parquet_shards_to_hub(
File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 5291, in _push_parquet_shards_to_hub
_retry(
File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/datasets/utils/file_utils.py", line 281, in _retry
return func(*func_args, **func_kwargs)
File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
return fn(*args, **kwargs)
File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/huggingface_hub/hf_api.py", line 826, in _inner
return fn(self, *args, **kwargs)
File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/huggingface_hub/hf_api.py", line 3205, in upload_file
commit_info = self.create_commit(
File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
return fn(*args, **kwargs)
File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/huggingface_hub/hf_api.py", line 826, in _inner
return fn(self, *args, **kwargs)
File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/huggingface_hub/hf_api.py", line 2711, in create_commit
commit_resp = get_session().post(url=commit_url, headers=headers, data=data, params=params)
File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/requests/sessions.py", line 637, in post
return self.request("POST", url, data=data, json=json, **kwargs)
File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/requests/adapters.py", line 486, in send
resp = conn.urlopen(
File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/urllib3/connectionpool.py", line 790, in urlopen
response = self._make_request(
File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/urllib3/connectionpool.py", line 536, in _make_request
response = conn.getresponse()
File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/urllib3/connection.py", line 454, in getresponse
httplib_response = super().getresponse()
File "/root/miniconda3/envs/py3.9/lib/python3.9/http/client.py", line 1377, in getresponse
response.begin()
File "/root/miniconda3/envs/py3.9/lib/python3.9/http/client.py", line 320, in begin
version, status, reason = self._read_status()
File "/root/miniconda3/envs/py3.9/lib/python3.9/http/client.py", line 281, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/root/miniconda3/envs/py3.9/lib/python3.9/socket.py", line 704, in readinto
return self._sock.recv_into(b)
File "/root/miniconda3/envs/py3.9/lib/python3.9/ssl.py", line 1242, in recv_into
return self.read(nbytes, buffer)
File "/root/miniconda3/envs/py3.9/lib/python3.9/ssl.py", line 1100, in read
return self._sslobj.read(len, buffer)
KeyboardInterrupt
In [ ]:
!mv last_run_prepared last_run_prepared_bak
In [ ]:
!python scripts/finetune.py examples/falcon/config-40b-qlora.yml --prepare_ds_only
Setting ds_accelerator to cuda (auto detect)
===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please run
python -m bitsandbytes
and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================
bin /root/miniconda3/envs/py3.9/lib/python3.9/site-packages/bitsandbytes-0.39.0-py3.9.egg/bitsandbytes/libbitsandbytes_cuda118.so
/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/bitsandbytes-0.39.0-py3.9.egg/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/usr/local/nvidia/lib'), PosixPath('/usr/local/nvidia/lib64')}
warn(msg)
/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/bitsandbytes-0.39.0-py3.9.egg/bitsandbytes/cuda_setup/main.py:149: UserWarning: /usr/local/nvidia/lib:/usr/local/nvidia/lib64 did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths...
warn(msg)
/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/bitsandbytes-0.39.0-py3.9.egg/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('module'), PosixPath('//matplotlib_inline.backend_inline')}
warn(msg)
/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/bitsandbytes-0.39.0-py3.9.egg/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDvnE4umHheXhWsDJbbukYvvyc47/mC4z8syS93btA72T90WDrQagOy5O+DrhdXOvr5i/JwsTlAImy57eLRrtRFOrQq73jyi7Dzo0tvrAiNLVgX2q2dFLoplRyXDXiVYLPmPieMWQOeUCLeSb8FC5zzllcocZwjMXpxScDerZqnlAR0ccpSkGyKIod4ZMkn/29A/C5kHEb/wT8cOAq+MWJ/2okZZgbiR0AMV4DynAkrtcx9JnJnTs9chiMyH+dyCS42Ai24sHWJBkQo6TfxXkyKo9GOpu3Y2WLgrHyaot9Lk5mA1mujyIWdlReD2nvjeCQKjl3KW3xZ73m4nD97MydWSWoJfEWlr+VZvk8EWsZk3CYLZCIBLdod6xXJJ0DD0pvTIq11c8VB7XkgVjapuU/sC8M6HFzHW/NBeE+xX/txPkZkIGqrnxeQ0AtBXdN9ukyNGhGzTkPYJNliiYpY0dCvVuz/BJ2FawFTQGnD1EHOenUCRajREFGCbKoYZqi40j8= utensil@Utensils-MacBook-Pro.local')}
warn(msg)
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/bitsandbytes-0.39.0-py3.9.egg/bitsandbytes/cuda_setup/main.py:149: UserWarning: Found duplicate ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] files: {PosixPath('/usr/local/cuda/lib64/libcudart.so.11.0'), PosixPath('/usr/local/cuda/lib64/libcudart.so')}.. We'll flip a coin and try one of these, in order to fail forward.
Either way, this might cause trouble in the future:
If you get `CUDA error: invalid device function` errors, the above might be the cause and the solution is to make sure only one ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] in the paths that we search based on your env.
warn(msg)
CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so.11.0
CUDA SETUP: Highest compute capability among GPUs detected: 8.6
CUDA SETUP: Detected CUDA version 118
CUDA SETUP: Loading binary /root/miniconda3/envs/py3.9/lib/python3.9/site-packages/bitsandbytes-0.39.0-py3.9.egg/bitsandbytes/libbitsandbytes_cuda118.so...
WARNING:root:`trust_remote_code` is set to true. Please make sure that you reviewed the remote code/model.
INFO:root:loading tokenizer... tiiuae/falcon-40b
Using bos_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using unk_token, but it is not set yet.
INFO:root:Checking for packed prepared dataset from hub... utensil/216aa4e2be3974d0a1d25d8c19db01a4
Downloading and preparing dataset parquet/utensil--216aa4e2be3974d0a1d25d8c19db01a4 to /root/.cache/huggingface/datasets/utensil___parquet/utensil--216aa4e2be3974d0a1d25d8c19db01a4-75ee8c7e5c8a0dd6/0.0.0/14a00e99c0d15a23649d0db8944380ac81082d4b021f398733dd84f3a6c569a7...
Downloading data files: 0%| | 0/1 [00:00<?, ?it/s]
Downloading data: 0%| | 0.00/108M [00:00<?, ?B/s]
Downloading data: 0%| | 52.2k/108M [00:00<06:10, 290kB/s]
Downloading data: 0%| | 296k/108M [00:00<01:57, 914kB/s]
Downloading data: 1%|▏ | 989k/108M [00:00<00:37, 2.83MB/s]
Downloading data: 2%|▍ | 1.93M/108M [00:00<00:21, 4.89MB/s]
Downloading data: 3%|▋ | 3.55M/108M [00:00<00:12, 8.36MB/s]
Downloading data: 6%|█▎ | 6.90M/108M [00:00<00:06, 16.0MB/s]
Downloading data: 11%|██▎ | 11.7M/108M [00:00<00:03, 25.6MB/s]
Downloading data: 16%|███▎ | 17.0M/108M [00:00<00:02, 34.2MB/s]
Downloading data: 20%|████▏ | 21.1M/108M [00:01<00:02, 32.9MB/s]
Downloading data: 24%|████▉ | 25.5M/108M [00:01<00:02, 35.8MB/s]
Downloading data: 28%|█████▊ | 29.8M/108M [00:01<00:02, 37.9MB/s]
Downloading data: 32%|██████▋ | 34.4M/108M [00:01<00:01, 40.2MB/s]
Downloading data: 36%|███████▌ | 38.9M/108M [00:01<00:01, 41.6MB/s]
Downloading data: 41%|████████▌ | 44.1M/108M [00:01<00:01, 44.7MB/s]
Downloading data: 45%|█████████▍ | 48.6M/108M [00:01<00:01, 44.4MB/s]
Downloading data: 50%|██████████▍ | 53.3M/108M [00:01<00:01, 45.2MB/s]
Downloading data: 54%|███████████▎ | 57.8M/108M [00:01<00:01, 39.1MB/s]
Downloading data: 58%|████████████▎ | 62.9M/108M [00:02<00:01, 42.2MB/s]
Downloading data: 63%|█████████████▏ | 67.3M/108M [00:02<00:00, 42.7MB/s]
Downloading data: 67%|██████████████ | 71.7M/108M [00:02<00:00, 42.1MB/s]
Downloading data: 71%|██████████████▉ | 76.3M/108M [00:02<00:00, 38.2MB/s]
Downloading data: 75%|███████████████▋ | 80.6M/108M [00:02<00:00, 39.4MB/s]
Downloading data: 79%|████████████████▌ | 84.6M/108M [00:02<00:00, 39.2MB/s]
Downloading data: 84%|█████████████████▌ | 89.8M/108M [00:02<00:00, 42.8MB/s]
Downloading data: 88%|██████████████████▍ | 94.2M/108M [00:02<00:00, 43.0MB/s]
Downloading data: 92%|███████████████████▎ | 98.6M/108M [00:02<00:00, 43.6MB/s]
Downloading data: 96%|█████████████████████ | 103M/108M [00:03<00:00, 38.3MB/s]
Downloading data: 100%|██████████████████████| 108M/108M [00:03<00:00, 34.0MB/s]
Downloading data files: 100%|█████████████████████| 1/1 [00:04<00:00, 4.67s/it]
Extracting data files: 100%|████████████████████| 1/1 [00:00<00:00, 1510.37it/s]
Dataset parquet downloaded and prepared to /root/.cache/huggingface/datasets/utensil___parquet/utensil--216aa4e2be3974d0a1d25d8c19db01a4-75ee8c7e5c8a0dd6/0.0.0/14a00e99c0d15a23649d0db8944380ac81082d4b021f398733dd84f3a6c569a7. Subsequent calls will reuse this data.
100%|████████████████████████████████████████████| 1/1 [00:00<00:00, 564.36it/s]
INFO:root:Finished preparing dataset. Exiting...
In [ ]:
!rm -rf /root/.cache/huggingface/datasets/utensil___parquet/
Chores¶
Clean up¶
In [ ]:
!rm -rf /content/axolotl/last_run_prepared
sharegpt_jokes Patch¶
https://github.com/OpenAccess-AI-Collective/axolotl/pull/192
In [ ]:
%cd /workspace/
/workspace
In [ ]:
!rm -rf axolotl-patch
In [ ]:
!git clone https://github.com/OpenAccess-AI-Collective/axolotl axolotl-patch
Cloning into 'axolotl-patch'... remote: Enumerating objects: 3119, done. remote: Counting objects: 100% (1332/1332), done. remote: Compressing objects: 100% (349/349), done. remote: Total 3119 (delta 1085), reused 1092 (delta 933), pack-reused 1787 Receiving objects: 100% (3119/3119), 1.42 MiB | 16.97 MiB/s, done. Resolving deltas: 100% (1953/1953), done.
In [ ]:
%cd axolotl-patch
/workspace/axolotl-patch
In [ ]:
!git checkout sharegpt-custom-prompt
Branch 'sharegpt-custom-prompt' set up to track remote branch 'sharegpt-custom-prompt' from 'origin'. Switched to a new branch 'sharegpt-custom-prompt'
In [ ]:
!git status
On branch sharegpt-custom-prompt Your branch is up to date with 'origin/sharegpt-custom-prompt'. nothing to commit, working tree clean
In [ ]:
%cd /workspace/
/workspace
In [ ]:
!cp -r axolotl-patch/* axolotl
In [ ]:
%cd axolotl
/workspace/axolotl
In [ ]:
!git status
On branch main Your branch is up to date with 'origin/main'. Changes not staged for commit: (use "git add <file>..." to update what will be committed) (use "git restore <file>..." to discard changes in working directory) modified: FAQS.md modified: README.md modified: docker/Dockerfile modified: docker/Dockerfile-base modified: examples/gptq-lora-7b/config.yml modified: examples/mpt-7b/config.yml modified: requirements.txt modified: scripts/finetune.py modified: scripts/runpod-entrypoint.sh modified: src/axolotl/datasets.py modified: src/axolotl/flash_attn.py modified: src/axolotl/prompt_strategies/alpaca_chat.py modified: src/axolotl/prompters.py modified: src/axolotl/utils/data.py modified: src/axolotl/utils/models.py modified: src/axolotl/utils/trainer.py modified: src/axolotl/utils/validation.py modified: src/axolotl/utils/wandb.py modified: tests/test_prompt_tokenizers.py modified: tests/test_validation.py Untracked files: (use "git add <file>..." to include in what will be committed) docker-compose.yaml examples/cerebras/ examples/falcon/config-7b-qlora.yml examples/falcon/ft.yml examples/falcon/lora.yml examples/falcon/qlora.yml examples/gptj/ examples/huggyllama/ examples/jeopardy-bot/ examples/openllama-3b/ examples/openllama/ examples/pythia/ image/axolotl-badge-web.png src/axolotl/monkeypatch/ src/axolotl/prompt_strategies/context_qa.py src/axolotl/prompt_strategies/sharegpt_jokes.py src/axolotl/prompt_strategies/sharegpt_simple.py tests/fixtures/alpaca/ tests/test_packed_dataset.py no changes added to commit (use "git add" and/or "git commit -a")
In [ ]: