1. Get Axolotl and install its dependencies¶
In [1]:
!git clone https://github.com/OpenAccess-AI-Collective/axolotl
Cloning into 'axolotl'... remote: Enumerating objects: 2526, done. remote: Counting objects: 100% (1226/1226), done. remote: Compressing objects: 100% (380/380), done. remote: Total 2526 (delta 883), reused 1062 (delta 780), pack-reused 1300 Receiving objects: 100% (2526/2526), 1.30 MiB | 11.20 MiB/s, done. Resolving deltas: 100% (1561/1561), done.
In [ ]:
%cd axolotl
/content/axolotl
In [ ]:
!pip3 install -e .
Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/
Obtaining file:///content/axolotl
Preparing metadata (setup.py) ... done
Collecting transformers@ git+https://github.com/huggingface/transformers.git (from axolotl==0.1)
Cloning https://github.com/huggingface/transformers.git to /tmp/pip-install-xxdqu_yh/transformers_a0d0e82c2ad1489a8a38e67638338220
Running command git clone --filter=blob:none --quiet https://github.com/huggingface/transformers.git /tmp/pip-install-xxdqu_yh/transformers_a0d0e82c2ad1489a8a38e67638338220
Resolved https://github.com/huggingface/transformers.git to commit 539e2281cd97c35ef4122757f26c88f44115fa94
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: bitsandbytes>=0.39.0 in /usr/local/lib/python3.10/dist-packages (from axolotl==0.1) (0.39.0)
Requirement already satisfied: accelerate in /usr/local/lib/python3.10/dist-packages (from axolotl==0.1) (0.20.0.dev0)
Requirement already satisfied: addict in /usr/local/lib/python3.10/dist-packages (from axolotl==0.1) (2.4.0)
Requirement already satisfied: fire in /usr/local/lib/python3.10/dist-packages (from axolotl==0.1) (0.5.0)
Requirement already satisfied: PyYAML==6.0 in /usr/local/lib/python3.10/dist-packages (from axolotl==0.1) (6.0)
Requirement already satisfied: datasets in /usr/local/lib/python3.10/dist-packages (from axolotl==0.1) (2.12.0)
Requirement already satisfied: sentencepiece in /usr/local/lib/python3.10/dist-packages (from axolotl==0.1) (0.1.99)
Requirement already satisfied: wandb in /usr/local/lib/python3.10/dist-packages (from axolotl==0.1) (0.15.3)
Requirement already satisfied: einops in /usr/local/lib/python3.10/dist-packages (from axolotl==0.1) (0.6.1)
Requirement already satisfied: xformers in /usr/local/lib/python3.10/dist-packages (from axolotl==0.1) (0.0.20)
Requirement already satisfied: bert-score==0.3.13 in /usr/local/lib/python3.10/dist-packages (from axolotl==0.1) (0.3.13)
Requirement already satisfied: evaluate==0.4.0 in /usr/local/lib/python3.10/dist-packages (from axolotl==0.1) (0.4.0)
Requirement already satisfied: rouge-score==0.1.2 in /usr/local/lib/python3.10/dist-packages (from axolotl==0.1) (0.1.2)
Requirement already satisfied: scipy in /usr/local/lib/python3.10/dist-packages (from axolotl==0.1) (1.10.1)
Requirement already satisfied: scikit-learn==1.2.2 in /usr/local/lib/python3.10/dist-packages (from axolotl==0.1) (1.2.2)
Requirement already satisfied: torch>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from bert-score==0.3.13->axolotl==0.1) (2.0.1+cu118)
Requirement already satisfied: pandas>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from bert-score==0.3.13->axolotl==0.1) (1.5.3)
Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from bert-score==0.3.13->axolotl==0.1) (1.22.4)
Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from bert-score==0.3.13->axolotl==0.1) (2.27.1)
Requirement already satisfied: tqdm>=4.31.1 in /usr/local/lib/python3.10/dist-packages (from bert-score==0.3.13->axolotl==0.1) (4.65.0)
Requirement already satisfied: matplotlib in /usr/local/lib/python3.10/dist-packages (from bert-score==0.3.13->axolotl==0.1) (3.7.1)
Requirement already satisfied: packaging>=20.9 in /usr/local/lib/python3.10/dist-packages (from bert-score==0.3.13->axolotl==0.1) (23.1)
Requirement already satisfied: dill in /usr/local/lib/python3.10/dist-packages (from evaluate==0.4.0->axolotl==0.1) (0.3.6)
Requirement already satisfied: xxhash in /usr/local/lib/python3.10/dist-packages (from evaluate==0.4.0->axolotl==0.1) (3.2.0)
Requirement already satisfied: multiprocess in /usr/local/lib/python3.10/dist-packages (from evaluate==0.4.0->axolotl==0.1) (0.70.14)
Requirement already satisfied: fsspec[http]>=2021.05.0 in /usr/local/lib/python3.10/dist-packages (from evaluate==0.4.0->axolotl==0.1) (2023.4.0)
Requirement already satisfied: huggingface-hub>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from evaluate==0.4.0->axolotl==0.1) (0.15.1)
Requirement already satisfied: responses<0.19 in /usr/local/lib/python3.10/dist-packages (from evaluate==0.4.0->axolotl==0.1) (0.18.0)
Requirement already satisfied: absl-py in /usr/local/lib/python3.10/dist-packages (from rouge-score==0.1.2->axolotl==0.1) (1.4.0)
Requirement already satisfied: nltk in /usr/local/lib/python3.10/dist-packages (from rouge-score==0.1.2->axolotl==0.1) (3.8.1)
Requirement already satisfied: six>=1.14.0 in /usr/local/lib/python3.10/dist-packages (from rouge-score==0.1.2->axolotl==0.1) (1.16.0)
Requirement already satisfied: joblib>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from scikit-learn==1.2.2->axolotl==0.1) (1.2.0)
Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn==1.2.2->axolotl==0.1) (3.1.0)
Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from accelerate->axolotl==0.1) (5.9.5)
Requirement already satisfied: pyarrow>=8.0.0 in /usr/local/lib/python3.10/dist-packages (from datasets->axolotl==0.1) (9.0.0)
Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (from datasets->axolotl==0.1) (3.8.4)
Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers@ git+https://github.com/huggingface/transformers.git->axolotl==0.1) (3.12.0)
Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers@ git+https://github.com/huggingface/transformers.git->axolotl==0.1) (2022.10.31)
Requirement already satisfied: tokenizers!=0.11.3,<0.14,>=0.11.1 in /usr/local/lib/python3.10/dist-packages (from transformers@ git+https://github.com/huggingface/transformers.git->axolotl==0.1) (0.13.3)
Requirement already satisfied: safetensors>=0.3.1 in /usr/local/lib/python3.10/dist-packages (from transformers@ git+https://github.com/huggingface/transformers.git->axolotl==0.1) (0.3.1)
Requirement already satisfied: termcolor in /usr/local/lib/python3.10/dist-packages (from fire->axolotl==0.1) (2.3.0)
Requirement already satisfied: Click!=8.0.0,>=7.0 in /usr/local/lib/python3.10/dist-packages (from wandb->axolotl==0.1) (8.1.3)
Requirement already satisfied: GitPython!=3.1.29,>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from wandb->axolotl==0.1) (3.1.31)
Requirement already satisfied: sentry-sdk>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from wandb->axolotl==0.1) (1.25.0)
Requirement already satisfied: docker-pycreds>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from wandb->axolotl==0.1) (0.4.0)
Requirement already satisfied: pathtools in /usr/local/lib/python3.10/dist-packages (from wandb->axolotl==0.1) (0.1.2)
Requirement already satisfied: setproctitle in /usr/local/lib/python3.10/dist-packages (from wandb->axolotl==0.1) (1.3.2)
Requirement already satisfied: setuptools in /usr/local/lib/python3.10/dist-packages (from wandb->axolotl==0.1) (67.7.2)
Requirement already satisfied: appdirs>=1.4.3 in /usr/local/lib/python3.10/dist-packages (from wandb->axolotl==0.1) (1.4.4)
Requirement already satisfied: protobuf!=4.21.0,<5,>=3.19.0 in /usr/local/lib/python3.10/dist-packages (from wandb->axolotl==0.1) (3.20.3)
Requirement already satisfied: pyre-extensions==0.0.29 in /usr/local/lib/python3.10/dist-packages (from xformers->axolotl==0.1) (0.0.29)
Requirement already satisfied: typing-inspect in /usr/local/lib/python3.10/dist-packages (from pyre-extensions==0.0.29->xformers->axolotl==0.1) (0.9.0)
Requirement already satisfied: typing-extensions in /usr/local/lib/python3.10/dist-packages (from pyre-extensions==0.0.29->xformers->axolotl==0.1) (4.5.0)
Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch>=1.0.0->bert-score==0.3.13->axolotl==0.1) (1.11.1)
Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch>=1.0.0->bert-score==0.3.13->axolotl==0.1) (3.1)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch>=1.0.0->bert-score==0.3.13->axolotl==0.1) (3.1.2)
Requirement already satisfied: triton==2.0.0 in /usr/local/lib/python3.10/dist-packages (from torch>=1.0.0->bert-score==0.3.13->axolotl==0.1) (2.0.0)
Requirement already satisfied: cmake in /usr/local/lib/python3.10/dist-packages (from triton==2.0.0->torch>=1.0.0->bert-score==0.3.13->axolotl==0.1) (3.25.2)
Requirement already satisfied: lit in /usr/local/lib/python3.10/dist-packages (from triton==2.0.0->torch>=1.0.0->bert-score==0.3.13->axolotl==0.1) (16.0.5)
Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->axolotl==0.1) (23.1.0)
Requirement already satisfied: charset-normalizer<4.0,>=2.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->axolotl==0.1) (2.0.12)
Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->axolotl==0.1) (6.0.4)
Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->axolotl==0.1) (4.0.2)
Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->axolotl==0.1) (1.9.2)
Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->axolotl==0.1) (1.3.3)
Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->axolotl==0.1) (1.3.1)
Requirement already satisfied: gitdb<5,>=4.0.1 in /usr/local/lib/python3.10/dist-packages (from GitPython!=3.1.29,>=1.0.0->wandb->axolotl==0.1) (4.0.10)
Requirement already satisfied: python-dateutil>=2.8.1 in /usr/local/lib/python3.10/dist-packages (from pandas>=1.0.1->bert-score==0.3.13->axolotl==0.1) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas>=1.0.1->bert-score==0.3.13->axolotl==0.1) (2022.7.1)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->bert-score==0.3.13->axolotl==0.1) (1.26.15)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->bert-score==0.3.13->axolotl==0.1) (2022.12.7)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->bert-score==0.3.13->axolotl==0.1) (3.4)
Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->bert-score==0.3.13->axolotl==0.1) (1.0.7)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/dist-packages (from matplotlib->bert-score==0.3.13->axolotl==0.1) (0.11.0)
Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib->bert-score==0.3.13->axolotl==0.1) (4.39.3)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->bert-score==0.3.13->axolotl==0.1) (1.4.4)
Requirement already satisfied: pillow>=6.2.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib->bert-score==0.3.13->axolotl==0.1) (8.4.0)
Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->bert-score==0.3.13->axolotl==0.1) (3.0.9)
Requirement already satisfied: smmap<6,>=3.0.1 in /usr/local/lib/python3.10/dist-packages (from gitdb<5,>=4.0.1->GitPython!=3.1.29,>=1.0.0->wandb->axolotl==0.1) (5.0.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch>=1.0.0->bert-score==0.3.13->axolotl==0.1) (2.1.2)
Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->torch>=1.0.0->bert-score==0.3.13->axolotl==0.1) (1.3.0)
Requirement already satisfied: mypy-extensions>=0.3.0 in /usr/local/lib/python3.10/dist-packages (from typing-inspect->pyre-extensions==0.0.29->xformers->axolotl==0.1) (1.0.0)
Installing collected packages: axolotl
Attempting uninstall: axolotl
Found existing installation: axolotl 0.1
Uninstalling axolotl-0.1:
Successfully uninstalled axolotl-0.1
Running setup.py develop for axolotl
Successfully installed axolotl-0.1
2. Generate default config for accelerate¶
In [ ]:
!accelerate config --config_file configs/accelerate/default_config.yaml default
2023-06-03 10:27:56.068481: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT Configuration already exists at /root/.cache/huggingface/accelerate/default_config.yaml, will not override. Run `accelerate config` manually or pass a different `save_location`.
In [ ]:
!wget https://raw.githubusercontent.com/utensil/axolotl/falcon-7b-qlora/examples/falcon/config-7b-qlora.yml
--2023-06-03 10:28:00-- https://raw.githubusercontent.com/utensil/axolotl/falcon-7b-qlora/examples/falcon/config-7b-qlora.yml Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ... Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 2238 (2.2K) [text/plain] Saving to: ‘config-7b-qlora.yml.1’ config-7b-qlora.yml 0%[ ] 0 --.-KB/s config-7b-qlora.yml 100%[===================>] 2.19K --.-KB/s in 0s 2023-06-03 10:28:00 (38.7 MB/s) - ‘config-7b-qlora.yml.1’ saved [2238/2238]
3. Use a well-tested falcon-7b qlora config and adjust it to 1b & Cloab¶
In [ ]:
!cp config-7b-qlora.yml config-1b-qlora.yml
!sed -i -e 's/falcon-7b/falcon-rw-1b/g' -e 's/wandb_project: falcon-qlora/wandb_project: /g' -e 's/wandb_project: falcon-qlora/wandb_project: /g' -e 's/bf16: true/bf16: false/g' -e 's/tf32: true/tf32: false/g' config-1b-qlora.yml
In [ ]:
!cat config-1b-qlora.yml
# 1b: tiiuae/falcon-rw-1b
# 40b: tiiuae/falcon-40b
base_model: tiiuae/falcon-rw-1b
base_model_config: tiiuae/falcon-rw-1b
# required by falcon custom model code: https://huggingface.co/tiiuae/falcon-rw-1b/tree/main
trust_remote_code: true
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer
load_in_8bit: false
# enable 4bit for QLoRA
load_in_4bit: true
gptq: false
strict: false
push_dataset_to_hub:
datasets:
- path: QingyiSi/Alpaca-CoT
data_files:
- Chain-of-Thought/formatted_cot_data/gsm8k_train.json
type: "alpaca:chat"
dataset_prepared_path: last_run_prepared
val_set_size: 0.01
# enable QLoRA
adapter: qlora
lora_model_dir:
sequence_len: 2048
max_packed_sequence_len:
# hyperparameters from QLoRA paper Appendix B.2
# "We find hyperparameters to be largely robust across datasets"
lora_r: 64
lora_alpha: 16
# 0.1 for models up to 13B
# 0.05 for 33B and 65B models
lora_dropout: 0.05
# add LoRA modules on all linear layers of the base model
lora_target_modules:
lora_target_linear: true
lora_fan_in_fan_out:
wandb_project:
wandb_watch:
wandb_run_id:
wandb_log_model:
output_dir: ./qlora-out
# QLoRA paper Table 9
# - 16 for 7b & 13b
# - 32 for 33b, 64 for 64b
# Max size tested on A6000
# - 7b: 40
# - 40b: 4
# decrease if OOM, increase for max VRAM utilization
micro_batch_size: 1
gradient_accumulation_steps: 2
num_epochs: 3
# Optimizer for QLoRA
optimizer: paged_adamw_32bit
torchdistx_path:
lr_scheduler: cosine
# QLoRA paper Table 9
# - 2e-4 for 7b & 13b
# - 1e-4 for 33b & 64b
learning_rate: 0.0002
train_on_inputs: false
group_by_length: false
bf16: false
fp16: false
tf32: false
gradient_checkpointing: true
# stop training after this many evaluation losses have increased in a row
# https://huggingface.co/transformers/v4.2.2/_modules/transformers/trainer_callback.html#EarlyStoppingCallback
early_stopping_patience: 3
resume_from_checkpoint:
auto_resume_from_checkpoints: true
local_rank:
logging_steps: 1
xformers_attention: true
flash_attention:
gptq_groupsize:
gptq_model_v1:
warmup_steps: 10
eval_steps: 5
save_steps: 10
debug:
deepspeed:
weight_decay: 0.000001
fsdp:
fsdp_config:
special_tokens:
pad_token: "<|endoftext|>"
bos_token: ">>ABSTRACT<<"
eos_token: "<|endoftext|>"
You can read the comments in the config and tweak it.
4. Install QLoRA dependencies¶
In [ ]:
!pip install bitsandbytes
!pip install git+https://github.com/huggingface/transformers.git
!pip install git+https://github.com/huggingface/peft.git
!pip install git+https://github.com/huggingface/accelerate.git
Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/ Requirement already satisfied: bitsandbytes in /usr/local/lib/python3.10/dist-packages (0.39.0) Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/ Collecting git+https://github.com/huggingface/transformers.git Cloning https://github.com/huggingface/transformers.git to /tmp/pip-req-build-504lkq2w Running command git clone --filter=blob:none --quiet https://github.com/huggingface/transformers.git /tmp/pip-req-build-504lkq2w Resolved https://github.com/huggingface/transformers.git to commit 539e2281cd97c35ef4122757f26c88f44115fa94 Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... done Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers==4.30.0.dev0) (3.12.0) Requirement already satisfied: huggingface-hub<1.0,>=0.14.1 in /usr/local/lib/python3.10/dist-packages (from transformers==4.30.0.dev0) (0.15.1) Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from transformers==4.30.0.dev0) (1.22.4) Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from transformers==4.30.0.dev0) (23.1) Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from transformers==4.30.0.dev0) (6.0) Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers==4.30.0.dev0) (2022.10.31) Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from transformers==4.30.0.dev0) (2.27.1) Requirement already satisfied: tokenizers!=0.11.3,<0.14,>=0.11.1 in /usr/local/lib/python3.10/dist-packages (from transformers==4.30.0.dev0) (0.13.3) Requirement already satisfied: safetensors>=0.3.1 in /usr/local/lib/python3.10/dist-packages (from transformers==4.30.0.dev0) (0.3.1) Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.10/dist-packages (from transformers==4.30.0.dev0) (4.65.0) Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.14.1->transformers==4.30.0.dev0) (2023.4.0) Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.14.1->transformers==4.30.0.dev0) (4.5.0) Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.30.0.dev0) (1.26.15) Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.30.0.dev0) (2022.12.7) Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.30.0.dev0) (2.0.12) Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->transformers==4.30.0.dev0) (3.4) Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/ Collecting git+https://github.com/huggingface/peft.git Cloning https://github.com/huggingface/peft.git to /tmp/pip-req-build-z02g4q7x Running command git clone --filter=blob:none --quiet https://github.com/huggingface/peft.git /tmp/pip-req-build-z02g4q7x Resolved https://github.com/huggingface/peft.git to commit fcff23f005fc7bfb816ad1f55360442c170cd5f5 Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... done Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from peft==0.4.0.dev0) (1.22.4) Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from peft==0.4.0.dev0) (23.1) Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from peft==0.4.0.dev0) (5.9.5) Requirement already satisfied: pyyaml in /usr/local/lib/python3.10/dist-packages (from peft==0.4.0.dev0) (6.0) Requirement already satisfied: torch>=1.13.0 in /usr/local/lib/python3.10/dist-packages (from peft==0.4.0.dev0) (2.0.1+cu118) Requirement already satisfied: transformers in /usr/local/lib/python3.10/dist-packages (from peft==0.4.0.dev0) (4.30.0.dev0) Requirement already satisfied: accelerate in /usr/local/lib/python3.10/dist-packages (from peft==0.4.0.dev0) (0.20.0.dev0) Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch>=1.13.0->peft==0.4.0.dev0) (3.12.0) Requirement already satisfied: typing-extensions in /usr/local/lib/python3.10/dist-packages (from torch>=1.13.0->peft==0.4.0.dev0) (4.5.0) Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch>=1.13.0->peft==0.4.0.dev0) (1.11.1) Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch>=1.13.0->peft==0.4.0.dev0) (3.1) Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch>=1.13.0->peft==0.4.0.dev0) (3.1.2) Requirement already satisfied: triton==2.0.0 in /usr/local/lib/python3.10/dist-packages (from torch>=1.13.0->peft==0.4.0.dev0) (2.0.0) Requirement already satisfied: cmake in /usr/local/lib/python3.10/dist-packages (from triton==2.0.0->torch>=1.13.0->peft==0.4.0.dev0) (3.25.2) Requirement already satisfied: lit in /usr/local/lib/python3.10/dist-packages (from triton==2.0.0->torch>=1.13.0->peft==0.4.0.dev0) (16.0.5) Requirement already satisfied: huggingface-hub<1.0,>=0.14.1 in /usr/local/lib/python3.10/dist-packages (from transformers->peft==0.4.0.dev0) (0.15.1) Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers->peft==0.4.0.dev0) (2022.10.31) Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from transformers->peft==0.4.0.dev0) (2.27.1) Requirement already satisfied: tokenizers!=0.11.3,<0.14,>=0.11.1 in /usr/local/lib/python3.10/dist-packages (from transformers->peft==0.4.0.dev0) (0.13.3) Requirement already satisfied: safetensors>=0.3.1 in /usr/local/lib/python3.10/dist-packages (from transformers->peft==0.4.0.dev0) (0.3.1) Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.10/dist-packages (from transformers->peft==0.4.0.dev0) (4.65.0) Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.14.1->transformers->peft==0.4.0.dev0) (2023.4.0) Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch>=1.13.0->peft==0.4.0.dev0) (2.1.2) Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->transformers->peft==0.4.0.dev0) (1.26.15) Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->transformers->peft==0.4.0.dev0) (2022.12.7) Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.10/dist-packages (from requests->transformers->peft==0.4.0.dev0) (2.0.12) Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->transformers->peft==0.4.0.dev0) (3.4) Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->torch>=1.13.0->peft==0.4.0.dev0) (1.3.0) Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/ Collecting git+https://github.com/huggingface/accelerate.git Cloning https://github.com/huggingface/accelerate.git to /tmp/pip-req-build-lr6c5fbq Running command git clone --filter=blob:none --quiet https://github.com/huggingface/accelerate.git /tmp/pip-req-build-lr6c5fbq Resolved https://github.com/huggingface/accelerate.git to commit eba6eb79dc2ab652cd8b44b37165a4852768a8ac Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... done Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from accelerate==0.20.0.dev0) (1.22.4) Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from accelerate==0.20.0.dev0) (23.1) Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from accelerate==0.20.0.dev0) (5.9.5) Requirement already satisfied: pyyaml in /usr/local/lib/python3.10/dist-packages (from accelerate==0.20.0.dev0) (6.0) Requirement already satisfied: torch>=1.6.0 in /usr/local/lib/python3.10/dist-packages (from accelerate==0.20.0.dev0) (2.0.1+cu118) Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch>=1.6.0->accelerate==0.20.0.dev0) (3.12.0) Requirement already satisfied: typing-extensions in /usr/local/lib/python3.10/dist-packages (from torch>=1.6.0->accelerate==0.20.0.dev0) (4.5.0) Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch>=1.6.0->accelerate==0.20.0.dev0) (1.11.1) Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch>=1.6.0->accelerate==0.20.0.dev0) (3.1) Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch>=1.6.0->accelerate==0.20.0.dev0) (3.1.2) Requirement already satisfied: triton==2.0.0 in /usr/local/lib/python3.10/dist-packages (from torch>=1.6.0->accelerate==0.20.0.dev0) (2.0.0) Requirement already satisfied: cmake in /usr/local/lib/python3.10/dist-packages (from triton==2.0.0->torch>=1.6.0->accelerate==0.20.0.dev0) (3.25.2) Requirement already satisfied: lit in /usr/local/lib/python3.10/dist-packages (from triton==2.0.0->torch>=1.6.0->accelerate==0.20.0.dev0) (16.0.5) Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch>=1.6.0->accelerate==0.20.0.dev0) (2.1.2) Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->torch>=1.6.0->accelerate==0.20.0.dev0) (1.3.0)
5. Set W&B to offline mode¶
In [ ]:
%env WANDB_MODE=offline
env: WANDB_MODE=offline
This is to skip some extra setup steps, you can also choose to login to your W&B account before training.
6. Start training and enjoy!¶
In [ ]:
!accelerate launch scripts/finetune.py config-1b-qlora.yml
2023-06-03 10:28:57.937147: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please run
python -m bitsandbytes
and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================
bin /usr/local/lib/python3.10/dist-packages/bitsandbytes/libbitsandbytes_cuda118.so
/usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: /usr/lib64-nvidia did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths...
warn(msg)
/usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/sys/fs/cgroup/memory.events /var/colab/cgroup/jupyter-children/memory.events')}
warn(msg)
/usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('http'), PosixPath('//172.28.0.1'), PosixPath('8013')}
warn(msg)
/usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('--logtostderr --listen_host=172.28.0.12 --target_host=172.28.0.12 --tunnel_background_save_url=https'), PosixPath('//colab.research.google.com/tun/m/cc48301118ce562b961b3c22d803539adc1e0c19/gpu-t4-s-mgx0u73sau1b --tunnel_background_save_delay=10s --tunnel_periodic_background_save_frequency=30m0s --enable_output_coalescing=true --output_coalescing_required=true')}
warn(msg)
/usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/env/python')}
warn(msg)
/usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//ipykernel.pylab.backend_inline'), PosixPath('module')}
warn(msg)
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
/usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: Found duplicate ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] files: {PosixPath('/usr/local/cuda/lib64/libcudart.so'), PosixPath('/usr/local/cuda/lib64/libcudart.so.11.0')}.. We'll flip a coin and try one of these, in order to fail forward.
Either way, this might cause trouble in the future:
If you get `CUDA error: invalid device function` errors, the above might be the cause and the solution is to make sure only one ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] in the paths that we search based on your env.
warn(msg)
CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so
CUDA SETUP: Highest compute capability among GPUs detected: 7.5
CUDA SETUP: Detected CUDA version 118
CUDA SETUP: Loading binary /usr/local/lib/python3.10/dist-packages/bitsandbytes/libbitsandbytes_cuda118.so...
2023-06-03 10:29:07.230483: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
WARNING:root:`trust_remote_code` is set to true. Please make sure that you reviewed the remote code/model.
INFO:root:loading tokenizer... tiiuae/falcon-rw-1b
Using pad_token, but it is not set yet.
INFO:root:Loading prepared dataset from disk at last_run_prepared/0ecc5b78e3ce4254b22e749b093712b4...
INFO:root:Prepared dataset loaded from disk...
INFO:root:loading model and peft_config...
INFO:root:converting PEFT model w/ prepare_model_for_int8_training
/usr/local/lib/python3.10/dist-packages/peft/utils/other.py:76: FutureWarning: prepare_model_for_int8_training is deprecated and will be removed in a future version. Use prepare_model_for_kbit_training instead.
warnings.warn(
INFO:root:found linear modules: ['query_key_value', 'dense', 'dense_h_to_4h', 'dense_4h_to_h']
trainable params: 50331648 || all params: 757911552 || trainable%: 6.6408339953630895
INFO:root:Compiling torch model
INFO:root:Pre-saving adapter config to ./qlora-out
INFO:root:Starting trainer...
INFO:root:Using Auto-resume functionality to start with checkpoint at qlora-out/checkpoint-100
wandb: Tracking run with wandb version 0.15.3
wandb: W&B syncing is set to `offline` in this directory.
wandb: Run `wandb online` or set WANDB_MODE=online to enable cloud syncing.
0% 0/11097 [00:00<?, ?it/s]You're using a GPT2TokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
{'loss': 0.8716, 'learning_rate': 0.00019996675698728198, 'epoch': 0.03}
{'loss': 1.1533, 'learning_rate': 0.00019996602239885048, 'epoch': 0.03}
{'loss': 1.2701, 'learning_rate': 0.00019996527978395872, 'epoch': 0.03}
{'loss': 1.0565, 'learning_rate': 0.00019996452914266633, 'epoch': 0.03}
{'loss': 1.416, 'learning_rate': 0.0001999637704750336, 'epoch': 0.03}
1% 105/11097 [00:06<03:10, 57.69it/s]
0% 0/75 [00:00<?, ?it/s]
3% 2/75 [00:00<00:05, 13.09it/s]
5% 4/75 [00:00<00:10, 6.71it/s]
7% 5/75 [00:00<00:12, 5.59it/s]
8% 6/75 [00:00<00:12, 5.66it/s]
9% 7/75 [00:01<00:13, 4.92it/s]
11% 8/75 [00:01<00:14, 4.56it/s]
12% 9/75 [00:01<00:13, 4.90it/s]
13% 10/75 [00:01<00:13, 4.89it/s]
15% 11/75 [00:02<00:12, 5.16it/s]
16% 12/75 [00:02<00:12, 5.07it/s]
17% 13/75 [00:02<00:12, 4.84it/s]
19% 14/75 [00:02<00:12, 4.80it/s]
20% 15/75 [00:02<00:11, 5.13it/s]
21% 16/75 [00:03<00:12, 4.60it/s]
23% 17/75 [00:03<00:12, 4.65it/s]
24% 18/75 [00:03<00:11, 5.07it/s]
25% 19/75 [00:03<00:11, 4.82it/s]
27% 20/75 [00:03<00:11, 4.70it/s]
28% 21/75 [00:04<00:11, 4.67it/s]
29% 22/75 [00:04<00:11, 4.69it/s]
31% 23/75 [00:04<00:11, 4.38it/s]
32% 24/75 [00:04<00:11, 4.48it/s]
33% 25/75 [00:05<00:10, 4.89it/s]
35% 26/75 [00:05<00:09, 5.13it/s]
36% 27/75 [00:05<00:09, 5.05it/s]
37% 28/75 [00:05<00:09, 4.88it/s]
39% 29/75 [00:05<00:09, 4.93it/s]
40% 30/75 [00:06<00:09, 4.57it/s]
41% 31/75 [00:06<00:09, 4.48it/s]
43% 32/75 [00:06<00:08, 4.81it/s]
44% 33/75 [00:06<00:08, 4.75it/s]
45% 34/75 [00:06<00:08, 4.57it/s]
47% 35/75 [00:07<00:09, 4.42it/s]
48% 36/75 [00:07<00:08, 4.45it/s]
49% 37/75 [00:07<00:08, 4.49it/s]
51% 38/75 [00:07<00:08, 4.54it/s]
52% 39/75 [00:08<00:07, 4.54it/s]
53% 40/75 [00:08<00:07, 4.58it/s]
55% 41/75 [00:08<00:07, 4.61it/s]
56% 42/75 [00:08<00:07, 4.58it/s]
57% 43/75 [00:08<00:06, 4.63it/s]
59% 44/75 [00:09<00:06, 4.96it/s]
60% 45/75 [00:09<00:06, 4.73it/s]
61% 46/75 [00:09<00:06, 4.40it/s]
63% 47/75 [00:09<00:06, 4.18it/s]
64% 48/75 [00:10<00:06, 4.16it/s]
65% 49/75 [00:10<00:06, 4.19it/s]
67% 50/75 [00:10<00:06, 4.15it/s]
68% 51/75 [00:10<00:05, 4.32it/s]
69% 52/75 [00:10<00:05, 4.48it/s]
71% 53/75 [00:11<00:05, 3.99it/s]
72% 54/75 [00:11<00:05, 4.00it/s]
73% 55/75 [00:11<00:04, 4.44it/s]
75% 56/75 [00:11<00:04, 4.40it/s]
76% 57/75 [00:12<00:04, 4.33it/s]
77% 58/75 [00:12<00:03, 4.48it/s]
79% 59/75 [00:12<00:03, 4.71it/s]
80% 60/75 [00:12<00:03, 4.50it/s]
1% 105/11097 [00:19<03:10, 57.69it/s]
83% 62/75 [00:13<00:03, 4.28it/s]
84% 63/75 [00:13<00:02, 4.31it/s]
85% 64/75 [00:13<00:02, 4.38it/s]
87% 65/75 [00:13<00:02, 4.50it/s]
88% 66/75 [00:14<00:01, 4.90it/s]
89% 67/75 [00:14<00:01, 5.16it/s]
91% 68/75 [00:14<00:01, 4.93it/s]
92% 69/75 [00:14<00:01, 4.91it/s]
93% 70/75 [00:14<00:00, 5.14it/s]
95% 71/75 [00:15<00:00, 4.98it/s]
96% 72/75 [00:15<00:00, 4.90it/s]
97% 73/75 [00:15<00:00, 5.02it/s]
99% 74/75 [00:15<00:00, 4.52it/s]
{'eval_loss': 1.2588105201721191, 'eval_runtime': 16.1808, 'eval_samples_per_second': 4.635, 'eval_steps_per_second': 4.635, 'epoch': 0.03}
1% 105/11097 [00:22<03:10, 57.69it/s]
100% 75/75 [00:16<00:00, 4.54it/s]
{'loss': 1.2957, 'learning_rate': 0.00019996300378112138, 'epoch': 0.03}
{'loss': 1.303, 'learning_rate': 0.00019996222906099132, 'epoch': 0.03}
{'loss': 1.5324, 'learning_rate': 0.00019996144631470554, 'epoch': 0.03}
{'loss': 1.1532, 'learning_rate': 0.00019996065554232697, 'epoch': 0.03}
{'loss': 1.1804, 'learning_rate': 0.00019995985674391904, 'epoch': 0.03}
1% 110/11097 [00:29<1:15:08, 2.44it/s]
0% 0/75 [00:00<?, ?it/s]
3% 2/75 [00:00<00:05, 12.25it/s]
5% 4/75 [00:00<00:11, 6.42it/s]
7% 5/75 [00:00<00:12, 5.55it/s]
8% 6/75 [00:01<00:12, 5.40it/s]
9% 7/75 [00:01<00:14, 4.73it/s]
11% 8/75 [00:01<00:14, 4.52it/s]
12% 9/75 [00:01<00:13, 4.84it/s]
13% 10/75 [00:01<00:13, 4.80it/s]
15% 11/75 [00:02<00:12, 5.04it/s]
16% 12/75 [00:02<00:12, 4.91it/s]
17% 13/75 [00:02<00:12, 4.78it/s]
19% 14/75 [00:02<00:12, 4.74it/s]
20% 15/75 [00:02<00:12, 4.92it/s]
21% 16/75 [00:03<00:12, 4.62it/s]
23% 17/75 [00:03<00:12, 4.70it/s]
24% 18/75 [00:03<00:11, 4.92it/s]
25% 19/75 [00:03<00:12, 4.66it/s]
27% 20/75 [00:04<00:12, 4.52it/s]
28% 21/75 [00:04<00:11, 4.51it/s]
29% 22/75 [00:04<00:11, 4.42it/s]
31% 23/75 [00:04<00:12, 4.25it/s]
32% 24/75 [00:04<00:11, 4.38it/s]
33% 25/75 [00:05<00:10, 4.77it/s]
35% 26/75 [00:05<00:09, 4.97it/s]
36% 27/75 [00:05<00:10, 4.79it/s]
37% 28/75 [00:05<00:10, 4.62it/s]
39% 29/75 [00:06<00:10, 4.50it/s]
40% 30/75 [00:06<00:10, 4.25it/s]
1% 110/11097 [00:37<1:01:57, 2.96it/s]
wandb: Waiting for W&B process to finish... (success).
wandb:
wandb: Run history:
wandb: eval/loss ▁
wandb: eval/runtime ▁
wandb: eval/samples_per_second ▁
wandb: eval/steps_per_second ▁
wandb: train/epoch ▁▁▁▁▁▁▁▁▁▁▁
wandb: train/global_step ▁▂▃▃▄▄▅▆▆▇█
wandb: train/learning_rate █▇▆▆▅▄▃▃▂▁
wandb: train/loss ▁▄▅▃▇▅▆█▄▄
wandb:
wandb: Run summary:
wandb: eval/loss 1.25881
wandb: eval/runtime 16.1808
wandb: eval/samples_per_second 4.635
wandb: eval/steps_per_second 4.635
wandb: train/epoch 0.03
wandb: train/global_step 110
wandb: train/learning_rate 0.0002
wandb: train/loss 1.1804
wandb:
wandb: You can sync this run to the cloud by running:
wandb: wandb sync /content/axolotl/wandb/offline-run-20230603_102953-69ojcgwa
wandb: Find logs at: ./wandb/offline-run-20230603_102953-69ojcgwa/logs
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /usr/lib/python3.10/subprocess.py:1959 in _wait │
│ │
│ 1956 │ │ │ │ │ with self._waitpid_lock: │
│ 1957 │ │ │ │ │ │ if self.returncode is not None: │
│ 1958 │ │ │ │ │ │ │ break # Another thread waited. │
│ ❱ 1959 │ │ │ │ │ │ (pid, sts) = self._try_wait(0) │
│ 1960 │ │ │ │ │ │ # Check the pid and loop as waitpid has been │
│ 1961 │ │ │ │ │ │ # return 0 even without WNOHANG in odd situat │
│ 1962 │ │ │ │ │ │ # http://bugs.python.org/issue14396. │
│ │
│ /usr/lib/python3.10/subprocess.py:1917 in _try_wait │
│ │
│ 1914 │ │ def _try_wait(self, wait_flags): │
│ 1915 │ │ │ """All callers to this function MUST hold self._waitpid_l │
│ 1916 │ │ │ try: │
│ ❱ 1917 │ │ │ │ (pid, sts) = os.waitpid(self.pid, wait_flags) │
│ 1918 │ │ │ except ChildProcessError: │
│ 1919 │ │ │ │ # This happens if SIGCLD is set to be ignored or wait │
│ 1920 │ │ │ │ # for child processes has otherwise been disabled for │
╰──────────────────────────────────────────────────────────────────────────────╯
KeyboardInterrupt
During handling of the above exception, another exception occurred:
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /usr/local/bin/accelerate:8 in <module> │
│ │
│ 5 from accelerate.commands.accelerate_cli import main │
│ 6 if __name__ == '__main__': │
│ 7 │ sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0]) │
│ ❱ 8 │ sys.exit(main()) │
│ 9 │
│ │
│ /usr/local/lib/python3.10/dist-packages/accelerate/commands/accelerate_cli.p │
│ y:45 in main │
│ │
│ 42 │ │ exit(1) │
│ 43 │ │
│ 44 │ # Run │
│ ❱ 45 │ args.func(args) │
│ 46 │
│ 47 │
│ 48 if __name__ == "__main__": │
│ │
│ /usr/local/lib/python3.10/dist-packages/accelerate/commands/launch.py:941 in │
│ launch_command │
│ │
│ 938 │ elif defaults is not None and defaults.compute_environment == Comp │
│ 939 │ │ sagemaker_launcher(defaults, args) │
│ 940 │ else: │
│ ❱ 941 │ │ simple_launcher(args) │
│ 942 │
│ 943 │
│ 944 def main(): │
│ │
│ /usr/local/lib/python3.10/dist-packages/accelerate/commands/launch.py:600 in │
│ simple_launcher │
│ │
│ 597 │ cmd, current_env = prepare_simple_launcher_cmd_env(args) │
│ 598 │ │
│ 599 │ process = subprocess.Popen(cmd, env=current_env) │
│ ❱ 600 │ process.wait() │
│ 601 │ if process.returncode != 0: │
│ 602 │ │ if not args.quiet: │
│ 603 │ │ │ raise subprocess.CalledProcessError(returncode=process.ret │
│ │
│ /usr/lib/python3.10/subprocess.py:1209 in wait │
│ │
│ 1206 │ │ if timeout is not None: │
│ 1207 │ │ │ endtime = _time() + timeout │
│ 1208 │ │ try: │
│ ❱ 1209 │ │ │ return self._wait(timeout=timeout) │
│ 1210 │ │ except KeyboardInterrupt: │
│ 1211 │ │ │ # https://bugs.python.org/issue25942 │
│ 1212 │ │ │ # The first keyboard interrupt waits briefly for the chil │
│ │
│ /usr/lib/python3.10/subprocess.py:1959 in _wait │
│ │
│ 1956 │ │ │ │ │ with self._waitpid_lock: │
│ 1957 │ │ │ │ │ │ if self.returncode is not None: │
│ 1958 │ │ │ │ │ │ │ break # Another thread waited. │
│ ❱ 1959 │ │ │ │ │ │ (pid, sts) = self._try_wait(0) │
│ 1960 │ │ │ │ │ │ # Check the pid and loop as waitpid has been │
│ 1961 │ │ │ │ │ │ # return 0 even without WNOHANG in odd situat │
│ 1962 │ │ │ │ │ │ # http://bugs.python.org/issue14396. │
╰──────────────────────────────────────────────────────────────────────────────╯
KeyboardInterrupt
^C
In [ ]: