In [5]:
%cd /workspace/axolotl
/workspace/axolotl
In [6]:
!accelerate config --config_file configs/accelerate/default_config.yaml default
Setting ds_accelerator to cuda (auto detect) Configuration already exists at /root/.cache/huggingface/accelerate/default_config.yaml, will not override. Run `accelerate config` manually or pass a different `save_location`.
Open files to modify:
/workspace/axolotl/examples/falcon/qlora.yml
#1 run-12: 4*2 - no max len¶
In [7]:
!cat examples/falcon/qlora.yml
base_model: tiiuae/falcon-7b
base_model_config: tiiuae/falcon-7b
trust_remote_code: true
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer
load_in_8bit: false
load_in_4bit: true
gptq: false
strict: false
push_dataset_to_hub:
datasets:
- path: QingyiSi/Alpaca-CoT
data_files:
- Chain-of-Thought/formatted_cot_data/gsm8k_train.json
type: "alpaca:chat"
dataset_prepared_path: last_run_prepared
val_set_size: 0.01
adapter: qlora
lora_model_dir:
sequence_len: 2048
max_packed_sequence_len:
lora_r: 64
lora_alpha: 16
lora_dropout: 0.05
lora_target_modules:
lora_target_linear: true
lora_fan_in_fan_out:
wandb_project: falcon-qlora
wandb_watch:
wandb_run_id:
wandb_log_model:
output_dir: ./qlora-out
micro_batch_size: 4
gradient_accumulation_steps: 2
num_epochs: 3
optimizer: paged_adamw_32bit
torchdistx_path:
lr_scheduler: cosine
learning_rate: 0.0002
train_on_inputs: false
group_by_length: false
bf16: true
fp16: false
tf32: true
gradient_checkpointing: true
# stop training after this many evaluation losses have increased in a row
# https://huggingface.co/transformers/v4.2.2/_modules/transformers/trainer_callback.html#EarlyStoppingCallback
early_stopping_patience: 3
resume_from_checkpoint:
auto_resume_from_checkpoints: true
local_rank:
logging_steps: 1
xformers_attention: false
flash_attention:
gptq_groupsize:
gptq_model_v1:
warmup_steps: 10
eval_steps: 5
save_steps: 10
debug:
deepspeed:
weight_decay: 0.000001
fsdp:
fsdp_config:
special_tokens:
pad_token: "<|endoftext|>"
bos_token: ">>ABSTRACT<<"
eos_token: "<|endoftext|>"
In [8]:
!accelerate launch scripts/finetune.py examples/falcon/qlora.yml
Setting ds_accelerator to cuda (auto detect)
===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please run
python -m bitsandbytes
and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================
bin /root/miniconda3/envs/py3.9/lib/python3.9/site-packages/bitsandbytes-0.39.0-py3.9.egg/bitsandbytes/libbitsandbytes_cuda118.so
/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/bitsandbytes-0.39.0-py3.9.egg/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/usr/local/nvidia/lib'), PosixPath('/usr/local/nvidia/lib64')}
warn(msg)
/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/bitsandbytes-0.39.0-py3.9.egg/bitsandbytes/cuda_setup/main.py:149: UserWarning: /usr/local/nvidia/lib:/usr/local/nvidia/lib64 did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths...
warn(msg)
/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/bitsandbytes-0.39.0-py3.9.egg/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//matplotlib_inline.backend_inline'), PosixPath('module')}
warn(msg)
/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/bitsandbytes-0.39.0-py3.9.egg/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIK1tFOFrWbmoa2ckCJYhzgBHKTSMeR/AeuScCCzugqlI utensilcandel@gmail.com')}
warn(msg)
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/bitsandbytes-0.39.0-py3.9.egg/bitsandbytes/cuda_setup/main.py:149: UserWarning: Found duplicate ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] files: {PosixPath('/usr/local/cuda/lib64/libcudart.so.11.0'), PosixPath('/usr/local/cuda/lib64/libcudart.so')}.. We'll flip a coin and try one of these, in order to fail forward.
Either way, this might cause trouble in the future:
If you get `CUDA error: invalid device function` errors, the above might be the cause and the solution is to make sure only one ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] in the paths that we search based on your env.
warn(msg)
CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so.11.0
CUDA SETUP: Highest compute capability among GPUs detected: 8.6
CUDA SETUP: Detected CUDA version 118
CUDA SETUP: Loading binary /root/miniconda3/envs/py3.9/lib/python3.9/site-packages/bitsandbytes-0.39.0-py3.9.egg/bitsandbytes/libbitsandbytes_cuda118.so...
Setting ds_accelerator to cuda (auto detect)
WARNING:root:`trust_remote_code` is set to true. Please make sure that you reviewed the remote code/model.
INFO:root:loading tokenizer... tiiuae/falcon-7b
Downloading (…)okenizer_config.json: 100%|█████| 220/220 [00:00<00:00, 35.1kB/s]
Downloading (…)/main/tokenizer.json: 100%|█| 2.73M/2.73M [00:00<00:00, 12.6MB/s]
Downloading (…)cial_tokens_map.json: 100%|██████| 281/281 [00:00<00:00, 409kB/s]
Using bos_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using unk_token, but it is not set yet.
INFO:root:Unable to find prepared dataset in last_run_prepared/31a4e867d804a957707db033c9abcd13
INFO:root:Loading raw datasets...
Downloading readme: 100%|██████████████████| 8.26k/8.26k [00:00<00:00, 15.9MB/s]
Downloading and preparing dataset json/QingyiSi--Alpaca-CoT to /root/.cache/huggingface/datasets/QingyiSi___json/QingyiSi--Alpaca-CoT-a2fee0ff0bfd6656/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4...
Downloading data files: 0%| | 0/1 [00:00<?, ?it/s]
Downloading data: 0%| | 0.00/4.45M [00:00<?, ?B/s]
Downloading data: 100%|████████████████████| 4.45M/4.45M [00:00<00:00, 32.4MB/s]
Downloading data files: 100%|█████████████████████| 1/1 [00:00<00:00, 1.66it/s]
Extracting data files: 100%|████████████████████| 1/1 [00:00<00:00, 1995.39it/s]
Dataset json downloaded and prepared to /root/.cache/huggingface/datasets/QingyiSi___json/QingyiSi--Alpaca-CoT-a2fee0ff0bfd6656/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4. Subsequent calls will reuse this data.
100%|████████████████████████████████████████████| 1/1 [00:00<00:00, 750.99it/s]
INFO:root:tokenizing, merging, and shuffling master dataset
INFO:root:Saving merged prepared dataset to disk... last_run_prepared/31a4e867d804a957707db033c9abcd13
INFO:root:loading model and peft_config...
Downloading (…)lve/main/config.json: 100%|██████| 950/950 [00:00<00:00, 197kB/s]
Downloading (…)/configuration_RW.py: 100%|█| 2.61k/2.61k [00:00<00:00, 2.53MB/s]
A new version of the following files was downloaded from https://huggingface.co/tiiuae/falcon-7b:
- configuration_RW.py
. Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.
Downloading (…)main/modelling_RW.py: 100%|█| 47.6k/47.6k [00:00<00:00, 43.5MB/s]
A new version of the following files was downloaded from https://huggingface.co/tiiuae/falcon-7b:
- modelling_RW.py
. Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.
Downloading (…)model.bin.index.json: 100%|█| 16.9k/16.9k [00:00<00:00, 20.6MB/s]
Downloading shards: 0%| | 0/2 [00:00<?, ?it/s]
Downloading (…)l-00001-of-00002.bin: 0%| | 0.00/9.95G [00:00<?, ?B/s]
Downloading (…)l-00001-of-00002.bin: 0%| | 10.5M/9.95G [00:00<02:33, 64.6MB/s]
Downloading (…)l-00001-of-00002.bin: 0%| | 31.5M/9.95G [00:00<01:44, 94.8MB/s]
Downloading (…)l-00001-of-00002.bin: 0%| | 41.9M/9.95G [00:00<01:48, 91.7MB/s]
Downloading (…)l-00001-of-00002.bin: 1%| | 62.9M/9.95G [00:00<01:38, 100MB/s]
Downloading (…)l-00001-of-00002.bin: 1%| | 83.9M/9.95G [00:00<01:34, 105MB/s]
Downloading (…)l-00001-of-00002.bin: 1%| | 105M/9.95G [00:01<01:32, 107MB/s]
Downloading (…)l-00001-of-00002.bin: 1%| | 126M/9.95G [00:01<01:30, 109MB/s]
Downloading (…)l-00001-of-00002.bin: 1%| | 147M/9.95G [00:01<01:29, 109MB/s]
Downloading (…)l-00001-of-00002.bin: 2%| | 168M/9.95G [00:01<01:29, 109MB/s]
Downloading (…)l-00001-of-00002.bin: 2%| | 189M/9.95G [00:01<01:29, 110MB/s]
Downloading (…)l-00001-of-00002.bin: 2%| | 210M/9.95G [00:01<01:28, 110MB/s]
Downloading (…)l-00001-of-00002.bin: 2%| | 231M/9.95G [00:02<01:31, 106MB/s]
Downloading (…)l-00001-of-00002.bin: 3%| | 252M/9.95G [00:02<01:29, 108MB/s]
Downloading (…)l-00001-of-00002.bin: 3%| | 273M/9.95G [00:02<01:29, 109MB/s]
Downloading (…)l-00001-of-00002.bin: 3%| | 294M/9.95G [00:02<01:28, 109MB/s]
Downloading (…)l-00001-of-00002.bin: 3%| | 315M/9.95G [00:02<01:27, 110MB/s]
Downloading (…)l-00001-of-00002.bin: 3%| | 336M/9.95G [00:03<01:27, 110MB/s]
Downloading (…)l-00001-of-00002.bin: 4%| | 357M/9.95G [00:03<01:26, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 4%| | 377M/9.95G [00:03<01:26, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 4%| | 398M/9.95G [00:03<01:26, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 4%|▏ | 419M/9.95G [00:03<01:25, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 4%|▏ | 440M/9.95G [00:04<01:25, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 5%|▏ | 461M/9.95G [00:04<01:25, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 5%|▏ | 482M/9.95G [00:04<01:25, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 5%|▏ | 503M/9.95G [00:04<01:25, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 5%|▏ | 524M/9.95G [00:04<01:25, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 5%|▏ | 545M/9.95G [00:05<01:26, 108MB/s]
Downloading (…)l-00001-of-00002.bin: 6%|▏ | 566M/9.95G [00:05<01:25, 109MB/s]
Downloading (…)l-00001-of-00002.bin: 6%|▏ | 587M/9.95G [00:05<01:25, 110MB/s]
Downloading (…)l-00001-of-00002.bin: 6%|▏ | 608M/9.95G [00:05<01:24, 110MB/s]
Downloading (…)l-00001-of-00002.bin: 6%|▏ | 629M/9.95G [00:05<01:24, 110MB/s]
Downloading (…)l-00001-of-00002.bin: 7%|▏ | 650M/9.95G [00:05<01:23, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 7%|▏ | 671M/9.95G [00:06<01:23, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 7%|▏ | 692M/9.95G [00:06<01:23, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 7%|▏ | 713M/9.95G [00:06<01:23, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 7%|▏ | 734M/9.95G [00:06<01:23, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 8%|▏ | 755M/9.95G [00:06<01:23, 110MB/s]
Downloading (…)l-00001-of-00002.bin: 8%|▏ | 776M/9.95G [00:07<01:22, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 8%|▏ | 797M/9.95G [00:07<01:22, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 8%|▏ | 818M/9.95G [00:07<01:22, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 8%|▎ | 839M/9.95G [00:07<01:22, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 9%|▎ | 860M/9.95G [00:07<01:21, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 9%|▎ | 881M/9.95G [00:08<01:21, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 9%|▎ | 902M/9.95G [00:08<01:21, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 9%|▎ | 923M/9.95G [00:08<01:21, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 9%|▎ | 944M/9.95G [00:08<01:21, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 10%|▎ | 965M/9.95G [00:08<01:20, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 10%|▎ | 986M/9.95G [00:09<01:21, 110MB/s]
Downloading (…)l-00001-of-00002.bin: 10%|▏ | 1.01G/9.95G [00:09<01:21, 109MB/s]
Downloading (…)l-00001-of-00002.bin: 10%|▏ | 1.03G/9.95G [00:09<01:21, 109MB/s]
Downloading (…)l-00001-of-00002.bin: 11%|▏ | 1.05G/9.95G [00:09<01:21, 110MB/s]
Downloading (…)l-00001-of-00002.bin: 11%|▏ | 1.07G/9.95G [00:09<01:20, 110MB/s]
Downloading (…)l-00001-of-00002.bin: 11%|▏ | 1.09G/9.95G [00:09<01:20, 110MB/s]
Downloading (…)l-00001-of-00002.bin: 11%|▏ | 1.11G/9.95G [00:10<01:19, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 11%|▏ | 1.13G/9.95G [00:10<01:19, 110MB/s]
Downloading (…)l-00001-of-00002.bin: 12%|▏ | 1.15G/9.95G [00:10<01:19, 110MB/s]
Downloading (…)l-00001-of-00002.bin: 12%|▏ | 1.17G/9.95G [00:10<01:19, 110MB/s]
Downloading (…)l-00001-of-00002.bin: 12%|▏ | 1.20G/9.95G [00:10<01:19, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 12%|▏ | 1.22G/9.95G [00:11<01:18, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 12%|▏ | 1.24G/9.95G [00:11<01:18, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 13%|▎ | 1.26G/9.95G [00:11<01:18, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 13%|▎ | 1.28G/9.95G [00:11<01:18, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 13%|▎ | 1.30G/9.95G [00:11<01:17, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 13%|▎ | 1.32G/9.95G [00:12<01:17, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 13%|▎ | 1.34G/9.95G [00:12<01:16, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 14%|▎ | 1.36G/9.95G [00:12<01:16, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 14%|▎ | 1.38G/9.95G [00:12<01:16, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 14%|▎ | 1.41G/9.95G [00:12<01:16, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 14%|▎ | 1.43G/9.95G [00:12<01:16, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 15%|▎ | 1.45G/9.95G [00:13<01:15, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 15%|▎ | 1.47G/9.95G [00:13<01:15, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 15%|▎ | 1.49G/9.95G [00:13<01:15, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 15%|▎ | 1.51G/9.95G [00:13<01:15, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 15%|▎ | 1.53G/9.95G [00:13<01:15, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 16%|▎ | 1.55G/9.95G [00:14<01:16, 110MB/s]
Downloading (…)l-00001-of-00002.bin: 16%|▎ | 1.57G/9.95G [00:14<01:16, 110MB/s]
Downloading (…)l-00001-of-00002.bin: 16%|▎ | 1.59G/9.95G [00:14<01:15, 110MB/s]
Downloading (…)l-00001-of-00002.bin: 16%|▎ | 1.61G/9.95G [00:14<01:15, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 16%|▎ | 1.64G/9.95G [00:14<01:14, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 17%|▎ | 1.66G/9.95G [00:15<01:14, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 17%|▎ | 1.68G/9.95G [00:15<01:14, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 17%|▎ | 1.70G/9.95G [00:15<01:14, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 17%|▎ | 1.72G/9.95G [00:15<01:14, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 17%|▎ | 1.74G/9.95G [00:15<01:13, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 18%|▎ | 1.76G/9.95G [00:16<01:13, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 18%|▎ | 1.78G/9.95G [00:16<01:12, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 18%|▎ | 1.80G/9.95G [00:16<01:13, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 18%|▎ | 1.82G/9.95G [00:16<01:12, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 19%|▎ | 1.85G/9.95G [00:16<01:12, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 19%|▍ | 1.87G/9.95G [00:16<01:12, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 19%|▍ | 1.89G/9.95G [00:17<01:12, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 19%|▍ | 1.91G/9.95G [00:17<01:12, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 19%|▍ | 1.93G/9.95G [00:17<01:12, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 20%|▍ | 1.95G/9.95G [00:17<01:12, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 20%|▍ | 1.97G/9.95G [00:17<01:11, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 20%|▍ | 1.99G/9.95G [00:18<01:11, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 20%|▍ | 2.01G/9.95G [00:18<01:11, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 20%|▍ | 2.03G/9.95G [00:18<01:11, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 21%|▍ | 2.06G/9.95G [00:18<01:10, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 21%|▍ | 2.08G/9.95G [00:18<01:10, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 21%|▍ | 2.10G/9.95G [00:19<01:10, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 21%|▍ | 2.12G/9.95G [00:19<01:10, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 21%|▍ | 2.14G/9.95G [00:19<01:10, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 22%|▍ | 2.16G/9.95G [00:19<01:09, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 22%|▍ | 2.18G/9.95G [00:19<01:09, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 22%|▍ | 2.20G/9.95G [00:19<01:09, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 22%|▍ | 2.22G/9.95G [00:20<01:09, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 23%|▍ | 2.24G/9.95G [00:20<01:08, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 23%|▍ | 2.26G/9.95G [00:20<01:08, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 23%|▍ | 2.29G/9.95G [00:20<01:08, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 23%|▍ | 2.31G/9.95G [00:20<01:08, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 23%|▍ | 2.33G/9.95G [00:21<01:08, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 24%|▍ | 2.35G/9.95G [00:21<01:08, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 24%|▍ | 2.37G/9.95G [00:21<01:07, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 24%|▍ | 2.39G/9.95G [00:21<01:08, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 24%|▍ | 2.41G/9.95G [00:21<01:07, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 24%|▍ | 2.43G/9.95G [00:22<01:07, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 25%|▍ | 2.45G/9.95G [00:22<01:07, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 25%|▍ | 2.47G/9.95G [00:22<01:07, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 25%|▌ | 2.50G/9.95G [00:22<01:07, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 25%|▌ | 2.52G/9.95G [00:22<01:06, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 26%|▌ | 2.54G/9.95G [00:22<01:06, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 26%|▌ | 2.56G/9.95G [00:23<01:06, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 26%|▌ | 2.58G/9.95G [00:23<01:06, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 26%|▌ | 2.60G/9.95G [00:23<01:05, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 26%|▌ | 2.62G/9.95G [00:23<01:05, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 27%|▌ | 2.64G/9.95G [00:23<01:05, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 27%|▌ | 2.66G/9.95G [00:24<01:05, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 27%|▌ | 2.68G/9.95G [00:24<01:05, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 27%|▌ | 2.71G/9.95G [00:24<01:05, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 27%|▌ | 2.73G/9.95G [00:24<01:04, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 28%|▌ | 2.75G/9.95G [00:24<01:04, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 28%|▌ | 2.77G/9.95G [00:25<01:04, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 28%|▌ | 2.79G/9.95G [00:25<01:04, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 28%|▌ | 2.81G/9.95G [00:25<01:03, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 28%|▌ | 2.83G/9.95G [00:25<01:03, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 29%|▌ | 2.85G/9.95G [00:25<01:03, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 29%|▌ | 2.87G/9.95G [00:25<01:03, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 29%|▌ | 2.89G/9.95G [00:26<01:03, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 29%|▌ | 2.92G/9.95G [00:26<01:02, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 30%|▌ | 2.94G/9.95G [00:26<01:02, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 30%|▌ | 2.96G/9.95G [00:26<01:02, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 30%|▌ | 2.98G/9.95G [00:26<01:02, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 30%|▌ | 3.00G/9.95G [00:27<01:01, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 30%|▌ | 3.02G/9.95G [00:27<01:01, 113MB/s]
Downloading (…)l-00001-of-00002.bin: 31%|▌ | 3.04G/9.95G [00:27<01:01, 113MB/s]
Downloading (…)l-00001-of-00002.bin: 31%|▌ | 3.06G/9.95G [00:27<01:00, 113MB/s]
Downloading (…)l-00001-of-00002.bin: 31%|▌ | 3.08G/9.95G [00:27<01:00, 114MB/s]
Downloading (…)l-00001-of-00002.bin: 31%|▌ | 3.10G/9.95G [00:28<01:00, 114MB/s]
Downloading (…)l-00001-of-00002.bin: 31%|▋ | 3.12G/9.95G [00:28<00:59, 114MB/s]
Downloading (…)l-00001-of-00002.bin: 32%|▋ | 3.15G/9.95G [00:28<00:59, 114MB/s]
Downloading (…)l-00001-of-00002.bin: 32%|▋ | 3.17G/9.95G [00:28<00:59, 114MB/s]
Downloading (…)l-00001-of-00002.bin: 32%|▋ | 3.19G/9.95G [00:28<00:59, 114MB/s]
Downloading (…)l-00001-of-00002.bin: 32%|▋ | 3.21G/9.95G [00:28<00:59, 114MB/s]
Downloading (…)l-00001-of-00002.bin: 32%|▋ | 3.23G/9.95G [00:29<00:59, 114MB/s]
Downloading (…)l-00001-of-00002.bin: 33%|▋ | 3.25G/9.95G [00:29<00:59, 113MB/s]
Downloading (…)l-00001-of-00002.bin: 33%|▋ | 3.27G/9.95G [00:29<01:00, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 33%|▋ | 3.29G/9.95G [00:29<00:59, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 33%|▋ | 3.31G/9.95G [00:29<00:59, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 34%|▋ | 3.33G/9.95G [00:30<00:59, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 34%|▋ | 3.36G/9.95G [00:30<00:59, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 34%|▋ | 3.38G/9.95G [00:30<00:59, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 34%|▋ | 3.40G/9.95G [00:30<00:58, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 34%|▋ | 3.42G/9.95G [00:30<00:58, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 35%|▋ | 3.44G/9.95G [00:31<00:58, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 35%|▋ | 3.46G/9.95G [00:31<00:58, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 35%|▋ | 3.48G/9.95G [00:31<00:57, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 35%|▋ | 3.50G/9.95G [00:31<00:57, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 35%|▋ | 3.52G/9.95G [00:31<00:57, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 36%|▋ | 3.54G/9.95G [00:31<00:57, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 36%|▋ | 3.57G/9.95G [00:32<00:57, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 36%|▋ | 3.59G/9.95G [00:32<00:56, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 36%|▋ | 3.61G/9.95G [00:32<00:56, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 36%|▋ | 3.63G/9.95G [00:32<00:56, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 37%|▋ | 3.65G/9.95G [00:32<00:56, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 37%|▋ | 3.67G/9.95G [00:33<00:56, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 37%|▋ | 3.69G/9.95G [00:33<00:56, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 37%|▋ | 3.71G/9.95G [00:33<00:55, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 38%|▊ | 3.73G/9.95G [00:33<00:55, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 38%|▊ | 3.75G/9.95G [00:33<00:55, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 38%|▊ | 3.77G/9.95G [00:34<00:55, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 38%|▊ | 3.80G/9.95G [00:34<00:55, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 38%|▊ | 3.82G/9.95G [00:34<00:55, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 39%|▊ | 3.84G/9.95G [00:34<00:55, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 39%|▊ | 3.86G/9.95G [00:34<00:54, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 39%|▊ | 3.88G/9.95G [00:34<00:54, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 39%|▊ | 3.90G/9.95G [00:35<00:54, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 39%|▊ | 3.92G/9.95G [00:35<00:53, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 40%|▊ | 3.94G/9.95G [00:35<00:53, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 40%|▊ | 3.96G/9.95G [00:35<00:53, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 40%|▊ | 3.98G/9.95G [00:35<00:53, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 40%|▊ | 4.01G/9.95G [00:36<00:53, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 40%|▊ | 4.03G/9.95G [00:36<00:52, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 41%|▊ | 4.05G/9.95G [00:36<00:52, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 41%|▊ | 4.07G/9.95G [00:36<00:52, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 41%|▊ | 4.09G/9.95G [00:36<00:53, 110MB/s]
Downloading (…)l-00001-of-00002.bin: 41%|▊ | 4.11G/9.95G [00:37<00:52, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 42%|▊ | 4.13G/9.95G [00:37<00:52, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 42%|▊ | 4.15G/9.95G [00:37<00:52, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 42%|▊ | 4.17G/9.95G [00:37<00:51, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 42%|▊ | 4.19G/9.95G [00:37<00:51, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 42%|▊ | 4.22G/9.95G [00:37<00:51, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 43%|▊ | 4.24G/9.95G [00:38<00:51, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 43%|▊ | 4.26G/9.95G [00:38<00:50, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 43%|▊ | 4.28G/9.95G [00:38<00:50, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 43%|▊ | 4.30G/9.95G [00:38<00:50, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 43%|▊ | 4.32G/9.95G [00:38<00:50, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 44%|▊ | 4.34G/9.95G [00:39<00:50, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 44%|▉ | 4.36G/9.95G [00:39<00:49, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 44%|▉ | 4.38G/9.95G [00:39<00:49, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 44%|▉ | 4.40G/9.95G [00:39<00:50, 109MB/s]
Downloading (…)l-00001-of-00002.bin: 44%|▉ | 4.42G/9.95G [00:39<00:50, 109MB/s]
Downloading (…)l-00001-of-00002.bin: 45%|▉ | 4.45G/9.95G [00:40<00:50, 110MB/s]
Downloading (…)l-00001-of-00002.bin: 45%|▉ | 4.47G/9.95G [00:40<00:49, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 45%|▉ | 4.49G/9.95G [00:40<00:49, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 45%|▉ | 4.51G/9.95G [00:40<00:48, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 46%|▉ | 4.53G/9.95G [00:40<00:48, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 46%|▉ | 4.55G/9.95G [00:40<00:48, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 46%|▉ | 4.57G/9.95G [00:41<00:48, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 46%|▉ | 4.59G/9.95G [00:41<00:48, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 46%|▉ | 4.61G/9.95G [00:41<00:47, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 47%|▉ | 4.63G/9.95G [00:41<00:47, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 47%|▉ | 4.66G/9.95G [00:41<00:47, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 47%|▉ | 4.68G/9.95G [00:42<00:47, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 47%|▉ | 4.70G/9.95G [00:42<00:46, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 47%|▉ | 4.72G/9.95G [00:42<00:46, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 48%|▉ | 4.74G/9.95G [00:42<00:46, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 48%|▉ | 4.76G/9.95G [00:42<00:46, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 48%|▉ | 4.78G/9.95G [00:43<00:46, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 48%|▉ | 4.80G/9.95G [00:43<00:46, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 48%|▉ | 4.82G/9.95G [00:43<00:45, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 49%|▉ | 4.84G/9.95G [00:43<00:45, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 49%|▉ | 4.87G/9.95G [00:43<00:45, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 49%|▉ | 4.89G/9.95G [00:43<00:45, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 49%|▉ | 4.91G/9.95G [00:44<00:45, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 50%|▉ | 4.93G/9.95G [00:44<00:45, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 50%|▉ | 4.95G/9.95G [00:44<00:44, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 50%|▉ | 4.97G/9.95G [00:44<00:44, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 50%|█ | 4.99G/9.95G [00:44<00:44, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 50%|█ | 5.01G/9.95G [00:45<00:44, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 51%|█ | 5.03G/9.95G [00:45<00:44, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 51%|█ | 5.05G/9.95G [00:45<00:44, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 51%|█ | 5.08G/9.95G [00:45<00:43, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 51%|█ | 5.10G/9.95G [00:45<00:43, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 51%|█ | 5.12G/9.95G [00:46<00:43, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 52%|█ | 5.14G/9.95G [00:46<00:42, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 52%|█ | 5.16G/9.95G [00:46<00:42, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 52%|█ | 5.18G/9.95G [00:46<00:42, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 52%|█ | 5.20G/9.95G [00:46<00:42, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 52%|█ | 5.22G/9.95G [00:46<00:42, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 53%|█ | 5.24G/9.95G [00:47<00:42, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 53%|█ | 5.26G/9.95G [00:47<00:42, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 53%|█ | 5.28G/9.95G [00:47<00:42, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 53%|█ | 5.31G/9.95G [00:47<00:41, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 54%|█ | 5.33G/9.95G [00:47<00:41, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 54%|█ | 5.35G/9.95G [00:48<00:41, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 54%|█ | 5.37G/9.95G [00:48<00:41, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 54%|█ | 5.39G/9.95G [00:48<00:41, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 54%|█ | 5.41G/9.95G [00:48<00:40, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 55%|█ | 5.43G/9.95G [00:48<00:40, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 55%|█ | 5.45G/9.95G [00:49<00:40, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 55%|█ | 5.47G/9.95G [00:49<00:40, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 55%|█ | 5.49G/9.95G [00:49<00:40, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 55%|█ | 5.52G/9.95G [00:49<00:40, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 56%|█ | 5.54G/9.95G [00:49<00:39, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 56%|█ | 5.56G/9.95G [00:50<00:39, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 56%|█ | 5.58G/9.95G [00:50<00:39, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 56%|█▏| 5.60G/9.95G [00:50<00:39, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 56%|█▏| 5.62G/9.95G [00:50<00:38, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 57%|█▏| 5.64G/9.95G [00:50<00:38, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 57%|█▏| 5.66G/9.95G [00:50<00:38, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 57%|█▏| 5.68G/9.95G [00:51<00:38, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 57%|█▏| 5.70G/9.95G [00:51<00:38, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 58%|█▏| 5.73G/9.95G [00:51<00:37, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 58%|█▏| 5.75G/9.95G [00:51<00:37, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 58%|█▏| 5.77G/9.95G [00:51<00:37, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 58%|█▏| 5.79G/9.95G [00:52<00:37, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 58%|█▏| 5.81G/9.95G [00:52<00:37, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 59%|█▏| 5.83G/9.95G [00:52<00:37, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 59%|█▏| 5.85G/9.95G [00:52<00:36, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 59%|█▏| 5.87G/9.95G [00:52<00:36, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 59%|█▏| 5.89G/9.95G [00:53<00:36, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 59%|█▏| 5.91G/9.95G [00:53<00:36, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 60%|█▏| 5.93G/9.95G [00:53<00:36, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 60%|█▏| 5.96G/9.95G [00:53<00:35, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 60%|█▏| 5.98G/9.95G [00:53<00:35, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 60%|█▏| 6.00G/9.95G [00:53<00:35, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 60%|█▏| 6.02G/9.95G [00:54<00:35, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 61%|█▏| 6.04G/9.95G [00:54<00:35, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 61%|█▏| 6.06G/9.95G [00:54<00:34, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 61%|█▏| 6.08G/9.95G [00:54<00:34, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 61%|█▏| 6.10G/9.95G [00:54<00:34, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 62%|█▏| 6.12G/9.95G [00:55<00:34, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 62%|█▏| 6.14G/9.95G [00:55<00:34, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 62%|█▏| 6.17G/9.95G [00:55<00:33, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 62%|█▏| 6.19G/9.95G [00:55<00:33, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 62%|█▏| 6.21G/9.95G [00:55<00:33, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 63%|█▎| 6.23G/9.95G [00:56<00:33, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 63%|█▎| 6.25G/9.95G [00:56<00:32, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 63%|█▎| 6.27G/9.95G [00:56<00:32, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 63%|█▎| 6.29G/9.95G [00:56<00:32, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 63%|█▎| 6.31G/9.95G [00:56<00:32, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 64%|█▎| 6.33G/9.95G [00:56<00:32, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 64%|█▎| 6.35G/9.95G [00:57<00:32, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 64%|█▎| 6.38G/9.95G [00:57<00:31, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 64%|█▎| 6.40G/9.95G [00:57<00:31, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 64%|█▎| 6.42G/9.95G [00:57<00:32, 110MB/s]
Downloading (…)l-00001-of-00002.bin: 65%|█▎| 6.44G/9.95G [00:57<00:31, 110MB/s]
Downloading (…)l-00001-of-00002.bin: 65%|█▎| 6.46G/9.95G [00:58<00:31, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 65%|█▎| 6.48G/9.95G [00:58<00:31, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 65%|█▎| 6.50G/9.95G [00:58<00:31, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 66%|█▎| 6.52G/9.95G [00:58<00:30, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 66%|█▎| 6.54G/9.95G [00:58<00:30, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 66%|█▎| 6.56G/9.95G [00:59<00:30, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 66%|█▎| 6.59G/9.95G [00:59<00:30, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 66%|█▎| 6.61G/9.95G [00:59<00:30, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 67%|█▎| 6.63G/9.95G [00:59<00:29, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 67%|█▎| 6.65G/9.95G [00:59<00:29, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 67%|█▎| 6.67G/9.95G [00:59<00:29, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 67%|█▎| 6.69G/9.95G [01:00<00:29, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 67%|█▎| 6.71G/9.95G [01:00<00:29, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 68%|█▎| 6.73G/9.95G [01:00<00:29, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 68%|█▎| 6.75G/9.95G [01:00<00:28, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 68%|█▎| 6.77G/9.95G [01:00<00:28, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 68%|█▎| 6.79G/9.95G [01:01<00:28, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 68%|█▎| 6.82G/9.95G [01:01<00:28, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 69%|█▎| 6.84G/9.95G [01:01<00:28, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 69%|█▍| 6.86G/9.95G [01:01<00:27, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 69%|█▍| 6.88G/9.95G [01:01<00:27, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 69%|█▍| 6.90G/9.95G [01:02<00:27, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 70%|█▍| 6.92G/9.95G [01:02<00:27, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 70%|█▍| 6.94G/9.95G [01:02<00:27, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 70%|█▍| 6.96G/9.95G [01:02<00:26, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 70%|█▍| 6.98G/9.95G [01:02<00:26, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 70%|█▍| 7.00G/9.95G [01:03<00:26, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 71%|█▍| 7.03G/9.95G [01:03<00:26, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 71%|█▍| 7.05G/9.95G [01:03<00:26, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 71%|█▍| 7.07G/9.95G [01:03<00:25, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 71%|█▍| 7.09G/9.95G [01:03<00:25, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 71%|█▍| 7.11G/9.95G [01:03<00:25, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 72%|█▍| 7.13G/9.95G [01:04<00:25, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 72%|█▍| 7.15G/9.95G [01:04<00:24, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 72%|█▍| 7.17G/9.95G [01:04<00:24, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 72%|█▍| 7.19G/9.95G [01:04<00:24, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 72%|█▍| 7.21G/9.95G [01:04<00:24, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 73%|█▍| 7.24G/9.95G [01:05<00:24, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 73%|█▍| 7.26G/9.95G [01:05<00:24, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 73%|█▍| 7.28G/9.95G [01:05<00:24, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 73%|█▍| 7.30G/9.95G [01:05<00:23, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 74%|█▍| 7.32G/9.95G [01:05<00:23, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 74%|█▍| 7.34G/9.95G [01:06<00:23, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 74%|█▍| 7.36G/9.95G [01:06<00:23, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 74%|█▍| 7.38G/9.95G [01:06<00:23, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 74%|█▍| 7.40G/9.95G [01:06<00:22, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 75%|█▍| 7.42G/9.95G [01:06<00:22, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 75%|█▍| 7.44G/9.95G [01:06<00:22, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 75%|█▌| 7.47G/9.95G [01:07<00:22, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 75%|█▌| 7.49G/9.95G [01:07<00:22, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 75%|█▌| 7.51G/9.95G [01:07<00:21, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 76%|█▌| 7.53G/9.95G [01:07<00:21, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 76%|█▌| 7.55G/9.95G [01:07<00:21, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 76%|█▌| 7.57G/9.95G [01:08<00:21, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 76%|█▌| 7.59G/9.95G [01:08<00:21, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 77%|█▌| 7.61G/9.95G [01:08<00:20, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 77%|█▌| 7.63G/9.95G [01:08<00:20, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 77%|█▌| 7.65G/9.95G [01:08<00:20, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 77%|█▌| 7.68G/9.95G [01:09<00:20, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 77%|█▌| 7.70G/9.95G [01:09<00:20, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 78%|█▌| 7.72G/9.95G [01:09<00:20, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 78%|█▌| 7.74G/9.95G [01:09<00:19, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 78%|█▌| 7.76G/9.95G [01:09<00:19, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 78%|█▌| 7.78G/9.95G [01:09<00:19, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 78%|█▌| 7.80G/9.95G [01:10<00:19, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 79%|█▌| 7.82G/9.95G [01:10<00:19, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 79%|█▌| 7.84G/9.95G [01:10<00:18, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 79%|█▌| 7.86G/9.95G [01:10<00:18, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 79%|█▌| 7.89G/9.95G [01:10<00:18, 113MB/s]
Downloading (…)l-00001-of-00002.bin: 79%|█▌| 7.91G/9.95G [01:11<00:18, 113MB/s]
Downloading (…)l-00001-of-00002.bin: 80%|█▌| 7.93G/9.95G [01:11<00:17, 113MB/s]
Downloading (…)l-00001-of-00002.bin: 80%|█▌| 7.95G/9.95G [01:11<00:17, 113MB/s]
Downloading (…)l-00001-of-00002.bin: 80%|█▌| 7.97G/9.95G [01:11<00:17, 113MB/s]
Downloading (…)l-00001-of-00002.bin: 80%|█▌| 7.99G/9.95G [01:11<00:17, 113MB/s]
Downloading (…)l-00001-of-00002.bin: 81%|█▌| 8.01G/9.95G [01:12<00:17, 113MB/s]
Downloading (…)l-00001-of-00002.bin: 81%|█▌| 8.03G/9.95G [01:12<00:16, 113MB/s]
Downloading (…)l-00001-of-00002.bin: 81%|█▌| 8.05G/9.95G [01:12<00:16, 113MB/s]
Downloading (…)l-00001-of-00002.bin: 81%|█▌| 8.07G/9.95G [01:12<00:16, 113MB/s]
Downloading (…)l-00001-of-00002.bin: 81%|█▋| 8.10G/9.95G [01:12<00:16, 113MB/s]
Downloading (…)l-00001-of-00002.bin: 82%|█▋| 8.12G/9.95G [01:12<00:16, 113MB/s]
Downloading (…)l-00001-of-00002.bin: 82%|█▋| 8.14G/9.95G [01:13<00:16, 113MB/s]
Downloading (…)l-00001-of-00002.bin: 82%|█▋| 8.16G/9.95G [01:13<00:15, 113MB/s]
Downloading (…)l-00001-of-00002.bin: 82%|█▋| 8.18G/9.95G [01:13<00:15, 113MB/s]
Downloading (…)l-00001-of-00002.bin: 82%|█▋| 8.20G/9.95G [01:13<00:15, 113MB/s]
Downloading (…)l-00001-of-00002.bin: 83%|█▋| 8.22G/9.95G [01:13<00:15, 113MB/s]
Downloading (…)l-00001-of-00002.bin: 83%|█▋| 8.24G/9.95G [01:14<00:15, 113MB/s]
Downloading (…)l-00001-of-00002.bin: 83%|█▋| 8.26G/9.95G [01:14<00:14, 113MB/s]
Downloading (…)l-00001-of-00002.bin: 83%|█▋| 8.28G/9.95G [01:14<00:14, 113MB/s]
Downloading (…)l-00001-of-00002.bin: 83%|█▋| 8.30G/9.95G [01:14<00:14, 113MB/s]
Downloading (…)l-00001-of-00002.bin: 84%|█▋| 8.33G/9.95G [01:14<00:14, 113MB/s]
Downloading (…)l-00001-of-00002.bin: 84%|█▋| 8.35G/9.95G [01:14<00:14, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 84%|█▋| 8.37G/9.95G [01:15<00:14, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 84%|█▋| 8.39G/9.95G [01:15<00:14, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 85%|█▋| 8.41G/9.95G [01:15<00:13, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 85%|█▋| 8.43G/9.95G [01:15<00:13, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 85%|█▋| 8.45G/9.95G [01:15<00:13, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 85%|█▋| 8.47G/9.95G [01:16<00:13, 113MB/s]
Downloading (…)l-00001-of-00002.bin: 85%|█▋| 8.49G/9.95G [01:16<00:12, 113MB/s]
Downloading (…)l-00001-of-00002.bin: 86%|█▋| 8.51G/9.95G [01:16<00:12, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 86%|█▋| 8.54G/9.95G [01:16<00:12, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 86%|█▋| 8.56G/9.95G [01:16<00:12, 113MB/s]
Downloading (…)l-00001-of-00002.bin: 86%|█▋| 8.58G/9.95G [01:17<00:12, 113MB/s]
Downloading (…)l-00001-of-00002.bin: 86%|█▋| 8.60G/9.95G [01:17<00:11, 113MB/s]
Downloading (…)l-00001-of-00002.bin: 87%|█▋| 8.62G/9.95G [01:17<00:11, 113MB/s]
Downloading (…)l-00001-of-00002.bin: 87%|█▋| 8.64G/9.95G [01:17<00:11, 113MB/s]
Downloading (…)l-00001-of-00002.bin: 87%|█▋| 8.66G/9.95G [01:17<00:11, 113MB/s]
Downloading (…)l-00001-of-00002.bin: 87%|█▋| 8.68G/9.95G [01:17<00:11, 113MB/s]
Downloading (…)l-00001-of-00002.bin: 87%|█▋| 8.70G/9.95G [01:18<00:11, 113MB/s]
Downloading (…)l-00001-of-00002.bin: 88%|█▊| 8.72G/9.95G [01:18<00:10, 113MB/s]
Downloading (…)l-00001-of-00002.bin: 88%|█▊| 8.75G/9.95G [01:18<00:10, 113MB/s]
Downloading (…)l-00001-of-00002.bin: 88%|█▊| 8.77G/9.95G [01:18<00:10, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 88%|█▊| 8.79G/9.95G [01:18<00:10, 113MB/s]
Downloading (…)l-00001-of-00002.bin: 89%|█▊| 8.81G/9.95G [01:19<00:10, 113MB/s]
Downloading (…)l-00001-of-00002.bin: 89%|█▊| 8.83G/9.95G [01:19<00:09, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 89%|█▊| 8.85G/9.95G [01:19<00:09, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 89%|█▊| 8.87G/9.95G [01:19<00:09, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 89%|█▊| 8.89G/9.95G [01:19<00:09, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 90%|█▊| 8.91G/9.95G [01:20<00:09, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 90%|▉| 8.93G/9.95G [01:20<00:10, 93.8MB/s]
Downloading (…)l-00001-of-00002.bin: 90%|▉| 8.94G/9.95G [01:20<00:11, 88.9MB/s]
Downloading (…)l-00001-of-00002.bin: 90%|▉| 8.95G/9.95G [01:20<00:11, 88.1MB/s]
Downloading (…)l-00001-of-00002.bin: 90%|▉| 8.97G/9.95G [01:20<00:11, 88.5MB/s]
Downloading (…)l-00001-of-00002.bin: 90%|▉| 8.98G/9.95G [01:20<00:10, 91.1MB/s]
Downloading (…)l-00001-of-00002.bin: 90%|▉| 9.00G/9.95G [01:21<00:09, 97.7MB/s]
Downloading (…)l-00001-of-00002.bin: 91%|█▊| 9.02G/9.95G [01:21<00:09, 103MB/s]
Downloading (…)l-00001-of-00002.bin: 91%|█▊| 9.04G/9.95G [01:21<00:08, 106MB/s]
Downloading (…)l-00001-of-00002.bin: 91%|█▊| 9.06G/9.95G [01:21<00:08, 109MB/s]
Downloading (…)l-00001-of-00002.bin: 91%|█▊| 9.08G/9.95G [01:21<00:07, 111MB/s]
Downloading (…)l-00001-of-00002.bin: 91%|█▊| 9.10G/9.95G [01:21<00:07, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 92%|█▊| 9.12G/9.95G [01:22<00:07, 113MB/s]
Downloading (…)l-00001-of-00002.bin: 92%|█▊| 9.14G/9.95G [01:22<00:07, 113MB/s]
Downloading (…)l-00001-of-00002.bin: 92%|█▊| 9.16G/9.95G [01:22<00:06, 114MB/s]
Downloading (…)l-00001-of-00002.bin: 92%|█▊| 9.19G/9.95G [01:22<00:06, 114MB/s]
Downloading (…)l-00001-of-00002.bin: 93%|█▊| 9.21G/9.95G [01:22<00:06, 114MB/s]
Downloading (…)l-00001-of-00002.bin: 93%|█▊| 9.23G/9.95G [01:23<00:06, 115MB/s]
Downloading (…)l-00001-of-00002.bin: 93%|█▊| 9.25G/9.95G [01:23<00:06, 107MB/s]
Downloading (…)l-00001-of-00002.bin: 93%|█▊| 9.27G/9.95G [01:23<00:06, 107MB/s]
Downloading (…)l-00001-of-00002.bin: 93%|█▊| 9.29G/9.95G [01:23<00:06, 108MB/s]
Downloading (…)l-00001-of-00002.bin: 94%|█▊| 9.31G/9.95G [01:23<00:05, 109MB/s]
Downloading (…)l-00001-of-00002.bin: 94%|█▉| 9.33G/9.95G [01:24<00:05, 110MB/s]
Downloading (…)l-00001-of-00002.bin: 94%|█▉| 9.35G/9.95G [01:24<00:05, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 94%|█▉| 9.37G/9.95G [01:24<00:05, 113MB/s]
Downloading (…)l-00001-of-00002.bin: 94%|█▉| 9.40G/9.95G [01:24<00:04, 113MB/s]
Downloading (…)l-00001-of-00002.bin: 95%|█▉| 9.42G/9.95G [01:24<00:04, 114MB/s]
Downloading (…)l-00001-of-00002.bin: 95%|█▉| 9.44G/9.95G [01:24<00:04, 114MB/s]
Downloading (…)l-00001-of-00002.bin: 95%|█▉| 9.46G/9.95G [01:25<00:04, 114MB/s]
Downloading (…)l-00001-of-00002.bin: 95%|█▉| 9.48G/9.95G [01:25<00:04, 114MB/s]
Downloading (…)l-00001-of-00002.bin: 95%|█▉| 9.50G/9.95G [01:25<00:03, 114MB/s]
Downloading (…)l-00001-of-00002.bin: 96%|█▉| 9.52G/9.95G [01:25<00:03, 114MB/s]
Downloading (…)l-00001-of-00002.bin: 96%|█▉| 9.54G/9.95G [01:25<00:03, 113MB/s]
Downloading (…)l-00001-of-00002.bin: 96%|█▉| 9.56G/9.95G [01:26<00:03, 113MB/s]
Downloading (…)l-00001-of-00002.bin: 96%|█▉| 9.58G/9.95G [01:26<00:03, 113MB/s]
Downloading (…)l-00001-of-00002.bin: 97%|█▉| 9.60G/9.95G [01:26<00:03, 114MB/s]
Downloading (…)l-00001-of-00002.bin: 97%|█▉| 9.63G/9.95G [01:26<00:02, 115MB/s]
Downloading (…)l-00001-of-00002.bin: 97%|█▉| 9.65G/9.95G [01:26<00:02, 114MB/s]
Downloading (…)l-00001-of-00002.bin: 97%|█▉| 9.67G/9.95G [01:26<00:02, 114MB/s]
Downloading (…)l-00001-of-00002.bin: 97%|█▉| 9.69G/9.95G [01:27<00:02, 114MB/s]
Downloading (…)l-00001-of-00002.bin: 98%|█▉| 9.71G/9.95G [01:27<00:02, 114MB/s]
Downloading (…)l-00001-of-00002.bin: 98%|█▉| 9.73G/9.95G [01:27<00:01, 115MB/s]
Downloading (…)l-00001-of-00002.bin: 98%|█▉| 9.75G/9.95G [01:27<00:01, 115MB/s]
Downloading (…)l-00001-of-00002.bin: 98%|█▉| 9.77G/9.95G [01:27<00:01, 115MB/s]
Downloading (…)l-00001-of-00002.bin: 98%|█▉| 9.79G/9.95G [01:28<00:01, 115MB/s]
Downloading (…)l-00001-of-00002.bin: 99%|█▉| 9.81G/9.95G [01:28<00:01, 114MB/s]
Downloading (…)l-00001-of-00002.bin: 99%|█▉| 9.84G/9.95G [01:28<00:01, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 99%|█▉| 9.86G/9.95G [01:28<00:00, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 99%|█▉| 9.88G/9.95G [01:28<00:00, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 99%|█▉| 9.90G/9.95G [01:28<00:00, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 100%|█▉| 9.92G/9.95G [01:29<00:00, 112MB/s]
Downloading (…)l-00001-of-00002.bin: 100%|██| 9.95G/9.95G [01:29<00:00, 111MB/s]
Downloading shards: 50%|████████████▌ | 1/2 [01:29<01:29, 89.55s/it]
Downloading (…)l-00002-of-00002.bin: 0%| | 0.00/4.48G [00:00<?, ?B/s]
Downloading (…)l-00002-of-00002.bin: 0%| | 10.5M/4.48G [00:00<00:44, 101MB/s]
Downloading (…)l-00002-of-00002.bin: 1%| | 31.5M/4.48G [00:00<00:41, 107MB/s]
Downloading (…)l-00002-of-00002.bin: 1%| | 52.4M/4.48G [00:00<00:40, 109MB/s]
Downloading (…)l-00002-of-00002.bin: 2%| | 73.4M/4.48G [00:00<00:39, 110MB/s]
Downloading (…)l-00002-of-00002.bin: 2%| | 94.4M/4.48G [00:00<00:39, 111MB/s]
Downloading (…)l-00002-of-00002.bin: 3%| | 115M/4.48G [00:01<00:38, 112MB/s]
Downloading (…)l-00002-of-00002.bin: 3%| | 136M/4.48G [00:01<00:38, 112MB/s]
Downloading (…)l-00002-of-00002.bin: 4%| | 157M/4.48G [00:01<00:38, 112MB/s]
Downloading (…)l-00002-of-00002.bin: 4%| | 178M/4.48G [00:01<00:38, 112MB/s]
Downloading (…)l-00002-of-00002.bin: 4%|▏ | 199M/4.48G [00:01<00:38, 112MB/s]
Downloading (…)l-00002-of-00002.bin: 5%|▏ | 220M/4.48G [00:01<00:38, 112MB/s]
Downloading (…)l-00002-of-00002.bin: 5%|▏ | 241M/4.48G [00:02<00:38, 110MB/s]
Downloading (…)l-00002-of-00002.bin: 6%|▏ | 262M/4.48G [00:02<00:38, 109MB/s]
Downloading (…)l-00002-of-00002.bin: 6%|▏ | 283M/4.48G [00:02<00:39, 108MB/s]
Downloading (…)l-00002-of-00002.bin: 7%|▏ | 304M/4.48G [00:02<00:38, 107MB/s]
Downloading (…)l-00002-of-00002.bin: 7%|▏ | 325M/4.48G [00:02<00:38, 107MB/s]
Downloading (…)l-00002-of-00002.bin: 8%|▏ | 346M/4.48G [00:03<00:38, 107MB/s]
Downloading (…)l-00002-of-00002.bin: 8%|▏ | 367M/4.48G [00:03<00:38, 106MB/s]
Downloading (…)l-00002-of-00002.bin: 9%|▎ | 388M/4.48G [00:03<00:38, 106MB/s]
Downloading (…)l-00002-of-00002.bin: 9%|▎ | 409M/4.48G [00:03<00:38, 106MB/s]
Downloading (…)l-00002-of-00002.bin: 10%|▎ | 430M/4.48G [00:03<00:38, 106MB/s]
Downloading (…)l-00002-of-00002.bin: 10%|▎ | 451M/4.48G [00:04<00:37, 106MB/s]
Downloading (…)l-00002-of-00002.bin: 11%|▎ | 472M/4.48G [00:04<00:37, 106MB/s]
Downloading (…)l-00002-of-00002.bin: 11%|▎ | 493M/4.48G [00:04<00:37, 106MB/s]
Downloading (…)l-00002-of-00002.bin: 11%|▎ | 514M/4.48G [00:04<00:37, 106MB/s]
Downloading (…)l-00002-of-00002.bin: 12%|▎ | 535M/4.48G [00:04<00:36, 107MB/s]
Downloading (…)l-00002-of-00002.bin: 12%|▎ | 556M/4.48G [00:05<00:36, 107MB/s]
Downloading (…)l-00002-of-00002.bin: 13%|▍ | 577M/4.48G [00:05<00:36, 107MB/s]
Downloading (…)l-00002-of-00002.bin: 13%|▍ | 598M/4.48G [00:05<00:36, 107MB/s]
Downloading (…)l-00002-of-00002.bin: 14%|▍ | 619M/4.48G [00:05<00:36, 107MB/s]
Downloading (…)l-00002-of-00002.bin: 14%|▍ | 640M/4.48G [00:05<00:35, 107MB/s]
Downloading (…)l-00002-of-00002.bin: 15%|▍ | 661M/4.48G [00:06<00:35, 107MB/s]
Downloading (…)l-00002-of-00002.bin: 15%|▍ | 682M/4.48G [00:06<00:35, 107MB/s]
Downloading (…)l-00002-of-00002.bin: 16%|▍ | 703M/4.48G [00:06<00:35, 108MB/s]
Downloading (…)l-00002-of-00002.bin: 16%|▍ | 724M/4.48G [00:06<00:35, 107MB/s]
Downloading (…)l-00002-of-00002.bin: 17%|▍ | 744M/4.48G [00:06<00:34, 107MB/s]
Downloading (…)l-00002-of-00002.bin: 17%|▌ | 765M/4.48G [00:07<00:34, 107MB/s]
Downloading (…)l-00002-of-00002.bin: 18%|▌ | 786M/4.48G [00:07<00:34, 107MB/s]
Downloading (…)l-00002-of-00002.bin: 18%|▌ | 807M/4.48G [00:07<00:34, 106MB/s]
Downloading (…)l-00002-of-00002.bin: 18%|▌ | 828M/4.48G [00:07<00:34, 106MB/s]
Downloading (…)l-00002-of-00002.bin: 19%|▌ | 849M/4.48G [00:07<00:34, 107MB/s]
Downloading (…)l-00002-of-00002.bin: 19%|▌ | 870M/4.48G [00:08<00:33, 107MB/s]
Downloading (…)l-00002-of-00002.bin: 20%|▌ | 891M/4.48G [00:08<00:33, 107MB/s]
Downloading (…)l-00002-of-00002.bin: 20%|▌ | 912M/4.48G [00:08<00:33, 106MB/s]
Downloading (…)l-00002-of-00002.bin: 21%|▌ | 933M/4.48G [00:08<00:33, 106MB/s]
Downloading (…)l-00002-of-00002.bin: 21%|▋ | 954M/4.48G [00:08<00:33, 106MB/s]
Downloading (…)l-00002-of-00002.bin: 22%|▋ | 975M/4.48G [00:09<00:32, 108MB/s]
Downloading (…)l-00002-of-00002.bin: 22%|▋ | 996M/4.48G [00:09<00:32, 108MB/s]
Downloading (…)l-00002-of-00002.bin: 23%|▍ | 1.02G/4.48G [00:09<00:31, 109MB/s]
Downloading (…)l-00002-of-00002.bin: 23%|▍ | 1.04G/4.48G [00:09<00:31, 109MB/s]
Downloading (…)l-00002-of-00002.bin: 24%|▍ | 1.06G/4.48G [00:09<00:31, 108MB/s]
Downloading (…)l-00002-of-00002.bin: 24%|▍ | 1.08G/4.48G [00:10<00:31, 109MB/s]
Downloading (…)l-00002-of-00002.bin: 25%|▍ | 1.10G/4.48G [00:10<00:30, 110MB/s]
Downloading (…)l-00002-of-00002.bin: 25%|▌ | 1.12G/4.48G [00:10<00:30, 110MB/s]
Downloading (…)l-00002-of-00002.bin: 25%|▌ | 1.14G/4.48G [00:10<00:30, 110MB/s]
Downloading (…)l-00002-of-00002.bin: 26%|▌ | 1.16G/4.48G [00:10<00:30, 110MB/s]
Downloading (…)l-00002-of-00002.bin: 26%|▌ | 1.18G/4.48G [00:10<00:29, 111MB/s]
Downloading (…)l-00002-of-00002.bin: 27%|▌ | 1.21G/4.48G [00:11<00:29, 111MB/s]
Downloading (…)l-00002-of-00002.bin: 27%|▌ | 1.23G/4.48G [00:11<00:29, 111MB/s]
Downloading (…)l-00002-of-00002.bin: 28%|▌ | 1.25G/4.48G [00:11<00:29, 111MB/s]
Downloading (…)l-00002-of-00002.bin: 28%|▌ | 1.27G/4.48G [00:11<00:28, 111MB/s]
Downloading (…)l-00002-of-00002.bin: 29%|▌ | 1.29G/4.48G [00:11<00:28, 111MB/s]
Downloading (…)l-00002-of-00002.bin: 29%|▌ | 1.31G/4.48G [00:12<00:28, 111MB/s]
Downloading (…)l-00002-of-00002.bin: 30%|▌ | 1.33G/4.48G [00:12<00:28, 111MB/s]
Downloading (…)l-00002-of-00002.bin: 30%|▌ | 1.35G/4.48G [00:12<00:28, 111MB/s]
Downloading (…)l-00002-of-00002.bin: 31%|▌ | 1.37G/4.48G [00:12<00:27, 111MB/s]
Downloading (…)l-00002-of-00002.bin: 31%|▌ | 1.39G/4.48G [00:12<00:27, 111MB/s]
Downloading (…)l-00002-of-00002.bin: 32%|▋ | 1.42G/4.48G [00:13<00:27, 111MB/s]
Downloading (…)l-00002-of-00002.bin: 32%|▋ | 1.44G/4.48G [00:13<00:27, 111MB/s]
Downloading (…)l-00002-of-00002.bin: 33%|▋ | 1.46G/4.48G [00:13<00:27, 110MB/s]
Downloading (…)l-00002-of-00002.bin: 33%|▋ | 1.48G/4.48G [00:13<00:27, 110MB/s]
Downloading (…)l-00002-of-00002.bin: 33%|▋ | 1.50G/4.48G [00:13<00:26, 111MB/s]
Downloading (…)l-00002-of-00002.bin: 34%|▋ | 1.52G/4.48G [00:13<00:26, 111MB/s]
Downloading (…)l-00002-of-00002.bin: 34%|▋ | 1.54G/4.48G [00:14<00:26, 111MB/s]
Downloading (…)l-00002-of-00002.bin: 35%|▋ | 1.56G/4.48G [00:14<00:26, 111MB/s]
Downloading (…)l-00002-of-00002.bin: 35%|▋ | 1.58G/4.48G [00:14<00:26, 111MB/s]
Downloading (…)l-00002-of-00002.bin: 36%|▋ | 1.60G/4.48G [00:14<00:25, 111MB/s]
Downloading (…)l-00002-of-00002.bin: 36%|▋ | 1.63G/4.48G [00:14<00:25, 112MB/s]
Downloading (…)l-00002-of-00002.bin: 37%|▋ | 1.65G/4.48G [00:15<00:25, 111MB/s]
Downloading (…)l-00002-of-00002.bin: 37%|▋ | 1.67G/4.48G [00:15<00:25, 112MB/s]
Downloading (…)l-00002-of-00002.bin: 38%|▊ | 1.69G/4.48G [00:15<00:25, 111MB/s]
Downloading (…)l-00002-of-00002.bin: 38%|▊ | 1.71G/4.48G [00:15<00:24, 111MB/s]
Downloading (…)l-00002-of-00002.bin: 39%|▊ | 1.73G/4.48G [00:15<00:24, 111MB/s]
Downloading (…)l-00002-of-00002.bin: 39%|▊ | 1.75G/4.48G [00:16<00:24, 111MB/s]
Downloading (…)l-00002-of-00002.bin: 40%|▊ | 1.77G/4.48G [00:16<00:24, 112MB/s]
Downloading (…)l-00002-of-00002.bin: 40%|▊ | 1.79G/4.48G [00:16<00:24, 112MB/s]
Downloading (…)l-00002-of-00002.bin: 40%|▊ | 1.81G/4.48G [00:16<00:23, 111MB/s]
Downloading (…)l-00002-of-00002.bin: 41%|▊ | 1.84G/4.48G [00:16<00:23, 112MB/s]
Downloading (…)l-00002-of-00002.bin: 41%|▊ | 1.86G/4.48G [00:16<00:23, 112MB/s]
Downloading (…)l-00002-of-00002.bin: 42%|▊ | 1.88G/4.48G [00:17<00:23, 112MB/s]
Downloading (…)l-00002-of-00002.bin: 42%|▊ | 1.90G/4.48G [00:17<00:23, 112MB/s]
Downloading (…)l-00002-of-00002.bin: 43%|▊ | 1.92G/4.48G [00:17<00:23, 112MB/s]
Downloading (…)l-00002-of-00002.bin: 43%|▊ | 1.94G/4.48G [00:17<00:22, 111MB/s]
Downloading (…)l-00002-of-00002.bin: 44%|▊ | 1.96G/4.48G [00:17<00:22, 112MB/s]
Downloading (…)l-00002-of-00002.bin: 44%|▉ | 1.98G/4.48G [00:18<00:22, 112MB/s]
Downloading (…)l-00002-of-00002.bin: 45%|▉ | 2.00G/4.48G [00:18<00:22, 112MB/s]
Downloading (…)l-00002-of-00002.bin: 45%|▉ | 2.02G/4.48G [00:18<00:21, 112MB/s]
Downloading (…)l-00002-of-00002.bin: 46%|▉ | 2.04G/4.48G [00:18<00:22, 110MB/s]
Downloading (…)l-00002-of-00002.bin: 46%|▉ | 2.07G/4.48G [00:18<00:21, 110MB/s]
Downloading (…)l-00002-of-00002.bin: 47%|▉ | 2.09G/4.48G [00:19<00:21, 111MB/s]
Downloading (…)l-00002-of-00002.bin: 47%|▉ | 2.11G/4.48G [00:19<00:21, 111MB/s]
Downloading (…)l-00002-of-00002.bin: 47%|▉ | 2.13G/4.48G [00:19<00:21, 112MB/s]
Downloading (…)l-00002-of-00002.bin: 48%|▉ | 2.15G/4.48G [00:19<00:20, 111MB/s]
Downloading (…)l-00002-of-00002.bin: 48%|▉ | 2.17G/4.48G [00:19<00:20, 111MB/s]
Downloading (…)l-00002-of-00002.bin: 49%|▉ | 2.19G/4.48G [00:20<00:20, 112MB/s]
Downloading (…)l-00002-of-00002.bin: 49%|▉ | 2.21G/4.48G [00:20<00:20, 111MB/s]
Downloading (…)l-00002-of-00002.bin: 50%|▉ | 2.23G/4.48G [00:20<00:20, 112MB/s]
Downloading (…)l-00002-of-00002.bin: 50%|█ | 2.25G/4.48G [00:20<00:20, 111MB/s]
Downloading (…)l-00002-of-00002.bin: 51%|█ | 2.28G/4.48G [00:20<00:19, 111MB/s]
Downloading (…)l-00002-of-00002.bin: 51%|█ | 2.30G/4.48G [00:20<00:19, 111MB/s]
Downloading (…)l-00002-of-00002.bin: 52%|█ | 2.32G/4.48G [00:21<00:19, 112MB/s]
Downloading (…)l-00002-of-00002.bin: 52%|█ | 2.34G/4.48G [00:21<00:19, 112MB/s]
Downloading (…)l-00002-of-00002.bin: 53%|█ | 2.36G/4.48G [00:21<00:19, 111MB/s]
Downloading (…)l-00002-of-00002.bin: 53%|█ | 2.38G/4.48G [00:21<00:18, 112MB/s]
Downloading (…)l-00002-of-00002.bin: 54%|█ | 2.40G/4.48G [00:21<00:18, 112MB/s]
Downloading (…)l-00002-of-00002.bin: 54%|█ | 2.42G/4.48G [00:22<00:18, 112MB/s]
Downloading (…)l-00002-of-00002.bin: 54%|█ | 2.44G/4.48G [00:22<00:18, 112MB/s]
Downloading (…)l-00002-of-00002.bin: 55%|█ | 2.46G/4.48G [00:22<00:18, 111MB/s]
Downloading (…)l-00002-of-00002.bin: 55%|█ | 2.49G/4.48G [00:22<00:17, 112MB/s]
Downloading (…)l-00002-of-00002.bin: 56%|█ | 2.51G/4.48G [00:22<00:17, 112MB/s]
Downloading (…)l-00002-of-00002.bin: 56%|█▏| 2.53G/4.48G [00:23<00:17, 112MB/s]
Downloading (…)l-00002-of-00002.bin: 57%|█▏| 2.55G/4.48G [00:23<00:17, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 57%|█▏| 2.57G/4.48G [00:23<00:16, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 58%|█▏| 2.59G/4.48G [00:23<00:16, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 58%|█▏| 2.61G/4.48G [00:23<00:16, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 59%|█▏| 2.63G/4.48G [00:23<00:16, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 59%|█▏| 2.65G/4.48G [00:24<00:16, 112MB/s]
Downloading (…)l-00002-of-00002.bin: 60%|█▏| 2.67G/4.48G [00:24<00:16, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 60%|█▏| 2.69G/4.48G [00:24<00:15, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 61%|█▏| 2.72G/4.48G [00:24<00:15, 112MB/s]
Downloading (…)l-00002-of-00002.bin: 61%|█▏| 2.74G/4.48G [00:24<00:15, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 62%|█▏| 2.76G/4.48G [00:25<00:15, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 62%|█▏| 2.78G/4.48G [00:25<00:15, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 62%|█▏| 2.80G/4.48G [00:25<00:14, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 63%|█▎| 2.82G/4.48G [00:25<00:14, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 63%|█▎| 2.84G/4.48G [00:25<00:14, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 64%|█▎| 2.86G/4.48G [00:25<00:14, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 64%|█▎| 2.88G/4.48G [00:26<00:14, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 65%|█▎| 2.90G/4.48G [00:26<00:13, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 65%|█▎| 2.93G/4.48G [00:26<00:13, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 66%|█▎| 2.95G/4.48G [00:26<00:13, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 66%|█▎| 2.97G/4.48G [00:26<00:13, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 67%|█▎| 2.99G/4.48G [00:27<00:13, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 67%|█▎| 3.01G/4.48G [00:27<00:13, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 68%|█▎| 3.03G/4.48G [00:27<00:12, 112MB/s]
Downloading (…)l-00002-of-00002.bin: 68%|█▎| 3.05G/4.48G [00:27<00:12, 112MB/s]
Downloading (…)l-00002-of-00002.bin: 69%|█▎| 3.07G/4.48G [00:27<00:12, 112MB/s]
Downloading (…)l-00002-of-00002.bin: 69%|█▍| 3.09G/4.48G [00:28<00:12, 112MB/s]
Downloading (…)l-00002-of-00002.bin: 69%|█▍| 3.11G/4.48G [00:28<00:12, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 70%|█▍| 3.14G/4.48G [00:28<00:11, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 70%|█▍| 3.16G/4.48G [00:28<00:11, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 71%|█▍| 3.18G/4.48G [00:28<00:11, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 71%|█▍| 3.20G/4.48G [00:28<00:11, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 72%|█▍| 3.22G/4.48G [00:29<00:11, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 72%|█▍| 3.24G/4.48G [00:29<00:11, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 73%|█▍| 3.26G/4.48G [00:29<00:11, 105MB/s]
Downloading (…)l-00002-of-00002.bin: 73%|█▍| 3.28G/4.48G [00:29<00:11, 104MB/s]
Downloading (…)l-00002-of-00002.bin: 74%|█▍| 3.30G/4.48G [00:29<00:11, 106MB/s]
Downloading (…)l-00002-of-00002.bin: 74%|█▍| 3.32G/4.48G [00:30<00:10, 108MB/s]
Downloading (…)l-00002-of-00002.bin: 75%|█▍| 3.34G/4.48G [00:30<00:10, 109MB/s]
Downloading (…)l-00002-of-00002.bin: 75%|█▌| 3.37G/4.48G [00:30<00:10, 110MB/s]
Downloading (…)l-00002-of-00002.bin: 76%|█▌| 3.39G/4.48G [00:30<00:09, 111MB/s]
Downloading (…)l-00002-of-00002.bin: 76%|█▌| 3.41G/4.48G [00:30<00:09, 112MB/s]
Downloading (…)l-00002-of-00002.bin: 76%|█▌| 3.43G/4.48G [00:31<00:09, 112MB/s]
Downloading (…)l-00002-of-00002.bin: 77%|█▌| 3.45G/4.48G [00:31<00:09, 112MB/s]
Downloading (…)l-00002-of-00002.bin: 77%|█▌| 3.47G/4.48G [00:31<00:09, 112MB/s]
Downloading (…)l-00002-of-00002.bin: 78%|█▌| 3.49G/4.48G [00:31<00:08, 112MB/s]
Downloading (…)l-00002-of-00002.bin: 78%|█▌| 3.51G/4.48G [00:31<00:08, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 79%|█▌| 3.53G/4.48G [00:32<00:08, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 79%|█▌| 3.55G/4.48G [00:32<00:08, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 80%|█▌| 3.58G/4.48G [00:32<00:08, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 80%|█▌| 3.60G/4.48G [00:32<00:07, 112MB/s]
Downloading (…)l-00002-of-00002.bin: 81%|█▌| 3.62G/4.48G [00:32<00:07, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 81%|█▌| 3.64G/4.48G [00:32<00:07, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 82%|█▋| 3.66G/4.48G [00:33<00:07, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 82%|█▋| 3.68G/4.48G [00:33<00:07, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 83%|█▋| 3.70G/4.48G [00:33<00:06, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 83%|█▋| 3.72G/4.48G [00:33<00:06, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 83%|█▋| 3.74G/4.48G [00:33<00:06, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 84%|█▋| 3.76G/4.48G [00:34<00:06, 114MB/s]
Downloading (…)l-00002-of-00002.bin: 84%|█▋| 3.79G/4.48G [00:34<00:06, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 85%|█▋| 3.81G/4.48G [00:34<00:05, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 85%|█▋| 3.83G/4.48G [00:34<00:05, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 86%|█▋| 3.85G/4.48G [00:34<00:05, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 86%|█▋| 3.87G/4.48G [00:34<00:05, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 87%|█▋| 3.89G/4.48G [00:35<00:05, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 87%|█▋| 3.91G/4.48G [00:35<00:05, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 88%|█▊| 3.93G/4.48G [00:35<00:04, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 88%|█▊| 3.95G/4.48G [00:35<00:04, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 89%|█▊| 3.97G/4.48G [00:35<00:04, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 89%|█▊| 4.00G/4.48G [00:36<00:04, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 90%|█▊| 4.02G/4.48G [00:36<00:04, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 90%|█▊| 4.04G/4.48G [00:36<00:03, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 91%|█▊| 4.06G/4.48G [00:36<00:03, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 91%|█▊| 4.08G/4.48G [00:36<00:03, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 91%|█▊| 4.10G/4.48G [00:37<00:03, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 92%|█▊| 4.12G/4.48G [00:37<00:03, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 92%|█▊| 4.14G/4.48G [00:37<00:03, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 93%|█▊| 4.16G/4.48G [00:37<00:02, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 93%|█▊| 4.18G/4.48G [00:37<00:02, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 94%|█▉| 4.20G/4.48G [00:37<00:02, 111MB/s]
Downloading (…)l-00002-of-00002.bin: 94%|█▉| 4.23G/4.48G [00:38<00:02, 111MB/s]
Downloading (…)l-00002-of-00002.bin: 95%|█▉| 4.25G/4.48G [00:38<00:02, 112MB/s]
Downloading (…)l-00002-of-00002.bin: 95%|█▉| 4.27G/4.48G [00:38<00:01, 112MB/s]
Downloading (…)l-00002-of-00002.bin: 96%|█▉| 4.29G/4.48G [00:38<00:01, 113MB/s]
Downloading (…)l-00002-of-00002.bin: 96%|█▉| 4.31G/4.48G [00:38<00:01, 112MB/s]
Downloading (…)l-00002-of-00002.bin: 97%|█▉| 4.33G/4.48G [00:39<00:01, 111MB/s]
Downloading (…)l-00002-of-00002.bin: 97%|█▉| 4.35G/4.48G [00:39<00:01, 110MB/s]
Downloading (…)l-00002-of-00002.bin: 98%|█▉| 4.37G/4.48G [00:39<00:01, 109MB/s]
Downloading (…)l-00002-of-00002.bin: 98%|█▉| 4.39G/4.48G [00:39<00:00, 109MB/s]
Downloading (…)l-00002-of-00002.bin: 98%|█▉| 4.41G/4.48G [00:39<00:00, 109MB/s]
Downloading (…)l-00002-of-00002.bin: 99%|█▉| 4.44G/4.48G [00:40<00:00, 109MB/s]
Downloading (…)l-00002-of-00002.bin: 99%|█▉| 4.46G/4.48G [00:40<00:00, 109MB/s]
Downloading (…)l-00002-of-00002.bin: 100%|██| 4.48G/4.48G [00:40<00:00, 111MB/s]
Downloading shards: 100%|█████████████████████████| 2/2 [02:10<00:00, 65.07s/it]
Loading checkpoint shards: 100%|██████████████████| 2/2 [00:18<00:00, 9.31s/it]
Downloading (…)neration_config.json: 100%|█████| 111/111 [00:00<00:00, 22.2kB/s]
INFO:root:converting PEFT model w/ prepare_model_for_int8_training
/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/peft/utils/other.py:76: FutureWarning: prepare_model_for_int8_training is deprecated and will be removed in a future version. Use prepare_model_for_kbit_training instead.
warnings.warn(
INFO:root:found linear modules: ['dense', 'dense_h_to_4h', 'dense_4h_to_h', 'query_key_value']
trainable params: 130547712 || all params: 3739292544 || trainable%: 3.4912409356543783
INFO:root:Compiling torch model
INFO:root:Pre-saving adapter config to ./qlora-out
INFO:root:Starting trainer...
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
- Avoid using `tokenizers` before the fork if possible
- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
- Avoid using `tokenizers` before the fork if possible
- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
- Avoid using `tokenizers` before the fork if possible
- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
wandb: Currently logged in as: utensil. Use `wandb login --relogin` to force relogin
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
- Avoid using `tokenizers` before the fork if possible
- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
wandb: Tracking run with wandb version 0.15.3
wandb: Run data is saved locally in /workspace/axolotl/wandb/run-20230601_090754-py4qg3pj
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run still-monkey-12
wandb: ⭐️ View project at https://wandb.ai/utensil/falcon-qlora
wandb: 🚀 View run at https://wandb.ai/utensil/falcon-qlora/runs/py4qg3pj
0%| | 0/2775 [00:00<?, ?it/s]You're using a PreTrainedTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
{'loss': 1.03, 'learning_rate': 2e-05, 'epoch': 0.0}
{'loss': 1.1617, 'learning_rate': 4e-05, 'epoch': 0.0}
{'loss': 0.972, 'learning_rate': 6e-05, 'epoch': 0.0}
{'loss': 1.2016, 'learning_rate': 8e-05, 'epoch': 0.0}
{'loss': 1.2021, 'learning_rate': 0.0001, 'epoch': 0.01}
0%| | 5/2775 [00:12<1:48:22, 2.35s/it]
0%| | 0/19 [00:00<?, ?it/s]
11%|████▋ | 2/19 [00:00<00:03, 5.45it/s]
16%|██████▉ | 3/19 [00:00<00:03, 4.59it/s]
21%|█████████▎ | 4/19 [00:00<00:03, 3.97it/s]
26%|███████████▌ | 5/19 [00:01<00:03, 3.61it/s]
32%|█████████████▉ | 6/19 [00:01<00:03, 3.35it/s]
37%|████████████████▏ | 7/19 [00:01<00:03, 3.32it/s]
42%|██████████████████▌ | 8/19 [00:02<00:03, 3.25it/s]
47%|████████████████████▊ | 9/19 [00:02<00:03, 3.20it/s]
53%|██████████████████████▋ | 10/19 [00:02<00:02, 3.28it/s]
58%|████████████████████████▉ | 11/19 [00:03<00:02, 3.32it/s]
63%|███████████████████████████▏ | 12/19 [00:03<00:02, 3.19it/s]
68%|█████████████████████████████▍ | 13/19 [00:03<00:01, 3.07it/s]
74%|███████████████████████████████▋ | 14/19 [00:04<00:01, 2.97it/s]
79%|█████████████████████████████████▉ | 15/19 [00:04<00:01, 3.02it/s]
84%|████████████████████████████████████▏ | 16/19 [00:04<00:01, 3.00it/s]
89%|██████████████████████████████████████▍ | 17/19 [00:05<00:00, 3.09it/s]
95%|████████████████████████████████████████▋ | 18/19 [00:05<00:00, 3.29it/s]
{'eval_loss': 1.1653213500976562, 'eval_runtime': 5.9718, 'eval_samples_per_second': 12.559, 'eval_steps_per_second': 3.182, 'epoch': 0.01}
0%| | 5/2775 [00:18<1:48:22, 2.35s/it]
100%|███████████████████████████████████████████| 19/19 [00:05<00:00, 3.47it/s]
{'loss': 1.0062, 'learning_rate': 0.00012, 'epoch': 0.01}
{'loss': 1.1272, 'learning_rate': 0.00014, 'epoch': 0.01}
{'loss': 1.0595, 'learning_rate': 0.00016, 'epoch': 0.01}
{'loss': 1.1408, 'learning_rate': 0.00018, 'epoch': 0.01}
{'loss': 1.0711, 'learning_rate': 0.0002, 'epoch': 0.01}
0%|▏ | 10/2775 [00:28<1:58:36, 2.57s/it]
0%| | 0/19 [00:00<?, ?it/s]
11%|████▋ | 2/19 [00:00<00:03, 5.46it/s]
16%|██████▉ | 3/19 [00:00<00:03, 4.60it/s]
21%|█████████▎ | 4/19 [00:00<00:03, 3.97it/s]
26%|███████████▌ | 5/19 [00:01<00:03, 3.62it/s]
32%|█████████████▉ | 6/19 [00:01<00:03, 3.35it/s]
37%|████████████████▏ | 7/19 [00:01<00:03, 3.33it/s]
42%|██████████████████▌ | 8/19 [00:02<00:03, 3.26it/s]
47%|████████████████████▊ | 9/19 [00:02<00:03, 3.21it/s]
53%|██████████████████████▋ | 10/19 [00:02<00:02, 3.28it/s]
58%|████████████████████████▉ | 11/19 [00:03<00:02, 3.34it/s]
63%|███████████████████████████▏ | 12/19 [00:03<00:02, 3.19it/s]
68%|█████████████████████████████▍ | 13/19 [00:03<00:01, 3.07it/s]
74%|███████████████████████████████▋ | 14/19 [00:04<00:01, 2.96it/s]
79%|█████████████████████████████████▉ | 15/19 [00:04<00:01, 3.02it/s]
84%|████████████████████████████████████▏ | 16/19 [00:04<00:01, 2.99it/s]
89%|██████████████████████████████████████▍ | 17/19 [00:05<00:00, 3.07it/s]
95%|████████████████████████████████████████▋ | 18/19 [00:05<00:00, 3.27it/s]
{'eval_loss': 0.9998868703842163, 'eval_runtime': 5.9727, 'eval_samples_per_second': 12.557, 'eval_steps_per_second': 3.181, 'epoch': 0.01}
0%|▏ | 10/2775 [00:34<1:58:36, 2.57s/it]
100%|███████████████████████████████████████████| 19/19 [00:05<00:00, 3.45it/s]
{'loss': 0.8395, 'learning_rate': 0.0001999999354524992, 'epoch': 0.01}
{'loss': 0.8302, 'learning_rate': 0.00019999974181008004, 'epoch': 0.01}
{'loss': 0.8494, 'learning_rate': 0.00019999941907299258, 'epoch': 0.01}
{'loss': 0.7828, 'learning_rate': 0.00019999896724165342, 'epoch': 0.02}
{'loss': 0.8007, 'learning_rate': 0.00019999838631664586, 'epoch': 0.02}
1%|▏ | 15/2775 [00:50<2:20:32, 3.06s/it]
0%| | 0/19 [00:00<?, ?it/s]
11%|████▋ | 2/19 [00:00<00:03, 5.46it/s]
16%|██████▉ | 3/19 [00:00<00:03, 4.59it/s]
21%|█████████▎ | 4/19 [00:00<00:03, 3.96it/s]
26%|███████████▌ | 5/19 [00:01<00:03, 3.60it/s]
32%|█████████████▉ | 6/19 [00:01<00:03, 3.34it/s]
37%|████████████████▏ | 7/19 [00:01<00:03, 3.32it/s]
42%|██████████████████▌ | 8/19 [00:02<00:03, 3.25it/s]
47%|████████████████████▊ | 9/19 [00:02<00:03, 3.20it/s]
53%|██████████████████████▋ | 10/19 [00:02<00:02, 3.27it/s]
58%|████████████████████████▉ | 11/19 [00:03<00:02, 3.34it/s]
63%|███████████████████████████▏ | 12/19 [00:03<00:02, 3.19it/s]
68%|█████████████████████████████▍ | 13/19 [00:03<00:01, 3.07it/s]
74%|███████████████████████████████▋ | 14/19 [00:04<00:01, 2.95it/s]
79%|█████████████████████████████████▉ | 15/19 [00:04<00:01, 3.02it/s]
84%|████████████████████████████████████▏ | 16/19 [00:04<00:01, 2.99it/s]
89%|██████████████████████████████████████▍ | 17/19 [00:05<00:00, 3.07it/s]
95%|████████████████████████████████████████▋ | 18/19 [00:05<00:00, 3.27it/s]
{'eval_loss': 0.8723135590553284, 'eval_runtime': 5.9832, 'eval_samples_per_second': 12.535, 'eval_steps_per_second': 3.176, 'epoch': 0.02}
1%|▏ | 15/2775 [00:56<2:20:32, 3.06s/it]
100%|███████████████████████████████████████████| 19/19 [00:05<00:00, 3.45it/s]
{'loss': 0.8119, 'learning_rate': 0.00019999767629871982, 'epoch': 0.02}
{'loss': 0.883, 'learning_rate': 0.00019999683718879195, 'epoch': 0.02}
{'loss': 0.8471, 'learning_rate': 0.00019999586898794544, 'epoch': 0.02}
{'loss': 0.7596, 'learning_rate': 0.00019999477169743024, 'epoch': 0.02}
{'loss': 0.6701, 'learning_rate': 0.00019999354531866285, 'epoch': 0.02}
1%|▎ | 20/2775 [01:07<2:07:18, 2.77s/it]
0%| | 0/19 [00:00<?, ?it/s]
11%|████▋ | 2/19 [00:00<00:03, 5.45it/s]
16%|██████▉ | 3/19 [00:00<00:03, 4.58it/s]
21%|█████████▎ | 4/19 [00:00<00:03, 3.95it/s]
26%|███████████▌ | 5/19 [00:01<00:03, 3.59it/s]
32%|█████████████▉ | 6/19 [00:01<00:03, 3.33it/s]
37%|████████████████▏ | 7/19 [00:01<00:03, 3.30it/s]
42%|██████████████████▌ | 8/19 [00:02<00:03, 3.24it/s]
47%|████████████████████▊ | 9/19 [00:02<00:03, 3.19it/s]
53%|██████████████████████▋ | 10/19 [00:02<00:02, 3.26it/s]
58%|████████████████████████▉ | 11/19 [00:03<00:02, 3.33it/s]
63%|███████████████████████████▏ | 12/19 [00:03<00:02, 3.18it/s]
68%|█████████████████████████████▍ | 13/19 [00:03<00:01, 3.06it/s]
74%|███████████████████████████████▋ | 14/19 [00:04<00:01, 2.95it/s]
79%|█████████████████████████████████▉ | 15/19 [00:04<00:01, 3.01it/s]
84%|████████████████████████████████████▏ | 16/19 [00:04<00:01, 2.98it/s]
89%|██████████████████████████████████████▍ | 17/19 [00:05<00:00, 3.07it/s]
95%|████████████████████████████████████████▋ | 18/19 [00:05<00:00, 3.27it/s]
{'eval_loss': 0.821587860584259, 'eval_runtime': 5.998, 'eval_samples_per_second': 12.504, 'eval_steps_per_second': 3.168, 'epoch': 0.02}
1%|▎ | 20/2775 [01:13<2:07:18, 2.77s/it]
100%|███████████████████████████████████████████| 19/19 [00:05<00:00, 3.45it/s]
{'loss': 0.7863, 'learning_rate': 0.00019999218985322653, 'epoch': 0.02}
{'loss': 0.8566, 'learning_rate': 0.00019999070530287103, 'epoch': 0.02}
{'loss': 0.78, 'learning_rate': 0.0001999890916695129, 'epoch': 0.02}
{'loss': 0.7803, 'learning_rate': 0.00019998734895523525, 'epoch': 0.03}
{'loss': 0.9027, 'learning_rate': 0.0001999854771622878, 'epoch': 0.03}
1%|▎ | 25/2775 [01:29<2:18:32, 3.02s/it]
0%| | 0/19 [00:00<?, ?it/s]
11%|████▋ | 2/19 [00:00<00:03, 5.43it/s]
16%|██████▉ | 3/19 [00:00<00:03, 4.57it/s]
21%|█████████▎ | 4/19 [00:00<00:03, 3.95it/s]
26%|███████████▌ | 5/19 [00:01<00:03, 3.59it/s]
32%|█████████████▉ | 6/19 [00:01<00:03, 3.34it/s]
37%|████████████████▏ | 7/19 [00:01<00:03, 3.31it/s]
42%|██████████████████▌ | 8/19 [00:02<00:03, 3.24it/s]
47%|████████████████████▊ | 9/19 [00:02<00:03, 3.19it/s]
53%|██████████████████████▋ | 10/19 [00:02<00:02, 3.27it/s]
58%|████████████████████████▉ | 11/19 [00:03<00:02, 3.33it/s]
63%|███████████████████████████▏ | 12/19 [00:03<00:02, 3.18it/s]
68%|█████████████████████████████▍ | 13/19 [00:03<00:01, 3.06it/s]
74%|███████████████████████████████▋ | 14/19 [00:04<00:01, 2.95it/s]
79%|█████████████████████████████████▉ | 15/19 [00:04<00:01, 3.01it/s]
84%|████████████████████████████████████▏ | 16/19 [00:04<00:01, 2.98it/s]
89%|██████████████████████████████████████▍ | 17/19 [00:05<00:00, 3.07it/s]
95%|████████████████████████████████████████▋ | 18/19 [00:05<00:00, 3.26it/s]
{'eval_loss': 0.7868093848228455, 'eval_runtime': 5.9997, 'eval_samples_per_second': 12.501, 'eval_steps_per_second': 3.167, 'epoch': 0.03}
1%|▎ | 25/2775 [01:35<2:18:32, 3.02s/it]
100%|███████████████████████████████████████████| 19/19 [00:05<00:00, 3.44it/s]
{'loss': 0.7481, 'learning_rate': 0.000199983476293087, 'epoch': 0.03}
{'loss': 0.7742, 'learning_rate': 0.0001999813463502158, 'epoch': 0.03}
{'loss': 0.6962, 'learning_rate': 0.00019997908733642388, 'epoch': 0.03}
{'loss': 0.7422, 'learning_rate': 0.00019997669925462755, 'epoch': 0.03}
{'loss': 0.6607, 'learning_rate': 0.00019997418210790965, 'epoch': 0.03}
1%|▍ | 30/2775 [01:45<2:01:49, 2.66s/it]
0%| | 0/19 [00:00<?, ?it/s]
11%|████▋ | 2/19 [00:00<00:03, 5.43it/s]
16%|██████▉ | 3/19 [00:00<00:03, 4.56it/s]
21%|█████████▎ | 4/19 [00:00<00:03, 3.94it/s]
26%|███████████▌ | 5/19 [00:01<00:03, 3.57it/s]
32%|█████████████▉ | 6/19 [00:01<00:03, 3.31it/s]
37%|████████████████▏ | 7/19 [00:01<00:03, 3.29it/s]
42%|██████████████████▌ | 8/19 [00:02<00:03, 3.22it/s]
47%|████████████████████▊ | 9/19 [00:02<00:03, 3.18it/s]
53%|██████████████████████▋ | 10/19 [00:02<00:02, 3.25it/s]
58%|████████████████████████▉ | 11/19 [00:03<00:02, 3.32it/s]
63%|███████████████████████████▏ | 12/19 [00:03<00:02, 3.17it/s]
68%|█████████████████████████████▍ | 13/19 [00:03<00:01, 3.05it/s]
74%|███████████████████████████████▋ | 14/19 [00:04<00:01, 2.93it/s]
79%|█████████████████████████████████▉ | 15/19 [00:04<00:01, 3.00it/s]
84%|████████████████████████████████████▏ | 16/19 [00:04<00:01, 2.97it/s]
89%|██████████████████████████████████████▍ | 17/19 [00:05<00:00, 3.05it/s]
95%|████████████████████████████████████████▋ | 18/19 [00:05<00:00, 3.26it/s]
{'eval_loss': 0.7816319465637207, 'eval_runtime': 6.0219, 'eval_samples_per_second': 12.454, 'eval_steps_per_second': 3.155, 'epoch': 0.03}
1%|▍ | 30/2775 [01:51<2:01:49, 2.66s/it]
100%|███████████████████████████████████████████| 19/19 [00:05<00:00, 3.43it/s]
{'loss': 0.8978, 'learning_rate': 0.00019997153589951973, 'epoch': 0.03}
{'loss': 0.8512, 'learning_rate': 0.0001999687606328739, 'epoch': 0.03}
{'loss': 0.7373, 'learning_rate': 0.00019996585631155492, 'epoch': 0.04}
{'loss': 0.7072, 'learning_rate': 0.00019996282293931203, 'epoch': 0.04}
{'loss': 0.7172, 'learning_rate': 0.00019995966052006127, 'epoch': 0.04}
1%|▍ | 35/2775 [02:09<2:24:10, 3.16s/it]
0%| | 0/19 [00:00<?, ?it/s]
11%|████▋ | 2/19 [00:00<00:03, 5.41it/s]
16%|██████▉ | 3/19 [00:00<00:03, 4.54it/s]
21%|█████████▎ | 4/19 [00:00<00:03, 3.93it/s]
26%|███████████▌ | 5/19 [00:01<00:03, 3.57it/s]
32%|█████████████▉ | 6/19 [00:01<00:03, 3.31it/s]
37%|████████████████▏ | 7/19 [00:01<00:03, 3.29it/s]
42%|██████████████████▌ | 8/19 [00:02<00:03, 3.22it/s]
47%|████████████████████▊ | 9/19 [00:02<00:03, 3.18it/s]
53%|██████████████████████▋ | 10/19 [00:02<00:02, 3.25it/s]
58%|████████████████████████▉ | 11/19 [00:03<00:02, 3.31it/s]
63%|███████████████████████████▏ | 12/19 [00:03<00:02, 3.16it/s]
68%|█████████████████████████████▍ | 13/19 [00:03<00:01, 3.05it/s]
74%|███████████████████████████████▋ | 14/19 [00:04<00:01, 2.94it/s]
79%|█████████████████████████████████▉ | 15/19 [00:04<00:01, 3.00it/s]
84%|████████████████████████████████████▏ | 16/19 [00:04<00:01, 2.97it/s]
89%|██████████████████████████████████████▍ | 17/19 [00:05<00:00, 3.05it/s]
95%|████████████████████████████████████████▋ | 18/19 [00:05<00:00, 3.25it/s]
{'eval_loss': 0.7573897838592529, 'eval_runtime': 6.0345, 'eval_samples_per_second': 12.428, 'eval_steps_per_second': 3.149, 'epoch': 0.04}
1%|▍ | 35/2775 [02:15<2:24:10, 3.16s/it]
100%|███████████████████████████████████████████| 19/19 [00:05<00:00, 3.42it/s]
{'loss': 0.7172, 'learning_rate': 0.00019995636905788508, 'epoch': 0.04}
{'loss': 0.7797, 'learning_rate': 0.00019995294855703266, 'epoch': 0.04}
{'loss': 0.5514, 'learning_rate': 0.00019994939902191964, 'epoch': 0.04}
{'loss': 0.7519, 'learning_rate': 0.0001999457204571283, 'epoch': 0.04}
{'loss': 0.871, 'learning_rate': 0.0001999419128674075, 'epoch': 0.04}
1%|▌ | 40/2775 [02:25<2:03:24, 2.71s/it]
0%| | 0/19 [00:00<?, ?it/s]
11%|████▋ | 2/19 [00:00<00:03, 5.42it/s]
16%|██████▉ | 3/19 [00:00<00:03, 4.56it/s]
21%|█████████▎ | 4/19 [00:00<00:03, 3.95it/s]
26%|███████████▌ | 5/19 [00:01<00:03, 3.58it/s]
32%|█████████████▉ | 6/19 [00:01<00:03, 3.32it/s]
37%|████████████████▏ | 7/19 [00:01<00:03, 3.30it/s]
42%|██████████████████▌ | 8/19 [00:02<00:03, 3.23it/s]
47%|████████████████████▊ | 9/19 [00:02<00:03, 3.18it/s]
53%|██████████████████████▋ | 10/19 [00:02<00:02, 3.25it/s]
58%|████████████████████████▉ | 11/19 [00:03<00:02, 3.30it/s]
63%|███████████████████████████▏ | 12/19 [00:03<00:02, 3.17it/s]
68%|█████████████████████████████▍ | 13/19 [00:03<00:01, 3.05it/s]
74%|███████████████████████████████▋ | 14/19 [00:04<00:01, 2.94it/s]
79%|█████████████████████████████████▉ | 15/19 [00:04<00:01, 3.01it/s]
84%|████████████████████████████████████▏ | 16/19 [00:04<00:01, 2.98it/s]
89%|██████████████████████████████████████▍ | 17/19 [00:05<00:00, 3.06it/s]
95%|████████████████████████████████████████▋ | 18/19 [00:05<00:00, 3.26it/s]
{'eval_loss': 0.7504977583885193, 'eval_runtime': 6.0154, 'eval_samples_per_second': 12.468, 'eval_steps_per_second': 3.159, 'epoch': 0.04}
1%|▌ | 40/2775 [02:31<2:03:24, 2.71s/it]
100%|███████████████████████████████████████████| 19/19 [00:05<00:00, 3.43it/s]
{'loss': 0.6751, 'learning_rate': 0.00019993797625767262, 'epoch': 0.04}
{'loss': 0.7127, 'learning_rate': 0.00019993391063300567, 'epoch': 0.05}
{'loss': 0.7182, 'learning_rate': 0.00019992971599865514, 'epoch': 0.05}
{'loss': 0.6566, 'learning_rate': 0.00019992539236003614, 'epoch': 0.05}
{'loss': 0.7533, 'learning_rate': 0.00019992093972273018, 'epoch': 0.05}
2%|▋ | 45/2775 [02:48<2:20:39, 3.09s/it]
0%| | 0/19 [00:00<?, ?it/s]
11%|████▋ | 2/19 [00:00<00:03, 5.43it/s]
16%|██████▉ | 3/19 [00:00<00:03, 4.57it/s]
21%|█████████▎ | 4/19 [00:00<00:03, 3.95it/s]
26%|███████████▌ | 5/19 [00:01<00:03, 3.59it/s]
32%|█████████████▉ | 6/19 [00:01<00:03, 3.32it/s]
37%|████████████████▏ | 7/19 [00:01<00:03, 3.30it/s]
42%|██████████████████▌ | 8/19 [00:02<00:03, 3.23it/s]
47%|████████████████████▊ | 9/19 [00:02<00:03, 3.19it/s]
53%|██████████████████████▋ | 10/19 [00:02<00:02, 3.26it/s]
58%|████████████████████████▉ | 11/19 [00:03<00:02, 3.33it/s]
63%|███████████████████████████▏ | 12/19 [00:03<00:02, 3.17it/s]
68%|█████████████████████████████▍ | 13/19 [00:03<00:01, 3.05it/s]
74%|███████████████████████████████▋ | 14/19 [00:04<00:01, 2.94it/s]
79%|█████████████████████████████████▉ | 15/19 [00:04<00:01, 3.01it/s]
84%|████████████████████████████████████▏ | 16/19 [00:04<00:01, 2.98it/s]
89%|██████████████████████████████████████▍ | 17/19 [00:05<00:00, 3.07it/s]
95%|████████████████████████████████████████▋ | 18/19 [00:05<00:00, 3.26it/s]
{'eval_loss': 0.7449343204498291, 'eval_runtime': 6.0069, 'eval_samples_per_second': 12.486, 'eval_steps_per_second': 3.163, 'epoch': 0.05}
2%|▋ | 45/2775 [02:54<2:20:39, 3.09s/it]
100%|███████████████████████████████████████████| 19/19 [00:05<00:00, 3.44it/s]
{'loss': 0.6953, 'learning_rate': 0.00019991635809248548, 'epoch': 0.05}
{'loss': 0.5515, 'learning_rate': 0.00019991164747521663, 'epoch': 0.05}
{'loss': 0.6033, 'learning_rate': 0.00019990680787700487, 'epoch': 0.05}
{'loss': 0.7761, 'learning_rate': 0.00019990183930409782, 'epoch': 0.05}
{'loss': 0.7589, 'learning_rate': 0.00019989674176290967, 'epoch': 0.05}
2%|▋ | 50/2775 [03:05<2:03:51, 2.73s/it]
0%| | 0/19 [00:00<?, ?it/s]
11%|████▋ | 2/19 [00:00<00:03, 5.43it/s]
16%|██████▉ | 3/19 [00:00<00:03, 4.56it/s]
21%|█████████▎ | 4/19 [00:00<00:03, 3.94it/s]
26%|███████████▌ | 5/19 [00:01<00:03, 3.59it/s]
32%|█████████████▉ | 6/19 [00:01<00:03, 3.32it/s]
37%|████████████████▏ | 7/19 [00:01<00:03, 3.30it/s]
42%|██████████████████▌ | 8/19 [00:02<00:03, 3.23it/s]
47%|████████████████████▊ | 9/19 [00:02<00:03, 3.18it/s]
53%|██████████████████████▋ | 10/19 [00:02<00:02, 3.25it/s]
58%|████████████████████████▉ | 11/19 [00:03<00:02, 3.32it/s]
63%|███████████████████████████▏ | 12/19 [00:03<00:02, 3.17it/s]
68%|█████████████████████████████▍ | 13/19 [00:03<00:01, 3.05it/s]
74%|███████████████████████████████▋ | 14/19 [00:04<00:01, 2.94it/s]
79%|█████████████████████████████████▉ | 15/19 [00:04<00:01, 3.00it/s]
84%|████████████████████████████████████▏ | 16/19 [00:04<00:01, 2.97it/s]
89%|██████████████████████████████████████▍ | 17/19 [00:05<00:00, 3.06it/s]
95%|████████████████████████████████████████▋ | 18/19 [00:05<00:00, 3.26it/s]
{'eval_loss': 0.740250825881958, 'eval_runtime': 6.0148, 'eval_samples_per_second': 12.469, 'eval_steps_per_second': 3.159, 'epoch': 0.05}
2%|▋ | 50/2775 [03:11<2:03:51, 2.73s/it]
100%|███████████████████████████████████████████| 19/19 [00:05<00:00, 3.44it/s]
{'loss': 0.8527, 'learning_rate': 0.0001998915152600211, 'epoch': 0.06}
{'loss': 0.636, 'learning_rate': 0.00019988615980217925, 'epoch': 0.06}
{'loss': 0.6423, 'learning_rate': 0.0001998806753962978, 'epoch': 0.06}
{'loss': 0.6367, 'learning_rate': 0.00019987506204945677, 'epoch': 0.06}
{'loss': 0.6775, 'learning_rate': 0.00019986931976890277, 'epoch': 0.06}
2%|▊ | 55/2775 [03:28<2:19:42, 3.08s/it]
0%| | 0/19 [00:00<?, ?it/s]
11%|████▋ | 2/19 [00:00<00:03, 5.43it/s]
16%|██████▉ | 3/19 [00:00<00:03, 4.57it/s]
21%|█████████▎ | 4/19 [00:00<00:03, 3.95it/s]
26%|███████████▌ | 5/19 [00:01<00:03, 3.59it/s]
32%|█████████████▉ | 6/19 [00:01<00:03, 3.33it/s]
37%|████████████████▏ | 7/19 [00:01<00:03, 3.30it/s]
42%|██████████████████▌ | 8/19 [00:02<00:03, 3.24it/s]
47%|████████████████████▊ | 9/19 [00:02<00:03, 3.19it/s]
53%|██████████████████████▋ | 10/19 [00:02<00:02, 3.26it/s]
58%|████████████████████████▉ | 11/19 [00:03<00:02, 3.32it/s]
63%|███████████████████████████▏ | 12/19 [00:03<00:02, 3.17it/s]
68%|█████████████████████████████▍ | 13/19 [00:03<00:01, 3.05it/s]
74%|███████████████████████████████▋ | 14/19 [00:04<00:01, 2.94it/s]
79%|█████████████████████████████████▉ | 15/19 [00:04<00:01, 3.00it/s]
84%|████████████████████████████████████▏ | 16/19 [00:04<00:01, 2.97it/s]
89%|██████████████████████████████████████▍ | 17/19 [00:05<00:00, 3.06it/s]
95%|████████████████████████████████████████▋ | 18/19 [00:05<00:00, 3.26it/s]
{'eval_loss': 0.7213495373725891, 'eval_runtime': 6.0107, 'eval_samples_per_second': 12.478, 'eval_steps_per_second': 3.161, 'epoch': 0.06}
2%|▊ | 55/2775 [03:34<2:19:42, 3.08s/it]
100%|███████████████████████████████████████████| 19/19 [00:05<00:00, 3.44it/s]
{'loss': 0.6645, 'learning_rate': 0.00019986344856204877, 'epoch': 0.06}
{'loss': 0.7062, 'learning_rate': 0.00019985744843647421, 'epoch': 0.06}
{'loss': 0.6675, 'learning_rate': 0.00019985131939992493, 'epoch': 0.06}
{'loss': 0.7105, 'learning_rate': 0.00019984506146031325, 'epoch': 0.06}
{'loss': 0.5274, 'learning_rate': 0.00019983867462571784, 'epoch': 0.06}
2%|▊ | 60/2775 [03:44<1:58:49, 2.63s/it]
0%| | 0/19 [00:00<?, ?it/s]
11%|████▋ | 2/19 [00:00<00:03, 5.42it/s]
16%|██████▉ | 3/19 [00:00<00:03, 4.56it/s]
21%|█████████▎ | 4/19 [00:00<00:03, 3.94it/s]
26%|███████████▌ | 5/19 [00:01<00:03, 3.59it/s]
32%|█████████████▉ | 6/19 [00:01<00:03, 3.33it/s]
37%|████████████████▏ | 7/19 [00:01<00:03, 3.30it/s]
42%|██████████████████▌ | 8/19 [00:02<00:03, 3.23it/s]
47%|████████████████████▊ | 9/19 [00:02<00:03, 3.18it/s]
53%|██████████████████████▋ | 10/19 [00:02<00:02, 3.25it/s]
58%|████████████████████████▉ | 11/19 [00:03<00:02, 3.32it/s]
63%|███████████████████████████▏ | 12/19 [00:03<00:02, 3.17it/s]
68%|█████████████████████████████▍ | 13/19 [00:03<00:01, 3.05it/s]
74%|███████████████████████████████▋ | 14/19 [00:04<00:01, 2.94it/s]
79%|█████████████████████████████████▉ | 15/19 [00:04<00:01, 3.00it/s]
84%|████████████████████████████████████▏ | 16/19 [00:04<00:01, 2.97it/s]
89%|██████████████████████████████████████▍ | 17/19 [00:05<00:00, 3.06it/s]
95%|████████████████████████████████████████▋ | 18/19 [00:05<00:00, 3.25it/s]
{'eval_loss': 0.715459942817688, 'eval_runtime': 6.0162, 'eval_samples_per_second': 12.466, 'eval_steps_per_second': 3.158, 'epoch': 0.06}
2%|▊ | 60/2775 [03:50<1:58:49, 2.63s/it]
100%|███████████████████████████████████████████| 19/19 [00:05<00:00, 3.43it/s]
{'loss': 0.6289, 'learning_rate': 0.00019983215890438379, 'epoch': 0.07}
{'loss': 0.6202, 'learning_rate': 0.00019982551430472255, 'epoch': 0.07}
{'loss': 0.6305, 'learning_rate': 0.00019981874083531198, 'epoch': 0.07}
{'loss': 0.7176, 'learning_rate': 0.00019981183850489628, 'epoch': 0.07}
{'loss': 0.5461, 'learning_rate': 0.000199804807322386, 'epoch': 0.07}
2%|▉ | 65/2775 [04:06<2:13:47, 2.96s/it]
0%| | 0/19 [00:00<?, ?it/s]
11%|████▋ | 2/19 [00:00<00:03, 5.42it/s]
16%|██████▉ | 3/19 [00:00<00:03, 4.56it/s]
21%|█████████▎ | 4/19 [00:00<00:03, 3.95it/s]
26%|███████████▌ | 5/19 [00:01<00:03, 3.59it/s]
32%|█████████████▉ | 6/19 [00:01<00:03, 3.33it/s]
37%|████████████████▏ | 7/19 [00:01<00:03, 3.30it/s]
42%|██████████████████▌ | 8/19 [00:02<00:03, 3.23it/s]
47%|████████████████████▊ | 9/19 [00:02<00:03, 3.19it/s]
53%|██████████████████████▋ | 10/19 [00:02<00:02, 3.26it/s]
58%|████████████████████████▉ | 11/19 [00:03<00:02, 3.32it/s]
63%|███████████████████████████▏ | 12/19 [00:03<00:02, 3.17it/s]
68%|█████████████████████████████▍ | 13/19 [00:03<00:01, 3.06it/s]
74%|███████████████████████████████▋ | 14/19 [00:04<00:01, 2.94it/s]
79%|█████████████████████████████████▉ | 15/19 [00:04<00:01, 3.01it/s]
84%|████████████████████████████████████▏ | 16/19 [00:04<00:01, 2.98it/s]
89%|██████████████████████████████████████▍ | 17/19 [00:05<00:00, 3.06it/s]
95%|████████████████████████████████████████▋ | 18/19 [00:05<00:00, 3.26it/s]
{'eval_loss': 0.7042242884635925, 'eval_runtime': 6.0106, 'eval_samples_per_second': 12.478, 'eval_steps_per_second': 3.161, 'epoch': 0.07}
2%|▉ | 65/2775 [04:12<2:13:47, 2.96s/it]
100%|███████████████████████████████████████████| 19/19 [00:05<00:00, 3.43it/s]
{'loss': 0.7021, 'learning_rate': 0.00019979764729685813, 'epoch': 0.07}
{'loss': 0.6482, 'learning_rate': 0.00019979035843755582, 'epoch': 0.07}
{'loss': 0.7068, 'learning_rate': 0.00019978294075388863, 'epoch': 0.07}
{'loss': 0.6977, 'learning_rate': 0.00019977539425543244, 'epoch': 0.07}
{'loss': 0.6442, 'learning_rate': 0.0001997677189519294, 'epoch': 0.08}
3%|▉ | 70/2775 [04:23<2:00:36, 2.68s/it]
0%| | 0/19 [00:00<?, ?it/s]
11%|████▋ | 2/19 [00:00<00:03, 5.43it/s]
16%|██████▉ | 3/19 [00:00<00:03, 4.56it/s]
21%|█████████▎ | 4/19 [00:00<00:03, 3.95it/s]
26%|███████████▌ | 5/19 [00:01<00:03, 3.59it/s]
32%|█████████████▉ | 6/19 [00:01<00:03, 3.32it/s]
37%|████████████████▏ | 7/19 [00:01<00:03, 3.30it/s]
42%|██████████████████▌ | 8/19 [00:02<00:03, 3.23it/s]
47%|████████████████████▊ | 9/19 [00:02<00:03, 3.18it/s]
53%|██████████████████████▋ | 10/19 [00:02<00:02, 3.25it/s]
58%|████████████████████████▉ | 11/19 [00:03<00:02, 3.32it/s]
63%|███████████████████████████▏ | 12/19 [00:03<00:02, 3.17it/s]
68%|█████████████████████████████▍ | 13/19 [00:03<00:01, 3.05it/s]
74%|███████████████████████████████▋ | 14/19 [00:04<00:01, 2.94it/s]
79%|█████████████████████████████████▉ | 15/19 [00:04<00:01, 3.00it/s]
84%|████████████████████████████████████▏ | 16/19 [00:04<00:01, 2.97it/s]
89%|██████████████████████████████████████▍ | 17/19 [00:05<00:00, 3.06it/s]
95%|████████████████████████████████████████▋ | 18/19 [00:05<00:00, 3.26it/s]
{'eval_loss': 0.7020356059074402, 'eval_runtime': 6.0154, 'eval_samples_per_second': 12.468, 'eval_steps_per_second': 3.159, 'epoch': 0.08}
3%|▉ | 70/2775 [04:29<2:00:36, 2.68s/it]
100%|███████████████████████████████████████████| 19/19 [00:05<00:00, 3.43it/s]
{'loss': 0.6472, 'learning_rate': 0.00019975991485328794, 'epoch': 0.08}
{'loss': 0.6003, 'learning_rate': 0.0001997519819695828, 'epoch': 0.08}
{'loss': 0.9422, 'learning_rate': 0.00019974392031105482, 'epoch': 0.08}
{'loss': 0.6847, 'learning_rate': 0.00019973572988811133, 'epoch': 0.08}
{'loss': 0.67, 'learning_rate': 0.00019972741071132567, 'epoch': 0.08}
3%|█ | 75/2775 [04:45<2:19:28, 3.10s/it]
0%| | 0/19 [00:00<?, ?it/s]
11%|████▋ | 2/19 [00:00<00:03, 5.43it/s]
16%|██████▉ | 3/19 [00:00<00:03, 4.57it/s]
21%|█████████▎ | 4/19 [00:00<00:03, 3.95it/s]
26%|███████████▌ | 5/19 [00:01<00:03, 3.59it/s]
32%|█████████████▉ | 6/19 [00:01<00:03, 3.33it/s]
37%|████████████████▏ | 7/19 [00:01<00:03, 3.30it/s]
42%|██████████████████▌ | 8/19 [00:02<00:03, 3.23it/s]
47%|████████████████████▊ | 9/19 [00:02<00:03, 3.19it/s]
53%|██████████████████████▋ | 10/19 [00:02<00:02, 3.26it/s]
58%|████████████████████████▉ | 11/19 [00:03<00:02, 3.32it/s]
63%|███████████████████████████▏ | 12/19 [00:03<00:02, 3.17it/s]
68%|█████████████████████████████▍ | 13/19 [00:03<00:01, 3.05it/s]
74%|███████████████████████████████▋ | 14/19 [00:04<00:01, 2.93it/s]
79%|█████████████████████████████████▉ | 15/19 [00:04<00:01, 3.00it/s]
84%|████████████████████████████████████▏ | 16/19 [00:04<00:01, 2.97it/s]
89%|██████████████████████████████████████▍ | 17/19 [00:05<00:00, 3.05it/s]
95%|████████████████████████████████████████▋ | 18/19 [00:05<00:00, 3.26it/s]
{'eval_loss': 0.6963744759559631, 'eval_runtime': 6.0167, 'eval_samples_per_second': 12.465, 'eval_steps_per_second': 3.158, 'epoch': 0.08}
3%|█ | 75/2775 [04:51<2:19:28, 3.10s/it]
100%|███████████████████████████████████████████| 19/19 [00:05<00:00, 3.42it/s]
{'loss': 0.6812, 'learning_rate': 0.00019971896279143755, 'epoch': 0.08}
{'loss': 0.6381, 'learning_rate': 0.00019971038613935272, 'epoch': 0.08}
{'loss': 0.734, 'learning_rate': 0.00019970168076614332, 'epoch': 0.08}
{'loss': 0.8738, 'learning_rate': 0.0001996928466830475, 'epoch': 0.09}
{'loss': 0.6175, 'learning_rate': 0.0001996838839014696, 'epoch': 0.09}
3%|█ | 80/2775 [05:02<2:04:38, 2.78s/it]
0%| | 0/19 [00:00<?, ?it/s]
11%|████▋ | 2/19 [00:00<00:03, 5.42it/s]
16%|██████▉ | 3/19 [00:00<00:03, 4.56it/s]
21%|█████████▎ | 4/19 [00:00<00:03, 3.95it/s]
26%|███████████▌ | 5/19 [00:01<00:03, 3.58it/s]
32%|█████████████▉ | 6/19 [00:01<00:03, 3.32it/s]
37%|████████████████▏ | 7/19 [00:01<00:03, 3.30it/s]
42%|██████████████████▌ | 8/19 [00:02<00:03, 3.23it/s]
47%|████████████████████▊ | 9/19 [00:02<00:03, 3.18it/s]
53%|██████████████████████▋ | 10/19 [00:02<00:02, 3.24it/s]
58%|████████████████████████▉ | 11/19 [00:03<00:02, 3.31it/s]
63%|███████████████████████████▏ | 12/19 [00:03<00:02, 3.16it/s]
68%|█████████████████████████████▍ | 13/19 [00:03<00:01, 3.04it/s]
74%|███████████████████████████████▋ | 14/19 [00:04<00:01, 2.94it/s]
79%|█████████████████████████████████▉ | 15/19 [00:04<00:01, 3.00it/s]
84%|████████████████████████████████████▏ | 16/19 [00:04<00:01, 2.96it/s]
89%|██████████████████████████████████████▍ | 17/19 [00:05<00:00, 3.04it/s]
95%|████████████████████████████████████████▋ | 18/19 [00:05<00:00, 3.24it/s]
{'eval_loss': 0.689468264579773, 'eval_runtime': 6.0271, 'eval_samples_per_second': 12.444, 'eval_steps_per_second': 3.152, 'epoch': 0.09}
3%|█ | 80/2775 [05:08<2:04:38, 2.78s/it]
100%|███████████████████████████████████████████| 19/19 [00:05<00:00, 3.42it/s]
{'loss': 0.6772, 'learning_rate': 0.00019967479243298013, 'epoch': 0.09}
{'loss': 0.676, 'learning_rate': 0.00019966557228931576, 'epoch': 0.09}
{'loss': 0.6174, 'learning_rate': 0.00019965622348237916, 'epoch': 0.09}
{'loss': 0.7213, 'learning_rate': 0.00019964674602423926, 'epoch': 0.09}
{'loss': 0.6746, 'learning_rate': 0.00019963713992713093, 'epoch': 0.09}
3%|█▏ | 85/2775 [05:25<2:20:19, 3.13s/it]
0%| | 0/19 [00:00<?, ?it/s]
11%|████▋ | 2/19 [00:00<00:03, 5.43it/s]
16%|██████▉ | 3/19 [00:00<00:03, 4.56it/s]
21%|█████████▎ | 4/19 [00:00<00:03, 3.95it/s]
26%|███████████▌ | 5/19 [00:01<00:03, 3.59it/s]
32%|█████████████▉ | 6/19 [00:01<00:03, 3.33it/s]
37%|████████████████▏ | 7/19 [00:01<00:03, 3.30it/s]
42%|██████████████████▌ | 8/19 [00:02<00:03, 3.23it/s]
47%|████████████████████▊ | 9/19 [00:02<00:03, 3.18it/s]
53%|██████████████████████▋ | 10/19 [00:02<00:02, 3.26it/s]
58%|████████████████████████▉ | 11/19 [00:03<00:02, 3.32it/s]
63%|███████████████████████████▏ | 12/19 [00:03<00:02, 3.17it/s]
68%|█████████████████████████████▍ | 13/19 [00:03<00:01, 3.05it/s]
74%|███████████████████████████████▋ | 14/19 [00:04<00:01, 2.94it/s]
79%|█████████████████████████████████▉ | 15/19 [00:04<00:01, 3.01it/s]
84%|████████████████████████████████████▏ | 16/19 [00:04<00:01, 2.97it/s]
89%|██████████████████████████████████████▍ | 17/19 [00:05<00:00, 3.06it/s]
95%|████████████████████████████████████████▋ | 18/19 [00:05<00:00, 3.26it/s]
{'eval_loss': 0.6823983192443848, 'eval_runtime': 6.0142, 'eval_samples_per_second': 12.47, 'eval_steps_per_second': 3.159, 'epoch': 0.09}
3%|█▏ | 85/2775 [05:31<2:20:19, 3.13s/it]
100%|███████████████████████████████████████████| 19/19 [00:05<00:00, 3.43it/s]
{'loss': 0.6452, 'learning_rate': 0.00019962740520345514, 'epoch': 0.09}
{'loss': 0.581, 'learning_rate': 0.00019961754186577902, 'epoch': 0.09}
{'loss': 0.5279, 'learning_rate': 0.0001996075499268356, 'epoch': 0.1}
{'loss': 0.6252, 'learning_rate': 0.00019959742939952392, 'epoch': 0.1}
{'loss': 0.7314, 'learning_rate': 0.00019958718029690911, 'epoch': 0.1}
3%|█▎ | 90/2775 [05:42<2:03:38, 2.76s/it]
0%| | 0/19 [00:00<?, ?it/s]
11%|████▋ | 2/19 [00:00<00:03, 5.42it/s]
16%|██████▉ | 3/19 [00:00<00:03, 4.56it/s]
21%|█████████▎ | 4/19 [00:00<00:03, 3.95it/s]
26%|███████████▌ | 5/19 [00:01<00:03, 3.59it/s]
32%|█████████████▉ | 6/19 [00:01<00:03, 3.32it/s]
37%|████████████████▏ | 7/19 [00:01<00:03, 3.30it/s]
42%|██████████████████▌ | 8/19 [00:02<00:03, 3.23it/s]
47%|████████████████████▊ | 9/19 [00:02<00:03, 3.18it/s]
53%|██████████████████████▋ | 10/19 [00:02<00:02, 3.25it/s]
58%|████████████████████████▉ | 11/19 [00:03<00:02, 3.32it/s]
63%|███████████████████████████▏ | 12/19 [00:03<00:02, 3.17it/s]
68%|█████████████████████████████▍ | 13/19 [00:03<00:01, 3.05it/s]
74%|███████████████████████████████▋ | 14/19 [00:04<00:01, 2.94it/s]
79%|█████████████████████████████████▉ | 15/19 [00:04<00:01, 3.00it/s]
84%|████████████████████████████████████▏ | 16/19 [00:04<00:01, 2.97it/s]
89%|██████████████████████████████████████▍ | 17/19 [00:05<00:00, 3.06it/s]
95%|████████████████████████████████████████▋ | 18/19 [00:05<00:00, 3.25it/s]
{'eval_loss': 0.6797629594802856, 'eval_runtime': 6.0165, 'eval_samples_per_second': 12.466, 'eval_steps_per_second': 3.158, 'epoch': 0.1}
3%|█▎ | 90/2775 [05:48<2:03:38, 2.76s/it]
100%|███████████████████████████████████████████| 19/19 [00:05<00:00, 3.43it/s]
{'loss': 0.68, 'learning_rate': 0.00019957680263222232, 'epoch': 0.1}
{'loss': 0.6972, 'learning_rate': 0.0001995662964188605, 'epoch': 0.1}
{'loss': 0.6269, 'learning_rate': 0.0001995556616703867, 'epoch': 0.1}
{'loss': 0.8012, 'learning_rate': 0.0001995448984005298, 'epoch': 0.1}
{'loss': 0.5582, 'learning_rate': 0.00019953400662318468, 'epoch': 0.1}
3%|█▎ | 95/2775 [06:05<2:19:49, 3.13s/it]
0%| | 0/19 [00:00<?, ?it/s]
11%|████▋ | 2/19 [00:00<00:03, 5.43it/s]
16%|██████▉ | 3/19 [00:00<00:03, 4.57it/s]
21%|█████████▎ | 4/19 [00:00<00:03, 3.95it/s]
26%|███████████▌ | 5/19 [00:01<00:03, 3.59it/s]
32%|█████████████▉ | 6/19 [00:01<00:03, 3.33it/s]
37%|████████████████▏ | 7/19 [00:01<00:03, 3.30it/s]
42%|██████████████████▌ | 8/19 [00:02<00:03, 3.23it/s]
47%|████████████████████▊ | 9/19 [00:02<00:03, 3.19it/s]
53%|██████████████████████▋ | 10/19 [00:02<00:02, 3.26it/s]
58%|████████████████████████▉ | 11/19 [00:03<00:02, 3.32it/s]
63%|███████████████████████████▏ | 12/19 [00:03<00:02, 3.18it/s]
68%|█████████████████████████████▍ | 13/19 [00:03<00:01, 3.05it/s]
74%|███████████████████████████████▋ | 14/19 [00:04<00:01, 2.94it/s]
79%|█████████████████████████████████▉ | 15/19 [00:04<00:01, 3.01it/s]
84%|████████████████████████████████████▏ | 16/19 [00:04<00:01, 2.98it/s]
89%|██████████████████████████████████████▍ | 17/19 [00:05<00:00, 3.06it/s]
95%|████████████████████████████████████████▋ | 18/19 [00:05<00:00, 3.26it/s]
{'eval_loss': 0.6793395280838013, 'eval_runtime': 6.0096, 'eval_samples_per_second': 12.48, 'eval_steps_per_second': 3.162, 'epoch': 0.1}
3%|█▎ | 95/2775 [06:11<2:19:49, 3.13s/it]
100%|███████████████████████████████████████████| 19/19 [00:05<00:00, 3.43it/s]
{'loss': 0.567, 'learning_rate': 0.0001995229863524121, 'epoch': 0.1}
3%|█▎ | 96/2775 [06:13<3:23:17, 4.55s/it]^C
#2 run-15: 16*2 + xformer¶
In [18]:
!cat examples/falcon/qlora.yml
base_model: tiiuae/falcon-7b
base_model_config: tiiuae/falcon-7b
trust_remote_code: true
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer
load_in_8bit: false
load_in_4bit: true
gptq: false
strict: false
push_dataset_to_hub:
datasets:
- path: QingyiSi/Alpaca-CoT
data_files:
- Chain-of-Thought/formatted_cot_data/gsm8k_train.json
type: "alpaca:chat"
dataset_prepared_path: last_run_prepared
val_set_size: 0.01
adapter: qlora
lora_model_dir:
sequence_len: 2048
max_packed_sequence_len:
lora_r: 64
lora_alpha: 16
lora_dropout: 0.05
lora_target_modules:
lora_target_linear: true
lora_fan_in_fan_out:
wandb_project: falcon-qlora
wandb_watch:
wandb_run_id:
wandb_log_model:
output_dir: ./qlora-out
micro_batch_size: 32
gradient_accumulation_steps: 2
num_epochs: 3
optimizer: paged_adamw_32bit
torchdistx_path:
lr_scheduler: cosine
learning_rate: 0.0002
train_on_inputs: false
group_by_length: false
bf16: true
fp16: false
tf32: true
gradient_checkpointing: true
# stop training after this many evaluation losses have increased in a row
# https://huggingface.co/transformers/v4.2.2/_modules/transformers/trainer_callback.html#EarlyStoppingCallback
early_stopping_patience: 3
resume_from_checkpoint:
auto_resume_from_checkpoints: true
local_rank:
logging_steps: 1
xformers_attention: true
flash_attention:
gptq_groupsize:
gptq_model_v1:
warmup_steps: 10
eval_steps: 5
save_steps: 10
debug:
deepspeed:
weight_decay: 0.000001
fsdp:
fsdp_config:
special_tokens:
pad_token: "<|endoftext|>"
bos_token: ">>ABSTRACT<<"
eos_token: "<|endoftext|>"
In [19]:
!accelerate launch scripts/finetune.py examples/falcon/qlora.yml
Setting ds_accelerator to cuda (auto detect)
===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please run
python -m bitsandbytes
and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================
bin /root/miniconda3/envs/py3.9/lib/python3.9/site-packages/bitsandbytes-0.39.0-py3.9.egg/bitsandbytes/libbitsandbytes_cuda118.so
/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/bitsandbytes-0.39.0-py3.9.egg/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/usr/local/nvidia/lib64'), PosixPath('/usr/local/nvidia/lib')}
warn(msg)
/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/bitsandbytes-0.39.0-py3.9.egg/bitsandbytes/cuda_setup/main.py:149: UserWarning: /usr/local/nvidia/lib:/usr/local/nvidia/lib64 did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths...
warn(msg)
/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/bitsandbytes-0.39.0-py3.9.egg/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//matplotlib_inline.backend_inline'), PosixPath('module')}
warn(msg)
/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/bitsandbytes-0.39.0-py3.9.egg/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIK1tFOFrWbmoa2ckCJYhzgBHKTSMeR/AeuScCCzugqlI utensilcandel@gmail.com')}
warn(msg)
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/bitsandbytes-0.39.0-py3.9.egg/bitsandbytes/cuda_setup/main.py:149: UserWarning: Found duplicate ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] files: {PosixPath('/usr/local/cuda/lib64/libcudart.so.11.0'), PosixPath('/usr/local/cuda/lib64/libcudart.so')}.. We'll flip a coin and try one of these, in order to fail forward.
Either way, this might cause trouble in the future:
If you get `CUDA error: invalid device function` errors, the above might be the cause and the solution is to make sure only one ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] in the paths that we search based on your env.
warn(msg)
CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so.11.0
CUDA SETUP: Highest compute capability among GPUs detected: 8.6
CUDA SETUP: Detected CUDA version 118
CUDA SETUP: Loading binary /root/miniconda3/envs/py3.9/lib/python3.9/site-packages/bitsandbytes-0.39.0-py3.9.egg/bitsandbytes/libbitsandbytes_cuda118.so...
Setting ds_accelerator to cuda (auto detect)
WARNING:root:`trust_remote_code` is set to true. Please make sure that you reviewed the remote code/model.
INFO:root:loading tokenizer... tiiuae/falcon-7b
Using bos_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using unk_token, but it is not set yet.
INFO:root:Loading prepared dataset from disk at last_run_prepared/31a4e867d804a957707db033c9abcd13...
INFO:root:Prepared dataset loaded from disk...
INFO:root:loading model and peft_config...
Loading checkpoint shards: 100%|██████████████████| 2/2 [00:15<00:00, 7.93s/it]
INFO:root:converting PEFT model w/ prepare_model_for_int8_training
/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/peft/utils/other.py:76: FutureWarning: prepare_model_for_int8_training is deprecated and will be removed in a future version. Use prepare_model_for_kbit_training instead.
warnings.warn(
INFO:root:found linear modules: ['dense', 'query_key_value', 'dense_h_to_4h', 'dense_4h_to_h']
trainable params: 130547712 || all params: 3739292544 || trainable%: 3.4912409356543783
INFO:root:Compiling torch model
INFO:root:Pre-saving adapter config to ./qlora-out
INFO:root:Starting trainer...
INFO:root:Using Auto-resume functionality to start with checkpoint at qlora-out/checkpoint-20
wandb: Currently logged in as: utensil. Use `wandb login --relogin` to force relogin
wandb: Tracking run with wandb version 0.15.3
wandb: Run data is saved locally in /workspace/axolotl/wandb/run-20230601_092648-wmol98sl
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run driven-resonance-15
wandb: ⭐️ View project at https://wandb.ai/utensil/falcon-qlora
wandb: 🚀 View run at https://wandb.ai/utensil/falcon-qlora/runs/wmol98sl
0%| | 0/348 [00:00<?, ?it/s]You're using a PreTrainedTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
{'loss': 0.7366, 'learning_rate': 0.0001994777921806989, 'epoch': 0.18}
{'loss': 0.7105, 'learning_rate': 0.00019937863245275304, 'epoch': 0.19}
{'loss': 0.7804, 'learning_rate': 0.0001992708874098054, 'epoch': 0.2}
{'loss': 0.7878, 'learning_rate': 0.0001991545663599448, 'epoch': 0.21}
{'loss': 0.7642, 'learning_rate': 0.00019902967935214082, 'epoch': 0.22}
7%|███ | 25/348 [01:26<33:49, 6.28s/it]
0%| | 0/3 [00:00<?, ?it/s]
67%|██████████████████████████████ | 2/3 [00:02<00:01, 1.10s/it]
100%|█████████████████████████████████████████████| 3/3 [00:02<00:00, 1.06it/s]
{'eval_loss': 0.7560953497886658, 'eval_runtime': 5.0947, 'eval_samples_per_second': 14.721, 'eval_steps_per_second': 0.589, 'epoch': 0.22}
7%|███ | 25/348 [01:31<33:49, 6.28s/it]
{'loss': 0.746, 'learning_rate': 0.00019889623717537564, 'epoch': 0.22}
{'loss': 0.7889, 'learning_rate': 0.0001987542513577122, 'epoch': 0.23}
{'loss': 0.722, 'learning_rate': 0.00019860373416529802, 'epoch': 0.24}
{'loss': 0.719, 'learning_rate': 0.00019844469860130572, 'epoch': 0.25}
{'loss': 0.6846, 'learning_rate': 0.0001982771584048096, 'epoch': 0.26}
9%|███▍ | 30/348 [02:58<1:13:37, 13.89s/it]
0%| | 0/3 [00:00<?, ?it/s]
67%|██████████████████████████████ | 2/3 [00:02<00:01, 1.10s/it]
100%|█████████████████████████████████████████████| 3/3 [00:02<00:00, 1.06it/s]
{'eval_loss': 0.7330535650253296, 'eval_runtime': 5.0921, 'eval_samples_per_second': 14.729, 'eval_steps_per_second': 0.589, 'epoch': 0.26}
9%|███▍ | 30/348 [03:03<1:13:37, 13.89s/it]
{'loss': 0.7376, 'learning_rate': 0.00019810112804959865, 'epoch': 0.27}
{'loss': 0.7493, 'learning_rate': 0.00019791662274292637, 'epoch': 0.28}
{'loss': 0.712, 'learning_rate': 0.00019772365842419677, 'epoch': 0.28}
{'loss': 0.7273, 'learning_rate': 0.00019752225176358757, 'epoch': 0.29}
{'loss': 0.6568, 'learning_rate': 0.00019731242016060983, 'epoch': 0.3}
10%|████ | 35/348 [04:32<1:29:09, 17.09s/it]
0%| | 0/3 [00:00<?, ?it/s]
67%|██████████████████████████████ | 2/3 [00:02<00:01, 1.10s/it]
{'eval_loss': 0.7157832980155945, 'eval_runtime': 5.0923, 'eval_samples_per_second': 14.728, 'eval_steps_per_second': 0.589, 'epoch': 0.3}
10%|████ | 35/348 [04:37<1:29:09, 17.09s/it]
100%|█████████████████████████████████████████████| 3/3 [00:02<00:00, 1.06it/s]
{'loss': 0.639, 'learning_rate': 0.0001970941817426052, 'epoch': 0.31}
{'loss': 0.6835, 'learning_rate': 0.00019686755536317945, 'epoch': 0.32}
{'loss': 0.6886, 'learning_rate': 0.00019663256060057393, 'epoch': 0.33}
{'loss': 0.7036, 'learning_rate': 0.00019638921775597424, 'epoch': 0.34}
{'loss': 0.6819, 'learning_rate': 0.0001961375478517564, 'epoch': 0.34}
11%|████▌ | 40/348 [06:04<1:35:26, 18.59s/it]
0%| | 0/3 [00:00<?, ?it/s]
67%|██████████████████████████████ | 2/3 [00:02<00:01, 1.10s/it]
{'eval_loss': 0.6986502408981323, 'eval_runtime': 5.0968, 'eval_samples_per_second': 14.715, 'eval_steps_per_second': 0.589, 'epoch': 0.34}
11%|████▌ | 40/348 [06:09<1:35:26, 18.59s/it]
100%|█████████████████████████████████████████████| 3/3 [00:02<00:00, 1.06it/s]
{'loss': 0.6565, 'learning_rate': 0.00019587757262967058, 'epoch': 0.35}
{'loss': 0.6407, 'learning_rate': 0.00019560931454896298, 'epoch': 0.36}
{'loss': 0.6636, 'learning_rate': 0.0001953327967844356, 'epoch': 0.37}
{'loss': 0.6172, 'learning_rate': 0.000195048043224444, 'epoch': 0.38}
{'loss': 0.6129, 'learning_rate': 0.00019475507846883377, 'epoch': 0.39}
13%|█████▏ | 45/348 [07:32<1:24:35, 16.75s/it]
0%| | 0/3 [00:00<?, ?it/s]
67%|██████████████████████████████ | 2/3 [00:02<00:01, 1.10s/it]
{'eval_loss': 0.6819314360618591, 'eval_runtime': 5.0971, 'eval_samples_per_second': 14.714, 'eval_steps_per_second': 0.589, 'epoch': 0.39}
13%|█████▏ | 45/348 [07:37<1:24:35, 16.75s/it]
100%|█████████████████████████████████████████████| 3/3 [00:02<00:00, 1.06it/s]
{'loss': 0.6559, 'learning_rate': 0.00019445392782681522, 'epoch': 0.4}
{'loss': 0.6797, 'learning_rate': 0.000194144617314777, 'epoch': 0.41}
{'loss': 0.6313, 'learning_rate': 0.00019382717365403854, 'epoch': 0.41}
{'loss': 0.6444, 'learning_rate': 0.0001935016242685415, 'epoch': 0.42}
{'loss': 0.6206, 'learning_rate': 0.00019316799728248075, 'epoch': 0.43}
14%|█████▋ | 50/348 [09:06<1:27:32, 17.63s/it]
0%| | 0/3 [00:00<?, ?it/s]
67%|██████████████████████████████ | 2/3 [00:02<00:01, 1.10s/it]
{'eval_loss': 0.6761702299118042, 'eval_runtime': 5.1012, 'eval_samples_per_second': 14.702, 'eval_steps_per_second': 0.588, 'epoch': 0.43}
14%|█████▋ | 50/348 [09:11<1:27:32, 17.63s/it]
100%|█████████████████████████████████████████████| 3/3 [00:02<00:00, 1.06it/s]
{'loss': 0.6976, 'learning_rate': 0.00019282632151787463, 'epoch': 0.44}
{'loss': 0.6352, 'learning_rate': 0.0001924766264920751, 'epoch': 0.45}
{'loss': 0.6153, 'learning_rate': 0.00019211894241521758, 'epoch': 0.46}
{'loss': 0.6334, 'learning_rate': 0.0001917533001876113, 'epoch': 0.47}
{'loss': 0.7208, 'learning_rate': 0.00019137973139706974, 'epoch': 0.47}
16%|██████▎ | 55/348 [10:35<1:21:44, 16.74s/it]
0%| | 0/3 [00:00<?, ?it/s]
67%|██████████████████████████████ | 2/3 [00:02<00:01, 1.10s/it]
{'eval_loss': 0.6638404130935669, 'eval_runtime': 5.0937, 'eval_samples_per_second': 14.724, 'eval_steps_per_second': 0.589, 'epoch': 0.47}
16%|██████▎ | 55/348 [10:40<1:21:44, 16.74s/it]
100%|█████████████████████████████████████████████| 3/3 [00:02<00:00, 1.06it/s]
{'loss': 0.6449, 'learning_rate': 0.0001909982683161817, 'epoch': 0.48}
{'loss': 0.6586, 'learning_rate': 0.00019060894389952328, 'epoch': 0.49}
{'loss': 0.6182, 'learning_rate': 0.00019021179178081105, 'epoch': 0.5}
{'loss': 0.6092, 'learning_rate': 0.0001898068462699964, 'epoch': 0.51}
{'loss': 0.6224, 'learning_rate': 0.00018939414235030134, 'epoch': 0.52}
17%|██████▉ | 60/348 [11:57<1:16:22, 15.91s/it]
0%| | 0/3 [00:00<?, ?it/s]
67%|██████████████████████████████ | 2/3 [00:02<00:01, 1.10s/it]
{'eval_loss': 0.6583952307701111, 'eval_runtime': 5.0975, 'eval_samples_per_second': 14.713, 'eval_steps_per_second': 0.589, 'epoch': 0.52}
17%|██████▉ | 60/348 [12:03<1:16:22, 15.91s/it]
100%|█████████████████████████████████████████████| 3/3 [00:02<00:00, 1.06it/s]
{'loss': 0.6197, 'learning_rate': 0.0001889737156751965, 'epoch': 0.53}
{'loss': 0.6879, 'learning_rate': 0.000188545602565321, 'epoch': 0.53}
{'loss': 0.5799, 'learning_rate': 0.00018810984000534458, 'epoch': 0.54}
{'loss': 0.6077, 'learning_rate': 0.00018766646564077265, 'epoch': 0.55}
{'loss': 0.5766, 'learning_rate': 0.00018721551777469396, 'epoch': 0.56}
19%|███████▍ | 65/348 [13:34<1:23:32, 17.71s/it]
0%| | 0/3 [00:00<?, ?it/s]
67%|██████████████████████████████ | 2/3 [00:02<00:01, 1.10s/it]
{'eval_loss': 0.6495146155357361, 'eval_runtime': 5.0998, 'eval_samples_per_second': 14.707, 'eval_steps_per_second': 0.588, 'epoch': 0.56}
19%|███████▍ | 65/348 [13:39<1:23:32, 17.71s/it]
100%|█████████████████████████████████████████████| 3/3 [00:02<00:00, 1.06it/s]
{'loss': 0.6615, 'learning_rate': 0.00018675703536447178, 'epoch': 0.57}
{'loss': 0.6234, 'learning_rate': 0.00018629105801837818, 'epoch': 0.58}
{'loss': 0.5719, 'learning_rate': 0.00018581762599217242, 'epoch': 0.59}
{'loss': 0.6227, 'learning_rate': 0.00018533678018562309, 'epoch': 0.59}
{'loss': 0.6539, 'learning_rate': 0.00018484856213897498, 'epoch': 0.6}
20%|████████ | 70/348 [15:05<1:21:43, 17.64s/it]
0%| | 0/3 [00:00<?, ?it/s]
67%|██████████████████████████████ | 2/3 [00:02<00:01, 1.10s/it]
{'eval_loss': 0.6446104645729065, 'eval_runtime': 5.0981, 'eval_samples_per_second': 14.711, 'eval_steps_per_second': 0.588, 'epoch': 0.6}
20%|████████ | 70/348 [15:10<1:21:43, 17.64s/it]
100%|█████████████████████████████████████████████| 3/3 [00:02<00:00, 1.06it/s]
^C
#3 run-15: 64*2 + xformer OOM¶
In [25]:
!cat examples/falcon/qlora.yml
base_model: tiiuae/falcon-7b
base_model_config: tiiuae/falcon-7b
trust_remote_code: true
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer
load_in_8bit: false
load_in_4bit: true
gptq: false
strict: false
push_dataset_to_hub:
datasets:
- path: QingyiSi/Alpaca-CoT
data_files:
- Chain-of-Thought/formatted_cot_data/gsm8k_train.json
type: "alpaca:chat"
dataset_prepared_path: last_run_prepared
val_set_size: 0.01
adapter: qlora
lora_model_dir:
sequence_len: 2048
max_packed_sequence_len:
lora_r: 64
lora_alpha: 16
lora_dropout: 0.05
lora_target_modules:
lora_target_linear: true
lora_fan_in_fan_out:
wandb_project: falcon-qlora
wandb_watch:
wandb_run_id:
wandb_log_model:
output_dir: ./qlora-out
micro_batch_size: 64
gradient_accumulation_steps: 2
num_epochs: 3
optimizer: paged_adamw_32bit
torchdistx_path:
lr_scheduler: cosine
learning_rate: 0.0002
train_on_inputs: false
group_by_length: false
bf16: true
fp16: false
tf32: true
gradient_checkpointing: true
# stop training after this many evaluation losses have increased in a row
# https://huggingface.co/transformers/v4.2.2/_modules/transformers/trainer_callback.html#EarlyStoppingCallback
early_stopping_patience: 3
resume_from_checkpoint:
auto_resume_from_checkpoints: true
local_rank:
logging_steps: 1
xformers_attention: true
flash_attention:
gptq_groupsize:
gptq_model_v1:
warmup_steps: 10
eval_steps: 5
save_steps: 10
debug:
deepspeed:
weight_decay: 0.000001
fsdp:
fsdp_config:
special_tokens:
pad_token: "<|endoftext|>"
bos_token: ">>ABSTRACT<<"
eos_token: "<|endoftext|>"
In [26]:
!accelerate launch scripts/finetune.py examples/falcon/qlora.yml
Setting ds_accelerator to cuda (auto detect)
===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please run
python -m bitsandbytes
and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================
bin /root/miniconda3/envs/py3.9/lib/python3.9/site-packages/bitsandbytes-0.39.0-py3.9.egg/bitsandbytes/libbitsandbytes_cuda118.so
/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/bitsandbytes-0.39.0-py3.9.egg/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/usr/local/nvidia/lib'), PosixPath('/usr/local/nvidia/lib64')}
warn(msg)
/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/bitsandbytes-0.39.0-py3.9.egg/bitsandbytes/cuda_setup/main.py:149: UserWarning: /usr/local/nvidia/lib:/usr/local/nvidia/lib64 did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths...
warn(msg)
/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/bitsandbytes-0.39.0-py3.9.egg/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//matplotlib_inline.backend_inline'), PosixPath('module')}
warn(msg)
/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/bitsandbytes-0.39.0-py3.9.egg/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIK1tFOFrWbmoa2ckCJYhzgBHKTSMeR/AeuScCCzugqlI utensilcandel@gmail.com')}
warn(msg)
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/bitsandbytes-0.39.0-py3.9.egg/bitsandbytes/cuda_setup/main.py:149: UserWarning: Found duplicate ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] files: {PosixPath('/usr/local/cuda/lib64/libcudart.so.11.0'), PosixPath('/usr/local/cuda/lib64/libcudart.so')}.. We'll flip a coin and try one of these, in order to fail forward.
Either way, this might cause trouble in the future:
If you get `CUDA error: invalid device function` errors, the above might be the cause and the solution is to make sure only one ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] in the paths that we search based on your env.
warn(msg)
CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so.11.0
CUDA SETUP: Highest compute capability among GPUs detected: 8.6
CUDA SETUP: Detected CUDA version 118
CUDA SETUP: Loading binary /root/miniconda3/envs/py3.9/lib/python3.9/site-packages/bitsandbytes-0.39.0-py3.9.egg/bitsandbytes/libbitsandbytes_cuda118.so...
Setting ds_accelerator to cuda (auto detect)
WARNING:root:`trust_remote_code` is set to true. Please make sure that you reviewed the remote code/model.
INFO:root:loading tokenizer... tiiuae/falcon-7b
Using bos_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using unk_token, but it is not set yet.
INFO:root:Loading prepared dataset from disk at last_run_prepared/31a4e867d804a957707db033c9abcd13...
INFO:root:Prepared dataset loaded from disk...
INFO:root:loading model and peft_config...
Loading checkpoint shards: 100%|██████████████████| 2/2 [00:16<00:00, 8.49s/it]
INFO:root:converting PEFT model w/ prepare_model_for_int8_training
/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/peft/utils/other.py:76: FutureWarning: prepare_model_for_int8_training is deprecated and will be removed in a future version. Use prepare_model_for_kbit_training instead.
warnings.warn(
INFO:root:found linear modules: ['dense_h_to_4h', 'dense_4h_to_h', 'query_key_value', 'dense']
trainable params: 130547712 || all params: 3739292544 || trainable%: 3.4912409356543783
INFO:root:Compiling torch model
INFO:root:Pre-saving adapter config to ./qlora-out
INFO:root:Starting trainer...
wandb: Currently logged in as: utensil. Use `wandb login --relogin` to force relogin
wandb: Tracking run with wandb version 0.15.3
wandb: Run data is saved locally in /workspace/axolotl/wandb/run-20230601_094418-8i9lr0h6
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run rich-wave-16
wandb: ⭐️ View project at https://wandb.ai/utensil/falcon-qlora
wandb: 🚀 View run at https://wandb.ai/utensil/falcon-qlora/runs/8i9lr0h6
0%| | 0/174 [00:00<?, ?it/s]You're using a PreTrainedTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
{'loss': 1.1007, 'learning_rate': 2e-05, 'epoch': 0.02}
1%|▏ | 1/174 [00:37<1:48:03, 37.48s/it]Traceback (most recent call last):
File "/workspace/axolotl/scripts/finetune.py", line 294, in <module>
fire.Fire(train)
File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/fire/core.py", line 141, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/fire/core.py", line 475, in _Fire
component, remaining_args = _CallAndUpdateTrace(
File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "/workspace/axolotl/scripts/finetune.py", line 281, in train
trainer.train(resume_from_checkpoint=resume_from_checkpoint)
File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/transformers/trainer.py", line 1696, in train
return inner_training_loop(
File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/transformers/trainer.py", line 1973, in _inner_training_loop
tr_loss_step = self.training_step(model, inputs)
File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/transformers/trainer.py", line 2805, in training_step
loss.backward()
File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/torch/_tensor.py", line 487, in backward
torch.autograd.backward(
File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/torch/autograd/__init__.py", line 200, in backward
Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 5.81 GiB (GPU 0; 47.54 GiB total capacity; 28.08 GiB already allocated; 2.73 GiB free; 42.43 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
wandb: Waiting for W&B process to finish... (failed 1). Press Control-C to abort syncing.
wandb:
wandb: Run history:
wandb: train/epoch ▁
wandb: train/global_step ▁
wandb: train/learning_rate ▁
wandb: train/loss ▁
wandb:
wandb: Run summary:
wandb: train/epoch 0.02
wandb: train/global_step 1
wandb: train/learning_rate 2e-05
wandb: train/loss 1.1007
wandb:
wandb: 🚀 View run rich-wave-16 at: https://wandb.ai/utensil/falcon-qlora/runs/8i9lr0h6
wandb: Synced 6 W&B file(s), 0 media file(s), 2 artifact file(s) and 0 other file(s)
wandb: Find logs at: ./wandb/run-20230601_094418-8i9lr0h6/logs
Traceback (most recent call last):
File "/root/miniconda3/envs/py3.9/bin/accelerate", line 8, in <module>
sys.exit(main())
File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/accelerate/commands/accelerate_cli.py", line 45, in main
args.func(args)
File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/accelerate/commands/launch.py", line 928, in launch_command
simple_launcher(args)
File "/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/accelerate/commands/launch.py", line 588, in simple_launcher
raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/root/miniconda3/envs/py3.9/bin/python3', 'scripts/finetune.py', 'examples/falcon/qlora.yml']' returned non-zero exit status 1.
#4 run-16: 40*2 + xformer¶
In [27]:
!cat examples/falcon/qlora.yml
base_model: tiiuae/falcon-7b
base_model_config: tiiuae/falcon-7b
trust_remote_code: true
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer
load_in_8bit: false
load_in_4bit: true
gptq: false
strict: false
push_dataset_to_hub:
datasets:
- path: QingyiSi/Alpaca-CoT
data_files:
- Chain-of-Thought/formatted_cot_data/gsm8k_train.json
type: "alpaca:chat"
dataset_prepared_path: last_run_prepared
val_set_size: 0.01
adapter: qlora
lora_model_dir:
sequence_len: 2048
max_packed_sequence_len:
lora_r: 64
lora_alpha: 16
lora_dropout: 0.05
lora_target_modules:
lora_target_linear: true
lora_fan_in_fan_out:
wandb_project: falcon-qlora
wandb_watch:
wandb_run_id:
wandb_log_model:
output_dir: ./qlora-out
micro_batch_size: 40
gradient_accumulation_steps: 2
num_epochs: 3
optimizer: paged_adamw_32bit
torchdistx_path:
lr_scheduler: cosine
learning_rate: 0.0002
train_on_inputs: false
group_by_length: false
bf16: true
fp16: false
tf32: true
gradient_checkpointing: true
# stop training after this many evaluation losses have increased in a row
# https://huggingface.co/transformers/v4.2.2/_modules/transformers/trainer_callback.html#EarlyStoppingCallback
early_stopping_patience: 3
resume_from_checkpoint:
auto_resume_from_checkpoints: true
local_rank:
logging_steps: 1
xformers_attention: true
flash_attention:
gptq_groupsize:
gptq_model_v1:
warmup_steps: 10
eval_steps: 5
save_steps: 10
debug:
deepspeed:
weight_decay: 0.000001
fsdp:
fsdp_config:
special_tokens:
pad_token: "<|endoftext|>"
bos_token: ">>ABSTRACT<<"
eos_token: "<|endoftext|>"
In [28]:
!accelerate launch scripts/finetune.py examples/falcon/qlora.yml
Setting ds_accelerator to cuda (auto detect)
===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please run
python -m bitsandbytes
and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================
bin /root/miniconda3/envs/py3.9/lib/python3.9/site-packages/bitsandbytes-0.39.0-py3.9.egg/bitsandbytes/libbitsandbytes_cuda118.so
/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/bitsandbytes-0.39.0-py3.9.egg/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/usr/local/nvidia/lib64'), PosixPath('/usr/local/nvidia/lib')}
warn(msg)
/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/bitsandbytes-0.39.0-py3.9.egg/bitsandbytes/cuda_setup/main.py:149: UserWarning: /usr/local/nvidia/lib:/usr/local/nvidia/lib64 did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths...
warn(msg)
/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/bitsandbytes-0.39.0-py3.9.egg/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//matplotlib_inline.backend_inline'), PosixPath('module')}
warn(msg)
/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/bitsandbytes-0.39.0-py3.9.egg/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIK1tFOFrWbmoa2ckCJYhzgBHKTSMeR/AeuScCCzugqlI utensilcandel@gmail.com')}
warn(msg)
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths...
/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/bitsandbytes-0.39.0-py3.9.egg/bitsandbytes/cuda_setup/main.py:149: UserWarning: Found duplicate ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] files: {PosixPath('/usr/local/cuda/lib64/libcudart.so.11.0'), PosixPath('/usr/local/cuda/lib64/libcudart.so')}.. We'll flip a coin and try one of these, in order to fail forward.
Either way, this might cause trouble in the future:
If you get `CUDA error: invalid device function` errors, the above might be the cause and the solution is to make sure only one ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] in the paths that we search based on your env.
warn(msg)
CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so.11.0
CUDA SETUP: Highest compute capability among GPUs detected: 8.6
CUDA SETUP: Detected CUDA version 118
CUDA SETUP: Loading binary /root/miniconda3/envs/py3.9/lib/python3.9/site-packages/bitsandbytes-0.39.0-py3.9.egg/bitsandbytes/libbitsandbytes_cuda118.so...
Setting ds_accelerator to cuda (auto detect)
WARNING:root:`trust_remote_code` is set to true. Please make sure that you reviewed the remote code/model.
INFO:root:loading tokenizer... tiiuae/falcon-7b
Using bos_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using unk_token, but it is not set yet.
INFO:root:Loading prepared dataset from disk at last_run_prepared/31a4e867d804a957707db033c9abcd13...
INFO:root:Prepared dataset loaded from disk...
INFO:root:loading model and peft_config...
Loading checkpoint shards: 100%|██████████████████| 2/2 [00:15<00:00, 7.77s/it]
INFO:root:converting PEFT model w/ prepare_model_for_int8_training
/root/miniconda3/envs/py3.9/lib/python3.9/site-packages/peft/utils/other.py:76: FutureWarning: prepare_model_for_int8_training is deprecated and will be removed in a future version. Use prepare_model_for_kbit_training instead.
warnings.warn(
INFO:root:found linear modules: ['dense_4h_to_h', 'dense', 'dense_h_to_4h', 'query_key_value']
trainable params: 130547712 || all params: 3739292544 || trainable%: 3.4912409356543783
INFO:root:Compiling torch model
INFO:root:Pre-saving adapter config to ./qlora-out
INFO:root:Starting trainer...
wandb: Currently logged in as: utensil. Use `wandb login --relogin` to force relogin
wandb: Tracking run with wandb version 0.15.3
wandb: Run data is saved locally in /workspace/axolotl/wandb/run-20230601_095042-jxjubssp
wandb: Run `wandb offline` to turn off syncing.
wandb: Syncing run exalted-wind-17
wandb: ⭐️ View project at https://wandb.ai/utensil/falcon-qlora
wandb: 🚀 View run at https://wandb.ai/utensil/falcon-qlora/runs/jxjubssp
0%| | 0/276 [00:00<?, ?it/s]You're using a PreTrainedTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
{'loss': 1.1418, 'learning_rate': 2e-05, 'epoch': 0.01}
{'loss': 1.0759, 'learning_rate': 4e-05, 'epoch': 0.02}
{'loss': 1.1256, 'learning_rate': 6e-05, 'epoch': 0.03}
{'loss': 1.1398, 'learning_rate': 8e-05, 'epoch': 0.04}
{'loss': 1.0967, 'learning_rate': 0.0001, 'epoch': 0.05}
2%|▋ | 5/276 [01:50<1:39:57, 22.13s/it]
0%| | 0/2 [00:00<?, ?it/s]
{'eval_loss': 1.1484076976776123, 'eval_runtime': 5.1673, 'eval_samples_per_second': 14.514, 'eval_steps_per_second': 0.387, 'epoch': 0.05}
2%|▋ | 5/276 [01:56<1:39:57, 22.13s/it]
100%|█████████████████████████████████████████████| 2/2 [00:02<00:00, 1.20s/it]
{'loss': 1.085, 'learning_rate': 0.00012, 'epoch': 0.06}
{'loss': 1.0476, 'learning_rate': 0.00014, 'epoch': 0.08}
{'loss': 1.0615, 'learning_rate': 0.00016, 'epoch': 0.09}
{'loss': 0.9535, 'learning_rate': 0.00018, 'epoch': 0.1}
{'loss': 0.9402, 'learning_rate': 0.0002, 'epoch': 0.11}
4%|█▍ | 10/276 [03:37<1:34:28, 21.31s/it]
0%| | 0/2 [00:00<?, ?it/s]
{'eval_loss': 0.9538711309432983, 'eval_runtime': 5.1813, 'eval_samples_per_second': 14.475, 'eval_steps_per_second': 0.386, 'epoch': 0.11}
4%|█▍ | 10/276 [03:42<1:34:28, 21.31s/it]
100%|█████████████████████████████████████████████| 2/2 [00:02<00:00, 1.20s/it]
{'loss': 0.9144, 'learning_rate': 0.00019999302568709547, 'epoch': 0.12}
{'loss': 0.8303, 'learning_rate': 0.00019997210372120274, 'epoch': 0.13}
{'loss': 0.8115, 'learning_rate': 0.00019993723702064852, 'epoch': 0.14}
{'loss': 0.8351, 'learning_rate': 0.0001998884304488584, 'epoch': 0.15}
{'loss': 0.8404, 'learning_rate': 0.00019982569081367844, 'epoch': 0.16}
5%|██▏ | 15/276 [05:26<1:28:58, 20.45s/it]
0%| | 0/2 [00:00<?, ?it/s]
{'eval_loss': 0.8272685408592224, 'eval_runtime': 5.1776, 'eval_samples_per_second': 14.486, 'eval_steps_per_second': 0.386, 'epoch': 0.16}
5%|██▏ | 15/276 [05:31<1:28:58, 20.45s/it]
100%|█████████████████████████████████████████████| 2/2 [00:02<00:00, 1.20s/it]
{'loss': 0.7913, 'learning_rate': 0.00019974902686642558, 'epoch': 0.17}
{'loss': 0.7478, 'learning_rate': 0.000199658449300667, 'epoch': 0.18}
{'loss': 0.7906, 'learning_rate': 0.00019955397075072838, 'epoch': 0.19}
{'loss': 0.8069, 'learning_rate': 0.00019943560578993168, 'epoch': 0.21}
{'loss': 0.7791, 'learning_rate': 0.00019930337092856243, 'epoch': 0.22}
7%|██▉ | 20/276 [07:20<1:33:44, 21.97s/it]
0%| | 0/2 [00:00<?, ?it/s]
{'eval_loss': 0.7739953398704529, 'eval_runtime': 5.1785, 'eval_samples_per_second': 14.483, 'eval_steps_per_second': 0.386, 'epoch': 0.22}
7%|██▉ | 20/276 [07:25<1:33:44, 21.97s/it]
100%|█████████████████████████████████████████████| 2/2 [00:02<00:00, 1.20s/it]
{'loss': 0.7645, 'learning_rate': 0.00019915728461156656, 'epoch': 0.23}
{'loss': 0.7884, 'learning_rate': 0.00019899736721597786, 'epoch': 0.24}
{'loss': 0.7534, 'learning_rate': 0.00019882364104807537, 'epoch': 0.25}
{'loss': 0.6921, 'learning_rate': 0.00019863613034027224, 'epoch': 0.26}
{'loss': 0.7654, 'learning_rate': 0.00019843486124773545, 'epoch': 0.27}
9%|███▌ | 25/276 [09:18<1:30:16, 21.58s/it]
0%| | 0/2 [00:00<?, ?it/s]
{'eval_loss': 0.7478851675987244, 'eval_runtime': 5.1763, 'eval_samples_per_second': 14.489, 'eval_steps_per_second': 0.386, 'epoch': 0.27}
9%|███▌ | 25/276 [09:23<1:30:16, 21.58s/it]
100%|█████████████████████████████████████████████| 2/2 [00:02<00:00, 1.20s/it]
{'loss': 0.7728, 'learning_rate': 0.00019821986184473755, 'epoch': 0.28}
{'loss': 0.7357, 'learning_rate': 0.00019799116212074078, 'epoch': 0.29}
{'loss': 0.6604, 'learning_rate': 0.00019774879397621385, 'epoch': 0.3}
{'loss': 0.6484, 'learning_rate': 0.00019749279121818235, 'epoch': 0.31}
{'loss': 0.727, 'learning_rate': 0.00019722318955551306, 'epoch': 0.32}
11%|████▎ | 30/276 [11:05<1:24:58, 20.73s/it]
0%| | 0/2 [00:00<?, ?it/s]
{'eval_loss': 0.7283483147621155, 'eval_runtime': 5.179, 'eval_samples_per_second': 14.482, 'eval_steps_per_second': 0.386, 'epoch': 0.32}
11%|████▎ | 30/276 [11:10<1:24:58, 20.73s/it]
100%|█████████████████████████████████████████████| 2/2 [00:02<00:00, 1.20s/it]
{'loss': 0.7156, 'learning_rate': 0.00019694002659393305, 'epoch': 0.34}
{'loss': 0.6982, 'learning_rate': 0.00019664334183078428, 'epoch': 0.35}
{'loss': 0.6883, 'learning_rate': 0.0001963331766495142, 'epoch': 0.36}
{'loss': 0.6488, 'learning_rate': 0.00019600957431390326, 'epoch': 0.37}
{'loss': 0.6389, 'learning_rate': 0.00019567257996203047, 'epoch': 0.38}
13%|█████ | 35/276 [13:05<1:27:27, 21.77s/it]
0%| | 0/2 [00:00<?, ?it/s]
{'eval_loss': 0.7099348902702332, 'eval_runtime': 5.1774, 'eval_samples_per_second': 14.486, 'eval_steps_per_second': 0.386, 'epoch': 0.38}
13%|█████ | 35/276 [13:11<1:27:27, 21.77s/it]
100%|█████████████████████████████████████████████| 2/2 [00:02<00:00, 1.20s/it]
{'loss': 0.6438, 'learning_rate': 0.00019532224059997692, 'epoch': 0.39}
{'loss': 0.6792, 'learning_rate': 0.00019495860509526934, 'epoch': 0.4}
{'loss': 0.6582, 'learning_rate': 0.00019458172417006347, 'epoch': 0.41}
{'loss': 0.6655, 'learning_rate': 0.0001941916503940694, 'epoch': 0.42}
{'loss': 0.6357, 'learning_rate': 0.00019378843817721854, 'epoch': 0.43}
14%|█████▊ | 40/276 [14:57<1:26:25, 21.97s/it]
0%| | 0/2 [00:00<?, ?it/s]
{'eval_loss': 0.6957049369812012, 'eval_runtime': 5.1749, 'eval_samples_per_second': 14.493, 'eval_steps_per_second': 0.386, 'epoch': 0.43}
14%|█████▊ | 40/276 [15:02<1:26:25, 21.97s/it]
100%|█████████████████████████████████████████████| 2/2 [00:02<00:00, 1.20s/it]
{'loss': 0.6956, 'learning_rate': 0.00019337214376207416, 'epoch': 0.44}
{'loss': 0.6584, 'learning_rate': 0.0001929428252159866, 'epoch': 0.45}
{'loss': 0.6088, 'learning_rate': 0.00019250054242299326, 'epoch': 0.46}
{'loss': 0.7277, 'learning_rate': 0.00019204535707546602, 'epoch': 0.48}
{'loss': 0.6507, 'learning_rate': 0.00019157733266550575, 'epoch': 0.49}
16%|██████▌ | 45/276 [16:48<1:20:39, 20.95s/it]
0%| | 0/2 [00:00<?, ?it/s]
{'eval_loss': 0.6799066662788391, 'eval_runtime': 5.176, 'eval_samples_per_second': 14.49, 'eval_steps_per_second': 0.386, 'epoch': 0.49}
16%|██████▌ | 45/276 [16:53<1:20:39, 20.95s/it]
100%|█████████████████████████████████████████████| 2/2 [00:02<00:00, 1.20s/it]
{'loss': 0.6571, 'learning_rate': 0.00019109653447608606, 'epoch': 0.5}
{'loss': 0.6153, 'learning_rate': 0.0001906030295719473, 'epoch': 0.51}
{'loss': 0.6453, 'learning_rate': 0.0001900968867902419, 'epoch': 0.52}
{'loss': 0.6531, 'learning_rate': 0.00018957817673093258, 'epoch': 0.53}
{'loss': 0.6646, 'learning_rate': 0.00018904697174694447, 'epoch': 0.54}
18%|███████▏ | 50/276 [18:39<1:22:50, 21.99s/it]
0%| | 0/2 [00:00<?, ?it/s]
{'eval_loss': 0.668805718421936, 'eval_runtime': 5.184, 'eval_samples_per_second': 14.467, 'eval_steps_per_second': 0.386, 'epoch': 0.54}
18%|███████▏ | 50/276 [18:44<1:22:50, 21.99s/it]
100%|█████████████████████████████████████████████| 2/2 [00:02<00:00, 1.20s/it]
{'loss': 0.5817, 'learning_rate': 0.0001885033459340731, 'epoch': 0.55}
{'loss': 0.5987, 'learning_rate': 0.0001879473751206489, 'epoch': 0.56}
{'loss': 0.6538, 'learning_rate': 0.0001873791368569603, 'epoch': 0.57}
{'loss': 0.6214, 'learning_rate': 0.00018679871040443632, 'epoch': 0.58}
{'loss': 0.626, 'learning_rate': 0.00018620617672459097, 'epoch': 0.59}
20%|███████▉ | 55/276 [20:35<1:21:39, 22.17s/it]
0%| | 0/2 [00:00<?, ?it/s]
{'eval_loss': 0.6674392223358154, 'eval_runtime': 5.1742, 'eval_samples_per_second': 14.495, 'eval_steps_per_second': 0.387, 'epoch': 0.59}
20%|███████▉ | 55/276 [20:41<1:21:39, 22.17s/it]
100%|█████████████████████████████████████████████| 2/2 [00:02<00:00, 1.20s/it]
{'loss': 0.6459, 'learning_rate': 0.00018560161846773002, 'epoch': 0.61}
{'loss': 0.5957, 'learning_rate': 0.00018498511996142253, 'epoch': 0.62}
{'loss': 0.6218, 'learning_rate': 0.0001843567671987383, 'epoch': 0.63}
{'loss': 0.6846, 'learning_rate': 0.00018371664782625287, 'epoch': 0.64}
{'loss': 0.5973, 'learning_rate': 0.0001830648511318223, 'epoch': 0.65}
22%|████████▋ | 60/276 [22:30<1:20:19, 22.31s/it]
0%| | 0/2 [00:00<?, ?it/s]
{'eval_loss': 0.6522707939147949, 'eval_runtime': 5.1812, 'eval_samples_per_second': 14.475, 'eval_steps_per_second': 0.386, 'epoch': 0.65}
22%|████████▋ | 60/276 [22:35<1:20:19, 22.31s/it]
100%|█████████████████████████████████████████████| 2/2 [00:02<00:00, 1.20s/it]
{'loss': 0.6599, 'learning_rate': 0.00018240146803212852, 'epoch': 0.66}
{'loss': 0.6471, 'learning_rate': 0.00018172659105999782, 'epoch': 0.67}
{'loss': 0.604, 'learning_rate': 0.00018104031435149364, 'epoch': 0.68}
{'loss': 0.6467, 'learning_rate': 0.00018034273363278614, 'epoch': 0.69}
{'loss': 0.6578, 'learning_rate': 0.00017963394620679944, 'epoch': 0.7}
24%|█████████▍ | 65/276 [24:16<1:09:36, 19.80s/it]
0%| | 0/2 [00:00<?, ?it/s]
{'eval_loss': 0.6418537497520447, 'eval_runtime': 5.1824, 'eval_samples_per_second': 14.472, 'eval_steps_per_second': 0.386, 'epoch': 0.7}
24%|█████████▍ | 65/276 [24:21<1:09:36, 19.80s/it]
100%|█████████████████████████████████████████████| 2/2 [00:02<00:00, 1.20s/it]
{'loss': 0.6562, 'learning_rate': 0.00017891405093963938, 'epoch': 0.71}
{'loss': 0.6601, 'learning_rate': 0.000178183148246803, 'epoch': 0.72}
{'loss': 0.6248, 'learning_rate': 0.00017744134007917196, 'epoch': 0.74}
{'loss': 0.6493, 'learning_rate': 0.00017668872990879176, 'epoch': 0.75}
{'loss': 0.6287, 'learning_rate': 0.00017592542271443887, 'epoch': 0.76}
25%|██████████▏ | 70/276 [26:04<1:13:51, 21.51s/it]
0%| | 0/2 [00:00<?, ?it/s]
{'eval_loss': 0.6374295353889465, 'eval_runtime': 5.1786, 'eval_samples_per_second': 14.483, 'eval_steps_per_second': 0.386, 'epoch': 0.76}
25%|██████████▏ | 70/276 [26:09<1:13:51, 21.51s/it]
100%|█████████████████████████████████████████████| 2/2 [00:02<00:00, 1.20s/it]
{'loss': 0.5956, 'learning_rate': 0.00017515152496697765, 'epoch': 0.77}
{'loss': 0.6031, 'learning_rate': 0.00017436714461450899, 'epoch': 0.78}
{'loss': 0.6082, 'learning_rate': 0.00017357239106731317, 'epoch': 0.79}
{'loss': 0.631, 'learning_rate': 0.00017276737518258863, 'epoch': 0.8}
{'loss': 0.6372, 'learning_rate': 0.00017195220924898883, 'epoch': 0.81}
27%|██████████▊ | 75/276 [27:57<1:10:52, 21.16s/it]
0%| | 0/2 [00:00<?, ?it/s]
{'eval_loss': 0.6326897144317627, 'eval_runtime': 5.1779, 'eval_samples_per_second': 14.485, 'eval_steps_per_second': 0.386, 'epoch': 0.81}
27%|██████████▊ | 75/276 [28:02<1:10:52, 21.16s/it]
100%|█████████████████████████████████████████████| 2/2 [00:02<00:00, 1.20s/it]
{'loss': 0.6298, 'learning_rate': 0.00017112700697095954, 'epoch': 0.82}
{'loss': 0.5889, 'learning_rate': 0.00017029188345287868, 'epoch': 0.83}
{'loss': 0.6185, 'learning_rate': 0.00016944695518300085, 'epoch': 0.84}
{'loss': 0.5932, 'learning_rate': 0.00016859234001720882, 'epoch': 0.85}
{'loss': 0.6464, 'learning_rate': 0.00016772815716257412, 'epoch': 0.86}
29%|███████████▌ | 80/276 [29:44<1:08:56, 21.11s/it]
0%| | 0/2 [00:00<?, ?it/s]
{'eval_loss': 0.6283237338066101, 'eval_runtime': 5.1816, 'eval_samples_per_second': 14.474, 'eval_steps_per_second': 0.386, 'epoch': 0.86}
29%|███████████▌ | 80/276 [29:50<1:08:56, 21.11s/it]
100%|█████████████████████████████████████████████| 2/2 [00:02<00:00, 1.20s/it]
{'loss': 0.6074, 'learning_rate': 0.00016685452716072946, 'epoch': 0.88}
{'loss': 0.5856, 'learning_rate': 0.00016597157187105474, 'epoch': 0.89}
{'loss': 0.6056, 'learning_rate': 0.00016507941445367935, 'epoch': 0.9}
{'loss': 0.5829, 'learning_rate': 0.00016417817935230318, 'epoch': 0.91}
{'loss': 0.6036, 'learning_rate': 0.00016326799227683804, 'epoch': 0.92}
31%|████████████▎ | 85/276 [31:44<1:13:02, 22.94s/it]
0%| | 0/2 [00:00<?, ?it/s]
{'eval_loss': 0.624378502368927, 'eval_runtime': 5.1809, 'eval_samples_per_second': 14.476, 'eval_steps_per_second': 0.386, 'epoch': 0.92}
31%|████████████▎ | 85/276 [31:49<1:13:02, 22.94s/it]
100%|█████████████████████████████████████████████| 2/2 [00:02<00:00, 1.20s/it]
{'loss': 0.576, 'learning_rate': 0.00016234898018587337, 'epoch': 0.93}
{'loss': 0.6358, 'learning_rate': 0.0001614212712689668, 'epoch': 0.94}
{'loss': 0.6356, 'learning_rate': 0.00016048499492876377, 'epoch': 0.95}
{'loss': 0.5574, 'learning_rate': 0.00015954028176294748, 'epoch': 0.96}
{'loss': 0.5923, 'learning_rate': 0.00015858726354602248, 'epoch': 0.97}
33%|█████████████ | 90/276 [33:34<1:06:13, 21.36s/it]
0%| | 0/2 [00:00<?, ?it/s]
{'eval_loss': 0.6224556565284729, 'eval_runtime': 5.1829, 'eval_samples_per_second': 14.471, 'eval_steps_per_second': 0.386, 'epoch': 0.97}
33%|█████████████ | 90/276 [33:39<1:06:13, 21.36s/it]
100%|█████████████████████████████████████████████| 2/2 [00:02<00:00, 1.20s/it]
{'loss': 0.579, 'learning_rate': 0.00015762607321093367, 'epoch': 0.98}
{'loss': 0.5839, 'learning_rate': 0.00015665684483052425, 'epoch': 0.99}
{'loss': 0.6368, 'learning_rate': 0.0001556797135988342, 'epoch': 1.01}
{'loss': 0.5569, 'learning_rate': 0.00015469481581224272, 'epoch': 1.02}
{'loss': 0.5773, 'learning_rate': 0.0001537022888504566, 'epoch': 1.03}
34%|█████████████▊ | 95/276 [35:26<1:04:42, 21.45s/it]
0%| | 0/2 [00:00<?, ?it/s]
{'eval_loss': 0.6141393184661865, 'eval_runtime': 5.183, 'eval_samples_per_second': 14.47, 'eval_steps_per_second': 0.386, 'epoch': 1.03}
34%|█████████████▊ | 95/276 [35:31<1:04:42, 21.45s/it]
100%|█████████████████████████████████████████████| 2/2 [00:02<00:00, 1.20s/it]
{'loss': 0.5776, 'learning_rate': 0.0001527022711573479, 'epoch': 1.04}
{'loss': 0.5554, 'learning_rate': 0.00015169490222164254, 'epoch': 1.05}
{'loss': 0.5591, 'learning_rate': 0.00015068032255746402, 'epoch': 1.06}
{'loss': 0.5689, 'learning_rate': 0.00014965867368473308, 'epoch': 1.07}
{'loss': 0.5492, 'learning_rate': 0.00014863009810942815, 'epoch': 1.08}
36%|██████████████▏ | 100/276 [37:15<1:02:18, 21.24s/it]
0%| | 0/2 [00:00<?, ?it/s]
{'eval_loss': 0.6113758087158203, 'eval_runtime': 5.1811, 'eval_samples_per_second': 14.476, 'eval_steps_per_second': 0.386, 'epoch': 1.08}
36%|██████████████▏ | 100/276 [37:20<1:02:18, 21.24s/it]
100%|█████████████████████████████████████████████| 2/2 [00:02<00:00, 1.20s/it]
{'loss': 0.5334, 'learning_rate': 0.00014759473930370736, 'epoch': 1.09}
{'loss': 0.5876, 'learning_rate': 0.00014655274168589634, 'epoch': 1.1}
{'loss': 0.5495, 'learning_rate': 0.00014550425060034367, 'epoch': 1.11}
{'loss': 0.5404, 'learning_rate': 0.0001444494122971476, 'epoch': 1.12}
{'loss': 0.543, 'learning_rate': 0.00014338837391175582, 'epoch': 1.14}
38%|███████████████▌ | 105/276 [39:05<59:44, 20.96s/it]
0%| | 0/2 [00:00<?, ?it/s]
{'eval_loss': 0.608830988407135, 'eval_runtime': 5.1793, 'eval_samples_per_second': 14.481, 'eval_steps_per_second': 0.386, 'epoch': 1.14}
38%|███████████████▌ | 105/276 [39:10<59:44, 20.96s/it]
100%|█████████████████████████████████████████████| 2/2 [00:02<00:00, 1.20s/it]
{'loss': 0.5846, 'learning_rate': 0.0001423212834444425, 'epoch': 1.15}
{'loss': 0.559, 'learning_rate': 0.00014124828973966394, 'epoch': 1.16}
{'loss': 0.5687, 'learning_rate': 0.00014016954246529696, 'epoch': 1.17}
{'loss': 0.5639, 'learning_rate': 0.00013908519209176228, 'epoch': 1.18}
{'loss': 0.5326, 'learning_rate': 0.000137995389871036, 'epoch': 1.19}
40%|████████████████▎ | 110/276 [40:52<56:59, 20.60s/it]
0%| | 0/2 [00:00<?, ?it/s]
{'eval_loss': 0.61087566614151, 'eval_runtime': 5.1801, 'eval_samples_per_second': 14.479, 'eval_steps_per_second': 0.386, 'epoch': 1.19}
40%|████████████████▎ | 110/276 [40:57<56:59, 20.60s/it]
100%|█████████████████████████████████████████████| 2/2 [00:02<00:00, 1.20s/it]
{'loss': 0.5994, 'learning_rate': 0.0001369002878155519, 'epoch': 1.2}
{'loss': 0.5733, 'learning_rate': 0.00013580003867699799, 'epoch': 1.21}
{'loss': 0.5773, 'learning_rate': 0.00013469479592500953, 'epoch': 1.22}
{'loss': 0.526, 'learning_rate': 0.0001335847137257623, 'epoch': 1.23}
{'loss': 0.5612, 'learning_rate': 0.00013246994692046836, 'epoch': 1.24}
42%|█████████████████ | 115/276 [42:47<57:37, 21.47s/it]
0%| | 0/2 [00:00<?, ?it/s]
{'eval_loss': 0.6073997020721436, 'eval_runtime': 5.1793, 'eval_samples_per_second': 14.481, 'eval_steps_per_second': 0.386, 'epoch': 1.24}
42%|█████████████████ | 115/276 [42:52<57:37, 21.47s/it]
100%|█████████████████████████████████████████████| 2/2 [00:02<00:00, 1.20s/it]
{'loss': 0.5361, 'learning_rate': 0.00013135065100377814, 'epoch': 1.25}
{'loss': 0.5548, 'learning_rate': 0.00013022698210209067, 'epoch': 1.26}
{'loss': 0.5428, 'learning_rate': 0.00012909909695177647, 'epoch': 1.28}
{'loss': 0.5824, 'learning_rate': 0.0001279671528773146, 'epoch': 1.29}
{'loss': 0.5201, 'learning_rate': 0.0001268313077693485, 'epoch': 1.3}
43%|█████████████████▊ | 120/276 [44:31<53:55, 20.74s/it]
0%| | 0/2 [00:00<?, ?it/s]
{'eval_loss': 0.6084761023521423, 'eval_runtime': 5.1838, 'eval_samples_per_second': 14.468, 'eval_steps_per_second': 0.386, 'epoch': 1.3}
43%|█████████████████▊ | 120/276 [44:37<53:55, 20.74s/it]
100%|█████████████████████████████████████████████| 2/2 [00:02<00:00, 1.20s/it]
{'loss': 0.5541, 'learning_rate': 0.00012569172006266193, 'epoch': 1.31}
{'loss': 0.5522, 'learning_rate': 0.00012454854871407994, 'epoch': 1.32}
{'loss': 0.577, 'learning_rate': 0.00012340195318029623, 'epoch': 1.33}
{'loss': 0.5464, 'learning_rate': 0.00012225209339563145, 'epoch': 1.34}
{'loss': 0.5488, 'learning_rate': 0.00012109912974972424, 'epoch': 1.35}
45%|██████████████████▌ | 125/276 [46:28<56:17, 22.37s/it]
0%| | 0/2 [00:00<?, ?it/s]
{'eval_loss': 0.6019806861877441, 'eval_runtime': 5.1784, 'eval_samples_per_second': 14.483, 'eval_steps_per_second': 0.386, 'epoch': 1.35}
45%|██████████████████▌ | 125/276 [46:33<56:17, 22.37s/it]
100%|█████████████████████████████████████████████| 2/2 [00:02<00:00, 1.20s/it]
{'loss': 0.5779, 'learning_rate': 0.00011994322306515927, 'epoch': 1.36}
{'loss': 0.6335, 'learning_rate': 0.00011878453457503465, 'epoch': 1.37}
{'loss': 0.5218, 'learning_rate': 0.00011762322590047219, 'epoch': 1.38}
{'loss': 0.5204, 'learning_rate': 0.00011645945902807341, 'epoch': 1.39}
{'loss': 0.5375, 'learning_rate': 0.0001152933962873246, 'epoch': 1.41}
47%|███████████████████▎ | 130/276 [48:18<51:13, 21.05s/it]
0%| | 0/2 [00:00<?, ?it/s]
{'eval_loss': 0.5993748903274536, 'eval_runtime': 5.1818, 'eval_samples_per_second': 14.474, 'eval_steps_per_second': 0.386, 'epoch': 1.41}
47%|███████████████████▎ | 130/276 [48:24<51:13, 21.05s/it]
100%|█████████████████████████████████████████████| 2/2 [00:02<00:00, 1.20s/it]
{'loss': 0.5576, 'learning_rate': 0.0001141252003279542, 'epoch': 1.42}
{'loss': 0.5301, 'learning_rate': 0.00011295503409724526, 'epoch': 1.43}
{'loss': 0.5406, 'learning_rate': 0.00011178306081730665, 'epoch': 1.44}
{'loss': 0.5908, 'learning_rate': 0.00011060944396230581, 'epoch': 1.45}
{'loss': 0.5387, 'learning_rate': 0.00010943434723566623, 'epoch': 1.46}
49%|████████████████████ | 135/276 [50:20<51:59, 22.13s/it]
0%| | 0/2 [00:00<?, ?it/s]
{'eval_loss': 0.5990074276924133, 'eval_runtime': 5.1831, 'eval_samples_per_second': 14.47, 'eval_steps_per_second': 0.386, 'epoch': 1.46}
49%|████████████████████ | 135/276 [50:25<51:59, 22.13s/it]
100%|█████████████████████████████████████████████| 2/2 [00:02<00:00, 1.20s/it]
{'loss': 0.5454, 'learning_rate': 0.00010825793454723325, 'epoch': 1.47}
{'loss': 0.5242, 'learning_rate': 0.00010708036999041072, 'epoch': 1.48}
{'loss': 0.5235, 'learning_rate': 0.00010590181781927229, 'epoch': 1.49}
{'loss': 0.5508, 'learning_rate': 0.00010472244242565035, 'epoch': 1.5}
{'loss': 0.5198, 'learning_rate': 0.00010354240831620541, 'epoch': 1.51}
51%|████████████████████▊ | 140/276 [52:09<47:33, 20.99s/it]
0%| | 0/2 [00:00<?, ?it/s]
{'eval_loss': 0.5957627892494202, 'eval_runtime': 5.1865, 'eval_samples_per_second': 14.461, 'eval_steps_per_second': 0.386, 'epoch': 1.51}
51%|████████████████████▊ | 140/276 [52:14<47:33, 20.99s/it]
100%|█████████████████████████████████████████████| 2/2 [00:02<00:00, 1.21s/it]
{'loss': 0.5344, 'learning_rate': 0.0001023618800894798, 'epoch': 1.52}
{'loss': 0.5637, 'learning_rate': 0.00010118102241293848, 'epoch': 1.54}
{'loss': 0.5176, 'learning_rate': 0.0001, 'epoch': 1.55}
{'loss': 0.5638, 'learning_rate': 9.881897758706156e-05, 'epoch': 1.56}
{'loss': 0.5298, 'learning_rate': 9.76381199105202e-05, 'epoch': 1.57}
53%|█████████████████████▌ | 145/276 [54:08<48:31, 22.22s/it]
0%| | 0/2 [00:00<?, ?it/s]
{'eval_loss': 0.5926047563552856, 'eval_runtime': 5.1828, 'eval_samples_per_second': 14.471, 'eval_steps_per_second': 0.386, 'epoch': 1.57}
53%|█████████████████████▌ | 145/276 [54:13<48:31, 22.22s/it]
100%|█████████████████████████████████████████████| 2/2 [00:02<00:00, 1.20s/it]
{'loss': 0.5225, 'learning_rate': 9.645759168379463e-05, 'epoch': 1.58}
{'loss': 0.5217, 'learning_rate': 9.527755757434968e-05, 'epoch': 1.59}
{'loss': 0.5476, 'learning_rate': 9.409818218072773e-05, 'epoch': 1.6}
{'loss': 0.563, 'learning_rate': 9.291963000958932e-05, 'epoch': 1.61}
{'loss': 0.5469, 'learning_rate': 9.174206545276677e-05, 'epoch': 1.62}
54%|██████████████████████▎ | 150/276 [55:59<46:29, 22.14s/it]
0%| | 0/2 [00:00<?, ?it/s]
{'eval_loss': 0.5915570259094238, 'eval_runtime': 5.1847, 'eval_samples_per_second': 14.466, 'eval_steps_per_second': 0.386, 'epoch': 1.62}
54%|██████████████████████▎ | 150/276 [56:05<46:29, 22.14s/it]
100%|█████████████████████████████████████████████| 2/2 [00:02<00:00, 1.20s/it]
{'loss': 0.5368, 'learning_rate': 9.056565276433378e-05, 'epoch': 1.63}
{'loss': 0.5304, 'learning_rate': 8.939055603769421e-05, 'epoch': 1.64}
{'loss': 0.5338, 'learning_rate': 8.821693918269334e-05, 'epoch': 1.65}
{'loss': 0.5057, 'learning_rate': 8.704496590275478e-05, 'epoch': 1.66}
{'loss': 0.5384, 'learning_rate': 8.587479967204584e-05, 'epoch': 1.68}
56%|███████████████████████ | 155/276 [57:55<43:45, 21.70s/it]
0%| | 0/2 [00:00<?, ?it/s]
{'eval_loss': 0.590203046798706, 'eval_runtime': 5.1796, 'eval_samples_per_second': 14.48, 'eval_steps_per_second': 0.386, 'epoch': 1.68}
56%|███████████████████████ | 155/276 [58:01<43:45, 21.70s/it]
100%|█████████████████████████████████████████████| 2/2 [00:02<00:00, 1.20s/it]
{'loss': 0.5219, 'learning_rate': 8.47066037126754e-05, 'epoch': 1.69}
{'loss': 0.4666, 'learning_rate': 8.35405409719266e-05, 'epoch': 1.7}
57%|███████████████████████▎ | 157/276 [58:44<44:43, 22.55s/it]^C
In [ ]:
Below are ad hoc cells handling issues during training¶
Force release VRAM¶
In [24]:
# First interupt the kernel, wait a few seconds then run this to kill finetune to release VRAM
!ps aux|grep python|grep finetune|awk '{print $2}'|xargs kill
Clean the finetuned model and all checkpoints¶
In [ ]:
# Only run this to start over
!rm -rf ./qlora-out
Zip the prepared dataset¶
In [ ]:
!apt install zip
!zip -r last_run_prepared.zip -xi last_run_prepared
Monitoring GPU¶
In [ ]:
# Run this in a seperate terminal
!nvitop -m full
Fix DISK FULL¶
In [ ]:
%cd /
In [ ]:
!du -d 2 -h|grep G
In [31]:
!rm -rf /root/.local/share/Trash/
In [32]:
!rm -rf /root/.local/share/wandb/
In [33]:
!rm -rf /root/.cache/wandb/
Check who is using GPU¶
In [ ]:
!apt install lsof
In [ ]:
!lsof /dev/nvidia*
Upload checkpoints to HF¶
In [ ]:
!cp -r /workspace/axolotl/qlora-out/checkpoint-120 /workspace/llm-playground/storage/falcon-qlora/
In [ ]:
!python /workspace/llm-playground/helper/storage.py -u
Update axolotl¶
In [ ]:
%cd /workspace/
In [ ]:
!git clone https://github.com/OpenAccess-AI-Collective/axolotl axolotl-update
In [ ]:
!cp -r axolotl-update/* axolotl
In [ ]:
%cd /workspace/axolotl
In [ ]:
!git status
In [ ]:
!pip install -e .
Quick buttons (not working yet)¶