Finetuning falcon-7b with Axolotl

Dark theme llm-playground axolotl Runpod notebooks axolotl configs Storage Edit config

%cd /workspace/axolotl

!accelerate config --config_file configs/accelerate/default_config.yaml default

#1

!cat examples/falcon/ft.yml

!accelerate launch scripts/finetune.py examples/falcon/ft.yml

#2

!cat examples/falcon/ft.yml

!accelerate launch scripts/finetune.py examples/falcon/ft.yml

#3

!cat examples/falcon/ft.yml

!accelerate launch scripts/finetune.py examples/falcon/ft.yml

#4

!cat examples/falcon/ft.yml

!accelerate launch scripts/finetune.py examples/falcon/ft.yml

Below are ad hoc cells handling issues during training

Force release VRAM

# First interupt the kernel, wait a few seconds then run this to kill finetune to release VRAM
!ps aux|grep python|grep finetune|awk '{print $2}'|xargs kill

Clean the finetuned model and all checkpoints

# Only run this to start over
!rm -rf ./ft-out

Zip the prepared dataset

!apt install zip
!zip -r last_run_prepared.zip -xi last_run_prepared

Monitoring GPU

# Run this in a seperate terminal
!nvitop -m full

Fix DISK FULL

%cd /

!du -d 2 -h|grep G

!du -d 2 -h /root/.local

!rm -rf /root/.local/share/Trash/

!rm -rf /root/.local/share/wandb/

!rm -rf /root/.cache/wandb/

Check who is using GPU

!apt install lsof

!lsof /dev/nvidia*

Upload checkpoints to HF

!mkdir -p /workspace/llm-playground/storage/falcon-ft/

!cp -r /workspace/axolotl/ft-out/checkpoint-120 /workspace/llm-playground/storage/falcon-ft/

!python /workspace/llm-playground/helper/storage.py -u

Update axolotl

%cd /workspace/

!git clone https://github.com/OpenAccess-AI-Collective/axolotl axolotl-update

!cp -r axolotl-update/* axolotl

%cd /workspace/axolotl

!git status

!pip install -e .

A new bash without tmux etc.

!bash --norc --noprofile

Clean up all checkpoints but last one

cd /workspace/axolotl/ft-out/ && ls -lhta |grep checkpoint- | awk 'NR > 1 {print $9}' | xargs rm -rf