Open In Colab

In [1]:
!pip install datasets

Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/
Collecting datasets
  Downloading datasets-2.13.0-py3-none-any.whl (485 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 485.6/485.6 kB 8.7 MB/s eta 0:00:00
Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from datasets) (1.22.4)
Requirement already satisfied: pyarrow>=8.0.0 in /usr/local/lib/python3.10/dist-packages (from datasets) (9.0.0)
Collecting dill<0.3.7,>=0.3.0 (from datasets)
  Downloading dill-0.3.6-py3-none-any.whl (110 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 110.5/110.5 kB 12.3 MB/s eta 0:00:00
Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from datasets) (1.5.3)
Requirement already satisfied: requests>=2.19.0 in /usr/local/lib/python3.10/dist-packages (from datasets) (2.27.1)
Requirement already satisfied: tqdm>=4.62.1 in /usr/local/lib/python3.10/dist-packages (from datasets) (4.65.0)
Collecting xxhash (from datasets)
  Downloading xxhash-3.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (212 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 212.5/212.5 kB 20.5 MB/s eta 0:00:00
Collecting multiprocess (from datasets)
  Downloading multiprocess-0.70.14-py310-none-any.whl (134 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 134.3/134.3 kB 11.5 MB/s eta 0:00:00
Requirement already satisfied: fsspec[http]>=2021.11.1 in /usr/local/lib/python3.10/dist-packages (from datasets) (2023.4.0)
Collecting aiohttp (from datasets)
  Downloading aiohttp-3.8.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.0/1.0 MB 33.6 MB/s eta 0:00:00
Collecting huggingface-hub<1.0.0,>=0.11.0 (from datasets)
  Downloading huggingface_hub-0.15.1-py3-none-any.whl (236 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 236.8/236.8 kB 15.9 MB/s eta 0:00:00
Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from datasets) (23.1)
Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from datasets) (6.0)
Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (23.1.0)
Requirement already satisfied: charset-normalizer<4.0,>=2.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (2.0.12)
Collecting multidict<7.0,>=4.5 (from aiohttp->datasets)
  Downloading multidict-6.0.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (114 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 114.5/114.5 kB 11.7 MB/s eta 0:00:00
Collecting async-timeout<5.0,>=4.0.0a3 (from aiohttp->datasets)
  Downloading async_timeout-4.0.2-py3-none-any.whl (5.8 kB)
Collecting yarl<2.0,>=1.0 (from aiohttp->datasets)
  Downloading yarl-1.9.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (268 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 268.8/268.8 kB 23.4 MB/s eta 0:00:00
Collecting frozenlist>=1.1.1 (from aiohttp->datasets)
  Downloading frozenlist-1.3.3-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (149 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 149.6/149.6 kB 11.8 MB/s eta 0:00:00
Collecting aiosignal>=1.1.2 (from aiohttp->datasets)
  Downloading aiosignal-1.3.1-py3-none-any.whl (7.6 kB)
Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0.0,>=0.11.0->datasets) (3.12.0)
Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0.0,>=0.11.0->datasets) (4.5.0)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests>=2.19.0->datasets) (1.26.15)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests>=2.19.0->datasets) (2022.12.7)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests>=2.19.0->datasets) (3.4)
Requirement already satisfied: python-dateutil>=2.8.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets) (2022.7.1)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.8.1->pandas->datasets) (1.16.0)
Installing collected packages: xxhash, multidict, frozenlist, dill, async-timeout, yarl, multiprocess, huggingface-hub, aiosignal, aiohttp, datasets
Successfully installed aiohttp-3.8.4 aiosignal-1.3.1 async-timeout-4.0.2 datasets-2.13.0 dill-0.3.6 frozenlist-1.3.3 huggingface-hub-0.15.1 multidict-6.0.4 multiprocess-0.70.14 xxhash-3.2.0 yarl-1.9.2
In [49]:
!git clone https://github.com/utensil/llm-playground

Cloning into 'llm-playground'...
remote: Enumerating objects: 1015, done.
remote: Counting objects: 100% (171/171), done.
remote: Compressing objects: 100% (170/170), done.
remote: Total 1015 (delta 117), reused 0 (delta 0), pack-reused 844
Receiving objects: 100% (1015/1015), 6.00 MiB | 20.97 MiB/s, done.
Resolving deltas: 100% (619/619), done.
In [50]:
%cd llm-playground

/content/axolotl/llm-playground
In [51]:
!python helper/download-dataset.py winglian/evals

Working directory changed to: /content/axolotl/llm-playground/helper/..
https://huggingface.co/api/datasets/winglian/evals/tree/main
b'[{"type":"directory","oid":"455a74625bb0d3b98b18e3b3f48a3a2fa2740eb4","size":0,"path":"custom"},{"type":"directory","oid":"831547ccd85eb2c166f39fe44de7a28dd0cf5cf4","size":0,"path":"hellaswag"},{"type":"directory","oid":"5941a5a7c4da62beb9ea4ed542aad9b1228dbed4","size":0,"path":"hf"},{"type":"directory","oid":"30060f75278460df5dd72aabc2da596272e7dfa0","size":0,"path":"mmlu"},{"type":"directory","oid":"e137e975f4c847cfe3a09baaafdb91bc0834200b","size":0,"path":"openai"},{"type":"file","oid":"4e8bc8604fef9c578fca4348c0b790242d43bde6","size":2475,"path":".gitattributes"},{"type":"file","oid":"898e6749676e6c8d4af62aa732f1a5deeeae9ccd","size":259,"path":"README.md"}]'
Gathering: custom
https://huggingface.co/api/datasets/winglian/evals/tree/main/custom
b'[{"type":"file","oid":"1017f5ae02a8341673b5dac66b8e819f2931f08e","size":624453,"path":"custom/classify-self-chat.jsonl"},{"type":"file","oid":"acfb41ed3b22a1d017ec09305dacc7d33243c327","size":625783,"path":"custom/classify-self-chat.sharegpt.jsonl"},{"type":"file","oid":"c95b8c1fba8deecfe4ce4fd68dcc0384318f1772","size":7848567,"path":"custom/coding-self-chat.jsonl"},{"type":"file","oid":"8720ba5bf54d5a62f6a5945ba000f45864a76a81","size":7868365,"path":"custom/coding-self-chat.sharegpt.jsonl"},{"type":"file","oid":"d6b82b83b168eeacc65a1bcbf90087151381eaba","size":79852,"path":"custom/context_insensitivity.jsonl"},{"type":"file","oid":"c2a2ed8cf512bde88f7da84fbb40a86092114465","size":7979073,"path":"custom/creative-w-critique.cleaned.jsonl"},{"type":"file","oid":"4fb0e6bdf6ad3743d8a03e96bd41c50d8142ad59","size":5296937,"path":"custom/guanaco-cleaned.en.jsonl"},{"type":"file","oid":"1b28d81e62dc226a16514e499176c9bfc2e688b6","size":1842270,"path":"custom/in_context_qa.jsonl"},{"type":"file","oid":"6d66653f6487fcabde372174fa6de1998e37e195","size":1582714,"path":"custom/jokes_explained_500up.jsonl"},{"type":"file","oid":"28c67166dbcc0d21ed867c6999bf45fe3578faf2","size":22694681,"lfs":{"oid":"7477da5a64eafe449fd7989389ea401d45a5864efc7f75c39a9765cbd1acb379","size":22694681,"pointerSize":133},"path":"custom/logic_inference_oa.jsonl"},{"type":"file","oid":"6b85304b539d8e0ddbaa38ddacb708b7e52eb89f","size":3094572,"path":"custom/long_in_context_qa.jsonl"},{"type":"file","oid":"373f2886cea8197464838791249f5e2834a7d1b7","size":394601,"path":"custom/misconceptions.jsonl"},{"type":"file","oid":"723f8b2a50ffb127c425544a0fac8d7883b450e8","size":171981,"path":"custom/n_task.jsonl"},{"type":"file","oid":"dd669ad212d9ea98d31b416af0c3feae5e5f8e9f","size":1116596,"path":"custom/prose-gpt4.jsonl"},{"type":"file","oid":"75c92971d4cf065c363c4f62d77d90efc33b4702","size":1120643,"path":"custom/prose-gpt4.sharegpt.jsonl"},{"type":"file","oid":"ccb103c376e246e3289b2dfc674717cce2d9d02a","size":1127244,"path":"custom/prose-rewrite-gpt4.jsonl"},{"type":"file","oid":"705a1a7eca67183f79aee84224e73e3384234176","size":1129695,"path":"custom/prose-rewrite-gpt4.sharegpt.jsonl"},{"type":"file","oid":"3463830faa6f720d1b4662edfae1c5a639306cfc","size":68778,"path":"custom/prose.jsonl"}]'
https://huggingface.co/api/datasets/winglian/evals/tree/main/custom?cursor=ZXlKbWFXeGxYMjVoYldVaU9pSmpkWE4wYjIwdmNISnZjMlV1YW5OdmJtd2lmUT09OjUw
b'{"error":"Tree Entry not found: custom"}'
Gathering: hellaswag
https://huggingface.co/api/datasets/winglian/evals/tree/main/hellaswag
b'[{"type":"file","oid":"9bd3a28289746d37302482122181df7f8558e99f","size":4198676,"path":"hellaswag/hellaswag-concise.jsonl"},{"type":"file","oid":"399a47913d1f77c55629390e27552169854ac0be","size":90564564,"lfs":{"oid":"4bf203666389ec638475b9fad5ad7809bc8fafe03c3d928ccb5906f0312f47eb","size":90564564,"pointerSize":133},"path":"hellaswag/hellaswag-v2.jsonl"},{"type":"file","oid":"a0942e76f52d5933cdb5026f09ef0fa177975de6","size":73133306,"path":"hellaswag/hellaswag.jsonl"}]'
https://huggingface.co/api/datasets/winglian/evals/tree/main/hellaswag?cursor=ZXlKbWFXeGxYMjVoYldVaU9pSm9aV3hzWVhOM1lXY3ZhR1ZzYkdGemQyRm5MbXB6YjI1c0luMD06NTA%3D
b'{"error":"Tree Entry not found: hellaswag"}'
Gathering: hf
https://huggingface.co/api/datasets/winglian/evals/tree/main/hf
b'[{"type":"file","oid":"55c0ba3bbd08831b853aa01ce972866e2777b140","size":2049206,"path":"hf/ARC-Challenge.jsonl"},{"type":"file","oid":"d02449bcd756f7bfad179d08a37021239befc97b","size":3222016,"path":"hf/ARC-Easy.jsonl"},{"type":"file","oid":"de4f357a8a64a0888f3f2626767c565b2a98ff6f","size":7790845,"path":"hf/gsm8k.jsonl"},{"type":"file","oid":"66b17d565c2892231a0b9d219b76d931b776a3f5","size":9051949,"path":"hf/piqa.jsonl"},{"type":"file","oid":"fb5086cc634a113ce27f1f768ed570a63c928cf8","size":3938149,"path":"hf/riddle_sense.jsonl"},{"type":"file","oid":"27c6a1dc6fb1270766da11f39d60b677be93542e","size":5894135,"path":"hf/winogrande.jsonl"}]'
https://huggingface.co/api/datasets/winglian/evals/tree/main/hf?cursor=ZXlKbWFXeGxYMjVoYldVaU9pSm9aaTkzYVc1dlozSmhibVJsTG1wemIyNXNJbjA9OjUw
b'{"error":"Tree Entry not found: hf"}'
Gathering: mmlu
https://huggingface.co/api/datasets/winglian/evals/tree/main/mmlu
b'[{"type":"file","oid":"75fd4e37f64f3155fe82573d88fc965b4f9d5749","size":141260,"path":"mmlu/abstract_algebra.jsonl"},{"type":"file","oid":"a4de859fb1f6e673616c527e504c90f481e096b7","size":312961,"path":"mmlu/conceptual_physics.jsonl"},{"type":"file","oid":"ee1a9fe43e779d548458deed6dae5131f19ff072","size":216604,"path":"mmlu/formal_logic.jsonl"},{"type":"file","oid":"3501ecfb6db44c0c3e0d9e1f00259453cc84bb91","size":284766,"path":"mmlu/high_school_physics.jsonl"},{"type":"file","oid":"5206e01eaee0c6b67580d5c11ef1858c74a1ba45","size":263345,"path":"mmlu/logical_fallacies.jsonl"}]'
https://huggingface.co/api/datasets/winglian/evals/tree/main/mmlu?cursor=ZXlKbWFXeGxYMjVoYldVaU9pSnRiV3gxTDJ4dloybGpZV3hmWm1Gc2JHRmphV1Z6TG1wemIyNXNJbjA9OjUw
b'{"error":"Tree Entry not found: mmlu"}'
Gathering: openai
https://huggingface.co/api/datasets/winglian/evals/tree/main/openai
b'[{"type":"file","oid":"d08e76a7808048bd640f2e28f9d79a47eb3696bb","size":2233732,"path":"openai/tldr.jsonl"}]'
https://huggingface.co/api/datasets/winglian/evals/tree/main/openai?cursor=ZXlKbWFXeGxYMjVoYldVaU9pSnZjR1Z1WVdrdmRHeGtjaTVxYzI5dWJDSjk6NTA%3D
b'{"error":"Tree Entry not found: openai"}'
https://huggingface.co/api/datasets/winglian/evals/tree/main?cursor=ZXlKbWFXeGxYMjVoYldVaU9pSlNSVUZFVFVVdWJXUWlmUT09OjUw
b'[]'
Downloading the dataset to datasets/winglian_evals
  0% 0/35 [00:00<?, ?it/s]
  0% 0.00/624k [00:00<?, ?iB/s, File: custom/classify-self-chat.jsonl]

  0% 0.00/79.9k [00:00<?, ?iB/s, File: custom/context_insensitivity.jsonl]


100% 79.9k/79.9k [00:00<00:00, 3.49MiB/s, File: custom/context_insensitivity.jsonl]


  0% 0.00/1.84M [00:00<?, ?iB/s, File: custom/in_context_qa.jsonl]
100% 624k/624k [00:00<00:00, 6.23MiB/s, File: custom/classify-self-chat.jsonl]


100% 626k/626k [00:00<00:00, 6.06MiB/s, File: custom/classify-self-chat.sharegpt.jsonl]
100% 624k/624k [00:00<00:00, 5.52MiB/s, File: custom/classify-self-chat.jsonl]
  3% 1/35 [00:00<00:08,  4.06it/s]
  0% 0.00/7.98M [00:00<?, ?iB/s, File: custom/creative-w-critique.cleaned.jsonl]


  0% 0.00/5.30M [00:00<?, ?iB/s, File: custom/guanaco-cleaned.en.jsonl]

 40% 729k/1.84M [00:00<00:00, 7.17MiB/s, File: custom/in_context_qa.jsonl]



100% 1.84M/1.84M [00:00<00:00, 11.5MiB/s, File: custom/in_context_qa.jsonl]


  0% 0.00/7.85M [00:00<?, ?iB/s, File: custom/coding-self-chat.jsonl]




  0% 0.00/1.58M [00:00<?, ?iB/s, File: custom/jokes_explained_500up.jsonl]
  7% 548k/7.98M [00:00<00:01, 5.48MiB/s, File: custom/creative-w-critique.cleaned.jsonl]


  9% 452k/5.30M [00:00<00:01, 4.49MiB/s, File: custom/guanaco-cleaned.en.jsonl]



  8% 664k/7.87M [00:00<00:01, 6.64MiB/s, File: custom/coding-self-chat.sharegpt.jsonl]

  5% 378k/7.85M [00:00<00:01, 3.78MiB/s, File: custom/coding-self-chat.jsonl]




 33% 524k/1.58M [00:00<00:00, 4.62MiB/s, File: custom/jokes_explained_500up.jsonl]





  0% 0.00/3.09M [00:00<?, ?iB/s, File: custom/long_in_context_qa.jsonl]


 25% 1.30M/5.30M [00:00<00:00, 6.76MiB/s, File: custom/guanaco-cleaned.en.jsonl]
 16% 1.28M/7.98M [00:00<00:01, 6.23MiB/s, File: custom/creative-w-critique.cleaned.jsonl]



 17% 1.33M/7.87M [00:00<00:01, 5.94MiB/s, File: custom/coding-self-chat.sharegpt.jsonl]

 13% 1.05M/7.85M [00:00<00:01, 5.52MiB/s, File: custom/coding-self-chat.jsonl]






  0% 0.00/395k [00:00<?, ?iB/s, File: custom/misconceptions.jsonl]





 17% 513k/3.09M [00:00<00:00, 5.13MiB/s, File: custom/long_in_context_qa.jsonl]




 88% 1.39M/1.58M [00:00<00:00, 6.06MiB/s, File: custom/jokes_explained_500up.jsonl]


 39% 2.05M/5.30M [00:00<00:00, 6.76MiB/s, File: custom/guanaco-cleaned.en.jsonl]
100% 1.58M/1.58M [00:00<00:00, 5.99MiB/s, File: custom/jokes_explained_500up.jsonl]




100% 395k/395k [00:00<00:00, 4.91MiB/s, File: custom/misconceptions.jsonl]


 20% 1.60M/7.85M [00:00<00:01, 5.24MiB/s, File: custom/coding-self-chat.jsonl]





 43% 1.33M/3.09M [00:00<00:00, 6.94MiB/s, File: custom/long_in_context_qa.jsonl]


 58% 3.09M/5.30M [00:00<00:00, 7.52MiB/s, File: custom/guanaco-cleaned.en.jsonl]
 39% 3.10M/7.98M [00:00<00:00, 7.32MiB/s, File: custom/creative-w-critique.cleaned.jsonl]



 35% 2.79M/7.87M [00:00<00:00, 6.86MiB/s, File: custom/coding-self-chat.sharegpt.jsonl]

 32% 2.48M/7.85M [00:00<00:00, 6.31MiB/s, File: custom/coding-self-chat.jsonl]





 74% 2.29M/3.09M [00:00<00:00, 8.15MiB/s, File: custom/long_in_context_qa.jsonl]


 78% 4.14M/5.30M [00:00<00:00, 8.51MiB/s, File: custom/guanaco-cleaned.en.jsonl]
 52% 4.13M/7.98M [00:00<00:00, 8.30MiB/s, File: custom/creative-w-critique.cleaned.jsonl]



 48% 3.77M/7.87M [00:00<00:00, 7.68MiB/s, File: custom/coding-self-chat.sharegpt.jsonl]




100% 3.09M/3.09M [00:00<00:00, 7.87MiB/s, File: custom/long_in_context_qa.jsonl]


100% 172k/172k [00:00<00:00, 2.83MiB/s, File: custom/n_task.jsonl]



 94% 4.99M/5.30M [00:00<00:00, 8.43MiB/s, File: custom/guanaco-cleaned.en.jsonl]




  0% 0.00/1.12M [00:00<?, ?iB/s, File: custom/prose-gpt4.jsonl]



 62% 4.88M/7.87M [00:00<00:00, 8.79MiB/s, File: custom/coding-self-chat.sharegpt.jsonl]
100% 5.30M/5.30M [00:00<00:00, 7.78MiB/s, File: custom/guanaco-cleaned.en.jsonl]


 61% 4.76M/7.85M [00:00<00:00, 8.86MiB/s, File: custom/coding-self-chat.jsonl]


  0% 0.00/22.7M [00:00<?, ?iB/s, File: custom/logic_inference_oa.jsonl]




 37% 411k/1.12M [00:00<00:00, 3.77MiB/s, File: custom/prose-gpt4.jsonl]



 79% 6.18M/7.87M [00:00<00:00, 9.86MiB/s, File: custom/coding-self-chat.sharegpt.jsonl]
100% 1.12M/1.12M [00:00<00:00, 7.89MiB/s, File: custom/prose-gpt4.jsonl]





  0% 0.00/1.12M [00:00<?, ?iB/s, File: custom/prose-gpt4.sharegpt.jsonl]

 73% 5.74M/7.85M [00:00<00:00, 9.10MiB/s, File: custom/coding-self-chat.jsonl]


  2% 424k/22.7M [00:00<00:05, 4.24MiB/s, File: custom/logic_inference_oa.jsonl]



 97% 7.60M/7.87M [00:00<00:00, 11.2MiB/s, File: custom/coding-self-chat.sharegpt.jsonl]
 93% 7.42M/7.98M [00:00<00:00, 9.54MiB/s, File: custom/creative-w-critique.cleaned.jsonl]





  0% 0.00/1.13M [00:00<?, ?iB/s, File: custom/prose-rewrite-gpt4.jsonl]






  0% 0.00/1.13M [00:00<?, ?iB/s, File: custom/prose-rewrite-gpt4.sharegpt.jsonl]




 43% 483k/1.12M [00:00<00:00, 4.83MiB/s, File: custom/prose-gpt4.sharegpt.jsonl]

100% 7.87M/7.87M [00:00<00:00, 8.71MiB/s, File: custom/coding-self-chat.sharegpt.jsonl]
100% 7.98M/7.98M [00:00<00:00, 8.42MiB/s, File: custom/creative-w-critique.cleaned.jsonl]

  0% 0.00/68.8k [00:00<?, ?iB/s, File: custom/prose.jsonl]


100% 68.8k/68.8k [00:00<00:00, 3.43MiB/s, File: custom/prose.jsonl]






100% 1.12M/1.12M [00:00<00:00, 6.05MiB/s, File: custom/prose-gpt4.sharegpt.jsonl]







 40% 453k/1.13M [00:00<00:00, 4.31MiB/s, File: custom/prose-rewrite-gpt4.sharegpt.jsonl]

100% 1.13M/1.13M [00:00<00:00, 7.17MiB/s, File: custom/prose-rewrite-gpt4.jsonl]
100% 7.85M/7.85M [00:00<00:00, 8.12MiB/s, File: custom/coding-self-chat.jsonl]
  9% 3/35 [00:01<00:14,  2.15it/s]


100% 1.13M/1.13M [00:00<00:00, 6.27MiB/s, File: custom/prose-rewrite-gpt4.sharegpt.jsonl]



 34% 7.65M/22.7M [00:00<00:00, 26.4MiB/s, File: custom/logic_inference_oa.jsonl]
  0% 0.00/2.05M [00:00<?, ?iB/s, File: hf/ARC-Challenge.jsonl]

  0% 0.00/4.20M [00:00<?, ?iB/s, File: hellaswag/hellaswag-concise.jsonl]


 68% 15.4M/22.7M [00:00<00:00, 44.6MiB/s, File: custom/logic_inference_oa.jsonl]
 51% 1.05M/2.05M [00:00<00:00, 10.5MiB/s, File: hf/ARC-Challenge.jsonl]



  0% 0.00/7.79M [00:00<?, ?iB/s, File: hf/gsm8k.jsonl]




100% 2.05M/2.05M [00:00<00:00, 13.7MiB/s, File: hf/ARC-Challenge.jsonl]


 15% 633k/4.20M [00:00<00:00, 6.24MiB/s, File: hellaswag/hellaswag-concise.jsonl]



  9% 670k/7.79M [00:00<00:01, 6.16MiB/s, File: hf/gsm8k.jsonl]




 15% 475k/3.22M [00:00<00:00, 4.75MiB/s, File: hf/ARC-Easy.jsonl]
  0% 0.00/9.05M [00:00<?, ?iB/s, File: hf/piqa.jsonl]

 41% 1.74M/4.20M [00:00<00:00, 9.07MiB/s, File: hellaswag/hellaswag-concise.jsonl]





  0% 0.00/90.6M [00:00<?, ?iB/s, File: hellaswag/hellaswag-v2.jsonl]



 26% 2.05M/7.79M [00:00<00:00, 10.5MiB/s, File: hf/gsm8k.jsonl]




 62% 1.99M/3.22M [00:00<00:00, 10.7MiB/s, File: hf/ARC-Easy.jsonl]
  7% 629k/9.05M [00:00<00:01, 6.19MiB/s, File: hf/piqa.jsonl]






  0% 0.00/3.94M [00:00<?, ?iB/s, File: hf/riddle_sense.jsonl]

 74% 3.09M/4.20M [00:00<00:00, 10.7MiB/s, File: hellaswag/hellaswag-concise.jsonl]





  0% 298k/90.6M [00:00<00:30, 2.95MiB/s, File: hellaswag/hellaswag-v2.jsonl]
 15% 1.33M/9.05M [00:00<00:01, 6.65MiB/s, File: hf/piqa.jsonl]






 13% 526k/3.94M [00:00<00:00, 5.26MiB/s, File: hf/riddle_sense.jsonl]




 95% 3.05M/3.22M [00:00<00:00, 8.91MiB/s, File: hf/ARC-Easy.jsonl]



100% 3.22M/3.22M [00:00<00:00, 8.61MiB/s, File: hf/ARC-Easy.jsonl]






  1% 1.01M/90.6M [00:00<00:16, 5.31MiB/s, File: hellaswag/hellaswag-v2.jsonl]

100% 4.20M/4.20M [00:00<00:00, 8.87MiB/s, File: hellaswag/hellaswag-concise.jsonl]



 87% 19.8M/22.7M [00:00<00:00, 20.3MiB/s, File: custom/logic_inference_oa.jsonl]
 22% 2.00M/9.05M [00:00<00:01, 6.48MiB/s, File: hf/piqa.jsonl]






 33% 1.31M/3.94M [00:00<00:00, 6.56MiB/s, File: hf/riddle_sense.jsonl]



 52% 4.02M/7.79M [00:00<00:00, 8.58MiB/s, File: hf/gsm8k.jsonl]





  2% 1.91M/90.6M [00:00<00:12, 6.97MiB/s, File: hellaswag/hellaswag-v2.jsonl]
 37% 3.36M/9.05M [00:00<00:00, 9.22MiB/s, File: hf/piqa.jsonl]

  0% 0.00/141k [00:00<?, ?iB/s, File: mmlu/abstract_algebra.jsonl]






100% 141k/141k [00:00<00:00, 3.07MiB/s, File: mmlu/abstract_algebra.jsonl]






  3% 3.11M/90.6M [00:00<00:09, 8.87MiB/s, File: hellaswag/hellaswag-v2.jsonl]



 63% 4.92M/7.79M [00:00<00:00, 8.06MiB/s, File: hf/gsm8k.jsonl]

  0% 0.00/5.89M [00:00<?, ?iB/s, File: hf/winogrande.jsonl]
 47% 4.28M/9.05M [00:00<00:00, 8.95MiB/s, File: hf/piqa.jsonl]






 78% 3.06M/3.94M [00:00<00:00, 7.50MiB/s, File: hf/riddle_sense.jsonl]





100% 22.7M/22.7M [00:01<00:00, 18.3MiB/s, File: custom/logic_inference_oa.jsonl]




 29% 10/35 [00:02<00:05,  4.82it/s]


  0% 0.00/313k [00:00<?, ?iB/s, File: mmlu/conceptual_physics.jsonl]

  9% 559k/5.89M [00:00<00:00, 5.59MiB/s, File: hf/winogrande.jsonl]
 57% 5.19M/9.05M [00:00<00:00, 8.53MiB/s, File: hf/piqa.jsonl]






100% 3.94M/3.94M [00:00<00:00, 7.49MiB/s, File: hf/riddle_sense.jsonl]
100% 313k/313k [00:00<00:00, 4.56MiB/s, File: mmlu/conceptual_physics.jsonl]




 84% 6.53M/7.79M [00:00<00:00, 7.74MiB/s, File: hf/gsm8k.jsonl]





  5% 4.96M/90.6M [00:00<00:10, 7.98MiB/s, File: hellaswag/hellaswag-v2.jsonl]

 26% 1.53M/5.89M [00:00<00:00, 8.01MiB/s, File: hf/winogrande.jsonl]


  0% 0.00/217k [00:00<?, ?iB/s, File: mmlu/formal_logic.jsonl]
100% 217k/217k [00:00<00:00, 3.28MiB/s, File: mmlu/formal_logic.jsonl]



  0% 0.00/285k [00:00<?, ?iB/s, File: mmlu/high_school_physics.jsonl]



 97% 7.57M/7.79M [00:00<00:00, 7.97MiB/s, File: hf/gsm8k.jsonl]




  0% 0.00/263k [00:00<?, ?iB/s, File: mmlu/logical_fallacies.jsonl]

 40% 2.33M/5.89M [00:00<00:00, 7.52MiB/s, File: hf/winogrande.jsonl]





100% 7.79M/7.79M [00:00<00:00, 7.93MiB/s, File: hf/gsm8k.jsonl]
100% 285k/285k [00:00<00:00, 4.19MiB/s, File: mmlu/high_school_physics.jsonl]

100% 263k/263k [00:00<00:00, 4.22MiB/s, File: mmlu/logical_fallacies.jsonl]


 60% 3.52M/5.89M [00:00<00:00, 9.15MiB/s, File: hf/winogrande.jsonl]





  7% 6.54M/90.6M [00:00<00:11, 7.33MiB/s, File: hellaswag/hellaswag-v2.jsonl]
 97% 8.80M/9.05M [00:00<00:00, 10.4MiB/s, File: hf/piqa.jsonl]


100% 2.48k/2.48k [00:00<00:00, 1.64MiB/s, File: .gitattributes]



100% 259/259 [00:00<00:00, 185kiB/s, File: README.md]
100% 9.05M/9.05M [00:01<00:00, 9.00MiB/s, File: hf/piqa.jsonl]

  0% 0.00/2.23M [00:00<?, ?iB/s, File: openai/tldr.jsonl]

 86% 5.06M/5.89M [00:00<00:00, 11.3MiB/s, File: hf/winogrande.jsonl]





100% 5.89M/5.89M [00:00<00:00, 10.8MiB/s, File: hf/winogrande.jsonl]

 31% 682k/2.23M [00:00<00:00, 6.82MiB/s, File: openai/tldr.jsonl]





100% 2.23M/2.23M [00:00<00:00, 14.6MiB/s, File: openai/tldr.jsonl]






 18% 15.9M/90.6M [00:01<00:03, 24.1MiB/s, File: hellaswag/hellaswag-v2.jsonl]





 20% 18.4M/90.6M [00:01<00:03, 23.7MiB/s, File: hellaswag/hellaswag-v2.jsonl]





 25% 22.8M/90.6M [00:01<00:02, 29.5MiB/s, File: hellaswag/hellaswag-v2.jsonl]





 33% 30.0M/90.6M [00:01<00:01, 41.9MiB/s, File: hellaswag/hellaswag-v2.jsonl]





 41% 37.4M/90.6M [00:01<00:01, 51.3MiB/s, File: hellaswag/hellaswag-v2.jsonl]





 49% 43.9M/90.6M [00:01<00:00, 55.5MiB/s, File: hellaswag/hellaswag-v2.jsonl]





 57% 51.2M/90.6M [00:01<00:00, 60.7MiB/s, File: hellaswag/hellaswag-v2.jsonl]





 64% 58.0M/90.6M [00:01<00:00, 62.7MiB/s, File: hellaswag/hellaswag-v2.jsonl]





 72% 65.0M/90.6M [00:01<00:00, 65.0MiB/s, File: hellaswag/hellaswag-v2.jsonl]





 81% 73.6M/90.6M [00:02<00:00, 71.4MiB/s, File: hellaswag/hellaswag-v2.jsonl]





 90% 81.6M/90.6M [00:02<00:00, 73.7MiB/s, File: hellaswag/hellaswag-v2.jsonl]





100% 90.6M/90.6M [00:02<00:00, 39.4MiB/s, File: hellaswag/hellaswag-v2.jsonl]
 57% 20/35 [00:04<00:02,  5.30it/s]
  0% 0.00/73.1M [00:00<?, ?iB/s, File: hellaswag/hellaswag.jsonl]
  1% 671k/73.1M [00:00<00:10, 6.65MiB/s, File: hellaswag/hellaswag.jsonl]
 10% 7.05M/73.1M [00:00<00:03, 18.3MiB/s, File: hellaswag/hellaswag.jsonl]
 13% 9.41M/73.1M [00:00<00:04, 15.5MiB/s, File: hellaswag/hellaswag.jsonl]
 16% 11.8M/73.1M [00:01<00:05, 10.2MiB/s, File: hellaswag/hellaswag.jsonl]
 19% 14.1M/73.1M [00:01<00:06, 9.13MiB/s, File: hellaswag/hellaswag.jsonl]
 24% 17.8M/73.1M [00:01<00:04, 13.3MiB/s, File: hellaswag/hellaswag.jsonl]
 27% 19.6M/73.1M [00:01<00:03, 14.2MiB/s, File: hellaswag/hellaswag.jsonl]
 38% 27.9M/73.1M [00:01<00:01, 27.9MiB/s, File: hellaswag/hellaswag.jsonl]
 44% 32.5M/73.1M [00:01<00:01, 31.8MiB/s, File: hellaswag/hellaswag.jsonl]
 57% 41.6M/73.1M [00:01<00:00, 46.2MiB/s, File: hellaswag/hellaswag.jsonl]
 68% 50.0M/73.1M [00:01<00:00, 55.8MiB/s, File: hellaswag/hellaswag.jsonl]
 81% 59.0M/73.1M [00:02<00:00, 64.9MiB/s, File: hellaswag/hellaswag.jsonl]
100% 73.1M/73.1M [00:02<00:00, 33.5MiB/s, File: hellaswag/hellaswag.jsonl]
100% 35/35 [00:06<00:00,  5.05it/s]
In [8]:
from datasets import load_dataset, get_dataset_split_names
In [9]:
get_dataset_split_names("./datasets/winglian_evals/hellaswag")
Out[9]:
['train']
In [10]:
ds = load_dataset("./datasets/winglian_evals/hellaswag")

Downloading and preparing dataset json/hellaswag to /root/.cache/huggingface/datasets/json/hellaswag-18236f2717ecccc6/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4...

Downloading data files:   0%|          | 0/1 [00:00<?, ?it/s]
Extracting data files:   0%|          | 0/1 [00:00<?, ?it/s]
Generating train split: 0 examples [00:00, ? examples/s]
Dataset json downloaded and prepared to /root/.cache/huggingface/datasets/json/hellaswag-18236f2717ecccc6/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4. Subsequent calls will reuse this data.

  0%|          | 0/1 [00:00<?, ?it/s]
In [13]:
ds
Out[13]:
DatasetDict({
    train: Dataset({
        features: ['question', 'choices', 'answer', 'explanation'],
        num_rows: 84911
    })
})
In [14]:
ds.shape
Out[14]:
{'train': (84911, 4)}
In [19]:
ds.data['train'].to_pandas().head()
Out[19]:
question choices answer explanation
0 Then, the man writes over the snow covering th... [, the man adds wax to the windshield and cuts... 3 The correct choice is ", the man continues rem...
1 He scrubs in the shampoo and then washes it of... [then rinses it off in the sink., lets his hai... 2 The correct choice is "uses an electric clippe...
2 A female chef in white uniform shows a stack o... [contain egg yolks and baking soda., are then ... 3 The correct choice is "are filled with pastrie...
3 He then combs it and blow dries his hair after... [places six screws on the front of the hair th... 1 The correct choice is "also trims the back and...
4 The man in the blue shirt sits on the chair ne... [drops the water faucet in the sink., then wal... 3 The correct choice is "scrubs in the shampoo a...
In [23]:
ds_by_dir = load_dataset("./datasets/winglian_evals", data_dir='hellaswag')

WARNING:datasets.builder:Found cached dataset json (/root/.cache/huggingface/datasets/json/winglian_evals-6e111d1b7c4f3a89/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4)

  0%|          | 0/1 [00:00<?, ?it/s]
In [25]:
ds_by_dir.shape
Out[25]:
{'train': (84911, 4)}
In [24]:
ds_by_dir.data['train'].to_pandas().head()
Out[24]:
question choices answer explanation
0 Then, the man writes over the snow covering th... [, the man adds wax to the windshield and cuts... 3 The correct choice is ", the man continues rem...
1 He scrubs in the shampoo and then washes it of... [then rinses it off in the sink., lets his hai... 2 The correct choice is "uses an electric clippe...
2 A female chef in white uniform shows a stack o... [contain egg yolks and baking soda., are then ... 3 The correct choice is "are filled with pastrie...
3 He then combs it and blow dries his hair after... [places six screws on the front of the hair th... 1 The correct choice is "also trims the back and...
4 The man in the blue shirt sits on the chair ne... [drops the water faucet in the sink., then wal... 3 The correct choice is "scrubs in the shampoo a...
In [28]:
ds_by_file = load_dataset("./datasets/winglian_evals/hellaswag",  data_files=['hellaswag.jsonl', 'hellaswag-v2.jsonl', 'hellaswag-concise.jsonl'])

Downloading and preparing dataset json/hellaswag to /root/.cache/huggingface/datasets/json/hellaswag-ad1562765da2a8e7/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4...

Downloading data files:   0%|          | 0/1 [00:00<?, ?it/s]
Extracting data files:   0%|          | 0/1 [00:00<?, ?it/s]
Generating train split: 0 examples [00:00, ? examples/s]
Dataset json downloaded and prepared to /root/.cache/huggingface/datasets/json/hellaswag-ad1562765da2a8e7/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4. Subsequent calls will reuse this data.

  0%|          | 0/1 [00:00<?, ?it/s]
In [29]:
ds_by_file.shape
Out[29]:
{'train': (84911, 4)}
In [30]:
ds_by_file.data['train'].to_pandas().head()
Out[30]:
question choices answer explanation
0 He also trims the back and sides of his head w... [poses with how to complete the look using the... 2 The correct choice is "uses some hair gel to s...
1 The man in the blue shirt sits on the chair ne... [drops the water faucet in the sink., then wal... 3 The correct choice is "scrubs in the shampoo a...
2 He uses scissors to trim the hair and give it ... [lifts his look off the camera., comes on the ... 3 The correct choice is "then explains how the h...
3 He uses an electric clipper to groom the sideb... [then picks up some lipstick on the table to a... 1 The correct choice is "uses scissors to trim t...
4 He then combs it and blow dries his hair after... [places six screws on the front of the hair th... 1 The correct choice is "also trims the back and...
In [31]:
%cd /content

/content
In [33]:
!git clone -b local_dataset https://github.com/utensil/axolotl.git

Cloning into 'axolotl'...
remote: Enumerating objects: 3141, done.
remote: Counting objects: 100% (1459/1459), done.
remote: Compressing objects: 100% (375/375), done.
remote: Total 3141 (delta 1175), reused 1240 (delta 1027), pack-reused 1682
Receiving objects: 100% (3141/3141), 1.43 MiB | 17.43 MiB/s, done.
Resolving deltas: 100% (1966/1966), done.
In [63]:
%cd /content/axolotl

/content/axolotl
In [64]:
!pip install -e .

Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/
Obtaining file:///content/axolotl
  Preparing metadata (setup.py) ... done
Collecting transformers@ git+https://github.com/huggingface/transformers.git (from axolotl==0.1)
  Cloning https://github.com/huggingface/transformers.git to /tmp/pip-install-uhfcgui4/transformers_4e503058d4964cd8a7298704ad7381c4
  Running command git clone --filter=blob:none --quiet https://github.com/huggingface/transformers.git /tmp/pip-install-uhfcgui4/transformers_4e503058d4964cd8a7298704ad7381c4
  Resolved https://github.com/huggingface/transformers.git to commit ee88ae59940fd4b2c8fc119373143d7a1175c651
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: bitsandbytes>=0.39.0 in /usr/local/lib/python3.10/dist-packages (from axolotl==0.1) (0.39.0)
Requirement already satisfied: accelerate in /usr/local/lib/python3.10/dist-packages (from axolotl==0.1) (0.20.3)
Requirement already satisfied: addict in /usr/local/lib/python3.10/dist-packages (from axolotl==0.1) (2.4.0)
Requirement already satisfied: fire in /usr/local/lib/python3.10/dist-packages (from axolotl==0.1) (0.5.0)
Requirement already satisfied: PyYAML==6.0 in /usr/local/lib/python3.10/dist-packages (from axolotl==0.1) (6.0)
Requirement already satisfied: datasets in /usr/local/lib/python3.10/dist-packages (from axolotl==0.1) (2.13.0)
Requirement already satisfied: sentencepiece in /usr/local/lib/python3.10/dist-packages (from axolotl==0.1) (0.1.99)
Requirement already satisfied: wandb in /usr/local/lib/python3.10/dist-packages (from axolotl==0.1) (0.15.4)
Requirement already satisfied: einops in /usr/local/lib/python3.10/dist-packages (from axolotl==0.1) (0.6.1)
Requirement already satisfied: xformers in /usr/local/lib/python3.10/dist-packages (from axolotl==0.1) (0.0.20)
Requirement already satisfied: optimum in /usr/local/lib/python3.10/dist-packages (from axolotl==0.1) (1.8.8)
Requirement already satisfied: bert-score==0.3.13 in /usr/local/lib/python3.10/dist-packages (from axolotl==0.1) (0.3.13)
Requirement already satisfied: evaluate==0.4.0 in /usr/local/lib/python3.10/dist-packages (from axolotl==0.1) (0.4.0)
Requirement already satisfied: rouge-score==0.1.2 in /usr/local/lib/python3.10/dist-packages (from axolotl==0.1) (0.1.2)
Requirement already satisfied: scipy in /usr/local/lib/python3.10/dist-packages (from axolotl==0.1) (1.10.1)
Requirement already satisfied: scikit-learn==1.2.2 in /usr/local/lib/python3.10/dist-packages (from axolotl==0.1) (1.2.2)
Requirement already satisfied: torch>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from bert-score==0.3.13->axolotl==0.1) (2.0.1+cu118)
Requirement already satisfied: pandas>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from bert-score==0.3.13->axolotl==0.1) (1.5.3)
Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from bert-score==0.3.13->axolotl==0.1) (1.22.4)
Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from bert-score==0.3.13->axolotl==0.1) (2.27.1)
Requirement already satisfied: tqdm>=4.31.1 in /usr/local/lib/python3.10/dist-packages (from bert-score==0.3.13->axolotl==0.1) (4.65.0)
Requirement already satisfied: matplotlib in /usr/local/lib/python3.10/dist-packages (from bert-score==0.3.13->axolotl==0.1) (3.7.1)
Requirement already satisfied: packaging>=20.9 in /usr/local/lib/python3.10/dist-packages (from bert-score==0.3.13->axolotl==0.1) (23.1)
Requirement already satisfied: dill in /usr/local/lib/python3.10/dist-packages (from evaluate==0.4.0->axolotl==0.1) (0.3.6)
Requirement already satisfied: xxhash in /usr/local/lib/python3.10/dist-packages (from evaluate==0.4.0->axolotl==0.1) (3.2.0)
Requirement already satisfied: multiprocess in /usr/local/lib/python3.10/dist-packages (from evaluate==0.4.0->axolotl==0.1) (0.70.14)
Requirement already satisfied: fsspec[http]>=2021.05.0 in /usr/local/lib/python3.10/dist-packages (from evaluate==0.4.0->axolotl==0.1) (2023.4.0)
Requirement already satisfied: huggingface-hub>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from evaluate==0.4.0->axolotl==0.1) (0.15.1)
Requirement already satisfied: responses<0.19 in /usr/local/lib/python3.10/dist-packages (from evaluate==0.4.0->axolotl==0.1) (0.18.0)
Requirement already satisfied: absl-py in /usr/local/lib/python3.10/dist-packages (from rouge-score==0.1.2->axolotl==0.1) (1.4.0)
Requirement already satisfied: nltk in /usr/local/lib/python3.10/dist-packages (from rouge-score==0.1.2->axolotl==0.1) (3.8.1)
Requirement already satisfied: six>=1.14.0 in /usr/local/lib/python3.10/dist-packages (from rouge-score==0.1.2->axolotl==0.1) (1.16.0)
Requirement already satisfied: joblib>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from scikit-learn==1.2.2->axolotl==0.1) (1.2.0)
Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn==1.2.2->axolotl==0.1) (3.1.0)
Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from accelerate->axolotl==0.1) (5.9.5)
Requirement already satisfied: pyarrow>=8.0.0 in /usr/local/lib/python3.10/dist-packages (from datasets->axolotl==0.1) (9.0.0)
Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (from datasets->axolotl==0.1) (3.8.4)
Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers@ git+https://github.com/huggingface/transformers.git->axolotl==0.1) (3.12.0)
Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers@ git+https://github.com/huggingface/transformers.git->axolotl==0.1) (2022.10.31)
Requirement already satisfied: tokenizers!=0.11.3,<0.14,>=0.11.1 in /usr/local/lib/python3.10/dist-packages (from transformers@ git+https://github.com/huggingface/transformers.git->axolotl==0.1) (0.13.3)
Requirement already satisfied: safetensors>=0.3.1 in /usr/local/lib/python3.10/dist-packages (from transformers@ git+https://github.com/huggingface/transformers.git->axolotl==0.1) (0.3.1)
Requirement already satisfied: termcolor in /usr/local/lib/python3.10/dist-packages (from fire->axolotl==0.1) (2.3.0)
Requirement already satisfied: coloredlogs in /usr/local/lib/python3.10/dist-packages (from optimum->axolotl==0.1) (15.0.1)
Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from optimum->axolotl==0.1) (1.11.1)
Requirement already satisfied: torchvision in /usr/local/lib/python3.10/dist-packages (from optimum->axolotl==0.1) (0.15.2+cu118)
Requirement already satisfied: Click!=8.0.0,>=7.0 in /usr/local/lib/python3.10/dist-packages (from wandb->axolotl==0.1) (8.1.3)
Requirement already satisfied: GitPython!=3.1.29,>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from wandb->axolotl==0.1) (3.1.31)
Requirement already satisfied: sentry-sdk>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from wandb->axolotl==0.1) (1.25.1)
Requirement already satisfied: docker-pycreds>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from wandb->axolotl==0.1) (0.4.0)
Requirement already satisfied: pathtools in /usr/local/lib/python3.10/dist-packages (from wandb->axolotl==0.1) (0.1.2)
Requirement already satisfied: setproctitle in /usr/local/lib/python3.10/dist-packages (from wandb->axolotl==0.1) (1.3.2)
Requirement already satisfied: setuptools in /usr/local/lib/python3.10/dist-packages (from wandb->axolotl==0.1) (67.7.2)
Requirement already satisfied: appdirs>=1.4.3 in /usr/local/lib/python3.10/dist-packages (from wandb->axolotl==0.1) (1.4.4)
Requirement already satisfied: protobuf!=4.21.0,<5,>=3.19.0 in /usr/local/lib/python3.10/dist-packages (from wandb->axolotl==0.1) (3.20.3)
Requirement already satisfied: pyre-extensions==0.0.29 in /usr/local/lib/python3.10/dist-packages (from xformers->axolotl==0.1) (0.0.29)
Requirement already satisfied: typing-inspect in /usr/local/lib/python3.10/dist-packages (from pyre-extensions==0.0.29->xformers->axolotl==0.1) (0.9.0)
Requirement already satisfied: typing-extensions in /usr/local/lib/python3.10/dist-packages (from pyre-extensions==0.0.29->xformers->axolotl==0.1) (4.5.0)
Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch>=1.0.0->bert-score==0.3.13->axolotl==0.1) (3.1)
Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch>=1.0.0->bert-score==0.3.13->axolotl==0.1) (3.1.2)
Requirement already satisfied: triton==2.0.0 in /usr/local/lib/python3.10/dist-packages (from torch>=1.0.0->bert-score==0.3.13->axolotl==0.1) (2.0.0)
Requirement already satisfied: cmake in /usr/local/lib/python3.10/dist-packages (from triton==2.0.0->torch>=1.0.0->bert-score==0.3.13->axolotl==0.1) (3.25.2)
Requirement already satisfied: lit in /usr/local/lib/python3.10/dist-packages (from triton==2.0.0->torch>=1.0.0->bert-score==0.3.13->axolotl==0.1) (16.0.5)
Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->axolotl==0.1) (23.1.0)
Requirement already satisfied: charset-normalizer<4.0,>=2.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->axolotl==0.1) (2.0.12)
Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->axolotl==0.1) (6.0.4)
Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->axolotl==0.1) (4.0.2)
Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->axolotl==0.1) (1.9.2)
Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->axolotl==0.1) (1.3.3)
Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->axolotl==0.1) (1.3.1)
Requirement already satisfied: gitdb<5,>=4.0.1 in /usr/local/lib/python3.10/dist-packages (from GitPython!=3.1.29,>=1.0.0->wandb->axolotl==0.1) (4.0.10)
Requirement already satisfied: python-dateutil>=2.8.1 in /usr/local/lib/python3.10/dist-packages (from pandas>=1.0.1->bert-score==0.3.13->axolotl==0.1) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas>=1.0.1->bert-score==0.3.13->axolotl==0.1) (2022.7.1)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->bert-score==0.3.13->axolotl==0.1) (1.26.15)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->bert-score==0.3.13->axolotl==0.1) (2022.12.7)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->bert-score==0.3.13->axolotl==0.1) (3.4)
Requirement already satisfied: humanfriendly>=9.1 in /usr/local/lib/python3.10/dist-packages (from coloredlogs->optimum->axolotl==0.1) (10.0)
Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->bert-score==0.3.13->axolotl==0.1) (1.0.7)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/dist-packages (from matplotlib->bert-score==0.3.13->axolotl==0.1) (0.11.0)
Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib->bert-score==0.3.13->axolotl==0.1) (4.39.3)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->bert-score==0.3.13->axolotl==0.1) (1.4.4)
Requirement already satisfied: pillow>=6.2.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib->bert-score==0.3.13->axolotl==0.1) (8.4.0)
Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib->bert-score==0.3.13->axolotl==0.1) (3.0.9)
Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->optimum->axolotl==0.1) (1.3.0)
Requirement already satisfied: smmap<6,>=3.0.1 in /usr/local/lib/python3.10/dist-packages (from gitdb<5,>=4.0.1->GitPython!=3.1.29,>=1.0.0->wandb->axolotl==0.1) (5.0.0)
Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch>=1.0.0->bert-score==0.3.13->axolotl==0.1) (2.1.2)
Requirement already satisfied: mypy-extensions>=0.3.0 in /usr/local/lib/python3.10/dist-packages (from typing-inspect->pyre-extensions==0.0.29->xformers->axolotl==0.1) (1.0.0)
Installing collected packages: axolotl
  Attempting uninstall: axolotl
    Found existing installation: axolotl 0.1
    Uninstalling axolotl-0.1:
      Successfully uninstalled axolotl-0.1
  Running setup.py develop for axolotl
Successfully installed axolotl-0.1
In [48]:
%env WANDB_MODE=offline

env: WANDB_MODE=offline
In [56]:
%cd /content/axolotl/

/content/axolotl
In [66]:
%%writefile falcon-1b-qlora.yml
# 1b: tiiuae/falcon-rw-1b
# 40b: tiiuae/falcon-40b
base_model: tiiuae/falcon-rw-1b
base_model_config: tiiuae/falcon-rw-1b
# required by falcon custom model code: https://huggingface.co/tiiuae/falcon-rw-1b/tree/main
trust_remote_code: true
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer
load_in_8bit: false
# enable 4bit for QLoRA
load_in_4bit: true
gptq: false
strict: false
push_dataset_to_hub:
datasets:
  - path: /content/axolotl/llm-playground/datasets/winglian_evals/hellaswag
    type: explainchoice:chat
dataset_prepared_path: last_run_prepared
val_set_size: 0.01
# enable QLoRA
adapter: qlora
lora_model_dir:
sequence_len: 2048
max_packed_sequence_len:

# hyperparameters from QLoRA paper Appendix B.2
# "We find hyperparameters to be largely robust across datasets"
lora_r: 64
lora_alpha: 16
# 0.1 for models up to 13B
# 0.05 for 33B and 65B models
lora_dropout: 0.05
# add LoRA modules on all linear layers of the base model
lora_target_modules:
lora_target_linear: true
lora_fan_in_fan_out:

wandb_project:
wandb_watch:
wandb_run_id:
wandb_log_model:
output_dir: ./qlora-out

# QLoRA paper Table 9
# - 16 for 7b & 13b
# - 32 for 33b, 64 for 64b
# Max size tested on A6000
# - 7b: 40
# - 40b: 4
# decrease if OOM, increase for max VRAM utilization
micro_batch_size: 1
gradient_accumulation_steps: 2
num_epochs: 3
# Optimizer for QLoRA
optimizer: paged_adamw_32bit
torchdistx_path:
lr_scheduler: cosine
# QLoRA paper Table 9
# - 2e-4 for 7b & 13b
# - 1e-4 for 33b & 64b
learning_rate: 0.0002
train_on_inputs: false
group_by_length: false
bf16: false
fp16: false
tf32: false
gradient_checkpointing: true
# stop training after this many evaluation losses have increased in a row
# https://huggingface.co/transformers/v4.2.2/_modules/transformers/trainer_callback.html#EarlyStoppingCallback
# early_stopping_patience: 3
resume_from_checkpoint:
auto_resume_from_checkpoints: true
local_rank:
logging_steps: 1
xformers_attention:
flash_attention:
flash_optimum:
gptq_groupsize:
gptq_model_v1:
warmup_steps: 10
eval_steps: 5
save_steps: 10
debug:
deepspeed:
weight_decay: 0.000001
fsdp:
fsdp_config:
special_tokens:
  pad_token: "<|endoftext|>"
  bos_token: ">>ABSTRACT<<"
  eos_token: "<|endoftext|>"

Overwriting falcon-1b-qlora.yml
In [58]:
!accelerate config default

2023-06-17 08:23:12.424042: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Configuration already exists at /root/.cache/huggingface/accelerate/default_config.yaml, will not override. Run `accelerate config` manually or pass a different `save_location`.
In [68]:
!accelerate launch scripts/finetune.py falcon-1b-qlora.yml --prepare_ds_only

2023-06-17 08:30:02.556480: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2023-06-17 08:30:11.564824: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please run

python -m bitsandbytes

 and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================
bin /usr/local/lib/python3.10/dist-packages/bitsandbytes/libbitsandbytes_cpu.so
/usr/local/lib/python3.10/dist-packages/bitsandbytes/cextension.py:34: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable.
  warn("The installed version of bitsandbytes was compiled without GPU support. "
/usr/local/lib/python3.10/dist-packages/bitsandbytes/libbitsandbytes_cpu.so: undefined symbol: cadam32bit_grad_fp32
CUDA SETUP: Loading binary /usr/local/lib/python3.10/dist-packages/bitsandbytes/libbitsandbytes_cpu.so...
WARNING:root:`trust_remote_code` is set to true. Please make sure that you reviewed the remote code/model.
INFO:root:loading tokenizer... tiiuae/falcon-rw-1b
Using pad_token, but it is not set yet.
INFO:root:Unable to find prepared dataset in last_run_prepared/c7a68403218431471fee2af3fe7750b3
INFO:root:Loading raw datasets...
INFO:root:No seed provided, using default seed of 42
Downloading and preparing dataset json/hellaswag to /root/.cache/huggingface/datasets/json/hellaswag-70dd5cc5956d23d2/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4...
Downloading data files: 100% 1/1 [00:00<00:00, 3883.61it/s]
Extracting data files: 100% 1/1 [00:00<00:00, 537.11it/s]
Dataset json downloaded and prepared to /root/.cache/huggingface/datasets/json/hellaswag-70dd5cc5956d23d2/0.0.0/e347ab1c932092252e717ff3f949105a4dd28b27e842dd53157d2f72e276c2e4. Subsequent calls will reuse this data.
100% 1/1 [00:00<00:00, 136.84it/s]
INFO:root:tokenizing, merging, and shuffling master dataset
INFO:root:Saving merged prepared dataset to disk... last_run_prepared/c7a68403218431471fee2af3fe7750b3
INFO:root:Finished preparing dataset. Exiting...
In [ ]: