In [1]:
%cd ../datasets
/workspaces/llm-playground/datasets
In [2]:
from huggingface_hub import Repository
In [3]:
!git lfs --help
GIT-LFS(1) GIT-LFS(1)
NAME
git-lfs - Work with large files in Git repositories
SYNOPSIS
git lfs <command> [<args>]
DESCRIPTION
Git LFS is a system for managing and versioning large files in
association with a Git repository. Instead of storing the large files
within the Git repository as blobs, Git LFS stores special "pointer
files" in the repository, while storing the actual file contents on a
Git LFS server. The contents of the large file are downloaded
automatically when needed, for example when a Git branch containing the
large file is checked out.
Git LFS works by using a "smudge" filter to look up the large file
contents based on the pointer file, and a "clean" filter to create a
new version of the pointer file when the large file’s contents change.
It also uses a pre-push hook to upload the large file contents to the
Git LFS server whenever a commit containing a new large file version is
about to be pushed to the corresponding Git server.
COMMANDS
Like Git, Git LFS commands are separated into high level ("porcelain")
commands and low level ("plumbing") commands.
High level porcelain commands
git-lfs-checkout(1)
Populate working copy with real content from Git LFS files.
git-lfs-dedup(1)
De-duplicate Git LFS files.
git-lfs-env(1)
Display the Git LFS environment.
git-lfs-ext(1)
Display Git LFS extension details.
git-lfs-fetch(1)
Download Git LFS files from a remote.
git-lfs-fsck(1)
Check Git LFS files for consistency.
git-lfs-install(1)
Install Git LFS configuration.
git-lfs-lock(1)
Set a file as "locked" on the Git LFS server.
git-lfs-locks(1)
List currently "locked" files from the Git LFS server.
git-lfs-logs(1)
Show errors from the Git LFS command.
git-lfs-ls-files(1)
Show information about Git LFS files in the index and working tree.
git-lfs-migrate(1)
Migrate history to or from Git LFS
git-lfs-prune(1)
Delete old Git LFS files from local storage
git-lfs-pull(1)
Fetch Git LFS changes from the remote & checkout any required
working tree files.
git-lfs-push(1)
Push queued large files to the Git LFS endpoint.
git-lfs-status(1)
Show the status of Git LFS files in the working tree.
git-lfs-track(1)
View or add Git LFS paths to Git attributes.
git-lfs-uninstall(1)
Uninstall Git LFS by removing hooks and smudge/clean filter
configuration.
git-lfs-unlock(1)
Remove "locked" setting for a file on the Git LFS server.
git-lfs-untrack(1)
Remove Git LFS paths from Git Attributes.
git-lfs-update(1)
Update Git hooks for the current Git repository.
git-lfs-version(1)
Report the version number.
Low level plumbing commands
git-lfs-clean(1)
Git clean filter that converts large files to pointers.
git-lfs-filter-process(1)
Git process filter that converts between large files and pointers.
git-lfs-merge-driver(1)
Merge text-based LFS files
git-lfs-pointer(1)
Build and compare pointers.
git-lfs-post-checkout(1)
Git post-checkout hook implementation.
git-lfs-post-commit(1)
Git post-commit hook implementation.
git-lfs-post-merge(1)
Git post-merge hook implementation.
git-lfs-pre-push(1)
Git pre-push hook implementation.
git-lfs-smudge(1)
Git smudge filter that converts pointer in blobs to the actual
content.
git-lfs-standalone-file(1)
Git LFS standalone transfer adapter for file URLs (local paths).
EXAMPLES
To get started with Git LFS, the following commands can be used.
1. Setup Git LFS on your system. You only have to do this once per
user account:
git lfs install
2. Choose the type of files you want to track, for examples all ISO
images, with git-lfs-track(1):
git lfs track "*.iso"
3. The above stores this information in gitattributes(5) files, so
that file needs to be added to the repository:
git add .gitattributes
4. Commit, push and work with the files normally:
git add file.iso
git commit -m "Add disk image"
git push
GIT-LFS(1)
In [4]:
# https://huggingface.co/datasets/hoskinson-center/proof-pile/
repo = Repository(local_dir="proof-pile", clone_from="hoskinson-center/proof-pile", repo_type='dataset', skip_lfs_files=True)
/workspaces/llm-playground/datasets/proof-pile is already a clone of https://huggingface.co/datasets/hoskinson-center/proof-pile. Make sure you pull the latest changes with `repo.git_pull()`.
In [5]:
!git lfs pull --help
git lfs pull [options] [<remote>] Download Git LFS objects for the currently checked out ref, and update the working copy with the downloaded content if required. This is equivalent to running the following 2 commands: git lfs fetch [options] [] git lfs checkout Options: -I <paths>: --include=<paths>: Specify lfs.fetchinclude just for this invocation; see "Include and exclude" -X <paths>: --exclude=<paths>: Specify lfs.fetchexclude just for this invocation; see "Include and exclude" Include and exclude ------------------- You can configure Git LFS to only fetch objects to satisfy references in certain paths of the repo, and/or to exclude certain paths of the repo, to reduce the time you spend downloading things you do not use. In your Git configuration or in a .lfsconfig file, you may set either or both of lfs.fetchinclude and lfs.fetchexclude to comma-separated lists of paths. If lfs.fetchinclude is defined, Git LFS objects will only be fetched if their path matches one in that list, and if lfs.fetchexclude is defined, Git LFS objects will only be fetched if their path does not match one in that list. Paths are matched using wildcard matching as per gitignore(5). Note that using the command-line options -I and -X override the respective configuration settings. Setting either option to an empty string clears the value. Default remote -------------- Without arguments, pull downloads from the default remote. The default remote is the same as for git pull, i.e. based on the remote branch you're tracking first, or origin otherwise.
In [6]:
%cd proof-pile
/workspaces/llm-playground/datasets/proof-pile
In [7]:
repo.git_pull()
In [8]:
!ls -lhta train
total 96K drwxrwxrwx+ 2 codespace codespace 4.0K Apr 16 08:16 . -rw-rw-rw- 1 codespace codespace 134 Apr 16 08:16 proofpile_train_8.jsonl.gz -rw-rw-rw- 1 codespace codespace 134 Apr 16 08:16 proofpile_train_9.jsonl.gz -rw-rw-rw- 1 codespace codespace 134 Apr 16 08:16 proofpile_train_19.jsonl.gz -rw-rw-rw- 1 codespace codespace 134 Apr 16 08:16 proofpile_train_2.jsonl.gz -rw-rw-rw- 1 codespace codespace 134 Apr 16 08:16 proofpile_train_20.jsonl.gz -rw-rw-rw- 1 codespace codespace 134 Apr 16 08:16 proofpile_train_3.jsonl.gz -rw-rw-rw- 1 codespace codespace 134 Apr 16 08:16 proofpile_train_4.jsonl.gz -rw-rw-rw- 1 codespace codespace 134 Apr 16 08:16 proofpile_train_5.jsonl.gz -rw-rw-rw- 1 codespace codespace 134 Apr 16 08:16 proofpile_train_6.jsonl.gz -rw-rw-rw- 1 codespace codespace 134 Apr 16 08:16 proofpile_train_7.jsonl.gz drwxrwxrwx+ 6 codespace codespace 4.0K Apr 16 08:16 .. -rw-rw-rw- 1 codespace codespace 1.4K Apr 16 08:16 .gitattributes -rw-rw-rw- 1 codespace codespace 134 Apr 16 08:16 proofpile_train_0.jsonl.gz -rw-rw-rw- 1 codespace codespace 134 Apr 16 08:16 proofpile_train_1.jsonl.gz -rw-rw-rw- 1 codespace codespace 134 Apr 16 08:16 proofpile_train_10.jsonl.gz -rw-rw-rw- 1 codespace codespace 134 Apr 16 08:16 proofpile_train_11.jsonl.gz -rw-rw-rw- 1 codespace codespace 134 Apr 16 08:16 proofpile_train_12.jsonl.gz -rw-rw-rw- 1 codespace codespace 134 Apr 16 08:16 proofpile_train_13.jsonl.gz -rw-rw-rw- 1 codespace codespace 134 Apr 16 08:16 proofpile_train_14.jsonl.gz -rw-rw-rw- 1 codespace codespace 134 Apr 16 08:16 proofpile_train_15.jsonl.gz -rw-rw-rw- 1 codespace codespace 134 Apr 16 08:16 proofpile_train_16.jsonl.gz -rw-rw-rw- 1 codespace codespace 134 Apr 16 08:16 proofpile_train_17.jsonl.gz -rw-rw-rw- 1 codespace codespace 134 Apr 16 08:16 proofpile_train_18.jsonl.gz
In [9]:
%%bash
git lfs pull --include="dev/proofpile_dev.jsonl.gz"
ls -lhta dev
total 155M -rw-rw-rw- 1 codespace codespace 155M Apr 16 08:28 proofpile_dev.jsonl.gz drwxrwxrwx+ 2 codespace codespace 4.0K Apr 16 08:28 . drwxrwxrwx+ 6 codespace codespace 4.0K Apr 16 08:16 .. -rw-rw-rw- 1 codespace codespace 59 Apr 16 08:16 .gitattributes
In [10]:
%%bash
git lfs pointer --file="dev/proofpile_dev.jsonl.gz"
ls -lhta dev
Git LFS pointer for dev/proofpile_dev.jsonl.gz
version https://git-lfs.github.com/spec/v1 oid sha256:9a33bde2feabb2421936bbde65ab15692ba814812bb2cf0e82a77b35e07e0b9b size 161818020 total 155M -rw-rw-rw- 1 codespace codespace 155M Apr 16 08:28 proofpile_dev.jsonl.gz drwxrwxrwx+ 2 codespace codespace 4.0K Apr 16 08:28 . drwxrwxrwx+ 6 codespace codespace 4.0K Apr 16 08:16 .. -rw-rw-rw- 1 codespace codespace 59 Apr 16 08:16 .gitattributes
In [11]:
!pwd
/workspaces/llm-playground/datasets/proof-pile
In [12]:
%cd ../../
/workspaces/llm-playground
In [13]:
import os
HF_TOKEN = os.environ.get("HUGGINGFACE_TOKEN")
In [14]:
# https://huggingface.co/datasets/utensil/storage
repo = Repository(local_dir="storage", clone_from="utensil/storage", repo_type='dataset', skip_lfs_files=True, use_auth_token=HF_TOKEN)
Cloning https://huggingface.co/datasets/utensil/storage into local empty directory.
In [15]:
!touch storage/test.txt
In [16]:
repo.push_to_hub()
In [17]:
!date --rfc-3339=seconds > storage/time.txt
In [18]:
!cat storage/time.txt
2023-04-16 12:03:25+00:00
In [19]:
!pwd
/workspaces/llm-playground
In [20]:
!python helper/upload.py
Working directory changed to: /workspaces/llm-playground/helper/.. /workspaces/llm-playground/storage is already a clone of https://huggingface.co/datasets/utensil/storage. Make sure you pull the latest changes with `repo.git_pull()`. To https://huggingface.co/datasets/utensil/storage 7ea436a..39774b0 main -> main Upload succeeded.
In [ ]: