From b0e77734fed75ee614b70bb592918dd5d046fbfc Mon Sep 17 00:00:00 2001 From: Szymon Tworkowski <44476995+syzymon@users.noreply.github.com> Date: Mon, 18 Mar 2024 09:11:17 -0700 Subject: [PATCH 1/5] Make download instruction more clear (#155) --- README.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index eaded5c..12570d7 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,8 @@ This repository contains JAX example code for loading and running the Grok-1 open-weights model. -Make sure to download the checkpoint and place `ckpt-0` directory in `checkpoint`. +Make sure to download the checkpoint and place `ckpt-0` directory in `checkpoint` - see [Downloading the weights](Downloading-the-weights) + Then, run ```shell @@ -24,6 +25,13 @@ You can download the weights using a torrent client and this magnet link: magnet:?xt=urn:btih:5f96d43576e3d386c9ba65b883210a393b68210e&tr=https%3A%2F%2Facademictorrents.com%2Fannounce.php&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969&tr=udp%3A%2F%2Ftracker.opentrackr.org%3A1337%2Fannounce ``` +or directly using HuggingFace: +``` +git clone https://github.com/xai-org/grok-1.git && cd grok-1 +pip install huggingface_hub[hf_transfer] +huggingface-cli download xai-org/grok-1 --repo-type model --include ckpt-0/* --local-dir checkpoints --local-dir-use-symlinks False +``` + # License The code and associated Grok-1 weights in this release are licensed under the From 1ff4435d254a77a91393f7c40a6865bd5b50e6d7 Mon Sep 17 00:00:00 2001 From: "Gareth Paul Jones (GPJ)" Date: Mon, 18 Mar 2024 09:36:24 -0700 Subject: [PATCH 2/5] Update README with Model Specifications (#27) Added an overview of the model as discussed in response to #14. Adding more info on the the model specs before they proceed to download the checkpoints should help folks ensure they have the necessary resources to effectively utilize Grok-1. --- README.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/README.md b/README.md index 12570d7..8b362bf 100644 --- a/README.md +++ b/README.md @@ -18,9 +18,26 @@ The script loads the checkpoint and samples from the model on a test input. Due to the large size of the model (314B parameters), a machine with enough GPU memory is required to test the model with the example code. The implementation of the MoE layer in this repository is not efficient. The implementation was chosen to avoid the need for custom kernels to validate the correctness of the model. +# Model Specifications + +Grok-1 is currently designed with the following specifications: + +- **Parameters:** 314B +- **Architecture:** Mixture of 8 Experts (MoE) +- **Experts Utilization:** 2 experts used per token +- **Layers:** 64 +- **Attention Heads:** 48 for queries, 8 for keys/values +- **Embedding Size:** 6,144 +- **Tokenization:** SentencePiece tokenizer with 131,072 tokens +- **Additional Features:** + - Rotary embeddings (RoPE) + - Supports activation sharding and 8-bit quantization +- **Maximum Sequence Length (context):** 8,192 tokens + # Downloading the weights You can download the weights using a torrent client and this magnet link: + ``` magnet:?xt=urn:btih:5f96d43576e3d386c9ba65b883210a393b68210e&tr=https%3A%2F%2Facademictorrents.com%2Fannounce.php&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969&tr=udp%3A%2F%2Ftracker.opentrackr.org%3A1337%2Fannounce ``` From 310e19eee25efa348fcd3ac614040791d012f23a Mon Sep 17 00:00:00 2001 From: Seth Junot Date: Mon, 18 Mar 2024 09:39:02 -0700 Subject: [PATCH 3/5] Corrected checkpoint dir name, download section link --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 8b362bf..f4d9b61 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ This repository contains JAX example code for loading and running the Grok-1 open-weights model. -Make sure to download the checkpoint and place `ckpt-0` directory in `checkpoint` - see [Downloading the weights](Downloading-the-weights) +Make sure to download the checkpoint and place the `ckpt-0` directory in `checkpoints` - see [Downloading the weights](#downloading-the-weights) Then, run From 7207216386e07206b2083c5c0be88db1add8e631 Mon Sep 17 00:00:00 2001 From: Lve Lvee <25785753+lvelvee@users.noreply.github.com> Date: Tue, 19 Mar 2024 02:01:17 +0800 Subject: [PATCH 4/5] Create .gitignore for checkpoints (#149) ignore the checkpoints files --- .gitignore | 2 ++ 1 file changed, 2 insertions(+) create mode 100644 .gitignore diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..24d0d7e --- /dev/null +++ b/.gitignore @@ -0,0 +1,2 @@ +checkpoints/* +!checkpoints/README.md From d6d9447e2d3c9bd81da571dc8681ea60009e4c03 Mon Sep 17 00:00:00 2001 From: Szymon Tworkowski <44476995+syzymon@users.noreply.github.com> Date: Mon, 18 Mar 2024 11:40:01 -0700 Subject: [PATCH 5/5] Update huggingface link --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index f4d9b61..f501a07 100644 --- a/README.md +++ b/README.md @@ -42,7 +42,7 @@ You can download the weights using a torrent client and this magnet link: magnet:?xt=urn:btih:5f96d43576e3d386c9ba65b883210a393b68210e&tr=https%3A%2F%2Facademictorrents.com%2Fannounce.php&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969&tr=udp%3A%2F%2Ftracker.opentrackr.org%3A1337%2Fannounce ``` -or directly using HuggingFace: +or directly using [HuggingFace 🤗 Hub](https://huggingface.co/xai-org/grok-1): ``` git clone https://github.com/xai-org/grok-1.git && cd grok-1 pip install huggingface_hub[hf_transfer]