From b0e77734fed75ee614b70bb592918dd5d046fbfc Mon Sep 17 00:00:00 2001 From: Szymon Tworkowski <44476995+syzymon@users.noreply.github.com> Date: Mon, 18 Mar 2024 09:11:17 -0700 Subject: [PATCH 1/3] Make download instruction more clear (#155) --- README.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index eaded5c..12570d7 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,8 @@ This repository contains JAX example code for loading and running the Grok-1 open-weights model. -Make sure to download the checkpoint and place `ckpt-0` directory in `checkpoint`. +Make sure to download the checkpoint and place `ckpt-0` directory in `checkpoint` - see [Downloading the weights](Downloading-the-weights) + Then, run ```shell @@ -24,6 +25,13 @@ You can download the weights using a torrent client and this magnet link: magnet:?xt=urn:btih:5f96d43576e3d386c9ba65b883210a393b68210e&tr=https%3A%2F%2Facademictorrents.com%2Fannounce.php&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969&tr=udp%3A%2F%2Ftracker.opentrackr.org%3A1337%2Fannounce ``` +or directly using HuggingFace: +``` +git clone https://github.com/xai-org/grok-1.git && cd grok-1 +pip install huggingface_hub[hf_transfer] +huggingface-cli download xai-org/grok-1 --repo-type model --include ckpt-0/* --local-dir checkpoints --local-dir-use-symlinks False +``` + # License The code and associated Grok-1 weights in this release are licensed under the From 1ff4435d254a77a91393f7c40a6865bd5b50e6d7 Mon Sep 17 00:00:00 2001 From: "Gareth Paul Jones (GPJ)" Date: Mon, 18 Mar 2024 09:36:24 -0700 Subject: [PATCH 2/3] Update README with Model Specifications (#27) Added an overview of the model as discussed in response to #14. Adding more info on the the model specs before they proceed to download the checkpoints should help folks ensure they have the necessary resources to effectively utilize Grok-1. --- README.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/README.md b/README.md index 12570d7..8b362bf 100644 --- a/README.md +++ b/README.md @@ -18,9 +18,26 @@ The script loads the checkpoint and samples from the model on a test input. Due to the large size of the model (314B parameters), a machine with enough GPU memory is required to test the model with the example code. The implementation of the MoE layer in this repository is not efficient. The implementation was chosen to avoid the need for custom kernels to validate the correctness of the model. +# Model Specifications + +Grok-1 is currently designed with the following specifications: + +- **Parameters:** 314B +- **Architecture:** Mixture of 8 Experts (MoE) +- **Experts Utilization:** 2 experts used per token +- **Layers:** 64 +- **Attention Heads:** 48 for queries, 8 for keys/values +- **Embedding Size:** 6,144 +- **Tokenization:** SentencePiece tokenizer with 131,072 tokens +- **Additional Features:** + - Rotary embeddings (RoPE) + - Supports activation sharding and 8-bit quantization +- **Maximum Sequence Length (context):** 8,192 tokens + # Downloading the weights You can download the weights using a torrent client and this magnet link: + ``` magnet:?xt=urn:btih:5f96d43576e3d386c9ba65b883210a393b68210e&tr=https%3A%2F%2Facademictorrents.com%2Fannounce.php&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969&tr=udp%3A%2F%2Ftracker.opentrackr.org%3A1337%2Fannounce ``` From 310e19eee25efa348fcd3ac614040791d012f23a Mon Sep 17 00:00:00 2001 From: Seth Junot Date: Mon, 18 Mar 2024 09:39:02 -0700 Subject: [PATCH 3/3] Corrected checkpoint dir name, download section link --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 8b362bf..f4d9b61 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ This repository contains JAX example code for loading and running the Grok-1 open-weights model. -Make sure to download the checkpoint and place `ckpt-0` directory in `checkpoint` - see [Downloading the weights](Downloading-the-weights) +Make sure to download the checkpoint and place the `ckpt-0` directory in `checkpoints` - see [Downloading the weights](#downloading-the-weights) Then, run