AMajor/grok-1
1
0
Fork 0
mirror of https://github.com/xai-org/grok-1.git synced 2025-04-03 18:00:10 +03:00
Code Issues Packages Projects Releases Wiki Activity

Compare commits

base: AMajor:72d3a0dd94bb9ab84c529514a085d61aca1e9886
Branches Tags
AMajor:main
AMajor:download-instruction
...
compare: AMajor:1094ba77eb5b498d8f46386d16731afd9e7d3be1
Branches Tags
AMajor:main
AMajor:download-instruction

3 Commits
72d3a0dd94 ... 1094ba77eb

Author SHA1 Message Date
Andrew Kean Gao
1094ba77eb
Merge 1f4350143d into 7050ed204b 2024-03-19 08:53:59 -07:00
Eddy
7050ed204b
Corrected name of package "cuda12-pip" (#194) 
The `cuda12-pip` package was wrongly named `cuda12_pip`
in requirements.txt
 2024-03-19 08:48:22 -07:00
Andrew Kean Gao
1f4350143d
Update README.md to specify the active parameters 
This is important because users may think they need to have the full 314B at one time, but only 86B which is much more manageable!
 2024-03-18 10:41:38 -07:00
2 changed files with 2 additions and 2 deletions
Show Stats Expand all files Collapse all files

2
README.md
View File

@ -22,7 +22,7 @@ The implementation of the MoE layer in this repository is not efficient. The imp
Grok-1 is currently designed with the following specifications: Grok-1 is currently designed with the following specifications:
- **Parameters:** 314B - **Parameters:** 314B (86B active)
- **Architecture:** Mixture of 8 Experts (MoE) - **Architecture:** Mixture of 8 Experts (MoE)
- **Experts Utilization:** 2 experts used per token - **Experts Utilization:** 2 experts used per token
- **Layers:** 64 - **Layers:** 64

2
requirements.txt
View File

@ -1,4 +1,4 @@
dm_haiku==0.0.12 dm_haiku==0.0.12
jax[cuda12_pip]==0.4.25 -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html jax[cuda12-pip]==0.4.25 -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html
numpy==1.26.4 numpy==1.26.4
sentencepiece==0.2.0 sentencepiece==0.2.0