Stable Code 3B is a coding model with instruct and code completion variants on par with models such as Code Llama 7B that are 2.5x larger.
106.6K Pulls Updated 8 months ago
Updated 8 months ago
8 months ago
3358af90609f · 5.6GB
Readme
Stable Code 3B is a 3 billion parameter Large Language Model (LLM), allowing accurate and responsive code completion at a level on par with models such as Code Llama 7b that are 2.5x larger.
Key Features
- NEW instruct model
ollama run stable-code - Fill in Middle Capability (FIM)
- Supports Long Context, trained with Sequences upto 16,384
| Model | Size | Python | C++ | Javascript | Java | PHP | Rust |
|---|---|---|---|---|---|---|---|
| Stable Code | 3B | 32.4% | 30.9% | 32.1% | 32.1% | 24.2% | 23.0% |
| CodeLLama | 7B | 30.0% | 28.2% | 32.5% | 31.1% | 25.7% | 26.3% |
| Deepseek Coder | 1.3B | 28.6% | 29.2% | 28.7% | 29.0% | 23.6% | 18.5% |
| Wizard Coder | 3B | 31.6% | 25.6% | 26.2% | 25.8% | 25.3% | 20.4% |
| StarCoder | 3B | 21.6% | 19.8% | 21.5% | 20.5% | 19.0% | 16.9% |
| Replit Code V1.5 | 3B | 23.0% | 25.9% | 26.2% | 23.6% | 23.2% | 21.5% |
| Deci Coder | 1B | 19.1% | 6.8% | 18.4% | 16.7% | 2.1% | 1.7% |
Model Details
- Developed by: Stability AI
- Model type: stable-code models are auto-regressive language models based on the transformer decoder architecture.
- Language(s): English, Code
- Contact: For questions and comments about the model, please email
[email protected]
Model Architecture
The model is a decoder-only transformer similar to the LLaMA (Touvron et al., 2023) architecture with the following modifications:
| Parameters | Hidden Size | Layers | Heads | Sequence Length |
|---|---|---|---|---|
| 2,796,431,360 | 2560 | 32 | 32 | 16384 |
- Position Embeddings: Rotary Position Embeddings (Su et al., 2021) applied to the first 25% of head embedding dimensions for improved throughput following Black et al. (2022).
- Tokenizer: We use a modified version of the GPTNeoX Tokenizer.
NeoX. We add special tokens to train for Fill in the Middle (FIM) capabilities like<FIM_PREFIX>and<FIM_SUFFIX>along with other special tokens.
Training
Training Dataset
The dataset is comprised of a filtered mixture of open-source large-scale datasets available on the HuggingFace Hub: Falcon RefinedWeb extract (Penedo et al., 2023), along with CommitPackFT and Github Issues (BigCode., 2023), and StarCoder (Li et al., 2023). We further supplement our training with data from mathematical domains (Azerbayev, Zhangir, et al., 2023 and, Yu, Longhui, et al., 2023).
Top 18 programming languages trained on: - C - CPP - Java - JavaScript - CSS - Go - HTML - Ruby - Rust - Markdown - Shell - Php - Sql - R - Typescript - Python - Jupyter-Clean - RestructuredText
Use and Limitations
Intended Use
The model is intended to be used as a foundational base model for application-specific fine-tuning. Developers must evaluate and fine-tune the model for safe performance in downstream applications.
Limitations and Bias
As a base model, this model may exhibit unreliable, unsafe, or other undesirable behaviors that must be corrected through evaluation and fine-tuning prior to deployment. The pre-training dataset may have contained offensive or inappropriate content, even after applying data cleansing filters, which can be reflected in the model-generated text. We recommend that users exercise caution when using these models in production systems. Do not use the models if they are unsuitable for your application, or for any applications that may cause deliberate or unintentional harm to others.