| | --- |
| | license: mit |
| | library_name: adapter-transformers |
| | --- |
| | Effi-13B AWQ is a quantization model of our [Effi-13B](https://huggingface.co/aiplanet/effi-13b) a reasoning model. |
| |
|
| | ## About AWQ |
| |
|
| | AWQ is an efficient, accurate and blazing-fast low-bit weight quantization method, currently supporting 4-bit quantization. Compared to GPTQ, it offers faster Transformers-based inference. |
| |
|
| | It is also now supported by continuous batching server vLLM, allowing use of AWQ models for high-throughput concurrent inference in multi-user server scenarios. |
| |
|
| | effi-13B parameters is a causal decoder-only model built by AI Planet based on Llama-2-13b-chat-hf and fine tuned using the 1.8 Million coversations from CoT dataset available in huggingface datasets. The model is made available under the Apache 2.0 license. |
| |
|
| | ## Why use effi-13B-Instruct? |
| |
|
| | - This is a ready to use chat/instruct model based on Llama-2-13b-chat-hf, which provides a rationale for the context provided. |
| | - Llama-2 is the best open-source model available. This is an instruct model, which may not be ideal for further finetuning. If you are interested in building your own instruct/chat model, we recommend starting from Llama-2-13b-chat-hf |
| | You will need at least 85-100GB of memory to run inference with effi-13b swiftly. |
| |
|
| | ## Our benchmarking |
| |
|
| | | Metric | Value | |
| | |--------------------|---------| |
| | | Perplexity | 5.529 | |
| | | MMLU | 50.90 | |
| | | Hella Swag (acc) | 59.38 | |
| | | Hella Swag (acc_norm) | 78.91 | |
| | | TruthfulQA | 38.24 | |
| | |
| | ## Direct Use |
| | |
| | effi-13b has been finetuned on a Chain of Thought dataset. |
| | |
| | ## Out-of-Scope Use |
| | |
| | Production use without adequate assessment of risks and mitigation; any use cases which may be considered irresponsible or harmful. |
| | |
| | ## Bias, Risks, and Limitations |
| | |
| | This model has been majorly trained on English data, and will not generalize appropriately to other languages. Furthermore, as it is trained on a large-scale corpora representative of the web, it will carry the stereotypes and biases commonly encountered online. |
| | |
| | ## Recommendations |
| | |
| | We recommend users of effi-13b to develop guardrails and take appropriate precautions for any production use. |
| | |
| | Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information is needed for further recommendations. |
| | |
| | ## Citations |
| | |
| | ``` |
| | @misc {lucifertrj, |
| | author = { {Tarun Jain} }, |
| | title = { Effi-13B-AWQ by AI Planet}, |
| | year = 2024, |
| | url = { https://huggingface.co/aiplanet/effi-13B-AWQ/ }, |
| | publisher = { Hugging Face } |
| | } |
| | ``` |
