StarChat2-15B-v0.1 | OpenRouter | Page 1

pliant stratus May 26, 2024, 2:30 AM

#

https://huggingface.co/HuggingFaceH4/starchat2-15b-v0.1

This model has benchmark performance comparable to DeepSeek Coder 6.7B Instruct and the other DeepSeek Coder 6.7B finetunes, but with better support for lower-resource coding languages and fully open data.

HuggingFaceH4/starchat2-15b-v0.1 · Hugging Face

pliant stratus Jun 12, 2024, 4:21 PM

#

@deep thunder The twitter post you mentioned says that this is also hosted by HuggingFace Inference API

pngwn (@evilpingwin) on X

@iamRezaSayar @Gradio @huggingface I made you a list:

HuggingFaceM4/idefics2-8b
codellama/CodeLlama-7b-hf
HuggingFaceH4/zephyr-7b-alpha
google/flan-t5-xxl
bigcode/octocoder
bigcode/santacoder
bigcode/starcoder
bigcode/starcoder2-15b
bigcode/starcoder2-3b
codellama/CodeLlama-13b-hf…

#

Its benchmark performance is way better than BigCode's instruct finetune of StarCoder2

deep thunder Jun 12, 2024, 4:21 PM

#

pliant stratus <@718785396485390346> The [twitter post](https://x.com/evilpingwin/status/178605...

yeah, I've shown this post to Lab in #1239440110559170620 thread

deep thunder Jun 12, 2024, 4:22 PM

#

pliant stratus Its benchmark performance is way better than BigCode's instruct finetune of Star...

interesting, it didn't have any SFT or DPO coding specific datasets
Only generalist ones

pliant stratus Jun 12, 2024, 4:25 PM

#

StarChat2 - 71.3 HumanEval+, 64.6 MBPP+
StarCoder2-Instruct - 60.4 HumanEval+, 65.1 MBPP+
Speechless-StarCoder2 - 62.8 HumanEval+, 62.1 MBPP+

#

oh, it's not as good at MBPP+

#

unfortunately we have no benchmark data from it

#

other than EvalPlus

#

also original MBPP is probably completely useless, since it contains many nonsense questions and questions with incorrect test cases

#StarChat2-15B-v0.1