turboderp/Cat-Llama-3-70B-instruct | OpenRouter | Page 1

calm vale May 16, 2024, 5:24 AM

#

Fine-tuned on over a billion tokens of data using a custom pipeline, Cat is one of the best performing Llama-3 fine tunes. It has highly advanced chain of thought prompting, as well as being resistant to rejections.

https://huggingface.co/turboderp/Cat-Llama-3-70B-instruct

Make sure to use the correct BOS token: <|begin_of_text|>

There is also an 8B version which you can find here:
https://huggingface.co/TheSkullery/llama-3-cat-8b-instruct-v1

Personally I think the model is amazing, and I'd love to hear what you think after trying it out.

For the system prompt, the model works best with statements like "The following is a conversation between..." or "Below is..." rather than "You are..." or "You must..." statements!

70B GGUF REPOS
https://huggingface.co/mradermacher/Cat-Llama-3-70B-instruct-i1-GGUF
https://huggingface.co/mradermacher/Cat-Llama-3-70B-instruct-GGUF

8B GGUF REPO
https://huggingface.co/bartowski/llama-3-cat-8b-instruct-v1-GGUF/tree/main

hazy finch May 16, 2024, 5:38 AM

#

here are the cots:

#

#

here is the ERP

crimson gyro May 17, 2024, 6:07 PM

#

cc @regal vortex wonder what you think about this model

regal vortex May 17, 2024, 6:07 PM

#

crimson gyro cc <@718785396485390346> wonder what you think about this model

Haven't tried it yet
Prob will try rn

regal vortex May 17, 2024, 6:27 PM

#

I can't get it to output more than one sentence💀

#

maybe 4bit exl2 quant is broken

crimson gyro May 17, 2024, 6:28 PM

#

rippp

regal vortex May 17, 2024, 7:05 PM

#

Got the official 4bpw EXL2 for 70B, still broken

#

Even with skip EOS token and skip special tokens enabled

#

I think it spams EOS tokens

#

oh boy

#

it spits out special tokens

crimson gyro May 17, 2024, 7:07 PM

#

interesting..

regal vortex May 17, 2024, 7:07 PM

#

crimson gyro interesting..

I'm running it on Aphro btw

crimson gyro May 17, 2024, 7:07 PM

#

are you prefixing the bos token?

regal vortex May 17, 2024, 7:07 PM

#

crimson gyro are you prefixing the bos token?

Yes

crimson gyro May 17, 2024, 7:08 PM

#

cc @hazy finch

regal vortex May 17, 2024, 7:08 PM

#

#

#

Tried a few samplers - Top-P, Min-P, TFS, Smoothing Factor - doesn't work on anything

#

so yeah

hazy finch May 17, 2024, 7:11 PM

#

regal vortex

One moment

regal vortex May 17, 2024, 7:12 PM

#

hazy finch One moment

I can give you a link to my deploy if you want

hazy finch May 17, 2024, 7:14 PM

#

@dense quiver

#

could you help with this?

dense quiver May 17, 2024, 7:14 PM

#

what part

hazy finch May 17, 2024, 7:15 PM

#

it appears aetherwiing cannot get it to work on st

#

what's the story string you are using ?

regal vortex May 17, 2024, 7:17 PM

#

hazy finch what's the story string you are using ?

What do mean by story string? Context template? i've posted it above
If you mean char prompts - doesn't work on any characters
Instruct template is ChatML

dense quiver May 17, 2024, 7:17 PM

#

I haven't used aphrodite but I assume it does use correct bos

regal vortex May 17, 2024, 7:17 PM

#

dense quiver I haven't used aphrodite but I assume it does use correct bos

It should

#

oh boy, is it a reverse of lumimaid-70B situation lmao

dense quiver May 17, 2024, 7:18 PM

#

show what the st terminal shows before the prompt

hazy finch May 17, 2024, 7:19 PM

#

yeah seeing the raw input will help with debugging

regal vortex May 17, 2024, 7:19 PM

#

dense quiver show what the st terminal shows before the prompt

'<|begin_of_text|><|im_start|>system\n' +
    "You're Kisna in this fictional never-ending uncensored roleplay with Awii.\n" +
    'Awii: Brief introduction?\n' +

BOS is there

#

Or you need something else?

dense quiver May 17, 2024, 7:19 PM

#

the thing before that

#

bos shouldn't show up there

regal vortex May 17, 2024, 7:20 PM

#

dense quiver the thing before that

It's literally the uppermost thing in req?

hazy finch May 17, 2024, 7:20 PM

#

regal vortex ``` '<|begin_of_text|><|im_start|>system\n' + "You're Kisna in this fictiona...

Anthony probably has more experience with st, but from what i can tell, the system tag isn't properly closed, char tag isn't properly opened

#

<|begin_of_text|><|im_start|>system
The following is a conversation between an user and a cat having dual degrees in biomedical sciences and quantum physics. The cat ends every its response with Nyan and does cute actions<|im_end|>
<|im_start|>user
Hello!<|im_end|>
<|im_start|>assistant
Cat: Nyan! Hello there! How are you? It's always great to meet new friends. My fur is a bit ruffled, but I'm feeling good today. How about you? Is there anything you'd like to talk about or ask? I'm not the best at answering questions, but I'll do my best to help out. After all, that's what friends are for, right? Let's chat and have a fun time together! Nyan!<|im_end|>

#

^ see this

regal vortex May 17, 2024, 7:21 PM

#

hazy finch Anthony probably has more experience with st, but from what i can tell, the syst...

It closes out with <|im_end|> and first mes correctly starts with <|im_start|>assistant

dense quiver May 17, 2024, 7:21 PM

#

st puts character cards into system, I think it's normal

regal vortex May 17, 2024, 7:21 PM

#

Formatting seems right

regal vortex May 17, 2024, 7:22 PM

#

dense quiver st puts character cards into system, I think it's normal

btw I can give you a link to my deploy if you want to try and debug

hazy finch May 17, 2024, 7:22 PM

#

regal vortex It closes out with <|im_end|> and first mes correctly starts with <|im_start|>as...

can i see the entire thing? Is this produced by/fed into the cat model?

dense quiver May 17, 2024, 7:22 PM

#

regal vortex It's literally the uppermost thing in req?

This is what shows up with tabby, but this is depending on the backend so aphrodite might not have it

regal vortex May 17, 2024, 7:23 PM

#

hazy finch can i see the entire thing? Is this produced by/fed into the cat model?

📎 message.txt

dense quiver May 17, 2024, 7:23 PM

#

either way what I'm looking for might not be there

#

oh it's at the bottom

regal vortex May 17, 2024, 7:23 PM

#

dense quiver This is what shows up with tabby, but this is depending on the backend so aphrod...

ye, Aphro doesn't do that normally

dense quiver May 17, 2024, 7:24 PM

#

add bos token is on so it should be working

dense quiver May 17, 2024, 7:24 PM

#

regal vortex

you don't need bos in there

regal vortex May 17, 2024, 7:25 PM

#

dense quiver you don't need bos in there

broken regardless of that, I've checked

dense quiver May 17, 2024, 7:25 PM

#

what I will point out is the system prompt having some first person wording

regal vortex May 17, 2024, 7:26 PM

#

dense quiver what I will point out is the system prompt having some first person wording

Even if I remove "You're" from sys prompt and replace it with "Below is.." stuf it still does that

#

Also it does that on char cards in various formats

#

e.g. W++

#

which doesn't have first person

dense quiver May 17, 2024, 7:27 PM

#

I'll try looking a bit though it more, give me sec

hazy finch May 17, 2024, 7:28 PM

#

prompt: '<|begin_of_text|><|im_start|>system\n' +
"Below is a conversation between Kisna and Awii.\n" +
"[Awii is a young male in his early 20's. He has chocolate brown hair and gray eyes. He has a pretty average build.]<|im_end|>\n" +
'<|im_start|>assistant\n' +
'Kisna: Your assistant Kisna walks on over to you, her blue eyes staring at you before she sits herself against the side of your desk. As she adjusts your hair, you notice her outfit which consists of a white, opened jacket, gray shorts, some long brown gloves, a brown bodysuit and a pair of brown, thigh-high boots. She crosses her legs before she looks down at you, sighing deeply before beginning to speak.\n' +
'\n' +
"Welcome back Awii. Why did you wake me up so early? ...What are you dawdling for? O-of course I rested well! It wasn't like I was thinking about you all night or anything! She crosses her arms towards her chest before pouting. Anyway, are we doing anything today or are you just going to have me sit here? Standing idle is boring you know?<|im_end|>\n" +
'<|im_start|>user\n' +
'Awii: Write a funny joke<|im_end|>\n' +
'<|im_start|>assistant\n' +
'Kisna:',

#

Could you please try this?

#

lets play it safe first

regal vortex May 17, 2024, 7:29 PM

#

hazy finch prompt: '<|begin_of_text|><|im_start|>system\n' + "Below is a conversation b...

Tried exactly that, but still having it abruptly ends the sequence

#

no spitting out of special tokens so far tho

hazy finch May 17, 2024, 7:30 PM

#

you are using the original fp16 or some sort of quant?

regal vortex May 17, 2024, 7:30 PM

#

hazy finch you are using the original fp16 or some sort of quant?

EXL2 quant from turboderp acc

#

4bpw

hazy finch May 17, 2024, 7:30 PM

#

regal vortex Tried exactly that, but still having it abruptly ends the sequence

May i see the raw input and output?

#

of that picture

regal vortex May 17, 2024, 7:31 PM

#

hazy finch May i see the raw input and output?

📎 message.txt

hazy finch May 17, 2024, 7:32 PM

#

hazy finch prompt: '<|begin_of_text|><|im_start|>system\n' + "Below is a conversation b...

It might not like the conversation in the system card. Could you try this system prompt?

regal vortex May 17, 2024, 7:33 PM

#

hazy finch It might not like the conversation in the system card. Could you try this system...

Like, deleting the whole card?

#

Also tried, same thing

dense quiver May 17, 2024, 7:33 PM

#

have you always used aprodite and it worked?

hazy finch May 17, 2024, 7:33 PM

#

regal vortex Like, deleting the whole card?

Can i see the raw input?

regal vortex May 17, 2024, 7:34 PM

#

hazy finch Can i see the raw input?

📎 message.txt

hazy finch May 17, 2024, 7:35 PM

#

regal vortex

This looks right, except i dont see a bos. What's the output of this and what happens if you add a bos?

regal vortex May 17, 2024, 7:36 PM

#

hazy finch This looks right, except i dont see a bos. What's the output of this and what ha...

Output is Well…

dense quiver May 17, 2024, 7:36 PM

#

bos doesn't show up on st log, only shows up as a parameter

#

it is enabled though

regal vortex May 17, 2024, 7:36 PM

#

and adding it to context template doesn't change anything

dense quiver May 17, 2024, 7:37 PM

#

also have you always used aphrodite?

regal vortex May 17, 2024, 7:37 PM

#

dense quiver it is enabled though

It's always enabled
No way to disable it fyi

regal vortex May 17, 2024, 7:37 PM

#

dense quiver also have you always used aphrodite?

Yes, no breakage with other models, including L3-70B finetunes

#

Even with problematic models like Lumimaid it worked fine

dense quiver May 17, 2024, 7:37 PM

#

oh okay

dense quiver May 17, 2024, 7:38 PM

#

regal vortex It's always enabled No way to disable it fyi

It's this one

hazy finch May 17, 2024, 7:38 PM

#

Could you give anthony the link to test?

regal vortex May 17, 2024, 7:38 PM

#

dense quiver It's this one

hmm I dont see that
doesn't matter ig

regal vortex May 17, 2024, 7:38 PM

#

hazy finch Could you give anthony the link to test?

Sure

dense quiver May 17, 2024, 7:38 PM

#

I haven't used 70b so I can't really tell

hazy finch May 17, 2024, 7:39 PM

#

dense quiver I haven't used 70b so I can't really tell

should functionally be similar to 8b

regal vortex May 17, 2024, 7:39 PM

#

dense quiver I haven't used 70b so I can't really tell

Sent in a dm

dense quiver May 17, 2024, 7:39 PM

#

it was trained differently

regal vortex May 17, 2024, 7:40 PM

#

hazy finch should functionally be similar to 8b

Lumimaid 8B and 70B also are supposed to similar
Yet 8B works fine on everything, and 70B breaks horribly on vLLM

#

so it happens sometimes

hazy finch May 17, 2024, 7:41 PM

#

Let me finish my lunch and i will look into this if Anthony couldn't find what's wrong

dense quiver May 17, 2024, 7:42 PM

#

yeah, I am getting the issue

regal vortex May 17, 2024, 7:43 PM

#

btw I have eager mode enabled and fp8 KV cache
It didn't cause any issues in the past

hazy finch May 17, 2024, 7:43 PM

#

Any possibility the model is corrupted?

regal vortex May 17, 2024, 7:43 PM

#

hazy finch Any possibility the model is corrupted?

I don't think so, I redownloaded it

#

It's runpod it has good internet

hazy finch May 17, 2024, 7:43 PM

#

what does eager mode do?

regal vortex May 17, 2024, 7:43 PM

#

Also an unofficial EXL2 4bpw quant was also broken

dense quiver May 17, 2024, 7:44 PM

#

oh yeah, aprodite doesn't show bos option in st

#

does aphrodite have some kind of log?

regal vortex May 17, 2024, 7:44 PM

#

hazy finch what does eager mode do?

disables CUDA graphs

regal vortex May 17, 2024, 7:45 PM

#

dense quiver does aphrodite have some kind of log?

I have max log length set to 0, but I can enable it

hazy finch May 17, 2024, 7:45 PM

#

that will be really useful

dense quiver May 17, 2024, 7:45 PM

#

try turning this off

#

it seemed to help

hazy finch May 17, 2024, 7:46 PM

#

Just "seems"?

dense quiver May 17, 2024, 7:46 PM

#

it gave me a complete respone with it off

regal vortex May 17, 2024, 7:47 PM

#

ok enabling logs

#

and restarting

hazy finch May 17, 2024, 7:47 PM

#

dense quiver it gave me a complete respone with it off

that sounds different

dense quiver May 17, 2024, 7:47 PM

#

dense quiver try turning this off

try that

regal vortex May 17, 2024, 7:47 PM

#

hazy finch that sounds different

that's weird, never seen that happen

regal vortex May 17, 2024, 7:47 PM

#

dense quiver try that

Will do once it restarts

#

oh f, it drops the weights on restart, will have to redownload

#

back up

dense quiver May 17, 2024, 7:49 PM

#

tell me if it makes a difference

regal vortex May 17, 2024, 7:49 PM

#

dense quiver tell me if it makes a difference

no difference, spitting tokens again

#

Aphro simply doesn't work I guess

#

First time since ever

dense quiver May 17, 2024, 7:50 PM

#

oh, I got a single complete response but yeah back to not working

regal vortex May 17, 2024, 7:51 PM

#

Was this model tested on like vLLM or Ooba? Or anything besides Tabby?

dense quiver May 17, 2024, 7:51 PM

#

only place I know is on her own private ui

#

@hazy finch

regal vortex May 17, 2024, 7:52 PM

#

what kind of backend this was tested on?

dense quiver May 17, 2024, 7:52 PM

#

I don't really know. I can't run over 20b

regal vortex May 17, 2024, 7:52 PM

#

Maybe it doesn't work because Aphro uses it own EXL2 implementation

#

Tho it didn't cause any issues for me previously

dense quiver May 17, 2024, 7:53 PM

#

maybe but probably not

hazy finch May 17, 2024, 7:54 PM

#

regal vortex Was this model tested on like vLLM or Ooba? Or anything besides Tabby?

it was tested on exllama

#

what log is aphro showing?

#

well gguf obviously also works

#

also the exl 2 was quanted with turbca's newest commit to exllama
so if aphro's implementation is old, it might not correctly parse the weights

regal vortex May 17, 2024, 7:56 PM

#

hazy finch what log is aphro showing?

nothing of particular interest sadly

📎 message.txt

regal vortex May 17, 2024, 7:56 PM

#

hazy finch also the exl 2 was quanted with turbca's newest commit to exllama so if aphro's ...

This might be the case, but Aphro updated like a week ago, improving EXL2 (tensor parallelism and all)

#

Aphro 0.5.3 btw

dense quiver May 17, 2024, 7:57 PM

#

yeah, not getting much luck with changing around settings

regal vortex May 17, 2024, 7:57 PM

#

hazy finch also the exl 2 was quanted with turbca's newest commit to exllama so if aphro's ...

I also used blockblockblock/Cat-Llama-3-70B-instruct-bpw4-exl2 quant too
also broken

hazy finch May 17, 2024, 7:57 PM

#

by any chance do you think you can try tabby?

#

could you pm me the st link?
I can test now

regal vortex May 17, 2024, 7:58 PM

#

hazy finch by any chance do you think you can try tabby?

Is there a template for it on Runpod?

regal vortex May 17, 2024, 7:58 PM

#

hazy finch could you pm me the st link? I can test now

The Api? sure

regal vortex May 17, 2024, 7:58 PM

#

hazy finch could you pm me the st link? I can test now

sent it

hazy finch May 17, 2024, 8:00 PM

#

https://github.com/theroyallab/tabbyAPI/wiki/1.-Getting-Started
this is a quick start for tabby @regal vortex

GitHub

1. Getting Started

An OAI compatible exllamav2 API that's both lightweight and fast - theroyallab/tabbyAPI

#

do you happen to be able to expose the st url?

regal vortex May 17, 2024, 8:00 PM

#

hazy finch do you happen to be able to expose the st url?

no, sry

hazy finch May 17, 2024, 8:01 PM

#

so @dense quiver was using his own UI with your api?

dense quiver May 17, 2024, 8:01 PM

#

yeah

#

st is just a node webapp

hazy finch May 17, 2024, 8:01 PM

#

that's weird

#

@calm vale could you help with @regal vortex with setting up cat api?
like you did on runpod the other day

hazy finch May 17, 2024, 8:02 PM

#

dense quiver st is just a node webapp

so it appears to me that it's just the api provider being uncooperative

dense quiver May 17, 2024, 8:02 PM

#

maybe

hazy finch May 17, 2024, 8:03 PM

#

the other day stanley setup the api and it worked perfectly with st

dense quiver May 17, 2024, 8:03 PM

#

he does use tabby too

#

I think

hazy finch May 17, 2024, 8:03 PM

#

yeah he mentioned it "you just run start.sh"

dense quiver May 17, 2024, 8:03 PM

#

yeah

hazy finch May 17, 2024, 8:04 PM

#

I think most likely stanley is getting his sister from school atm

calm vale May 17, 2024, 8:04 PM

#

I got back a little while ago

hazy finch May 17, 2024, 8:04 PM

#

bruh _

hazy finch May 17, 2024, 8:04 PM

#

calm vale I got back a little while ago

could you help?

regal vortex May 17, 2024, 8:04 PM

#

No template for Tabby on Runpod bruh

calm vale May 17, 2024, 8:05 PM

#

yeah I just use pytorch 2.2.0

#

then install it all

dense quiver May 17, 2024, 8:05 PM

#

does runpod allow for terminal usage like ssh or something

regal vortex May 17, 2024, 8:05 PM

#

dense quiver does runpod allow for terminal usage like ssh or something

yes

#

I guess I kill the aphro spot and start a pytorch one

dense quiver May 17, 2024, 8:05 PM

#

should be easy to install manually then

#

just git clone tabby and setup config then start

regal vortex May 17, 2024, 8:06 PM

#

ok

#

tho for the future releases - I really recommend testing them on like Ooba, vLLM and Aphro

#

most ppl use those

#

(vLLM is irrelevant for EXL2)

calm vale May 17, 2024, 8:07 PM

#

I was able to get them to work on ooba 😅

regal vortex May 17, 2024, 8:08 PM

#

calm vale I was able to get them to work on ooba 😅

hm, maybe we should try Ooba?

hazy finch May 17, 2024, 8:08 PM

#

bet a potato it's aphro's problem

#

instead of the frontend

regal vortex May 17, 2024, 8:08 PM

#

hazy finch bet a potato it's aphro's problem

first one I've seen in 50 models, but looks like

#

lemme try Ooba

hazy finch May 17, 2024, 8:09 PM

#

and also I was worrying about the BOS
none of the debug log shows bos being added
if you are confident aphro adds it secretly AND manually adding it won't work...then ok

dense quiver May 17, 2024, 8:10 PM

#

@calm vale what did you use to expose tabby api publicly?

regal vortex May 17, 2024, 8:10 PM

#

hazy finch and also I was worrying about the BOS none of the debug log shows bos being adde...

I'm like 98% sure
I ran L3 models other day w/ no problems

hazy finch May 17, 2024, 8:10 PM

#

regal vortex I'm like 98% sure I ran L3 models other day w/ no problems

many finetunes weren't trained with a bos, so they don't react as strongly towards a lack of one

regal vortex May 17, 2024, 8:11 PM

#

hazy finch many finetunes weren't trained with a bos, so they don't react as strongly towar...

L3-70B-Instruct definetely was trained with BOS tho, and it worked

hazy finch May 17, 2024, 8:11 PM

#

dense quiver <@1212999708931526656> what did you use to expose tabby api publicly?

basically just instruct it to bind to 0.0.0.0

#

then in pod forward the port

#

try try tabby _

dense quiver May 17, 2024, 8:12 PM

#

I was just wondering if it allowed port forwarding or if he used some kind of reverse proxy

regal vortex May 17, 2024, 8:12 PM

#

ok, I'm trying Ooba, if it breaks, I'll try Tabby
If Ooba works, just add smth like "Aphro =<0.5.3 doesn't work" to model card

hazy finch May 17, 2024, 8:13 PM

#

dense quiver I was just wondering if it allowed port forwarding or if he used some kind of re...

yeah it's just port forwarding

dense quiver May 17, 2024, 8:13 PM

#

ah okay

regal vortex May 17, 2024, 8:13 PM

#

dense quiver I was just wondering if it allowed port forwarding or if he used some kind of re...

Setting it up on blank PyTorch template was kinda an ass when I tried it with lcpp one time
So I'm just a bit reluctant to spend extra money on spot while I'm figuring this out

hazy finch May 17, 2024, 8:14 PM

#

stanley mentioned it was like very easy
I think he got one set up in 10min from scratch

#

the day before yesterday?

dense quiver May 17, 2024, 8:14 PM

#

yeah, I just use something like ngrok as reverse proxy

#

my router doesn't support port forwarding

hazy finch May 17, 2024, 8:15 PM

#

openwrt can support port forwarding

dense quiver May 17, 2024, 8:16 PM

#

I am not installing linux on my router

hazy finch May 17, 2024, 8:16 PM

#

too bad

regal vortex May 17, 2024, 8:20 PM

#

to think about it - all this ~200msg crisis wouldn't had happened if there was an AWQ quant sayuwuLul

hazy finch May 17, 2024, 8:22 PM

#

last time awq was mentioned I heard ppl saying gguf and exl2 were the only thing that's relevant

#

_

dense quiver May 17, 2024, 8:23 PM

#

tried awq and it's just kind of slow for me

regal vortex May 17, 2024, 8:23 PM

#

hazy finch last time awq was mentioned I heard ppl saying gguf and exl2 were the only thing...

I love AWQ, zero issues with them 100% of the time

regal vortex May 17, 2024, 8:23 PM

#

dense quiver tried awq and it's just kind of slow for me

idk same speed on A6000

#

@hazy finch IT WOOORKS on Ooba

dense quiver May 17, 2024, 8:23 PM

#

it was a long time ago though

#

nice

calm vale May 17, 2024, 8:24 PM

#

uh...

regal vortex May 17, 2024, 8:24 PM

#

dense quiver nice

So, pls add to EXL2 model card that it doesn't work with Aphro =<0.5.3

calm vale May 17, 2024, 8:24 PM

#

do you guys still need the API key and url then

regal vortex May 17, 2024, 8:24 PM

#

calm vale do you guys still need the API key and url then

I guess not

dense quiver May 17, 2024, 8:24 PM

#

not my model

calm vale May 17, 2024, 8:24 PM

#

lol ok

hazy finch May 17, 2024, 8:24 PM

#

regal vortex <@550000146289524737> IT WOOORKS on Ooba

you can do many crazy things with system prompt
and it will highlight the ability of the model
although you might neeed to reword your system card a bit
can you think of a crazy systemcard you want to test?

regal vortex May 17, 2024, 8:24 PM

#

calm vale do you guys still need the API key and url then

We found out it was a backend issue

calm vale May 17, 2024, 8:24 PM

#

thanks for testing it out, aether

#

and letting us know

dense quiver May 17, 2024, 8:25 PM

#

regal vortex So, pls add to EXL2 model card that it doesn't work with Aphro =<0.5.3

@hazy finch I guess tell turbo to add that to the readmes

hazy finch May 17, 2024, 8:26 PM

#

yeah I can do it tonight
I think turbca is taking noon nap atm

dense quiver May 17, 2024, 8:26 PM

#

sleepy cat

#

I think he did wake up earlier today

hazy finch May 17, 2024, 8:27 PM

#

before falling into sleep again

hazy finch May 17, 2024, 8:29 PM

#

hazy finch you can do many crazy things with system prompt and it will highlight the abilit...

please let e know if you have further problems
@regal vortex

regal vortex May 17, 2024, 8:29 PM

#

hazy finch please let e know if you have further problems <@718785396485390346>

Seems to work fine, and it is pretty good so far

#

Trying some more complex system prompts /w CoT and complex cards

#

Actually, it already produced some of the best output I've seen from l3 finetunes I think

hazy finch May 17, 2024, 8:31 PM

#

please let me know if you cant get it to work
since the model doesnt take instruction system card like "you are/you must"
it takes "The {{char}} is, The {{char}} will always"

dense quiver May 17, 2024, 8:32 PM

#

also yeah, the 70b looks so good from that screenshot

regal vortex May 17, 2024, 8:32 PM

#

hazy finch please let me know if you cant get it to work since the model doesnt take instru...

I have most cards written this way (cause another model I happen to like prefers it like that), so not a problem

dense quiver May 17, 2024, 8:32 PM

#

wish we could've gotten a 13b l3

regal vortex May 17, 2024, 8:34 PM

#

dense quiver wish we could've gotten a 13b l3

we've gotten a 16B deepseek
close enough

hazy finch May 17, 2024, 8:34 PM

#

ok I will go back to doing surgeries
let me know if you have any difficulties

regal vortex May 17, 2024, 9:04 PM

#

So, review of sorts - really good model, and a really unusual one. Has L3-70B-instruct level of smarts when it comes to writing, but is completely (I mean, completely) uncesored and has no positivity bias. Writing quality is quite good, pretty much no GPT-isms or such stuff. Also, it's not lazy like many L3 finetunes, and writes decent amount of text. It's a bit picky when it comes to prompts, especially system ones, but handles stats, infoboards and other persistent details quite well. It's especially strong on RPG and CYOA type of cards (it does well even on ones where all other local models failed), as they directly play into it's strengths. A prompt for CoT is mandatory it seems, w/o it will generate CoT anyway, but will have a bit of a random structure (a more structured one leads to much better gens, ofc). It also picks up on small details really well (picks up on stuff that only Midnight-Rose and WinterGoddess managed to pick on, and it's a high praise). Overall, a strong model, tho I'd say it's a bit less of a RP one than a RPG one, so to speak.

calm vale May 17, 2024, 9:11 PM

#

thank you for the review, it means a lot. it's nice to hear that it does well with small details and is uncensored...

regal vortex May 17, 2024, 9:18 PM

#

calm vale thank you for the review, it means a lot. it's nice to hear that it does well wi...

I probably should have added that it was totally worth the struggle with Aphro lol
It's probably the most interesting L3-70B finetune I've seen so far. Like, it's memorable in a good way unlike all other ones I tried

calm vale May 17, 2024, 9:19 PM

#

a lot of effort was put into it, especially with the code that was developed to generate the data for it

#

very proud of kat for working so hard

regal vortex May 17, 2024, 9:19 PM

#

Lack of laziness is godsent after e.g Lumimaid which barely spits out ~100 tokens often.

regal vortex May 17, 2024, 9:19 PM

#

calm vale very proud of kat for working so hard

Kudos to everyone who worked on it

hazy finch May 18, 2024, 12:26 AM

#

regal vortex Lack of laziness is godsent after e.g Lumimaid which barely spits out ~100 token...

maybe need to let @crimson gyro know this _

crimson gyro May 18, 2024, 3:25 AM

#

Just woke up, looking pretty fun here!

#turboderp/Cat-Llama-3-70B-instruct