Chat guide? | OpenRouter | Page 1

spring ore Jan 5, 2025, 7:44 PM

#

Now that OpenRouter's chat has web search built-in, I see much less of a reason to use a separate client for general needs. (The localStorage thing is a bit scary but I'll take backups… Many others like big-AGI also do this, anyway.) That said, is there a user guide detailing each of these features somewhere? I couldn't find any in the Docs section and some things take a while to figure out.

For example, for all I know there might be a way to configure a prompt to enable (Anthropic) prompt caching, to dramatically reduce the cost of a big message + followups scenario, but I couldn't find anything. Are there / commands, perhaps?

tardy mango Jan 5, 2025, 7:46 PM

#

spring ore Now that OpenRouter's chat has web search built-in, I see much less of a reason ...

All prompt caching is automatic. Providers will automatically detect reaccuring prompts and cache it

#

Be warned that providers do not share prompt caches, so if DeepInfra caches your prompt (DeepInfra **doesn't **support prompt caching btw. This is just an example) and Hyperbolic serves your next request, HyperBolic won't have the prompt cache and so you'll be paying at a higher rate (HyperBolic also **doesn't **support prompt caching btw. This is just an example

spring ore Jan 5, 2025, 7:48 PM

#

tardy mango All prompt caching is automatic. Providers will automatically detect reaccuring ...

Nope, Anthropic requires flagging messages (no more than 4) with an "ephemeral" cache type to be considered.

#

I had a test session with Claude 3.5 Sonnet (self-moderated) and my Activity confirms that caching was not used. (Both messages were sent to the same back-end.) Perhaps the web search plugin interferes. I'll experiment further…

tardy mango Jan 5, 2025, 8:12 PM

#

spring ore Nope, Anthropic requires flagging messages (no more than 4) with an "ephemeral" ...

Oops. Turns out I didn't lnow what I was talking about

hoary nacelle Jan 5, 2025, 8:25 PM

#

@spring ore in my experience, the only way caching discounts will be applied in the openrouter chat interface is if the provider enables it automatically. im not sure web search interferes, ive never seen cache discounts being applied in any of my conversations with claude models in the activity section when using the openrouter chat room. the most frequent provider i see cache discounts applied to is the openai models, but this is anecdotal. just my experience.

spring ore Jan 5, 2025, 9:05 PM

#

Makes sense. Okay so I guess my question is two-fold then:

Is there a user's guide for https://openrouter.ai/chat which details all its features to make sure I'm not missing out?
I guess I should file a feature request to get some kind of checkbox when sending a prompt if I want it to be a caching checkpoint. 🤔
I realize I could just use another chat client, but OpenRouter's own is becoming good enough to not want to bother. 😉

#Chat guide?