#Do you know roughly how many tokens/
gloomy stirrup
Unfortunately not offhand, but I can get you it! Is there a method to check the count by chance? Using both the vertex and generativeai libraries so either will work.
wary mulch
There is a countTokens endpoint, and I'm pretty sure libraries on both platforms give a function to use it.
gloomy stirrup
How would I include function defs as well?
Because it seems my prompt is only ~2000ish tokens
wary mulch
That's a quesiton I've been wondering, too.
How many function definitions do you have?
calm cloud
not a definitive answer, but tools are not included in the contents field of the API, so I'm not sure if function declarations are relevant for input tokens. will follow up if I hear otherwise. https://ai.google.dev/api/rest/v1beta/models/generateContent
surprising if the prompt length alone is causing issues. I assume you're using 1.0-pro if you're using function calling?
gloomy stirrup
Correct
gloomy stirrup
That's a quesiton I've been wondering, too.
How many function definitions do you...
Just 1, and it's not crazy long either.
gloomy stirrup
surprising if the prompt length alone is causing issues. I assume you're using 1...
Yeah, when I do 1 item, it's fine, but 4-5 items (which are JSON blobs) + the prompt seems to be a no-go
calm cloud
Are you getting a specific error?
gloomy stirrup
[<AioRpcError of RPC that terminated with:
status = StatusCode.INTERNAL
details = "An internal error has occurred. Please retry or report in https://developers.generativeai.google/guide/troubleshooting"
debug_error_string = "UNKNOWN:Error received from peer ipv4:[REDACTED IP]:443 {created_time:"2024-03-27T23:37:35.6521582+00:00", grpc_status:13, grpc_message:"An internal error has occurred. Please retry or report in https://developers.generativeai.google/guide/troubleshooting"}"
>]```Hm, are you able to share the specific prompt that causes the error for me to reproduce? You can DM or I can give you an email to send it to, if you prefer.
gloomy stirrup
Absolutely! I'll send you a DM
calm cloud
not a definitive answer, but tools are not included in the `contents` field of t...
Following up. 🙂 function declarations/tools do take up tokens in the input, but the current count tokens call does not reflect them.
wary mulch
Any good method to estimate? (Send it with the JSON of the tools as a text part?)
calm cloud
I can check what the best way is, but that does sound reasonable. Do you have a specific use case or goal in mind?
wary mulch
What's the use case for having a countTokens endpoint at all? {:
Broadly speaking, I'm looking at building something with a lot of function definitions. And I don't have a sense of how big they can get.