#tal-wordware_api
tepid windBOT
👋 Welcome to your new thread!
⏲️ We'll be here soon! Typically we respond in a few minutes, but sometimes we might take a bit longer if the server is busy or if you have a particularly tricky question.
⏱️ We close idle threads, which makes them read-only. Once a thread is closed it won't be reopened, but you can always start a new thread if you have another question.
🔗 This thread will always be available, even after it's closed. You can find it again using Discord's search, or you can save this link: https://discord.com/channels/841573134531821608/1359292411536216185
📝 Have more to share? Add more details, code, screenshots, videos, etc. below.
Below are links to other discussions we've had with you in the past week in case you want to review that information. If your question is related to one of these previous discussions, please provide a comprehensive summary of the current state and what you need help with now. We help many users simultaneously, so a summary allows us to resolve your issue as soon as possible.
- tal-wordware_api, 17 hours ago, 5 messages
- tal-wordware_error, 1 day ago, 31 messages
daring garnet
Oh shoot so sorry. I'm digging into this now.
sly rune
If helpful, here are the metering events coming in for customer stripe-18 in test mode, and the equivalent preview invoice showing no usage
sly rune
Not entirely sure
We can probably go generating invoices for everyone and find the most recent, unless there's a better way at finding that out
daring garnet
So sorry.
So metered events can take a bit of time to flow through to being available, though creating a (real) invoice, or updating the Subscription, should trigger a 'recompute' and let you see them ~right away.
You can also use a Test Clock to be able to go forward in time and create the actual next Invoice, to confirm that the metered events land as expected.
sly rune
Wait we're relying on preview invoices to know if users are out of credit, and the docs say preview invoices are eventually consistent with a max 1 minute delay. Is that not the case?
daring garnet
It says max 1 minute?
sly rune
I can see if I can dig up the source, but this was the number I had
daring garnet
Ok. It says 'asynchronous' but isn't specific beyond that, at least as far as I know (but I could be wrong).
Did these work previously? If so when did they stop working?
sly rune
I ran a series of simulations last night testing this that seemed to work fine
The openai line is our calculated results based on preview invoices and credit balance summaries
daring garnet
And that's not working today?
sly rune
The lines below are my eyeball for what they should be, where I stopped updating them past the invoice expiration
Yes newly not working
daring garnet
Can you share the code behind that?
sly rune
I'm rerunning the simulation I just sent you now
Here:
Ok this is working
Something is different between running this simulation, which works, and manually adding metering events through the dashboard, or through the typescript API, which doesn't
sly rune
No the dashboard-added events don't show up in the invoices
But they do show up in the metering events activity page
meter is mtr_test_61QhYlTu7O8BceebX41LnmNuDWyYU1HM
price is price_1PbZBtLnmNuDWyYUqNRFVjzV
And you can see that price I'm reading from is connected to the meter I'm writing to
daring garnet
Wait, so this works in Python but not in Node?
sly rune
seemingly
Something seems fishy
Both sets of events appear under the meter, but only the Python ones make it into the preview invoice
daring garnet
You're doing something with Credit Balances there too, that you aren't doing with the other, ya?
sly rune
I created a new metering event via CLI while the simulation was running against the ephemeral customer created by the simulation
--event-name=llm_usage \
-d "payload[value]"=2500000000 \
-d "payload[stripe_customer_id]"=cus_S5xaVVqgtnukhp```And did those get included?
sly rune
It appeared in the dashboard but not the simulation results, which were calculated from preview invoices and credit balance summaries
daring garnet
Ah
tepid windBOT
sly rune
Found something?
tepid windBOT
spice comet
👋 Hi there. Taking over and gimme some time to catch up
sly rune
Perfect timing lol, I get the sense @daring garnet had just figured this out 😄
There may be an issue with Stripe API v18 where metering events from the new API don't get metered against accounts correctly if the account hasn't been upgraded to version v18 in prod
Thought it's certainly possible it's not related to the API version
spice comet
Hi there, normally the Meter events are asynced and can be reflected in 3 minutes. However if you use Test clock, you may want to wait for it reflected before moving up the Test clock
sly rune
Looks like the delay is at least a few hours
I'm not seeing any new events since earlier today
spice comet
We would want to look a bit closer into that. Are you seeing it in Live mode or Test mode, or both?
sly rune
Or rather the metering events appear but they're not being priced into the invoices
Both
spice comet
On Test mode do you use Test Clocks?
If you use Test Clocks and advance the simulated time for 10 mins, then 30 mins or more, do you see it is reflected on Upcoming Invoice?
sly rune
Not currently using test clocks, I can try
Just created user Preview Invoices Test (cus_S5yezDgBXmDKTR) and used this:
--event-name=llm_usage \
-d "payload[value]"=1234567890 \
-d "payload[stripe_customer_id]"=cus_S5yezDgBXmDKTR
Looks like the usage was successfully logged
And incorporated into the upcoming invoice
Now I'm going to try with a different customer (cus_S5u0VTUGggLfMo)
^ Before
Created metering event with CLI:
--event-name=llm_usage \
-d "payload[value]"=1234567890 \
-d "payload[stripe_customer_id]"=cus_S5u0VTUGggLfMo```And didn't work
I did exactly the same process with both, except the one that worked was with the test clock (didn't advance it though, left it at the same time), and the one that didn't work was with an existing customer on the same subscription type
Any idea what's going on?
spice comet
Hmm so as long as Test clock chime in you suddenly have it working, but as of a normal/regular flow you never see the events
Weird...
It does make sense though, because I used Test Clock and that worked for me
sly rune
I just bumped our API version in Workbench to 2025-03-31.basil
To see if that makes a difference
We've already switched all our requests over
spice comet
Which version were you on before?
sly rune
2023-10-16
spice comet
Um I am on 2024-11-20.acacia and it's working for me
sly rune
Can you try 2025-03-31.basil?
spice comet
Sure. But before that, could you share the Subscription Id which didn't work for you?
sly rune
sub_1RBiDSLnmNuDWyYUQ3Q4dySX
spice comet
Umm I see
sly rune
?
spice comet
I mean I can see the meter is not reflected in your Upcoming Invoice. Trying again in on my end with 2025-03-31.basil
sly rune
One thought I had is there could be some constraint on the preview invoice endpoint that causes it to break after too many calls, cause we're using it quite heavily
spice comet
Like how heavy is it 🙂 I think there isn't any constraint but trying to gather as much as I can
Hi, could you try (just try) to have a Price with much larger unit amount? ie $5 per usage
sly rune
We bill for LLM token usage, and many models are a few cents per million tokens, so we internally represent prices in billionths of a dollar per token 🙂
sly rune
Ah you mean send a test metering event against a new test price with a much higher unit amount?
spice comet
Yes
sly rune
I'll do that, but if it's helpful you're totally free to add new products and prices on your end for testing
As long as it's in test and not live mode
We're robust to the existence of new products and new prices in existing products as long as the new prices aren't set to be default
spice comet
We basically won't and shouldn't do anything on your account 🙂
We only see the objects and the view as read-only
sly rune
Just sent a metering event, about $6.2B charged to Tom Smith
Ok that worked, though it took 30+ seconds
(to appear on the preview invoice)
sly rune
Created the meter event via CLI on version 1.26.1 which is from 5 days ago
But I wonder if it's about the fact that Tom's new
Ok that worked and appeared on the prev inv after 50 seconds
So perhaps it's subscriptions created before the API version bump?
spice comet
To sumarize:
- 2023-10-16 / Live mode -> not reflected
- 2023-10-16 / Test mode -> not reflected
- 2023-10-16 / Test mode + Test mode -> reflected
- 2025-03-31.basil / Test mode -> reflected
Does this look correct?
Hmm
If you report to any Subscription created before 2025-03-31, no matter which Price, mode, it won't be reflected?
sly rune
Let's find one
Mar 6
Just sent the metering event w/ CLI
Though that test customer also has a different seat-based subscription item to go with the usage-based subscription item we're metering on
Ok that wokred
*worked
spice comet
Weird this is definitely something on and off
sly rune
We're doing our next press launch on Thursday morning, can we rely on this?
spice comet
That's close. Can we do another test? Looking back at sub_1RBjGhLnmNuDWyYUvjxvHPYq which you saw the issue earlier
Can you try again now? register one more usage
sly rune
Alternatively do you know of another way of knowing how much of a customer's credit grants they've used in the current billing cycle or if they're out of credit grants? We're doing prepaid billing using credit grants
Sure here's before for sub_1RBjGhLnmNuDWyYUvjxvHPYq
Sent
spice comet
(Could you share the sent request id as well? req_xxx)
spice comet
Umm same Meter, just different Customer and different Subscription
sly rune
The only thing I changed in the request via CLI was the stripe customer id
First and second
spice comet
Now let's use cus_S5uv8L6MqJ0Vcd (the problematic one), would you be able to create a new Price + Subscription + Meter?
Keep using the exact same customer, but generate a completely different set of (Meter/Price/Sub)
Sorry for all the trouble. We are really stuck on own side as we can't reproduce, and seems it is also only 1 pattern in your account
sly rune
Alternatively do you know of another way of knowing how much of a customer's cre...
Definitely still curious about this
Wait a minute
I could've sworn I saw the usage appear on here
Just re-sent the request, and it worked again, we'll see if it disappears
Very strange. I re-ran the request from my request history, so you can see that it was sent and received, but it was at zero until I sent it again, and it may or may not have originally worked. Very odd
spice comet
So the 2nd time after you saw it in above screenshot, is it still there?
Something stucked there and this cleared it probably?
sly rune
Looks good now
For the test price + meter + existing customer
I can try a metering event matching our actual meters/prices
spice comet
Now can you back to the original Meter which didn't work, send an event there
sub_1RBjGhLnmNuDWyYUvjxvHPYq
sly rune
So I never removed the first one, just added the second and metered against it
just sent to the new one
*old one
Before
req_Q2m7tgRHOL6oS4
spice comet
This is odd
So I thought
- Meter isn't the issue, because Meter works with different Customer
- Customer isn't the issue, because Customer works. with different Meter
Now can we test with the same customer, the same Meter but with a new Price?
If that also works, it means only a combination of this Customer + this Price + this Meter can rerproduce
Btw, do you have any other non-working example of Subscription? Beside sub_1QzmjrLnmNuDWyYU7ZFNDUof
spice comet
Ty!
sly rune
sub_1RBpgOLnmNuDWyYUwDIfhMNv
Sending request now
Before
So this should be interesting as there are now two prices connected to this same meter
^ New subscription with new price on old meter
spice comet
Oh that works
sly rune
^ Old subscription with old price on old meter
spice comet
Uhhmm so we have been probing enough to say this is the combination of the exact Subscription sub_1RBjGhLnmNuDWyYUvjxvHPYq
Sorry the last question, did you notice this happen on any other Subscription as well?
tepid windBOT
spice comet
Sorry I looked back and at a time you mentioned it worked in Python but not in Node. I wonder if you use Python, on the exact problematic Subscription + Meter + Customer, does it work for you?
To rule out the differences between programming language
(I believe languages doesn't matter though)