invalid characters in json response | Bun | Page 1

knotty sandal Oct 14, 2022, 10:37 PM

#

I have a HonoJS webserver that returns a random st string from a json file. Sometimes it returns an invalid json response with "soh" or "nul" characters inside. I've checked the file and it's not the issue, so I believe it to be an issue with bun.

#

below is the code i use for the api route

const jokes = await Bun.file("jokes.json").json()

app.get("/api/joke", (ctx: Context) => {
    return ctx.json({
        status: 200,
        joke: jokes[Math.floor(Math.random() * jokes.length)]
    })
})

proper reef Oct 14, 2022, 10:40 PM

#

I'm seeing this as well ^^^ (not using Hono/another external lib, rather using Bun.serve directly)

knotty sandal Oct 14, 2022, 10:41 PM

#

i only started getting this issue after upgrading to 0.2.0 i believe

proper reef Oct 14, 2022, 10:41 PM

#

When trying to parse the JSON via Go: invalid character '<' looking for beginning of value; doesn't seem to happen initially, but rather after ~10 min of pinging endpoint every 100ms

#

And can confirm this is new as of v0.2.0 for me as well ^^

soft tree Oct 15, 2022, 2:04 AM

#

interesting

#

If you make the request body > 96 bytes long, does it still happen?

#

I’m surprised all the tests I wrote didn’t catch this

#

Maybe the tests check long input too much

#

Instead of short input

proper reef Oct 15, 2022, 2:09 AM

#

soft tree If you make the request body > 96 bytes long, does it still happen?

I’m hitting it running Shumai distributed training benchmarks

soft tree Oct 15, 2022, 2:10 AM

#

96 is relevant because there’s an optimization where it stores the body inline in the Response object itself if it’s less than or equal to that

#

so I’m wondering if somewhere in there it’s a pointer to stack memory which is dead

proper reef Oct 15, 2022, 2:10 AM

#

soft tree 96 is relevant because there’s an optimization where it stores the body inline i...

Bet, let me test when I’m back on my PC

soft tree Oct 15, 2022, 2:11 AM

#

and this is specific to JSON right

#

Or are you using ArrayBuffer for your training data?

proper reef Oct 15, 2022, 2:13 AM

#

soft tree Or are you using ArrayBuffer for your training data?

We’re using arraybuffer, I think that’s playing a part in the issue for sure

soft tree Oct 15, 2022, 2:13 AM

#

It’s most likely something with this 96 bytes thing

proper reef Oct 15, 2022, 2:15 AM

#

For context, I’m working on a dashboard to chart analytics for distribute training network built w shumai, so frontend pings the central server /statistics endpoint 10x per sec from Go. After running the distributed demo for a bit, this error occurs in Go when requesting stats

#

Yea, that would def be consistent with my experience testing earlier today trying to ID the issue ^^

proper reef Oct 15, 2022, 3:33 AM

#

soft tree It’s most likely something with this 96 bytes thing

So, printing the # of bytes of the response Body in Go, it looks as though the stats JSON is ranging between 278 to 283 bytes give or take prior to feeding data into the network; after feeding data into the network, response body is a lot larger as we're returning stats on each operation (as well as which device executes the operation), bytes: 21069.

I'm gonna dig into this more; it's definitely not appearing right away, but after running for a bit. I'll typically run the demo client script, bash examples/distributed/client.sh (which starts 3 clients that feed data to the network), a few times before I start hitting this ^^

proper reef Oct 15, 2022, 3:57 AM

#

Trying to replicate, I got a BusError running the same demo (and pinging stats endpoint @ same freq from frontend) as previously:


BusError at 0x14C62D5EF7040000


----- bun meta -----
Bun v0.2.0 (b542921f) macOS Silicon 21.6.0
AutoCommand: 
Elapsed: 1528941ms | User: 89753ms | Sys: 59767ms
RSS: 1.56GB | Peak: 1.93GB | Commit: 0.39GB | Faults: 3
----- bun meta -----

Ask for #help in https://bun.sh/discord or go to https://bun.sh/issues

soft tree Oct 15, 2022, 4:00 AM

#

i think bun needs to start running longer tests

#

all the tests run for like 1 minute max

#

that's my best guess for why the current tests aren't catching this sort of thing still

#

will play around with this in a bit

#

i'm currently implementing websocket server

#

almost done

#

or at least, almost have a working prototype

proper reef Oct 15, 2022, 4:09 AM

#

soft tree that's my best guess for why the current tests aren't catching this sort of thin...

That would definitely make sense

soft tree Oct 15, 2022, 4:11 AM

#

out of respect for your time -- i don't think it's worth digging into more unless doing so with a debugger connected to Bun. I'm happy to dig into this later this evening with a debugger

proper reef Oct 15, 2022, 4:11 AM

#

soft tree i'm currently implementing websocket server

Oh hell yea; we're definitely really looking forward to getting to leverage Bun WS server w Shumai haha

proper reef Oct 15, 2022, 4:13 AM

#

soft tree out of respect for your time -- i don't think it's worth digging into more unles...

Yea, was having issues w lldb attaching last I tried (using bun-profile); and no sweat just having fun playing w Bun + Shumai + Svelte

#

Addtl context, and potentially save time in debugging this a bit, you can increase batch size in the demo in examples/distributed/data.ts from const input = sm.randn([1, 8]),such const input = sm.randn([32, 8]) causes it to crash w BusError faster (w/o pinging /statistics from the shumai-dataviz app); think we might have actually hit this BusError when stress testing this demo on prev versions of Bun, so not necessarily introduced recently


BusError at 0x14C62DDEE7020000


----- bun meta -----
Bun v0.2.0 (b542921f) macOS Silicon 21.6.0
AutoCommand: 
Elapsed: 5023ms | User: 826ms | Sys: 410ms
RSS: 0.17GB | Peak: 0.17GB | Commit: 54.35MB | Faults: 0
----- bun meta -----

Ask for #help in https://bun.sh/discord or go to https://bun.sh/issues

soft tree Oct 15, 2022, 12:46 PM

#

haven't been able to repro yet, trying to install shumai says "unable to run on this architecture"

#

not sure if that's a bun problem or a shumai problem

#

i tried just sending gigabytes of data

#

and it didn't do anything

proper reef Oct 15, 2022, 3:20 PM

#

soft tree haven't been able to repro yet, trying to install shumai says "unable to run on ...

Oof, yea I haven't seen that one before (running locally on Darwin/arm64)

soft tree Oct 15, 2022, 3:21 PM

#

Same

#

maybe it’s a bun problem? Maybe the file failed to download?

#

Or like was corrupted?

#

WebSocket server seems to work now btw, though it has barely any test coverage. Fortunately it’s a lot less code than the rest of the http server, it mostly relies on uWebSockets directly

proper reef Oct 15, 2022, 3:23 PM

#

soft tree WebSocket server seems to work now btw, though it has barely any test coverage. ...

Is it working in current release of Bun so that I can play around w it now or need to wait on canary release? (If it's the former, ik what my weekend plans look like now haha)

proper reef Oct 15, 2022, 3:24 PM

#

soft tree maybe it’s a bun problem? Maybe the file failed to download?

I think @trail sun or @rapid sail would likely be able to provide more insight into the install failure ^^

soft tree Oct 15, 2022, 4:07 PM

#

proper reef Is it working in current release of Bun so that I can play around w it now or ne...

bun upgrade —canary

knotty sandal Oct 15, 2022, 4:10 PM

#

I've also tried the canary build and I'm still facing the same issue.

soft tree Oct 15, 2022, 9:23 PM

#

knotty sandal I've also tried the canary build and I'm still facing the same issue.

Can you give me a complete reproduction?

#

That’s the best way to help me get this fixed

knotty sandal Oct 15, 2022, 9:29 PM

#

above is the issue and below is the server code i use:

import { Hono, Context } from "hono"

const app: Hono = new Hono()

const jokes = await Bun.file("jokes.json").json()

app.get("/", (ctx: Context) => {
    return ctx.json({
        status: 200,
        joke: jokes[Math.floor(Math.random() * jokes.length)]
    })
})

export default {
    port: 1234,
    fetch: app.fetch,
}```

soft tree Oct 15, 2022, 9:29 PM

#

Can you include the json file?

knotty sandal Oct 15, 2022, 9:29 PM

#

sure

#

📎 jokes.json

soft tree Oct 15, 2022, 9:32 PM

#

Thank you

knotty sandal Oct 15, 2022, 9:36 PM

#

no problem

soft tree Oct 16, 2022, 2:47 AM

#

I think I narrowed it down

#

There are two issues

#

One is something with invalid data in some cases

#

Another is worth an issue with error handling code changes

soft tree Oct 16, 2022, 4:36 AM

#

This is definitely still happening...but only on release builds

#

It smells like a memory aliasing bug

#

okay

#

I have narrowed it down to

#

The bug is in fetch

#

not in the HTTP server

#

which is good

#

much lower impact

#

sitll important

knotty sandal Oct 16, 2022, 12:39 PM

#

any eta on this getting fixed?

soft tree Oct 16, 2022, 12:41 PM

#

I fixed a crash that could happen in this case but as far as I can tell it’s currently an HTTP client issue and not a server issue. It doesn’t repro unless I’m using bun’s fetch and not node/Deno fefch

#

But

#

That might be because I did fix it and what I’m seeing is something unrelated

#

Can you give it another try in canary?

#

bun upgrade —canary

knotty sandal Oct 16, 2022, 6:24 PM

#

soft tree bun upgrade —canary

it didn't help unfortunately.

#

#

still returns response with invalid characters

#

#

output of uname -a if it helps ```Linux mail 6.0.1-1-default #1 SMP PREEMPT_DYNAMIC Wed Oct 12 09:01:29 UTC 2022 (94ab0ef) x86_64 x86_64 x86_64 GNU/Linux

soft tree Oct 16, 2022, 11:34 PM

#

I'm better able to repro now

#

it only happens when the contents changes

#

like if you do jokes[0] it doesn't happen

soft tree Oct 17, 2022, 2:13 AM

#

alright

#

fix is pushed

#

@knotty sandal please retry after this is green

#

(~30 minutes)

soft tree Oct 17, 2022, 2:44 AM

#

hm

#

it might not have fixed it

soft tree Oct 17, 2022, 3:31 AM

#

alright

#

It is fixed

knotty sandal Oct 17, 2022, 12:39 PM

#

it seems to be working now

#

thanks

trail sun Oct 17, 2022, 4:13 PM

#

soft tree haven't been able to repro yet, trying to install shumai says "unable to run on ...

if you're on arm linux, bun add @shumai/linux_arm64_shumai_flashlight_cpu to shim in a working backend

#invalid characters in json response