Grom - a Gleamy Discord API Library 🌟

[nocturne ridge]

Hello everyone!

Over the past couple months (I think since April), I've been working on my biggest project to date. 👻

Grom is a Discord API library for Gleam! It aims to be the de-facto replacement of discord_gleam, which has been lacking in my opinion.
It features:

• A gleamy API
• ~95% coverage of the REST API, ~99% coverage of the Gateway API ||(numbers taken straight out of you know where)||
• Modern Discord features, such as Components V2 and fully featured interaction support.
• Easy intent/other bitflag management

Grom is very experimental, so I probably wouldn't migrate MEE6 to it
Grom is also quite unstable by design, there are many changes that happen to the Discord API that constitute breaking changes in Gleam. ||(one of them being the recent addition of new permissions, so be prepared for v2.0.0)||
I've been working very hard for it to work well, and in my opinion, it is ready to have its first big bot written using it!

https://hexdocs.pm/grom/index.html
https://gitlab.com/grom-gleam/grom

[shadow pilot]

Whoa!!!

Amazing

26k lines, pretty big!

It'd be cool if the functions had type annotations, it's a bit harder to read without them

[nocturne ridge]

Thanks! 💜

[nocturne ridge]

Yeah, some are missing. I could probably do a quick sweep and put them wherever they're missing

[shadow pilot]

There’s a code action to add all missing ones in a module in the RC version of Gleam atm 👀

[ember hornet]

this is so exciting

[raw rock]

That looks nice!
I spent this weekend writing a bot in Python that takes rust code blocks and returns a link to the gleam playground.
I’ll try rewriting it with this and see how it goes!

[nocturne ridge]

Released the earlier-mentioned v2.0.0

[nocturne ridge]

Thanks a lot by the way! I'd probably still be at it without this tip lol

[tired socket]

great work!!!

[mild lava]

Hell yeah! Would love to try migrating my yugioh bot.

[dawn surge]

When will sharding be implemented?

[nocturne ridge]

Soon, I hope 🙃 I'll probably get around to it soon

I really hope I'll be able to write it without having to rewrite the entire gateway

I already did that once, lol

[dawn surge]

The creator of discord.js rewrote the code for the voice getaway 8 times before being satisfied with it

I feel like rewriting is just inevitable

I rewrote my discord bot 8 times too, ironically

[nocturne ridge]

And then discord added E2EE lmao

[mild lava]

Oh yeah, I need sharding 😬

[ember hornet]

used this to make a little (but very helpful to me) bot!
https://github.com/graphiteisaac/pablo_pixarto

[shadow pilot]

That’s lovely!

[nocturne ridge]

Awesome!!

[timber mortar]

just had an idea for a little personal-use discord bot last night, so this is incredible timing haha

I'm looking forward to using this!

[nocturne ridge]

Just pushed v3 with sharding support, and some more goodies!

Next on the roadmap: typed snowflakes, maybe voice gateway but I don't know if I'm smart enough to do it

Note: Help wanted! Please try to replicate and help fix this thing if possible. I found this nasty bug and I have no idea what to do with it. It's very random from what I've experienced.
https://gitlab.com/grom-gleam/grom/-/issues/13

[timber mortar]

hari@seafoam ~/d/g/izzy (trunk)> gleam update grom
  Resolving versions

The following dependencies have new major versions available:

Package  Current  Latest
-------  -------  ------
grom     2.0.0    3.0.0

🫡

[nocturne ridge]

Mandatory 3.0.1 release with a README update because that gets bundled to hex 😂

[granite cloud]

gleam docs publish 🙂

[nocturne ridge]

TIL

[timber mortar]

Error(CouldNotDecode(UnableToDecode([DecodeError("Dict", "List", []), DecodeError("Dict", "List", []), DecodeError("Dict", "List", []), DecodeError("Dict", "List", []), DecodeError("Dict", "List", []), DecodeError("Dict", "List", ["bot"]), DecodeError("Dict", "List", ["system"]), DecodeError("Dict", "List", ["mfa_enabled"]), DecodeError("Dict", "List", ["banner"]), DecodeError("Dict", "List", ["accent_color"]), DecodeError("Dict", "List", ["locale"]), DecodeError("Dict", "List", ["flags"]), DecodeError("Dict", "List", ["premium_type"]), DecodeError("Dict", "List", ["public_flags"]), DecodeError("Dict", "List", ["avatar_decoration_data"]), DecodeError("Dict", "List", ["collectibles"]), DecodeError("Dict", "List", ["primary_guild"])])))

@nocturne ridge is this a bug in Grom?

  echo current_user.get_guilds(client, [])

This is what I'm trying to do

[nocturne ridge]

definitely a bug on my part

gonna fix hold on

may me had a stroke or something, how is get_guilds supposed to return a user 😭

[timber mortar]

happens to the best of us

[nocturne ridge]

version 4 it is

[timber mortar]

is it really a major version if you're changing a function signature that was fundamentally broken?

[nocturne ridge]

semver says so

backwards incompatible bugfix

[timber mortar]

like, it's not a breaking change because no one was using get_guilds in a way that worked

[nocturne ridge]

🤔

[timber mortar]

can't break something that's already broken beyond use

[shadow pilot]

Changing a type signature would cause programs that reference it to fail to compile

It's a breaking change

But also, you should not be scared to do major versions

[nocturne ridge]

yeah it's just a number in the end

[shadow pilot]

Normally a higher version number means a better quality library, as it shows the author is taking semver seriously and so likely is taking the rest of library authorship seriously

[granite cloud]

i think causing programs to stop compiling is a good thing in that case tbh

[nocturne ridge]

systemd got it right

[granite cloud]

but yeah if you follow semver as written its a breaking change

[timber mortar]

systemd got many things right

[shadow pilot]

It really is not a good idea to break people's projects without warning, come on

You have absolutely no context to say that the programs were broken before, what a silly thing to say.

Libraries cannot know how they will be used

That's how we end up with an npm where everything rots at high speed

[timber mortar]

no fair, I understand all of that. I agree with everything you've said so far, but I was just asking in this case because the function was trying to decode the wrong thing and so wouldn't work at all. i thought that because it was already broken, it wouldn't break anyone's program that was using it because they couldn't be using it

but yeah, i suppose they could pass the function to something without ever calling it and that might break their program?

[shadow pilot]

There's no way to know if it's used or not

(honestly kinda want a world where versioning is done at function level rathre than package level)

[timber mortar]

maybe OT for this thread, but isn't unison doing something like that with their function hashing thing

[granite cloud]

i don't think it's possible to make the problem as bad in gleam as it is on npm 😅
anyways i'll move on

[coral briar]

fn my function() // dont use i'm broken :3
fn my_function_v2(foo: String)
😎

[nocturne ridge]

released @timber mortar

[timber mortar]

Ok([PartialGuild("948857185197248533", "Just me", Some("0ca6e6c86f7a7a8fad4d4066fa6d6ac6"), None, False, [CreateInstantInvite, AddReactions, Stream, ViewChannel, SendMessages, EmbedLinks, AttachFiles, ReadMessageHistory, MentionEveryone, UseExternalEmojis, Connect, Speak, UseVoiceActivityDetection, ChangeNickname, UseApplicationCommands, RequestToSpeak, CreatePublicThreads, CreatePrivateThreads, UseExternalStickers, SendMessagesInThreads, UseEmbeddedActivities, UseSoundboard, CreateGuildExpressions, CreateEvents, UseExternalSounds, SendVoiceMessages, SendPolls, UseExternalApps], [UnknownFeature], None, None)])

[nocturne ridge]

how I love UnknownFeature 🙄

it's either unknown or known but unannounced

[timber mortar]

Thank you! That was an incredible response time

[nocturne ridge]

i really need to keep track of the discord api commits

[nocturne ridge]

yeah i was just about to write a bot for myself 😭

[timber mortar]

same, i had an idea while brushing my teeth two nights ago and i'm chasing it down before the New Idea Motivation dies off haha

[nocturne ridge]

@wet cradle could i nag for some feedback? :)

[wet cradle]

wdym?

it's just like, too low level? dunno how to say it :d

[coral briar]

it doesn't seem any lower level than previous discord libraries i have used, just more verbose since it's gleam/functional

[nocturne ridge]

yeah, i tried to make it as close to an actor as possible, since the gateway is just an actor lol

also you can just not use the gateway if you don't need to listen to events or appear online

[wet cradle]

nah, I don't mind the functional part, but I had to see/use a lot of stuff that I wasn't used, touch some events that are kinda internal, stuff like that

[timber mortar]

one quick question: if I want to send a message I can use the http create endpoint, but to listen to a reply on that message, do i need a gateway or a webhook?

this is very much a docs question, so I apologise in advance, but also the discord docs are a bit overwhelming since apps are so much more complicated than what I need

[nocturne ridge]

yeah, exactly

events are done through the gateway

[timber mortar]

ah gotcha, thanks!

[wet cradle]

btw, the library is cool, I'm just too stupid for it :d

[nocturne ridge]

only some requests are done through the gateway (update presence, update voice state, request guild members, request soundboard sounds)

[nocturne ridge]

if you've ever used an actor, the gateway is just that

the rest are just discord HTTP endpoints

the grom.Client is just an early design decision that i thought would do something but ended up being just a store for the token

[ember hornet]

my pixel art theme bot has been stable since launch now :)

only thing is the online status disappears after not too long, but I assume that's my fault lol

[timber mortar]

  message.create(
    client,
    channel_id,
    message.Create(..message.new_create(), content: Some("hello")),
  )

is this how I'm supposed to use the create endpoint?

[nocturne ridge]

yup

[nocturne ridge]

interesting

do the commands still work? if you have them off

ofc

[ember hornet]

there's no commands, only checking bluesky and posting,the source is available if you'd like to see, but please don't waste any time on it because it works perfectly for what I really need right now ahah

https://github.com/graphiteisaac/pablo_pixarto

[nocturne ridge]

no because that would mean the gateway crashes/fails to reconnect 🫣

i'll put a bot online for a little bit longer, see all the logs

[coral briar]

Was thinking about the voice gateway, I think the simplest way would be to use ffmpeg? I believe nostrum, the elixir library, does this

[nocturne ridge]

I haven't really thought about that yet. I'm sure the open telecom platform has got some cool solutions

[nocturne ridge]

Just dropped v5.0.0:

• New modal components: the radio group, the checkbox group and the checkbox
• New endpoints for invites (discord really vibe coded them, I refused to implement one of them because of horrible documentation. It doesn't seem important though)

[timber mortar]

can't wait to get back to my thing after this semester's done

[nocturne ridge]

I'm gonna write some docs for this

In the meantime

[ember hornet]

amazing!!

I'm still getting the offline status problem with my bot, but functionally it's working great

[nocturne ridge]

broke the gateway with this one somehow
kinda glad nobody uses it yet

[nocturne ridge]

v5.1.0:

• Added gateway.get_shard_for_guild to the public API. Useful for a nice health check command.
• Fixed an off-by-one bug caused by switching to int.range, which caused shard 0 to never spawn

[nocturne ridge]

I'm investigating this rn, after viewing your code I see you're not doing much with the gateway besides online status, which is fine, but that's why you're not seeing any other issues. I started up a bot and am gonna see if I see this issue, this could honestly be due to a reconnect not working. I'll see though. It might be kinda hard because I don't have a VPS at the moment though 😂

[nocturne ridge]

Bit of an update, it's been 2 hours and I've so far had no disconnects, the bot is still online. I'll keep it on overnight, I have a watcher written in go which will notify me if it goes offline

[ember hornet]

thanks so much for looking into it! appreciate it and love the library

it's not a big deal at all anyway because the actual bot functionality has been rock solid

been so solid and a pleasure to work on when I've needed to

[nocturne ridge]

After 2.5 hours at 23:45 it went down. Heartbeats aren't coming out anymore, it didn't emit any messages about reconnecting. My watcher bot had internet, so this isn't that.
Seems like reconnecting is messed up somewhere. I can't put a finger as to whether:

• Discord initiated the reconnect through an event and I failed to respond properly
• I somehow killed the connection with bad code
• The connection was killed prematurely and I failed to respond to that appropriately

I'll probably get on it tomorrow, I've got winter break right now so I can work on this lol
A problem I have is the long waiting time for the issue to occur, 2.5 hours seems reasonable for discord to do this (reconnects are to be expected on the gateway for long-standing connections)

[nocturne ridge]

Yeah, well, your bot isn't using the gateway lol 😅

It would've affected functionality if you were listening for events

[mild lava]

-# Could've been a webhook 🤭

[open ferry]

I'm using grom to make a Discord bot but I couldn't figure out how to get the user who ran a slash command. It doesn't appear to be in Interaction or SlashCommandExecution. Can someone help me?

Wait, I could use the InteractionInfo type

[open ferry]

Is it possible to send a custom message to the returned Subject?

[nocturne ridge]

Not really custom, just the 4 - request guild members, request soundboard sounds, update presence, update voice state

Why do you need custom messages?

[open ferry]

I was thinking of sending messages to users based on a timer, but for that I'd need to access the State

Because I keep the list of users in it

But I'll be migrating to a db anyways, so I don't think I'll need it

[nocturne ridge]

I suppose you could start your own actor which has the gateway connection in its state

[open ferry]

I'll see if I need to do something like this

Another thing, after I get a Channel, is there an easy way to get it's id? One of it's variants contains a Thread, so I can't directly do channel.id

[nocturne ridge]

I think the only way is using a case statement in that situation

[open ferry]

yeah, that's what I'm doing right now. I think a get_id function could be provided

[nocturne ridge]

Decided I'll vibe code the hell out of this one - possible fix: https://github.com/folospior/grom/pull/25
I'll see if it works

Also wondering if switching to github makes sense:

• Gitlab has virtually no AI features (at least those I could use) (I have Copilot Pro)
• Github is more approachable (it's the google of git)
• Pretty much the only reason I chose Gitlab is for the better font

[nocturne ridge]

Update: For the first run, I unfortunately didn't turn on echoing messages to the terminal, so that's a miss. I did however turn on wireshark to see when messages stop and the good news is I noticed that they did stop and switch to a different destination IP + source port, which would mean a successful reconnect. I'll try to reproduce it once again, this time with echoing messages to see what and when I receive/send

On the other hand, we now have resuming working!

[nocturne ridge]

Released as v5.1.1

[nocturne ridge]

I'm gonna take some time to work on this and add some features that I really want:

• merging components into a single module
• typed IDs
• more examples
• maybe more docpages

Also, I'm thinking of removing what I called the half-builder pattern:

client
|> message.create(in: guild_id, using: 
  message.Create(..message.new_create(), content: option.Some("hello")),
)

in favour of the full-builder pattern:

client
|> message.create(in: guild_id, using: 
  message.new_create()
  |> message.create_with_content("hello")
)```

I also would like some feedback from y'all regarding the library, except the chaotic development - what would you like to see?

[shadow pilot]

More examples that explain each concept! The current one is a bit big and hard to understand for me.
And lots of comments added to each block of code in the examples

And annotations added to all functions

[nocturne ridge]

I think I did that a while back, went and did a sweep of all files and just used the annotate all top-level definitions code action

[shadow pilot]

Code-wise I think my focus would be on the API. There's a HUGE number of modules, most of them seem to have nothing in them. It would be great to shrink it down to (ideally) just one public module with a very clear and focused API

[shadow pilot]

In the example I meant there

[nocturne ridge]

Oh yeah

[nocturne ridge]

One module would seem weird to me, that'd essentially be prefixing all the functions

[shadow pilot]

Having loads of modules makes it hard for people to understand the library, and once you do understand it makes it annoying to work with as instead of having 1 import you have to manage many

[shadow pilot]

I've been clicking on modules at random to try and understand and most of them don't have any functions in them?

and the ones that do seem like they could be moved without collisions easily

Most the stuff doesn't have any documentation so it's hard to understand what it is

[timber mortar]

The more conventional builder pattern would be easier to get started with. tbf what's there currently isn't too hard to understand but without documentation I was a bit confused. So I agree with louis, docs would be amazing

[shadow pilot]

Looking at the implementation now, you're mostly not using supervision

[nocturne ridge]

I'll try to take a stab at it, it'll definitely be lots of work though (some stuff is terribly implemented, like query parameters)

[shadow pilot]

Def make sure everything is supervised, and you should have a supervised function be the main entrypoint to the program rather than start

[shadow pilot]

This is such an ambitious project, it's very impressive!

[nocturne ridge]

I'll also try to make it somehow possible for people to choose the HTTP client, right now it's httpc-only, one thing I saw suggested by you in comms was returning a Request object, making the library user send the request and then providing a parse function. This seems kind of convoluted though

[shadow pilot]

This library is too complex for the sans-io pattern really

[nocturne ridge]

Going from

let assert Ok(current_user) = 
  client
  |> current_user.get_user

to

let assert Ok(current_user) =
  client
  |> current_user.get_user_request
  |> hackney.send
  |> user.parse``` really does seem too complex

actually this wouldn't really work because you'd first have to assert the response, but you get the point

[nocturne ridge]

I was thinking perhaps something like wisp_mist/wisp_ewe, with grom_httpc/grom_hackney

Could even be grom_fetch if all you're doing is HTTP requests and not touching the gateway

A JS gateway would be possible but I don't think there are many advantages tbh

Perhaps then a client would actually make sense 😂 Right now it just holds the token

[shadow pilot]

Maybe keep it as is with one client until there’s clear reason to support more

Can always add it later

[shadow pilot]

I meant for the stuff that is running inside the actors

[nocturne ridge]

Oh I mean that's really just stratus at the moment for websocket clients lol

[open ferry]

I'm having some problems with subcommands, where the subcommand's parameters aren't registered in Discord

Has someone else run into this?

[nocturne ridge]

no, probably not, let me see

yeah that doesn't work

let me see what's going on with the to_json function

so, my to_json function works fine, but discord seems to be rejecting these parameters
perhaps i'm skipping over some required parts of the parameter, and that makes discord just remove them?
honestly, i'd think discord would return an error in this case, but i guess not

ok i found the issue

it's in my to_json

discord calls parameters options in their api
i preferred to rename them to parameters because there's a gleam/option type and it could:
a) be confusing
b) annoy me (i'd have to alias gleam/option in my command module)

[nocturne ridge]

Fixed in v5.1.2

-# after 2 or 3 force pushes to main because i have no idea on how to change versions on my own project

[open ferry]

Thanks!

[open ferry]

Another thing, it looks like you forgot to set the type field while serializing modal components to json

So it doesn't work

[nocturne ridge]

I'll get around to try and fix it soon, PRs welcome tho

[open ferry]

I'm planning to make PRs both to fix this and add supervision. Do you want me to open issues first?

[nocturne ridge]

No need for issues, also I've just gotten around to fixing components, so you can just add supervision if you want :)

Components are fixed on main, looking forward to your supervision PR for 5.2.0 or 6.0.0. Also please note I've switched back to GitHub

[open ferry]

Oh, thanks!

I'll try implementing supervision then :D

[turbid marlin]

isnt that just https://tangled.org/nuv.sh/grom-fluxer/blob/main/src/grom/gateway.gleam#L2450

[nocturne ridge]

i haven't really dived into it, your code seems fine, but i have some concerns:

• the subject to the process which receives user messages is returned in gateway.new_with_initializer
• if we supervise that process and it gets restarted, my understanding is that that subject will no longer be valid, so no user messages will work
• this would create a dependency loop:
  • we need a constant subject which will always receive messages
  • we want supervision, which means that the subject is no longer constant

i'm not really a master of OTP, so if subjects remain valid upon restarts, that'd be cool, but i don't think they do

[turbid marlin]

itd have to be a named subject

i forgot that you can send stuff to the gateway

or well

a name

[nocturne ridge]

my thinking behind just returning Subject(gateway.Message) was that that process simply won't die - its entire job is to receive messages from shards and pass them on to shards, and shards themselves are supervised internally

[nocturne ridge]

yeah i'd have to check out named subjects

[turbid marlin]

https://hexdocs.pm/gleam_erlang/gleam/erlang/process.html#named_subject

[nocturne ridge]

and then just return a Name(gateway.Message) in gateway.new_with_initializer?

[open ferry]

Ideally you’d pass a name to gateway.new

[nocturne ridge]

that's a good idea

allows for multiple discord bots

[turbid marlin]

ye; names should be defined at application startup https://hexdocs.pm/gleam_erlang/gleam/erlang/process.html#new_name

[turbid marlin]

names are always unique
identical calls to new_name will yield different names

[nocturne ridge]

cool cool

makes sense

[nocturne ridge]

I've gotten supervision with named processes working on my local branch, some problems I have are:

• user state resets back to initial state upon restarts
• i would really really like to have reconnects/resumes working on restarts

I mean, reconnects work

not resumes lol

[ember hornet]

what state is resetting?

[ember hornet]

bit of a noob question but any advice here?

[nocturne ridge]

you should do that in on events, the AllShardsReadyEvent is what you're looking for

[nocturne ridge]

technically the entire gateway, so everything that's:

• only for the gateway shard pool, such as user state
• shard state - heartbeat counts and whatnot

[ember hornet]

that makes sense, ty

[nocturne ridge]

or you could make a state variable outside of the gateway initializer

fyi, you don't need the initializer really, you can use gateway.identify_with_presence with your identify object

[shadow pilot]

nit: Shouldn't be using unsupervised processes

[ember hornet]

yeah that's my next step to fixing this up haha

[nocturne ridge]

some bugfixes on v5.1.3

i'm gonna work more heavily on the API rewrite for v6.0.0 (hopefully) soon
should be a totally different API

I spent way too much time typing this message because I was wondering if extending the grom module to be REST-only to make it target-agnostic is possible

It's not, JavaScript function coloring destroys the concept

And in my mind it's not worth it to allow switching between httpc and hackney unless someone really really needs it

[open ferry]

I recently deployed a bot and it’s working great! However, it suddenly went offline, even with the VM on. No error or warning was logged.

Has someone stumbled on this?

[nocturne ridge]

What version are you on?

I thought I fixed that in 5.1.1, unless it's another bug

[open ferry]

5.1.3

[nocturne ridge]

Ooh

That's freaky

How long did it take to go down?

[open ferry]

Well, I don’t know exactly, because I was sleeping, but it took less than 4 hours and 30 minutes

[nocturne ridge]

I know what I'm doing today then

[open ferry]

Is there's anything I can do to help you debug?

[nocturne ridge]

Some info (if you can):

• How many guilds is your bot in?
• How many members total?
• How many members were using the bot (if you know)
• Did you set a shard count manually?

[open ferry]

• 2
• A hundred at most
• 1
• no

[nocturne ridge]

Alright, that's cool then

I'll set up a bot, try to run it for a couple hours and then we'll see

[open ferry]

If you want, the repo is public: https://github.com/mine-tech-oficial/one_on_one

[gentle vine]

It's not 404

[open ferry]

I'll also keep a look at whether it goes offline

oh

done

[nocturne ridge]

Alright, this could be some freaky SSL stuff.
I have this bug in this issue over on the old GitLab repo: https://gitlab.com/grom-gleam/grom/-/work_items/13
But not on the GitHub repo.

I tried to start my test bot up 3 times, and only on the 4th attempt did it not throw me that SSL error.
Either I got insanely (un)lucky with it throwing 3 times in a row, or there was a regression (which I'd assume would be in stratus/gramps)

Whenever Discord reconnects, grom makes a new websocket connection, so this might be the bug.

The warning in the repo is similar to what I have now, but not the same: (new)

=WARNING REPORT==== 27-Mar-2026::11:41:12.074281 ===
Actor discarding unexpected message: Ssl(Sslsocket(//erl(#Port<0.5>), //erl(<0.137.0>), //erl(<0.136.0>), GenTcp, TlsGenConnection, //erl(#Ref<0.1859040561.2961571847.204808>), Undefined), <<129, 126, 0, 128, 123, 34, 116, 34, 58, 110, 117, 108, 108, 44, 34, 115, 34, 58, 110, 117, 108, 108, 44, 34, 111, 112, 34, 58, 49, 48, 44, 34, 100, 34, 58, 123, 34, 104, 101, 97, 114, 116, 98, 101, 97, 116, 95, 105, 110, 116, 101, 114, 118, 97, 108, 34, 58, 52, 49, 50, 53, 48, 44, 34, 95, 116, 114, 97, 99, 101, 34, 58, 91, 34, 91, 92, 34, 103, 97, 116, 101, 119, 97, 121, 45, 112, 114, 100, 45, 97, 114, 109, 45, 117, 115, 45, 101, 97, 115, 116, 49, 45, 99, 45, 122, 119, 49, 108, 92, 34, 44, 123, 92, 34, 109, 105, 99, 114, 111, 115, 92, 34, 58, 48, 46, 48, 125, 93, 34, 93, 125, 125>>)```

I'm worried about the fact that you didn't receive any warning at all. This would suggest a different bug.

I hooked up my bot to the `spectator` tool, I'm gonna keep my PC on for a few hours and see what it gets me.

Discord reconnects are expected, happen once every few hours and (at least in theory) are handled in grom

Well, nothing that screams regression, honestly

Might've gotten unlucky

[nocturne ridge]

That error looks different because I didn't configure logging 😭

[nocturne ridge]

Resume #1 - resumed successfully (11:50 - 13:33)

Neeeeeeevermind

[open ferry]

I just checked and the bot's offline again

[nocturne ridge]

src/grom/gateway.gleam:2995
User(StartSendHeartbeat)
src/grom/gateway.gleam:2995
Text("{\"t\":null,\"s\":null,\"op\":11,\"d\":null}")
src/grom/gateway.gleam:2995
Text("{\"t\":null,\"s\":null,\"op\":7,\"d\":null}")
src/grom/gateway.gleam:2995
User(StartSendResume(ResumingInfo(Subject(//erl(<0.155.0>), //erl(#Ref<0.731568588.1351614468.47151>)), Some(1), "wss://gateway-us-east1-d.discord.gg", "c95adcdb6e60db376b580c04cd807447")))
src/grom/gateway.gleam:2995
Text("{\"t\":null,\"s\":null,\"op\":10,\"d\":{\"heartbeat_interval\":41250,\"_trace\":[\"[\\\"gateway-prd-arm-us-east1-d-h1gb\\\",{\\\"micros\\\":0.0}]\"]}}")
src/grom/gateway.gleam:2995
User(StartIdentify(Welcomed("wss://gateway.discord.gg", Subject(//erl(<0.150.0>), //erl(#Ref<0.731568588.1351614468.47029>)), Subject(//erl(<0.147.0>), //erl(#Ref<0.731568588.1351614468.47008>)), IdentifyMessage(Shard(0, 1), "token", IdentifyProperties("linux", "grom", "grom"), False, None, None, []), Subject(//erl(<0.4340.0>), //erl(#Ref<0.731568588.1351614468.117520>)), None)))
src/grom/gateway.gleam:2995
Text("{\"t\":\"RESUMED\",\"s\":2,\"op\":0,\"d\":{\"_trace\":[\"[\\\"gateway-prd-arm-us-east1-d-h1gb\\\",{\\\"micros\\\":5141,\\\"calls\\\":[\\\"id_created\\\",{\\\"micros\\\":0,\\\"calls\\\":[]},\\\"session_lookup_time\\\",{\\\"micros\\\":3802,\\\"calls\\\":[]},\\\"session_lookup_finished\\\",{\\\"micros\\\":9,\\\"calls\\\":[]},\\\"sessions-prd-gcp-us-east1-d-78\\\",{\\\"micros\\\":15}]}]\"]}}")```

1. shouldn't be identifying when resuming (doesn't hurt, but still technically shouldn't)
2. no heartbeats after resuming

I have no idea whether the TCP connection still exists

Actually I can check that

Technically shouldn't be

[nocturne ridge]

I swear this lib is like a diesel in winter

It takes 3 tries to start up a bot 😭

Looking for recreations: https://github.com/folospior/grom/issues/32

[nocturne ridge]

Wanted to test out a mist server immediately sending a message to a stratus client... remembered there's no SSL on localhost 🙈

[nocturne ridge]

Apparently stratus doesn't like self-signed certificates as well

Aaaand I put claude into a hallucination loop

Well apparently someone was quicker: https://github.com/rawhat/stratus/issues/22

[open ferry]

So it should work with the latest stratus commit?

[nocturne ridge]

No, it's two separate bugs, one on startup and one on reconnect

[open ferry]

Oh, damn

[nocturne ridge]

I had copilot make me a bugfix for the reconnect one, the discord overlords are choosing not to reconnect me though

[nocturne ridge]

We have reconnects fully working

Took just under 2 hours from having the code to confirming it works

[open ferry]

Could you please ping me whenever it's released?

[nocturne ridge]

Out now as v5.1.4

[open ferry]

Let's hope it works now!

[nocturne ridge]

Should work for 2 hours

Honestly my fault for not noticing this the last time

[nocturne ridge]

Right here I should've waited for heartbeats and noticed something was off when there was no heartbeats

[nocturne ridge]

@open ferry Any issues so far?

[open ferry]

No! The bot is working perfectly and hasn’t gone offline!

[nocturne ridge]

That's amazing

[open ferry]

Thanks so much for making this library and being so helpful!

[nocturne ridge]

I hope v6 will bring more joy, lol

Gonna take a couple months for sure

[nocturne ridge]

https://github.com/folospior/grom/issues/33
Made this mega-issue for v6

[open ferry]

How is supervision going?

[nocturne ridge]

I essentially have a plan on how to do it:

• named subject to the gateway actor
• move gateway state to ets
• actor which holds gateway state in ets
• static supervisor (rest for one) - ets + gateway actors
• supervised function which returns a child specification for that previous supervisor

I mostly put it off to the rewrite (which I unfortunately named "6.0.0" on the GitHub)

[open ferry]

Oh, somehow I didn't read that part of the issue lol

[nocturne ridge]

I edited it in so you mightve read it before

[open ferry]

It's great that you have it planned out already!

[nocturne ridge]

Might honestly just make v6 this gateway change and then the rewrite will be somewhere down the line

[open ferry]

oh yeah, just checked the dates and that's what happened lol

Btw, my bot is randomly going offline, although I don't know why. It does keep itself online for longer periods of time tho

I don't know how to debug tbf

[nocturne ridge]

It could be the race condition in stratus

Do you get any warnings?

Also could be the gateway actor dying somehow

[open ferry]

Idk, Fly deletes logs older than 2 days

[open ferry]

I’ll try restarting with spectator to monitor it

[nocturne ridge]

Isn't fly serverless

Well actually shouldn't matter because theres always a connection

[open ferry]

I don’t think so?

[shadow pilot]

What do you mean by serverless?

[nocturne ridge]

that it fires up when called upon

anyway it shouldn't matter much here

[nocturne ridge]

I'll fire up a bot on a VPS so I get my logs stored locally
I assume what's happening is that one of the reconnects doesn't receive the HELLO message from Discord because of the stratus race condition (every reconnection is a new stratus connection)

What to do:

• Wait until https://github.com/rawhat/stratus/issues/22 gets resolved
• If you really can't wait, fork grom, set grom's stratus dependency to the main branch and use that fork as your bot's dependency

The fix is in main, rawhat seems to want to test some more stuff

I'll monitor my bot, but if you don't have the logs this seems like the most probable cause of your issues

[open ferry]

Hey @nocturne ridge, just letting you know that earlier I restarted the bot and I just noticed it's offline, although I didn't check whether it actually went online. I believe the problem might be the race condition. Nothing was logged to the console btw

I guess I'll fork grom in the meantime

[nocturne ridge]

o

Nothing to the console would mean it's not that

[open ferry]

Sorry, I meant no errors logged to the console. The "overwriting commands" log was sent

[nocturne ridge]

That means it hit the READY event

(if that's where your "overwriting commands" part is)

[nocturne ridge]

That would mean there's a similar bug in the reconnect handler

Reconnects and resumes are handled differently

[open ferry]

I don't know what's happening with my bot. I'm trying to start it up and it just doesn't go online, neither with the version on hex or my fork with updated stratus

[nocturne ridge]

Could you try the bugfix/invalid-reconnect-request branch?

[open ferry]

Yeah, I'll try it!

[nocturne ridge]

So far I've had one successful resume on my bot, no reconnects yet

[open ferry]

I'm getting a lot of things echoed

[nocturne ridge]

That's expected

[open ferry]

Mainly messages with t: "READY" and StartIdentify

But it doesn't go online

Let me try running the example bot

[nocturne ridge]

No heartbeats?

Should take ~40 seconds between heartbeats

[open ferry]

Something "heartbeat_interval"?

If so, a lot

[nocturne ridge]

No

Like this

[open ferry]

No

[nocturne ridge]

Can I see? (Censor your token!)

[open ferry]

Is there anything else which I should censor?

[nocturne ridge]

There are also geo-ordered RTC regions in the READY message, if you'd prefer not to leak which one you're closest to

Or at least where your server is the closest to

[open ferry]

nah, np

[nocturne ridge]

Opcode 9 is an invalid session

This suggests an invalid token

[open ferry]

oh, what?

[nocturne ridge]

What's interesting is that it's trying to reconnect with d set to false

[open ferry]

that's weird

let me reset my bot token

[nocturne ridge]

Discord resets bot tokens if you've accidentally leaked them somewhere 😄

Check if you've got a DM from a system account

Could be that

Looks like this

[open ferry]

oh, this is cool

but I received nothing

and it ain't working with the new one

[nocturne ridge]

No idea then, maybe you accidentally removed a character

Hm

Invalid session after Identify + Ready

Oh your bot has grown a little

You seem to require verification for the guild members intent

[open ferry]

what?

[nocturne ridge]

You got the GatewayGuildMembersLimited flag

[open ferry]

it's only on two servers lol

[nocturne ridge]

Got caught on my own lack of documentation

You're right - limited is for bots under 100 servers

Hmmm

Are you sure you don't have two instances active?

You get doubled-up messages

[open ferry]

I don't see how I'd have tbf. The one running on Fly is stopped, and there's only one on my machine

[nocturne ridge]

Could you update and retry

I just added an echo to the close reason 😂

[open ferry]

The close reason is "Already authenticated."

[nocturne ridge]

Looks like a bug, wondering why I'm not getting that though

[open ferry]

I wonder if the Fly.io machine didn't stop gracefully, so Discord thinks it's still online?

[nocturne ridge]

It'd show up as online on discord

Huh

[nocturne ridge]

@open ferry I started up a new branch with just the echo stuff + main, without my reconnect fixes
Could you try bugfix/not-starting-up, but please do gleam clean before running your bot

[nocturne ridge]

I'm wondering why you weren't able to connect before you switched to the bugfix branch

[open ferry]

Huh, it worked first try

[nocturne ridge]

Yeah that bugfix branch introduced another bug

[open ferry]

So main should work too?

[nocturne ridge]

Main will work but you will still have it die after a few hours

And you may encounter the stratus race condition

So basically it mustve been a one-off thing when your bot didn't start up 😄

[open ferry]

Huh, I thought that was fixed?

[nocturne ridge]

It's been "fixed" twice already, lol
You reported it dying again yesterday

Although I'm questioning whether it's an issue with reconnects or another issue with resumes

@open ferry Hi, so what happened with that invalid session thing and doubled-up events is apparently something in stratus' main branch 👀

At least that's what copilot's telling me

Or I'm using stratus main incorrectly

[open ferry]

Oh, that’s interesting! I’ll try when I’m back home

[nocturne ridge]

And now that I somewhat know what's happening, I can't reproduce it

Heisenbug 😂

[open ferry]

I just got back and whenever I run on main it works, but on my fork (with updated stratus) it doesn't, so it's probably stratus

[nocturne ridge]

It's definitely stratus

I was gonna write an issue about it but got carried away

[open ferry]

Guess I'll revert back to the normal version for now then

[nocturne ridge]

I published v5.1.5 which should fix the failure to reconnect issue. Do mind that the stratus race condition is still in there, so you might have to start your bot up a couple times before it actually turns on.

[open ferry]

Can the race condition apply on reconnects?

[nocturne ridge]

Possibly, yeah. I haven't had that happen, so perhaps stratus only needs to initialize TLS once on app startup, but I can't guarantee that it doesn't happen.

Also, v6 will be stratus update, not the rewrite 🙃
Might add supervision properly before, or might just work on the rewrite

[open ferry]

I'll try the new version then. If it disconnects I'll tell ya (if you don't mind)

[nocturne ridge]

For sure

[open ferry]

Although since I restarted yesterday it hasn't come down yet, so the bug is probably less likely to happen

[nocturne ridge]

The worst thing was that the reconnect function returns Nil (because it's used in stratus.on_close), so the error you had just passed quietly

[nocturne ridge]

Reconnects really only happen if your session gets invalidated or hearbeats don't equal heartbeat acknowledgements

All other times it's resumes

[open ferry]

oh, semantics lol

[nocturne ridge]

Yeah, reconnects are supposed to be rare because they cause downtime

Resumes send you the events you missed, so they only cause delays

[nocturne ridge]

@open ferry Any news? I hope it's still up 👀

[open ferry]

It went offline again
Do you think running observer with it will make it easier to debug?

[nocturne ridge]

what's observer?

[white tulip]

https://www.erlang.org/doc/apps/observer/observer.html

[nocturne ridge]

I hate to be testing in prod :(
Tell you what, I'll hook up my test bot to all the bells and whistles and leave it running, I'll come back in a couple days with an answer as to why it's crashing

[open ferry]

Thanks so much! If there’s anything I can do to help, do tell me!

[nocturne ridge]

So far, I've learned:

• we need supervision and ETS state holding, for now I've noticed a shard crash and get restarted, that caused a reconnect though while it could be a resume
• 8 hours in, 1 reconnect and 4 resumes - bot still online
  Will continue to monitor

[shadow pilot]

Remember that a crash is supposed to mean state loss

If state if always retained then it becomes impossible to reset to a known-good state

recovering from state corruption then becomes impossible

Incremental shedding of state is one of the primary purposes of supervision trees

[nocturne ridge]

I don't think I mentioned this before: please, if you want to write a working bot, don't use this (unless you're HTTP-only) - the gateway won't be a fun time. Rewrite underway, ETA: perhaps July/August realistically

But will probably take longer (that ETA is if I work on it, which is hard for me)

[open ferry]

About that. Is it possible to use HTTP if I just need slash commands and message components?

I don't want to rewrite in JS lol

[nocturne ridge]

No, not really. HTTP-only interactions do exist on discord, but I didn't implement them as I thought they're kind of outside the scope (they're essentially webhooks)

https://docs.discord.com/developers/interactions/overview#preparing-for-interactions

[open ferry]

I think I got supervision working locally. At least everything is either a supervisor or an actor (there was before an actor with a supervisor inside, which I don't know is correct)

[open ferry]

It's up on Github, if someone wants to take a look: https://github.com/mine-tech-oficial/grom

I won't open a PR for now, as I don't know whether it's correct

[nocturne ridge]

I doubt supervision is useful at this point. I am at a point where messages just stop coming at a certain point, yet the processes don't die. I don't know if it's my or stratus's fault (probably mine).

iirc I tried to fix this through vibe coding it, but the main branch now has build warnings so I'm not sure if it's good enough 😂

I don't remember what copilot thought was the caveat here, but I'm certain it won't work

I haven't actually checked on stratus v3, I'll fire up a bot on the new version and see, but I'm not really optimistic

Took a few days last time I think

[shadow pilot]

Supervision is wanted for all processes !

[ember hornet]

yeah issues with the heartbeat not withstanding you definitely want supervision

have you looked at what elixir discord libs do? might be a good thing to dig through for some ideas

i've played with nostrum before

[nocturne ridge]

I don't think the issue is necessairly supervision; supervision is implemented in grom at the shard level. What's missing is supervision of the top "gateway-shard manager" actor, to which you send and receive messages.

I can actually see supervision working because the sockets just close for some reason sometimes. (see attch.)

The issue I was describing before was just getting radio silence at one point. Not a supervisor crash log, not an attempted resume, nothing. I have spectator set up with my test bot, and when I checked it, I saw that both the main actor and the shard actor were alive, yet I wasn't getting messages about heartbeats getting sent.

Right now, after about 5 hours of running, it still hasn't reached that point of radio silence. That doesn't mean it won't hit that bug, though I hope my last attempts at vibe-debugging and the latest stratus changes have fixed this.

I was saying that I doubt supervision (at the top level) was useful because I wasn't super into the idea of trying to revive the current version, but I think I'm getting convinced:)

The gateway requires some work, notably:

• actors are used as a way of managing mutable state, so something like the delayed start of a shard has an actor, the heartbeat counter has one, etc.
• supervision doesn't exist at the top level
• shard crashes result in reconnects without resuming the previous session, which means possible missed events

This is a lot of work, pretty much requires rewiring everywhere, so I thought of just ignoring it, calling the thing unusable and fixing it along the rewrite.

I don't think I was very rational in my decision to call grom unusable, while, sure, in its current state (with the radio silence bug), I wouldn't use it for anything, that doesn't mean it can't be fixed, but it came after a few weeks of lack of motivation. Since yesterday I've been working on the rewrite branch and to be honest it was pleasant, so I hope I'll pick myself up to fix up main during the week :)

I don't think I'll touch it if the radio silence bug has been fixed though, seems like that'd just be silly and the internal fixes could be included in the rewrite

But if the bug does exist, I don't think I'll be able to fix it without the gateway module internal restructure

Nevermind, seems like it hit the point of radio silence about half an hour ago, I just didn't notice it🙃

[open ferry]

Mine hit this point too, very odd

Guess I’ll have to rewrite in another language in the meantime

If I can help you with anything, I’d be glad to!

[nocturne ridge]

I killed the "Connection Manager" actor, which was used to keep the stratus.Connection object in sync between the shard and its heartbeat manager, while playing around in spectator.

1. That revived it. (triggered a death chain, restarted the whole shebang)
2. It contained a PID in its state which didn't exist in my process list. Most likely, that stratus connection just didnt exist and grom was trying to send heartbeat messages to an orphaned mailbox. Stratus somehow died and that didn't kill the connection manager/shard.
3. This is actually a debugging failure - I was only logging in one place - the stratus event handler. If I was to log in the heartbeat manager/shard too, I'd see it trying to send heartbeats to stratus.
4. Seems like the fault was actually because of me splitting actors incorrectly, I doubt this would've happened if the same actor held the Connection object, automatically sent heartbeats and listened to events from stratus, instead of silly-split actors.

I'm actually starting to wonder whether this is a bug in stratus maybe

Or maybe I called stratus.stop somewhere

[ember hornet]

this might be heresy to say but is it worth trying to build out your own websocket implementation, or maybe vendoring stratus and trying to fix it?

[nocturne ridge]

There's collie, which passes all the websocket tests. Also, I'll first have to check if this isn't something I'm doing wrong. I'd trust myself less than I'd trust rawhat 🥲

[tired socket]

i didnt look into what exactly is the problem here, but i am certain that in specific cases, stratus can freeze for quite some time, possibly forever. At the comparison https://vshakitskiy.github.io/collie you can clearly see some of the 9.x tests times out

so it is possible that the main issue is within the stratus

[ember hornet]

wow that's a really handy page

how much work is it to swap over from stratus?

[ember hornet]

not hard as it turns out 😅

if anyone would like to try it out by pinning to that github version perhaps, could see if that helps

[tired socket]

yep, i made sure the api is not that different, so it is not hard to migrate when in need

and besides the api is very clean already to introduce something different

[ember hornet]

yeah you've done a great job, it was primarily a :%s/stratus/collie/g

[shadow pilot]

Wow, super clean

"one way to do things" paying off

[nocturne ridge]

Sure, let me try

[nocturne ridge]

These tests mostly relate to large payloads and compression, which I doubt are the case. For one, I don't have compression enabled, and two, the largest messages I receive are probably the READY messages. Collie will be good to have anyway

[tired socket]

mmm yeaah, most of the problem is usually on the large payloads and compression side.. There are still some testcases in which stratus hangs up, but I dont think it is what is happening here

so the communication is usually happening with no fragmented frames from the discord side, right?

i think i need to have more context on what is the problem, i will read the top messages 😅

[nocturne ridge]

Yeah. I think what's more likely is I'm calling stratus.stop somewhere I shouldn't

[tired socket]

for now the only thing i would advice is to try to use collie. in the case the problem will be still there, we can be 100% certain it is not the websocket implementation problem

[ember hornet]

i might be able to spin up a quick bot and run it overnight to see if my fork works

[nocturne ridge]

I'm already running one 🤭

[ember hornet]

awesome

[tired socket]

by looking at the logs, i assume the websocket connection is closing uncleanly or even fail to close properly. I would expect to receive close frame rather than just silent closure of the socket

[nocturne ridge]

Yeah, but it picks up after that and reconnects

[nocturne ridge]

That's no bueno

[ember hornet]

wowzers

[nocturne ridge]

It appears I have around 300k processes spawned

[tired socket]

woa

[nocturne ridge]

Do y'all think adding wisp as a dependency would make the project too bloated? I want to add HTTP interactions later on, and the way I thought of doing that is just creating a wisp middleware responsible for responding to pings and parsing interactions.

[ember hornet]

it does feel quite heavy to me to include that in the main package

i would be putting it in grom_http or something

[nocturne ridge]

Problem with that is I'd have it have access to the grom interaction decoder or have to copy it into that other package

[ember hornet]

why would that be necessary?

[nocturne ridge]

Discord just sends interactions through HTTP instead of the gateway in this case, so in order to parse them, you need a decoder

[ember hornet]

i would opt for copy-pasting

to be totally honest

i just feel like shipping an entire webserver alongside a bot framework, especially when the HTTP interaction api is seldom-used by regular users, would be making some pretty big assumptions for the end-user

I am quite confident in that position because I maintain some bigger bots and I would personally be a bit miffed if I found out my bot lib was pulling in Wisp lol

you could even think about splitting off some decoders and types into their own package if you're too concerned about duplication, but I don't think it would be particularly bad to just copy-paste, especially because it is a public API that is documented (at least theoretically) well

[nocturne ridge]

I feel like splitting decoders and types into their own packages is just like creating a types.gleam file

I'd much rather copy and paste

[ember hornet]

yeah that'd be my choice too

copy paste is goated

[nocturne ridge]

I actually think this can be done without importing wisp with request.map and response.map

I'll have to write something up

I'm having so many ideas and the library barely works at the moment 😆

[shadow pilot]

It's not, having decoders and encoders in their own thing is good because they are useful on their own

[nocturne ridge]

That's one more thing for the rewrite then!

[nocturne ridge]

I've been working on HTTP webhook-based interactions on the http-interactions branch.

I have a problem though: Discord gives 3 seconds from the time they send out an interaction to either respond or defer a response.

The flow goes like this:

• receive a request
• pattern match on the path
• if it's the Discord path:
  • validate security headers (requests are signed with an app-specific Ed25519 key)
  • if all goes well, give the interaction to the library user to handle
  • now, on the library user's side, find the interaction handler actor by name
  • spawn a worker actor
  • that worker child sends a message to itself on startup
  • on message, it sends an interaction response
  • concurrently (after spawning the child), an HTTP OK response is sent in response to the original webhook request

I seem to always miss this 3 second window, and I have a couple theories why:

• I'm self-hosting this bot on a low-power APU and 200 Mb/s bandwidth, with the API proxied through cloudflare
  • Possible problem 1: security validation takes too long
  • Possible problem 2: network is too slow
  • Possible problem 3: spawning the worker takes too long
  • Possible problem 4: cloudflare proxy takes too long

To verify these claims, I'd need a stronger server, and preferably more bandwidth. I could use my personal PC, but that wouldn't change the bandwidth issue.

Could I ask someone to try hosting this on their own? I'm not home right now, so if anyone needs them, I can post my dockerfile and compose.yaml.

It involves hosting a public API with wisp. I have no idea whether discord requires a domain or HTTPS, but I think most people have those already.

The code is in the http-interactions branch, two commits are the most important, one's just a bugfix :)

[nocturne ridge]

I'll try hosting it on my PC, that's gonna make sure security validation and spinning up the worker doesnt take too long since these are both CPU tasks I assume

[shrewd estuary]

add some timings around key parts of the flow, can be as simple as sprinkle some use <- pocket_watch.simple("request_handler") in your code

[nocturne ridge]

I can also get the creation time of the interaction when I get it

[nocturne ridge]

Apparently, the problem was that the Discord API expected me to return a 202, instead of a 200, which makes more sense semantically, but was hidden behind a wall of text on another documentation page 😩

I assume it'll now work on my server, but damn that was a really bad bug, especially with the error message suggesting a timeout (see pic)

[nocturne ridge]

Just pushed grom v6.0.0:

• small breaking change: stratus removed its InitializationError type and I had that in one of my error variants, so that's that
• deprecated grom/gateway (it's broken anyway)
• added a way to handle interactions through HTTP (bots using just commands (which I think is all grom bots at this point) can use this to keep using gleam and grom for their bot struggles)

See the docs at https://github.com/folospior/grom/tree/main/examples/http_interactions (I honestly think these might be the best docs I ever wrote)

[open ferry]

Oh, interesting! Time to rewrite with HTTP then!

I also want to see if I can figure the gateway bug, but that might not be a big deal as you’ll rewrite it anyway

[nocturne ridge]

The rewrite will take a couple months, so if you can figure it out then go ahead lol

[nocturne ridge]

I can't seem to think of a way of doing this without creating looped dependencies, unless I create a true types.gleam where types, decoders and encoders live together

[shadow pilot]

Name it after what it is in the business domain, not the kind of code thing it is

[open ferry]

Hey, I just finished rewriting using HTTP, and it seems to be working!

For some reason it won't change its status to online automatically, but that's fine

It is not possible to use any of the intents with the HTTP api, right?

[nocturne ridge]

Yeah, that's correct. It shows up on the member list in the online section, but doesn't have a green circle

[nocturne ridge]

Nope, the only event you're receiving is InteractionCreated, essentially

[open ferry]

ok, makes sense. Is this a Discord restriction?

[nocturne ridge]

That's just how that endpoint works, lol

Everything else is done through the gateway

[open ferry]

oh, lol

ok, thanks!

[open ferry]

It doesn't even show up as online, but it works 🤷‍♂️

[nocturne ridge]

I think it needs a role

[open ferry]

Btw, has someone tested switching stratus for collie? I saw there was some discussion above, but wasn't sure whether your test was just swapping stratus or had smth else

[nocturne ridge]

Yeah, just switched to collie, ended up getting 300k spawned processes

[open ferry]

oh, that's odd

[tired socket]

So something is wrong with the collie also? 😅 Maybe i will need to look up what was the cause of it

[ember hornet]

I've just written a very very basic bot (literally only handling a couple of events) using collie and it stayed online for 7 hours before I force quit it

so I don't think so

[nocturne ridge]

Really? Mine wasn't handling any events

[ember hornet]

all mine does is echo when a message gets sent, heartbeat, and resume

but it stayed online the whole time, yeah

[nocturne ridge]

Yeah, that's what mine did too

Might just check once again

[ember hornet]

if it's of any help i just have one actor for collie and a gateway handler which spawns handlers with a factory_supervisor

https://gist.github.com/graphiteisaac/acee3f5cced286b9425e56c2d5a019fd

[nocturne ridge]

Ahhh you built your own thing

I thought grom works lol

[ember hornet]

oh yeah I didn't specify that did I

woops hahah

[nocturne ridge]

I had some sleep over it (and an AI chat), and I realized that the way grom is currently structured is actually better than I thought.

I have this idea where grom is just a place where the main primitives live, and other packages, working in a rust feature type of way, provide other functionality, such as REST API calls, the gateway, voice support, etc. This will allow me to create packages that don't pull in dependencies that are useless for many people.

So, for example:

• grom - provides the User, Application, Guild, Channel, Message, etc. types, each in their own module
• grom_rest / grom_api - provides a rate limiting, failure preventing HTTP client, as well as all the REST endpoints, so something like user_api.get_current_user(client: http_client)
• grom_gateway - provides a gateway client, pretty much the same as the grom/gateway module nowadays but, yk, working
• grom_voice - for the future, would provide a voice client
• grom_wisp - would provide a more automated way to handle HTTP interactions (being able to import wisp, it could use a lot less boilerplate)

My questions are:

• Would you consider this a move for the better?
• I haven't previously taken this into consideration, but would you prefer grom/discord_user to grom/user? I feel like a lot of the grom modules are taking up module names that you use in your application, example user, application, role, etc.
• Fun names for every new package or staying with the plain old grom_feature format?

[shadow pilot]

Are these multiple packages?

Not sure what all these grom_* names are

Oh yes they are packages, sorry, bad at reading.

What's the advantage of breaking it up into multiple packages?

RE grom_wisp, how come it is tied to Wisp and not regular gleam_http?

[nocturne ridge]

If you don't use the gateway, you don't need to add it, so no need to have sth like collie as a dependency

[shadow pilot]

That makes sense

What about _voice?

[nocturne ridge]

Right now, I use gleam_http, but there's some boilerplate associated with that. If I use wisp, I can just take in a wisp.Request and return a wisp.Response

[shadow pilot]

Yeah but then it only works with Wisp

A small amount of boilerplate is preferable imo. The README can show you the code to copy paste for popular frameworks

[nocturne ridge]

That'd probably use mug, but I haven't checked it out yet. That's for the far-fetched future

[nocturne ridge]

Oh yeah I just remembered there's glimr (I think that's what it was called)

[shadow pilot]

For the API one, I think that design is not great because it forces a HTTP client, takes control away from the programmer, won't work on both targets, and it's not possible for it to implement rate limiting in a useful way

I think all the API stuff should use the sans-io pattern and should live in grom

Then grom would be useful on all targets in applications

[nocturne ridge]

Discord responds every time with some rate limiting headers that other libraries use to avoid rate limits

[shadow pilot]

Yeah but without information about what the rest of the cluster is doing you can't do rate limiting

You need coordination between the whole deployment

But also, if you use the sans-io pattern then the programmer doesn't have the control taken away from them, so they can do rate limiting in whatever way is best and you don't have to make a super complex system that both takes away control but also lets the programmer have enough input in how rate limiting etc works

imo always do sans-io first

A package that provides all the parsing and serialising and request construction is useful everywhere. It can be the foundation for both applications and also more specialised libraries

The less that is forced on the programmer the better. Libraries have almost no context so they can at-best aim for a lowest-common-demoninator solution, and anyone who can't tolerate that compromise is stuck.

[nocturne ridge]

I guess I could provide a rate limiting client for single-node bots

In a separate package, that is

[shadow pilot]

What would it offer over the existing generic rate limiting libraries?

A really nice thing about not taking control away from the programmer is that they can use generic libraries and things don’t need to be reinvented for each specific library

[nocturne ridge]

1. Specific to Discord's rate limiting system - Discord rate limits are per endpoint & bucket, based on which the bot is supposed to rate limit itself
2. Preventing bad requests - an incorrect token will always return a 401, so we stop the bot from sending any more requests with that token (not related to rate limiting, but still something that should be part of a grom HTTP client)
3. Ease of use - it's much easier to use grom's pre-made rate limiter instead of having to go through Discord's docs and configuring one yourself
4. You can still use your own client - the main grom library would now follow the sans-io pattern, so the grom HTTP client would just be an optional addition

[shadow pilot]

But you wouldn’t be able to use it outside of anything but a single node, which is very limiting

Any production usage would likely be multi-node

If instead of doing that you provided whatever helpers is needed to use any existing rate limiting system then that is less code and works for everyone

What are the discord specific bits? I’ve found rate limiting to be very generic

[nocturne ridge]

I'll give it some thoughts. I think the solution here is a factory supervisor which spawns new children per endpoint, since rate limits are per endpoint. I don't know if I'd want to write my own HTTP client with rate limiting and failure prevention, synced across multiple nodes just to write a Discord bot. Grom is already more complicated than other discord bot libraries, and only very large bots require splitting across multiple nodes (or if you're like really concerned about uptime of your small bot)

[shadow pilot]

You have to because it’s not something a library can do for you unless it is going to force a particular state management system, which isn’t an option

[crude geyser]

@nocturne ridge Hello, is grom broken for now if I just want to listen to events like messages from a server?

[nocturne ridge]

Unfortunately, yeah

[crude geyser]

ah, roger, it seems gleam_discord is out of date as well

think there might not be any working Discord gleam library right now then

[nocturne ridge]

That's correct

[crude geyser]

is it just due to Discord changing their API stuff?

[nocturne ridge]

Nope, my implementation's just buggy as hell and probably shouldn't have been released

[crude geyser]

well at least you tried 👍

[agile basin]

i believe discord_gleam should work rn, if not lmk what issues ur having and ill take a look

[crude geyser]

ok good to know! i actually just assumed it might have had issues

[agile basin]

i havent worked on it for a while but if its broken i can find some time to fix it

[crude geyser]

@agile basin sorry for the bother, but I am trying to run the example for discord_gleam right on the README.md for the github repo and it seems like my bot isn't coming online or responding to the !ping or getting the message. I set logging to debug

   Compiled in 0.75s
    Running discord_scriptable_bot.main
DEBG Starting event loop
DEBG Sending start message
DEBG Received start message
DEBG Creating websocket client builder
DEBG Making request to gateway.discord.gg at 443
DEBG Sent upgrade request, waiting 5000
DEBG Handshake successful
DEBG Calling user initializer
WARN #(charlist.from_string("Actor discarding unexpected message: ~s"), [charlist.from_string("Ssl(Sslsocket(//erl(#Port<0.4>), //erl(<0.127.0>), //erl(<0.126.0>), GenTcp, TlsGenConnection, //erl(#Ref<0.2628963925.475398145.55436>), Undefined), <<129, 126, 0, 128, 123, 34, 116, 34, 58, 110, 117, 108, 108, 44, 34, 115, 34, 58, 110, 117, 108, 108, 44, 34, 111, 112, 34, 58, 49, 48, 44, 34, 100, 34, 58, 123, 34, 104, 101, 97, 114, 116, 98, 101, 97, 116, 95, 105, 110, 116, 101, 114, 118, 97, 108, 34, 58, 52, 49, 50, 53, 48, 44, 34, 95, 116, 114, 97, 99, 101, 34, 58, 91, 34, 91, 92, 34, 103, 97, 116, 101, 119, 97, 121, 45, 112, 114, 100, 45, 97, 114, 109, 45, 117, 115, 45, 101, 97, 115, 116, 49, 45, 98, 45, 113, 107, 103, 120, 92, 34, 44, 123, 92, 34, 109, 105, 99, 114, 111, 115, 92, 34, 58, 48, 46, 48, 125, 93, 34, 93, 125, 125>>)")])```

[agile basin]

hm thats odd, im about to go to sleep but ill take a look tmrw

[crude geyser]

ya np im not in a rush anyways, thank you!

[turbid root]

{"t":null,"s":null,"op":10,"d":{"heartbeat_interval":41250,"_trace":["[\"gateway-prd-arm-us-east1-b-qkgx\",{\"micro\":0.0}]"]}}

btw

[crude geyser]

oh that's what those bytes are, the <<129, 126,... ?

thanks that's helpful actually

wait do i need to open ports on my router or anything for this?

didnt even think about that

[turbid root]

yah, i think the null 0 byte there meant it wasnt rendered as a gleam string but yah

[shadow pilot]

I’m surprised that wasn’t printed as a string

[nocturne ridge]

looks like discord_gleam is using an old stratus version

this should be fixed in v2

[agile basin]

nope

[agile basin]

hm ill take a look at upgrading

[agile basin]

can you upgrade to discord_gleam 2.1.2 and try again?

[crude geyser]

i actually just started looking at this stuff and saw you updated lol

[agile basin]

peak

[crude geyser]

...
DEBG Making request to gateway.discord.gg at 443
DEBG Sent upgrade request, waiting 5000
DEBG Handshake successful
DEBG Calling user initializer
DEBG Gateway text msg: {"t":null,"s":null,"op":10,"d":{"heartbeat_interval":41250,"_trace":["[\"gateway-prd-arm-us-east1-b-n2ws\",{\"micros\":0.0}]"]}}
DEBG WebSocket closing: CustomCloseReason(4014, "Disallowed intent(s).")
DEBG The webhook was closed
EROR Disallowed intents used, did you remember to enable any priveleged intents you used in the Discord Developer Portal (https://discord.dev)? Not reconnecting
DEBG Sending heartbeat: {"op":1,"d":null}
DEBG Sending heartbeat: {"op":1,"d":null}

I need to figure out what to do with the intents for the bot

[agile basin]

you need to enable the message contents intent

[crude geyser]

alright sweet i got it!

[agile basin]

lmk if u have any more issues :D

[agile basin]

updated to v3 and seems to be working fine now, ty

[crude geyser]

question, if I'm running this example that uses the supervisor and then does sleep_forever and run via gleam run in the terminal how do you end it cleanly? When I just do ctrl-c a couple times I think it's causing an issue with a runaway process probably or something thats detached from the terminal so its not actually getting killed

because when I just do ctrl-c a couple times and then rerun I get the same sort of

DEBG Making request to gateway.discord.gg at 443
DEBG Sent upgrade request, waiting 5000
DEBG Handshake successful
DEBG Calling user initializer
WARN #(charlist.from_string("Actor discarding unexpected message: ~s"), [charlist.from_string("Ssl(Sslsocket(//erl(#Port<0.4>), //erl(<0.127.0>), //erl(<0.126.0>), GenTcp, TlsGenConnection, //erl(#Ref<0.2100475275.2268200961.178239>), Undefined), <<129, 126, 0, 128, 123, 34, 116, 34, 58, 110, 117, 108, 108, 44, 34, 115, 34, 58, 110, 117, 108, 108, 44, 34, 111, 112, 34, 58, 49, 48, 44, 34, 100, 34, 58, 123, 34, 104, 101, 97, 114, 116, 98, 101, 97, 116, 95, 105, 110, 116, 101, 114, 118, 97, 108, 34, 58, 52, 49, 50, 53, 48, 44, 34, 95, 116, 114, 97, 99, 101, 34, 58, 91, 34, 91, 92, 34, 103, 97, 116, 101, 119, 97, 121, 45, 112, 114, 100, 45, 97, 114, 109, 45, 117, 115, 45, 101, 97, 115, 116, 49, 45, 100, 45, 57, 49, 49, 106, 92, 34, 44, 123, 92, 34, 109, 105, 99, 114, 111, 115, 92, 34, 58, 48, 46, 48, 125, 93, 34, 93, 125, 125>>)")])```

and it appears the bot never goes offline so its still running somewhere

[agile basin]

huhh

did you update?

[crude geyser]

yes

[agile basin]

i think cleanest is ctrl + c then type a (abort) and press enter, not completly sure though

[agile basin]

hmm

whats your gleam version?

[nocturne ridge]

ctrl + backslash

[crude geyser]

ok interesting if I do ctrl-c i gotta wait a while like 30 seconds or something before rerunning it to not encounter it getting stuck with the ouptut up above but if i do ctrl + baskslash I can immediately rerun and its able to connect no problem

[agile basin]

oh thats odd

[crude geyser]

ya im not sure what that's about

guess ctrl+\ sends SIGQUIT which probably makes a difference

[agile basin]

hm, if i ctrl + z or ctrl + c twice it works fine for me

really odd

[crude geyser]

arch linux but in WSL within the built in terminal in zed editor if that makes any difference lol

ctrl+\ works to quickly restart tho

[agile basin]

almost same as me

im testing on endeavouros (arch) with zed

[crude geyser]

in WSL?

[agile basin]

nah

[open ferry]

Hey @nocturne ridge, today my bot went into prod (well, it’s an internal thing), using the http interactions, and it’s working great!

I haven’t had time to look into the bug, as I originally planned, but the http events is all I need

[shadow pilot]

Awesome!

[nocturne ridge]

I'd love to devote some more time to this library, couldn't have for the past few weeks because of school. I'll definitely work on grom over the summer vacations

[nocturne ridge]

I've been wondering about this for a loooooong time.

• Single-module API - grom.get_user(Id), grom.MessageComponent
  • Hell of a large file.
  • Less organization in the docs (can be remedied with categories)
  • I don't think this fits into "Split modules per business domain" - users are one part, messages are another.
• Multi-module API (less than the current version) - user.get(Id), message.Component
  • Requires really small modules for some stuff - SKUs, the singular RestError type, etc.
  • Actually splits stuff per business domain

In both cases, additional modules (like grom/gateway) would be added via other packages.

I'm just wondering whether splitting per business domain would actually make sense for this library.

let assert Ok(user) = 
  grom.get_current_user_request(token)
  |> httpc.send
  |> result.try(grom.get_current_user_response)

seems a bit verbose

still not convinced on this whole sans-io thing

and that's with let assert 😂

pub type Error {
  RequestSendingFailed(httpc.HttpError)
  ErrorneousDiscordResponseReceived(grom.RestError)
}

use user_response <- result.try(
  grom.get_current_user_request(token)
  |> httpc.send
  |> result.map_error(RequestSendingFailed)
)

use user <- result.try(
  user_response
  |> grom.get_current_user_response
  |> result.map_error(ErrorneousDiscordResponseReceived)
)```

This is a lot of code just to get the current user, whereas in many other libraries I could just do ```js
const user = await client.getCurrentUser();```

Without sans-io, it could also just be

use user <- result.try(grom.get_current_user(token))

(you'd probably still want your own error type here but yeah)

[shadow pilot]

The sans io is much nicer

The opinionated one you have there only works for one target, mandates a particular http client, and gives little control to the programmer

Slightly more concise code is a very weak benefit for those extreme disadvantages

[nocturne ridge]

I don't know, I feel like that's too low-level and overkill for most usecases

I guess it's future-proofing for an actual big bot using the library, one that would require a multi-node setup with a convoluted rate limiting and statistics collection system, but this would enable a level of control over the requests that I personally haven't met in other libraries.

[turbid root]

the nice thing is as an app user you can go "this feels too low-level at the callsite" and write your own helper, once, and use it for all sans-io libraries

[shadow pilot]

Other way around. Forcing a bunch of stuff is too much for most use cases

A good library is one that doesn’t restrict the user

A concise library isn’t a better library

If you want you can make a second package that is an opinionated IO version of the main package, which is ok as that restriction is then optional.

[nocturne ridge]

That'd make more sense, honestly. I believe it's also a requirement to be a library recognized by Discord for their community resources page, since you need to somehow implement rate limiting at the library level, most libraries meet this requirement by just doing a single-node rate limiter without redis or sth

[shadow pilot]

We’ve been over this before more than once, forcing one particular restrictive rate limiting solution is worse than having no restrictions and then having an implementation one can chose to use in another package

[nocturne ridge]

I think I'll stick by the single-file API though. Makes more sense in my head, even if it turns out to be a huge file.

Sure, but the two-package solution gives you actually more freedom:

• comfort, if you're making a small bot, since you don't need to configure all the knobs and dials (and that gives my lib a checkmark from discord if it gets big enough)
• freedom, if your bot gets big enough and suddenly you want your own rate limiter or a custom HTTP client or something

And who's to say you can't use the more liberal solution for small bots

I'd just be providing an optional, simple & opinionated solution to your HTTP-clienting needs

[shadow pilot]

Yeah, having options is best