pubSub realtime : issue on endpoint and backgroundJob

[hard elm]

Hi, first, thank you for all the work.
I'm settings up realTime (using pubSub from redwood).
I manage to get think working in some way : frontend subscribe and publish successfully.

My trouble come on 2 situations :

• when using my publishService from a api/src/function/<endpoint>, my context doesnt have pubSub attribut in it. So i cannot publish...

Here is how i create my function

import type { APIGatewayEvent } from 'aws-lambda'

import { useRequireAuth } from '@redwoodjs/graphql-server'

import { getCurrentUser, isAuthenticated, authDecoder } from 'src/lib/auth'
import { commitProject } from 'src/lib/commitFunction'

/**
 * The handler function processes HTTP request events.
 *
 * @typedef { import('aws-lambda').APIGatewayEvent } APIGatewayEvent
 * @typedef { import('aws-lambda').Context } Context
 * @param { APIGatewayEvent } event - Information from the invoker.
 * @param { Context } _context - Contains invocation, function, and environment info.
 */

const myHandler = async (event: APIGatewayEvent) => {
  if (!isAuthenticated()) {
    return { statusCode: 403 }
  }

  const user = context.currentUser
  const body = event.body ? JSON.parse(event.body) : null

  if (!body) {
    throw new Error('Invalid body')
  }

  const res = await commitProject(body, user.username)
  if (res) {
    return res
  }

  return {
    statusCode: 200,
    headers: {
      'Content-Type': 'application/json',
    },
    body: JSON.stringify(body),
  }
}

export const handler = useRequireAuth({
  handlerFn: myHandler,
  getCurrentUser,
  authDecoder,
})

I got currentUser without trouble, but not the pubSub inside ...

The 2nd point is :

• I would like to publish from a backgroundJob, how to do it ? Since context is not available neither. I need this because i have a periodic backgroundJob that will publish ....

Thank you !!

Here is the publiscation api/services/<projects>

import type { PublishProjectChangeInput } from 'types/graphql'

import { db } from 'src/lib/db'
import { logger } from 'src/lib/logger'
import type { ProjectChangeChannelType } from 'src/subscriptions/projectChange/projectChange'

// export const projectChange = ({ projectUid }) => [projectUid]

export const publishProjectChange = async (
  { input }: { input: PublishProjectChangeInput },
  { context }: { context: { pubSub: ProjectChangeChannelType } }
) => {
  logger.debug({ input }, 'publishing project change')

  const { projectUid } = input

  const project = await db.project.findUnique({
    where: { projectUid },
  })

  if (project) {
    logger.debug({ projectUid }, `publishing project ${project.id} change`)
    context.pubSub.publish('projectChange', projectUid, project)
  } else {
    logger.warn(`Project with projectUid ${projectUid} not found`)
  }

  return project
}

[minor escarp]

@floral wraith anything you can help with here?

[hard elm]

Any help or piece of information is appreciated. Perhapse i'm doing wrong, perhapse it's just impossible, or perhapse i'm not looking into the correct direction. Thanks again for the time.

[floral wraith]

I never did much with realtime. So, sorry for playing hot potato here, but I'll pass this along to @brazen lagoon, who implemented subscription support. See if he has any input.
If this turns out to be an actual bug in the framework I can see about fixing it 🙂

[brazen lagoon]

Hi @hard elm it’s been awhile but the way I published from a job was via webhooks. The job runs in a completely different process than the graphql server or api. So I essentially send a webhook to that api with some reference to the thing and the webhook then can publish. See https://github.com/dthyresson/pixeez/blob/d9059d67f1736e152a16ab6cc2ed4c783cbf8d94/api/src/jobs/CreateThumbnailJob/CreateThumbnailJob.ts#L30

And https://github.com/dthyresson/pixeez/blob/d9059d67f1736e152a16ab6cc2ed4c783cbf8d94/api/src/jobs/webhooks.ts

I then have a service to accept the webhook. Since it’s a GraphQl resolver it can easily publish. The webhook itself is a gql call https://github.com/dthyresson/pixeez/blob/d9059d67f1736e152a16ab6cc2ed4c783cbf8d94/api/src/services/webhooks/webhooks.ts

@floral wraith I don’t think this is a bug. Jobs run in a different process than the api or GraphQL server so it can’t do gql things. The webhook approach I did was the best solution… it’s kind of treating the publish like a q. The other way i might do it would be some queue service but seems too complicated. I’d suggest the person maybe add some retry etc for durability

Note also that there is just very basic secret auth on the webhook consumer

[hard elm]

@brazen lagoon it's perfect wekhoob. Clean and fast. I have never used redWoodSDK but with your links i should be able to reproduce and understand. really really tahnks you. ❤️

For my trouble of api/src/functions not having the pubSub object ? Do you have any idea ?

[brazen lagoon]

I used live queries in that app https://github.com/dthyresson/pixeez/blob/d9059d67f1736e152a16ab6cc2ed4c783cbf8d94/api/src/lib/realtime.ts

And invalidated the key

[hard elm]

umm but it's liveQuery and not pubSub. Do you know if it's "normal" behavior or no ?

to not have pubsub in context ?

[brazen lagoon]

Should be able to do the same publish … I’d just call the service that you’d do normally from the webhook

I have to recall but I’m pretty sure I put the pub sub on the content via the plugin…

[hard elm]

I mean when i call my service from my AWS fonction, pubSub is empty. This is weird

Ok, i posted how i create the function, perhpase i have badly done something ?

(in my 1st message)

[brazen lagoon]

Yeah I do here https://github.com/redwoodjs/graphql/blob/427e92a5b9aac06a53b69eabdb66aa09fde4c016/packages/realtime/src/graphql/plugins/useRedwoodRealtime.ts#L245

[hard elm]

i have console.log it and so, its empty i dont get it

Cursor tells me its because in src/api/function its not the same context and GraphQl is not loaded but you know... cant trust the AI ^^

[brazen lagoon]

Have you gotten basic sub and pub to work via the examples in https://docs.redwoodjs.com/docs/realtime

Yeah the pub sub is already in context

[hard elm]

It works well, when i call my service from web part. My trouble is calling the service from api/src/function

export const publishProjectChange = async (
  { input }: { input: PublishProjectChangeInput },
  { context }: { context: { pubSub: ProjectChangeChannelType } }
) => {
  logger.debug({ input }, 'publishing project change')

  const { projectUid } = input

  const project = await db.project.findUnique({
    where: { projectUid },
  })

  if (project) {
    logger.debug({ projectUid }, `publishing project ${project.id} change`)
    context.pubSub.publish('projectChange', project)
  } else {
    logger.warn(`Project with projectUid ${projectUid} not found`)
  }

  return project
}

[brazen lagoon]

Have a look at https://github.com/redwoodjs/graphql/tree/main/packages/cli/src/commands/setup/realtime/templates/subscriptions

[hard elm]

it crash on context.pubSub, telling its undefined

I do have look, but its the front that publish or the CLI, not from a function (endpoint)

[brazen lagoon]

You are calling the service from a function?

[hard elm]

yes

[brazen lagoon]

Yeah then it won’t go through the GraphQL lifecycle and setup the context

[hard elm]

because i have a component that call a function like POST on /api/foo/bar, that fct need to publish

[brazen lagoon]

Try via a GraphQL mutation or query

This is why the webhook I did calls the graphql server and not just a webhook function

[hard elm]

ok thanks, this is was i though,

so i could call the webhook from the function .

it shoulds work no .? Or the mutation directly ?

[brazen lagoon]

Mmmkmmm mmm I’d do the mutation directly

[hard elm]

never done it, im not pro att mutation yet ^^ Thanks for the hint !

i need to step up then

(so the mutation will do the publish ? )

[brazen lagoon]

Well I’d try direct first and then the function … with the function you’d make two requests and really just need one

[hard elm]

what do you mean by direct ?

[brazen lagoon]

The mutation is sent from the job https://github.com/dthyresson/pixeez/blob/d9059d67f1736e152a16ab6cc2ed4c783cbf8d94/api/src/jobs/webhooks.ts

Then there is a service that consumes it and that’s when you do the publish

Job says here is a webhook to call graohql with some id and some resolver.

That service resolver executed and grabs id or other params and does the publish on it Maybe it needs to fetch something with that id

This in the job makes the gql request https://github.com/dthyresson/pixeez/blob/d9059d67f1736e152a16ab6cc2ed4c783cbf8d94/api/src/jobs/webhooks.ts#L27

[hard elm]

ok thanks for all, i will digest everything slowly. I think you gave me all the resource needed ❤️ thank you very much !!

[brazen lagoon]

executeGraphQLWebhook makes a post with one of the mutations

No problem. It’s a bit involved but even in some other system like Sidekiq or a cloudflare queue you’d do something similar to pass info between a job and something else.

[hard elm]

Yes, i'm not surprised. There no RedwoodMagic ✨ here, that's all :p

(it cant be simple everywhere)

[brazen lagoon]

These are two different processes. The GraphQL server is separate from each instance of the job runner. Even in WebSocket pub sub you’d need to send info or connect to that socket

In fact I need to figure this out for rwsdk and durable objects so send a notification from a job too. I haven’t tried but need to do this too

[hard elm]

ok !!
On my side, i have to publish dockerStats to clients front-end, and lock project on export to avoid conflict

pooling is, inded, not possible to update the UI ...

[brazen lagoon]

Have you considered just simple polling?

If you don’t need immediate or one second response it’s waaaaaay simpler to poll for new stats every 30 seconds or so

You can use GraphQL caching to reduce load if no data has changed

Much much much much simpler

And just as effective

Wait polling isn’t possible? Why not?

A pub sub literally gets new data and a live query is just an on demand refresh

[hard elm]

i can try but i want real time, i need 2-3 sec max delai

and pooling for this is not good

but if you think cachin can be enough, because data changed not so ofter, i can give it a try

but this mean i have to store my dockerStat result in database, i dont need this extra cost, i just want the last state, and doing so will need deepth refacto of code in our side. Pub was the cleanest way to notify modifcation

but i will considere what you propose.

[brazen lagoon]

Nope you don’t need to store it in the database - just have your service make the api call and respond with the stats. Then add caching so it pulls from the cache rather than making the service and api call

But then I am confused why you need a job at all

If you don’t store the data when the job runs

I imagine you are not running the job every 2 seconds

[hard elm]

Umm. the cache will works on front & backend call (from web and api ? )

I guess you are right, i need to redraw my schema properly

it's not clear to me, that cache thing. I mean, i have multiple source that can "update" my data and ask the user to refetch. It can come from front/back or background job (i have a 5min background job that lock a ressource and need to display in front for example)

[hard elm]

and does the cache is sahred accross user ? I mean its on the client side or backend side ? I guess backend

Anyway, response is user specific due to permission. that will mess with backend cache in some way

[brazen lagoon]

It can be user or session specific with various ttl and ways to invalidate. See https://the-guild.dev/graphql/yoga-server/docs/features/response-caching

[hard elm]

um ok. thanks for your time really. I need to step up on this

i'm still too much old school

[brazen lagoon]

What I don’t understand is if your job fetches data every 5 minutes and you do not persist it you could be doing work that no user ever sees. Instead have no jobs and no added compute that to one sees and just get the info when needed. Even if you fetch cached every 30’seconds while a user is viewing it’ll have faster updates than a 2-3 second update that updates every 5 minutes

It sounds like you are doing a piggyback or pass through to an api to fetch stats

[hard elm]

there is a mix, backgroudn job update part of info, and user action update randomly some parts

but you are right, its too messy anyway

[brazen lagoon]

Ok so job can update and poll fetch can … fetch

I think you can simplify it and still offer a good user experience

[hard elm]

yes yes

i have to separe the concern

[brazen lagoon]

Cool. If I get some time I can maybe draw something out

The GraphQL real-time is nice (I did the implementation last year or before? ) but it does have some moving parts

[hard elm]

because you ask, i have a side bar, displayign list of project (list of tools). users can update projects (mulitple user on same project) they can change color, name, permission etc... and i have to add state of some tools, that are, in fact docker running, so i want to monitor if its up/down, running, unheatly etc...

the pubSub subscription i have add for editing, allowing to removing the pooling of project list/update works

[brazen lagoon]

Hence this diagram of how real-time do you really really need https://docs.redwoodjs.com/docs/realtime#how-do-i-choose-subscriptions-or-live-queries

[hard elm]

but project can be locked when user export them (its take more than 5 min on locked state for example)

[brazen lagoon]

consider polling for data updates on a reasonable interval. According to Apollo, in most cases, your client should not use subscriptions to stay up to date with your backend. Instead, you should poll intermittently with queries or re-execute queries on demand when a user performs a relevant action, such as clicking a button

https://www.apollographql.com/docs/react/data/subscriptions#when-to-use-subscriptions

[hard elm]

yes but other user can update, so i cannot bind this to click on front

A update project FOO, that is displayed to X users. thaths why the pubSub was nice

user are subscribing to project's channel, and so, get update when a "change" is made

easy to understand

[brazen lagoon]

Got it

Then I think the job webhook pattern is what you want

[hard elm]

i can split the "tools state" that i wanted to run on backgroudn job. But perhapse its wrong and i shoudl get the tools's state not in a backgroudn job

but all you tell me is right, it's due to my lack of knowledge on prisma and graphQL

[brazen lagoon]

Hard to know. But if it is state then it strikes me odd that you don’t persist it since that is what state is

[hard elm]

i'm too beggining on this, so my mind still map old concept

yes, i guess persist the last state is better

at least, when user 'jump' on homepage, they will need state, and dont wait the next "publish" or pooling to get data

[brazen lagoon]

What does the app do?

[hard elm]

my redWoodApp you mean ? or tools in "project" ?

[brazen lagoon]

The thing that uses the job and why and what the user does and sees and why

[hard elm]

Well, its running docker, and user can access it using VNC or iframe

when user 'click' on the tool, there is a manager to start/restart docker if not started yet

[brazen lagoon]

Right but what does it do? Why do users need it? Why build it?

[hard elm]

user need to have information about tool, url, token, tools state

[brazen lagoon]

A web interface to restart a docker container?

[hard elm]

kinda, but more complicated

you can see it like this its valid

[brazen lagoon]

A web interface to your docker container to fetch info about it and interact with the container? A management interface?

What is the container doing?

And users have deployed their own containers? And this is the way they manage it?

[hard elm]

it's tool, it can be a firefox, or a VNC, or other webTool displayed in an iframe

[hard elm]

yes they can pick container/tools to plug them in project

[brazen lagoon]

Oh ok so you have a browser extension or something that talks to a redwood api over GraphQL and that extension shows info about some container they need to monitor

Is it up or down memory use etc etc

[hard elm]

yes for this

[brazen lagoon]

And the extension is meant to respond to a GraphQL subscription? As the main ui?

[hard elm]

you lost me on this. Its redwood that manage the containers, and redwood UI that display states/tools and so.

[brazen lagoon]

Because it renders a page in some iframe

Ok so it’s just a browser rendered page but you might display it in Firefox or other things that can make a request

And to get the states/tools from docker there is some api call you make

[hard elm]

I think i lost you by speaking about firefox. Tools are docker running app. It can be a Website, or a "software" like i dont know a full linux , or so. This is not the important part

[hard elm]

yes

[brazen lagoon]

Ok you mentioned Firefox

[hard elm]

yes that's my mistake its confusing sorry

[brazen lagoon]

No worries

Just trying to get a handle on things to see what other options might be

[hard elm]

lol you are dedicated. I'm not asking that much ^^

[brazen lagoon]

But if you have the plumbing and such for subscriptions working and can test it then sounds like webhook way might be best

Appreciate that. Since I created that real-time thing figure I need to help anyone trying to use it 🤓

[hard elm]

ahah got it. thank you.

we use too much function and not enough service/graphql

[brazen lagoon]

See if you can get subscription updates working “normally “ first . So one client subscribes and then in some other client have a mutation that then issues a publish

[hard elm]

it's due to our bad comprehension of the concept and our old concept comming from REST or websocket

[brazen lagoon]

And if that works then the webhook pattern will work

[hard elm]

this looks ok at the moment,

[brazen lagoon]

You can move your function logic into a service . Just manually create a sdl for share of the response (the type) and the query or mutation. You’ll have much more control over auth too

Make sure your functions follow https://docs.redwoodjs.com/docs/serverless-functions

[hard elm]

I know you are right. but SDL and so is more complicated in my head to apprenhend :p

[hard elm]

you mean, i can keep my function but move all logic into service instead . And funciton is just a shell. Thats why you mean ?

[brazen lagoon]

No function at all. Just a GraphQL query or mutation

[hard elm]

ok 👍

[brazen lagoon]

Instead of a function handler it’s a GraphQL resolver

[hard elm]

yeap. Thank you.

[brazen lagoon]

The GraphQL server is actually a function … it just hands off logic to the things that can perform the task the request asks for

[hard elm]

ok*

[brazen lagoon]

But you need a type sdl and a service to tell it what it can do and how

[hard elm]

yeap

[brazen lagoon]

As part of the GraphQL schema

[hard elm]

got it. Cursor help a lot for that ^^

[brazen lagoon]

But the nice thing is you have access to the context and everything else graphql can do

[hard elm]

and the caching

[brazen lagoon]

Yup

[hard elm]

i know we are doing bad thing at the moment

thats why i'm working on the '"realTime" pub sub, but i need to rewrite lot of stuff in fact because some part of code is 2 years old and project has evold since (and RW too lol)

[brazen lagoon]

Nothings bad . Everything is a trade off and can be better if need . Or not if it’s not critical. If it works and users are happy then all good

[hard elm]

i think we are on it since reddwood 1

[brazen lagoon]

Oh the glory days!

If you rewrite it though I would consider rwsdk

[hard elm]

yeah, i was happy to see the backgroudn job when i needed it. just in time

[brazen lagoon]

Jobs and real-time are soooo much easier

[hard elm]

i dont know, it will cost a lot i guess

[brazen lagoon]

Ok well let me know how it goes. Happy to help if can

[hard elm]

you always help. It's a real pleasur. Thank you so much. i cannot share what we are working on, but I would be happy ^^