Does anyone know where I can find a pinojs module for NestJS? | NestJS | Page 1

pastel spire Oct 26, 2022, 7:11 PM

#

If not, can any advise on how to build one. I have found nestjs-pino, but it doesn't support transports.

tired jay Oct 26, 2022, 7:15 PM

#

nestjs-pino is the popular one. But yeah, looks like it only supports transports for the pino-http part of the logger

pastel spire Oct 26, 2022, 7:16 PM

#

It's weird how no one has built this yet. Pino is the fastest production grade logger out there. I'm not really why the docs recommend winston anymore.

tired jay Oct 26, 2022, 7:16 PM

#

Personally I use #822179861791375451, but I did write that with Nest in mind (it's benchmarks are actually pretty close to Pino too 🙂 )

pastel spire Oct 26, 2022, 7:17 PM

#

Interesting...

#

let me check it out.

#

Tbh, don't really care as long as I can provide a transport and it's abstracted behind the nest logger.

tired jay Oct 26, 2022, 7:18 PM

#

Looks like the options for pino-http get passed to pino as well, as it's just a plugin on top of pino, but I'm just guessing here.

#

I originally wrote #822179861791375451 as an academic challenge for myself, just a learning process, but it turned out to be a pretty well written implementation so I keep up with it now

pastel spire Oct 26, 2022, 7:19 PM

#

Well I asked on the issues board, but the creator of the package told me that an issue was already open for my request, when I looked at the other issue, the creator wanted a RFC drafted for it. I mean, it seems like a relatively simple change to require a RFC.

#

@tired jay Basically... I'm looking to add a logger which supports multiple transports as workers. My intention was to use pino to log standard info and debug level stuff to Logtail whilst transporting errors etc to sentry.

tired jay Oct 26, 2022, 7:20 PM

#

/shrug Oh wow. That seems like total overkill to me

pastel spire Oct 26, 2022, 7:20 PM

#

Does ogma support this?

tired jay Oct 26, 2022, 7:21 PM

#

pastel spire <@524281921115652106> Basically... I'm looking to add a logger which supports mu...

Multiple transports, not quite easily. I mean, it's possible, and I could absolutely walk you through what to do for it, but there's not a direct config for setting up "errors go to one transport while all other logs go to another"

#

Again, I could help you set this idea up and it wouldn't be too difficult, but you have to write some code to set up the transport splitting

pastel spire Oct 26, 2022, 7:22 PM

#

Well I mean... I dont mind if all goes to sentry or logtail, would just be nice to be able to use sentry for true in-depth looks into the errors and how frequent they occur

#

logtail is simply for well... logs 😛

#

@tired jay depends how easy the implementation would be... I know there are sentry transports already available for pinojs and I've already written a pinojs transport for logtail. I mean, my efforts might be better served to just create a module wrapper for pino.

#

I'm undecided at this point.

#

Half tempted to fork that guys package and make the changes I need to publish.

#

Just looking at the quickest way to get around this issue for now.

tired jay Oct 26, 2022, 7:26 PM

#

So, Ogma supports a stream option which you can use for a custom output. By default it's process.stdout, but you can write to file or any other "stream" you want. That stream just needs to implement a write method that takes in a a "string" (typed as unknown, but I'm pretty sure I can change that to string) and returns unknwon (void, any, promise, whatever).

Ogma will end up calling this.options.stream.write(logToWrite) so you can check that log string for the level string ERROR, FATAL, etc, and send it to sentry and otherwise send it to logtail, or do whatever else you could want for it

pastel spire Oct 26, 2022, 7:27 PM

#

So I know pino does this in a separate worker thread. Is this something Ogma does?

tired jay Oct 26, 2022, 7:28 PM

#

Ogma does not use worker threads, no. That's something I could and probably eventually should look into

pastel spire Oct 26, 2022, 7:28 PM

#

Damn... I guess I'll just have to figure out a way to implement PinoJS as a module then. Bummer.

#

@tired jay keep me posted on the work you do with Ogma. I'd be interested on creating a product similar to Flareapp/Logtail if that's something you ever wish to officially support. I could help out 🙂

#

I also love the name 😉

tired jay Oct 26, 2022, 7:31 PM

#

Thank you! I've had people successfully integrate it with DataDog in the past. I'll need to look into Flareapp and Logtail and see what those are about. Are they other Logregators?

pastel spire Oct 26, 2022, 7:32 PM

#

Flareapp is more of a error aggregator, Logtail is just a fancy way to store logs and archive them to s3. Logtail gives a bit more support with looking up particular log messages too using SQL.

tired jay Oct 26, 2022, 7:33 PM

#

Awesome, good to know

pastel spire Oct 26, 2022, 7:34 PM

#

I've looked into things like DataDog and I mean the tools are great... but just too much for small businesses. Over complicated or over featured shall I say, built for true enterprise.

cerulean oak Feb 2, 2023, 5:11 AM

#

Still looking for a quality PinoJS module as well. I'm wondering if it's ok to simply create a custom file and import as needed, or perhaps a "singleton" service that manages it. The existing nestjs-pino is simultaneously vague and confusing. Also, the lack of explicit information on non-http transport configuration makes my eyes twitch. 😆

tired jay Feb 2, 2023, 5:13 AM

#

Are you married to using pino or would you be open to trying out another logger?

cerulean oak Feb 2, 2023, 6:32 AM

#

I need speed and multiple streams (e.g. stdout, file, web service)

#

But I'm open to other things. I saw Winston, but I also see several "interesting" issues in GH

#

What do you recommend?

tired jay Feb 2, 2023, 6:33 AM

#

Next major update of ogma will have speeds comparable to pino and the stream option is customizable enough to send logs whenever you need, just saying 😉 (I'm also the author of ogma, so I am biased about that)

#

I'm waiting on a PR for nest to get merged so I can update the ogma NestJS module before doing this major bump that is gonna impact many of the packages for ogma.

cerulean oak Feb 2, 2023, 6:35 AM

#

Hrm, anything I can work with immediately, or should I wait? Got an ETA? I really need something "up and running" in the next 48 hours. 😰

tired jay Feb 2, 2023, 6:38 AM

#

I mean, your can go ahead and integrate ogma now if you want to give it a shot and see what you think. Just saying that there's a major update that really pushes it's speeds in the pipeline.

Otherwise it currently has:

JSON logging support
Support for request logging (can when with all of nest's official transports)
Optional request scoped logger to automatically add correlation ids (requires the above request logging)
A CLI to turn the json logs back into human readable logs
Log levels
Log level name mapping
Basic json value masking based on key name

cerulean oak Feb 2, 2023, 6:40 AM

#

Yup, most of the logging is generated internally @cron() and the like . . . so no need for interceptors/etc at the moment. Just good old JSON logging and sending of that data to different locations.

Will the update require any major refactoring if I dive in tonight?

tired jay Feb 2, 2023, 6:43 AM

#

For ogma itself, there's gonna be a slight change to the json log format. No real API changes, but because the output changes I need to call it a major update.

There'll be a slight confusion change for the NestJS module, but nothing major. It's actually gonna be much easier to work with once it's done

cerulean oak Feb 2, 2023, 6:45 AM

#

Gotcha. Alright, thanks for the recommendation, I'll give it a go.

tired jay Feb 2, 2023, 6:46 AM

#

You run into any problems let me know either on the #1058055360994353172 channel or in a GitHub discussion

cerulean oak Feb 2, 2023, 6:49 AM

#

Will do, thx. First order of business, see how quickly I can discern from the docs how to get output to a specific file. 🤪

tired jay Feb 2, 2023, 6:49 AM

#

stream: fs.createWriteStream('fileName.log') should do it

#

Any node writable steam should be valid

cerulean oak Feb 2, 2023, 7:10 AM

#

Cannot find module '@ogma/nestjs-module' or its corresponding type declarations.

package.json: "@ogma/nestjs-module": "^4.2.2",

#

I'll do some digging and log a bug if I can't figure it out.

#

I don't need the interceptor or the express parser so I didn't import those . . . . . .they're not required are they?

tired jay Feb 2, 2023, 7:14 AM

#

They aren't necessary. They're add-ons for request logging

#

Hmm, it should be there. Does a build currently work or do you get the same error?

cerulean oak Feb 2, 2023, 7:21 AM

#

Local build works . . . might be an issue related to Docker. Just nuked everything and restarted . . . . 😨

#

So, local build works, local nest start --watch fails because it doesn't like my "currently nonexistent" output file location. Both folder and file exist with 777 just in case.

#

Gonna sleep on it and hopefully something comes to me overnight. 😖

#

Thanks again!

tired jay Feb 2, 2023, 7:29 AM

#

Interesting. Let me know if I need to take a deeper look

cerulean oak Feb 2, 2023, 7:32 AM

#

import { ConfigService } from '@nestjs/config';
import { OgmaModule } from '@ogma/nestjs-module';
import { createWriteStream } from 'fs';

@Module({
  imports: [
    OgmaModule.forRootAsync({
      useFactory: (config: ConfigService) => ({
        service: {
          json: true,
          // stream: createWriteStream('../data-general/file.json'),
          application: 'MyAwesomeApp',
        },
        interceptor: {
          http: false,
          ws: false,
          gql: false,
          rpc: false,
        },
      }),
      inject: [ConfigService],
    }),
    ...,
    ...,
    ...
  ],
  controllers: [...],
  providers: [...],
})

#

I have a fairly vanilla setup. Just commented out the stream line, just in case that somehow magically mattered (knew it wouldn't), and it's still the same thing.

#

Yeah, feel free to take a look if you get a chance. I'll circle back in the morning.

cerulean oak Feb 2, 2023, 8:22 PM

#

Hrm, still no progress on this. any thoughts?

tired jay Feb 2, 2023, 8:40 PM

#

What's the current error?

cerulean oak Feb 2, 2023, 9:10 PM

#

Same as before Cannot find module inside Docker. Literally the only package having that issue. 🤔

tired jay Feb 2, 2023, 9:11 PM

#

Can you force rebuild the docker container?

Or at the very least run -it sh and check that the node_modules does or doesn't contain the package?

cerulean oak Feb 2, 2023, 9:11 PM

#

Yup, I nuked everything then system prune my volumes as well. Like completely nuked it all. 😆

Same issue. 😢

#

I'll do it again real quick though.

tired jay Feb 2, 2023, 9:15 PM

#

It seems like there's a package.json or a lockfile that's not properly updated

cerulean oak Feb 2, 2023, 9:19 PM

#

Dockerfile:

...
COPY ./apps/awesome-app/package.json ./
# rm just in case
RUN rm -rf node_modules
RUN yarn cache clean
RUN yarn install
...

#

I don't even copy the lock file over. And I flush everything, just to be sure. But nope, it appears that @ogma isn't being installed.

#

Just manually ran yarn inside the container and it works. 🤦‍♂️

#

And, somehow deleting my local yarn.lock file and nuking again fixed it. I have absolutely no idea why -- never even copy it to the docker. There must have been something cached somewhere from an old image.

#

Apologies for the time suck. 🫠

tired jay Feb 2, 2023, 9:24 PM

#

Glad it was figured out!

cerulean oak Feb 2, 2023, 10:26 PM

#

Working like a charm. Unfortunately Docker doesn't display any colors in the log by default, but that's quite alright.

Any tips/docs on highly opinionated data routing? Something like:

streamDebits: ...
streamCredits: ...
streamInterest: ...

. . .or perhaps even a degree of specificity with regard to each call log, error, etc. ?

tired jay Feb 2, 2023, 11:25 PM

#

You've got options there. As ogma's stream method passes the entire message that will be logged to whatever stream is given you could use some regex or an .includes() check to determine where to send the log and have multiple streams inside to route the output. I don't have anything as complex as Winston's transports due to wanting to keep the API simple

cerulean oak Feb 10, 2023, 2:43 AM

#

So, just an update . . .

Got OGMA configured and it's working reasonably well. I have it logging and I'm actually capturing those logs via Vector(docker.sock) -> Grafana/Loki. Works well so far.

One question though: Is there no better way in the NestJS ecosystem for creating a truly "global" instance than to have to configure per-module and use OgmaModule.forFeature(MyCustomService)? Hopefully I'm just missing something, but it would be nice to be able to simply declare logger in app.module and use it anywhere. I'm ok with having to "inject" it if necessary, but the extra step of having to add it to each module that uses when it's already being piped in "from the top" seems off. 🤔

tired jay Feb 10, 2023, 2:56 AM

#

You can just use OgmaService directly without getting a context. The forFeature is to create a context specific logger, but you're able to pass the context to each log if you want

cerulean oak Feb 10, 2023, 3:39 PM

#

tired jay You can just use `OgmaService` directly without getting a context. The `forFeatu...

But is there any way to use it without import:[] in every module that calls it?

tired jay Feb 10, 2023, 3:43 PM

#

Like I said, just use the forRoot and inject OgmsService without the decorator. You lose out on the auto context, but there logger itself is available

#Does anyone know where I can find a pinojs module for NestJS?