#Logging and tracing

1 messages · Page 1 of 1 (latest)

graceful shoal
#

How do y'all handle logging and tracing in self hosted production? Was trying to use grafana+victorialogs but every multi-line log was getting its own log line in grafana/vmlogs instead of getting grouped together (bit hard to explain)
So I just stopped collecting logs for nextjs for now but its unfortunate not being able to reference logs from traces

slender bladeBOT
#
Post created!

🔎 This post has been indexed in our web forum and will be seen by search engines so other users can find it outside Discord

🕵️ Your user profile is private by default and won't be visible to users outside Discord, if you want to be visible in the web forum you can add the "Public Profile" role in id:customize

✅ You can mark a message as the answer for your post with Right click -> Apps -> Mark Solution
(if you don't see the option, try refreshing Discord with Ctrl + R)

naive lagoon
# graceful shoal How do y'all handle logging and tracing in self hosted production? Was trying to...

I like to use the LGTM stack with otel-collector. Metrics then help me to identify that there is an issue, traces tell me then what happened and why it happened and the logs tell me exact details about the specific request.

When it comes to monitoring your mind model need to change a bit. Before: I log everything, that helps me. After: split between metrics, tracing and logging to have the most consistent and needed information

There are serveral tutorials on how you setup the LGTM stack with otel-collector

graceful shoal
#

My issue specifically was the fact that by default the way nextjs logs anything, even startup logs, arent grouped together in a single line. Theyre split by.. idek. Vibes i guess? Vercel just wanted to make them look pretty visually but when u try to collect them they show up as individual messages in grafana. Ill send a screenshot example later

naive lagoon
# graceful shoal My issue specifically was the fact that by default the way nextjs logs anything,...

Inside the nextjs app I like to use the @vercel/otel package as it configures everything for nextjs. Like that, all the nextjs relevant stuff is already handled.

For my own logs that I have inside an api route or when I want to debug something or log an id or whatever, then I am using the Pino-logger. Pino in general is already pretty nice and there is a package that automatically sends them to OTel when the .env variables are correctly configured. Like that custom logs are also collected

graceful shoal
naive lagoon
# graceful shoal Oh that sounds pretty nice. Its server side only i assume?

It is, yea. It reads the .env variables and I have for my otel collector a basic auth header so no unknown apps or persons can ingest data. But only the server has access to those variables.

Technically you can expose it, but I would be very very careful what the client sends you. Imagine in the middle of the night you wake up due to alerts that were triggered by a evil client.. not good

graceful shoal
#

Anyway ill check out pino. Would you be okay with sharing your utility function to trigger pino if you use one?

naive lagoon
# graceful shoal Anyway ill check out pino. Would you be okay with sharing your utility function ...

sure, this is my logger file:

import pino from 'pino';

const isProduction = process.env.NODE_ENV === 'production';

function createLogger(): pino.Logger {
    const level = process.env.LOG_LEVEL ?? 'info';

    // biome-ignore lint/correctness/noConstantCondition: <explanation>
    if (1 > 0) {
        const transport = pino.transport({
            level,
            target: 'pino-opentelemetry-transport',
            // options: {
            //     formatters: {
            //         log(object: any) {
            //             const span = trace.getSpan(context.active());
            //             if (!span) return object;

            //             const spanContext = span.spanContext();
            //             return {
            //                 ...object,
            //                 trace_id: spanContext.traceId,
            //                 span_id: spanContext.spanId,
            //                 trace_flags: spanContext.traceFlags,
            //             };
            //         },
            //     },
            // },
        });

        return pino(transport);
    }

    return pino({
        level,
        transport: {
            target: 'pino-pretty',
            options: { colorize: true },
        },
    });
}

let loggerInstance: pino.Logger | null = null;

export function getLogger(): pino.Logger {
    if (!loggerInstance) {
        loggerInstance = createLogger();
    }

    return loggerInstance;
}

Some of the commented out things may be needed for you. For example to mix in the current running trace or span id (or if you get something from sentry, then sentry details).

After that you can use pino like normally:

const logger = getLogger();
logger.info({something: "cool"}, "Oh hell no: something cool appeared")

Make sure to install the pino-opentelemetry-transpor package

graceful shoal
#

Cool project btw i saw in your bio, it looks clean

naive lagoon
#

which project? 👀

graceful shoal
#

Im about to launch a (game) server hosting platform myself and am slightly terrified. Lot of stuff i havent even thought about yet like i just saw on ur site u display the vat, getting that to work across regions gotta be hard. Im sure theres a bunch more like that too

graceful shoal
naive lagoon
#

ah yea, can be quite hard sometimes tbh

slender bladeBOT
graceful shoal
naive lagoon
#

yea, international stuff is a whole different thing. If you want to start, start slow in your region, then get bigger and bigger and eventually you will be a big player as well

graceful shoal
#

Appreciate the help man