Prometheus metrics journey in Remix

[zealous crag]

I wanted to post some details about getting Prometheus metrics working. ⚠️ This post will bit-rot like other things on the Internet.

I found this post to be a good starting point but it relies on an archived express middleware. https://scottsmerchek.com/blog/setting-up-production-monitoring-for-remix-on-fly-io
I liked its use of an ENV and used that to start it up locally so we can test it out even in dev.

This will expose a /metrics route which prometheus can scrape. Ultimately, I do not want my metrics to be public so this config will not work verbatim. We need to start another server on a different port and then it's up to you how to protect this. You could, alternatively protect this port based off a path. I'm assuming a very plain server environment and not from a PaaS or something.

So I want :3001/metrics to be my host. Easy enough to do with express. Create another express app and treat it as another app listening on another port. Unfortunately, this will have side-effects which I will address later.

First, let's talk about a faster feedback loop. You might think we need a prometheus + grafana instance to see if this is working (I did). In the end, I realized that this is just HTTP. curl localhost:3001/metrics | grep foo_counter will be a realistic test to see if our Counter named foo_counter is working. We don't need to test with Grafana.

My first attempt involved configuring express-prometheus-middleware and following along with the server.ts from the blues stack. It works fine for a default config and prometheus setup but it did not show how to configure a custom Gauge or Counter. This leads me to the core of this post.

We need a global.

The prometheus client really, really wants to use globals. It has a Registry API that lets you register different metrics. If you register a metric twice, it explodes. In other words: you can't create a metric in a loader.

It will load the first time but it will error out on the second page load with

metric foo_counter is already registered

So, we need a global. But we can't have one. There's no middleware.

So, my first attempt involved trying to create a Singleton as something like metrics.ts which would hold all the metrics (Counter, Gauge), etc. This did not work. Because, of the scope and express.

My Singleton was working fine as plain objects. I could hold a string and a single value across the app. I could create it in server.ts but it would not work with the prometheus and express stuff. I'm not 100% on this because it is pretty confusing what scope is where. But ultimately, my branch ended up with what I thought was right but the counter would always stay at 0. It's like the memory was off in lala land. Maybe I was registering the wrong express app. I even tried to manage the prometheus Registry myself but this also did not work.

I switched to express-prom-bundle and started over (git reset). You can basically follow their install guide in the README.

server.ts

const app = express()

// order is important here
// "only the routes registered after the express-prom-bundle will be measured"
if (process.env.ENABLE_METRICS) {
  const metricsPort = process.env.METRICS_PORT || 9999
  const metricsApp = express()
  const metricsMiddleware = promBundle({
    autoregister: false,
    includeMethod: true,
    includePath: true,
    metricsApp: metricsApp,
    promClient: {
      collectDefaultMetrics: {},
    },
  })

  app.use("/((?!build|images))*", metricsMiddleware)

  metricsApp.listen(metricsPort, () => {
    console.log(`✅ metrics ready: http://localhost:${metricsPort}/metrics`)
  })

We're using the ENV trick from the scottsmerchek blog post. Note the order we are doing this in. I want to filter out the build and images paths (for now). After metricsApp.listen you can register metrics (like Gauge). But we cannot do a counter here because of the same global problems. If we create a counter in server.ts, how would we call .inc() on it from a loader? So, in my case I'm going to punt on this for now but use this middleware's path buckets to essentially histogram the URL paths.

But as I said before, there's a side effect of having two apps. And that's why we need that metricsApp: metricsApp option. So it can register itself with app but then serve from the other port.

more in a bit: build command, screenshot and more config snippets

[zealous crag]

Prometheus metrics journey in Remix

[molten pumice]

How did you go with this in the end? Did you get it working together with being able to define custom metrics?

[molten pumice]

I'm adding some more details here for the next person to follow on the journey, as I found a way to add further metrics from within the app. In my case, I wanted to add prisma metrics to the default set of metrics from the prometheus middleware.

Note that these steps work for appending the metrics coming from other libraries rather than from counters that we need to directly register in the server's prometheus. An example is the prisma library's prometheus metrics: https://www.prisma.io/docs/orm/prisma-client/observability-and-logging/metrics

The one issue I've run into is that of course by doing this as part of the request handler context, the additional metrics only get added to the list after the first page load/request that reaches the server.

Step 1: Create a function to add a callback to a list of callbacks to return metrics for:
The handleRequest function that is returned by default from entry.server.tsx takes as its 5th parameter the AppLoadContext - which is an object that is defined within the server.ts file. Through this object, you can pass a registration function to add a new metrics source to the list of metrics to be reported when calling the /metrics endpoint.

Since you need to be able to restrict the number of times a particular metric is added to the list, you need to be able to keep track of ones that have already been added - and to prevent those being re-added when not desired. How I've done this is to create an object in server.ts that is of type { [key: string]: () => Promise<string> }, then create a function that adds new metrics to that list as required:

type MetricsFunctions = {
  [key: string]: () => Promise<string>;
};
const addedMetricsFunctions: MetricsFunctions = {};

const addMetrics = (name: string, fn: () => Promise<string>) => {
  if (!addedMetricsFunctions[name]) {
    logger.debug({ msg: `Adding metrics for ${name}` });
    addedMetricsFunctions[name] = fn;
  }
};

The metrics endpoint then appends the resolved string value for each registered function to the end of the metrics response:

telemetry.get('/metrics', (req, res) => {
  res.set('Content-Type', promClient.register.contentType);
  promClient.register
    .metrics()
    .then(async (metrics) => {
      // Collect additional metrics
      const additionalMetrics = (
        await Promise.all(
          Object.values(addedMetricsFunctions)
                .map((fn) => fn())
          )
        ).join('\n');

      // return promClient metrics + additional metrics
      return res.end(metrics + '\n' + additionalMetrics);
    })
    .catch((error: unknown) => {
      logger.error({ msg: 'Error getting metrics', error });
    });
});

Step 2: Pass the addMetrics function to the remix app request handler:
The next step is to add the addMetrics function to the return value from the getLoadContext function passed into createRequestHandler in the route handler for the main application.

function getRequestHandler(_build: ServerBuild): RequestHandler {
  const getLoadContext: GetLoadContextFunction = (_, res) => {
    return {
      cspNonce: res.locals.cspNonce, // from EpicStack
      addMetrics,
    };
  };

  return createRequestHandler({ build: _build, mode: MODE, getLoadContext });
}

app.all(
  '*',
  MODE === 'development' ? (req, res, next) => getRequestHandler(devBuild)(req, res, next) : getRequestHandler(build)
);

This means that the addMetrics function is now available inside your remix app's entry.server.tsx default function:

export default function handleRequest(
  request: Request,
  responseStatusCode: number,
  responseHeaders: Headers,
  context: EntryContext,
  loadContext: AppLoadContext
) {
  // nonce from epic stack, which showed how to pass values through the context
  const nonce = typeof loadContext.cspNonce === 'string' ? loadContext.cspNonce : '';
  if ('addMetrics' in loadContext && typeof loadContext.addMetrics === 'function') {
    // register any self-contained sets of metrics created by other libraries
    // in my case, from prisma.
    loadContext.addMetrics('prisma', getDbMetrics);
  }

Step 3: create the functions that returns the promise of metrics strings:
Prisma has it's own prometheus metrics output, so we just need to be hooking into our prisma db's prometheus metrics. We do want to have prisma be registered only once in our dev server, so it's recommended to use some sort of singleton to ensure it is only registered once.

const db = singleton('prisma', () => {
  const prisma = new PrismaClient();
  prisma.$connect().catch((e) => {
    console.error(e);
    throw e;
  });

  return prisma;
});

export const getDbMetrics = () => db.$metrics.prometheus();

The getDbMetrics function returns a promise to the db's prometheus metrics output string. This gets passed as a constant function to our addMetrics on each request - but once we've added it to the addedMetricsFunctions list under the key prisma once, we'll ignore any further additions. When someone hits the metrics/telemetry server's /metrics endpoint from server.ts, we call that getDbMetrics function and append the returned string to the end of the prometheus client's list of metrics that it was tracking.