Inconsistent application performance

[visual pier]

I have a site in production which is having heavy oscillation in performance seemingly at random.
For example here is a page scan with extremely good performance
https://pagespeed.web.dev/analysis/https-codicerisparmio-it/k5gancc7ra?form_factor=mobile

and another scan with faily poor performance
https://pagespeed.web.dev/analysis/https-codicerisparmio-it/xl21vfq5ik?form_factor=mobile

I'm running everything on a CCX23 VPS on Hetzner.

I tried practically everything: caching aggressively through redis, changing settings in cloudflare, changing configs for opcache and phpfpm, tweaking mariadb config, optimizing the js and css and while this helped swinging the score toward the upper limit it still swings down heavily toward 60 at random.
I ran various profilers but all I could gather was that the problem isn't the db since the queries are executed extremely fast barely hitting 90ms total.

Another metric i found odd is that the spike in performance loss seems tied to memory consumption .

[hollow veldt]

Looking at the score and the page, the issue doesn't look like to be your webserver directly. Rather, it's due to content that's loaded. You're loading GTM, you're loading swiper from a CDN, loading lots of images immediately. So if the page would need to wait in order for it to be able to show something, that's going to contribute a lot towards the first paint. So you'd want to look at improving that.
Look at the insights pagespeed gives you, it also gives tips on how to improve that. A first step could be to improve your images, reduce them in size, only render what's visible, or even load those images after the page has loaded (https://developer.mozilla.org/en-US/docs/Web/Performance/Guides/Lazy_loading). I tried a simple online optimizer with one of the images (the 500x200 one for mobile), and it reduced the size by about 17 percent, from 32kB to 26kB

[visual pier]

i took the index by example because that's probably the page with the most content (aka it's the worst case scenario) but i have the same issues also on pages with barely any image, swiper is loaded through alpine

  <div x-data="SwiperSlider()" data-dispatch="slider-assets-loaded" x-load
      x-load-css="['https://cdn.jsdelivr.net/npm/swiper@11/swiper-bundle.min.css']"
      x-on:slider-assets-loaded-css.window="init" class="homepageSlider">

[hollow veldt]

Looking at that, it seems like the controller occasionally has to do more work. For example, if you cache query results, that cache is empty at some point, so then it checks the cache, fires a query to get the result, stores the result back into cache. This all contributes to memory usage and consuming more time. That wouldn't be a bad thing in itself, but you might be able to optimize that some more?
Laravel quite recently added things like "stale while revalidate" https://laravel.com/docs/12.x/cache#swr, this gives a window where "old" data is served, while in the background the cache is refreshed

[visual pier]

I use cache flexible everywhere on the site, also my caches are set fairly aggressive (around 2-3 minutes) hence why i found weird the swings in performances are barely 1 minute apart.

[hollow veldt]

Not sure about that graph, but does it show actual response times, or the total request time, including what happens after the response? If things happen async, after the request has been sent, then those numbers are kinda meaningless 😅
Perhaps also check if deferred functions properly work on your system, there are some requirements in order for it to do its thing after the request. Think it would fall back to performing it synchronously

[visual pier]

i would need to implement more telemetry to be completely sure and looking at their docs it's not exactly painless.
For now it shows around 450-600ms of Time which if matched with other tools is the TTFB of which around 90-100ms are cache hits and db queries and then a big blank. I do not use deferred functions currently in my application.

as for other metrics after the TTFB i am running the browser tooling with the recommended slowdowns

[hollow veldt]

I do not use deferred functions currently in my application.
Cache::flexible() uses deferred functions. So when the data is considered stale, it would update the cache in a deferred function. If that doesn't work properly then it would execute in sync with the request

[visual pier]

Oh i thought you meant the actual defer function, how can i be sure it's working properly then ?

[hollow veldt]

Just something like;

Route::post('/test', function (Request $request) {
    Log::info('Request started');
    defer(function () {
        sleep(10);
        Log::info('Defer done');
    );
 
    return response()->noContent();
});

If that request takes 10 seconds to complete you know deferred functions don't work. If it replies instantly you'll know it's doing its thing in the background and you should see a log pop up after 10 seconds

[visual pier]

i tried something like this since most of the caches are in get requests and it's printing the first Log but not the one inside defer

Route::get('/test-defer', function () {
    Log::info('Defer Test started');
    defer(function () {
        sleep(10);
        Log::info('Defer test ended');
    });

    return response()->noContent();
});

[hollow veldt]

Does it work locally?

[visual pier]

Doesn't seem like it

[hollow veldt]

It relies on fastcgi_finish_request (so might not work on artisan serve) along with the InvokeDeferredCallbacks middleware. So perhaps those are something to look into?

[visual pier]

i'm using sail locally

and in production it's a server provisioned through forge

[hollow veldt]

Unfortunately Sail uses artisan serve under the hood 😅 So not sure if that would work

[visual pier]

being a laravel 12 application i should be adding the middleware in bootstrap/app right ?

    ->withMiddleware(function (Middleware $middleware) {
     
        $middleware->append(InvokeDeferredCallbacks::class);

[hollow veldt]

It should be added automatically

[visual pier]

so i'm guessing the issue is on the fastcgi_finish_request ?

shouldn't have forge setup everything correctly ?

[hollow veldt]

It should be part of the default global middleware https://github.com/laravel/framework/blob/9bc12a3455f3fbc32ad16f704522371117b8a793/src/Illuminate/Foundation/Configuration/Middleware.php#L455
You're not overwriting the global middleware, right?

[visual pier]

No

this is my app.php

<?php

use Inspector\Laravel\Middleware\WebRequestMonitoring;
use Illuminate\Foundation\Application;
use Illuminate\Foundation\Configuration\Exceptions;
use Illuminate\Foundation\Configuration\Middleware;

return Application::configure(basePath: dirname(__DIR__))
    ->withRouting(
        web: __DIR__.'/../routes/web.php',
        api: __DIR__.'/../routes/api.php',
        commands: __DIR__.'/../routes/console.php',
        channels: __DIR__.'/../routes/channels.php',
        health: '/up',
    )
    ->withMiddleware(function (Middleware $middleware) {
        $middleware->validateSignatures(except:[
            '_se'
        ]);
        $middleware->append(\App\Http\Middleware\CustomRobotsMiddleware::class);

        $middleware
            ->appendToGroup('web', WebRequestMonitoring::class);

    })
    ->withExceptions(function (Exceptions $exceptions) {
        if (config('app.debughandler.sentry')) {
            Sentry\Laravel\Integration::handles($exceptions);
        }
    })->create();

[visual pier]

@hollow veldt
digging into the php error logs on forge i noticed that everytime defer is called it throws an error

[25-Jul-2025 10:37:59] WARNING: [pool www] child 3875706 exited with code 255 after 9.739257 seconds from start
[25-Jul-2025 10:37:59] NOTICE: [pool www] child 3875825 started
[25-Jul-2025 12:25:05] WARNING: [pool www] child 3879179 exited with code 255 after 5685.631784 seconds from start
[25-Jul-2025 12:25:05] NOTICE: [pool www] child 3895546 started
[25-Jul-2025 12:29:02] WARNING: [pool www] child 3879174 exited with code 255 after 5922.564006 seconds from start
[25-Jul-2025 12:29:02] NOTICE: [pool www] child 3896420 started

well, another extremely weird behavior
if i call defer it defaults to the one in the helpers which calls \Illuminate\Support\defer
this however throws a error 502 in production and causes that error, calling \Illuminate\Support\defer directly it works as intended

I am not using octane however i noticed there's a swoole.ini file in my config directory

[sour dagger]

Hi, @visual pier, I have the same issue. Did you find out the reason or just using \Illuminate\Support\defer?

[visual pier]

to the defer call or to the wonky app performance ?

[sour dagger]

I'm interested in both. I still dont understand, why defer function not working, but \Illuminate\Support\defer works. I also use Forge and Hetzner. So I am also curious about performance.

[visual pier]

no to both

the defer call isn't a big issue, since the system seems to work fine regardless

as for the app performance

i'm left only with the nuclear option of trying to move it to hostinger and change db from maria to postgres

and see if it makes a solid difference

[sour dagger]

How do you measure performance in production?