Error deploying during installation of Playwright dependencies | Trigger.dev | Page 1

little plaza Sep 12, 2024, 11:55 AM

#

Hello Trigger team,

In our project, we have a task that generates images from PDFs using Playwright.
We added the following line to our configuration to install Playwright dependencies:
postInstall: "npx -y [email protected] install --with-deps"

During the Docker image build, Playwright fails to install the browsers:

#18 41.22 Installing dependencies...
#18 41.23 Switching to root user to install dependencies...
#18 41.27 Password:
su: Authentication failure
#18 42.27 Failed to install browsers
#18 42.27 Error: Installation process exited with code: 1

Do you have any ideas on how to fix this?

near delta Sep 12, 2024, 2:48 PM

#

Ah yes that won't work. Playwright requires changes to the container to run or root privileges for postinstall which is unusual.

We are just launching a new build system which makes "extensions" like ffmpeg, Playwright etc possible.

First though, I should tell you that we don't currently allow scraping sites that you don't own because you can get our IP addresses blocked (this has happened before).

If that's what you're doing then I recommend you use something like BrightData with Trigger: #1253278287199866900 message

#

Are you scraping sites you have permission to scrape to create the PDFs? If so I can help get Playwright working with our new build system

little plaza Sep 13, 2024, 8:26 AM

#

Hello @near delta ,
We’re using Playwright to open a Chromium browser, inject pdj.js scripts, open a PDF from a buffer, and generate a base64 image from the PDF preview

near delta Sep 13, 2024, 1:14 PM

#

Out of interest, how come you're doing this with Playwright? Is it because you're creating PDFs from web pages and then turning that into images?

#

We're going to create tutorials and extensions for using Playwright/Puppeteer and others with our new build system in the next couple of weeks.

It can be done right now because we just added support for adding binaries with Docker commands.

#

I answered a similar question earlier today, full info here: #1283763713748238440 message

little plaza Sep 13, 2024, 3:51 PM

#

Thank you for the information.
We are importing PDFs into the app and generating image previews of them. Later, we will implement PDF generation from web pages.

I tried using the new build system to integrate Puppeteer:

extensions: [
      {
        name: "puppeteer",
        onBuildComplete: async (context: BuildContext, manifest: BuildManifest) => {
          if (context.target === "dev") {
            return;
          }

          context.addLayer({
            id: "puppeteer",
            image: {
              instructions: [
                `RUN apt-get update && apt-get install curl gnupg -y \
  && curl --location --silent https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - \
  && sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list' \
  && apt-get update \
  && apt-get install google-chrome-stable -y --no-install-recommends \
  && rm -rf /var/lib/apt/lists/*`,
              ],
            },
          });
        },
      },
    ],

The build works, and we can deploy the image. However, when we try to run a task using puppeteer, we encounter the following error:

UnexpectedExitError: Unexpected exit with code -1
    at #handleExit (file:///app/Library/Caches/pnpm/dlx/3m6akg4oukkueqcm6tco2nuxhu/191ebcb6ff4-bd3d/node_modules/.pnpm/[email protected]/node_modules/trigger.dev/dist/esm/entryPoints/deploy-run-controller.mjs:11980:20)
    at ChildProcess.emit (node:events:519:28)
    at ChildProcess._handle.onexit (node:internal/child_process:294:12)

near delta Sep 13, 2024, 4:01 PM

#

My guess is that Node.js has run out of memory. We try and detect this and show a good error message but Node does weird things sometimes and just does a -1 exit.

The default "machine" is designed for doing really simple operations like OpenAI requests. For Playwright you're going to need more power. You can easily change it by following the docs here: https://trigger.dev/docs/machines

Trigger.dev

Machines - Trigger.dev

Configure the number of vCPUs and GBs of RAM you want the task to use.

#

I recommend trying one of the large ones to start with just to see if this was the problem, then you can refine it to a smaller/cheaper one.

#

We're going to add CPU/memory graphs at some point which will make it easier to determine the right machine.

little plaza Sep 13, 2024, 4:23 PM

#

It works, thank you very much

near delta Sep 13, 2024, 4:32 PM

#

@prisma fossil @outer surge Playwright working

little plaza Sep 13, 2024, 4:35 PM

#

@near delta I made Puppeteer working, I haven't tried with playwright

near delta Sep 13, 2024, 4:36 PM

#

Ah sorry 👍

prisma fossil Sep 13, 2024, 4:37 PM

#

little plaza <@947824835755646976> I made Puppeteer working, I haven't tried with playwright

Nice!

outer surge Sep 13, 2024, 4:47 PM

#

Woah awesome

little plaza Sep 13, 2024, 4:49 PM

#

I have forked Trigger repository to add the puppeteer extension, will open a PR soon

coarse finch Dec 17, 2024, 4:08 PM

#

@near delta can you please share how you got playwright working? We are trying to use this in our platform via browserbase's playwright-core package which exports a chromium instance (import { chromium } from "playwright-core") and we're getting the error:

ERROR] Could not resolve "chromium-bidi/lib/cjs/cdp/CdpConnection"

    node_modules/playwright-core/lib/server/bidi/bidiOverCdp.js:8:56:
      8 │ ...equireWildcard(require("chromium-bidi/lib/cjs/cdp/CdpConnection"));
        ╵                           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  You can mark the path "chromium-bidi/lib/cjs/cdp/CdpConnection" as external to exclude it from the bundle, which will remove this error and leave the unresolved path in the bundle. You can also surround this "require" call with a try/catch block to handle this failure at run-time instead of bundle-time.

near delta Dec 17, 2024, 4:23 PM

#

If you add chromium-bidi to external in trigger.config does it fix this problem? https://trigger.dev/docs/config/config-file#external

Trigger.dev

The trigger.config.ts file - Trigger.dev

This file is used to configure your project and how it's built.

#

I haven't got Playwright working personally. There was someone the other day who was talking about this: #general message

#

We don't have a guide for it yet

#

We have Puppeteer but I'm sure you saw that already: https://trigger.dev/docs/guides/examples/scrape-hacker-news

Trigger.dev

Scrape the top 3 articles from Hacker News and email yourself a sum...

This example demonstrates how to scrape the top 3 articles from Hacker News using BrowserBase and Puppeteer, summarize them with ChatGPT and send a nicely formatted email summary to yourself every weekday using Resend.

coarse finch Dec 17, 2024, 4:29 PM

#

Yea saw pupeteer already but a bunch of our code is already in playwright 😦

#

Adding chromium-bidi as external didn't seem to fix it, here's the bottom of my config (I may be doing something incorrectly)

build: {
external: ["chromium-bidi"],
extensions: [
additionalPackages({
packages: ["playwright-core", "chromium-bidi"],
}),
// additionalFiles({
// files: [
// "./actions//*.ts",
// "./constants//.ts",
// "./data/**/.ts",
// "./db//*.ts",
// "./lib//.ts",
// "./utils/**/.ts",
// "./types/**/*.ts",
// ],
// }),
],
},
postInstall: "npx -y playwright-core install --with-deps",

#

I see this in the trigger.dev github repo
https://github.com/triggerdotdev/trigger.dev/blob/main/playwright.config.ts

GitHub

trigger.dev/playwright.config.ts at main · triggerdotdev/trigger.dev

Trigger.dev is the open source background jobs platform. - triggerdotdev/trigger.dev

near delta Dec 17, 2024, 4:48 PM

#

Oh that's us using Playwright to test our web app, not related to running Playwright on top of our service.

#

@outer surge any ideas on using Playwright?

outer surge Dec 17, 2024, 5:45 PM

#

Unfortunately I do not 😭

coarse finch Dec 17, 2024, 6:03 PM

#

Interesting, changing the build part of the config to just

build: {
    external: ["playwright-core", "chromium-bidi"],
}

seems to have made it work, at least locally

#

if it worked in dev can i assume it will work in prod?

coarse finch Dec 17, 2024, 6:21 PM

#

it works!

#Error deploying during installation of Playwright dependencies