#How to specify a file url that is generated at build time, but is hashed?

1 messages · Page 1 of 1 (latest)

sharp halo
#

I have a very large JSON file that is generated at build time with data.
I need to fetch this JSON file in my frontend code.

After looking through the documentation, I believe I need to use a plugin to do this.

Here's my plugin:

export function buildJsonFilesPlugin(): VitePlugin {
  const myJSON = buildMyJson();
  const source = JSON.stringify(myJSON);
  return {
    name: "vite-plugin-my-json-builder",
    configureServer(server) {
      server.middlewares.use(async (req, res, next) => {
        if (req.url?.includes("generated-data/FILE.json")) {
          res.setHeader("Content-Type", "application/json");
          res.end(source);
          return;
        }
        next();
      });
    },

    generateBundle() {
      console.log("Building JSON files...");
      const output = this.emitFile({
        type: "asset",
        needsCodeReference: true,
        source,
        name: "FILE.json",
      });
      console.log("Emitted target JSON file with reference ID:", output);
      console.log("Finished building JSON files!");
    },
  };
}

Then, I try to reference it in my code like so:

const url = new URL("generated-data/FILE.json", import.meta.url).toString();
await fetch(url).then(res => res.json()).then((json) => {
    console.log("Fetched FILE at: ", url);
    console.log("Contents are: ", json);
  });

Now, because of the configureServer, this does work with local development. However, the filename in the code snippet is not replaced in the bundled output (when I run vite build).
Perplexingly, the file itself is output to dist/assets/generated-data/FILE-[hash].json (with [HASH]) suitably replaced.

I have been wresting with this trying to figure out how to get vite to not say that new URL("FILE.json", import.meta.url) "doesn't exist at build time"

rugged relic
#

can you not just have the JSON be fetched before vite does the build?

sharp halo
#

What do you mean?

rugged relic
#

sorry, i misread your question

#

sec, i'll try to reproduce

sharp halo
#

here's what I've been trying now... I think i'm making progress?

    load(id) {
      if (id === "virtual:FILE-json-url") {
        referenceId ??= this.emitFile({
          type: "asset",
          needsCodeReference: true,
          source,
          name: "generated-data/FILE.json",
        });
        return `export const FILE_JSON_URL = import.meta.ROLLUP_FILE_URL_${referenceId};`;
      }
    }
    resolveId(id) {
      referenceId ??= this.emitFile({
        type: "asset",
        needsCodeReference: true,
        source,
        name: "generated-data/FILE.json",
      });
      if (id === "virtual:FILE-json-url") {
        return `export const FILE_JSON_URL = import.meta.ROLLUP_FILE_URL_${referenceId};`;
      }
    },
#

Won't work, since resolveID returns the wrong thing... not sure. 🤔

rugged relic
#

ok, let's start over, what's buildMyJson here? what are you trying to achieve?

sharp halo
#

It returns a pretty large JSON object that I construct from some javascript code

#

It's like 500kB

#

But it is generated from stuff that is edited close to our other code

rugged relic
#

me and a friend of mine wrote this plugin some time ago, might be what you're looking for?

#
sharp halo
#

I mean i need the file name to be hashed

#

based on its content

#

Yeah I don't think this is what I'm looking for.

rugged relic
#

so why do you need the file name to be hashed?

sharp halo
#

In short, cache busting.

rugged relic
#

right

sharp halo
#

We've been having problems with stale data already lol

rugged relic
#

relatable

sharp halo
#

I feel like I'm close but I'm just doing something in the wrong order? I see roll-up mentioning this import.meta.ROLLUP_FILE_URL but that seems to be for injections from the plugin...

#

And hashes don't seem to be computed until the generateBundle step..

rugged relic
sharp halo
#

Basically, there's this map of information edited by a different team, that is referenced in typescript code. It's edited by one of our teams (not the dev team).

So like, imagine we have this typescript file that's a huge map of objects, and it's all type safe.
It uses enums, etc.

#

The object itself is right now just lying in typescript code and imported directly, and it literally takes up 120k lines of source code, and contributes something like 10MB of memory

#

So I'm refactoring it to not be left around

#

I don't even want the code to be part of the JavaScript because of how massive it is.

#

Anyway, so I have this builder that just creates a json for it.

All that works. With what I have in generateBundle, the file is being written in dist with the correct content. It's just I can't resolve it to a url I can fetch.

The configureServer is also getting it to work with the dev server

rugged relic
#

this JSON needs to be available in both dev and during build?

sharp halo
#

...and during vitest, but I plan to mock

rugged relic
#

so why isn't this JSON just something you push to a cloud storage ...?

sharp halo
#

Getting JSON to be available during dev is already working. It's the build that isn't.

#

It can change on a whim while the other team edits it, and they use vite start to test their changes

#

Idk what you mean by cloud storage

rugged relic
#

errr maybe i'm missing some context of why you need to do it this way

sharp halo
#

Basically, this is for a web based game. We have this huge list of item compatibility with characters. Our balance team will tweak it like, every update. And we have a beta site so any changes are reflected quickly.

#

So when they change which characters can use which items, e.g., the Json file needs to be regenerated.

#

And when testing the game locally, the item compatibility needs to be correct, based on the version they wrote.

rugged relic
#

where i work, the JSON would be managed in a seperate repo (specifically for storing assets), and changing the JSON and pushing the change would trigger a pipeline to update the file in S3

sharp halo
#

Yeah that's not what we have here.

#

In any case, that's not really the relevant part. The relevant part is how do I resolve a URL to an asset that I generate during build time.

#

A "virtual asset" if you will

sharp halo
rugged relic
rugged relic
#

you know, the only reason you're doing this whole thing is because you want a hash to be part of the file name right?

#
import { defineConfig } from "vite";
import type { Plugin as VitePlugin } from "vite";
import react from "@vitejs/plugin-react-swc";

function buildJsonFilesPlugin(): VitePlugin {
  const virtualModuleId = "virtual:json-file";
  const resolvedVirtualModuleId = "\0" + virtualModuleId;

  const myJSON = { name: "dummy" };
  const source = JSON.stringify(myJSON);

  return {
    name: "vite-plugin-my-json-builder",
    resolveId(id) {
      if (id === virtualModuleId) {
        return resolvedVirtualModuleId;
      }
    },

    load(id) {
      if (id === resolvedVirtualModuleId) {
        return `export const LARGE_DATA = JSON.parse('${source}');`;
      }
    },
  };
}

// https://vite.dev/config/
export default defineConfig({
  plugins: [react(), buildJsonFilesPlugin()],
});

because this technically works, just by using virtual modules

#
import { useState } from "react";
import reactLogo from "./assets/react.svg";
import viteLogo from "/vite.svg";
import "./App.css";
// @ts-expect-error lol
import { LARGE_DATA } from "virtual:json-file";

function App() {
  const [count, setCount] = useState(0);

  console.log(" >>", LARGE_DATA);
sharp halo
#

Well sort of. I still want vite to be able to serve the file

rugged relic
#

i was able to get it working in during build, but apparently in dev, this.emitFile doesn't work

sharp halo
rugged relic
#

you could probably use fs.writeFile and get the hash yourself using sha256 or something

rugged relic
# sharp halo During dev, I use `configureServer` to intercept it. That part I already have wo...

well, try this then

function buildJsonFilesPlugin(): VitePlugin {
  const virtualModuleId = "virtual:json-file";
  const resolvedVirtualModuleId = "\0" + virtualModuleId;

  const myJSON = { name: "dummy" };
  const source = JSON.stringify(myJSON);
  let referenceId: string | undefined = undefined;

  return {
    name: "vite-plugin-my-json-builder",
    resolveId(id) {
      if (id === virtualModuleId) {
        return resolvedVirtualModuleId;
      }
    },

    load(id) {
      if (id === resolvedVirtualModuleId) {
        return `export const FILE_JSON_URL = import.meta.ROLLUP_FILE_URL_${referenceId}`;
      }
    },

    configureServer(server) {
      server.middlewares.use(async (req, res, next) => {
        if (req.url?.includes("generated-data/FILE.json")) {
          res.setHeader("Content-Type", "application/json");
          res.end(source);
          return;
        }
        next();
      });
    },

    buildStart() {
      console.log("Building JSON files...");
      referenceId ??= this.emitFile({
        type: "asset",
        needsCodeReference: true,
        source,
        name: "FILE.json",
      });
      console.log("Emitted target JSON file with reference ID:", referenceId);
      console.log("Finished building JSON files!");
    },
  };
}
sharp halo
#

Oh I can put that in build start??

rugged relic
#

yeah, that works in both dev and build

#

but yeah, you get the emitFile warning

sharp halo
rugged relic
#
Building JSON files...
[plugin:vite-plugin-my-json-builder] context method emitFile() is not supported in serve mode. This plugin is likely not vite-compatible.
Emitted target JSON file with reference ID: 
Finished building JSON files!
Port 5173 is in use, trying another one...

  VITE v7.1.3  ready in 63 ms

  ➜  Local:   http://localhost:5174/
  ➜  Network: use --host to expose
  ➜  press h + enter to show help
sharp halo
#

Oh it doesn't need to lol

rugged relic
#

hm? how does the JSON get created in dev then

sharp halo
#

The json file doesn't need to exist.

#

In dev

#

The server just has to serve the json

#

See the configureServer from my initial post

rugged relic
#

right

sharp halo
#

Btw thank you for all the help

rugged relic
#

np 🙂

sharp halo
rugged relic
#

you play radical red too? 🙂

sharp halo
#

yea

#

A lot lol

#

Actually the project I'm asking about.. It's for pokerogue

sharp halo
# rugged relic np 🙂

In case you're interested, here's my final, working solution:

export function buildJsonFilesPlugin(): VitePlugin {
  // Build and write the TM data JSON file
  const tmsJSON = buildTMJson();
  // Ensure that the keys are always in a consistent order when hashing to ensure consistent filenames
  const source = JSON.stringify(tmsJSON);
  const virtualModuleId = "virtual:tms-json-url";
  const resolvedVirtualModuleId = "\0" + virtualModuleId;
  let referenceId: string | undefined;
  let isBuild = false;
  return {
    name: "pokerogue-build-json",
    configResolved(config) {
      isBuild = config.command === "build";
    },
    // This ensures that the development server will serve the generated JSON file
    configureServer(server) {
      server.middlewares.use(async (req, res, next) => {
        if (req.url?.endsWith("generated-data/tms.json")) {
          res.setHeader("Content-Type", "application/json");
          res.end(source);
          return;
        }
        next();
      });
    },
    resolveId(id) {
      if (id === virtualModuleId) {
        return resolvedVirtualModuleId;
      }
    },

    buildStart() {
      this.addWatchFile("src/data-generators/tms/serialize-tms.ts");
      this.addWatchFile("src/data-generators/tms/tm-species.ts");
      this.addWatchFile("src/data-generators/tms/tm-pool-tiers.ts");
      if (!isBuild) {
        return;
      }
      referenceId ??= this.emitFile({
        type: "asset",
        needsCodeReference: true,
        source,
        name: "generated-data/tms.json",
      });
    },
    load(id) {
      if (id === resolvedVirtualModuleId) {
        if (!isBuild) {
          return `export default "generated-data/tms.json";`;
        }
        if (referenceId === undefined) {
          console.log("Generating TMs JSON...");
          referenceId = this.emitFile({
            type: "asset",
            needsCodeReference: true,
            source,
            name: "generated-data/tms.json",
          });
        }
        return `export default import.meta.ROLLUP_FILE_URL_${referenceId};`;
      }
    },
  };
}

(with the real context)

rugged relic
#

that’s super cool!

#

is pokerogue open source?

sharp halo
rugged relic