tsc not finding imports in node_modules? | TypeScript Community | Page 1

gentle burrow May 26, 2024, 8:13 PM

#

I'm using buf with protoc-gen-es to generate protobuf TS. tsc and vscode are happy with that generated code. However, that generated code imports types from @bufbuild/protobuf and the generated types extend those types. When I run tsc it errors on methods inherited from @bufbuild/protobuf.

Some of my code:

        // BusMessage is a generated type
        let bm = new akpb.BusMessage();
        // type is a property on BusMessage; tsc likes this
        bm.type = akpb.ExternalMessageType.SUBSCRIBE;
        // toBinary is inherited from types in @bufbuild/protobuf
        // TS2339: Property 'toBinary' does not exist on type 'BusMessage'.
        this.#socket.send(bm.toBinary());

This is for the browser and I'm trying to use ES modules. protobuf-gen-es defaults to ES-style modules/imports.

Here's the generated code doing its imports:

import type { BinaryReadOptions, FieldList, JsonReadOptions, JsonValue, PartialMessage, PlainMessage } from "@bufbuild/protobuf";
import { Message, proto3 } from "@bufbuild/protobuf";

Here's my tsconfig.json:

{
    "compilerOptions": {
        "module": "es2022",
        "noImplicitAny": true,
        "target": "es2018"
    },
    "include": [
        "src/*ts"
    ],
}

and my package.json:

{
  "name": "ak",
  "main": "main.js",
  "type": "module",
  // ...
  "dependencies": {
    "@bufbuild/buf": "^1.32.1",
    "@bufbuild/protobuf": "^1.9.0",
    "@bufbuild/protoc-gen-es": "^1.9.0"
  },
  "devDependencies": {
    "typescript": "^5.4.5",
    // ...
  }
}

My code lives in ./src/ relative to ./package.json and ./tsconfig.json.

I'm very experienced in other languages and understand the basics of JS and TS, but am totally new to TS tooling.

raw portal May 26, 2024, 8:22 PM

#

if you hover over bm to get type info, what do you see?

#

i'm trying to reproduce the error but don't have enough details to do so. how is akpb defined?

gentle burrow May 26, 2024, 8:27 PM

#

let bm: akpb.BusMessage

#

import * as akpb from "../pb/autonomouskoi_pb";

#

In vscode, bm -> Go To Type Definition takes me to ../pb/autonomouskoi_pb.d.ts, which is generated. That's what's doing the imports from @bufbuild/protobuf

raw portal May 26, 2024, 8:31 PM

#

thanks but that's still not enough info for me to reproduce the error. what does ../pb/autonomouskoi_pb.d.ts (or its source file) look like?

#

there's presumably a BusMessage class defined in there. that's what i need to see

gentle burrow May 26, 2024, 8:32 PM

#

Yup:

#

/**
 * @generated from message autonomouskoi.BusMessage
 */
export declare class BusMessage extends Message<BusMessage> {
  /**
   * @generated from field: string topic = 1;
   */
  topic: string;

  /**
   * @generated from field: int32 type = 2;
   */
  type: number;

  /**
   * @generated from field: optional autonomouskoi.Error error = 3;
   */
  error?: Error;

  /**
   * @generated from field: optional bytes message = 4;
   */
  message?: Uint8Array;

  /**
   * @generated from field: optional int64 reply_to = 5;
   */
  replyTo?: bigint;

  constructor(data?: PartialMessage<BusMessage>);

  static readonly runtime: typeof proto3;
  static readonly typeName = "autonomouskoi.BusMessage";
  static readonly fields: FieldList;

  static fromBinary(bytes: Uint8Array, options?: Partial<BinaryReadOptions>): BusMessage;

  static fromJson(jsonValue: JsonValue, options?: Partial<JsonReadOptions>): BusMessage;

  static fromJsonString(jsonString: string, options?: Partial<JsonReadOptions>): BusMessage;

  static equals(a: BusMessage | PlainMessage<BusMessage> | undefined, b: BusMessage | PlainMessage<BusMessage> | undefined): boolean;
}

raw portal May 26, 2024, 8:33 PM

#

well, this works:

tawdry sinewBOT May 26, 2024, 8:33 PM

#

mkantor#0

Playground Link

Preview:```ts
import {Message} from "@bufbuild/protobuf"

export declare class BusMessage extends Message<BusMessage> {}

const bm = new BusMessage()
bm.toBinary()```

gentle burrow May 26, 2024, 8:33 PM

#

And I can find definitions for Message in node_modules/@bufbuild/protobuf/dist/esm/message.d.ts

raw portal May 26, 2024, 8:34 PM

#

tawdry sinew

so the issue is probably a configuration difference between that playground and your project (which you might have already suspected)

gentle burrow May 26, 2024, 8:34 PM

#

Yup. I'm 95% confident this is me not understanding the tools

raw portal May 26, 2024, 8:38 PM

#

is TS2339: Property 'toBinary' does not exist on type 'BusMessage'. the only error you see when you build?

gentle burrow May 26, 2024, 8:38 PM

#

yes. Same between tsc and vscode

#

and vscode doesn't know what to do for Go to Type Defintion on Message in the definition of BusMessage

raw portal May 26, 2024, 8:41 PM

#

gentle burrow and vscode doesn't know what to do for _Go to Type Defintion_ on `Message` in th...

ah, that's interesting. i know i only asked to see BusMessage before, but can you also share any imports in that file?

gentle burrow May 26, 2024, 8:42 PM

#

The entirety of imports in pb/autonomouskoi_pb.d.ts:

#

import type { BinaryReadOptions, FieldList, JsonReadOptions, JsonValue, PartialMessage, PlainMessage } from "@bufbuild/protobuf";
import { Message, proto3 } from "@bufbuild/protobuf";

#

same as above

raw portal May 26, 2024, 8:42 PM

#

huh, so looks like it is the right Message

gentle burrow May 26, 2024, 8:42 PM

#

In pb/autonomouskoi.js:

#

import { proto3 } from "@bufbuild/protobuf";

#

I've been flailing madly at this for some time. I've tried building with gulp, not knowing what I'm doing. I've changed a bunch of things in tsconfig.json. Is there a cache that might be populated that's confusing things?

raw portal May 26, 2024, 8:44 PM

#

just as a sanity check, if you write this code in the same file as where you saw your original problem, does it typecheck?

declare class Test extends Message<BusMessage> {}
const bm = new Test();
bm.toBinary()

raw portal May 26, 2024, 8:45 PM

#

gentle burrow I've been flailing madly at this for some time. I've tried building with `gulp`,...

probably not if you see the same error in vscode and when running tsc from the command line

gentle burrow May 26, 2024, 8:47 PM

#

I have bus.ts (hand-written) imports ../pb/autonomouskoi_pb (generated) imports @bufbuild/protobuf (in node_modules). Pasting your sample in bus.ts errors on basically everything because BusMessage and Message are not imported/defined

raw portal May 26, 2024, 8:48 PM

#

gentle burrow I have `bus.ts` (hand-written) imports `../pb/autonomouskoi_pb` (generated) impo...

ah sorry, what about this then?

import { Message } from "@bufbuild/protobuf"
declare const bm: Message
bm.toBinary()

gentle burrow May 26, 2024, 8:49 PM

#

A clue!

#

error TS2792: Cannot find module '@bufbuild/protobuf'. Did you mean to set the 'moduleResolution' option to 'nodenext', or to add aliases to the 'paths' option?

3 import { Message } from "@bufbuild/protobuf"

#

I'm not trying to do node imports though. Basically finished system will load code dynamically so I don't know at JS/TS build time what I'll have

#

but that's an orthogonal issue.

raw portal May 26, 2024, 8:51 PM

#

ah, yeah that is a good clue. i don't think i've used "module": "es2022" to target browsers (usually i'm in backend-land and targeting node, or when i do dabble in frontend stuff it's in projects that use a bundler)

#

maybe try "moduleResolution": "node16"? even if that's not actually what you want, i'm curious if it solves the issue

gentle burrow May 26, 2024, 8:53 PM

#

Where would "moduleResolution": "node16" go. tsconfig.json compilerOptions?

raw portal May 26, 2024, 8:54 PM

#

yep. see https://www.typescriptlang.org/tsconfig/#moduleResolution

gentle burrow May 26, 2024, 8:54 PM

#

Oh yeah, that changes things. One sec

#

error TS2835: Relative import paths need explicit file extensions in ECMAScript imports when '--moduleResolution' is 'node16' or 'nodenext'. Did you mean '../pb/autonomouskoi_pb.js'?

1 import * as akpb from "../pb/autonomouskoi_pb";

#

But now it's happy with bm.toBinary()

raw portal May 26, 2024, 8:55 PM

#

ah yes, you probably do want .js on your import paths in any case if you're doing native ESM

#

e.g. change import * as akpb from "../pb/autonomouskoi_pb" to import * as akpb from "../pb/autonomouskoi_pb.js"

gentle burrow May 26, 2024, 8:56 PM

#

Doing that and tsc generates src/bus.js with no errors

raw portal May 26, 2024, 8:56 PM

#

nice

#

okay so i don't know if this is the actual setup you should use. but in order to figure that out i need to know more about:

finished system will load code dynamically

can you give me more details about that? how is the code actually going to be loaded at runtime?

gentle burrow May 26, 2024, 9:00 PM

#

I'm building a framework for bots, initially oriented at Twitch streamers. It's modular, loading plugins as WASM (via WASI). Modules pass messages on an internal bus as protobufs. The bus is externalized via websocket so browser-based stuff can play along; this is primarily useful for OBS overlays.

#

Each module defines its own protobuf schemas, so modules that interact with a given module have to be aware of that schema. For stuff running in the browser, the JS types get generated from the proto schema. Each will require the runtime library for common proto types and serialization code. Technically, each module could come with all the types it needs for other modules, but I'd rather not have complex installs have many copies of the same code.

#

In my head, a module provides the JS for its proto types and the plugin host makes the proto types available from one path.

#

I'm hoping when I get to that point I have a contributor who understands how things work in the browser 😂

raw portal May 26, 2024, 9:06 PM

#

gotta run, sorry. i'll probably be back online later though

gentle burrow May 26, 2024, 9:06 PM

#

I really appreciate you getting me to something that works

raw portal May 26, 2024, 9:44 PM

#

back now, though i'm not sure i can offer any specific advice based on that explanation. i guess one thing is that i think it's not uncommon for packages like what you're describing to end up needing different builds for different runtimes

#

would each plugin have code like what you shared baked into it? or is that part of a single supporting library that is external to the plugin itself?

gentle burrow May 26, 2024, 10:35 PM

#

The program hosting the plugins is itself an HTTP server and it can adjust the content it serves dynamically. Let's say you write a plugin that can fetch user profile data. You define a protobuf schema that includes a request and a response for that profile data. I write a plugin that interacts with yours. I'll need to build against your schema to be able to interact with your code. I package up my plugin with a manifest saying what dependencies it has. When the end user loads the plugin, I'd like it if that plugin could just include your generated code from the HTTP server rather than each plugin having to ship with a copy of the generated code of each other plugin they want to interact with

#

This would (probably) function properly if every plugin included a copy of everything it needed, but that's messy and inelegant.

#

I'm not really ready to try to tackle that problem though, so don't sweat it. I do have a tangential question though. export declare class BusMessage extends Message<BusMessage> {} seems like it's defined in terms of itself. What's up with that?

#

!resolved

raw portal May 26, 2024, 11:37 PM

#

gentle burrow I'm not really ready to try to tackle that problem though, so don't sweat it. I ...

it is, but that's generally fine. here's another example of that sort of thing:

#tsc not finding imports in node_modules?