Custom Multer streaming StorageEngine and FileInterceptor | NestJS | Page 1

lament agate Dec 19, 2024, 3:48 PM

I am trying to create a custom Multer StorageEngine to simply return the Readable stream from Multer in req.file, rather than a Buffer. I then pass this Readable stream through my app to another package form-data which reads the stream to be sent in an HTTP request via Nodes fetch. I know this form-data library works with streams because if I manually create a Readable within the controller (Readable.from(createReadStream("./50Mbfile")) the library unpauses and reads the stream as expected and the file is correctly sent via fetch.

Looking at the @nestjs/platform-express package, multer appears to be set at "multer": "1.4.4-lts.1". I found a comment that states file.stream was added in a 2.x.x version. I have installed Nest 10.4.14 and in my custom Engine it appears that the stream is available (IDE inspection states it is a FileStream) so I assume the comment above isn't quite correct.

Regardless, I am having a difficult time with the StorageEngine definition and I don't know if the fault is with Multer, NestJS or my implementation. I am using a Nest Interceptor and a variation of the custom engine example within the Multer repository

import { FileInterceptor } from "@nestjs/platform-express";
import { Req, Controller, UseInterceptors } from "@nestjs/common";
import { StreamEngine } from "./engine"

@Controller
class MyController
  @UseInterceptors(FileInterceptor("file", { storage: new StreamEngine() }))
  async myConroller(@Req() request) {
    // If FileInterceptor options are passed as the above, 
    // execution never reaches here with large files  
    request.file.stream
  }

For whatever reason, it appears the callback to Multer doesn't seem to resolve.

class StreamEngine implements StorageEngine {
  _handleFile(
    req: Request,
    file: Express.Multer.File,
    callback: (error: Error | null, info?: Partial<AsyncFile>) => void,
  ): void {
    const passThrough = new PassThrough();
    file.stream.pipe(passThrough);

    // Execution reaches this callback, but the execution 
    // doesn't continue in the Nest controller.
    callback(null, { fieldname: file.fieldname, stream: passThrough });
  })
  _removeFile(
    req: Request,
    file: AsyncFile,
    callback: (error: Error | null) => void,
  ): void {
    callback(null);
  }
}

I have removed the error handling and cleaning up of streams to make the example more terse, but I have tried a lot of things, and I cannot figure this out. If I remove the pipe and just return the stream I have the same issue.

Whats more, a red herring that got me for ages was that small files (generally a few hundred Kb) will "work" which really confused me. In that case I thought it was an issue of backpressure - hence the PassThrough, but i'm not sure how much that makes sense because it appears the form-data library has logic to pause on pressure.

lament agate Dec 19, 2024, 4:06 PM

Does the Engine have to be promise based?

#Custom Multer streaming StorageEngine and FileInterceptor