Excessive Mongo Connections in Deployed Firebase Cloud Functions | NestJS | Page 1

agile thorn Apr 16, 2024, 5:10 PM

#

Hi everyone,

I am working through a persistent issue where Nest’s MongooseModule allows more connections to be created than what is passed the into the MongooseModuleOption’s maxPoolSize property. For example, a single Firebase function instance will open 200+ connections despite having a maxPoolSize of 25. This issue occurs when the app is deployed on a GCP Firebase cloud function. Locally, the number of connections are exactly as one would expect (up to 5-10 on concurrent db calls). For context, the NestJs app in question is a backend API for a web app.

Here are some of the things I tried:

•    To diagnose the issue, I set the minimum and maximum number of cloud functions to “1”. This confirms that a single instance is creating more connections than it should.

•    Based on answers to other Discord/StackOverflow posts that discussed similar issues, I implemented a durable data source provider that uses a random UUID to uniquely defined each Firebase function instance. With a Scope.DEFAULT scope, it did not seem to have an impact on the excessive connections… When using Scope.REQUEST The issue was made significantly worse (1000+ connections).

•    Based on related research, I removed all concurrent database calls from all services, and removed all concurrent requests from the frontend. The rationale was that the number of connections could potentially exceed the set limit if there are multiple synchronous calls to the database.

Has anyone run into this type of issue before? Based on existing threads, it sounds like using a durable provider is the silver bullet… However, since we aren’t using multi-tenancy or a micro-service architecture, I don’t fully understand how this should be implement nor how it achieves anything more favorable than the default behavior of Nest.

I am posting code examples in a subsequent comment since I hit the 2,000 character cap.

Thank you in advance for any help or input.

#

We are using monorepo, so I was hoping to avoid posting a complete repo, but if anyone feels like it would help stir insight, I would be willing to spin something up. Otherwise, here are snippets from the app.module, data-source.module, and data-source.service applicable files:

app.module.ts

@Module({
  imports: [
    MongooseModule.forRoot(process.env.MONGO_URI, {
      minPoolSize: 10,
      maxPoolSize: 25,
      socketTimeoutMS: 10000,
      maxIdleTimeMS: 5000,
    }),
    ManyOthersModule,
  ],
  providers: [SeveralOtherProvider],
})
export class AppModule {}

data-source.module.ts

@Module({
  imports: [
    MongooseModule.forFeature([
      { name: User.name, schema: UsersSchema },
    ]),
  ],
  providers: [DataSourceService],
  exports: [DataSourceService],
})
export class DataSourceModule {}

data-source.service.ts

@Injectable()
export class DataSourceService {
  constructor(
    @InjectModel(User.name) readonly userModel: Model<UserDocument>,
    // and many other models!
    @Inject(REQUEST) readonly requestContext: TRequestContext,
  ) {}
}

outer blaze Apr 17, 2024, 5:50 AM

#

You are more than likely seeing the connections in MongoDB, right? When you say "single firebase function instance" how are you making that determination? How do you know your serverless function isn't getting restarted multiple times, thus creating multiple new connections?

agile thorn Apr 17, 2024, 2:40 PM

#

Hi @outer blaze, thanks for the reply. Yes, that's correct - to clarify, I am seeing the connections on Mongo's Atlas metrics graphs.

That's a great question - I am reasonably confident that the serverless function isn't just being restarted based on the GCP logs. I am able to see the standard Nest bootstrapping logs from the single instance. During this test, I am confirming that there is only one cycle Nest bootstrap logs

outer blaze Apr 17, 2024, 3:56 PM

#

Hmm... 🤔 . Then I don't know what to suggest.

agile thorn Apr 17, 2024, 4:20 PM

#

Ahh well, thank you for trying. At one point, the NestJs docs seemed to somewhat discourage the use of serverless architecture when deploying a Nest App. I can't seem to find that specific verbiage anymore, so maybe this is just a manufactured memory... but I still can't help but feel like this issue is caused by some interaction between the serverless environment and NestJs.

My suspicion is that this wouldn't be an issue if the app was running on a bonafide server... At a glance, would you agree?

outer blaze Apr 17, 2024, 7:42 PM

#

That could very well be. Is there some way to set the firebase function's TTL in some way? I'm not serverless savvy, so I'm sort of poking in the dark. 🤷🏻‍♂️

#

The section about serverless mentions the "cold start" issues that arise with Nest. This is why a normal Nest app isn't a great match for serverless.

#

With serverless, you want small and quick functions. 🙂

agile thorn Apr 17, 2024, 8:01 PM

#

Yeahhh, there is actually - and we have it set at 5 minutes but the problem is the initial burst of connections. I think the TTL would be a solid solution if there was an incremental connection leak, but since the issue arises from the sudden burst of connections on instance instantiation, I don't think shortening it will help much.

That makes sense - and up until now, our strategy has just been to keep a minimum number of instances alive.. Which in hindsight, I think is really just another sign that we need to move over to a server.

outer blaze Apr 17, 2024, 8:04 PM

#

You might get some insight from the Firebase support team too.

#

Maybe....

agile thorn Apr 17, 2024, 8:33 PM

#

Sounds good, I'll explore that avenue next. Thank you for your input and help, @outer blaze!

agile thorn Apr 22, 2024, 3:14 PM

#

For anyone who comes across this post in the future with the same issue...

We ended up resolving this challenge by switching from Google Cloud Function to Cloud Runner. In virtually all ways, we found that this proved to be a better infrastructure for hosting the Nest App (response time, robustness under torture testing, Mongo connections, redis connections, etc). The transition wasn't too bad - we just had to containerize the app via Docker (there are many ways to accomplish the task, this was just the route we took).

TLDR If you want a serverless infrastructure for your Nest application, do yourself a favor and use Google Cloud Runner or something similar from one of the other providers.

outer blaze Apr 22, 2024, 3:45 PM

#

@agile thorn - Nice!!!

#Excessive Mongo Connections in Deployed Firebase Cloud Functions