error: javascript memory leap while doing client side axios requests (recursive function calling) | Node.js | Page 1

subtle sinew May 3, 2023, 4:33 PM

#

@broken knoll here is the code

#

let me look in the logs for the complete error

broken knoll May 3, 2023, 4:34 PM

#

bruh you couldve at least put it in a pastebin or at least a js file painpeko

#

where's the recursive request in question?

subtle sinew May 3, 2023, 4:35 PM

#

oh, im sorry.

#

let me do that

#

😄

broken knoll May 3, 2023, 4:36 PM

#

also, have you tried reducing elements?

subtle sinew May 3, 2023, 4:36 PM

#

hope this is better

📎 memory_leap_cause.js

#

nope, would love to

broken knoll May 3, 2023, 4:37 PM

#

that would probably be a much easier place to start, given that you have an environment and the handling code

#

also this is on the browser, right? have you tried checking the network tab?

subtle sinew May 3, 2023, 4:38 PM

#

no, server threw that error

broken knoll May 3, 2023, 4:41 PM

#

wasn't the recursive call on the client?

subtle sinew May 3, 2023, 4:41 PM

#

 FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
2023-05-03T10:02:10.094048+00:00 app[web.1]:  1: 0xb7a940 node::Abort() [/app/.heroku/node/bin/node]
2023-05-03T10:02:10.094462+00:00 app[web.1]:  2: 0xa8e823  [/app/.heroku/node/bin/node]
2023-05-03T10:02:10.094933+00:00 app[web.1]:  3: 0xd5c940 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/app/.heroku/node/bin/node]
2023-05-03T10:02:10.095393+00:00 app[web.1]:  4: 0xd5cce7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/app/.heroku/node/bin/node]
2023-05-03T10:02:10.095919+00:00 app[web.1]:  5: 0xf3a3e5  [/app/.heroku/node/bin/node]
2023-05-03T10:02:10.096454+00:00 app[web.1]:  6: 0xf4c8cd v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/app/.heroku/node/bin/node]

#

this is the error

broken knoll May 3, 2023, 4:41 PM

#

i was assuming that recursion would be the source of the memory leak

subtle sinew May 3, 2023, 4:41 PM

#

wait.

#

i understand your point now 😄

#

let me show you the client side code, that leads to recurssion, i believe

#

window.onload = async function () {
  showLoader();
  let skip = 0;
  let pages = 0;
  const counts = {
    adds: 0,
    updates: 0,
  };
  const pageLimit = Infinity;
  const base = "https://saptopd.herokuapp.com";
  //const base = "http://localhost:8080";
  const endpoint = `beone/create-deals`;
  const quotationsAddCount = document.querySelector(".quotations-add-count");
  const quotationsUpdateCount = document.querySelector(".quotations-update-count");
  const quotationsAddText = document.querySelector(".quotations-add-text");
  if (quotationsAddCount == null) {
    throw new Error("quotations add count element missing.");
  }
  if (quotationsUpdateCount == null) {
    throw new Error("quotations update count element missing.");
  }

  const confirmation = document.querySelector(".confirmation");
  if (confirmation == null) {
    throw new Error("Confirmation container missing");
  }
  let spinner = document.querySelector(".spinner-border");
  if (spinner == null) {
    throw new Error("Spinner missing");
  }
  while (pages < pageLimit) {
    pages += 1;
    const url = `${base}/${endpoint}/${skip}`;
    const response = await axios.get(url);
    counts.adds += response.data.counts.adds;
    counts.updates += response.data.counts.updates;
    quotationsAddCount.innerHTML = `${String(counts.adds)} Sales Quotations added to Pipedrive`;
    quotationsUpdateCount.innerHTML = `${String(counts.updates)} Sales Quotations updated in Pipedrive`;
    if (response.data.nextLink) {
      const skipIndex = response.data.nextLink.indexOf("skip=");
      skip = response.data.nextLink.substring(skipIndex + 5);
    } else {
      break;
    }
  }
  confirmation.classList.remove("d-none");
  quotationsAddText?.remove();
  spinner.classList.add("d-none");
};

this is the client side code

broken knoll May 3, 2023, 4:46 PM

#

i'm not seeing any recursion here

subtle sinew May 3, 2023, 4:46 PM

#

oh, i thought its the recursion that the while loop is calling it over and over again

broken knoll May 3, 2023, 4:46 PM

#

(btw, the String in the template strings there is unnecessary)

broken knoll May 3, 2023, 4:47 PM

#

subtle sinew oh, i thought its the recursion that the while loop is calling it over and over ...

that isn't recursion, that's just a normal loop

#

recursion is when a function calls itself

subtle sinew May 3, 2023, 4:47 PM

#

mhh, thought this is also recursion 😄

#

thanks, makes sense

broken knoll May 3, 2023, 4:47 PM

#

anyways if you check the network tab from here, are you seeing a lot of requests?

subtle sinew May 3, 2023, 4:47 PM

#

let me check this

#

broken knoll May 3, 2023, 4:49 PM

#

did you restart the server

subtle sinew May 3, 2023, 4:49 PM

#

will restart the server, moment

broken knoll May 3, 2023, 4:51 PM

#

actually not sure if that'll have a significant effect but sounds like a good idea if it doesn't take too long

subtle sinew May 3, 2023, 4:52 PM

#

its itneresting, cant connect anymore to the server

#

so my server is running and restart jumps into it in seconds

#

still testing

#

second please

#

new issues, dont know why they are coming from .... unbelievable 😄

#

i do geterror 407, which sounds like a proxy error, right?

#

ok, now i do get the same error of 407 with different server, which cant be right

#

so this is deployed on heroku i will check this on localhost, just gimme a moment please. by te way: do you still have time or is it taking too long for you already?

broken knoll May 3, 2023, 5:15 PM

#

i might have to go do some other stuff, and it's pretty late, but it's not too time-intensive for me

#

i have no idea how to deal with a 407, sorry

subtle sinew May 3, 2023, 5:16 PM

#

this is concerning

#

this error just came out of the nowhere

#

let me try to dig deeper into it

#

@broken knoll not allowed to tell you this, omg

#

error of 407 happens due to max amount of requests per month is reached, omg

broken knoll May 3, 2023, 5:25 PM

#

sounds like maybe there was a ton of requests going on on the client then

subtle sinew May 3, 2023, 5:26 PM

#

have 25k requests per month

#

upgraded to 1million requests per month

#

will retry now

#

to get back to the initial problem

#

omg 😄 love coding, haha

#

now we are back, yes

#

broken knoll May 3, 2023, 5:29 PM

#

same error?

#

can you try checking the network tab anyways (and reload so it can record)

subtle sinew May 3, 2023, 5:30 PM

#

open the network tab and retry then, right?

broken knoll May 3, 2023, 5:31 PM

#

yeah

subtle sinew May 3, 2023, 5:33 PM

#

#

probably was this causing the memory leap error

#

its a infinite loop

#

#

network tab

#

broken knoll May 3, 2023, 5:35 PM

#

subtle sinew probably was this causing the memory leap error

yeah that definitely seems like an issue

subtle sinew May 3, 2023, 5:37 PM

#

    if (error.code === 401 || error.response.status === 401) {
      console.log("getAllQuotations error: 401 - User Session timed out, retrying...");
      //get data from redis
      const company_id = await redisClient.call("JSON.GET", `user:${req.session.user.email}`, ".company_id");
      const s_url = await redisClient.call("JSON.GET", `company:${company_id}`, ".businessone.s_url");
      const beone_susername = await redisClient.call("JSON.GET", `company:${company_id}`, ".businessone.s_username");
      const beone_spassword = await redisClient.call("JSON.GET", `company:${company_id}`, ".businessone.s_password");
      const beone_scompanydb = await redisClient.call("JSON.GET", `company:${company_id}`, ".businessone.s_companydb");
      //authenticateServiceLayer
      await authenticateServiceLayer(company_id, s_url, beone_susername, beone_spassword, beone_scompanydb);

broken knoll May 3, 2023, 5:38 PM

#

is that recursive?

subtle sinew May 3, 2023, 5:38 PM

#

yes

broken knoll May 3, 2023, 5:38 PM

#

might be better to fail after say 20 tries

subtle sinew May 3, 2023, 5:39 PM

#

catch (error) {
    if (error.code === 401 || error.response.status === 401) {
      console.log("getAllQuotations error: 401 - User Session timed out, retrying...");
      //get data from redis
      const company_id = await redisClient.call("JSON.GET", `user:${req.session.user.email}`, ".company_id");
      const s_url = await redisClient.call("JSON.GET", `company:${company_id}`, ".businessone.s_url");
      const beone_susername = await redisClient.call("JSON.GET", `company:${company_id}`, ".businessone.s_username");
      const beone_spassword = await redisClient.call("JSON.GET", `company:${company_id}`, ".businessone.s_password");
      const beone_scompanydb = await redisClient.call("JSON.GET", `company:${company_id}`, ".businessone.s_companydb");
      //authenticateServiceLayer
      await authenticateServiceLayer(company_id, s_url, beone_susername, beone_spassword, beone_scompanydb);
      await getAllQuotations(req, docTotal, lowerDate, upperDate, skip);
      return;
    } else {
      console.log("getAllQuotations error", error.code, error.message, error.stack);
      // Handle errokr by throwing an error message
      return {
        status: 500,
        message: error.message,
      };
    }
  }```

#

this is the full catch block

broken knoll May 3, 2023, 5:39 PM

#

what function is this in

subtle sinew May 3, 2023, 5:40 PM

#

getAllQuotations

#

so its calling itself again

#

uploading it, mom

#

📎 getAllQuotations.js

#

there you go

#

i added connection_attempts and it should stop after 10 retries. so i would say, its fine now with the amount of retries. what is a bit of uncertainty is the fact that its not able to reauthenticate, even the data is correct

broken knoll May 3, 2023, 5:42 PM

#

that won't work, as the amount of tries won't propagate

#

you'll have to add it to the recursive function

subtle sinew May 3, 2023, 5:43 PM

#

what you mean by that?

#

oh, as a parameter as the function will reset it to 0

#

got it

broken knoll May 3, 2023, 5:43 PM

#

yeah

#

function recurse(..., attempts = 0) {
  ...
  recurse(..., attempts + 1);
}

subtle sinew May 3, 2023, 5:44 PM

#

should i add it to the getallquotations or authenticateservicelayer function

broken knoll May 3, 2023, 5:44 PM

#

getAllQuotations, that's the recursive one

subtle sinew May 3, 2023, 5:44 PM

#

ok

#

async function getAllQuotations(
  req,
  docTotal,
  lowerDate = "",
  upperDate = "",
  skip = 0,
  querySinglePage = false,
  connection_attempts = 0
) { ... }
await getAllQuotations(req, docTotal, lowerDate, upperDate, skip, false, connection_attempts + 1);

#

like that ?

broken knoll May 3, 2023, 5:46 PM

#

sure

#

should be camelCase though

#

(as should the other snake_cased variables)

subtle sinew May 3, 2023, 5:47 PM

#

ah, ty

#

changed to camelCase, thanks.

#

Now lets see whats going on there

broken knoll May 3, 2023, 5:48 PM

#

(shouldn't you be propagating querySinglePage there instead of using false?)

subtle sinew May 3, 2023, 5:48 PM

#

this is my first project where i write functions this way, i really dont know

#

thought false is the correct value for the value of querySinglePage

broken knoll May 3, 2023, 5:51 PM

#

if it's always false then it shouldn't really be a parameter to begin with

subtle sinew May 3, 2023, 5:52 PM

#

no, its not

#

i use this to make tests

#

to only get a singlepage

#

otherwise amount of data could be huge

broken knoll May 3, 2023, 5:53 PM

#

so it'd be false for normal use and true for tests?

#

so it could be either value, in that case it should be propagated yeah

subtle sinew May 3, 2023, 5:54 PM

#

ok, thanks

#

so its correct that way, i assume 😄

#

ok, theres something wrong

#

heavily wrong

#

the route which caused the memory leap

#

now doesnt return any data

#

its the getAllQuotations function that should run and return X amount of datarecords

#

ahhh, wait

#

theres an object that i do send back to the client

#

also the server shoud proceed it

#

and also the client doesnt that data at all tbh

broken knoll May 3, 2023, 6:02 PM

#

try adding some console.log checkpoints throughout where the code should be proceeding?

subtle sinew May 3, 2023, 6:03 PM

#

yep.
it started to console.log that the response got back

#

but not more

#

added on more i believe i know why no data is synchronized at this place

#

yup, so code is executing as expected but not working / solving problem for now

#

also

#

i could see the memory leap if the object is too big that i sent back, could that be possible?

broken knoll May 3, 2023, 6:06 PM

#

if it's trying to buffer and send multiple at once, maybe? but that doesn't seem like the main issue to me

subtle sinew May 3, 2023, 6:07 PM

#

i dont see

#

where this data is used

#

so this is useless to res.json it back tbh

#

i removed it and trying it again

#

the problem is not a real problem. i just dont create new quotations if the company isnt created in the database before

#

perfect. still working with less data

subtle sinew May 3, 2023, 6:33 PM

#

thanks for your help. I would love to stop here first so you dont waste more time. thank you very much, will probably get back to you because of this error. have a great night!

subtle sinew May 8, 2023, 12:06 PM

#

@steel kraken here it is 😄

@broken knoll we are back to the problem

#

the error came up, after trying to get 340 datarecords through an api, which is already (i thought so) paginated:


2023-05-08T11:29:27.729683+00:00 app[web.1]: FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
2023-05-08T11:29:27.730240+00:00 app[web.1]:  1: 0xb7a940 node::Abort() [/app/.heroku/node/bin/node]
2023-05-08T11:29:27.730641+00:00 app[web.1]:  2: 0xa8e823  [/app/.heroku/node/bin/node]
2023-05-08T11:29:27.731102+00:00 app[web.1]:  3: 0xd5c940 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/app/.heroku/node/bin/node]
2023-05-08T11:29:27.731560+00:00 app[web.1]:  4: 0xd5cce7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/app/.heroku/node/bin/node]
2023-05-08T11:29:27.732094+00:00 app[web.1]:  5: 0xf3a3e5  [/app/.heroku/node/bin/node]
2023-05-08T11:29:27.732615+00:00 app[web.1]:  6: 0xf4c8cd v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/app/.heroku/node/bin/node]
2023-05-08T11:29:27.733127+00:00 app[web.1]:```

broken knoll May 8, 2023, 12:10 PM

#

subtle sinew <@100302176047337472> here it is 😄 <@628241756235890708> we are back to the pr...

oh no fubuki_ohno

subtle sinew May 8, 2023, 12:11 PM

#

i dont know how this happens tbh

#

how to share 100 lines of code without boring or bothering you, lol

#

its too much code it looks like, discord auto inserts it as file

📎 getAllQuotationsFromAPI.js

#

above is the nodejs function

#

This is the browser js, that calls the API

📎 getAllDealsClientSide.js

#error: javascript memory leap while doing client side axios requests (recursive function calling)