#CDY3 Out of Memory

1 messages · Page 1 of 1 (latest)

wicked garden
#

Please continue the discussion here...

Sorry I'm not sure I fully understand that comment. What do you mean by "Removing that and leaving the low memory mode sections appears to work fine." Are you saying that a particular gcode instruction is causing the problem? If you have a small gcode file that causes the problem please post it here.

From the stack trace it looks like you are running out of memory while trying to allocate space for a move segment. That is pretty unusual. Are you changing any of the queue lengths at all (using M595)? Are you using segmented moves (set using the S and T parameters to M669)?

Can you try the following test for me:
Install 3.5.0.rc.3 and reboot the board. Run any startup macros etc. you would normally use then run M122 and save the output), then run as small a test gcode file that know will cause the problem with 3.5.1 and after it has completed run M122 again. Post both M122 outputs here.

Then install 3.5.1 and go through the same start up process (but do not run the gcode file that causes the crash) then run M122 and post that output here as well.

lusty vigil
#

If I delete the sections marked with "(Linearized non-major-plane arc move)" then the gcode runs without crashing.

image.png
lusty vigil
wicked garden
#

Can you also check my questions about M669 and M595 usage. Can you also confirm that the same gcode file runs on 3.5.0-rc.3?

lusty vigil
#

Same gcode file for 3.5.0-rc.3

#

the segment moves is set M669 K0 S5 T0.5 but it's the same for both versions

wicked garden
#

You may want to try increasing T to be T1 and/or lowering S to say S2 and see if that allows the code to run. I suspect you are very close to running out of memory on 3.5.0-rc.3 and something in the later build has tipped things over the limit.

lusty vigil
lusty vigil
shell shoreBOT
#

GG @lusty vigil, you just advanced to level 2!

wicked garden
#

Can you also post the M122 from 3.5.1-rc.3 after running the program that causes the problem.

lusty vigil
lusty vigil
lusty vigil
wicked garden
#

Sorry I thought you said that 3.5.0-rc.3 worked ok?

lusty vigil
wicked garden
#

I want to see the M122 output from 3.5.0-rc.3 after successfully running the gcode file that fails when you use 3.5.1.

lusty vigil
lusty vigil
#

3.5.0-rc.3+101 - success gcode crashed after 60%, but I left DWC upgraded to 3.5.1, going to do it again with all the versions the same.

📎 message.txt
wicked garden
#

That looks like it had a similar problem (out of memory) to 3.5.1, I'm not surprised as the previous M122 showed that there was very little CCMRAM available.

I've created a test build that I think will fix this problem. You can find it here: https://drive.google.com/drive/folders/1qR-3o9RQ84-lDX0PHIc6iQtQYoeqZhwV?usp=sharing Note that I've not tested this at all so please use with great care. This build is a development build and so has a 10 second delay on startup. Please do not share this build with anyone without checking with me first.

lusty vigil
wicked garden
#

Sorry I'm a little confused, did the new build allow your test file to run? When was that M122 taken? It seems to be reporting different values for the number of segments created from previous reports.

lusty vigil
#

the M122 just after startup

#

running the gcode, now

#

... well, it hasn't crashed yet. it would crash immediatly before. fingers crossed

shell shoreBOT
#

GG @lusty vigil, you just advanced to level 3!

wicked garden
#

If it does run to completion please post the M122 output from after it has finished.

lusty vigil
#

probably 5-10 minutes

#

btw i did leave the segment moves as M669 K0 S2 T1

#

@wicked garden any other tests you want me to try?

wicked garden
#

You should probably return the M669 settings to your previous ones and test again (another M122 output if that test works would also be good).

lusty vigil
#

just did that and finished the gcode 😄

#

I'll generate the a new gcode for the CNC job, right now the setting for low memory in Fusion 360

#

with Fusion 360- low memory mode disable- success!!! @wicked garden

📎 message.txt
lusty vigil
#

@wicked garden JasonfromLDO said thanks and made a donation to you.

lusty vigil
#

@wicked garden can I let people in LDO test the firmware? Also how long will it take to be released?

wicked garden
#

Yes you can let them try it, it is not very different to 3.5.1, as to when it will be released I'm not sure yet, depends if any other problems with 3.5.1 show up or not and if the Duet folks decided to make a release not not.

lusty vigil
#

I was able to successfully build the RRF, but I still got these warnings, I feel like maybe one of my dependencies isn't the right version.

wicked garden
#

Sorry I've no idea why you are getting those warnings, I've never seen that. Are you sure you are using the correct version of gcc? I'm not at home right now so I can't provide the details of my setup. Make sure you have the v3.5-dev (note the v at the start) branches of each repo checked out.

lusty vigil
#

gcc version 13.2.1 20231009

wicked garden
#

Yep you really need to use the correct version (or you need to test and fix any problems that using a different version may cause). We will typically update something like gcc at the start of a development cycle to make sure we have plenty of time to identify any changes that the new compiler may cause. RRF is a fairly complex real time system, so small changes can sometimes have an unexpected consequence.

lusty vigil
#

ok, 12.3.Rel1 builds fine

lusty vigil
wicked garden
lusty vigil
shell shoreBOT
#

GG @lusty vigil, you just advanced to level 4!

wicked garden
#

Yes but doesn't it say to use 12.2 not 12.3?

lusty vigil
#

Also is it ok to share it with the Milo team?

wicked garden
#

I guess so, are they actually seeing any problems? We have not had any other reports of this problem.

lusty vigil
wicked garden
#

If you do pass this on to anyone else, please make them aware it is a test build and that I don't want them to pass it on to others.

lusty vigil
floral valve
# wicked garden I guess so, are they actually seeing any problems? We have not had any other rep...

I'd say we had 3-5 different people with various OOM related issues - some were triggered due to arc moves (which is why the 'low memory' setting in the post-processor that linearizes them exists), but I think CameronFromLDO might be the first person to see issues without any Arc moves in the gcode. It's seemed oddly intermittent, some people have been using the latest versions of MOS on their CDYv3s with every RRF version from like 3.5.0-rc2 up to 3.5.1 with no memory issues at all 🤷‍♂️

wicked garden
#

From what I could see, the amount of CCSRAM was very close to being exhausted in the case of it working. I suspect this is down to the amount of CCSRAM being used by the various scripts that are being used during startup and perhaps other operations (I think you had some problems with this previously). During print execution RRF sometimes needs to allocate additional memory to hold segmented moves and it was this allocation that was failing. This means that the likelihood of failure was a combination of the type of gcode being used and perhaps other scripts that may have been run during configuration and prior to the gcode being started. This failure mode is specific to the stm32f4 version of RRF because of the way that on the f4 memory is split into different regions (some of which can only be used for certain operations). I've modified the allocator to use main RAM if no more CSSRAM is available.

floral valve
# wicked garden From what I could see, the amount of CCSRAM was very close to being exhausted i...

Gotcha - I did suspect you'd done something like that (I assume this is what you were talking about potentially changing pre-3.5.0-release). With the produced gcode there's pretty static steps for each file - output a list of tools, trigger a probing cycle where the work co-ordinate system needs to change, trigger a tool change where necessary, and then start the spindle and run each operation. So in the simplest case, by the time you execute the first "cutting" operation, all of the CCSRAM that will be used up by the "setup" (MillenniumOS) has already been allocated. I was definitely very close to the limit on the Spider King and it was linearising all the arcs in post that allowed me to run a full job reliably.

Where it varies is in the probing cycle / tool change part of things - depending on the probe cycle type the operator selects, a different routine is executed and they all have different requirements for how to calculate a position and run different numbers of probing points. Same with tool changes, there's a number of variables in how 'complex' the tool change is based on the tool being used... so it's perfectly conceivable that the pathways that trigger OutOfMemory are only being encountered by a couple of people in certain circumstances.

RE: The fix, I assume this has performance implications as main RAM isn't core coupled and that's what needs to be okayed before releasing (as well as making sure stm32 / duet versions match up)?

Regardless, thanks for looking into this and sorry to be partly to blame for it 😂

wicked garden
#

I doubt if it will have a measurable impact on performance. The main downside of the fix is that in theory you could exhaust all of main memory by using complex scripts etc. which could cause unexpected results.

floral valve
#

I see, good to know. Is that where we're at on non-f4 chips? (i.e. we'd just run out of main memory given that there's no separation of the memory types)

wicked garden
#

Yes but there is a lot more memory on chips like the STM32H743 so it is much less likely to be an issue.

lusty vigil
#

Any idea on when the Duet team will decide to implement the fix or not?

wicked garden
#

@lusty vigil What fix is that? If you mean the one for out of memory, that has nothing to do with the Duet folks, that is purely an issue on the STM32F4 version.

wicked garden
#

That fix is already in our source code, but I'm currently waiting to see if there will be an update from the Duet folks to fix some other issues. Making a release takes about a week of my time so not something I want to do too frequently.