#Java 11 JIT HotSpot compiler inlining

1 messages · Page 1 of 1 (latest)

outer drum
#

hello, question about c1 and c2 compilers. if the server jvm is the default jvm that is being used, can c1 compilation still occur or is c2 the only compiler that is available? If it can occur is this any different than the c1 from client jvm? Im trying to hunt down a c2 compilerthread0 exception crash when its trying to inline methods and it's only occurring in production. If i exclude inline optimization from the compiler, the crash does not occur. Very recently I also noticed that the crash does not occur if the computer running the application has a pagefile virtual memory above 5000 mb +/-. the crash appears to me as an out of memory crash being manifested as something else but its hard to tell since the application crashes at seemingly random times. anywhere from 20 minutes to over several hours. when the application does crash, An event in windows event viewer is created a few minutes before warning about memory exhaustion and java is one of the programs being mentioned. Using jconsole there does not appear to be any memory leaks. I do have hs_err_pid & replay_pid logs available and the args used are -ea -Xmx512m -Xss2m -XX:CompileThreshold=1500

forest cairnBOT
#

⌛ This post has been reserved for your question.

Hey @outer drum! Please use /close or the Close Post button above when you're finished. Please remember to follow the help guidelines. This post will be automatically closed after 300 minutes of inactivity.

TIP: Narrow down your issue to simple and precise questions to maximize the chance that others will reply in here.

simple marten
#

Java uses tiered compilation by default for quite some time now

#

it first starts in interpreter mode and once some code is hot enough, it is JITed with C1

#

once it's executed even more often, it's further JITed with C2

#

for diagnosing the error, checkthe hs_err_pid file

#

it should tell you where the error occured

#

and make sure you're using the latest version of Java 11

#

and if you can,maybe also try with Java 17

outer drum
#

I think I understand my confusion with c1 and c2 now. As for the crash, I know what method it’s trying to compile/inline when it crashes. I just don’t understand why the jvm would even crash in the first place when c2 is being used. I tried viewing the addresses that occurred in the stack but I don’t have symbols to get any information on the method that’s being executed. This does occur when using java 17 as well.

simple marten
#

Can you show the contents of the hs err pid file?
If there is any sensitive data in it, make sure to redact that

outer drum
#

the 3 native functions at the end of the stack frame are for creating threads iirc

simple marten
#

Does your onGameTick method do anything special?

outer drum
#

it's a subscribed/listener method that handles the majority of the code within the project. Its a rather large method

simple marten
#

it seems like the latest version of Java 11 from coretto is 11.0.20.9.1

#

you might want to try updating to that

#

or even better try with Java 17 or even 20 maybe

outer drum
simple marten
#

Does the error also occur without the -XX:CompileThreshold=1500?

outer drum
#

yes

#

without any args it is still possible to crash

simple marten
#

What is the KeeganToAPlugin#onGameTick method doing?

#

Is that method pretty long?

#

If the method is pretty big, try to split it in smaller methods

outer drum
#

lots of array/list management, pathfinding, invoking methods to run on other subscribed/listened method. its a little over 2k lines

simple marten
#

Normally, the JIT wouldn't even try to compile a single method like this

#

Are you sure it still crashes without the -XX:CompileThreshold=1500?

outer drum
#

with or without this arg it still crashes. it takes longer without

#

the onGameTick is called about every .6 seconds

simple marten
#

Is it really a single method that has almost 2k lines of code?

#

if so, you should split that into multiple methods

outer drum
#

Yes

simple marten
#

Try doing that so that you have no single method with more than ~100 lines of code

outer drum
#

Is it trying to inline the method or could it be trying to inline the methods used inside of it?

simple marten
simple marten
outer drum
#

I have another log somewhere where the crash does not originate from my own code, would you want to examine this one too?

simple marten
#

the JIT works better with small methods

#

like if you have a few methods calling each other that are 2k lines together, it shouldn't be a problem

#

but with a single method that's that big, there are some difficulties

#

I think you might be pretty close to the limit where your method wouldn't be JITed at all

outer drum
#

@simple marten logs provided from a mac user lead me here because the stack contained method names. I considered it a dead end since it was fixed in java 9. Are reoccurring bugs possible? I will try making methods smaller. May I ask what made you suspicious of large methods ?

simple marten
#

8000 bytes is when the JIT normally completely ignores a method

outer drum
#

Do you know if I’m able to easily access symbols to view the jvm in something like IDA to search the addresses that show in the stack? Mac was able to produce the method names but not windows, but I do not know if it’s the same issue for windows

simple marten
#

What do you mean with method names?

#

you could probably extract some things from the DLL

#

but I don't feel like that would help you much

outer drum
#

In the log where it has jvm + address, on a mac machine it shows the method names after the address

simple marten
#

with the DLL/SO file?

outer drum
#

Jvm dll

#

Or whatever mac uses

simple marten
#

I would recommend you to still split up the method first

#

and check whether it still crashes

outer drum
#

Okay I’ll try that. Thank you for taking the time to help 🙏

forest cairnBOT
simple marten
#

Alternatively, you could try to create a cpy of that project and remove stuff from the copy as long as you can reproduce the error

#

you can use git for making checkpoints so you can quickly go back to a point where you can reproduce it if you're familiar with git

outer drum
#

@simple marten do you think its possible this issue could be an oom error that is manifesting as something else? as if the compiler is blowing up before it has a chance to crash with an oom error ? im not sure what it could mean if increasing virtual memory stops the crash, i feel like its related to memory usage. or like if the c2 is demanding to much memory when it begins ? I wont be able to test splitting the code into multiple methods until later, but i will get back to you on that

simple marten
#

maybe even that the off-heap memory is insufficient

#

and even in that case, splitting up your methods into smaller method could help because that would cause the JIT to require less memory to process the method

outer drum
#

okay thats good to know. wasn't sure if it was just a coincidence or not. you mentioned the compilethreshold, can this result in any issues with the compilers if its set to 1500? i noticed that the value is the same that the client jvm uses. maybe its too low to determine if something is "hot" or not

simple marten
#

it changes how fast the JIT activates IIRC

#

I think a lower number means stuff is JITed faster/earlier

#

Also I think that client VM means C1 and server VM means C2

#

though by default, it should use both

outer drum
#

thats what i was confused with earlier, server jvm is the only dll i can see that the application has a handle too

#

wasnt sure if the server jvm has its own client compiler inside of it or not

simple marten
#

but your idea that it goes OOM when during JIT compilation could easily be the cause of your problem here

simple marten
#

the C1 is also called the client JIT/client compiler whiole C2 is the server JIT/server compiler

#

by default, it first runs C1 once the code is sufficiently hot and then runs C2 at a later point in time once it gets even hotter

outer drum
#

so if c1 and c2 have the same compilethreshold, could this be a problem? its hard to find such specific answers on this stuff

simple marten
#

maybe though I don't really think so

outer drum
#

again, ty for the help 🙏 i could pick your brain all day, but i'll stop here until the method splitting is tested

#

well just one more question lol, could large classes being used also be a factor in the crashing?

#

the entire class is 2600 lines

simple marten
#

unlikely

#

but if you have classes that big, you should really consider refactoring them

crisp scaffold
simple marten
#

If so, does it still crash?

crisp scaffold
#

Not yet, just saw @outer drum's comments today it was never actually 2k. He doesn't have the actual source code so was probably just a guess on his end or something I mistakenly told him earlier on

#

Should be no big deal to slice it into smaller chunks I'll give it a shot

#

Basically this onGameTick is housing a whole boatload of other methods that it calls and that's grown to this size.. If you think it would make a difference I can group them together so the onGameTick just calls a function that calls 50 of them, and another function that calls 50 of them, etc

crisp scaffold
#

Again though, thanks for the help I'll split things up and see how it goes before worrying about anything else

forest cairnBOT
#

💤 Post marked as dormant

This post has been inactive for over 300 minutes, thus, it has been archived.
If your question was not answered yet, feel free to re-open this post or create a new one.

crisp scaffold
#

Only time will tell for certain, but it does seem like that may have done the trick, no one has crashed YET as far as I am aware since simply splitting up that method into a few different methods that it calls instead

simple marten
#

if you are able to create a minimal reproducer without any libraries, you might be able to report it to OpenJDK