I have two models (NEOBASE99 and NEOBASE99T) which are exactly the same but for the former is run locally and uploaded each day (until 10ish? rounds ago) and the latter was uploaded starting with round 534. This was a test to confirm that the results are the same. For the first two completed rounds, the metrics were exactly the same. For round 536, the results were moderately different. Is there any way I can troubleshoot this from my side to determine the difference? Some of the upcoming rounds are also slightly different, but most are exactly the same.
#Difference between model run locally and uploaded model
calm quartz
manic ivy
hi @calm quartz good question, the way i would debug this is
-
download your submission and verify that the generated predictions are the same as the ones you are generating locally to reproduce the issue. see announcements and latest numerapi for examples of how to do this. I am assuming these will be different but good just to double check to make sure this is not a scoring issue.
-
check if you have any indeterminism in your code - do you use any (psuedo) random number generators anywhere? if so, make sure to set the same random seed ie https://numpy.org/doc/stable/reference/random/generated/numpy.random.seed.html
-
check the dependency version numbers between your local env and numerai-predict (https://github.com/numerai/numerai-predict/blob/master/requirements.txt). Be aware that there are multiple "releases" or "tags" of numerai predict with different version numbers https://github.com/numerai/numerai-predict/tags so make sure to double check which version you are on for model upload.
calm quartz
I'm not seeing a submissionDownloadAuth in the NumerAPI Python module as suggested by the announcement.
frail aspen
cc @livid star could we add a function for this, please?
livid star
Sure! However, I am on vacation for 2 more weeks