#Why FNC scores for the same predictions are different?

1 messages · Page 1 of 1 (latest)

north nimbus
#

https://signals.numer.ai/i_am_legend
Round 1015 has the same predictions as round 1014 (churn is 0.)
Both are unresolved rounds, which I suspect means they are evaluated with the current "unresolved" target, which supposedly is the same for all unresolved rounds, right?

While the difference in MMC is understandable (someone skipping predictions just for 1 day can skew the results,) why does FNC for the exact same predictions with seemingly exact same targets (due to unresolved rounds), can have almost 2x difference: 1.66% vs 3%? Thank you.

image.png
random ravine
#

Why would the targets be the same for unresolved rounds? Unresolved rounds start out as Corr vs 1 day return, then 2 day return, etc, until you finally get the resolved score with the 20 day neutralized return corr

north nimbus
#

Good point. So for unresolved rounds, it's more like predicted price / current price / price on the day of submission and not predicted price / current price?

scarlet zephyr
#

The first 2 days after submission are ignored -- so the starting reference point price is post-submission. It could make a giant move in those two days and it would just be ignored. (Just to mess with you -- I think that's how it works.) Then the following 20 days returns are added each day for 20 trading days. So submissions from consecutive days (even if they are the same predictions) are being scored with a 1-day difference in accumulated returns until they both get to 20 (but they get to 20 on different days). So...they're different. If you submit identical predictions on two different slots on the same day, then you'd see identical scores.