Validation Accuracy extremely high but Evaluation Accuracy 50% | TensorFlow Plus | Page 1

late raven Jul 20, 2023, 6:42 PM

#

I'm trying to make a model that can differentiate between lists of completely random values between 0 and 1 and similar lists but taking the square root of each value instead.

I'm using this model:

model = tf.keras.Sequential([
    tf.keras.layers.Dense(100, activation='relu'),
    tf.keras.layers.Dense(100, activation='relu'),
    tf.keras.layers.Dense(1, activation='sigmoid')
])

The validation accuracy is always basically immediately 100% (as you can see in the graph), which I think makes sense because the data classes are pretty different. But when I try to evaluate the resulting model, the accuracy is always 50%, in fact it always predicts values very close to 0 (like 7.337504e-10).

I don't even get how this happens since I thought a high validation accuracy also means the actual accuracy should be high, especially in a clear case like this, where the data is just randomly generated anyway. Hope you can help me, I have no idea what to do now

desert wagon Jul 21, 2023, 8:40 AM

#

late raven I'm trying to make a model that can differentiate between lists of completely ra...

trying to differentiate between lists of completely random values? how will that work?

late raven Jul 21, 2023, 8:41 AM

#

one list has completely random values, for the other one I generate a list of random values and then take the square root of each one of them

desert wagon Jul 21, 2023, 8:43 AM

#

late raven one list has completely random values, for the other one I generate a list of ra...

I doubt its possible with with just Dense layer.... if you think about it roots relation is not linear in any sense

#

a linear relation ie y = w*x + b is easily replicated in a dense NN because a is basically the weight and b is bias

#

roots are very difficult to be expressed in this kind of setup...

late raven Jul 21, 2023, 8:46 AM

#

ok that makes sense, I was just trying to create data that is pretty clearly different but still fairly random

desert wagon Jul 21, 2023, 8:46 AM

#

https://www.ga-ccri.com/can-a-neural-net-learn-the-quadratic-formula @late raven you can refer to this... the ways it can be done

GA-CCRi

Can a Neural Net Learn the Quadratic Formula?

At first glance, this problem seems trivial. Neural nets are sophisticated technical constructs capable of advanced feats of machine learning, and you learned the quadratic formula in middle school. But an interesting property of classifiers was revealed trying to solve this issue.

late raven Jul 21, 2023, 8:46 AM

#

but I still don't get why the validation accuracy is so high then?

desert wagon Jul 21, 2023, 8:49 AM

#

late raven but I still don't get why the validation accuracy is so high then?

have you tried taking the model and predicting for some set of inputs like 2, 3, 4, 5 in a loop what's the results for them?

late raven Jul 21, 2023, 8:51 AM

#

how do you mean exactly?
I did this but I'm not sure if that's what you mean:

Data Avg: 0.4775797046445496 | Sqrt: False | Prediction: tf.Tensor(3.4331475e-07, shape=(), dtype=float32)
Accuracy: 1.0
Data Avg: 0.6436547380302116 | Sqrt: True | Prediction: tf.Tensor(5.212499e-09, shape=(), dtype=float32)
Accuracy: 0.5
Data Avg: 0.5033663893421858 | Sqrt: False | Prediction: tf.Tensor(1.3627226e-07, shape=(), dtype=float32)
Accuracy: 0.6666666666666666
Data Avg: 0.5139690135268022 | Sqrt: False | Prediction: tf.Tensor(1.2826035e-07, shape=(), dtype=float32)
Accuracy: 0.75
Data Avg: 0.6937483271019897 | Sqrt: True | Prediction: tf.Tensor(4.5154125e-10, shape=(), dtype=float32)
Accuracy: 0.6
Data Avg: 0.7035093223694211 | Sqrt: True | Prediction: tf.Tensor(2.89484e-10, shape=(), dtype=float32)
Accuracy: 0.5
Data Avg: 0.6772444358622769 | Sqrt: True | Prediction: tf.Tensor(4.848934e-10, shape=(), dtype=float32)
Accuracy: 0.42857142857142855
Data Avg: 0.49570931812970637 | Sqrt: False | Prediction: tf.Tensor(2.1911086e-07, shape=(), dtype=float32)
Accuracy: 0.5
Data Avg: 0.6404610695222839 | Sqrt: True | Prediction: tf.Tensor(2.1279773e-09, shape=(), dtype=float32)
Accuracy: 0.4444444444444444
Data Avg: 0.6556066880117803 | Sqrt: True | Prediction: tf.Tensor(1.4314768e-09, shape=(), dtype=float32)
Accuracy: 0.4
Data Avg: 0.6755432661588106 | Sqrt: True | Prediction: tf.Tensor(2.642594e-09, shape=(), dtype=float32)
Accuracy: 0.36363636363636365
Data Avg: 0.5625638236971383 | Sqrt: False | Prediction: tf.Tensor(1.087761e-08, shape=(), dtype=float32)
Accuracy: 0.4166666666666667
Data Avg: 0.5093235831690369 | Sqrt: False | Prediction: tf.Tensor(1.9760425e-07, shape=(), dtype=float32)
Accuracy: 0.46153846153846156
Data Avg: 0.5145146987704189 | Sqrt: False | Prediction: tf.Tensor(2.4959823e-07, shape=(), dtype=float32)
Accuracy: 0.5

#

This is the code:

while True:
    random_data = np.random.rand(1, num_features)
    sqrt = False
    if np.random.rand() > 0.5:
        sqrt = True
        random_data = np.sqrt(random_data)

    print(f'Data Avg: {np.average(random_data)} | Sqrt: {sqrt} | Prediction: ', end='')

    prediction = model(random_data)
    print(prediction[0][0])
    if sqrt:
        if prediction > 0.5:
            correct_predictions += 1
    else:
        if prediction < 0.5:
            correct_predictions += 1
    total_predictions += 1

    print('Accuracy: ' + str(correct_predictions / total_predictions))

desert wagon Jul 21, 2023, 8:57 AM

#

aaaa what are you trying to achieve with this i have clearly missunderstood what are trying to achieve here```py
random_data = np.random.rand(1, num_features)
sqrt = False
if np.random.rand() > 0.5:
sqrt = True
random_data = np.sqrt(random_data)

print(f'Data Avg: {np.average(random_data)} | Sqrt: {sqrt} | Prediction: ', end='')

prediction = model(random_data)

#

this means you giving the models ? sometimes the sqaure root? and sometimes the normal number... and you are expecting what?

late raven Jul 21, 2023, 9:03 AM

#

That code is just a test for the model I am trying to create.
I'm trying to differentiate between two types of inputs:

Completely random numbers
Completely random numbers but with each number having it's square root taken

I'm trying to create a simple "anti-cheat" for a rhythm game, where I want to differentiate between random offsets to inputs and actual human offsets. I thought vastly different lists of values, where even the average value is completely different (0.5 and ~0.707) would be some good trial data for trying to make such a model

#

And it seems to be working, since the validation accuracy is 100%, but then when I actually try to use the model to make predictions the values are always basically 0

topaz crest Jul 21, 2023, 9:04 AM

#

late raven I'm trying to make a model that can differentiate between lists of completely ra...

Yeah, can I see your dataset?

desert wagon Jul 21, 2023, 9:05 AM

#

that's the dataset... a list of random numbers ... a list of randoms numbers but square root

topaz crest Jul 21, 2023, 9:05 AM

#

I mean is he using tf.data.Dataset.from_tensor_slices

#

And especially, are you calling shuffle on your dataset @late raven

late raven Jul 21, 2023, 9:06 AM

#

yeah, I'm doing this:

# Shuffle data
p = np.random.permutation(len(training_labels))
training_data = training_data[p]
training_labels = training_labels[p]

late raven Jul 21, 2023, 9:07 AM

#

topaz crest I mean is he using `tf.data.Dataset.from_tensor_slices`

what does that do?

desert wagon Jul 21, 2023, 9:09 AM

#

my biggest point is... isn't every number square and sqaure root of something... how is it possible to have a relation between them... also I would suggest for classification models you have 2 outputs so that you can have activation level as confidence to check if model is just perplexed

#

but that's beside the point

topaz crest Jul 21, 2023, 9:11 AM

#

late raven This is the code: ```python while True: random_data = np.random.rand(1, num_...

Yeah I guess the best solution imo would be to classify any number lower than the square root of 1 as a squared number and any number above the square root of 1 as a non square rooted number

late raven Jul 21, 2023, 9:13 AM

#

but I can't get any values above 1.0 no?

desert wagon Jul 21, 2023, 9:13 AM

#

topaz crest Yeah I guess the best solution imo would be to classify any number lower than th...

ye something along those lines although probably not square root of 1 as it is one.

desert wagon Jul 21, 2023, 9:13 AM

#

late raven but I can't get any values above 1.0 no?

ye...

late raven Jul 21, 2023, 9:16 AM

#

ok hold up that actually works

#

setting the threshold to a really small value get's my accuracy to over 99%

threshold = 1.0e-08

    if sqrt:
        if prediction < threshold:
            correct_predictions += 1
    else:
        if prediction > threshold:
            correct_predictions += 1
    total_predictions += 1

#

so I guess the validation finds the right threshold on its own somehow? I really don't know enough about NNs lol

desert wagon Jul 21, 2023, 9:19 AM

#

I don't think its possible in theoretical sense but maybe NN is finding some pattern in your data which should be possible normally

#

you can just think as... 2 is a number but is square root too of 4 but it also has a square root 1.41...

#

that's true for all values

#

for 1.41 there exists a sqaure and a sqaure root too... there is no way to differentiate sqaures from their root

late raven Jul 21, 2023, 9:22 AM

#

hmm I don't think so, the validation has to somehow decide if the predicted value means the predicted result leans towards class 1 or class 2, and the threshold for that has to be figured out somehow, since it has an accuracy of 100%

#

now I just have to find the right threshold and i'm golden

#

@desert wagon It doesn't even make a difference if I simplify the model like this:

model = tf.keras.Sequential([
    tf.keras.layers.Dense(10, activation='relu'),
    tf.keras.layers.Dense(1, activation='sigmoid')
])```

#Validation Accuracy extremely high but Evaluation Accuracy 50%