model has very high validation and training accuracy with low loss, but performs badly when tested

[amber hinge]

I am training a convolutional neural network to classify signal data. it performs very well when training, and achieves 97% validation and training accuracy with training loss of 0.04 and validation loss of 0.14. i am using the keras binary cross entropy loss function.

However, when I test my model manually on individual signals, it appears to classify everything as one class, basically achieving 0% accuracy.

I made a confusion matrix out of 100 test cases out of never before seen testing data immediately after training the model. it achieves very good results with roughly 90% accuracy and around 5 false positive and negatives.

Im not sure why my model performs so well when trained and tested in batches, but performs extremely poorly when tested on individual test cases.

any ideas?

[quaint vapor]

Are you feeding it signals that have been sequentially sampled? In other words, are you feeding it datasets of temporally correlated samples? Are you, for example, classifying ordered batches of smaller windows of a continuously sampled signal?
If your convolutions are not defined properly you might leak data across samples in a batch... can you share the network topology?

[amber hinge]

I managed to fix the issue but I do not understand how the fix works at all

here is an example of what input the model was trained on:

the model was trained to analyze the region of this signal between the two red bars

ignore the blue section

when training, it does great on training and testing data, achieving 97.74% training accuracy and 95.19% validation accuracy.

however, when i feed the model individual signals which appear like this, it does very badly.

however, i found that flipping the signal upside down by subtracting one, results in the model performing extremely well

when testing the model on individual cases, if i modify the signal to appear like this, the model now performs very well despite the fact that it was not trained on data that looks like this

[amber hinge]

the signal is not a temporal signal

this is the network topology

[amber hinge]

also weirdly, when i run a prediction in a different program using the same data that looks like this, it does extremely bad. this is super weird because both programs are running predictions on the same model using the same data, but they perform totally different.

okay never mind, i feel dumb now.
it turns out that the assumption that both models were recieving the same dataset was incorrect. this issue has been resolved.
i have deleted the figures for privacy reasons