Please help. I can't seem to do one simple Image classification ML

[round swift]

Whati have is something like these images:

all of these are tagged 25 as 25% is what the blue bar occupies in the picture.

I've 2-300 of these images labeled and I've a verification set also labeled with a B inside (check image). So when asking around they told me that if i get the neural net to train over "C" or "A" (I've images for "A" as well) and the verification is on "B" since the white letter is strong the neural net is going to struggle.

So what i did is draw a black filled circle and basically blacken-out the letter (Both in validation and train set). Still no luck.

I build my labels and set programmatically like so.

files = list(filter(lambda k: '.jpg' in k, os.listdir("training/point_capture_c_defence")))
for idx, x in enumerate(files):
    img = cv2.imread("training/point_capture_c_defence/" + x)
    img = cv2.circle(img,(22 ,21), 15, (0,0,0), -1)
    labels.append(100-round(idx/len(files)*100))
    arr.append(img)

then i create the layers and the model like so.

arr = numpy.asarray(arr)
labels = numpy.asarray(labels)
labelsVerif = numpy.asarray(labelsVerif)
verif = numpy.asarray(verif)

model = tf.keras.Sequential([
    tf.keras.layers.Rescaling(1./255, input_shape=(42, 44, 3)),
    layers.Conv2D(16, 3, padding='same', activation='relu'),
    layers.MaxPooling2D(),
    layers.Conv2D(32, 3, padding='same', activation='relu'),
    layers.MaxPooling2D(),
    layers.Conv2D(64, 3, padding='same', activation='relu'),
    layers.MaxPooling2D(),
    layers.Flatten(),
    layers.Dense(128, activation='relu'),
    tf.keras.layers.Dense(101, activation=tf.nn.softmax)
])

and compile+ fit like so.

callback = tf.keras.callbacks.EarlyStopping(monitor="val_loss", patience=50)
model.compile(optimizer = 'adam', loss = 'sparse_categorical_crossentropy')

model.fit(arr, labels, epochs=5000,validation_data=(verif, labelsVerif), callbacks=[callback])

predictions = model.predict(verif)
for idx, x in enumerate(predictions):
    print("Test" + str(idx) + " - " + str(numpy.argmax(x)))

Note that the images are 44x42 but for some strange reason the input shape I need to set is 42,44,3

[undone quail]

I'm confused about the question here

[trail radish]

Your code has a lot of errors.
First of all, this can be done without using DL. Image learning is quite hard. Read about curse of dimensionality. You need a lot of images to reliably classify these things. In this case, I would mask out the ring, then make everything super vibrant so that they are either full red (255,0,0), or full blue. Then I would just basically do simple BLUE/ALLPIXELS.
If you need to do it with DL, dont do it categorically. That just seems very weird to me. Do regression. So, change the last Dense layer activation to linear, with only one output, and loss function to MSE.
You are also normalizing in the model, but you are not also normalizing in validation or predictions. Divide the verification array by 255.0 as well.
You could also try normalizing the output as well, I'm just going to go off topic here and this is quite advanced, but if the output is like big numbers, model can break easily because gradient flows are too big. Try to do it without normalizing the outputs, since thats the recommended way to do it, but if you see that loss function goes to inf, then maybe also try stuff like lowering the learning rate, changing the optimizer, changing loss function, and normalizing the output.

Regarding the input shape, its because your image has 3 channels, red, green and blue. If it was grayscale, TF would still need you to pass it as (44,42,1). In TF, always remember, its (batch_size, image_height, image_width, image_channels). This is different in PyTorch.