Embedding time series for Transformer Encoder.

[ripe moth]

hi guys, I'm trying to add embedding to time series samples, i have samples shape(1,60,1), with values that goes from 0 to 120, i want to add embeddings correctly to the samples, so when i pass something like this to the embedding layer:
input=
shape(1,60,1)
1-2
2-2
3-4
3-4
5-45
..........................................
60-45

the output is something like this=
shape(1,60,4)

1-[0.22, 0.53, 0.11, 023]

2-[0.22, 0.53, 0.11, 023]

3-[-0.12, 0.64, -0.11, 0.93]

4-[-0.12, 0.64, -0.11, 0.93]]

5-[-0.17, -0.66, -0.21, -072]
............................................
60-[-0.17, -0.66, -0.21, -072]

I'm already trying to do it with the code in the (image uploaded), but when i train the model, the train accuracy is always close to 0.5000 and validation accuracy is always 0.5000, so i guess I'm doing something wrong in the embedding or positional encoding part, couse when i tried to solve the problem with keras transformer for time series model (https://keras.io/examples/timeseries/timeseries_classification_transformer/) validation accuracy never got stuck at 0.5000 and it reached at least 0.5400.

[twin mountain]

You mentioned one of the metrics you're using is accuracy, have you tried looking at the outputs, maybe their stuck at one number e.g [0.22, 0,22. 0.22, 0.22] and when compared to your dataset (which am assuming is evenly distributed/is binary) you get an accuracy of 0.50 and its stuck because your outputs are stuck also. But I highly doubt it since you're doing nlp.

For now try and print x/output and see if they are changing during training

[ripe moth]

I'm actually printing some Embeddings and Positional Encoder values and the sum of them, when i tried generating dummy data with shape (6,1) the embeddings and positional encodings values seems correct and the sum of both, but when I try my original data which is shape (1,6,1) or dummy data with the same shape (1,6,1) the embedding and positional encoding values are also right but the sum of them(Embeddings + Positional Encodings) makes no sense, like in the images.

[twin mountain]

Honestly I don't know what's happening, it's especially suspicious when there is no error. Can you send two more pictures with the tensorflow, numpy and random seed set to a certain value. My best guess is that when you change from your original data (1,6,1) to dummy data (6,1) it conflicts with inputs2 (1,6) by broadcasting or some sort of operation that behaves abnormally when you change the data.
Here's an example code of how you can set a fixed seed it helps

import tensorflow as tf
import numpy as np
import random

# Set seeds for reproducibility
seed = 42
tf.random.set_seed(seed)
np.random.seed(seed)
random.seed(seed)

If I think of something more concrete I'll tell you

What does "inputs shape: (1, 6, 1)" and "inputs shape2: (1, 6)" stand for, maybe am confusing something. Same for the other screenshot "inputs shape: (6, 1)" and "inputs shape2: (1, 6)"

[ripe moth]

i solved it, thanks

[twin mountain]

Can you explain what was the bug?