save_pretrained problem

[hidden vault]

Hey , last week i was finetuning bert i came across this problem like i made this class

class BERTForClassification(tf.keras.Model):

def __init__(self, bert_model):
    super().__init__()
    self.bert = bert_model
    self.fc = tf.keras.layers.Dense(1, activation='sigmoid')

but i wasn't able to save_pretrained(path)
and when saved using keras i wasn't able push it on huggingface hub since it needs config.js and model

i tried to extract weights still there were so many problems that i couldn't figure out
Can anyone tell me what the possible problem could be?

[young yew]

When you load the pre trained BERT model, are you specify using TF? Default model is in Pytorch.

[hidden vault]

Like
Model = from_pretrained
Here??

[young yew]

Well, you can load a pretrained model from huggingface as such:

import torch
from transformers import AutoTokenizer, BertForSequenceClassification

tokenizer = AutoTokenizer.from_pretrained("textattack/bert-base-uncased-yelp-polarity")
model = BertForSequenceClassification.from_pretrained("textattack/bert-base-uncased-yelp-polarity")

inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")

with torch.no_grad():
    logits = model(**inputs).logits

To use the Tensorflow implementation, use

from transformers import AutoTokenizer, TFBertForSequenceClassification
import tensorflow as tf

tokenizer = AutoTokenizer.from_pretrained("ydshieh/bert-base-uncased-yelp-polarity")
model = TFBertForSequenceClassification.from_pretrained("ydshieh/bert-base-uncased-yelp-polarity")

inputs = tokenizer("Hello, my dog is cute", return_tensors="tf")

logits = model(**inputs).logits

Reference the following for that source code:
https://huggingface.co/docs/transformers/model_doc/bert
https://huggingface.co/docs/transformers/v4.28.1/en/model_doc/bert#transformers.BertForSequenceClassification
https://huggingface.co/docs/transformers/v4.28.1/en/model_doc/bert#transformers.TFBertForSequenceClassification

Alternatively, you can also import BERT from tensorflow hub:
https://tfhub.dev/google/collections/bert/1

I personally used the tensorflow hub copy of BERT for text embeddings in my BERT Database repo, but feel free to use whatever method you prefer

[hidden vault]

i used bert_model = TFAutoModel.from_pretrained("bert-base-uncased")

[young yew]

What version of transformers is that on?

[hidden vault]

i used kaggle i did'nt check

[young yew]

check your transformers version

they change stuff frequently between versions so some arguments may not apply

[hidden vault]

ill check but do you think there could be issue with instance of that class that we cant save instances or something

[young yew]

You should have that class come up as a layer if you use model.summary()

There is also this blog post which seems to do a lot of what you are trying to do:
https://huggingface.co/blog/tensorflow-philosophy

And circling back to your question, you can only save Tensorflow models as a SavedModel object or H5 file (which contains weights only)

When you add that layer & encapsulate that into a tf.keras.Model or some other Tensorflow model, you lose the ability to save that bert model with huggingface save_pretrained(). You'd have to save the whole model to a native Tensorflow format and try to either export that to ONNX (or whatever format may be accepted by huggingface).