tawdry frost
Hi all. I'm working on a thesis evaluator and I want to train Mixtral to predict whether a thesis deserves an A or not. My dataset consists of theses which were graded A and others which were not graded A. I would like to use binary classification to solve this problem, but I can't find any helpful online resources for using an LLM to achieve this. My current implementation is using RAG to feed the model the dataset and using the following prompt to retrieve the data.
'''Below is an instruction that describes a task, paired with an input that provides further context. Write response that appropriately completes the request.
Instruction: Label the thesis based on this question: "Does this thesis deserve a grade A or not?" Below are example labeled theses. Read each thesis mentioned first. Learn from the examples and think step by step before responding. Start your response by printing a "A/Not A" statement first as the label. Then explain why you chose that label.
Thesis: Name of thesis
Label: A
.
.
.
Thesis: Name of thesis
Label: Not A
.
.
.
Input:
Thesis: Name of thesis
Label:
Response: '''
But the output seems to be too biased towards giving As. Any advice on how to go about this? I also have access to a basic marking scheme for a thesis and was wondering if I could maybe add more to my current prompt to make use of this so the model can better evaluate it. Thoughts on this approach?
boreal mural
brave scaffold