Struggling to fine-tune DaVinci for custom language output

[crystal sorrel]

I'm trying to fine-tune DaVinci to output a custom language that I use for describing workflows. Here's an example input and desired output:

Input: "Make an apple pie. Include homemade dough. Low sugar. 8 persons"
Desired Output (simplified):
START "make a low sugar apple pie for 8 persons"; ACTOR1 "Cook"; STEP1 ACTOR1 "Gather ingredients"; <other commands> END;

I've created a fine-tuned model with 500 prompt/completion pairs, but its performance is worse than just using ChatGPT with prompt engineering (which still doesn't perform well enough for this task). All the models (raw DaVinci, fine-tuned DaVinci, and ChatGPT) can generate relevant workflow ideas based on the prompt, but they struggle with outputting the exact format I want.

Am I missing something, or is there a better approach to achieve my desired results?

[brazen talon]

That's a fairly normal result from fine-tuning, the base davinci model is pretty bad

[crystal sorrel]

do you know if there is some better way to achieve that kind of result ?

[brazen talon]

Well I think there are 3 options:

1. Fine-tuning
2. Embeddings
3. Prompt engineering

Which one would work best for you? Hard to say

[crystal sorrel]

I guess I have yet to try embeddings 😄

[brazen talon]

Not sure how they would mesh with your use-case

Could you describe your use-case a bit more?

[crystal sorrel]

I build a workflow engine for a BPM collaboration product https://lapala.io (kind of zapier like but which focuses more on human tasks). I want to use openai to generate workflows inside my tool from a user prompt, like in the example I provided in my question. I have found that gpt3.5 and 4 are very good to find good workflow ideas, with good logic etc. But I need the output to be structured so I can parse it into code and use them in my app.

I starting to think that I could just let GPT output the semi-valid outputs it manages to do, and find another way to correct them automatically

[brazen talon]

Yeah probably, GPT-4 is typically very good at following a template you tell it to follow

The way I personally do tasks like that is store the templates/instructions in embeddings and then call those as necessary

But that might be overkill

[crystal sorrel]

you mean you use embeddings as a "giant" prompt engineering ?

[brazen talon]

Well kind of, I just use embeddings to store essentially large prompts (include instructions, examples etc.) and then I can call them with my queries when I need to use them

It's just a fancier way of doing prompt engineering