#https discord com channels

44 messages · Page 1 of 1 (latest)

timber garnet
#

@dull grove can you test this and let me know how it works out? ideally link to threads or provide a test case and i could run it on the api if that's what your doing. I don't have gpt4 api key tho.

#

@outer cypress yo can you test it and let us know how well this works for your use case?

outer cypress
#

I'm on 3.5 turbo.

timber garnet
#

ok, if it works on that, it should definitely work on four. if it's too long lmk, we could shorten it

#

@rough dome yo you mentioned json formatting issues, can you test this?

outer cypress
#

Okay, I might need to modify this for my needs

timber garnet
#

you definitely at least need to input your desired schema and context message at the bottom. if you want we could set up nodemon to automatically insert the content from your codebase into the relevant sections whenever you save the files you need to put there.

#

sometimes it's annoying to fiddle with code and copy it into a prompt

dull grove
#

hi, It seems to provide the strict format that is desired. It looks good, I'll test it during the day. Now, what I was doing was to organize more information, but I solved it by making another call to gpt with the existing information
Now think the next step is to see if it breaks in any case for this prompt?

timber garnet
outer cypress
#

Is it absolutely necessary to tell it "You are a master AI JSON Generator"? I'm already giving it an instruction telling it it's an NPC

timber garnet
#

i'm just following directions from andrej karpathy on that, you can fiddle with it. The task specific context like NPC identity would be ideal for the ### Context section inside ## User Input; then you could change the identities and the json formatting mastery can remain

outer cypress
#

Okay, I'll see

timber garnet
#

yeah the lines where you specify the keys, you can convert that into json schema, and that's also the schema you can use to programatically test to ensure the responses have the keys and they are the correct types

#

you could copy all that stuff into context and then make a schema by copying one of the example responseJsonSchemas and tinkering with it in a code editor

# 
{
  "type": "object",
  "properties": {
    "score": {
      "type": "integer",
      "minimum": 0,
      "maximum": 100
    },
    "message": {
      "type": "string",
      "not": {
        "anyOf": [
          {
            "pattern": "NPC"
          },
          {
            "pattern": "\\*"
          }
        ]
      }
    }
  },
  "required": ["score", "message"],
  "additionalProperties": false
}
#

i asked GPT4 in a thread with context about json schema "can you make a json schema example with a score integer 0-100 and a message string and ensure the message does not contain the substring "NPC" or any "*" characters?"

#

you could change the names of the keys but "response" might be confusing since the response is also the term for the whole generated text with the code block in it. also i wrote the prompt for chat, we could mess with it if you don't need the code block
```json
```

#

basically we could just delete the backticks from the example responses, and we could delete the instructions to put a code block, and the responses can just be the json directly. the code blocks are nice for chat ui but not for api i guess

outer cypress
#

I don't feel like the bit explaining what various data type schema is necessary. When it decides to respond in JSON it responds in proper JSON, it just doesn't always decide to listen to that instruction

timber garnet
#

feel free to mess with it, shortening it would be good, translating it to json would be good

#

we could programatically ablate each instruction bullet point and try to see how it impacts the success rate

#

another way to dramatically increase the success rate could be to use a simpler (i.e. flat) format like CSV of (path, value) and then unflatten that to json on the client side. it'd be more likely to get csv correct since it's a simpler format

outer cypress
#

so far it appears to be working very well.

#

Oh, actually I was working out of the wrong file, so it appears that it just started acting right between yesterday and today.

#

Ah, no, my catch code is just working well

timber garnet
#

nice! that looks fun! what game engine are you using?

outer cypress
#

This is just Minecraft on a PaperMC server with CommandHelper running on it, all the ChatGPT handling is through MethodScript and CommandHelper

rough dome
timber garnet
#

It's possible that, since your examples are structured as ## User Input and ## Result, then if you want to avoid formatting issues with the response, you need to mirror the same structure instead of switching it up. Also, if you're using an API, then the 2nd version of the prompt is better if you don't ened the code block. Here is it in this context:

📎 strict_json_generator_api.md
#

Also, recall they pay close attention to the number of hashes in the sub-headings to denote nesting. So you might want to tinker with your subheadings and your Begin Response line to ensure you get a great response in the correct format. 1sec ill show you exactly what i mean

timber garnet
#

Also, note the instructions mention a schema, but you didn't put a responseJsonSchema; Would you be open minded to try a more strict json schema? In general, for consistent results and correctness in programming, you will get a lot better results with a stronger / more clear type system @rough dome.

{
  "description": "Schema for a person. A person is defined by an entity name, their profession, and their nationality.",
  "type": "object",
  "properties": {
    "type": {
      "type": "string",
      "const": "person",
      "description": "The type of entity. In this case, it's always 'person'."
    },
    "entity": {
      "type": "string",
      "description": "The entity represents the person's name."
    },
    "profession": {
      "type": "string",
      "description": "The person's profession, e.g., pirate, corsair, governor, king."
    },
    "nationality": {
      "type": "string",
      "description": "The person's nationality, e.g., French, English, Ottoman."
    }
  },
  "required": ["type", "entity", "profession", "nationality"],
  "additionalProperties": false
}
# 
{
  "description": "Schema for a place. A place is defined by an entity name, the country it belongs to, and its type.",
  "type": "object",
  "properties": {
    "type": {
      "type": "string",
      "const": "place",
      "description": "The type of entity. In this case, it's always 'place'."
    },
    "entity": {
      "type": "string",
      "description": "The entity represents the place's name."
    },
    "country": {
      "type": "string",
      "description": "The country to which the place belongs."
    },
    "placeType": {
      "type": "string",
      "description": "The type of place, e.g., fort, city, port."
    }
  },
  "required": ["type", "entity", "country", "placeType"],
  "additionalProperties": false
}
#

what sort of properties do you need for these various schemas @rough dome? how do you use them, is it a realistic historical game? what programming language are you using? more info can help me get these to a point that's working consistently for you.

const actionJsonSchema = {
  "description": "Schema for an action. An action is defined by an entity name, a description, and its duration.",
  "type": "object",
  "properties": {
    "type": {
      "type": "string",
      "const": "action",
      "description": "The type of entity. In this case, it's always 'action'."
    },
    "entity": {
      "type": "string",
      "description": "The entity represents the action's name."
    },
    "description": {
      "type": "string",
      "description": "A brief summary of the action."
    },
    "duration": {
      "type": "number",
      "description": "The duration of the action in seconds, allowing for partial seconds (e.g., 1.5 seconds)."
    }
  },
  "required": ["type", "entity", "description", "duration"],
  "additionalProperties": false
}
#

Here's a resource description, that ones a bit easier since we can put quantity and units.

{
  "description": "Schema for a resource. A resource is defined by an entity name, a description, its quantity, and the unit of measurement.",
  "type": "object",
  "properties": {
    "type": {
      "type": "string",
      "const": "resource",
      "description": "The type of entity. In this case, it's always 'resource'."
    },
    "entity": {
      "type": "string",
      "description": "The entity represents the resource's name."
    },
    "description": {
      "type": "string",
      "description": "A brief summary of the resource."
    },
    "quantity": {
      "type": "integer",
      "description": "The quantity of the resource, always a whole number.",
      "minimum": 0,
      "maximum": null
    },
    "unit": {
      "type": "string",
      "description": "The unit in which the resource quantity is measured."
    }
  },
  "required": ["type", "entity", "description", "quantity", "unit"],
  "additionalProperties": false
}
#

JSON Schema is pretty powerful; if you want specific properties on specific object categories, then you can definitely make that happen. The descriptions can help instruct the AI LLM about what business logic to consider when it generates each property, so the output will meet your needs better.

# 
    "unit": {
      "type": "string",
      "enum": ["kg", "liters", "pieces"],
      "description": "The unit in which the resource quantity is measured. Can only be 'kg', 'liters', or 'pieces'."
    }

Also, if you only support specific options for one of those strings, then enum is what you need. It constrains the string from crazy town, to be one of a given list. Then your code can just handle the unit options you provide in your schema, and oyu can me more confident the AI won't choose other unsupported units for resources

#

works on gpt-3.5-turbo

#

I even forgot ### JSON: ! whoops 🤣