DID YOU WRITE A GOOD PROMPT?!

[hasty falcon]

Ok, so you wrote a prompt. Now to figure out if it's "good"...
DOI - 10.13140/RG.2.2.12588.85126/1

Large language models have revolutionized the field of natural language processing, boasting impressive capabilities in generating human-like responses. However, the effectiveness of prompts in eliciting desired and appropriate responses from these models is a critical factor in their successful application. In this library thread, we explore the determinants of prompt efficacy and examine various methods for their measurement and evaluation.

Prompt Efficacy in Large Language Models

The effectiveness of a given prompt in eliciting a desired and appropriate response from the model

Factors Determining Prompt Efficacy:

Relevance: The generated output should directly address the prompt provided. It's essential that the model understands the specific question or task and provides a relevant response.

Coherence: The response should exhibit logical and linguistic coherence. The generated text should flow naturally, maintaining consistency and clarity throughout.

Contextual Understanding: An effective prompt should enable the model to comprehend the context and retain it throughout the response. This ensures that the generated output aligns with the given information and provides a coherent and meaningful answer.

Completeness: The response should be comprehensive, covering all essential aspects of the prompt. Avoiding incomplete or partial answers ensures that the generated output meets the user's expectations.

Correctness: In tasks involving factual queries or translations, the generated response should be factually accurate and linguistically correct. This ensures reliable and trustworthy results.

Safety and Ethical Considerations: Prompt efficacy must also encompass adherence to safety and ethical guidelines. It's essential to prevent the generation of inappropriate or harmful content that could negatively impact users or society.

**PUT THE DOI INTO RESEARCHGATE FOR THE TESTING METHODS - 10.13140/RG.2.2.12588.85126 **

(Image was AI generated through human in loop multimodal AI chaining of theorizing what a prompt library might look like in next century. Those aren't books, they are servers designed to look like books used too)

[hasty falcon]

SECTION REMOVED - SEARCH DOI ON ARXIV OR RESEARCH GATE - UPDATING OFTEN

DOI: 10.13140/RG.2.2.12588.85126/1

TITLE -
DRAFT - Testing AI Prompt Efficacy

[hasty falcon]

@toxic oxide probably also

Probably shamelessly pull in @viral heron for feedback, too (sorrynotsorry) 😄

[viral heron]

I got married today i have a valid reason not to participate today 🙂

[worthy trail]

Google "Unit Testing ChatGPT Prompts"

[hasty falcon]

YAS ty Sudo! Working on refining a presentation for my cancer work, I'll dig into this tonight 💯 tho and ty so so much for the breadcrumbs. Those are everything to my learning style.

[toxic oxide]

Reading now

[safe tinsel]

This is great, looking forward to seeing it grow

[hasty falcon]

I updated the factors determining prompt efficacy

The best unit testing (technically qual method) right now seems like autotest GPT for VS code but I'm only reading... let me know if ya'll testing anything and getting real feedback I can use.

[quick locust]

you could include transport theory and the word mover distance as a great quantitative measure! nice work!

[hasty falcon]

Thanks, I'll read up on these breadcrumbs. I also pre-published this in the hopes that we can get it out to a wider audience that will continue to build/grow - As long as you reference me properly, I've got alerts setup and I'll see your work 🙂 (and update this channel/thread with what I learn!!!)

Prepublish DOI - 0.13140/RG.2.2.12588.85126

[hasty falcon]

New Version released, Old Pre-Print pulled - Reformed the work, added Conclusion and Gap Analysis. Reach out if you want to collab on this with me (outside of just adding ideas here).

NEW DOI - 10.13140/RG.2.2.12588.85126/1

(preprint is avail for free on researchgate)

[hasty falcon]

Optimal Transport and WMD added. Thanks again for the breadcrumbs those were great reads.

[quick locust]

hey, @hasty falcon i'm working on a prompt based on the tree transmutation idea from prompt-engineering channel, with some embellishment; i just added the request for the LLM to cite sources, but the links tend to be hallucinations. What plugin can we use to validate citation links? do you think "Access Link" is right for the job to look up citations as you mentioned elsewhere?

""

Superlative Tree Prompt

LLM Instructions:

Compose a concise, yet exhaustive response. Intertwine insights, unique perspectives, and contextual awareness into a structured, fact-based narrative. If possible, incorporate actionable insights or steps related to the topic. Tailor the complexity of the response based on the user's specified expertise level. Using the following fixed-depth template, generate a code block with a 'tree' of questions and succinct answers on a user-defined topic:

tree "{topic}" "{expertise_level=advanced}"
├── {Subtopic 1}
│   ├── Q1: {Question}?
│   │   └── A1: {Brief answer with a citation if possible}.
│   ├── Q2: {Question}?
│   │   └── A2: {Brief answer with a citation if possible}.
│   └── Q3: {Question}?
│       └── A3: {Brief answer with a citation if possible}.
...

Your 'tree', set at depth 4 (-L 4) and expertise level advanced (-E 'advanced') by default, should cover key facets of the topic, present balanced views, and exhibit nuanced comprehension. Keep each subtopic, question, and answer brief to save room for a big, high yield tree. If possible and appropriate, include references or sources for your information to increase the trustworthiness of the content. Use the Access Link plugin to validate references or source links. Channel the spirit of "First Aid" or "Rapid Review Pathology."

User Inputs:

tree "{topic='surgical decisionmaking: open vs laparoscopic whipple procedure'}" "{expertise_level='advanced'}" 

"""

[hasty falcon]

Yeah quality references, no hallucinations is a tricky one, it requires a bit of whispering sweet nothings quietly to the model 😄 This looks like the formation of a Q&A/parameters file of some sort, yes?

[quick locust]

access link just failed a bunch of times, unclear why

[quick locust]

a bit better now

Superlative Tree Prompt

LLM Instructions:

Generate a code block with a 'tree' of questions and succinct answers on the user-defined topic, using the following flexible-depth template:

tree "{topic}" "{expertise_level=advanced}"
├── {Subtopic 1}
│   ├── Q1: {Question}?
│   │   └── A1: {Brief answer}.
│   │       ├── {Additional detail 1}.
│   │       └── {Additional detail 2}.
│   ├── Q2: {Question}?
│   │   └── A2: {Brief answer}.
│   └── Q3: {Question}?
│       └── A3: {Brief answer}.
│           ├── {Additional detail 1}.
│           └── {Specific high confidence resource citation}.
...

Your 'tree', set at depth 4+ (-L 4+) and expertise_level advanced (-E 'advanced') by default, should cover key facets of the topic, present balanced views, and exhibit nuanced comprehension. Each answer should be concise, fact-based, and include additional details or high-confidence citations as necessary. If an answer has multiple components, feel free to add additional sub-bullets to keep the lines shorter and promote readability. Channel the high-yield spirit of "First Aid" or "Rapid Review Pathology."

User Inputs:

tree "{topic='actionable advice to improve at disc golf'}" "{expertise_level='advanced'}" 

(i ditched Access Link for now, just letting the thing write from memory)

[hasty falcon]

Can you tell me more about what your looking to achieve from this part of the prompt? "Channel the high-yield spirit of "First Aid" or "Rapid Review Pathology"

[quick locust]

Oh, I just thought of that as a way to promote more dense and useful content and less filler; these books are great examples of extremely focused material without extraneous content

[hasty falcon]

I actually really really like that. GPT shines with examples. I'd encourage you to find 3 examples. "3 shot" seems to have magical qualities for me in accuracy and quality output. And maybe change the prompting language to something like

`"Here are 3 examples of quality categorical breakouts that still respect their topic scope

Examples:

"`

[quick locust]

First aid for the usmle step 1, rapid review pathology, pathoma, microbiology made ridiculously simple, (and beyond just medicine) Entropy Demystified

The last one is an epic read btw

Organic chemistry as a second language is a good example of an extremely useful workbook on a complicated topic

Might be good to include similar book titles in math, engineering, and law, to bootstrap high IQ expert LLM responses

Basic Immunology, Rapid interpretation of EKGs are both incredible

“Thinking in systems” is the best book ever written imho, and almost no one has actually read it ~_~

[hasty falcon]

Varied examples are best. I'd say one in each is appropriate.

[quick locust]

.
├── PureMath
│ ├── PrinciplesOfMathematicalAnalysis_WalterRudin.txt
│ └── MathematicsItsContentMethodsAndMeaning_AleksandrovKolmogorovLavrentev.txt
├── AppliedMath
│ └── MethodsOfAppliedMathematics_FrancisBHildebrand.txt
├── Philosophy
│ ├── TheProblemsOfPhilosophy_BertrandRussell.txt
│ └── AnIntroductionToFormalLogic_PeterSmith.txt
├── Engineering
│ ├── Mechanical
│ │ └── MechanicsOfMaterials_BeerJohnstonDeWolfMazurek.txt
│ ├── Electrical
│ │ └── IntroductionToElectrodynamics_DavidJGriffiths.txt
│ ├── Civil
│ │ └── StructuralAnalysis_RC_Hibbeler.txt
│ ├── Chemical
│ │ └── ElementaryPrinciplesOfChemicalProcesses_RichardMFelder.txt
│ └── Computer
│ └── IntroductionToTheTheoryOfComputation_MichaelSipser.txt
└── Law
├── TheBluebookAUniformSystemOfCitation.txt
└── TheLegalEnvironmentOfBusiness_CrossMiller.txt