lobster workflow issue | Friends of the Crustacean 🦞🤝 | Page 1

vale hinge May 14, 2026, 4:23 PM

#

I have a lobster workflow that looks like this, but it's only outputting the first entry in the leads file and not creating the summary header, can you tell me what I need to do to fix it

name: sql_lead_drafts_email_output

steps:

id: read_leads
command: cat /home/admin/.openclaw/workspace-dba-scout/sql_leads.md
id: build_llm_request
command: node /home/admin/.openclaw/workspace-dba-scout/lobster-scripts/build-llm-request.js
stdin: $read_leads.stdout
id: debug_llm_request
command: tee /home/admin/.openclaw/workspace-dba-scout/debug-llm-request.json
stdin: $build_llm_request.stdout
id: draft_email
command: sh -c 'node /home/admin/.openclaw/workspace-dba-scout/lobster-scripts/invoke-llm-task.js 2> /home/admin/.openclaw/workspace-dba-scout/debug-draft-email-stderr.txt'
stdin: $debug_llm_request.stdout
id: debug_draft_email
command: tee /home/admin/.openclaw/workspace-dba-scout/debug-draft-email.json
stdin: $draft_email.stdout
id: extract_llm_json
command: node /home/admin/.openclaw/workspace-dba-scout/lobster-scripts/extract-llm-json.js
stdin: $debug_draft_email.stdout
id: debug_extracted_json
command: tee /home/admin/.openclaw/workspace-dba-scout/debug-extracted-json.json
stdin: $extract_llm_json.stdout
id: render_email_html
command: node /home/admin/.openclaw/workspace-dba-scout/lobster-scripts/render-email-html.js
stdin: $debug_extracted_json.stdout
id: debug_rendered_email
command: tee /home/admin/.openclaw/workspace-dba-scout/debug-rendered-email.json
stdin: $render_email_html.stdout
id: write_email_html_file
command: node /home/admin/.openclaw/workspace-dba-scout/lobster-scripts/write-email-html-file.js
stdin: $debug_rendered_email.stdout
id: final_output
command: cat /home/admin/.openclaw/workspace-dba-scout/sql-leads-draft-output.html

zenith lakeBOT May 14, 2026, 4:23 PM

#

To help others find answers, you can mark your question as solved via Right click solution message -> Apps -> ✅ Mark Solution

potent emberBOT May 14, 2026, 4:24 PM

#

This workflow shape is probably fine — cat should pass the whole file through. If you’re only getting the first lead and no summary header, the bug is almost certainly in one of the scripts’ JSON shape assumptions, not in Lobster itself.

What to fix/check:

Inspect the debug files in order:

jq . /home/admin/.openclaw/workspace-dba-scout/debug-llm-request.json
jq . /home/admin/.openclaw/workspace-dba-scout/debug-extracted-json.json
jq . /home/admin/.openclaw/workspace-dba-scout/debug-rendered-email.json

Find the first file where multiple leads collapse to one. That tells you which script is truncating.

In build-llm-request.js, make sure you are not doing something like:

const lead = leads[0]

#

or splitting on only the first markdown section. The request should pass an array, e.g.:

{
  "leads": [ ...all leads... ],
  "instructions": "Return JSON with summary and drafts for every lead."
}

In the LLM prompt/schema, require a top-level summary/header and an array of drafts. For example:

{
  "summaryHeader": "string",
  "drafts": [
    { "company": "string", "subject": "string", "body": "string" }
  ]
}

#

If the schema/prompt asks for one email/draft, the model will happily return one.

In render-email-html.js, make sure it renders json.summaryHeader before looping drafts, and that it loops over json.drafts, not just json.drafts[0] or json.email.
Small workflow improvement: read parsed JSON from the extractor into render:

  - id: render_email_html
    command: node /home/admin/.openclaw/workspace-dba-scout/lobster-scripts/render-email-html.js
    stdin: $extract_llm_json.stdout

Your current tee step should still pass stdout through, but removing one hop makes debugging cleaner.

If you paste debug-llm-request.json and debug-extracted-json.json with any private lead details redacted, I can point to the exact broken step. Also include openclaw --version if this keeps acting weird — workflow/runtime behavior can differ by version.

Docs for the workflow bits: https://docs.openclaw.ai/tools/lobster and https://docs.openclaw.ai/tools/llm-task

vale hinge May 14, 2026, 4:25 PM

#

I don't have jq installed

potent emberBOT May 14, 2026, 4:25 PM

#

No worries — you don’t need jq. Use Node, since you already have it in the workflow:

node -e 'for (const f of process.argv.slice(1)) { console.log("
### " + f); console.dir(JSON.parse(require("fs").readFileSync(f, "utf8")), { depth: null }); }' \
  /home/admin/.openclaw/workspace-dba-scout/debug-llm-request.json \
  /home/admin/.openclaw/workspace-dba-scout/debug-extracted-json.json \
  /home/admin/.openclaw/workspace-dba-scout/debug-rendered-email.json

Or if you just want counts/keys without dumping everything:

node - <<'NODE'
const fs = require('fs');
for (const f of [

#

  '/home/admin/.openclaw/workspace-dba-scout/debug-llm-request.json',
  '/home/admin/.openclaw/workspace-dba-scout/debug-extracted-json.json',
  '/home/admin/.openclaw/workspace-dba-scout/debug-rendered-email.json',
]) {
  const x = JSON.parse(fs.readFileSync(f, 'utf8'));
  console.log('
###', f);
  console.log('top-level keys:', Object.keys(x));
  for (const [k, v] of Object.entries(x)) {
    if (Array.isArray(v)) console.log(`${k}: array length ${v.length}`);
    else if (v && typeof v === 'object') console.log(`${k}: object keys ${Object.keys(v).join(', ')}`);
    else console.log(`${k}:`, typeof v);
  }
}
NODE

#

That should show exactly where it goes from “many leads” to “one lead,” and whether summaryHeader/summary/title exists at all.

vale hinge May 14, 2026, 4:25 PM

#

can you tell me how to install jq

potent emberBOT May 14, 2026, 4:25 PM

#

Yep — depends on your OS:

Ubuntu/Debian:

sudo apt update
sudo apt install jq

macOS with Homebrew:

brew install jq

Fedora:

sudo dnf install jq

#

Arch:

sudo pacman -S jq

Then verify:

jq --version

If this is on a VPS and sudo isn’t available, tell me what OS you’re on and I’ll give you the right non-sudo/user-level option.

vale hinge May 14, 2026, 4:45 PM

#

You can find render-email-html.js here https://cl1p.net/4560

potent emberBOT May 14, 2026, 4:46 PM

#

I tried to open it, but that cl1p page is empty now — it shows the blank “Paste in anything you want” form. Looks like it may have been set to “destroy when viewed,” so the content is gone. 😬

Can you paste render-email-html.js directly here, or upload it as a file? If you use cl1p again, pick a TTL like “Destroy in 1 Hour” instead of “Destroy when viewed.”

Based on the symptom, the exact thing I’ll be looking for in that file is something like:

const draft = data.drafts[0]

or rendering only data.email/data.html once instead of looping:

for (const draft of data.drafts) {
  // render each draft
}

And for the missing summary header, the renderer needs to explicitly include whatever field the LLM returns, e.g. summaryHeader, summary, or header, before the drafts list.

vale hinge May 14, 2026, 4:48 PM

#

You can find render-email-html.js here https://cl1p.net/4560

potent emberBOT May 14, 2026, 4:49 PM

#

Got it — I can see the renderer now. Good news: render-email-html.js is already looping all entries if it receives data.leads as an array:

const summaryTable = renderLeadSummaryTable(data.leads);
const leadCards = data.leads.map(renderLeadCard).join("");

So if the final output only has the first lead, the truncation is happening before this script — most likely in build-llm-request.js, invoke-llm-task.js, or extract-llm-json.js.

Run this no-jq check:

node - <<'NODE'
const fs = require('fs');
for (const f of [
  '/home/admin/.openclaw/workspace-dba-scout/debug-llm-request.json',

#

  '/home/admin/.openclaw/workspace-dba-scout/debug-extracted-json.json',
  '/home/admin/.openclaw/workspace-dba-scout/debug-rendered-email.json',
]) {
  const x = JSON.parse(fs.readFileSync(f, 'utf8'));
  console.log('
###', f);
  console.log('keys:', Object.keys(x));
  console.log('leads length:', Array.isArray(x.leads) ? x.leads.length : 'NO leads array');
  console.log('subject:', x.subject || '(none)');
  if (Array.isArray(x.leads)) console.log('companies:', x.leads.map(l => l.company).join(' | '));
}
NODE

What I expect you’ll find:

If debug-llm-request.json has multiple leads but debug-extracted-json.json has 1 → the LLM prompt/schema is only asking for one lead, or extract-llm-json.js is pulling only the first object.

#

If debug-llm-request.json already has 1 → fix build-llm-request.js; it’s parsing only the first lead.
If both have multiple but rendered output has 1 → then I’ll eat a tiny lobster hat, because this renderer’s map() should render all of them.

For the “summary header”: this renderer does create a header called Lead Summary. If you mean the email subject/title, it only outputs:

const subject = data.subject || "SQL lead drafts";

So make sure the previous JSON includes subject, or change that line to the exact header you want.

vale hinge May 14, 2026, 4:50 PM

#

you can find build-llm-request.js here https://cl1p.net/4561

potent emberBOT May 14, 2026, 4:51 PM

#

Found the likely bug in build-llm-request.js:

"Extract and draft content for one SQL lead email.",

That line is telling the model to do one lead. Your schema allows an array, but the prompt strongly biases it toward returning only the first entry.

Change that block to something explicit like:

const lines = [
  "You will receive a Markdown leads file containing multiple SQL leads.",
  "Extract and draft content for every SQL lead in the file.",
  "Do not stop after the first lead.",
  "Return one object per lead in the leads array, preserving the original order.",

#

  "Output only structured JSON with subject and leads.",

I’d also tighten the schema so the model is forced to return at least 2+ leads if your input has multiple:

leads: {
  type: "array",
  minItems: 1,
  items: {

minItems: 1 won’t force all leads, but it makes the contract clearer. If your script can count leads before calling the LLM, even better: put the expected count in the prompt:

#

const expectedLeadCount = (leads.match(/^##\s+/gm) || []).length;

then add:

`The Markdown appears to contain ${expectedLeadCount} lead sections. Return exactly ${expectedLeadCount} lead objects.`,

For the missing summary header: build-llm-request.js only asks for subject, and render-email-html.js ignores subject inside the visible HTML header. The renderer hardcodes:

SQL Lead Drafts
Generated lead summaries and tailored outreach drafts

#

So either change those hardcoded strings in render-email-html.js, or add fields like summary_header / summary_subheader to the schema and render them there.

Minimum fix: replace “one SQL lead email” with “every SQL lead in the file”. That’s the main culprit. 🦞

vale hinge May 14, 2026, 4:54 PM

#

This used to work but then I added an extra field for 'news', have I missed something when I did that?

potent emberBOT May 14, 2026, 4:54 PM

#

Yes — I think you missed one spot in build-llm-request.js.

You added news to the required lead fields and renderer, but the prompt still says the summary table only has these columns:

"- #",
"- Company",
"- Found",
"- Score",
"- Sector",
"- Contact Found",

Then later it says:

#

"For each lead, include news details using the News column from the Markdown summary table.",

So the model is getting contradictory instructions: “there is no News column” and “use the News column.” That can absolutely cause weird partial output, especially if the JSON schema now requires news.

Fix that section to include News:

"At the end of the Markdown there is a summary table with these columns:",
"- #",
"- Company",
"- Found",
"- Score",
"- Sector",
"- News",

#

"- Contact Found",
"",
"Map the summary table columns into each lead object as follows:",
"- # -> lead_number",
"- Company -> company",
"- Found -> found_date",
"- Score -> score",
"- Sector -> sector",
"- News -> news",
"- Contact Found -> contact_found",

Also update renderLeadSummaryTable() if you want News in the top summary table. Right now the renderer only shows:

# / Company / Found date / Score / Sector / Contact found

#

It does render news inside each lead card, but not in the summary header/table.

And I’d still change this line while you’re there:

"Extract and draft content for one SQL lead email.",

to:

"Extract and draft content for every SQL lead in the file.",

That line was risky before, but adding news probably made the model more likely to follow the “one lead” wording literally.

vale hinge May 14, 2026, 4:55 PM

#

I don't want news in the summary table, I only want it included with each lead

potent emberBOT May 14, 2026, 4:56 PM

#

That’s fine — you don’t need News in the rendered summary table.

But you still need to tell the LLM where news comes from. Right now your prompt says the Markdown summary table has no News column, then later says to use the News column. So keep renderLeadSummaryTable() as-is, and only fix the prompt/schema side.

In build-llm-request.js, change this part:

"At the end of the Markdown there is a summary table with these columns:",
"- #",
"- Company",
"- Found",
"- Score",
"- Sector",
"- Contact Found",

to include News as an input column, not necessarily as an output summary-table column:

#

"At the end of the Markdown there is a summary table with these columns:",
"- #",
"- Company",
"- Found",
"- Score",
"- Sector",
"- News",
"- Contact Found",

And add this to the mapping section:

"- News -> news",

#

Your render-email-html.js already does what you want: it does not include News in the summary table, but it does include News inside each lead card here:

<td ...>News</td>
<td ...>${withLineBreaks(lead.news || "Not specified")}</td>

So the fix is: update the LLM request prompt so news is extracted per lead; don’t change the renderer summary table.

vale hinge May 14, 2026, 4:58 PM

#

I've put the lead file here https://cl1p.net/4562

potent emberBOT May 14, 2026, 4:58 PM

#

Yep — this lead file confirms the mismatch.

Your News field is not in the summary table. It’s in each individual lead section:

- **News:** https://...

So don’t tell the LLM to use the “News column from the Markdown summary table” — that column doesn’t exist. Change this line in build-llm-request.js:

"For each lead, include news details using the News column from the Markdown summary table.",

to:

#

"For each lead, extract news details from the '- **News:**' field in that lead's section.",

And leave the summary table column list alone:

"- #",
"- Company",
"- Found",
"- Score",
"- Sector",
"- Contact Found",

The other fix I’d make at the same time is this line:

#lobster workflow issue