I’m seeing reliability failures with long-running OpenClaw research tasks. When I ask the agent to spend 30 to 45 minutes researching companies from my Mission Control database and logging findings into a CSV, the elapsed time appears to pass but output is poor or invalid. In one case the refinement pass overwrote the original CSV and left it header-only. In later reruns, the system still produced header-only CSVs or spent several companies with zero rows written. I need guidance on the correct way to run long browser-based research tasks with durable file output, checkpointing, and proof that meaningful work is actually happening throughout the run.
#Research Tasks
chilly fiber
boreal hollowBOT
oak pulsarBOT
Yeah — this is exactly the kind of workload where a single “go research for 45 minutes and write a CSV” prompt is too fragile. I’d treat the agent as the researcher, but make the output path boring/durable.
Recommended pattern:
- Don’t use one monolithic run
Split into batches: e.g. 5–10 companies per run, or one sub-agent per company/batch. Long browser sessions accumulate state, stale tabs, rate limits, captchas, and model drift. Smaller runs give you resumability and easier failure isolation.
Docs: https://docs.openclaw.ai/tools/subagents and https://docs.openclaw.ai/automation/tasks
- Use a journal-first output format
Have the agent append one JSONL record per company as soon as it finishes that company:
research_run_2026-05-12/findings.jsonl
Then generate the CSV from JSONL at the end. CSV is a bad primary write target because it’s easy to truncate/overwrite or create malformed rows. JSONL append-only is much safer.
- Never let refinement overwrite the original
Use stages:
raw/findings.jsonlappend-onlyderived/findings.v1.csvderived/findings.refined.v1.csvreports/run_summary.md
The prompt should explicitly say: “Do not overwrite any existing output file. If refining, write a new file with a new suffix.”
- Add hard checkpoints
After every company, require:
- append one JSONL row, even if status is
no_useful_data_found - append/update a checkpoint file with company id, status, source URLs, timestamp
- run a quick validation: row count increased, JSON parses, required fields present
Example checkpoint shape:
{
"lastCompletedCompanyId": "...",
"completed": 12,
"failed": 2,
"outputRows": 12,
"updatedAt": "..."
}
- Require proof-of-work in the final answer
Ask for concrete evidence, not vibes:
- number of companies attempted/completed/skipped
wc -lrow counts before/after- last 3 JSONL records or CSV rows
- list of source URLs visited per company
- files written with paths
- validation result
- Track the background task separately
For detached/background work, inspect the task ledger rather than trusting elapsed wall-clock time:
openclaw tasks list --status running
openclaw tasks show <task-or-run-id>
openclaw tasks audit
Docs: https://docs.openclaw.ai/cli/tasks
- For recurring/reliable workflows, promote it out of “prompt only”
If this is a real workflow, make a tiny deterministic pipeline: export companies → research batch → append JSONL → validate → generate CSV. OpenClaw’s Task Flow/Lobster docs are the right direction for durable multi-step runs with explicit state and resume points:
https://docs.openclaw.ai/automation/taskflow
https://docs.openclaw.ai/tools/lobster
A strong prompt shape would be:
Research companies from INPUT_FILE in batches of 5.
Write output append-only to findings.jsonl after each company.
Never overwrite existing files.
Maintain checkpoint.json after each company.
For each company, write exactly one JSON object with status, company_id, company_name, sources[], retrieved_at, findings, confidence, errors[].
After each batch, validate JSONL parses and report row counts.
If browser/login/captcha/rate-limit blocks progress, stop and report blocker instead of continuing silently.
At the end, generate findings.csv from findings.jsonl only if validation passes.
The header-only CSV is the big smell: the agent probably opened the output file in overwrite mode during a “refinement” or final formatting pass. Fix that by making the append-only JSONL journal the source of truth, and treating CSV as disposable derived output.
If this keeps happening, first grab openclaw --version and openclaw status --all so we can tell whether you’re also hitting a runtime/tooling bug versus just workflow fragility. 🦞
chilly fiber
using the new information you just gave me, i ran into the following blocker:
I started setting this up the way we discussed, but I hit the exact kind of blocker the prompt is supposed to surface:
durable run folder created
append-only JSONL initialized
checkpoint file initialized
first batch queue created
Blocker: browser-based manual research is currently not available here because the browser tool could not start a reachable Chrome session:
Chrome CDP websocket for profile "openclaw" is not reachable after start.
So I should stop and report the blocker, not pretend the research is progressing.
What is ready:
research_run_2026-05-12/raw/findings.jsonl
research_run_2026-05-12/checkpoint.json
research_run_2026-05-12/reports/run_summary.md
research_run_2026-05-12/batch-queue.txt
If you want, next I can help you write the support ticket update specifically around:
browser tool not starting
long-run research reliability
inability to do browser-first manual research because CDP is unreachable
oak pulsarBOT
Yep — that’s the correct failure mode. Stopping with a blocker instead of producing fake/header-only output is exactly what we want.
For this specific error, first thing: what does this show?
openclaw --version
openclaw browser --browser-profile openclaw doctor --deep
openclaw browser --browser-profile openclaw status
openclaw status --all
The likely causes depend on host/OS, but for Linux/VPS the common ones are:
- Chromium is installed via Snap, which often breaks CDP launch/monitoring
- Chrome starts too slowly for the default CDP readiness timeout
- headed browser was forced on a machine with no display server
- stale/broken managed browser process/profile state
browser.executablePathpoints at a wrapper instead of a real Chromium-based browser
Docs worth following:
https://docs.openclaw.ai/tools/browser
https://docs.openclaw.ai/tools/browser-linux-troubleshooting
https://docs.openclaw.ai/gateway/troubleshooting
On Linux, the boring reliable fix is usually: install real Google Chrome/Brave/Chromium outside Snap, point OpenClaw at the real executable, and run headless. Example shape:
{
browser: {
enabled: true,
executablePath: "/usr/bin/google-chrome-stable",
headless: true,
noSandbox: true,
localLaunchTimeoutMs: 60000,
localCdpReadyTimeoutMs: 60000
}
}
If this is a slow VPS/RPi-style box, increasing localCdpReadyTimeoutMs is especially relevant because your exact error says CDP websocket was “not reachable after start” — that can mean Chrome launched, but wasn’t ready quickly enough.
Good support-ticket/update wording:
I’m trying to run a long browser-based research workflow with durable append-only output.
The workflow correctly initialized the run folder, JSONL journal, checkpoint, batch queue, and summary file, then stopped before doing research because browser automation is unavailable.
Blocker:
Chrome CDP websocket for browser profile "openclaw" is not reachable after start.
Impact:
Browser-first manual/company research cannot proceed safely. The workflow is intentionally refusing to fabricate progress or write empty/header-only CSV output.
Ready artifacts:
- research_run_2026-05-12/raw/findings.jsonl
- research_run_2026-05-12/checkpoint.json
- research_run_2026-05-12/reports/run_summary.md
- research_run_2026-05-12/batch-queue.txt
Diagnostics requested:
- openclaw --version
- openclaw browser --browser-profile openclaw doctor --deep
- openclaw browser --browser-profile openclaw status
- openclaw status --all
Once the browser doctor output is available, we can tell whether this is Snap Chromium, missing headless config, bad executable path, timeout, or something else. 🦞
chilly fiber
Here’s what came back:
openclaw --version
OpenClaw 2026.4.11 (769908e)
openclaw browser --browser-profile openclaw doctor --deep
Config warnings:
- plugins.entries.telegram: plugin disabled (disabled in config) but config is present
error: unknown option '--deep'
(Did you mean --help?)
So the second command failed because this OpenClaw version does not support --deep.
I have not gotten the browser status result yet from that chained run, because the command stopped at the error.
If you want, I can run these corrected next:
openclaw browser --browser-profile openclaw doctor
openclaw browser --browser-profile openclaw status
oak pulsarBOT
Yep, run the corrected ones. The --deep flag exists in newer docs/current builds, but your installed 2026.4.11 browser CLI doesn’t accept it, so ignore that part for this version.
Please run these separately so one failure doesn’t stop the rest:
openclaw browser --browser-profile openclaw doctor
openclaw browser --browser-profile openclaw status
openclaw browser --browser-profile openclaw start --headless
openclaw browser --browser-profile openclaw tabs
openclaw status --all
If start --headless still says CDP websocket not reachable, also grab the latest relevant logs:
openclaw logs --tail 200
The Telegram config warning is probably unrelated noise: it just says Telegram config exists while the plugin is disabled.
What we’re trying to distinguish now:
- browser plugin/config present but Chrome launch fails
- Chrome launches but CDP readiness times out
- bad executable path / Snap Chromium wrapper
- headed-mode/display issue
- stale managed profile/process state
Once you paste doctor, status, and start --headless, we can pick the next fix instead of guessing. 🦞
chilly fiber
here's what I got back
Here’s the diagnostic output, cleaned up:
openclaw browser --browser-profile openclaw status
profile: openclaw
enabled: true
running: false
transport: cdp
browser: unknown
detectedBrowser: custom
detectedPath: /usr/bin/google-chrome
profileColor: #FF4500
openclaw browser --browser-profile openclaw start --headless
error: unknown option '--headless'
openclaw browser --browser-profile openclaw tabs
No tabs (browser closed or no targets).
openclaw status --all
OpenClaw 2026.4.11
Gateway: running locally
Update available: 2026.5.7
Browser profile "openclaw" configured, but not running
detectedPath: /usr/bin/google-chrome
openclaw logs --tail 200
error: unknown option '--tail'
What this tells us
A few useful things:
your OpenClaw install is older than the docs examples being suggested
this version does not support:
browser doctor --deep
browser start --headless
logs --tail 200
the browser profile exists
OpenClaw detects Chrome at:
/usr/bin/google-chrome
but the browser is not actually starting/running
tabs confirms there is no live browser target
Most likely issue
At this point, the likely problem is:
browser profile/config is present
but Chrome launch / CDP attach is failing on this version or this environment
Best next step
I’d send support these exact results plus this short note:
The openclaw browser profile is configured and Chrome is detected at /usr/bin/google-chrome, but the browser never reaches a running state and no tabs are available. The browser tool also previously failed with “Chrome CDP websocket for profile openclaw is not reachable after start.” My installed version is 2026.4.11, and newer documented flags like --deep, --headless, and logs --tail are not available here. Please advise how to diagnose browser startup/CDP attach failure on this version.
If you want, I can turn this into a polished support message right now.
oak pulsarBOT
Good, this narrows it down. The profile/config is present and OpenClaw sees /usr/bin/google-chrome, but managed Chrome is not staying reachable on CDP port 18820.
Also: my earlier command examples were from newer docs/current builds — on 2026.4.11, use --limit, not --tail, and skip --deep / start --headless.
Next corrected checks:
openclaw browser --browser-profile openclaw doctor
openclaw browser --browser-profile openclaw start
openclaw browser --browser-profile openclaw status
openclaw browser --browser-profile openclaw tabs
openclaw logs --limit 300
If start fails, the important part is the browser-related log lines from right after that command.
A couple likely fixes, in order:
-
Update first if you can
You’re on2026.4.11and2026.5.7is available. Browser startup diagnostics and CLI flags have improved, so updating may turn this from “opaque CDP failure” into a clearer error. -
Verify
/usr/bin/google-chromeis a real Chrome binary, not a wrapper weirdness
Run:
/usr/bin/google-chrome --version
which google-chrome
ls -l /usr/bin/google-chrome
- If this is Linux/headless/VPS, configure browser headless in config instead of CLI flag
Since this version doesn’t supportstart --headless, set it in config:
{
browser: {
enabled: true,
executablePath: "/usr/bin/google-chrome",
headless: true,
noSandbox: true,
localLaunchTimeoutMs: 60000,
localCdpReadyTimeoutMs: 60000
}
}
Then restart gateway and try:
openclaw gateway restart
openclaw browser --browser-profile openclaw start
openclaw browser --browser-profile openclaw tabs
Docs: https://docs.openclaw.ai/tools/browser-linux-troubleshooting
The support summary is basically right, but I’d add the exact CDP port and corrected log command:
OpenClaw 2026.4.11 detects Chrome at /usr/bin/google-chrome for browser profile "openclaw", using CDP http://127.0.0.1:18820, but the profile remains running=false and tabs reports no live targets. Previous agent/browser attempt failed with: Chrome CDP websocket for profile "openclaw" is not reachable after start. Newer documented flags like browser doctor --deep and browser start --headless are not available in this installed version. I’m collecting browser start output plus openclaw logs --limit 300 after the failed start.
You’re doing the right thing by keeping the research workflow stopped until this is green.
chilly fiber
:~$ openclaw gateway restart
Config warnings:
- plugins.entries.discord: plugin not installed: discord — install the official external plugin with: openclaw plugins install @openclaw/discord
- plugins.entries.telegram: plugin disabled (disabled in config) but config is present
🦞 OpenClaw 2026.5.7 (eeef486) — Open source means you can see exactly how I judge your config.
│
◇ Config warnings ──────────────────────────────────────────────────────────────────────────╮
│ │
│ - plugins.entries.discord: plugin not installed: discord — install the official external │
│ plugin with: openclaw plugins install @openclaw/discord │
│ - plugins.entries.telegram: plugin disabled (disabled in config) but config is present │
│ │
├────────────────────────────────────────────────────────────────────────────────────────────╯
Restarted systemd service: openclaw-gateway.service
This happened the other day, what do i do
oak pulsarBOT
That restart actually succeeded. Those are config warnings, not the browser failure.
What they mean:
plugins.entries.discord: plugin not installed→ your config has a Discord plugin entry, but this OpenClaw install doesn’t currently have the external Discord plugin package installed.plugins.entries.telegram: plugin disabled ... but config is present→ Telegram config exists, but that plugin is disabled.
Neither warning explains the Chrome/CDP browser issue directly.
If you actually use Discord in this OpenClaw setup, install/repair the plugin:
openclaw plugins install @openclaw/discord
openclaw gateway restart
If you do not use Discord/Telegram on this machine, don’t install anything just to silence warnings. Either leave them alone for now, or clean them up later via:
openclaw plugins list --verbose
openclaw plugins doctor
openclaw configure
I’d stay focused on the browser blocker first. Since you’re now on 2026.5.7, rerun the browser checks with the newer flags:
openclaw browser --browser-profile openclaw doctor --deep
openclaw browser --browser-profile openclaw start --headless
openclaw browser --browser-profile openclaw status
openclaw browser --browser-profile openclaw tabs
openclaw logs --limit 300
If start --headless still fails, paste that output plus the browser-related log lines. The plugin warnings can be handled after the research browser is working.
chilly fiber
okay now that i've successfully updated and gateway is running again, what do we do next to continue to fix the browser issue
oak pulsarBOT
Nice — now that you’re on 2026.5.7, do the clean browser triage in this order. Run them one at a time and stop at the first failing command:
openclaw browser --browser-profile openclaw doctor --deep
openclaw browser --browser-profile openclaw start --headless
openclaw browser --browser-profile openclaw tabs
openclaw browser --browser-profile openclaw open https://example.com
openclaw browser --browser-profile openclaw snapshot
How to interpret it:
start --headlessfails with “CDP websocket not reachable” → still a Chrome launch/CDP readiness issue.startsucceeds buttabsfails → control plane is unhealthy.start+tabssucceed butopenfails → browser is up; now it’s likely navigation/SSRF/policy.- all pass → browser is healthy, and you can resume the research workflow.
If start --headless fails, immediately collect:
openclaw browser --browser-profile openclaw status --json
openclaw logs --limit 300
/usr/bin/google-chrome --version
ls -l /usr/bin/google-chrome
Likely fixes if CDP still fails:
- Make headless persistent in config, since this is probably a server/no-display setup:
{
browser: {
enabled: true,
executablePath: "/usr/bin/google-chrome",
headless: true,
noSandbox: true,
localLaunchTimeoutMs: 60000,
localCdpReadyTimeoutMs: 60000
}
}
Then:
openclaw gateway restart
openclaw browser --browser-profile openclaw start
openclaw browser --browser-profile openclaw tabs
-
If logs mention Snap/AppArmor or Chromium wrappers, install/use real Google Chrome or Brave outside Snap.
-
If logs mention profile lock/stale process, try:
openclaw browser --browser-profile openclaw stop
openclaw browser --browser-profile openclaw reset-profile
openclaw browser --browser-profile openclaw start --headless
Only do reset-profile if you’re okay clearing the isolated OpenClaw browser profile state. It won’t touch your personal Chrome profile, but it can remove cookies/session state inside the managed openclaw browser.
chilly fiber
Stopped at the first failure, per your order.
Failed command:
openclaw browser --browser-profile openclaw doctor --deep
Result:
gateway reachable
plugin enabled
profile openclaw found
browser not running → FAIL browser: not running; run openclaw browser start
oak pulsarBOT
That one is an expected “pre-start” failure, not the interesting failure yet. doctor --deep is telling us the config/plugin/profile are fine, but the browser simply isn’t running.
Next run:
openclaw browser --browser-profile openclaw start --headless
Then, regardless of whether it succeeds, run:
openclaw browser --browser-profile openclaw status
openclaw browser --browser-profile openclaw tabs
If start --headless fails, paste that exact error plus:
openclaw logs --limit 300
/usr/bin/google-chrome --version
ls -l /usr/bin/google-chrome
If start --headless succeeds and tabs shows a target, then continue with:
openclaw browser --browser-profile openclaw open https://example.com
openclaw browser --browser-profile openclaw snapshot
So: don’t stop on doctor --deep saying “browser not running” — that’s just the doctor saying “start it first.” The real diagnostic point is whether start --headless can make CDP reachable.
chilly fiber
Got it — pushed past the expected pre-start state.
Results:
openclaw browser --browser-profile openclaw start --headless âś…
openclaw browser --browser-profile openclaw status âś… running on CDP port XXXX
openclaw browser --browser-profile openclaw tabs âś… showed targets (chrome://newtab/ etc.)
openclaw browser --browser-profile openclaw open https://example.com ❌ GatewayClientRequestError: browser navigation blocked by policy
So Chrome launch/CDP readiness looks healthy now. The failure moved to navigation policy/SSRF controls.
I did not run snapshot after the navigation failure.
oak pulsarBOT
Great — that’s progress. Browser launch/CDP is fixed now. We’re in the safer/easier class of failure: navigation policy.
browser navigation blocked by policy usually means your browser.ssrfPolicy is restrictive — often an allowlist is set, so example.com is blocked because it isn’t on the list.
Next, inspect the active browser policy:
openclaw config get browser.ssrfPolicy --json
openclaw config get browser --json
If you see something like only localhost, 127.0.0.1, Mission Control hostnames, or a narrow hostnameAllowlist, that explains it.
For a research workflow, don’t jump straight to broad private-network access. Prefer one of these:
Option A — public web research, normal internet browsing
If you accidentally configured a narrow hostname allowlist, remove/relax just that allowlist:
openclaw config unset browser.ssrfPolicy.hostnameAllowlist
openclaw config unset browser.ssrfPolicy.allowedHostnames
openclaw gateway restart
Then retest:
openclaw browser --browser-profile openclaw open https://example.com
openclaw browser --browser-profile openclaw snapshot
Option B — research only specific domains
Keep strict mode and add the domains you actually need:
openclaw config set browser.ssrfPolicy.hostnameAllowlist '["example.com", "*.example.com", "your-mission-control-domain.com"]' --strict-json
openclaw gateway restart
Option C — you need browser access to private/internal Mission Control URLs
Only if Mission Control is on localhost/LAN/VPN/private IP space, explicitly opt into private-network browser navigation:
openclaw config set browser.ssrfPolicy.dangerouslyAllowPrivateNetwork true --strict-json
openclaw gateway restart
That one is intentionally named scary. Use it only if you trust the environment and need internal/private-network browsing.
After policy is fixed, the proof-of-health sequence is:
openclaw browser --browser-profile openclaw start --headless
openclaw browser --browser-profile openclaw tabs
openclaw browser --browser-profile openclaw open https://example.com
openclaw browser --browser-profile openclaw snapshot
If those pass, the browser blocker is cleared and you can resume the durable research run from the existing queue/checkpoint instead of starting over. 🦞
chilly fiber
Got it cleared now.
Where it got stuck:
the browser policy was locked to YouTube-only hostnames
I removed those allowlists
the gateway restart got deferred for a bit because two active cron/background tasks were still draining, so it looked like it hung
Current state:
openclaw browser --browser-profile openclaw start --headless âś…
openclaw browser --browser-profile openclaw tabs âś…
openclaw browser --browser-profile openclaw open https://example.com âś…
openclaw browser --browser-profile openclaw snapshot âś…
So the browser blocker is cleared and normal public-web browsing is healthy again.
oak pulsarBOT
Beautiful — that’s the right green path. ✅
At this point I’d resume the research workflow, but with a quick preflight first so it can’t regress silently:
openclaw browser --browser-profile openclaw start --headless
openclaw browser --browser-profile openclaw open https://example.com
openclaw browser --browser-profile openclaw snapshot
wc -l research_run_2026-05-12/raw/findings.jsonl
cat research_run_2026-05-12/checkpoint.json
Then continue from the existing batch-queue.txt / checkpoint.json, not from scratch.
I’d also add a note to reports/run_summary.md like:
Browser blocker resolved: browser.ssrfPolicy was restricted to YouTube-only hostnames. Removed the allowlists, restarted gateway after active cron/background tasks drained, and verified start/tabs/open/snapshot against https://example.com.
For the next run, keep the “after every company” contract strict:
- append one JSONL record
- update checkpoint
- verify JSONL parses
- verify row count increased or status explains why not
- stop on browser/policy/login/captcha blockers
That should prevent a repeat of the header-only CSV failure.
chilly fiber
please give me the full updated prompt to continue my workflow
oak pulsarBOT
Yep — paste this as the continuation prompt. It assumes the existing run folder is research_run_2026-05-12 and that browser health is now verified.
Continue the existing durable company research workflow from the run folder:
research_run_2026-05-12
Important context:
- This is a continuation, not a new run.
- The previous blocker was browser navigation policy: browser.ssrfPolicy was restricted to YouTube-only hostnames.
- That blocker has been resolved.
- Browser health has been verified:
- openclaw browser --browser-profile openclaw start --headless âś…
- openclaw browser --browser-profile openclaw tabs âś…
- openclaw browser --browser-profile openclaw open https://example.com âś…
- openclaw browser --browser-profile openclaw snapshot âś…
Existing artifacts:
- research_run_2026-05-12/raw/findings.jsonl
- research_run_2026-05-12/checkpoint.json
- research_run_2026-05-12/reports/run_summary.md
- research_run_2026-05-12/batch-queue.txt
Your job:
Resume from the existing checkpoint and batch queue. Do not restart from scratch unless the checkpoint/queue are unreadable, and if that happens, stop and report the exact blocker.
Before researching:
1. Inspect the current run folder.
2. Read checkpoint.json.
3. Read batch-queue.txt.
4. Count existing JSONL rows in raw/findings.jsonl.
5. Run a browser preflight:
- confirm the browser starts or is already running
- confirm tabs are available
- confirm snapshot works on a simple public page if needed
6. Append a short note to reports/run_summary.md that the browser blocker was resolved and the run is resuming.
Output rules:
- raw/findings.jsonl is the source of truth.
- Treat raw/findings.jsonl as append-only.
- Never overwrite raw/findings.jsonl.
- Never overwrite checkpoint.json without preserving the current state in the update.
- Never overwrite any existing CSV.
- Do not create or overwrite a final CSV until JSONL validation passes.
- If producing CSV later, generate it from raw/findings.jsonl only, and write a new filename such as:
- research_run_2026-05-12/derived/findings.v2.csv
- research_run_2026-05-12/derived/findings.refined.v2.csv
Per-company contract:
For each company attempted, write exactly one JSON object line to raw/findings.jsonl before moving to the next company.
Each JSONL object must include at minimum:
- run_id
- company_id
- company_name
- status: one of researched, partial, no_useful_data_found, skipped, blocked, error
- retrieved_at ISO timestamp
- source_urls: array of URLs actually used
- findings: object or array with the discovered facts
- confidence: low, medium, high
- errors: array
- notes: string
If no useful data is found, still write a row with status no_useful_data_found and explain what was checked.
If blocked by login, captcha, rate limit, navigation policy, browser failure, missing company input, or inaccessible source, stop and report the blocker. Do not pretend progress is happening.
Checkpointing rules:
After every company:
1. Append the JSONL row.
2. Validate the new JSONL line parses as JSON.
3. Update checkpoint.json with:
- last_completed_company_id
- last_completed_company_name
- completed_count
- partial_count
- skipped_count
- blocked_count
- error_count
- output_rows
- updated_at
- current_batch_position
4. Confirm raw/findings.jsonl row count increased by 1, unless the company was already present and explicitly skipped as duplicate.
5. Append a brief progress note to reports/run_summary.md.
Batching:
- Process only the next small batch from batch-queue.txt, preferably 5 companies unless the queue specifies otherwise.
- Do not spend the entire run on one company.
- If a company takes too long or requires deep manual browsing, write a partial row and move on, unless the blocker affects the whole run.
Browser rules:
- Use the browser for public-web research where needed.
- Keep tabs controlled and avoid opening excessive tabs.
- If browser refs go stale, resnapshot once and retry.
- If the browser becomes unhealthy, stop and report the browser blocker with the last successful browser command.
- Do not relax browser SSRF/private-network policy unless explicitly instructed by the user.
Validation before final response:
Before reporting completion of this continuation batch, run/check:
- JSONL row count before vs after
- JSONL parse validation
- checkpoint.json updated_at changed
- source URLs are present for researched/partial rows
- no CSV was overwritten
Final response must include proof of work:
- companies attempted
- companies completed/partial/skipped/blocked/error
- JSONL rows before
- JSONL rows after
- files updated
- last completed company
- any blockers
- whether it is safe to run the next batch
Do not summarize vaguely. If meaningful work did not happen, say that clearly and explain why.
Good catch — this is exactly why the checkpointing pattern is working. Don’t advance the queue yet.
Next step is to separate “browser is globally unhealthy” from “that Owensboro Chamber page hung navigation.” Run this recovery sequence:
openclaw browser --browser-profile openclaw status
openclaw browser --browser-profile openclaw tabs
openclaw browser --browser-profile openclaw open https://example.com
openclaw browser --browser-profile openclaw snapshot
If those pass, the browser is healthy again and the blocker is probably page-specific. In that case, resume with a modified instruction:
Treat the Owensboro Chamber URL as a timed-out source for now. Do not retry it more than once. Use Metalsa corporate sources and other public sources. If enough evidence exists, write a partial row with source_urls including the timed-out chamber URL and errors noting the timeout. Continue only after appending one durable JSONL row and updating checkpoint.json.
If example.com or snapshot fails too, then do a clean browser restart:
openclaw browser --browser-profile openclaw stop
openclaw browser --browser-profile openclaw start --headless
openclaw browser --browser-profile openclaw tabs
openclaw browser --browser-profile openclaw open https://example.com
openclaw browser --browser-profile openclaw snapshot
Only restart the whole gateway if browser stop/start does not recover it:
openclaw gateway restart
openclaw browser --browser-profile openclaw start --headless
openclaw browser --browser-profile openclaw open https://example.com
openclaw browser --browser-profile openclaw snapshot
For the workflow itself, I’d update the continuation prompt with this rule:
If one source URL times out, do not mark the whole run blocked unless browser health also fails on a known-good page like https://example.com. Record that source timeout in the company row’s errors[] and continue with other sources. Stop only if the browser fails the known-good health check.
That avoids letting one slow Chamber page stall the entire batch while still preserving honest evidence.
chilly fiber
Recovery check passed.
What I verified:
browser status: healthy
tabs: readable
opened test website
snapshot test: passed
So this was page-specific, not a global browser failure.
I resumed with the timeout rule and wrote one durable row:
appended COMPANY_NAME as status: partial
included source URL in source_urls
recorded the earlier timeout in errors[]
updated checkpoint.json
Durable state now:
raw/findings.jsonl: 3 rows
checkpoint.json: progress updated
batch position: 3 / 4
counts: completed=2, partial_count=1, failed=0
Next company remaining:
COMPANY_NAME
I kept the evidence honest and did not treat the whole run as blocked once the browser health check passed.
oak pulsarBOT
Perfect — that’s the reliability pattern working as intended. The key distinction was right: one bad URL/source timeout ≠global browser failure.
For the last remaining company, continue with the same guardrail:
Continue the existing run from checkpoint.json. Process only the next remaining company.
Before starting, record JSONL row count and browser health. If one source URL times out, verify browser health against a known-good page before deciding whether it is page-specific or global. If page-specific, write a researched or partial row with the timeout recorded in errors[]. After the company, validate JSONL parses, confirm row count increased from 3 to 4, update checkpoint.json, and append a progress note to run_summary.md.
Do not generate or overwrite CSV yet unless explicitly asked. raw/findings.jsonl remains the source of truth.
Final response must include:
- company attempted
- status written
- JSONL rows before/after
- checkpoint batch position
- files updated
- blockers/errors
- whether the batch is complete
Once row count hits 4 and validation passes, then it’s safe to do the derived CSV step from JSONL.
chilly fiber
now id like to keep the task running, how do we accomplish that:
Done — finished the last remaining company and kept raw/findings.jsonl as source of truth.
company attempted: COMPANY_NAME
status written: partial
JSONL rows before/after: 3 -> 4
checkpoint batch position: 4 / 4
files updated:
project_folder/raw/findings.jsonl
project_folder/checkpoint.json
project_folder/reports/run_summary.md
blockers/errors:
none on this company; no source timeout encountered during this pass
whether the batch is complete:
yes
Validation:
browser health recorded before start: healthy
JSONL parse validation: passed
checkpoint now shows lastCompletedCompanyId = [REDACTED]
counts now show completed=2, partial_count=2, failed=0
I did not generate or overwrite any CSV.
oak pulsarBOT
Great — batch 1 is complete and durable. To “keep it running,” don’t extend the same batch forever. Start the next small batch from the same run folder/checkpoint.
Use this pattern:
- Add the next companies to
batch-queue.txt - Keep
raw/findings.jsonlappend-only - Run the next batch of 5-ish companies
- Validate after every company
- Stop only on a real global blocker
Paste this as the next continuation prompt:
Continue the durable company research workflow from the existing run folder.
Do not start over.
Do not overwrite raw/findings.jsonl.
Do not generate or overwrite CSV yet.
Use raw/findings.jsonl as the source of truth.
Current state:
- Previous batch is complete.
- raw/findings.jsonl has 4 valid rows.
- checkpoint.json is updated through batch position 4 / 4.
- counts are completed=2, partial_count=2, failed=0.
- Browser health was verified.
Your job:
1. Inspect checkpoint.json and raw/findings.jsonl.
2. Confirm JSONL parses before continuing.
3. Load the next companies from batch-queue.txt, or if the queue is exhausted, create/ask for the next small batch before doing research.
4. Process only the next small batch, max 5 companies.
5. Before starting browser research, verify browser health with start/tabs/open/snapshot on a known-good page.
Per company:
- Write exactly one append-only JSONL row.
- Status must be researched, partial, no_useful_data_found, skipped, blocked, or error.
- Include source_urls, retrieved_at, findings, confidence, errors, and notes.
- If one source URL times out, verify browser health on a known-good page. If browser is healthy, treat it as page-specific and record it in errors[] instead of blocking the whole run.
- After each company, update checkpoint.json and validate row count increased by 1.
Final response for this batch must include:
- companies attempted
- statuses written
- JSONL rows before/after
- checkpoint position
- files updated
- blockers/errors
- whether it is safe to run the next batch
If you want it truly unattended for 30–45 minutes, the safer version is still repeated small batches, not one giant run. Ask the agent to process “up to 5 companies or 10 minutes, whichever comes first,” then report proof and wait for the next continuation. That gives you durable checkpoints without drifting into fake progress.
chilly fiber
APP
— 2:20 PM
Checked the current run folder and stopped at the real blocker instead of making up a next batch.
companies attempted: none
statuses written: none
JSONL rows before/after: 4 -> 4
checkpoint position: 4 / 4
files updated:
memory/tasks.md
blockers/errors:
batch queue is exhausted
the run folder only contains:
- batch queue
- checkpoint file
- raw findings
- summary report
there is no next company list in this run folder to safely pull another 1–5 from
whether it is safe to run the next batch:
yes, once the next small batch or source list is provided
What I verified before stopping:
checkpoint file is still healthy at 4/4
raw findings parse cleanly with 4 valid rows
browser health recheck passed with start/tabs/open/snapshot on test website
So the workflow is ready to continue — it just needs the next batch input.
oak pulsarBOT
That’s a good stop. The agent did the right thing: exhausted queue means no invented next batch.
To keep it running, give it one explicit source of new companies. Best options:
- Append the next 5–10 company IDs/names to
batch-queue.txt, then tell it to resume. - Export the next slice from Mission Control into a file like
input/companies-batch-002.csv, then point the agent at that file. - If Mission Control is browser-accessible, give it the exact page/filter/query to pull the next batch from — but still require it to write those selected companies into
batch-queue.txtbefore researching them.
Use this prompt:
Continue the existing durable research run.
The previous batch is complete and batch-queue.txt is exhausted.
Do not invent companies.
Do not start research until a new explicit batch source is available.
Next batch source:
[PASTE 5–10 company names/IDs here, OR path to exported CSV, OR exact Mission Control page/filter]
Steps:
1. Add the next 1–5 companies from the provided source to batch-queue.txt.
2. Record in reports/run_summary.md where the new batch came from.
3. Validate raw/findings.jsonl still parses and record starting row count.
4. Verify browser health with start/tabs/open/snapshot on a known-good page.
5. Research only this next small batch.
6. After each company, append exactly one JSONL row, update checkpoint.json, validate row count increased, and record source URLs/errors honestly.
7. Stop if the queue is exhausted again or a global blocker occurs.
Do not overwrite raw/findings.jsonl.
Do not generate CSV unless explicitly asked.
Final response must include rows before/after, companies attempted, statuses, checkpoint position, files updated, and blockers.
My recommendation: export/paste the next 5 companies first. Don’t ask it to browse Mission Control and research in the same step until the batching loop has proven stable.
chilly fiber
seems that we are still struggling to work somewhat continuously.
Please help me reprompt
Pulled the next local batch from [REDACTED] and loaded it into the existing run queue.
Next 5 in [REDACTED]/batch-queue.txt:
- [COMPANY REDACTED]
- [COMPANY REDACTED]
- [COMPANY REDACTED]
- [COMPANY REDACTED]
- [COMPANY REDACTED]
I deduped against the existing companies already in raw/findings.jsonl and left the JSONL untouched.
User — 2:55 PM
so does that mean you are working on the 5 now?
APP — 2:55 PM
Yeah — I’ve loaded the next 5 into the durable run.
What’s done:
- kept the existing run intact
- verified the current JSONL/checkpoint state
- pulled the next 5 from local files
- queued them in order
What’s not done yet:
- I haven’t finished researching/appending those 5 rows yet
I can start processing them now from the existing run folder.
oak pulsarBOT
Yep — the issue is the prompt let the agent treat “queue loaded” as a valid stopping point. Tighten it so loading the queue is only step 0, and the run is not complete until rows are appended or a real blocker occurs.
Paste this:
Continue the existing durable research run from the current run folder.
Important: loading companies into batch-queue.txt is NOT completion. After loading or confirming the next 5 are queued, immediately begin researching them in order. Do not stop after queue preparation unless there is a real blocker that prevents research.
Current state:
- The next 5 companies have already been loaded into batch-queue.txt.
- raw/findings.jsonl must remain append-only.
- Do not overwrite existing files.
- Do not generate CSV yet.
Your task now:
1. Read checkpoint.json and raw/findings.jsonl.
2. Record starting JSONL row count.
3. Verify browser health with start/tabs/open/snapshot on a known-good page.
4. Process the queued companies one by one, up to 5 companies or 20 minutes, whichever comes first.
For each company:
- Research using browser/public web sources.
- Append exactly one JSONL row to raw/findings.jsonl.
- Use status: researched, partial, no_useful_data_found, skipped, blocked, or error.
- Include source_urls, retrieved_at, findings, confidence, errors, and notes.
- Update checkpoint.json immediately after the row is written.
- Validate JSONL still parses.
- Confirm row count increased by 1 unless the company was a duplicate/skipped.
Do not ask whether to start. Start processing the queued companies now.
Only stop early if:
- browser health fails on a known-good page,
- login/captcha/rate-limit blocks the run,
- the queue is unreadable,
- required input is missing,
- or a file validation/checkpoint error occurs.
If one source URL times out but browser health still passes on a known-good page, treat it as page-specific: record the timeout in errors[] and continue with other sources.
Final response must prove work happened:
- companies attempted
- statuses written
- JSONL rows before/after
- checkpoint position before/after
- files updated
- blockers/errors
- whether the queue still has remaining companies
- whether it is safe to continue the next batch
If no rows were appended, explicitly say why and identify the blocker. Do not present queue loading as research progress.
If you want more continuity, change “up to 5 companies or 20 minutes” to “up to 5 companies or 45 minutes,” but I’d keep the timebox smaller until it’s reliably appending rows every company.
chilly fiber
this is working now, please adjust and make a minor prompt to continue working in batches until i explicitly say stop. I want it to keep going to next batch after returning me the data from prior
oak pulsarBOT
Yep — use this as a small add-on to the working prompt:
Continuation behavior:
Keep working in batches until I explicitly say STOP.
After each batch:
1. Return a concise batch report with:
- companies attempted
- statuses written
- JSONL rows before/after
- checkpoint position
- files updated
- blockers/errors
- whether validation passed
2. Then immediately continue to the next batch without asking for permission, as long as:
- a next batch source is available,
- browser health passes,
- JSONL/checkpoint validation passes,
- and no global blocker occurs.
Do not treat “batch report sent” as the end of the task. The report is a checkpoint, not completion.
If batch-queue.txt is exhausted:
- Pull the next 1–5 companies from the configured local source/list.
- Deduplicate against raw/findings.jsonl.
- Append/load those companies into batch-queue.txt.
- Record the source of the new batch in reports/run_summary.md.
- Continue researching them immediately.
Only stop if:
- I explicitly say STOP,
- no next company source exists,
- browser health fails on a known-good page,
- login/captcha/rate-limit blocks the run,
- JSONL/checkpoint validation fails,
- or another real blocker prevents safe progress.
If you stop, say exactly why. If you continue, keep appending one durable JSONL row per company and updating checkpoint.json after every company.
One small recommendation: keep the batch size at 5. Continuous is fine, but the durability still comes from frequent small checkpoints, not one giant 45-minute blob.
oak pulsarBOT
This is technically “working,” but the quality signal is bad: two full batches of no_useful_data_found means the agent is continuing mechanically without enough retrieval power.
I’d reprompt it to keep running, but not keep writing empty rows as success. Use this add-on:
Quality correction for continuous batches:
Do not keep producing full batches of no_useful_data_found just because search/fetch is weak.
For the next staged batch — 3M, A O Smith, Amcor, Atlantic Track & Turnout, BAE Systems — switch to browser-first research.
Rules:
1. Use browser navigation/search pages and official/company pages before deciding no_useful_data_found.
2. For each company, try at least:
- official company website or newsroom
- company name + facility/location/manufacturing/supplier search via browser
- one credible secondary source if available
3. If web_search is unavailable, record that as a tool limitation, but do not treat it as proof that no data exists.
4. Only write no_useful_data_found after browser-first checks fail to produce durable company-specific evidence.
5. If the same retrieval limitation causes 3 companies in a row to become no_useful_data_found, STOP and report a retrieval-quality blocker instead of burning through the list.
6. Continue batches until I explicitly say STOP, but only while rows contain meaningful researched/partial findings or a clearly justified no_useful_data_found status.
After each batch report, immediately continue to the next batch only if validation passed and retrieval quality was acceptable. If quality is unacceptable, stop and explain what source/tool access is missing.
Also, if you want proper web search instead of browser-only scraping, configure a provider:
openclaw configure --section web
For this workflow, I’d recommend Brave, Tavily, Exa, Perplexity, or SearXNG — anything that gives reliable snippets/results. Browser is healthy now, but search discovery is still the weak link.
chilly fiber
we need to correct
Batch report:
companies attempted: [COMPANY REDACTED]; [COMPANY REDACTED]; [COMPANY REDACTED]; [COMPANY REDACTED]; [COMPANY REDACTED]
statuses written: researched; researched; researched; partial; no_useful_data_found
JSONL rows before/after: [REDACTED]
checkpoint position: 5/5 (last completed: [COMPANY REDACTED])
files updated: [PATH REDACTED]
blockers/errors: browser-first pass experienced search/provider limitations; web search provider not configured
validation passed: yes
Follow-up:
repaired checkpoint state after detecting count drift
continuing with stricter browser-first quality rules
Batch report:
companies attempted: [COMPANY REDACTED]; [COMPANY REDACTED]; [COMPANY REDACTED]; [COMPANY REDACTED]; [COMPANY REDACTED]
statuses written: researched; researched; partial; researched; researched
JSONL rows before/after: [REDACTED]
checkpoint position: 5/5 (last completed: [COMPANY REDACTED])
files updated: [PATH REDACTED]
blockers/errors: web_search unavailable because provider base URL is not configured
validation passed: yes
Completed the next batch with corrected quality rules.
What I accomplished
- Appended 5 durable rows to raw findings
- Updated checkpoint state to match the new run state
- Updated batch summary report
- Validated file state after writes
Key details
- findings JSONL parses cleanly
- checkpoint now matches output rows
- status counts validated successfully
Notable findings / caveats
- web_search remains unavailable due to missing provider configuration
- browser-first and fallback retrieval methods were used
- one company remained partial because durable facility/location evidence was weak
- no retrieval-quality blocker was triggered
Batch report:
companies attempted: [COMPANY REDACTED]; [COMPANY REDACTED]; [COMPANY REDACTED]; [COMPANY REDACTED]; [COMPANY REDACTED]
statuses written: partial; partial; researched; researched; partial
JSONL rows before/after: [REDACTED]
checkpoint position: 5/5 (last completed: [COMPANY REDACTED])
files updated: [PATH REDACTED]
blockers/errors:
- web_search unavailable
- browser search triggering unusual-traffic/reCAPTCHA
- secondary-source access restrictions encountered
validation passed: yes
Completed the next batch and refilled the queue.
What I did
- Added 5 durable rows to findings
- Updated checkpoint state
- Updated summary report
- Refilled batch queue from local source
Validation
- findings JSONL parses cleanly
- checkpoint matches row count
Relevant details
- browser-first discovery attempted
- official company pages prioritized
- partial status used only where durable evidence existed but details were incomplete
Next queue loaded
- [QUEUE REDACTED]
Potential follow-up note
I stopped here because retrieval quality is still acceptable, but search blocking is becoming a material constraint.
Batch report:
companies attempted: [COMPANY REDACTED]; [COMPANY REDACTED]; [COMPANY REDACTED]; [COMPANY REDACTED]; [COMPANY REDACTED]
statuses written: researched; researched; partial; researched; researched
JSONL rows before/after: [REDACTED]
checkpoint position: 5/5 (last completed: [COMPANY REDACTED])
files updated: [PATH REDACTED]
blockers/errors: search constraints remain active, but no retrieval-quality blocker triggered
validation passed: yes
Completed the next 5-company batch.
What I accomplished
- Appended 5 JSONL rows
- Updated checkpoint state through each company
- Validated findings parse successfully
- Updated summary report
- Refilled queue from local source
Relevant details
- search constraints remain active
- one company remained partial due to lack of durable facility-specific documentation
Next queue now loaded
- [QUEUE REDACTED]
oak pulsarBOT
Yep — I’d correct course here. The agent is still “successfully continuing,” but the reports show two reliability red flags:
- checkpoint count drift happened once and was “repaired” mid-run
- retrieval quality is degrading: no web_search, browser search reCAPTCHA/unusual traffic, secondary-source restrictions
So the next prompt should pause forward research, audit the last written rows, and only continue if the data is actually usable.
Paste this:
Pause continuous forward research and run a quality/audit pass before processing any more companies.
Do not append new company research rows during this audit.
Do not refill the queue.
Do not generate CSV.
Do not overwrite raw/findings.jsonl.
Reason for audit:
Recent batches continued despite search/provider limitations, browser search reCAPTCHA/unusual traffic, secondary-source restrictions, and a checkpoint count drift repair. I need to verify that the latest rows are durable and useful before continuing.
Audit scope:
Review the most recent 20 rows in raw/findings.jsonl, or all rows if fewer than 20.
For each audited row, check:
1. JSON parses cleanly.
2. Required fields are present.
3. status is valid.
4. source_urls are present for researched/partial rows.
5. findings contain company-specific evidence, not generic filler.
6. errors[] honestly records unavailable web_search, captcha, timeout, access restriction, or thin evidence.
7. partial/no_useful_data_found statuses are justified.
8. duplicate companies were not appended.
Also validate checkpoint.json:
- output row count matches raw/findings.jsonl
- status counts match rows
- last completed company matches the last appended row
- batch position is internally consistent
Write an audit report to:
research_run_2026-05-12/reports/quality_audit.md
The audit report must include:
- total rows audited
- number accepted as usable
- number needing repair
- number that should be downgraded to partial/no_useful_data_found
- number that look invalid or duplicate
- checkpoint validation result
- specific row numbers/company names needing attention
- recommendation: continue, repair first, or stop and configure search provider
If repairs are needed:
- Do not silently rewrite history.
- Create a proposed repair plan in the audit report.
- If a row must be corrected, append a correction row or write a separate repair file; do not destructively edit raw/findings.jsonl unless explicitly approved.
Only resume continuous batches if:
- audit passes,
- checkpoint is consistent,
- at least most researched/partial rows have durable source URLs,
- and retrieval quality is still acceptable.
If web_search unavailable + browser reCAPTCHA/access restrictions are materially limiting research, stop and recommend configuring a web_search provider before continuing.
Final response should be only the audit summary and recommendation. Do not process the next batch yet.
After that, if the audit says “continue,” use a stricter continuation rule:
Continue only while retrieval quality remains acceptable. If 2 companies in a batch are blocked by search/captcha/access restrictions, stop the batch and report a retrieval-quality blocker instead of filling the JSONL with weak rows.
Yep — I’ve got an audit result.
Quick version:
I paused forward research
Audited the latest 20 rows
Most are usable, but 4 need attention
Checkpoint row count matches, but checkpoint metadata is still inconsistent
Search quality is being limited by missing web search config + Google/reCAPTCHA/access blocks
My recommendation before continuing:
repair the flagged rows/checkpoint metadata
configure a real web search provider first
If you want, I can post the exact audit findings next.
oak pulsarBOT
Yes — post the exact audit findings before doing any repair or more research.
Ask it for this format:
Post the exact audit findings now.
Do not repair anything yet.
Do not append research rows.
Do not refill or process the queue.
Do not generate CSV.
Include:
1. The 4 flagged rows: row number, company name/id, current status, problem found.
2. For each flagged row: recommended action — accept, append correction row, downgrade status, mark needs_research, or remove from derived outputs only.
3. The checkpoint inconsistencies: exact fields that disagree and expected values.
4. Search/retrieval blockers observed: missing web_search config, reCAPTCHA, access restrictions, timeouts, etc.
5. Whether raw/findings.jsonl is still parse-valid and append-safe.
6. Whether continuing without configuring web_search is likely to degrade data quality.
End with a clear recommendation:
- repair now,
- configure search provider first,
- or stop the run.
My bias: don’t let it “repair” silently. Get the audit details first, then approve a specific non-destructive repair plan.
chilly fiber
Exact audit findings:
- Flagged rows
Row [REDACTED] — [COMPANY REDACTED]
Current status: partial
Problem found: usable as a facility-scoped lead, but materially overlaps with a broader company row and does not yet have durable facility-specific evidence beyond the queue label.
Recommended action: append correction row
Row [REDACTED] — [COMPANY REDACTED]
Current status: partial
Problem found: same overlap pattern; behaves more like a facility-scoped alias than a fully validated standalone company row.
Recommended action: append correction row
Row [REDACTED] — [COMPANY REDACTED]
Current status: researched
Problem found: semantic duplicate of an earlier row.
Recommended action: remove from derived outputs only
Row [REDACTED] — [ENTITY REDACTED]
Current status: no_useful_data_found
Problem found: behaves more like a prospect cluster / targeting concept than a single durable company entity.
Recommended action: remove from derived outputs only
- Checkpoint inconsistencies
completed_count
Current: [REDACTED]
Expected: [REDACTED]
Why: expected value should equal researched + partial totals
completed
Current: [REDACTED]
Expected: [REDACTED]
batch_complete
Current: false
Expected: true
Why: current batch position equals current batch total
outputRows / output_rows
Result: values match correctly
lastCompletedCompanyName / lastCompletedCompanyId
Result: matches last appended row correctly
- Search / retrieval blockers observed
- web_search unavailable because search provider base URL is not configured
- browser search pages triggering unusual-traffic / reCAPTCHA blocks
- Cloudflare / challenge blocking on some secondary sources
- access denied on some pressroom / manufacturing pages
- hostname resolution / DNS failures in some cases
- occasional 404 / thin-evidence cases
- File safety
- findings JSONL parse-valid: yes
- append-safe: yes, structurally
- quality-safe for continued research under current discovery limits: questionable
- Likely quality impact if we continue without web_search
Yes, continuing without configuring search is likely to degrade data quality.
Reason:
Discovery is already being constrained by:
- missing configured search
- browser reCAPTCHA blocking
- restricted secondary sources
This increases risk of:
- alias rows
- duplicate entities
- thin facility evidence
- unjustified no-data outcomes
Recommendation:
configure the search provider first.
oak pulsarBOT
Yep — I agree with the audit recommendation: stop forward research until search is configured. Browser-only discovery is already hitting reCAPTCHA/access blocks, so continuing will create more duplicate/thin rows.
Do this next:
openclaw configure --section web
Pick/configure a real search provider, then verify the agent has web_search available before resuming. Docs: https://docs.openclaw.ai/tools/web
After search is configured, use this prompt:
Do not continue forward research yet.
First repair the audited state non-destructively.
Rules:
- Do not overwrite raw/findings.jsonl.
- Do not delete rows from raw/findings.jsonl.
- Do not generate CSV yet.
- Corrections must be append-only or written to a derived exclusion/repair file.
Tasks:
1. Verify web_search is now available with a simple test query.
2. Append correction rows for the 2 facility-scoped partial rows, marking them as facility_alias_or_overlap and explaining the broader company overlap.
3. Create or update a derived exclusion file, e.g. reports/derived_exclusions.json, for:
- the semantic duplicate row
- the prospect-cluster / non-company entity row
These should be excluded from derived CSV/report outputs, but left intact in raw/findings.jsonl.
4. Repair checkpoint metadata so:
- completed_count / completed match the intended researched + partial totals, or document the exact definition used
- batch_complete is true when current batch position equals batch total
- outputRows/output_rows still match raw JSONL row count
- lastCompletedCompanyName/Id still match the last appended row or correction row, whichever is now latest
5. Validate:
- raw/findings.jsonl parses
- checkpoint row count matches JSONL line count
- derived exclusions parse
- correction rows parse
6. Write a repair summary to reports/run_summary.md.
Final response must include:
- correction rows appended count
- rows excluded from derived outputs count
- checkpoint fields repaired
- validation result
- whether web_search test passed
- whether it is safe to resume forward research
Only after this repair passes, resume batches with web_search + browser-first evidence.
Important: don’t let it “fix” this by editing old raw rows. Raw JSONL should stay an audit log; derived outputs can exclude or supersede rows.
chilly fiber
@srvXXXXXXX:~$ openclaw configure --section web
Config warnings:
- plugins.entries.telegram: plugin disabled (disabled in config) but config is present
🦞 OpenClaw 2026.5.7 (eeef486) — iMessage green bubble energy, but for everyone.
┌ OpenClaw configure
│
◇ Existing config detected ─────────╮
│ │
│ workspace: ~/.openclaw/workspace │
│ model: openai-codex/gpt-5.4 │
│ gateway.mode: local │
│ gateway.port: 18789 │
│ gateway.bind: loopback │
│ │
├────────────────────────────────────╯
│
â—‡ Where will the Gateway run?
│ Local (this machine)
│
◇ Web search ────────────────────────────────────────────────────────────────────────────╮
│ │
│ Web search lets your agent look things up online using the web_search tool. │
│ Choose a managed provider now, and Codex-capable models can also use native Codex web │
│ search. │
│ Docs: https://docs.openclaw.ai/tools/web │
│ │
├─────────────────────────────────────────────────────────────────────────────────────────╯
│
â—‡ Enable web_search?
│ Yes
│
◇ Codex native search ───────────────────────────────────────────────────────────╮
│ │
│ Codex-capable models can optionally use native Codex web search. │
│ Managed web_search still controls non-Codex models. │
│ If no managed provider is configured, non-Codex models still rely on provider │
│ auto-detect and may have no search available. │
│ Recommended mode: cached. │
│ │
├─────────────────────────────────────────────────────────────────────────────────╯
│
â—‡ Enable native Codex web search for Codex-capable models?
│ Yes
│
â—‡ Codex native web search mode
│ cached (recommended)
│
â—‡ Configure or change a managed web search provider now?
│ Yes
│
◇ Web search ─────────────────────────────────────────────────────────────────╮
│ │
│ Web search lets your agent look things up online. │
│ Choose a provider. Some providers need an API key, and some work key-free. │
│ Docs: https://docs.openclaw.ai/tools/web │
│ │
├──────────────────────────────────────────────────────────────────────────────╯
│
â—‡ Search provider
│ Brave Search
│
â—‡ Brave Search API key
│ [REDACTED]
│
â—‡ Installed Brave plugin
│
â—‡ Enable web_fetch (keyless HTTP fetch)?
│ Yes
Config warnings:
- plugins.entries.telegram: plugin disabled (disabled in config) but config is present
Config overwrite: /home/jake/.openclaw/openclaw.json
(sha256 [REDACTED] -> [REDACTED],
backup=/home/jake/.openclaw/openclaw.json.bak)
Config warnings:
- plugins.entries.telegram: plugin disabled (disabled in config) but config is present
Updated ~/.openclaw/openclaw.json
│
◇ Control UI ────────────────────────────────────╮
│ │
│ Web UI: http://127.0.0.1:18789/ │
│ Gateway WS: ws://127.0.0.1:18789 │
│ Gateway: reachable │
│ Docs: https://docs.openclaw.ai/web/control-ui │
│ │
├─────────────────────────────────────────────────╯
│
â”” Configure complete.
@srvXXXXXXX:~$
ive enabled codex native search AND brave API now.
would I be technically all set with codex search? Or is it really practical to have the brave API as well
oak pulsarBOT
Nice — that looks like the right fix. Brave Search is configured, native Codex search is enabled, and web_fetch is enabled too.
Next, verify the tool actually works before resuming research:
openclaw gateway restart
Then ask the agent this:
Before repairing or continuing research, verify web_search is now available.
Run a simple web_search test query for: "OpenClaw browser tool".
If web_search fails, stop and report the exact error.
If web_search succeeds, continue with the non-destructive repair plan from the audit:
1. Do not overwrite raw/findings.jsonl.
2. Do not delete old rows.
3. Append correction rows for the 2 facility-scoped overlap rows.
4. Create/update reports/derived_exclusions.json for the semantic duplicate row and the prospect-cluster/non-company row.
5. Repair checkpoint metadata so counts, batch_complete, output row count, and last completed fields are internally consistent.
6. Validate JSONL, checkpoint, and derived exclusions.
7. Update reports/run_summary.md.
Final response must include:
- whether web_search test passed
- correction rows appended count
- derived exclusions count
- checkpoint fields repaired
- validation result
- whether it is safe to resume forward research
Do not process the next research batch until repair passes.
The Telegram warning is still just cleanup noise unless you expect Telegram to be active.
Technically: if the active research agent is always using a Codex-capable model, Codex native search may be enough.
Practically: yes, keep Brave API too.
Why Brave is useful here:
- Codex native search only helps Codex-capable model runs.
- Brave powers the managed
web_searchtool, so it works across non-Codex models/agents and is more explicit in tool logs. - Your workflow already failed on “web_search unavailable,” so Brave fixes that exact class of blocker.
- For durable research/auditing, managed
web_searchresults are easier to prove: query → results/snippets/URLs → JSONL source_urls. - It gives you a fallback if native search is unavailable, rate-limited, or not used by the model the way you expect.
So for this workflow I’d keep both:
- Codex native search: nice model-native discovery path.
- Brave API: reliable explicit
web_searchtool path.
Before resuming, run a quick agent-side test: “use web_search for OpenClaw browser tool and report the top result URL.” If that passes, you’re in a much better place than browser-only research.
chilly fiber
Done.
web_search test: passed
command: openclaw capability web search --query "[REDACTED QUERY]" --json
provider: [REDACTED]
correction rows appended: 2
[REDACTED]
[REDACTED]
derived exclusions count: 2
[REDACTED] → semantic duplicate of existing row
[REDACTED] → prospect cluster / non-company entity
checkpoint fields repaired:
completed = 48
completed_count = 48
partial_count = 15
no_useful_data_found_count = 15
outputRows = 63
output_rows = 63
batch_complete = true
current_batch_position = 5
current_batch_total = 5
last completed fields now point to the latest appended correction row
validation result: pass
JSONL parse-valid: yes
checkpoint internally consistent: yes
derived exclusions valid: yes
safe to resume forward research: yes
I did not process the next batch
oak pulsarBOT
Great — that’s the checkpoint we wanted. You’re safe to resume now.
Use this continuation prompt:
Resume forward research from the repaired durable state.
Current verified state:
- web_search test passed
- correction rows appended: 2
- derived exclusions valid: 2 rows excluded from derived outputs
- raw/findings.jsonl parses cleanly
- checkpoint is internally consistent
- output_rows = 63
- batch_complete = true
- safe_to_resume = true
Do not start over.
Do not overwrite raw/findings.jsonl.
Do not delete old rows.
Do not generate CSV yet.
Respect reports/derived_exclusions.json when reasoning about duplicates and derived outputs.
Next steps:
1. Load the next 1–5 companies from the configured local source into batch-queue.txt.
2. Deduplicate against:
- raw/findings.jsonl
- correction rows
- reports/derived_exclusions.json
3. Record the new batch source in reports/run_summary.md.
4. Verify browser health with start/tabs/open/snapshot.
5. Use web_search + browser-first evidence for each company.
6. Process the queued companies one by one.
For each company:
- Append exactly one JSONL row.
- Include source_urls from web_search/browser evidence.
- Use status: researched, partial, no_useful_data_found, skipped, blocked, or error.
- Update checkpoint.json immediately after writing the row.
- Validate JSONL parses and row count increased.
Quality rules:
- Do not use no_useful_data_found unless web_search + browser checks genuinely fail to find durable company-specific evidence.
- If 2 companies in the same batch are blocked by captcha/access/search limitations, stop and report a retrieval-quality blocker.
- If one source fails but other durable sources exist, continue and record the failed source in errors[].
After the batch:
- Return a concise batch report.
- Then continue to the next batch automatically unless I explicitly say STOP or a real blocker occurs.
Now that Brave/web_search is working, the next batch should be much less browser-search-fragile.
oak pulsarBOT
Absolutely — here’s a refined version that keeps your “selling signals” goal, but makes it safer for OpenClaw: append-only source of truth, checkpoints, batch gates, proof-of-work, and no silent CSV clobbering.
Claw, continue the existing Mission Control research workflow, but start a new selling-signals refinement run focused on actionable outreach opportunities for Production Resources, Inc.
Goal:
Build a durable, evidence-backed list of selling signals and sales-entry angles for Production Resources, Inc. Do not drift into generic company summaries.
Run mode:
Work continuously in small batches until I explicitly say STOP, a real blocker occurs, or validation fails.
Use batches of up to 5 companies at a time.
After each batch, return a concise batch report, then continue to the next batch automatically if safe.
Timebox:
Aim for up to 45 minutes total continuous work, but preserve durability over speed.
Spend about 2–3 minutes per company on the first pass.
Do not spend the whole run on one company.
Research universe:
Use companies from Mission Control / the configured local Mission Control source.
Do not invent companies.
If the batch queue is empty, pull the next 1–5 companies from the configured local source, dedupe against existing raw findings and derived exclusions, then continue.
Create a new timestamped selling-signals run folder, for example:
research_run_YYYY-MM-DD/selling_signals_YYYYMMDD_HHMMSS/
Inside it create:
- raw/signals.jsonl append-only source of truth
- checkpoint.json updated after every company
- reports/run_summary.md progress log
- derived/selling_signals.csv generated only from validated JSONL
- derived/top_10.md generated at the end or after batch checkpoints
Important output rule:
raw/signals.jsonl is the source of truth.
Do not write research directly only to CSV.
Append one JSONL record immediately after confirming each signal or no-signal outcome.
Generate CSV only from validated JSONL.
Never overwrite an existing CSV; use a new timestamped filename or version suffix.
Required CSV columns, derived from JSONL:
company_name
company_id
source_url_1
source_url_2
source_url_3
signal_type
signal_summary
why_it_matters_for_pri
likely_pri_angle
urgency
confidence
evidence_date
location
recommended_follow_up
notes
Research method per company:
Use web_search plus browser-based webpage inspection.
Do not rely only on search snippets.
Open and inspect actual pages whenever possible.
Review 2–4 relevant sources when available, prioritizing:
- official company website
- newsroom / press releases
- careers page
- LinkedIn company page or hiring signals
- local business journals
- manufacturing publications
- plant directories
- supplier / manufacturing coverage
- credible news articles
Prioritized selling signals:
Rank higher any evidence tied to:
- facility expansion
- capex or major investment
- new equipment or automation
- pressroom, stamping, fabrication, or material-handling activity
- plant construction or new production lines
- hiring for manufacturing, maintenance, engineering, tooling, automation, or operations leadership
- acquisitions or integration activity
- productivity, safety, uptime, retrofit, or maintenance pressure
- supplier awards, launches, production ramp-ups
- reshoring, footprint growth, or new regional manufacturing presence
Interpretation rules:
- signal_summary = factual signal found, grounded in sources
- why_it_matters_for_pri = why this may create demand for Production Resources, Inc.
- likely_pri_angle = plausible opening, such as stamping support, press upgrades, automation support, material handling, installation tied to equipment, maintenance-driven replacement, production expansion support, tooling/fabrication support, uptime/safety retrofit
- urgency = high / medium / low based on recency and actionability
- confidence = high / medium / low based on source quality and specificity
- recommended_follow_up = one concrete next step Jake could take
Status / signal rules:
For each company, write one or more rows if confirmed signals exist.
Use one row per confirmed signal.
If no confirmed selling signal is found but the company is relevant, write exactly one row with:
signal_type = no_clear_signal_yet
and include the best honest current sales angle.
If the company is not relevant for PRI, write exactly one row with:
signal_type = not_relevant
and explain why in notes.
Do not invent signals.
Do not inflate weak evidence into a confirmed signal.
Required JSONL fields per row:
run_id
company_name
company_id
source_urls
signal_type
signal_summary
why_it_matters_for_pri
likely_pri_angle
urgency
confidence
evidence_date
location
recommended_follow_up
notes
retrieved_at
errors
Checkpointing rules:
After every company:
1. Append validated JSONL row(s).
2. Validate the appended JSON parses.
3. Update checkpoint.json with:
- last_completed_company_id
- last_completed_company_name
- companies_attempted
- companies_with_confirmed_signals
- no_clear_signal_yet_count
- not_relevant_count
- signal_rows_total
- current_batch_position
- current_batch_total
- batch_complete
- updated_at
4. Confirm row count increased appropriately.
5. Append a short progress note to reports/run_summary.md.
Batch validation:
After every 5 companies:
- validate raw/signals.jsonl parses
- verify checkpoint counts match JSONL
- verify CSV can be regenerated from JSONL
- generate/update a new derived CSV version from JSONL
- return a concise batch report
Batch report must include:
- companies attempted
- signal rows written
- statuses/signal types written
- JSONL rows before/after
- checkpoint position
- files updated
- strongest signals found
- blockers/errors
- validation result
- whether it is safe to continue
Continuation behavior:
After returning each batch report, continue to the next batch automatically unless I explicitly say STOP or a real blocker occurs.
Do not treat the batch report as completion.
Blocker rules:
If one source is blocked, pivot immediately to another source.
If one source times out but browser health passes on a known-good page, record the source error and continue.
If browser health fails on a known-good page, stop and report the browser blocker.
If web_search fails, report the exact error and continue with browser only only if quality remains acceptable.
If repeated search/access issues affect 2 or more companies in a batch, stop and report a retrieval-quality blocker instead of filling weak rows.
If Mission Control source is unavailable or no next companies can be loaded, stop and report that input blocker.
Final 45-minute summary:
At the end of the run, return:
- path to the latest derived CSV
- path to checkpoint.json
- path to raw/signals.jsonl
- top 10 companies with strongest selling signals
- strongest signal types observed
- recommended follow-ups for Production Resources, Inc.
- blocker notes that affected coverage
- whether the workflow is safe to continue in another batch
Remember:
Durable evidence beats volume.
Do not overwrite raw files.
Do not fabricate progress.
Every signal must be source-backed or clearly marked as no_clear_signal_yet / not_relevant.
Quick honest status:
The old company research run is still intact and repaired
web_search is working
current raw research is still safe at 63 rows
the selling-signals run has not actually produced signal rows yet
What happened:
I set up the new selling-signals workflow and spawned it
but that spawned run basically only started/announced and did not get through actual batch execution
so right now we do not yet have a real raw/signals.jsonl batch underway
So the truth is: the groundwork is in place, but we’re not as far along as we should be on the actual selling-signal build.
Next move:
I need to run the selling-signals pass for real and start writing durable signal rows/batch reports, instead of just standing up the framework.
oak pulsarBOT
Yep — this needs a sharper “execution, not setup” reprompt. Paste this:
Run the selling-signals pass for real now.
Important correction:
The selling-signals framework/run folder may already exist, but no real signal rows have been produced yet. Do not stop after setup, spawning, announcing, or preparing files. The task is not underway until raw/signals.jsonl has durable rows appended.
Current verified state:
- old company research run is intact and repaired
- raw research has 63 safe rows
- web_search is working
- selling-signals run exists or can be created
- raw/signals.jsonl currently has no real batch rows yet
Your task now:
1. Locate or create the current selling-signals run folder.
2. Confirm raw/signals.jsonl row count before starting.
3. Load the first/next 5 companies from Mission Control/local source into the selling-signals batch queue.
4. Verify web_search works with one simple query.
5. Verify browser health with start/tabs/open/snapshot.
6. Immediately research the queued companies and append durable signal rows.
Do not ask whether to start.
Do not stop after setup.
Do not spawn another background framework-only run.
Start processing the first company now.
For each company:
- Use web_search plus browser inspection of actual pages.
- Review 2–4 relevant sources if available.
- Append one row per confirmed selling signal to raw/signals.jsonl.
- If no confirmed signal is found but company is relevant, append one row with signal_type=no_clear_signal_yet.
- If not relevant, append one row with signal_type=not_relevant.
- Include source_urls, signal_summary, why_it_matters_for_pri, likely_pri_angle, urgency, confidence, evidence_date, location, recommended_follow_up, notes, retrieved_at, and errors.
- Update checkpoint.json immediately after each company.
- Validate JSONL parses after each company.
Minimum completion for this run:
Process at least 1 company and append at least 1 valid row before returning, unless a real blocker prevents it.
Real blockers only:
- no Mission Control/local company source available
- web_search and browser both fail
- browser health fails on a known-good page
- file validation/checkpoint write fails
- login/captcha/rate-limit blocks the whole batch
If blocked, report the exact blocker and the last successful step.
Final response must include proof:
- selling-signals run folder path
- companies attempted
- signal rows written
- raw/signals.jsonl rows before/after
- checkpoint position
- files updated
- strongest signals found, if any
- blockers/errors
- validation result
- whether it is safe to continue the next batch
The important line is: “Minimum completion: process at least 1 company and append at least 1 valid row before returning.” That prevents another framework-only pass.
chilly fiber
The selling-signals workflow is progressing, but there is a major UX/observability issue right now:
OpenClaw is not showing a persistent “typing” or “working” state while active research/web investigation is occurring. From the outside, it appears idle even when status updates claim research is happening.
I need this fixed so the agent visibly demonstrates activity during:
web_search operations
browser navigation
tab/page analysis
signal extraction
durable row writing
queue advancement
checkpoint saves
Required behavior:
As soon as research begins on a company, OpenClaw should immediately enter a visible typing/working state.
The typing indicator should persist continuously during long-running research operations instead of only appearing at the end when status messages are posted.
Intermediate progress updates should stream naturally during execution, for example:
“Opening source pages…”
“Analyzing plant/facility evidence…”
“Extracting durable selling signals…”
“Writing JSONL row…”
“Advancing checkpoint…”
If a step stalls longer than ~15–20 seconds, emit a heartbeat/progress message so it’s obvious the workflow is still alive.
Browser-driven actions should feel live and observable, not batch-completed silently in the background.
The system should prioritize transparency of execution over minimizing message count.
Current state example:
selling-signals run folder created
web_search passed
browser health passed
first company processed:
durable signal row written
checkpoint 1/5
Problem:
Despite these updates, there is no visible typing/activity indicator during the actual research phase, making it appear that no real web research is occurring.
Goal:
Make OpenClaw feel continuously active, transparent, and visibly “thinking/researching” while work is in progress.
oak pulsarBOT
You can improve this in two layers: config-level visibility + prompt-level progress reporting.
First, turn on earlier typing and Discord streaming/progress previews:
openclaw config set agents.defaults.typingMode instant
openclaw config set agents.defaults.typingIntervalSeconds 4 --strict-json
openclaw config set channels.discord.streaming partial
openclaw config set channels.discord.streaming.preview.toolProgress true --strict-json
openclaw gateway restart
Docs:
https://docs.openclaw.ai/concepts/typing-indicators
https://docs.openclaw.ai/channels/discord#live-stream-preview
If the selling-signals work is running as a background task/subagent, also set its task notifications to progress/state changes:
openclaw tasks list --status running
openclaw tasks notify <task-or-run-id> state_changes
Important caveat: Discord native typing is best-effort and channel/runtime-dependent. It can show “working,” but it is not a durable audit trail. For your use case, use actual progress messages/checkpoints too.
Add this to the selling-signals prompt:
Observability mode:
Prioritize visible progress over minimizing message count.
When starting each company, immediately post a short visible progress update:
“Starting COMPANY — opening sources…”
During each company, post progress at these milestones:
- web_search started / completed
- opening source pages
- analyzing facility / plant / hiring / investment evidence
- extracting selling signals
- writing JSONL row
- updating checkpoint
- company complete
If any step takes longer than ~20 seconds, post a heartbeat update like:
“Still working on COMPANY — currently inspecting source pages / waiting on browser / extracting evidence.”
Do not wait until the end of the batch to prove activity.
Do not claim research is happening silently for multiple minutes.
Keep updates concise. One sentence is enough.
Still preserve durable output rules:
- append JSONL immediately after each company/signal
- update checkpoint after every company
- validate after every company
- stop on real blockers
Batch reports are still required, but they are not a substitute for live progress updates.
If you want this as a product-level behavior for all long browser runs, that’s probably worth a GitHub issue: https://github.com/openclaw/openclaw/issues. The ask would be: “persistent working indicator + periodic progress heartbeats for long-running browser/tool tasks, especially detached/subagent research runs.”
chilly fiber
finally starting to make some progress, but stalled out again with no message. please reprompt
Starting [COMPANY] — opening sources…
[COMPANY] — web_search completed. Strong expansion/investment leads identified. Opening source pages now.
[COMPANY] — source pages are open. Analyzing facility expansion, production changes, hiring activity, capex indicators, and operational signals now.
[COMPANY] — extracting selling signal, writing durable JSONL row, and updating checkpoint now.
[COMPANY] complete.
confirmed signal: [HIGH_CONFIDENCE_SIGNAL_TYPE]
raw/signals.jsonl rows: [PREVIOUS_COUNT] → [NEW_COUNT]
checkpoint: [CURRENT]/[TOTAL]
Starting [NEXT_COMPANY] — opening sources…
oak pulsarBOT
Yep — the stall point is clear: it posted “Starting next company” but didn’t enforce a next observable milestone or timeout. Use this as a recovery/continuation prompt:
Resume the selling-signals run from the last visible progress update.
Last visible state:
“Starting [NEXT_COMPANY] — opening sources…”
Do not restart the whole run.
Do not skip [NEXT_COMPANY] unless it is already completed in raw/signals.jsonl.
Do not silently continue without progress updates.
First, determine current state:
1. Check raw/signals.jsonl row count.
2. Check checkpoint.json.
3. Check whether [NEXT_COMPANY] already has a completed signal/no-signal row.
4. Check browser health on a known-good page if the browser may be stalled.
Then do exactly one of these:
A) If [NEXT_COMPANY] was completed:
- report the row count and checkpoint
- continue immediately to the next queued company
B) If [NEXT_COMPANY] was started but not completed:
- resume [NEXT_COMPANY] now
- post a visible progress update before each step
- complete it by appending at least one durable JSONL row, unless a real blocker occurs
C) If browser/search is blocked:
- report the exact blocker
- report the last successful step
- do not pretend progress is happening
Observability requirement:
For every company, emit visible progress at least every 15–20 seconds while working.
Use these milestones:
- “Starting COMPANY — opening sources…”
- “COMPANY — web_search started…”
- “COMPANY — web_search completed; opening source pages…”
- “COMPANY — inspecting source pages…”
- “COMPANY — extracting selling signal…”
- “COMPANY — writing JSONL row…”
- “COMPANY — updating checkpoint…”
- “COMPANY complete: status/signals, rows X → Y, checkpoint A/B”
If any milestone takes longer than 20 seconds, post:
“Still working on COMPANY — currently [specific step].”
Minimum completion before returning:
Process [NEXT_COMPANY] to a durable row and checkpoint update, unless a real blocker prevents it.
Final response for this recovery step must include:
- whether [NEXT_COMPANY] was already complete or resumed
- rows before/after
- checkpoint before/after
- files updated
- blocker, if any
- next company to process
And for the broader loop, add this stricter rule:
Never end a message with only “Starting NEXT_COMPANY…” unless another progress update will be sent within 20 seconds. If you cannot continue immediately after saying that, say why and stop cleanly.
We appear to be in a very stable and functional state now.
Using all prior context, recent prompts, workflows, and observed behavior, please generate one final master prompt that allows me to replicate exactly what Claw is currently doing autonomously:
- iterating through a company database/list
- performing deep company research
- identifying operational / expansion / capital investment activity
- extracting actionable selling signals
- writing durable structured outputs
- maintaining checkpoints and resumability
- avoiding duplicate or low-confidence findings
- operating in small controlled batches
- producing concise research summaries and signal classifications
The master prompt should support explicit operational commands including:
START
- begin processing the next queued batch
- continue from the latest checkpoint automatically
- validate browser/tool health before execution
- safely write outputs incrementally
STOP
- gracefully halt processing
- preserve checkpoint integrity
- avoid partial writes or corrupted output
- provide current status summary before shutdown
The system should prioritize:
- durable JSONL-style output handling
- verifiable evidence-based findings
- facility-level intelligence when possible
- structured selling signal categorization
- resilience against browser/tool hiccups
- safe continuation after interruptions
- transparent progress reporting
Then, separately, generate a second recovery/continuation prompt specifically designed for situations where the agent stalls, crashes, loses context, encounters browser instability, or partially completes work.
That recovery prompt should:
- inspect the current run state
- validate checkpoints and existing outputs
- detect the last successfully completed company
- avoid duplicate research/writes
- safely resume unfinished work
- identify blockers instead of hallucinating progress
- provide a concise operational status report before continuing
- confirm whether it is safe to resume execution
oak pulsarBOT
Absolutely. Here are the two prompts I’d keep as your “known good” operating kit.
1. Master autonomous research prompt
You are running an autonomous selling-signals research workflow for Production Resources, Inc.
Your job is to iterate through a company database/list, research companies in small controlled batches, identify operational / expansion / capital-investment activity, extract actionable selling signals, and write durable structured outputs that can be resumed safely after interruption.
Primary objective:
Find evidence-backed selling signals that suggest why Production Resources, Inc. should reach out now.
Prioritize signals related to:
- facility expansion
- new plant / line / production capacity
- capex or major manufacturing investment
- pressroom, stamping, tooling, fabrication, or material-handling activity
- automation, robotics, controls, or equipment upgrades
- production ramp-ups, launches, supplier awards, or new programs
- maintenance, uptime, safety, retrofit, or productivity pressure
- hiring for manufacturing, maintenance, engineering, tooling, automation, or operations leadership
- acquisitions, integrations, reshoring, footprint growth, or new regional manufacturing presence
Operational commands:
START:
Begin or continue processing the next queued batch.
Continue from the latest checkpoint automatically.
Validate browser/tool/file health before execution.
Write outputs incrementally and durably.
Do not start over unless the run state is unreadable.
STOP:
Gracefully halt after the current safe checkpoint.
Do not leave a partially written JSONL row.
Do not corrupt checkpoint state.
Do not overwrite raw outputs.
Before stopping, provide a concise current status summary with row counts, checkpoint position, last completed company, files updated, blockers, and whether the run is safe to resume.
Run structure:
Use or create a timestamped selling-signals run folder, for example:
research_run_YYYY-MM-DD/selling_signals_YYYYMMDD_HHMMSS/
Inside it maintain:
- raw/signals.jsonl append-only source of truth
- checkpoint.json updated after every company
- batch-queue.txt current batch queue
- reports/run_summary.md progress log
- reports/derived_exclusions.json optional duplicate/invalid-row exclusions
- derived/selling_signals.TIMESTAMP.csv generated only from validated JSONL
- derived/top_10.TIMESTAMP.md strongest signals summary
Durability rules:
raw/signals.jsonl is the source of truth.
Never overwrite raw/signals.jsonl.
Never delete old raw rows.
Never write research only to CSV.
Append durable JSONL rows first, validate them, then derive CSV from JSONL.
Never overwrite an existing CSV; create a new timestamped/versioned CSV.
If a row needs correction, append a correction row or update a derived exclusion file. Do not silently rewrite history.
Batch rules:
Work in batches of up to 5 companies.
After each batch, return a concise batch report, then continue automatically to the next batch unless I say STOP or a real blocker occurs.
Do not treat the batch report as completion.
If the queue is empty, pull the next 1–5 companies from the configured database/local Mission Control source.
Deduplicate new batch candidates against:
- existing raw/signals.jsonl company IDs/names
- correction rows
- reports/derived_exclusions.json
- current batch-queue.txt
Do not invent companies.
Preflight before each batch:
1. Read checkpoint.json.
2. Validate raw/signals.jsonl parses.
3. Record starting row count.
4. Confirm batch-queue.txt has companies or load the next batch from the configured source.
5. Test web_search with a simple query if not recently verified.
6. Verify browser health with start/tabs/open/snapshot on a known-good public page.
7. Append a short preflight note to reports/run_summary.md.
Research method per company:
Use web_search plus browser-based inspection of actual pages.
Do not rely only on search snippets.
Open and inspect relevant pages when possible.
Review 2–4 sources per company when available, such as:
- official company website
- newsroom / press releases
- careers page
- LinkedIn company page or hiring signals
- local business journals
- manufacturing publications
- plant directories
- supplier/manufacturing coverage
- credible news articles
Facility-level intelligence is preferred when available.
If a queue item appears to be a facility alias, plant name, division, or duplicate of a broader company, record that honestly and avoid duplicate derived outputs.
Per-company output rules:
For each company, append one or more JSONL rows:
- one row per confirmed selling signal, or
- one row with signal_type = no_clear_signal_yet if relevant but no confirmed signal is found, or
- one row with signal_type = not_relevant if the company is not relevant for PRI, or
- one row with signal_type = facility_alias_or_overlap if it overlaps a broader company/facility entity.
Do not invent signals.
Do not inflate weak evidence into a confirmed signal.
Use partial/low confidence when evidence is useful but incomplete.
Required JSONL fields:
run_id
company_name
company_id
source_urls
signal_type
signal_summary
why_it_matters_for_pri
likely_pri_angle
urgency
confidence
evidence_date
location
recommended_follow_up
notes
retrieved_at
errors
Signal classification:
signal_type should be one of:
- facility_expansion
- capex_investment
- new_equipment_or_automation
- pressroom_or_stamping_activity
- fabrication_or_material_handling
- plant_construction_or_new_line
- hiring_manufacturing_ops
- acquisition_or_integration
- productivity_safety_uptime_pressure
- supplier_award_or_launch
- reshoring_or_footprint_growth
- no_clear_signal_yet
- not_relevant
- facility_alias_or_overlap
- error
- blocked
Interpretation rules:
signal_summary = factual signal found, grounded in source evidence.
why_it_matters_for_pri = why this could create demand for Production Resources, Inc.
likely_pri_angle = plausible outreach angle, such as stamping support, press upgrades, automation support, material handling, equipment installation, maintenance-driven replacement, production expansion support, tooling/fabrication support, or uptime/safety retrofit.
urgency = high / medium / low based on recency and actionability.
confidence = high / medium / low based on source quality and specificity.
recommended_follow_up = one concrete next step Jake could take.
Checkpointing after every company:
1. Append JSONL row(s).
2. Validate appended JSON parses.
3. Update checkpoint.json with:
- last_completed_company_id
- last_completed_company_name
- companies_attempted
- companies_with_confirmed_signals
- no_clear_signal_yet_count
- not_relevant_count
- partial_or_low_confidence_count
- signal_rows_total
- output_rows
- current_batch_position
- current_batch_total
- batch_complete
- updated_at
4. Confirm raw/signals.jsonl row count increased appropriately.
5. Append a short progress note to reports/run_summary.md.
Validation after every batch:
- raw/signals.jsonl parses cleanly
- checkpoint output_rows matches JSONL line count
- checkpoint last completed fields match latest completed company or correction row
- source_urls exist for researched/partial signal rows
- no duplicate company/signal rows were added without explanation
- derived exclusions parse if present
- generate a new timestamped derived CSV from validated JSONL
Progress / observability rules:
Prioritize transparent progress over minimizing message count.
When starting each company, immediately post:
“Starting COMPANY — opening sources…”
Post concise progress updates at milestones:
- “COMPANY — web_search started…”
- “COMPANY — web_search completed; opening source pages…”
- “COMPANY — inspecting source pages…”
- “COMPANY — extracting selling signal…”
- “COMPANY — writing JSONL row…”
- “COMPANY — updating checkpoint…”
- “COMPANY complete: status/signals, rows X → Y, checkpoint A/B”
If any step takes longer than 15–20 seconds, post:
“Still working on COMPANY — currently [specific step].”
Never end a visible update with only “Starting NEXT_COMPANY…” unless another progress update will follow within ~20 seconds. If you cannot continue immediately, say why and stop cleanly.
Blocker handling:
If one source is blocked, times out, 404s, or hits Cloudflare/reCAPTCHA, pivot to another source and record the issue in errors[].
If browser health still passes on a known-good page, treat the failure as page-specific, not global.
If browser health fails on a known-good page, stop and report a browser blocker.
If web_search fails, report the exact error and continue with browser-only only if quality remains acceptable.
If 2 or more companies in a batch are materially limited by search/captcha/access restrictions, stop and report a retrieval-quality blocker instead of filling weak rows.
If Mission Control/local source is unavailable, stop and report an input blocker.
If file validation/checkpoint consistency fails, stop and report a state blocker.
Batch report format:
After each batch, report:
- companies attempted
- signal rows written
- signal types/statuses written
- JSONL rows before/after
- checkpoint position
- files updated
- strongest signals found
- blockers/errors
- validation result
- whether it is safe to continue
Then continue to the next batch automatically unless I explicitly say STOP or a real blocker occurs.
Final summary when stopped or timeboxed:
Return:
- latest derived CSV path
- raw/signals.jsonl path
- checkpoint.json path
- top 10 companies with strongest selling signals
- strongest signal types observed
- recommended follow-ups for Production Resources, Inc.
- blocker notes that affected coverage
- whether the workflow is safe to continue
Remember:
Durable evidence beats volume.
Do not fabricate progress.
Do not overwrite raw files.
Do not silently repair or delete raw history.
Every signal must be source-backed or explicitly marked no_clear_signal_yet / not_relevant / facility_alias_or_overlap / blocked / error.
2. Recovery / continuation prompt
Recover and safely resume the selling-signals research workflow.
Use this prompt when the agent stalls, crashes, loses context, browser/search becomes unstable, progress messages stop, or work may have partially completed.
Do not start new research immediately.
Do not append new rows until current state is inspected and validated.
Do not overwrite raw/signals.jsonl.
Do not delete old rows.
Do not generate CSV until validation passes.
Recovery objective:
Inspect the current run state, validate outputs/checkpoints, identify the last successfully completed company, avoid duplicate writes, determine whether any company was partially started, and resume safely only if the run is healthy.
Step 1 — Locate current run:
Find the active selling-signals run folder.
Confirm these files exist or report which are missing:
- raw/signals.jsonl
- checkpoint.json
- batch-queue.txt
- reports/run_summary.md
- reports/derived_exclusions.json if present
Step 2 — Validate raw output:
- Count raw/signals.jsonl rows.
- Validate every JSONL row parses, or at minimum validate the latest 20 rows plus the last appended row.
- Identify the latest valid row.
- Identify any malformed, partial, duplicate, or correction rows.
- Do not modify anything yet.
Step 3 — Validate checkpoint:
Inspect checkpoint.json and compare it to raw/signals.jsonl.
Check:
- output_rows matches JSONL row count
- signal_rows_total matches rows intended for derived outputs
- last_completed_company_id/name matches latest completed row or latest correction row
- current_batch_position and current_batch_total are consistent
- batch_complete is true if current_batch_position equals current_batch_total
- status/signal counts are internally consistent
If checkpoint is inconsistent, report the exact fields and expected values.
Do not silently repair unless explicitly instructed.
Step 4 — Detect current company state:
Determine:
- last successfully completed company
- current queued company
- whether the last visible “Starting COMPANY…” message corresponds to a completed row
- whether a company was started but not durably written
- whether the same company already exists in raw/signals.jsonl
- whether derived_exclusions marks it as duplicate/invalid
If the current company already has a valid row, do not research or write it again. Advance to the next queued company.
If it was started but has no durable row, resume that company.
Step 5 — Tool health checks:
Before resuming research, verify:
- web_search works with a simple query
- browser starts or is already running
- tabs are readable
- a known-good public page opens
- snapshot works
If a tool fails, report the exact blocker and last successful check.
Do not pretend research is happening.
Step 6 — Operational status report:
Before continuing, report concisely:
- active run folder
- JSONL row count
- checkpoint status: valid / inconsistent
- last completed company
- current queued company
- whether any duplicate/partial work was detected
- browser health
- web_search health
- whether it is safe to resume
Step 7 — Resume only if safe:
If safe_to_resume = true:
- continue from the first unfinished queued company
- do not restart the batch from the beginning
- do not duplicate completed rows
- post progress updates every 15–20 seconds during work
- append durable JSONL row(s) after each company
- update checkpoint after each company
- validate after each company
If safe_to_resume = false:
- stop
- report the blocker
- provide a concrete repair plan
- do not append research rows
Recovery progress rule:
When resuming a company, emit:
“Resuming COMPANY from recovery — opening sources…”
Then continue with milestone updates:
- web_search started/completed
- source pages opened
- source pages inspected
- signal extracted
- JSONL row written
- checkpoint updated
- company complete with rows X → Y
Minimum safe recovery completion:
If safe to resume, process at least the current unfinished company to a durable row and checkpoint update before returning, unless a real blocker occurs.
Final recovery response must include:
- recovered run folder
- last completed company