#MEMORY.md bloat/curation/trimming

1 messages · Page 1 of 1 (latest)

simple hatch
#

@runic fractal , openclaw doctor is reporting the following:

Bootstrap file size
• MEMORY.md is at 94% of its per-file limit

It suggests trimming. This doesn’t seem like it should be a manual process, though. Isn’t this supposed to be handled automatically by dreaming and memory-core?

If not, are there commands to run to force automatic trimming/pruning/synthesis/consolidation to keep MEMORY.md within its limits?

hasty tideBOT
#

To help others find answers, you can mark your question as solved via Right click solution message -> Apps -> âś… Mark Solution

runic fractalBOT
#

Short answer: no, not today.

That doctor warning is about the workspace bootstrap prompt budget, not an automatic MEMORY.md maintenance pass.

What the current docs/code show:

  • MEMORY.md is auto-injected into the prompt every session.
  • Doctor warns when a bootstrap file is near the per-file injection cap.
  • Default caps are:
    • agents.defaults.bootstrapMaxChars = 20000
    • agents.defaults.bootstrapTotalMaxChars = 150000
  • The “near limit” warning kicks in before truncation, so 94% just means “close”, not “already broken”.

And on dreaming specifically:

  • Light phase: never writes to MEMORY.md
  • REM phase: never writes to MEMORY.md
#
  • Deep phase: appends promoted entries to MEMORY.md

So dreaming is a promotion/consolidation pipeline into MEMORY.md, not an auto-prune/rewrite/shrink pipeline for MEMORY.md itself.

Also, I did not find a CLI command that does automatic:

  • trim
  • prune
  • rewrite
  • synthesize-down
  • consolidate-existing-MEMORY.md

The current commands are more like:

  • openclaw memory promote --apply → manually run promotion now, adds to MEMORY.md
  • openclaw memory index --force → rebuilds search index
  • openclaw memory status --fix / openclaw doctor --fix → repairs recall/dreaming artifacts, not MEMORY.md size
#

So if you want to stop that warning right now, the real options are:

  1. Manually curate MEMORY.md
  2. Raise the bootstrap limits if the larger file is intentional:
    • agents.defaults.bootstrapMaxChars
    • agents.defaults.bootstrapTotalMaxChars

Docs:

If you want, paste your openclaw --version and I can sanity-check whether this changed in your specific build.

simple hatch
#

I;m on v2026.4.15

runic fractalBOT
#

Yep, on v2026.4.15 I’d still read it the same way.

This is a bootstrap-context budget warning, not “dreaming failed to compact memory.”

So on your version:

  • No built-in auto-pruner for MEMORY.md that I could find
  • dreaming / memory-core promote into MEMORY.md, they do not rewrite it downward
  • openclaw memory promote --apply adds
  • openclaw memory status --fix and openclaw doctor --fix repair artifacts, not MEMORY.md size
  • openclaw memory index --force only rebuilds the index

So the practical choices are still:

  1. Trim/curate MEMORY.md manually, or
  2. Increase the bootstrap budget if your durable memory is intentionally larger:
    • agents.defaults.bootstrapMaxChars
#
  • agents.defaults.bootstrapTotalMaxChars

I’m with you, this does feel like something that should eventually have an automatic maintenance path. If you want that behavior, that’s probably a feature request: https://github.com/openclaw/openclaw/issues

If you want, I can help sketch what a “good” MEMORY.md curation strategy looks like so it stays useful without constant babysitting.

simple hatch
#

Please do.

runic fractalBOT
#

My rule of thumb: treat MEMORY.md like a durable index, not an archive.

A good strategy is:

Put in MEMORY.md

Only things that are all or mostly true:

  • durable
  • high value
  • likely to matter again
  • hard to recover from context/search

Examples:

  • stable preferences
  • recurring constraints
  • long-lived environment quirks
  • important relationship/context notes
#
  • repeated failure modes + fix
  • canonical terminology

Keep out of MEMORY.md

Move these to daily notes, docs, or wiki-style storage instead:

  • one-off task status
  • temporary project details
  • meeting notes
  • long examples/logs
  • speculative ideas
  • stale incident timelines
  • repeated bullets saying the same thing slightly differently

A nice shape

Keep it short, sectioned, and deduped, for example:

#
  • Durable facts / preferences
  • Repeated pain points
  • Environment quirks
  • Important relationships / naming
  • Known gotchas

Good compression pattern

Instead of 5 bullets like:

  • user hit auth bug on Mac
  • auth bug happened with nvm
  • old node path caused issue
  • gateway used stale global install
  • reinstall fixed it

compress to 1 durable bullet:

#
  • macOS + nvm: LaunchAgent can keep pointing at an old global Node/OpenClaw path after Node upgrades; reinstall the gateway service from the intended Node version.

That keeps the recall value without the sprawl.

Easy maintenance pass

When MEMORY.md gets big, do this:

  1. Delete stale items
  2. Merge duplicates
  3. Collapse incident clusters into one canonical bullet
  4. Move bulky detail to memory/YYYY-MM-DD.md or external docs
  5. Keep only the distilled takeaway in MEMORY.md

Practical split I’d recommend

  • MEMORY.md = durable distilled truths
  • memory/YYYY-MM-DD.md = working notes / recent context
  • DREAMS.md = review surface
#
  • memory-wiki = better if you want a more maintained knowledge base without stuffing everything into prompt bootstrap

If you want to stay under the limit

Aim for bullets that are:

  • one idea each
  • 1 to 3 lines
  • no raw logs
  • no repeated context
  • no “history of the issue” unless the history itself matters

If you want, I can also give you a copy-paste rubric for “should this go in MEMORY.md or not?”

simple hatch
#

All of that is helpful. Thanks. The problem seems to be with OpenClaw's memory promotion. Looking at the bloat, it's all entries like this:

openclaw-memory-promotion:memory:memory/2026-04-01.md:1:18

PENDING TASKS - 2026-04-01 - No critical pending tasks carried forward. Wednesday, April 1, 2026 > Log reconstructed — original was not written. Derived from git history and nightly audit report. System Fixes - Bootstrap Vault Sync timeout increased from 30s to 120s (14a7f27) — audit caught a timeout error on the prior run. - Hooks agent prompt fix — corrected 'message' parameter name in hooks agent prompt (372d97c). Nightly Integrity Audit (April 1, ~06:48 EST) - Flagged new Pango-Butler Storefront Audit cron job as uncommitted (added March 31). - Flagged openclaw.json browser plugin enable as uncommitted. - Bootstrap Vault Sync noted as erroring (resolved by timeout fix above). - All model governance checks passed. Session Summary Reconstructed. [score=0.821 recalls=13 avg=0.380 source=memory/2026-04-01.md:1-18]

runic fractalBOT
#

Yeah, I think you found the real issue.

That example looks less like “MEMORY.md is too big” and more like promotion is selecting the wrong unit of memory.

What I’m seeing in that blob:

  • it promoted a whole daily-note block: memory/2026-04-01.md:1-18
  • that block contains:
    • pending-task status
    • reconstructed session log text
    • audit output
    • one-off fixes
    • timestamps / commit refs
  • and it got promoted because it had enough recall/signal to clear deep-phase thresholds

That’s exactly the kind of thing I would not want in durable memory.

#

Important nuance

Current docs say:

  • deep promotion re-reads the live daily note before writing
  • so if you edit or delete a noisy source block in memory/YYYY-MM-DD.md, it should stop future promotions from using stale old recall-store text

That part is good news.

What I’d do right now

  1. Turn off dreaming temporarily if it’s auto-promoting junk
    /dreaming off
    or set plugins.entries.memory-core.config.dreaming.enabled: false

  2. Preview promotion candidates manually instead of letting auto deep promotion write:

# 
   openclaw memory promote
   openclaw memory promote-explain 2026-04-01.md
   openclaw memory rem-harness

  1. Use stricter one-off manual thresholds when applying:

    openclaw memory promote --min-score 0.9 --min-recall-count 5 --min-unique-queries 5

    That won’t fix granularity, but it can reduce junk getting through.

  2. Rewrite/remove the source daily-note blobs

    • split giant summaries into smaller bullets
    • keep durable facts separate from audit logs / reconstructions / task carry-forward text
    • avoid giant “session reconstructed” blocks in promotable daily memory
#
  1. Manually clean existing bad MEMORY.md entries
    because there isn’t a built-in auto-pruner for those yet

The underlying problem

This smells like a promotion-quality / candidate-granularity bug, not just operator error.

In other words, OpenClaw is technically doing what the pipeline says, but the promoted candidate shape is too coarse. A line range like :1-18 full of operational noise should probably not graduate as one durable memory item.

One annoying limitation

The docs currently say the scheduled dreaming thresholds are internal, not user-configurable in config.
So today the best controls are mostly:

  • disable auto dreaming
  • preview manually
  • use memory promote CLI threshold overrides for one-off runs
  • clean the source daily notes
#

If you want, I can help you turn that example into a tight GitHub issue report. I think this is issue-worthy.

simple hatch
#

Please do.

runic fractalBOT
#

Yep, here’s a draft you can paste into GitHub:

https://github.com/openclaw/openclaw/issues

## Summary

`memory-core` / dreaming is promoting large mixed-content daily-note blocks into `MEMORY.md` as single durable memory entries.

This leads to `MEMORY.md` bloat and eventually triggers `openclaw doctor` bootstrap-budget warnings like:

- `Bootstrap file size`
- `MEMORY.md is at 94% of its per-file limit`

From the current docs/CLI behavior, there also does not appear to be a built-in command to automatically prune/rewrite/synthesize `MEMORY.md` back down after these promotions.
# 
## Version

`openclaw v2026.4.15`

## What I expected

I expected long-term promotion to prefer **atomic durable facts** and avoid promoting whole operational blobs containing things like:

- pending task carry-forward
- reconstructed session summaries
- audit output
- one-off timestamps
- commit references
- transient fix logs
# 
If a block is too broad/noisy, I’d expect it either:
- not to be promoted, or
- to be distilled into smaller durable facts

## What actually happened

Large line-range blocks from `memory/YYYY-MM-DD.md` are getting promoted into `MEMORY.md` wholesale.

Representative promoted item shape:

```text
openclaw-memory-promotion:memory:memory/2026-04-01.md:1:18

PENDING TASKS - 2026-04-01 - No critical pending tasks carried forward. Wednesday, April 1, 2026 > Log reconstructed — original was not written. Derived from git history and nightly audit report. System Fixes - Bootstrap Vault Sync timeout increased from 30s to 120s (`14a7f27`) — audit caught a timeout error on the prior run. - Hooks agent prompt fix — corrected `'message'` parameter name in hooks agent prompt (`372d97c`). Nightly Integrity Audit (April 1, ~06:48 EST) - Flagged new ... [score=0.821 recalls=13 avg=0.380 source=memory/2026-04-01.md:1-18]
# 

That kind of blob does not seem appropriate for durable memory, but it still passes promotion and gets appended to `MEMORY.md`.

## Why this matters

This causes:

- `MEMORY.md` to accumulate low-signal operational noise
- bootstrap prompt bloat
- `openclaw doctor` warnings about `MEMORY.md` nearing the per-file limit
- lower-quality long-term memory, because transient daily-note material gets treated as durable knowledge

## Reproduction sketch

1. Enable dreaming / deep promotion.
# 
2. Keep large mixed-content blocks in `memory/YYYY-MM-DD.md`.
3. Let those blocks get recalled enough times.
4. Run the normal dreaming sweep or `openclaw memory promote --apply`.
5. Observe that entire broad line ranges get appended into `MEMORY.md`.

## Notes

A few related observations from docs/CLI behavior:

- deep phase is the only phase that writes to `MEMORY.md`
- scheduled dreaming thresholds appear to be internal, not user-configurable
- `openclaw memory promote --apply` can manually apply promotions, but I couldn’t find a built-in command to auto-prune/rewrite `MEMORY.md`
- `openclaw memory status --fix` / `openclaw doctor --fix` seem to repair artifacts, not curate durable memory content

## Possible improvements
# 

Any of these would help:

1. Promote smaller / more atomic candidates instead of broad line-range blobs
2. Add section-aware exclusion rules for task logs, audit sections, reconstructed summaries, etc.
3. Add configurable scheduled promotion thresholds/policies
4. Add a built-in `MEMORY.md` curation/consolidation pass
5. Add a “preview bad promotions” or “lint durable memory” workflow

## Question

Is this current behavior intended, or is this a bug in promotion granularity / candidate selection?

If you want, I can also make this shorter and more maintainer-ish before you post it.

simple hatch
#

Yes

runic fractalBOT
#

Yep, here’s a tighter version:

## Bug: deep promotion appends broad daily-note blobs into `MEMORY.md`, causing durable-memory bloat

### Version

`openclaw v2026.4.15`

### Summary

`memory-core` deep promotion appears to promote large mixed-content line ranges from `memory/YYYY-MM-DD.md` into `MEMORY.md` as single durable entries.

This bloats `MEMORY.md` and eventually triggers bootstrap-budget warnings from `openclaw doctor`, for example:

- `Bootstrap file size`
# 
- `MEMORY.md is at 94% of its per-file limit`

### Actual behavior

A promoted entry can look like this:

```text
openclaw-memory-promotion:memory:memory/2026-04-01.md:1:18
...
[score=0.821 recalls=13 avg=0.380 source=memory/2026-04-01.md:1-18]

The promoted block contains transient operational content like:

  • pending-task carry-forward
# 
- reconstructed session summary text
- audit output
- timestamps
- commit refs
- one-off fixes

This entire block gets appended to `MEMORY.md` even though it does not look like durable long-term memory.

### Expected behavior

Deep promotion should prefer atomic durable facts, or refuse/distill overly broad daily-note blocks instead of promoting them wholesale.

### Repro

1. Enable dreaming / deep promotion.
# 
2. Keep large mixed-content blocks in `memory/YYYY-MM-DD.md`.
3. Let them accumulate enough recall/signal.
4. Run the normal dreaming sweep or `openclaw memory promote --apply`.
5. Observe large line-range blobs appended to `MEMORY.md`.

### Impact

- low-signal durable memory
- `MEMORY.md` prompt bloat
- bootstrap budget warnings
- manual cleanup required

### Notes

From current docs/CLI behavior:
# 

- deep phase is the only phase that writes to `MEMORY.md`
- scheduled dreaming thresholds appear to be internal, not config-tunable
- I could not find a built-in command to auto-prune/rewrite/synthesize `MEMORY.md` after bad promotions

### Request

Please confirm whether this is expected behavior.

If not, this looks like a promotion-granularity / candidate-selection bug. A fix could be any of:

- smaller candidate units
- section-aware exclusions
- durable-fact extraction instead of whole-block promotion
- built-in durable-memory curation/consolidation