#Openclaw Internet Problem

54 messages · Page 1 of 1 (latest)

dull bone
#

Hi @timid hill, i have question. Currently my setup of my openclaw, searxng, and internet are like:

  1. Current Architecture
    The system consists of three components:
    OpenClaw
    LLM reasoning is working correctly
    The model understands tool usage instructions
    However, it does NOT have a working native tool execution layer in the current setup Node.js Tool Server
    Node.js
    Custom HTTP tool server implemented
    /search endpoint integrated with SearXNG
    /tools/search_web endpoint working correctly
    Search Backend
    SearXNG
    Fully functional
    Returns valid search results

  2. Core Problem

OpenClaw currently lacks a proper tool execution layer inside the agent loop.
Although the LLM is capable of reasoning about tool usage, it cannot actually execute external HTTP tools.
As a result, tool calls are never executed and the model falls back to generating responses from its internal knowledge.

  1. Expected vs Actual Behavior
    Expected (Agent Loop Design):
    LLM → Tool selection → Tool execution → Tool result → Final answer
    Actual behavior:
    LLM → Reasoning → Final answer (no tool execution)

  2. Impact
    The model claims it will use tools, but cannot execute them
    No real-time web data retrieval happens
    Responses are generated purely from model memory

  3. Root Cause
    The OpenClaw setup is currently functioning as:
    A language model runtime only, not a full agent runtime with tool execution capabilities.

pine skiffBOT
cunning beaconBOT
#

To help others find answers, you can mark your question as solved via Right click solution message -> Apps -> âś… Mark Solution

pine skiffBOT
timid hillBOT
#

Your root cause is slightly off. OpenClaw does have a real agent loop with tool execution: model → tool call → execution → result → final answer. Docs: https://docs.openclaw.ai/concepts/agent-loop

What’s usually broken here is one of these:

  1. Your custom Node HTTP server is not registered as an OpenClaw tool. OpenClaw does not auto-execute arbitrary /search or /tools/search_web endpoints.
  2. web_search is not enabled/allowlisted. If you use allowlists, add web_search or group:web.
  3. You added an extra layer you probably do not need. OpenClaw can talk to SearXNG directly as its web_search provider: https://docs.openclaw.ai/tools/searxng-search and https://docs.openclaw.ai/tools/web
  4. The model/provider is weak at tool calling, especially some local or OpenAI-compatible setups.

If your goal is just SearXNG search, the simplest config is:

{
  tools: {
    web: {
      search: {
# 
        enabled: true,
        provider: "searxng",
      },
    },
    allow: ["group:web"], // only if you use allowlists
  },
  plugins: {
    entries: {
      searxng: {
        config: {
          webSearch: {
            baseUrl: "http://localhost:8888",
          },
        },
      },
# 
    },
  },
}

If you want to keep the custom Node server, make it an MCP server or a plugin tool. Plain HTTP endpoints alone are not enough. MCP docs: https://docs.openclaw.ai/cli/mcp

Please paste these next:

  • openclaw --version
  • openclaw status --all
  • openclaw models status

and the relevant config for:

  • tools
#
  • plugins.entries.searxng
  • mcp.servers (if you use MCP)

Then I can tell you exactly which piece is missing.

dull bone
#

My SearXNG instance is working correctly and returns valid results when tested directly.
However, when used inside OpenClaw’s web search tool, it appears to be restricted or not executed.

What could cause this restriction in OpenClaw’s tool execution layer?

Specifically:

Is the SearXNG provider being blocked by tool allowlists (e.g. group:web)?
Could this be caused by missing tool permissions or disabled web.search configuration?
Or is the issue related to the model’s tool-calling capability rather than the search backend itself?

What are the exact configuration requirements for OpenClaw to successfully execute SearXNG search tools?

timid hillBOT
#

Yes, all three can cause it, but for SearXNG the important detail is:

SearXNG is not the tool. web_search is the tool.
SearXNG is just the provider behind web_search.

So the exact requirements are:

  1. web_search must be allowed

    • tools.profile must include web tools (coding or full)
    • if you use allowlists, allow web_search or group:web
    • provider/agent rules can also block it:
      • tools.byProvider.*
      • agents.list[].tools.*
      • agents.list[].tools.byProvider.*

  2. Web search must be enabled

    • tools.web.search.enabled: true
#

  1. SearXNG must be configured as the provider

    • either:
      • tools.web.search.provider: "searxng"
    • and/or:
      • SEARXNG_BASE_URL=...
      • plugins.entries.searxng.config.webSearch.baseUrl: ...

  2. The SearXNG URL must pass validation

    • http:// is only okay for loopback/private hosts
    • public SearXNG endpoints must use https://
    • SearXNG must have JSON format enabled

  3. The model must actually call tools

    • if no web_search call is emitted at all, that is usually a model/provider issue
    • smaller/local OpenAI-compatible models are the most common offenders

Minimal working shape:

# 
{
  tools: {
    profile: "coding", // or allow: ["group:web"]
    web: {
      search: {
        enabled: true,
        provider: "searxng",
      },
    },
  },
  plugins: {
    entries: {
      searxng: {
        config: {
          webSearch: {
# 
            baseUrl: "http://localhost:8888",
          },
        },
      },
    },
  },
}

Two super common mistakes:

  • allowing searxng, web.search, or search_web instead of web_search
  • SearXNG works directly, but the model never calls the tool, so it looks like “OpenClaw blocked it” when it’s actually a tool-calling/model issue

Docs:

dull bone
#

"tools": {
"profile": "coding",
"web": {
"search": {
"provider": "searxng",
"enabled": true,
"openaiCodex": {}
},
"fetch": {
"enabled": true
}
},
},
"plugins": {
"entries": {
"browser":{
"enabled": true
},
"searxng": {
"enabled": true,
"config": {
"webSearch": {
"baseUrl": "http://localhost:8888"
}
}
}
}
}

timid hillBOT
#

That config looks basically correct.

So with just this snippet, the problem is probably not:

  • tools.profile being too restrictive, because coding already includes group:web
  • tools.web.search.enabled, because you set it to true
  • missing SearXNG provider config, because provider: "searxng" + baseUrl is present

That narrows it down to the real suspects:

  1. localhost:8888 is wrong from the Gateway’s point of view

    • If OpenClaw runs in Docker, a VM, or another machine, localhost points to that runtime, not your host
    • This is the most common gotcha
    • In Docker, you usually want the service name or another reachable network address, not localhost

  2. SearXNG JSON API is not enabled

    • OpenClaw needs SearXNG’s JSON endpoint, not just the HTML search page
#
  • Test this from the same machine/container where OpenClaw runs:
    curl 'http://localhost:8888/search?q=test&format=json'
    • If that fails or returns non-JSON, OpenClaw cannot use it

  1. The model is not actually calling web_search

    • Your config can be perfect, and the model can still choose not to call the tool
    • This is especially common with smaller/local OpenAI-compatible models

  2. There is another override elsewhere

    • agents.list[].tools.*
    • tools.byProvider.*
    • agents.list[].tools.byProvider.*

Also, one important point: your custom Node tool server is not part of this path.
With this config, OpenClaw should use:

#

web_search → SearXNG directly

So the Node /tools/search_web layer is irrelevant unless you wired it in separately as MCP/plugin.

What I’d check next:

  • openclaw --version
  • openclaw status --all
  • openclaw logs --follow

Then ask OpenClaw: “search the web for X”

Interpretation:

  • No web_search activity in logs → model/tool-calling issue
  • web_search appears, then connection/refused/error → SearXNG reachability / URL / JSON issue
  • tool blocked/unavailable warning → hidden policy override
dull bone
#

the code is fine, the problem is when openclaw trying to use web_fetch, it is blocked by the site security from cloudflare. So the data is not presentable and less accurate

timid hillBOT
#

Yep, that makes sense, and that’s a different issue than SearXNG/tool allowlists.

web_search and web_fetch are separate tools.

  • web_search with SearXNG = gets search results/snippets
  • web_fetch = tries to fetch the actual page content

So if the destination site is behind Cloudflare / anti-bot, then:

  • SearXNG can still return a result
  • but web_fetch can fail or return junk/blocked HTML
  • which makes the final answer look weak or inaccurate

That is not OpenClaw blocking the tool. It is the target site blocking plain HTTP fetches.

Docs confirm this:

  • web_fetch is a plain HTTP GET and does not execute JavaScript
#

Minimal Firecrawl fallback setup:

{
  tools: {
    web: {
      fetch: {
        provider: "firecrawl",
      },
    },
  },
  plugins: {
    entries: {
      firecrawl: {
        enabled: true,
        config: {
# 
          webFetch: {
            apiKey: "FIRECRAWL_API_KEY_HERE",
          },
        },
      },
    },
  },
}

Important takeaway

Your SearXNG config can be correct, and OpenClaw tool execution can also be correct, while page extraction still fails because Cloudflare blocks web_fetch.

So the real diagnosis now is:

#
  • search works
  • fetch is blocked by site security
  • use browser or Firecrawl, not plain web_fetch, for those sites

If you want, I can help you choose the better route between:

  • browser-first
  • Firecrawl fallback
  • or disable web_fetch and rely on search snippets only
dull bone
#

i would choose browser-first. How to configure it?

timid hillBOT
#

Yep, browser-first is the right move here.

Your current setup needs one important change first:

tools.profile: "coding" does not include the browser tool.
So you must explicitly allow it.

Minimal browser-first config

{
  tools: {
    profile: "coding",
    allow: ["browser"], // important: coding does not include browser
    web: {
      search: {
# 
        provider: "searxng",
        enabled: true,
      },
      fetch: {
        enabled: true,
      },
    },
  },

  plugins: {
    entries: {
      browser: {
        enabled: true,
      },
      searxng: {
# 
        enabled: true,
        config: {
          webSearch: {
            baseUrl: "http://localhost:8888",
          },
        },
      },
    },
    // if you use plugins.allow, include "browser" there too
  },

  browser: {
    enabled: true,
    defaultProfile: "openclaw",
    headless: false
# 
  }
}

Then restart the Gateway

After changing config:

openclaw gateway restart

Quick smoke test

# 
openclaw browser --browser-profile openclaw status
openclaw browser --browser-profile openclaw start
openclaw browser --browser-profile openclaw open https://example.com
openclaw browser --browser-profile openclaw snapshot

If that works, the agent can use the browser tool.

Recommendation

Start with the managed profile:

  • defaultProfile: "openclaw"
  • headless: false

That is the easiest browser-first setup and usually better for Cloudflare debugging.

#

If a site still needs your real logged-in browser

Later, you can switch to profile="user" / existing-session mode, but I would not start there unless you specifically need cookies/login state.

Docs:

If you want, next I can give you the exact “browser-first + searxng” full config based on the snippet you already posted, so you can paste it directly.

dull bone
#

is this correct?

"plugins": {
"entries": {
"browser":{
"enabled": true
},
"searxng": {
"enabled": true,
"Profile": "openclaw",
"headless": false,
"config": {
"webSearch": {
"baseUrl": "http://localhost:8888"
}
}
}
}
}

timid hillBOT
#

Not quite.

"Profile": "openclaw" and "headless": false do not belong under plugins.entries.searxng.

Those are browser settings, so they belong under top-level browser.

Also, Profile should not be capitalized.

Use this shape instead:

{
  tools: {
    profile: "coding",
    allow: ["browser"],
    web: {
# 
      search: {
        provider: "searxng",
        enabled: true,
      },
      fetch: {
        enabled: true,
      },
    },
  },

  plugins: {
    entries: {
      browser: {
        enabled: true,
      },
# 
      searxng: {
        enabled: true,
        config: {
          webSearch: {
            baseUrl: "http://localhost:8888",
          },
        },
      },
    },
  },

  browser: {
    enabled: true,
    defaultProfile: "openclaw",
    headless: false,
# 
  },
}

Key fix

Wrong:

plugins.entries.searxng.Profile
plugins.entries.searxng.headless

Right:

# 
browser.defaultProfile
browser.headless

Then restart:

openclaw gateway restart

Docs: https://docs.openclaw.ai/tools/browser

If you want, I can format your full final config from what you’ve posted so far, ready to paste.

dull bone
#

the code is fine now, so to reset the openclaw brain, i need to give him commands like : "Reset, from now if you are asked to go searching online, use browser plugins first". Is that alright?

timid hillBOT
#

You can say that in chat, but I would not rely on it as the main fix.

Better approach:

  1. Put the rule in AGENTS.md so it is persistent
  2. Then do /reset (or /new) to start a fresh session if you want a clean slate

Why: OpenClaw rebuilds its system prompt from workspace files like AGENTS.md, so that is the right place for long-term behavior. Docs: https://docs.openclaw.ai/concepts/system-prompt and https://docs.openclaw.ai/reference/templates/AGENTS

Better wording

Instead of:

"use browser plugins first"

I’d write something like:

# 
## Web browsing preference

When asked to search online, use `web_search` for discovery.
For reading page contents, prefer the `browser` tool first on public websites, especially Cloudflare-protected or JS-heavy sites.
Use `web_fetch` only when the page is directly fetchable and readable.

That is better because:

  • web_search is still good for finding links
  • browser is better for protected pages
  • web_fetch is still useful when a page is simple

Short answer

  • Yes, you can use /reset plus a chat instruction for testing
#
  • But for permanent behavior, put it in AGENTS.md

If you want, I can help you write the exact AGENTS.md block for your setup.

dull bone
#

now, another issue, how can we get real-time access to API in every website? when i asked openclaw to get me about today's weather report. It cant doit, because no access for API real time

timid hillBOT
#

Short answer: you can’t give OpenClaw real-time API access to every website automatically.

A website is not the same as an API.

OpenClaw has 3 different paths:

  1. web_search
    Finds public search results

  2. browser / web_fetch
    Reads website pages

  3. API integration
    Only works when you add a real tool/integration for that API, usually via:

    • MCP
    • plugin
    • your own custom tool wrapper
#

So for weather, scraping websites is usually the wrong path.
The better path is a dedicated weather API/tool.

Best practice for weather

Use a real weather source, for example:

and expose it to OpenClaw as a tool.

Since you already have a Node tool server

The cleanest setup is:

#
  • keep SearXNG for web_search
  • keep browser for website reading
  • add a dedicated weather tool
  • back that tool with a real weather API

But important: OpenClaw still will not magically call raw HTTP endpoints unless they are wired in as MCP/plugin/tool integration.

Recommended architecture

  • Search → web_search + SearXNG
  • Protected websites → browser
  • Structured live data like weather, stock, exchange rate, news API → MCP/plugin/custom tool

Docs:

#

So the answer is basically:

  • No, not “API access in every website”
  • Yes, real-time data is possible, but only by adding the right tool for that data source

If you want, I can show you the simplest weather setup next, probably:

  1. MCP weather tool, or
  2. your Node server + Open-Meteo wrapped properly for OpenClaw.
dull bone
#

Ok, its fixed. But @timid hill I want to ask again, in my chat, when I send : “Find today’s gold price.” Then the reply is “Searching…”. After that, it becomes inactive. Then after I send another chat, “Is the result ready?”, only then it responds, even though the result was already available an hour ago. How do I configure it so that it responds automatically immediately without needing to be triggered again?

timid hillBOT
#

I’m checking the exact docs path for the delayed-reply behavior, because this sounds more like follow-up delivery/streaming than search itself.

#

The docs moved a bit in this checkout, so I’m using the current help/faq.md path instead of the old start/faq.md.

#

Yes, this is usually a delivery context problem, not a web_search problem.

What is happening:

  • "Searching..." is just the preview/progress message
  • the async completion wakes the same session
  • but the run has no valid external Discord target, so the followup stays session-only
  • then your next message reopens that session, and only then you see the result

Most important detail

If you are sending turns through POST /v1/chat/completions, that path currently runs with deliver: false. So later async followups do not auto-push back to Discord.

So the fix is:

Option A, best

If the user is chatting in Discord, let the turn originate from Discord directly.

#

Do not proxy that Discord message through /v1/chat/completions if you expect OpenClaw to push the later result back into Discord automatically.

Option B, if you call the agent yourself

Call the agent with explicit outbound delivery metadata on the first turn:

{
  message: "Find today’s gold price.",
  sessionKey: "your-session-key",
  deliver: true,
  channel: "discord",
  to: "your-discord-target-id",
  threadId: "your-thread-id" // if this is a Discord thread/forum post
}

While debugging, keep:

# 
{
  bestEffortDeliver: false
}

That way it fails loudly instead of silently downgrading to session-only.

Option C, if your Node.js app finishes the search outside the agent turn

When the result is ready, explicitly send it to Discord with message.send / message tool, instead of waiting for a session wake.

What is not the main fix

  • /queue is probably not the issue here
  • channels.discord.streaming is also not the issue, it only controls the preview UX

Please check these next

#

First, send:

openclaw --version
openclaw status --all
openclaw logs --follow

Then reproduce once, and look for log lines like:

  • agent delivery downgraded to session-only
  • no deliverable channel

If you paste those logs, I can tell you exactly which field is missing.

Docs: