#🔩・node-operators

1 messages · Page 3 of 1

rose solar
#

yoo

dry hamlet
#

Hi, I’m trying to connect my Golang client code to an Injective node. It works for the default mainnet servers but it is difficult to figure out the Network configuration URLs for a local node. Looking for some help, any idea how to get this working?

dapper folio
#

Has anyone got a node running with the indexer that I can use to run pyinjective API calls on?
I can't depend on sentry.exchange.grpc.injective.network:443 as the its constantly down, and can't yet justify running my own node.

old heron
left linden
#

Any injective dev can clarify recommended EBS throughput and IOPS? Seeing massive slow sync speed on our nodes and we're running m6i 2xlarge and nothing being used.

Sync speed is 0 point 7 and causing our nodes to slow down massively.

#

(Auto mod though I was linking..)

#

Would rocksdb improve perf?

shell gorge
#

Hey. I run my own injective node. Everything worked fine but last week my node started lagging behind. As i understand from injectived logs my node cannot connect to peers to be up to date. I retrieve many errors from p2p module. Is anyone encountered the same situation?

left linden
#

I’m the same above, we need a updated peer list

civic kettle
#

Hey all,

I am running injective node and it exposes metrics on I think 10337 but it is in json format , can you please help me on how to use it in Prometheus ?

civic kettle
#

Is anyone using json exporter to get metrics out of 10337/metrics to Prometheus?

I want help in preparing config yml for json exporter, if anyone has it , would be a great help

lucid wigeon
civic kettle
#

Hello Everyone,

I’ve been running an Injective RPC node that was always in sync until recently. Over the past few days, the node has started to lag behind significantly and takes much longer to catch up. My hardware specifications are more than sufficient:

RAM: 128 GB
CPU: 12 cores
Storage: 3TB NVMe
I’ve attached a graph that shows the node’s performance over time—it was consistently in sync before, but the lag has become noticeable now.

Does anyone know what might be causing this sudden delay? Are there any known issues or configuration changes that could be contributing to the problem, and what steps might help resolve it?

Thanks in advance for your insights.

stray hound
# civic kettle Hello Everyone, I’ve been running an Injective RPC node that was always in sync...

hello,

if there is no software issue like resource exhaustion (in case you run other nodes or something on this server), or a hardware problem e.g. NVMe on its way out -- you can check the disks health with smartctl or nvme-cli --, which may cause of lot of iowait

then it may be related to a peering issue, like not enough outbound peers. Maybe you can grab an updated addrbook and see if that helps

also, disabling the index and pruning while the node catches may help make it faster

mild summit
#

Hi Folks,

#

I've built a full Injective Dex stack but after my first snapshot refresh, my event provider is no longer pulling events? Any thoughts? The node is in sync

mild summit
civic kettle
#

Hey, thanks for the responses!

I have a couple of questions:

Addrbook Update:
I was under the impression that the addrbook is automatically created when seed peers are provided and the node starts. How can I obtain an updated addrbook after the node is running?

Catching_up vs. Block Height Difference:
When I check my node's status, it shows catching_up: false. However, I noticed that the latest block height of my node is about 6,000 blocks behind that of a community node. I understood that catching_up: false indicates the node is fully synced and not lagging behind. Could you please clarify why this discrepancy might occur?

I’m running the enterprise edition of NVMe drives and have confirmed that disk health is fine. Any insights would be appreciated!

stray hound
# civic kettle Hey, thanks for the responses! I have a couple of questions: Addrbook Update: ...

sometimes the node does not realize that it is lagging behind , but if you restart it, you should see "catching_up: true" (it's not going to affect anything else though)

Seeds : yes, although sometimes your node won't be able to connect. Personally I prefer having no seeds and no persistent peers, letting the node get them through pex

you can check the number of peers that you are connected to with curl localhost:26657/net_info |jq |grep n_peers -- my validator has 81 right now for example

to get the count of outbound peers (which are the ones you need) : curl localhost:26657/net_info |jq |grep '"is_outbound": true' |wc -l

if you don't have many, then maybe you can grab an updated addrbook e.g. at https://polkachu.com/addrbooks/injective (to apply it you need to restart the node)

civic kettle
stray hound
# civic kettle Hey , Thanks I did no seeds and no persistant peers and letting node get throug...

35 outbound peers isn't bad already, that should be fine

one thing to accelerate the process also, can be to update your config.toml in the [consensus] section : skip_timeout_commit = true

also, disabling logging might help : ̀log_level = "panic"`

next you can install glances and use it to check the current load of the server including the iowait.
or just resync from a snapshot if maybe the data is gigantic

I can't think of further explanations except the bandwidth being limited but I doubt this is the case 🤷‍♂️

mild summit
#

I'm getting the error "process exited with error: failed to process events: failed to get corresponding market for position. marketID" for KAITO/USDT PERP from indexer-exchange-process I did a snapshot refresh this morning after shutting my stack down and only restarted the stack in stages after the node had synced, followed by indexer-eventprovider-process . How can I track down and correct the failing lookup?

mild summit
distant wyvern
#

Hello, I'm trying to build a node archive but when i launch it i got this error error during handshake error on replay wrong app version 0, upgrade handler is missing for v1 12 0 upgrade plan.
If anyone can help it would be great !

lucid wigeon
distant wyvern
lucid wigeon
distant wyvern
civic kettle
# stray hound 35 outbound peers isn't bad already, that should be fine one thing to accelerat...

Hi Thomas,

Thank you very much for your previous help—it’s been truly appreciated.
According to the system details I gathered using Glances, everything appears to be operating normally. However, I’m still experiencing persistent lag issues that sometimes spike up to 50k.
Attaching the glances output screenshot below.
Any insights on potential causes or recommendations to help resolve these lag spikes would be extremely helpful.

Thanks again for your support

dull totem
stray hound
# civic kettle Hi Thomas, Thank you very much for your previous help—it’s been truly appreciat...

hello,

so from a quick look at this screenshot :

  • that cpu you have has a limited single core performance, which is what's important with Injective... see the comparison with the 4464P I'm using on our validator : https://www.cpubenchmark.net/compare/3591vs6073/AMD-EPYC-7402P-vs-AMD-EPYC-4464P -- the single core performance is double
  • it may just be related to the very moment at which you took the screenshot, but the write speed is quite low and perhaps that NVMe isn't super fast -- can you paste the output of nvme list to get an idea of what the hardware actually is ?

(I've come to realize that i/o performance can wildly differ from one manufacturer / model to another, despite what the official specs say)

civic kettle
stray hound
# civic kettle Thanks Thomas, Really appreciate the help on this. Btw I had another question, ...

hello,

the upgrade-info.json file will appear in the data folder when the upgrade time is reached, and cosmovisor will trigger the binary change

there's an official tutorial on how to set it up actually : https://docs.injective.network/nodes/getting-started/running-a-node/cosmovisor

it's fairly easy, the only thing that is out of the ordinary is that you need to take libwasmvm.x86_64.so into consideration as there's usually a new version with each upgrade, and the new binary won't typically work with the previous one.
it's specified in steps 5 and 6 of the guide

civic kettle
stray hound
# civic kettle The new binary, download and keeping it at right place has to be done manually ?...

hem, so
yes, in theory

in the service file, if you set Environment="DAEMON_ALLOW_DOWNLOAD_BINARIES=true", cosmovisor will attempt to download the binary from the path specified in upgrade-info.json, and switch onto it.

2 things :

  • this is generally frowned upon for security reasons (if the upstream github is hacked somehow and a malicious actor distributes an altered binary and forces a state change, you get the idea -- afaik this never, ever happened so risk is low. But still, its preferable to prepare the upgrades manually).
  • for Injective, I'm not sure it can work at all, because of the libwasmvm.x86_64.so requirement (it would have worked for today's upgrade as the lib did not change, but usually it won't)
civic kettle
stray hound
civic kettle
stray hound
# civic kettle because i came to know as my node stopped and i have an alert on that and that i...

I had made a python script to monitor proposals and send discord alerts a while back : https://github.com/HighStakesSwitzerland/Discord_Alerts_Cosmos_Proposals

it's a simplified version of the tool we use internally so you could potentially have it run on your server and set it up to send you messages..

pretty sure there are existing TG bots or something though, maybe more sophisticated and easier to subscribe to

civic kettle
civic kettle
# stray hound I had made a python script to monitor proposals and send discord alerts a while ...

Hi Thomas,

I restarted my node and resumed syncing, and my graph showed that I was closing the gap to the head block. However, it suddenly stalled, and the blocks-per-second rate dropped. Since my CPU, memory, and disk I/O are all normal, I suspect it might be related to peer connectivity. Have you encountered this before, and do you have any recommendations on how to address it? Sometimes my node lags and takes a while to catch up to the tip, so any insights would be much appreciated

stray hound
civic kettle
stray hound
#

oh I see, normal cpu then

peering looks fine too actually, I guess it will increase further over time

civic kettle
stray hound
errant gate
#

I'm trying to build Injective node binary myself, but I'm not able to figure out where I can get the appropriate source from

#

Ok - I'm battling with the spam check here

#

Any relevant info I'm trying to provide gets blocked 😅

dull totem
#

Do not reply to random dms

errant gate
#

Not my first rodeo

#

Let's try a screenshot...

#

Well that went through

lucid wigeon
stray hound
#

@dull totem, I was DM'd by @abrahamtitus who suggested that I join another server to open a "support ticket" 🙄

might be worth kicking him out of here..

errant gate
fading robin
#

Hi, there's no linux-amd64_zip file for the latest Corfu-v1_15_0-beta-1742307638 release

mild summit
#

Who handles the dumping of mongo db to s3://injective-snapshots/mainnet/weekly/mongo and how can we get more recent dumps?

mild summit
#

Or, are there alternative sources of dumps for eventProviderV2 and exchangeV2?

carmine pilot
#

vps req

floral oar
#

Are invariant checks enabled for most validators on mainnet?

light aspen
#

Hello is there a channel for node upgrade announcements ?

stray hound
frank rain
#

Can I please get the validator role? @gusty jetty

tardy hornet
#

Could I please be given the validator role. @gusty jetty Thanks.

loud jetty
frail badger
#

Hi can you please provide with websites offer private injective nodes , similar to quick nodes ‘ only has testnet ?

frail badger
light aspen
stray hound
solid basin
dull totem
solid basin
solid basin
#

Thank you... I'll come back to check 😎

loud jetty
#

Hi, this channel is reserved for validators. If you're a validator please ask someone from your team to make the request to add you in.

strange atlas
#

Hi, is anyone running an injective Injective RPC node with historical data starting from block 107,987,551?

uncut elm
#

peggo question: it looks like peggo requires a alchemy subscription (as it requires alchemy_filteredNewFullPendingTransactions ) on their websocket.
do I need to use the same alchemy endpoint for the RPC node (as I have regular gEth clients running locally, which i'd prefer to use) and just have peggo listening on the alchemy endpoint ?

(is this question better raised as an issue on the peggo GH?)

light aspen
#

Helo @simple olive is it possible to add me to the node annoucement channel, to ensure we can follow upgrade ?

lean whale
#

Anyone can share the more up to date snapshot for archive node ? I an not able to sync from 59693872

last grotto
frail badger
#

Hi there I have question question about running full node non validator is it possible and is it good for trading

zealous seal
#

@sick heron

sick heron
fringe echo
#

Hey I am trying to run injective node can anyone help me with the correct documentation for running it . I just have an empty server

vagrant canopy
#

Hey, is state sync working?

jagged juniper
vagrant canopy
worn juniper
#

Hi, this is Alex from Endorphine Stake.
I was wondering if the Foundation's delegation program is still active, or has it ended?

lilac fjord
worn juniper
smoky yacht
stray hound
smoky yacht
#

Comparison with the main network block height, and I restart at 23:00

stray hound
# smoky yacht I think the resources sould be fine.

not necessarily -- if the cpu has insufficient single core performance it might cause exactly that

do you know the exact make and model of the cpu ? cat /proc/cpuinfo |grep "model name"should give you the info otherwise

you can also check the evolution of the number of peers that you are connected to with grafana, might give some insight if it is correlated to the lag

smoky yacht
stray hound
# smoky yacht and the config:

can't see anything really wrong
your external ip isn't set apparently so you mustn't have any inbound peers but that shouldn't affect the uptime

you have the index enabled in config.toml, as well as the api / grpc / evm-rpc in app.toml -- the node maybe needs to process requests that can be heavy ?

and prometheus = false so you can't actually get the metrics
instead you can run ̀curl 127.0.0.1:26657/net_info |grep n_peers` to check the number of peers you are connected to

smoky yacht
#

ok, the cpu is 8259CL, and the n_peers are 50.

smoky yacht
#

This issue occurs approximately once every 1 to 3 days. 😂

stray hound
stray hound
stray hound
smoky yacht
#

OK, thank you. I will look at the cpu model and prometheus.

stray hound
dry jackal
#

hey is node run incentivise ?

dry jackal
#

nice try to scam

radiant mortar
smoky yacht
# stray hound I had made a guide to set up grafana for Elys, which should be mostly valid stil...

I’m currently observing that I seem to have lost many validators, on this metrics: metric_cometbft_consensus_missing_validators.

This is expected behavior.
,"injective-node","2025-07-30 16:00:00","1"
,"injective-node","2025-07-30 16:01:00","0"
,"injective-node","2025-07-30 16:02:00","0"
...
,"injective-node","2025-07-30 16:59:00","0"
,"injective-node","2025-07-30 17:00:00","0"

This is abnormal:
,"injective-node","2025-07-30 19:00:00","0"
,"injective-node","2025-07-30 19:01:00","0"
,"injective-node","2025-07-30 19:02:00","0"
,"injective-node","2025-07-30 19:03:00","6"
,"injective-node","2025-07-30 19:04:00","19"
,"injective-node","2025-07-30 19:05:00","15"
,"injective-node","2025-07-30 19:06:00","22"
...
,"injective-node","2025-07-30 20:00:00","1"

stray hound
smoky yacht
radiant mortar
#

upgrade in progress? moved to Version v1.16.0 (daf90f4) but nothing happening.

#

don't see that one, lots of this tho.

ERR Failed to send Reset message module=mempool
rose adder
#

chain has stopped ?!

lilac fjord
# rose adder chain has stopped ?!

Nope, the chain has not stopped. We are currently going through a big upgrade which is why txs are not going through right now.

Please wait a bit, everything will be back to normal very soon

radiant mortar
#

appers to be moving -- just out of interest, how come there is nobody commenting here at all during an upgrade?

naive olive
#

wondering the same. I'm just monitoring and waiting

#

I see MintScan is already at block 127,250,787

regal mountain
#

we keep getting this with the new version, tried a post-upgrade snapshot and still getting the same error:

failed to initialize database: L0: 414508: file 414508 (type 2) unknown to the objstorage provider: file does not exist
L0: 615519: file 615519 (type 2) unknown to the objstorage provider: file does not exist
L0: 979602: file 979602 (type 2) unknown to the objstorage provider: file does not exist

Anyone hit this?

worn summit
#

I'm running MM Stack, the injective-core container started correctly and is apparently synced, however the event provider (v1.16.53) crashes continuously with this error, each time it restart it try to start sync from 127250000 block
someone encounters the same issue?

dapper folio
worldly vault
torpid meteor
#

Our mainnet RPC nodes are stuck again since an hour ago. I just wanted to confirm that this is network wide?

sonic sleet
#

@loud jetty hey, Andrea from Crosnest here, could you add me to the validstor channel please?

obsidian depot
#

please add me too 🙏 @loud jetty

obtuse turtle
#

Could you add me too? I am working with the Wormhole team and operating Injective nodes.

smoky yacht
#

please add me too @loud jetty

slender summit
#

can you please add me as well @loud jetty

errant ivy
#

Same for me and @regal mountain @worldly vault please

prisma parcel
#

Can you add me as well pls?
@loud jetty

wise harbor
#

If anyone from Injective is around, may I get added into the validator channel ?

loud jetty
#

Hey guys, please ask your respective teams to make the request so we can add you in

slender summit
radiant mortar
#

@loud jetty pls add me

loud jetty
#

Slack or if they have discord role already that works too

fluid river
#

can i please be added to the validator channel?

hexed oriole
#

How to run node is it free???

zenith frost
#

can you please add me as well

radiant mortar
#

has this thing stalled again?

timid vine
#

@loud jetty hey ! can you add me to the validator channel ?

reef girder
#

@loud jetty Could you add me to the validator channel too?

unique palm
#

@loud jetty Could you add me to the validator channel?

radiant mortar
#

@loud jetty we run nodes for a dapp, need visibility for stalls like this, pls add me to validator channel pepe_pray

atomic thicket
#

@loud jetty Hi, we also run Injective nodes and would like to get informed about network stalls082 Can you pls add me to the validator channel?

idle bay
radiant mortar
pale atlas
radiant mortar
pale atlas
#

If the issue is about node stopping syncing the blocks then just restarting the node should help.

worn juniper
radiant mortar
#

oh dear

quartz tundra
#

Hello, it seems the chain is halted at block 127,692,146, do we have any official communication on this incident?

dull totem
muted dew
#

I am running an INJ mainnet RPC node locally, but today it cannot connect to peers and cannot sync blocks to the local node. Restarting the application does not resolve the issue. The logs are as follows. Could anyone kindly help me with this?

willow brook
#

am i too late to run the node or early. Want some opinions from devs

brisk trellis
warm prairie
pale atlas
#

There aren't any benefits of running a node, though if you run a full node and become an active validator then you can earn through the staking commissions.

odd wraith
#

Hello everyone! I'm trying optimise my validator node to miss less blocks, anybody have some recommedations for tuning config or system for better SLI?

worn juniper
sly tapir
#

Hello everyone,
We’re having trouble getting statesync to complete on an Injective node. Tendermint discovers only way too old snapshots and rejects them with light block not found because public RPCs don’t serve the required heights

We have a tested trust height of "last_height - 2000" to "last_height - 10000"

We have the last version of injectived (1.16.1)

We’ve tested several seeds and rpc (we also tested the list from the GitHub seeds.txt (https://raw.githubusercontent.com/InjectiveLabs/mainnet-config/refs/heads/master/10001/seeds.txt)

Anyone successfully run statesync recently and can share what worked?

Example log :

10:00AM INF Discovered new snapshot format=3 hash=6673E0... height=114589500 module=statesync
10:00AM INF error from light block request from primary, replacing... error="light block not found" height=114589501 module=light primary=http{https://injective-rpc.polkachu.com:443}
10:00AM INF failed to fetch and verify app hash err="light block not found" module=statesync
10:00AM INF Snapshot rejected format=3 hash=6673E0... height=114589500 module=statesync
stray hound
# sly tapir Hello everyone, We’re having trouble getting statesync to complete on an Injecti...

Statesync is complicated with Injective

A surefire solution is to have your own RPC providing the snapshot chunks (and adding it as a persistent peer, with the node id of the statesyncing node as an "unconditional peer" in this RPC node).
Relying on 3rd parties is unlikely to succeed in my experience.

When I need to statesync (e.g. I did that when deploying a new pebbledb node last week), I temporarily activate the snapshots on one of my backup nodes and do it from there.

sly tapir
potent wadi
#

how to run node sir?

#

Scam?

#

scam

winged garden
stuck harbor
#

any body here who has experience in maintaining injective private node? frm bare metal

#

i am having issues with starting indexer container

#

is this a scam?

#

what?

#

yes

idle bay
stuck harbor
#

when i run indexer-eventprovider-process, it causes error of coundnt connect to 26657, where I can access that locally

stuck harbor
#

i just exactly followed the documentaton for the main net set up,

#

can I assume I dont have to make special changes for the working?

warm prairie
#

hi is there any benefit to run a node?

abstract walrus
#

any guide

warm prairie
flat iron
#

pls help me how to run a node

idle bay
junior tundra
#

How I can fix this error. I don't know this error "injectived[4124487]: L3: 013850: file 013850 (type 2) unknown to the objstorage provider: file does not exist" although I had checked I had in ~/.injectived/data/application that I had extract snapshot before. I had checkout the right version and right chain . I run node in testnet

Thanks everyone.

hearty ingot
#

Hi, I restored my node from snapshot and its not failing with the below error
panic: failed to initialize database: L0: 000002: file 000002 (type 2) unknown to the objstorage provider: file does not exist

wheat wagon
carmine scroll
#

Can I run the injective node

lilac fjord
mental parrot
#

Hey I am trying to run a testnet node, and I keep getting this error message: panic: failed to load latest version: failed to load store: wanted to load target 96396676 but only found up to 96396675

Also, is there a place where we can download snapshots for testnet (like older snapshots)?

near urchin
#

How much bandwidth does mainnet node typically consume? I’m not looking for an exact number — I just want to confirm whether it uses an unusually large amount of bandwidth, like Solana does.

simple sparrow
livid latch
#

GM everyone, Joe here from LinkPool 👋
We’ve seen the Injective EVM go live with Chainlink Data Streams this week, impressive launch.
We've ran nodes on it since 2024 and run Chainlink Data Streams for injective and would like to speak to the foundation around validators.
Who’s the best person or channel to speak with about validator onboarding and ecosystem partnerships?
Appreciate any pointers.

livid latch
dull totem
#

@loud jetty @polar vector

coral kite
#

Hello, I'm spinning up a new RPC and see "Finalizing commit of block" and also err="invalid height value"

#

And earliest_block_height is stalled

#

Any Idea about why is it happening?

coral kite
#

Yes

livid latch
real plinth
#

gm! anyone with a snapshot/node around height 141150000 from the last update?

livid latch
#

@loud jetty @polar vector are you there?

hearty nacelle
#

Hi! I'm getting the following error after upgrading to v1 17 1:

error during handshake: error on replay: wrong app version 0, upgrade handler is missing for v1 17 0 upgrade plan

I ran the same upgrade process on two nodes with the same config: one worked fine, the other one is showing the error and failing to start.

hearty nacelle
#

I think I'll need to wait until we reach block #144210000 before upgrading. Not sure why my other node is not showing the same error 🤔

radiant mortar
#

upgrade now

spare dirge
#

Hi! I’m running an Injective node with pruning = "nothing". It works fine, but my disk fills up very fast. How are you handling disk growth?

fiery slate
#

Hi guys the whole network look like its missing lots of blocks.

woven coyote
fiery slate
fiery slate
#

testnet hub injective network/stake i was seeing main nodes missing blocks.

lunar pollen
#

Hello 👋 I am having issues running my Injective Node, maybe someone here could help?

grim meadow
#

We have been running a node since December with pruning disabled "cosmos/bank/v1beta1/supply/by_denom" returns 0 for our node but has the correct value on the public RPC node. Did I miss something in the configuration?

gilded sinew
#

Hi!
Who can I contact from the team to discuss onboarding and providing validator capabilities?

slim tendon
#

is there an upgrade soon ?
iv never done one before, is it hard?
choice has a demo tomorrow lol

#

setting up cosmovisor 🙂

#

when chain release upload?

#

id like to sleep while upgrade happens

blissful scarab
blissful scarab
maiden crescent
#

Hello )
please advise where can I frind the source code for the updcoming harfork ? official repo has latest release of v 1 17 2

slim tendon
slim tendon
hollow wharf
#

Hey guys, could you tell me please how to join #validators channel?

#

Some community members pointed me onto this channel for validators support. Is there any updates on ongoin updgrade process? I see the chain is halted for about 40 minutes

fiery slate
idle bay
hollow wharf
desert wave
#

@idle bay is there a validators channel?

idle bay
desert wave
#

can you get me added please ?

desert wave
#

i believe we are in that channel only, is there no deicated validator channel , asking around the upgrade

idle bay
hollow wharf
fiery slate
idle bay
fiery slate
#

Which version should we be at?

idle bay
fiery slate
sinful heart
#

1.19.0

fiery slate
#

So we wait til the chain gets unhalted?

hollow wharf
hollow wharf
#

@idle bay tell me please, how the upgrade is going?

idle bay
idle bay
hollow wharf
#

@fiery slate here we go, chain seems to be online

fiery slate
idle bay
muted dew
#

panic: failed to load latest version: version of store chainlink mismatch root store's version; expected 164424839 got 0; new stores should be added using StoreUpgrades

gusty jetty
proven ridge
#

hi @loud jetty can you please add me to the validator channel ?

rugged acorn
#

Hi, over the past two days on the testnet, all foundation validators have been missing blocks at a higher rate. Is there an issue?

lunar girder
#

Good afternoon, I wanted to know if you are currently recruiting new validators and do you have a delegation program?

fiery slate
#

Is there anything wrong with mainnet

dull totem