Newbie question about (unbounded) channels

[true pelican]

I'm making an importer for my own project, where I work with a lot of biology related API's that are all pretty badly optimized. Most endpoints requests can take over a second, I want to optimize this by creating queue's for each source I'm using to fetch data.

So I have a request that fetches data relating the tree of life (the skeleton of the project) that will return the name, the type of node it is: MRCA / Normal (this is domain specific terminology) and children.

Based on name I want to send this through a channel to another importer client that will retrieve context related to this name. So this node with name: Gorilla gorilla gorilla, will return a context descriptor: mammalia . After fetching this context descriptor I making a decision which external sources are interesting to call: wikipedia, encyclopedia of life, silva (for DNA related records), etc. So I was think to make channels for each of these clients as well.

I was thinking of making all channels unbounded because they shouldn't really care if they are behind or not I don't need to block other importers until the current one is done for my use case. But I read that this can run out of memory? I'm not sure how, it should only take up 1 task each time anyway right, and after finishing then it looks for the next one?

So my questions are:
Do I have to bound these channels?
If so what would be a reasonable amount?
Why can an unbounded channel run out of memory when only 1 task is processed each time?

[sour mulch]

The concern is that you can enqueue entries in the queue faster than you consume them, and if you let this run indefinitely then that queue will grow indefinitely

If you know for a fact that this import will not run indefinitely, then you can adjust accordingly. It might be better to split this processsing up though and have a fast ingest pipeline that imports to disk/database, then a separate pipeline that consumes from the disk/database for processing.

[true pelican]

Ok so the concern is basically the list inside that can hold too many items? Why are they often limited to 64 or 100? And not to something like 100_000 or whatever that still shouldn't really be a lot of memory?

I will create an import_task table do i still need channels, or can I just run a loop in a thread with a timeout and then try to read from db? Or are the channels still useful

[sour mulch]

For typical uses of channels, you try to balance input and output flow rates. Rates are never consistent, so some buffering helps smooth out spikes. Sometimes more inputs arrive, sometimes output processing is faster. And sometimes output processing can be batched using recg_many.

If the input rate is consistently faster than the output rate, buffering 200 items gives no more throughput benefit over buffering 100 items, but it does make your processing latencies increase (related to the concept of buffer bloat).

If processing latency doesn't matter, increasing the buffer size is fine. But you do need to eventually balance the input rate vs the output rate.

When you observe a channel is full, you can decide to handle that in a few ways. You can wait (which effectively just increases the buffer size by how many concurrent tasks you can handle). Or you can error. This might not seem ideal but it's useful to acknowledge that an input entry might expire before it's actually processed. And you can feed the error back up the chain and quickly tell the client to come back later, or the client can confirm they are happy to wait a little longer.

[true pelican]

I guess I will use both channels and a JSON file which I already have implemented for the main loop. The other channels will be automatically filled by the main loop, claude says my code is already throttled so I guess unbounded is not really at risk here:

 Channels vs DB: Use channels. The DB queue only buys you crash recovery for enrichment tasks — but you already have
  tol_queue.json for the main tree walk. If the process crashes, you re-run enrichment for the resumed nodes. Channels
  are simpler and sufficient here.

  Bounded vs unbounded: Your main loop is already naturally throttled — it does a upsert_edits + tx.commit() before it
  queues the next enrichment task. The producer can't race ahead faster than your DB writes allow. An unbounded channel
  won't flood memory here.

  So the straightforward design is:

  let (tx, rx) = tokio::sync::mpsc::unbounded_channel::<ClientTask>();

  // spawn one worker per source
  tokio::spawn(pbdb_worker(rx_pbdb, pool.clone()));
  tokio::spawn(gbif_worker(rx_gbif, pool.clone()));

  // in the main loop, after the DB write:
  tx_pbdb.send(ClientTask { db_row_id, lookup: LookupKind::ByName(name) })?;

  Each source gets its own channel/worker, the main loop just fires and forgets, and the workers drain at their own pace.
   No mutex, no Arc<VecDeque>, no DB table needed.  

I will try to write the code myself of course for learning purposes, I think the AI's assumptions are correct here in that it can't overflow (easily) because the producer is limited by itself. It seems the risk with memory here has more to do with over flooding the database with requests than the internals of the channels. I will still limit the channel but to something like 100_000 because there is no point in letting 1 importer run that far ahead. the program still has to wait for all the importers anyway.

Thanks for taking the time and detailed explanation 👍 .

[round marlin]

If you send a million messages and don't receive them, they have to be stored in memory

[true pelican]

Yes I know so maybe 100_000 is a good number. I have 3 fields 2 f64's and 1 bool. So it will take up a few mb's only. Even with a couple extra importers it shouldn't be too bad