#we should have taken this to a thread o_

1 messages · Page 1 of 1 (latest)

gray cypress
#

so my app is successfully

  • maintaining an adafruit_httpserver.Server instance (listening / **poll(..)**ing for requests
  • three active Websocket connections
  • responding to an incoming REST request
    all at the same time. That ought to be five sockets (one for the lister, three for the websocket sessions, and number five for handling the REST request)
#

I can load a fourth browser page (which should be using socket(s) while handling the response) which then fires up a fourth websocket session. However, any additional requests fail (even a simple REST call), but all four active websocket sessions continue to function properly

#

Also worth noting: these are all HTTP, not HTTPS

#

If I close one of the UI browser pages (which the websocket session detects and the closes it's socket instance) then the REST call will work again

#

(technically that's not exactly what happened - the fifth browser UI page which had failed to load earlier woke up and realized "hey, I can load now" and finished loading, and it's websocket session spun up properly which meant I still had four active websocket sessions. But after closing that page, there were only three so there was finally a "free socket" for new requests)

#

So it seems that 10.0.0-beta.3 on a Lolin S2 Mini can handle more then four, but maybe less than eight. By my count (HTTP listener + four active websocket sessions) that ought to be five. Still not eight though.

eager stirrup
#

I'm was still testing a bit ago, and it seemd that somehow I could make more servers than I thought I should be able to with 4 sockets, but not enough clients (socket is identical though). But try my code above... simply creating 5 stream sockets fails on my S3

gray cypress
#

also wondering if the web workflow might share from the same socket pool. It shouldn't be running in my app - I don't use any CIRCUITPY_WIFI_* settings in settings.toml (it manualy fires up the wifi)

#

which code?

#

hmm, if the server "listener socket" uses something other than an instance from the ...[CONFIG_LWIP_MAX_SOCKETS] tables, then my tests would only have shown a max of four at once (either four active WS sessions or one for handling an HTTP request + three active WS sessions)

#

that might be plausible, if the "listener" is actually something in the C/C++ ESP-IDF TCP stack

eager stirrup
#

web workflow definitely takes from the same pool, I don't have it enabled

#

could be... I always assumed that all sockets came from the same source, constrained by CONFIG_LWIP_MAX_SOCKETS, haven't seen anything break that since I started looking at it with https://github.com/adafruit/circuitpython/pull/6021 but there could be something my narrow uses may have missed

gray cypress
#

on the S3 mini I also get four "live" pages

#

and HTTP requests fail with those four working

#

doesn't seem like this would be it since it's under samples, but circuitpython\ports\espressif\esp-idf\tools\ldgen\samples\sdkconfig contains CONFIG_LWIP_MAX_SOCKETS=4

#

hmm, might help if I synced up with the 10.0.3-beta branch

eager stirrup
#

it's possible to override that in a narrower config file (like a particular board, or chip type like S3, or flash or ram size I think - but I haven't found any overriding definitions)

gray cypress
#

Ok, now I think I'm on the latest ports/espressif code

#

some good news - looks like socketpool is a CircuitPython specific API, so there's not a python "standard library" version we need to mimic

eager stirrup
#

yup

#

it behaves like a subset of socket

gray cypress
#

so adding a simple call like def available_socket_count(self) -> int: ... would hopefully be acceptable

eager stirrup
#

well, it should mimic CPython socket

#

one of the core devs could say for sure

gray cypress
#

and adding an optional 'size' parameter to the SocketPool constructor should also be unobtrusive (of course it can't really do much without switching all the CONFIG_LWIP_MAX_SOCKETS bits away from static arrays)

eager stirrup
#

we are constrained by ESP-IDF

gray cypress
eager stirrup
#

(or pico-sdk in that case)

#

does socketpool have anything that doesn't mimic CPython socket?

#

perhaps the naming is b/c it's such a drastic subset? not sure

gray cypress
#

not that this is necessarily a bad thing - it forces you to be more aware of just how and when you're allocating sockets, which in a resource constrained system makes a lot of sense

eager stirrup
#

you can run CircuitPython socket code on CPython, but not necessarily vice versa

gray cypress
eager stirrup
#

i don't know what that is, it's a 3rd-party thing, has a very different API than CircuitPython / CPython

#

oh, you may have been alluding earlier to available_socket_count in ConnectionManager... still reading to see how it figures that out

#

I think it's a count of only the sockets it's managing... you have to use ConnectionManager to create a new socket to add it to the count, but there could be un-created sockets still potentially available

#

did you get a chance to run the code snippet above? it's the crux of the issue (CP9 vs. CP10)

gray cypress
#

in standard python, "socket" is a module, so you create sockets directly

import socket

# the public network interface
HOST = socket.gethostbyname(socket.gethostname())

# create a raw socket and bind it to the public interface
s = socket.socket(socket.AF_INET, socket.SOCK_RAW, socket.IPPROTO_IP)
s.bind((HOST, 0))
eager stirrup
#

s = socketpool.socket(...

gray cypress
#

there's no intermediate "pool" you have to create first

eager stirrup
#

we don't have infinite sockets like a big computer 🙁

gray cypress
#

yeah, which is why I think adding this additional layer, even though it breaks compatibility with the way you allocate sockets in "normal" CPython code, was a good choice

eager stirrup
#

but yeah, we have to grab from a finite pool that depends on which hardware we're using

#

espressif, raspberrypi, ethernet, or esp32spi

eager stirrup
gray cypress
#

I was just pointing out that since it already diverges from the way "normal" python sockets are allocated, modifying the SocketPool class API shouldn't have any "downsides" as long as it doesn't break existing code, and whatever functionality is added justifies the increase in firmware size

eager stirrup
#

not sure I can answer that one - that's a core dev call

#

there would have to be esp-idf functions to extract that info

gray cypress
eager stirrup
#

right, I was only comparing socket to socketpool

gray cypress
#

and the "apples to oranges" bit is that python "socket.socket(...)" calls a function in a module, where socketpool.SocketPool(...).socket(...) calls a method on a class instance

eager stirrup
# 
>>> s1 = socketpool.SocketPool(wifi.radio).socket(socketpool.SocketPool.AF_INET, socketpool.SocketPool.SOCK_STREAM)
>>> s2 = socketpool.SocketPool(wifi.radio).socket(socketpool.SocketPool.AF_INET, socketpool.SocketPool.SOCK_STREAM)
>>> 
``` 🤪
#

and ConnectionManager has its own magic in that it can manage multiple pools of the same type of radio across multiple radio types... So. Many. Sockets.

gray cypress
#

I'm not terribly surprised that "worked", but (at least according to the docs), it shouldn't work. i.e. there will probably be issues when you try and use both s1 and s2

#

"Only one SocketPool can be created for each radio."

eager stirrup
#

right, but you can have multiple radios of the same and different types

eager stirrup
gray cypress
eager stirrup
#

yeah, it manages ssl-contexts too

#

I also like CM very much... Justin did an awesome job with that one

gray cypress
# eager stirrup CircuitPython with 7 "radios" using Connection Manager <https://gist.github.com/...

ok, I'm actually specifically interested in that... My current project (I've only been working with CircuitPython since March) draws a lot of design ideas from a similar (and much larger) C++ firmware project. One of the things I did there is run a mesh network protocol between each controller "node" which had some very nice advantages. The down side is that something somewhere had to connect to both the mesh (which runs on ESP32 wifi hardware, but is not standard TCP/IP over wifi) and a "traditional" Wifi connection to provide a bridge for normal REST/HTTP/Websockets to interact with the mesh

#

I managed it with by adding a USB wifi adapter to a Raspberry PI, so it had two radios (one for TCP/IP, the other for the mesh)

eager stirrup
# 
import os
import wifi
import socketpool
import ssl

HOST = "example.com"
PATH = "/"
PORT = 443
MAXBUF = 64

wifi.radio.connect(os.getenv('WIFI_SSID'), os.getenv('WIFI_PASSWORD'))
pool = socketpool.SocketPool(wifi.radio)
s1 = socketpool.SocketPool(wifi.radio).socket(socketpool.SocketPool.AF_INET, socketpool.SocketPool.SOCK_STREAM)
s2 = socketpool.SocketPool(wifi.radio).socket(socketpool.SocketPool.AF_INET, socketpool.SocketPool.SOCK_STREAM)

context = ssl.create_default_context()
ss1 = context.wrap_socket(s1, server_hostname=HOST)
ss2 = context.wrap_socket(s2, server_hostname=HOST)

for sock in (ss1, ss2):
    print(f'Connecting to {HOST}:{PORT}')
    sock.connect((HOST, PORT))
    size = sock.send(f"GET {PATH} HTTP/1.1\r\nHost: {HOST}:{PORT}\r\n\r\n".encode())
    buf = bytearray(MAXBUF)
    size = sock.recv_into(buf)  # just get the first hunk and call it a day
    print(f'Received {size} bytes {buf[:size]}')
    sock.close()

code.py output:
Connecting to example.com:443
Received 64 bytes bytearray(b'HTTP/1.1 200 OK\r\nContent-Type: text/html\r\nETag: "84238dfc8092e5d')
Connecting to example.com:443
Received 64 bytes bytearray(b'HTTP/1.1 200 OK\r\nContent-Type: text/html\r\nETag: "84238dfc8092e5d')

Code done running.

🤷‍♂️

#

(I find that I have to try these things b/c about the only thing I believe is running code )

#

Have you looked at ESP-Now? not sure if that helps your model

gray cypress
#

I at least partly agree there, I'm not going to trust it until I've thrown a few rocks

eager stirrup
gray cypress
#

But even if it works, if the docs say it doesn't I remain quite wary that it might not necessarily stay working, or work well in most CircuitPython builds (maybe I just won MCU bingo with the specific controller I'm using today)

eager stirrup
#

lol, very true

gray cypress
#

Espressif only is less than ideal, OTOH I've yet to try that code on anything other than ESP32 family (and originally ESP8266) controllers

eager stirrup
#

I should step away again... this is more interesting, but I need to make more progress with my project in the garage

gray cypress
#

but ESP-Now also has some data size and rate caps that look a bit low for what I'd want

eager stirrup
gray cypress
#

I'm familar with (and have used) quite a few tricks like that, but it would really be a pain with painlessMesh. It's all based on JSON, so it's shipping around variable length packets (which are sometimes hundreds of bytes, and can be more), and it's adhoc peer to peer so it would be quite possible for multiple nodes to "send" close enough in time that the "splits" would end up interleaved.

#

And I know there are ways to "handle" all that, but it's more than a little convenient to just let the TCP/IP stack do it for you.

#

All that said, I'm afraid painlessMesh might not be a great choice for CircuitPython. It probably wouldn't play well with the web workflow, and there are some serious issues with using lots of JSON (in your ongoing "main loop" - fine for initial setup) in CircuitPython

#

@eager stirrup It seems you enjoy, and have spent a lot of time, pushing the edge of CircuitPython networking...

#

I have some "frustrations" related to things I know it could do better, and I even know how to do most of it. But I haven't been using CircuitPython long, and I've only used it on ESP32 variants, and with a limited set of use cases. So a major "disincentive" to exploring those possibilities is that I could easily end up with something that works fine for ESP32s with my use cases but fails elsewhere.

#

I can't commit to working "full time" on it, but if you were willing to work with me on "testing" things, I thinkg we might eventually be able to work some magic.

#

There are (at least) two main areas which could be significantly improved (three if you include select, but that might require firmware mods). The first is building an "async" version of adafruit_httpserver. I think the HTTP part would be easy enough, but I haven't done anything with SSL (or HTTPS) yet, and even if I could "port" that over it would need a lot of testing.

#

The second is providing an extended, CircuitPython specific API layer specifically to deal with garbage collection. Currently, just about anything you do with networking (and definitely with adafruit_httpserver) ends up leaving a trail of allocated litter which the GC system has to collect. There are patterns which can eliminate most of this (potentially all with firmware changes, but...), but it would mean using new/alternate methods to read/write things (basically, you'd always pass in on object which maintains internal "buffers" and can be reused for multiple operations). That would also need testing, and ideally some feedback from someone who's explored far more deeply into the dark corners of CircuitPython networking than I have.

dry coral
#

@eager stirrup please open an issue with a simple example so we can track down when the limit change. Also try on a board with PSRAM if you have not already.

gray cypress
#

The ideal end result would be that apps willing to do the "extra" setup and work to use the async version with the buffer-aware calls could support serving REST and HTML browser requests and running chatty Websocket sessions with zero allocations the GC needs to cleanup

gray cypress
dry coral
#

i am talking about the socket limits

gray cypress
#

oh, that wasn't ... got it

dry coral
#

i didn't read through the whole thread

gray cypress
dry coral
#

we didn't mean to lower the number of sockets

gray cypress
#

I know @eager stirrup has looked, and I've done quite a bit of searching too. AFAIK, neither of us have seen any changes in the CircuitPython codebase which explaing the socket limit change.

eager stirrup
#

GC is a black box to me, but I often chime in on core and library networking issues and PRs so I'd be happy to test PRs. We may have discussed it already, but there has been some discussion of async HTTPServer (must've been Discord, I don't see an open issue), and Requests https://github.com/adafruit/Adafruit_CircuitPython_Requests/issues/134. Full disclosure though: I'm not a seasoned developer in Python (or any other environment).

dry coral
#

i am just looking for an example that can do >4 sockets before some version and now cannot. Not asking for a diagnosis

eager stirrup
#

btw, HTTPS Server only requires a few straightforward additions to code, it's all working in the library.

dry coral
#

i have a vague memory of some issue where you ran into some socket limit a year or two ago but not sure what it was

eager stirrup
#

@dry coral I'll file an issue, it's just a few lines of code CP9 vs. CP10.

#

the older issue was on Pi Pico, closed but those limits remain.

gray cypress
eager stirrup
#

we may luck out and it will be transparent if connect and accept are treated as the granular unit to await upon

#

I mean... Everything is async if the granularity is large enough 😉

gray cypress
#

@eager stirrup you probably wouldn't need to worry about porting it, but I'd definitely need help testing it

gray cypress
#

Whether they work reliably or not (and if so, how they manage it without a socketpool) remains to be seen

eager stirrup
#

oh, I played with those from the asyncio library yesterday, but didn't get very far... got exceptions that didn't seem within my control... someone may know better how to use them

gray cypress
#

You can support "asynchronous behavior" (as opposed to specifically async/await python coroutines) similar to what that feature request describes with a server like that - IIRC there were HTTP server implementations that did so even back in Python 2.x days - but that might require multithreading under the hoods (not an option on CircuitPython)

#

Also, the reason I brought up the possibility of collaborating was not because I assumed you were "seasoned developer", I'd like to think (as much as I can do so objectively) I've got that end covered. What caught my attention were some of the crazy examples you've shared, which strongly indicated some deep curiosity, an interest in "exploring" just for the joy of it, and a fairly comprehensive (if perhaps eclectic) degree of experience with CircuitPython networking. That (well, at least the CircuitPython networking experience - I'm a bit similar on the other two) potentially offsets the biggest missing piece I'd have attempting to do this solo.

eager stirrup
#

I'm always happy to comment on networking issues or review (mostly requirements or testing) PRs where I think I can contribute something (and as time permits).

HTTP servers are written all kinds of ways. 100% optimal may have some matching of client and server, but I had assumed one could be improved (e.g., async) without the other as long as they adhere to the time-worn specs.

gray cypress
#

the catch (ore least one of them) is that with async/await/asyncio, if/whenever you reach a point where you want to "wait" for something to happen, whatever you're (a)waiting on (as well as the function/method with the code asking to wait) also has to be async

#

So if you want an async HTTP response handler, it needs to be using an async socket, which means the HTTP server is running an async listener - can't (or even if you could figure out a hack, you probably would eventually regret it) really mix & match both traditional sync and async in the same server.

eager stirrup
#

if it's just a byte on a socket, does it matter how the byte is delivered?

#

oh... within the same server. I was picturing separate hosts

gray cypress
gray cypress
eager stirrup
#

but yeah, I think I get where you're coming from now... for a given CircuitPython application, you can async some of the things, but will be at the mercy of blocking with any non-async stuff mixed in

eager stirrup
gray cypress
#

This ultimately ties back to the conversation on select. You can do blocking "inside" handling something via async/await - my project has a async "thread" dedicated to the HTTP server. But that "thread" has to keep looping over server.poll() for handling HTTP requests and servicing websocket sessions.

#

With an async server, it could just "sleep" with registered select "listeners" that would wake it up as soon as any new data comes in on any of the websocket session sockets or a new connection appears on the HTTP listener socket.

#

and the select system can quite happily serve multiple servers this way at the same time

eager stirrup
#

conveniently, the io_queues in the asyncio library have select baked in

gray cypress
#

combine that with async/await/asyncio streams and you can build up some pretty serious networking support with multiple servers running effeiciently, with minum latency and none of them having to "run their own loop"

gray cypress
# eager stirrup conveniently, the `io_queue`s in the asyncio library have `select` baked in

AFIAK "io_queue" isn't part of "standard" python asyncio, and it isn't in the CircuitPython asyncio docs. However, if it looks/behaves like asyncio.Queue it certainly would be handy.

Python documentation
Queues

Source code: Lib/asyncio/queues.py asyncio queues are designed to be similar to classes of the queue module. Although asyncio queues are not thread-safe, they are designed to be used specifically i...

#

Not sure if a Queue cah play well with others, i.e. be one of multiple things being waited on. My guess, based on the name and "select baked in" is that io_queues are able to do so nicely

eager stirrup
#

looks like the API is quite different, I don't know the origins of the CP "IOQueue" version (which at least in part comes from MP)

#

(but I don't see IOQueue there)

gray cypress
#

Where are you seeing "io_queue"? If you only found it by looking at the asyncio source, or doing dir(asyncio) at the REPL, that falls into the "kids, don't try this at home" category. Not that this should stop you from exploring....

eager stirrup
gray cypress
#

ok, found it in asyncio/copr.py

#

that looks like something meant to be an "internal implementation detail"

#

it's not exported, and it appears to be a "helper" for the Loop class.

eager stirrup
#

yeah, it's not exposed in the API

gray cypress
#

Most importantly, it uses select as an implementation detail, it doesn't provide a way for an IOQueue to coordinate with other things happening in a external select instance - that's where the real scalability starts

eager stirrup
#

no, I take that back, it is exposed

gray cypress
#

That said, doesn't look like it would be too hard to "fix" that

eager stirrup
#

I'd suggest opening an issue for discussion on the asyncio library repo, there are folks with history with this stuff, in the library and in the core (tons of PRs in the core to enable asyncio)

gray cypress
#

FWIW, my definition of "exposed" in this context is that it's listed in the docs, or at least included in the __all__=["foo","bar"] section. Pretty much everything in python is "exposed" in the sense that you can import / access it if you're stubborn and persistent enough....

eager stirrup
#

I haven't even fully grokked using the asyncio library, let alone strategizing about requirements changes (that may also need core changes)

eager stirrup
#

I picked it up b/c I saw it in a PR and wanted to try it out

#

I haven't even fully grokked using the asyncio library (yet?), let alone strategizing about requirements changes (that may also need core changes), so an issue is the best path forward

gray cypress
#

little tidbit you might not know, anything name starting with a single underscore in python is considered to be "private" by convention - as in it's a "gentleman's agreement" but Python itself doesn't care

eager stirrup
#

right, that's caried over into CircuitPython, you'll see admonitions about using private methoids (subject to change without notice, and all that)

#

CircuitPython community is pretty good about adhering to CPython conventions and resource-savvy subsets for APIs

gray cypress
#

However, names starting with a double underscore (sometimes callled dunders) are recognized by CPython and "privatized". Especially for classes - when you use self.__foo inside a class method, it works, but if you try instance.__foo from the outside it won't. That's because the actual member name will have an additional prefix, and CPython will magically add that prefix for you when you use self.__foo - but only in the class methods

#

CircuitPython, however, doesn't support this. Which occasionally causes some incompatibilty between CircuitPython and CPython code (but only when, for example, class B inherits from class A and they both use __foo - with CPython methods in A and methods in B would each see their own class-specific __foo, but in CircuitPython there is only one __foo

eager stirrup
#

had not encountered that limitation yet