#circuitpython-dev
manic glacierBOT
mortal kernel
@tulip sleet I walking up on the error now with JTAG. It's happening very early in esp-idf initialization, somewhere in newlib STDIN, STDOUT, STDERR console initialization, or soon after.
slender iron
https://github.com/adafruit/circuitpython/pull/10277 is ready for review. It improves audio playback on rp2 and prevents playing random memory
slender iron
It's internal: https://github.com/adafruit/circuitpython/blob/443f829262c7f2083779187b9b70e47b7b714419/ports/raspberrypi/common-hal/picodvi/Framebuffer_RP2350.c#L65 Setting that to 71 will lead to 70hz refresh rate
@lone axle do you have a version of pathlib to use with the launcher?
mortal kernel
@tulip sleet The failure is occurring during an attempt to fopen() STDERR. This is where things get a little strange. The reason this code runs at all is because CP is taking the default CONFIG_VFS_SUPPORT_IO=y. I'm thinking that's possibly not correct.
lone axle
<@382939733107408897> do you have a version of pathlib to use with the launcher?
yes, here it is. fwiw I think that some other parts of this that aren't used by the launcher wouldn't work. glob() in particular I think is expecting different behavior from os.listdir(), perhaps other parts too.
Do you think we whould make an adafruit_pathlib library from this?
slender iron
yes, here it is. fwiw I think that some other parts of this that aren't used by ...
Yup, we should make our own
tiny peak
It's internal: <https://github.com/adafruit/circuitpython/blob/443f829262c7f2083...
ah... okay. I just got my rev B board today. Was kinda planning to just roll with 320x240 or 640x480 and do some Playground guide projects using IR and I2C peripherals. If you want me to try a custom uf2 build to see what happens against my capture card, I could do that. 640x480 is good enough for me though.
slender iron
I'm curious to see if it works
tiny peak
ok sure. if you've got a build you want me to try, I can do that. I'll go find the capture card.
slender iron
lemme build one
devout jolt
CircuitPython ReadTheDocs question: is there a way to link to the expanded "Available on these boards" list? Alternatively, is there another place I can link to that has this list for any particular core module?
lone axle
CircuitPython ReadTheDocs question: is there a way to link to the expanded "Avai...
it's a different page and the format is a lot more expanded, so perhaps less convenient to look/scroll through, but the support matrix allows you to pass a filter URL parameter i.e. https://docs.circuitpython.org/en/latest/shared-bindings/support_matrix.html?filter=tilepalettemapper then the first column is a list of devices that support that module.
devout jolt
it's a different page and the format is a lot more expanded, so perhaps less con...
that may work, thanks!
tulip sleet
anchor tags could be added to https://github.com/adafruit/circuitpython/blob/main/docs/autoapi/templates/python/module.rst
jaunty juniper
yes, here it is. fwiw I think that some other parts of this that aren't used by ...
note that we have micropython's ilistdir in storage.VfsFat
It's poorly documented in CP, but I believe it should be the same as
https://docs.micropython.org/en/latest/library/os.html#os.ilistdir
(it returns an iterator with additional data)
slender iron
ok sure. if you've got a build you want me to try, I can do that. I'll go find t...
working on getting a tested build for you
slender iron
fixing a bug with something I just merged
https://github.com/adafruit/circuitpython/pull/10263 is ready for review
slender iron
@tiny peak fj rev b
with extended back porch to slow refresh rate of 720x400 to 70hz
tiny peak
<@1220648290941669377> fj rev b
capture card saw the signal but displays its "Not Supported" message.
slender iron
which card is it?
tiny peak
AverMedia GC551G2
same one that I dumped the EDID info for in comments at https://github.com/adafruit/circuitpython/issues/10242
based on what I've read in forums and seen in reviews, this card doesn't do particularly well at handling older CRT style video modes. But, it is readily available, has HDMI passthrough, and isn't super expensive.
slender iron
it does seem like the xr1 lite is discontinued
tiny peak
also, the latency is pretty low and compression doesn't introduce too many artifacts (unlike some of the cheapest cards)
slender iron
I don't think it is worth fixing unfortunately
we'll want our demos to work at both 720x400 and 640x480
tiny peak
yeah, 640x480 at 60Hz is fine
as long as it's actually 60 Hz instead of something drastically faster
might be worth putting a caution box in the docs about how TVs are generally pretty tolerant of video modes but capture cards can be more picky.
oh, actually, I'm gonna go test your firmware build on my old Samsung TV to see if it likes it. I'm expecting it probably will
tiny peak
@slender iron works great on the old Samsung TV.
slender iron
<@252717193496756235> works great on the old Samsung TV.
That's 640x480. Are you forcing it? Is it a 4:3 TV?
tiny peak
no, that's just how it came up when I plugged it in and turned it on
tiny peak
yeah... I mean, it looks great
slender iron
@tulip sleet this is ready for review too https://github.com/adafruit/circuitpython/pull/10263 I'll need the size changes for the gc PR
tiny peak
one thing about that TV though is it will take a fairly wide range of refresh rates and bin them as "60Hz", "70Hz", "72Hz" or whatever
it's a lot more forgiving than the capture card
slender iron
I'll look at the wifi power PR now
tulip sleet
thaks
devout jolt
anchor tags could be added to <https://github.com/adafruit/circuitpython/blob/ma...
That's a great idea. I just tried the simple approach (adding an id attribute to summary or the ul) and it doesn't work. Appears that expanding a <details> block would require some JS.
tulip sleet
That's a great idea. I just tried the simple approach (adding an `id` attribute ...
does it jump to the right place but not expand?
lone axle
note that we have micropython's `ilistdir` in [storage.VfsFat](https://docs.circ...
Ooh, nice. Thank you, I did not know about that.
jaunty juniper
Ooh, nice. Thank you, I did not know about that.
I found that in my files, I don't use it but I remember needing ilistdir() once to go through a (very) long list of files on an SD card on a small board where listdir() wouldn't fit in memory
full plume
@mortal kernel I ended up checking out the source from https://github.com/eightycc/circuitpython/tree/issue-10191 to test with ESP-IDF 5.4.1. The loliin_s2_mini firmware works fine so far, but so far I haven't been able to get the lolin_s3_mini to work. I restart in uf2 mode (getting LOLIN3MBOOT instead of CIRCUITPY), copy in firmware.uf2, and after reset neither CIRCUITPY nor the USB REPL are accessible. I went back and built lolin_s3_mini from the main source (with ESP-IDF 5.3) and it starts up fine after copying in that frrmware.uf2. FWIW I'm testing with actual Lolin/Wemos boards from the Wemos online shop. Somewhere around here I have an ancient USB JTAG adapter that I managed to get working with an ESP32 C++ build a while back, but I haven't tried setting it up for CircuitPython yet (and I'm not even sure the S3 Mini makes all the pins available).
mortal kernel
@tulip sleet I've tracked down the component causing the double fault. It's ESP-IDF's vfs, which as far as I can tell CP does not use and never registers. But, it's getting included in the build and as part of its initialization ESP-IDF attempts a call to fopen SYSERR on /dev/console. This attempt results in a double fault. By commenting out the fopen, CP successfully starts. Unless there's a reason for vfs to be included that I'm missing, stripping it out will clear up this problem.
tulip sleet
it was included in the 5.3.2 builds, though, right, and didn't crash there
btw I am working on the TLS issue right now. Not so clear it is cert format change. That change was minor. But I have good logging in mbedtls in both 5.3.2 and 5.4.1 and will compare the logs
mortal kernel
it was included in the 5.3.2 builds, though, right, and didn't crash there
Uninitialized memory can cause build-dependent failures.
tulip sleet
I can believe that. I think I I saw some mysterious stuff about this in NINA-FW, which I just removed as not needed.
https://github.com/adafruit/nina-fw/blob/dfc0c299719891002a8b99c187c6532f2d1ce8f9/main/sketch.ino.cpp#L64 all this stuff about stdin stdout and stderr
maybe it's unrelated but it soudned familiar
mortal kernel
See https://docs.espressif.com/projects/esp-idf/en/stable/esp32/api-reference/storage/vfs.html for a description of the vfs component. I'm thinking CP never needed it, but it got included by default.
tulip sleet
if it builds without it great, great sleuthing!
you still get debug logs without it, hopefully
full plume
Uninitialized memory can cause build-dependent failures.
resolving static initialization order ambiguity for C++ startup (pre-main) code can be ... challenging
mortal kernel
I've not yet tested with it completely stripped. That's my next step.
mortal kernel
resolving static initialization order ambiguity for C++ startup (pre-main) code ...
I would not be surprised if the problem with the lolin board you just mentioned has the same root cause.
full plume
if there's something quirky in the initialization, even a minor tool chain update can let latent bugs run loose, and IIRC ESP-IDF 5.3 to 5.4.1 moves from GCC 13 to GCC 14 - much more than a minor update. OTOH, it's only the S3 build that's failing to start - the lolin_s2_mini build in the exact same local repo (checkout from your fork with ESP-IDF 5.4.1) and even in the same shell instance (i.e. only one call to . esp-idf/export.sh) works fine, so if it is the same problems it's probably subtle.
But of course sometimes that's exactly where the unitialized memory matters - accessing a global instance before it's constructor gets called might not cause noticeable problems if the RNG pixies left the right bits in the right spots...
mortal kernel
@tulip sleet This gets more intriguing. The long-standing double fault problem with sockets might also be related. On my first build with vfs completely stripped, I'm finding CP socket code that hooks itself up to vfs file descriptor events. Let me see why that's there...
full plume
I've noticed that CP appears to do some "man behind the current" magic with sockets - the most obvious impact is you have to import socketpool instead of import sockets, and then work with a "there can be only one" (pool per radio) constraint (which actually makes sense for most things one would be doing with CP). Plus rigging up all the web based workflow bits.
So is the vfs support in CP only for external storage like SD cards (i.e. the support for uf2/CIRCUITPY flash file system access goes through something other than vfs) ?
mortal kernel
So is the **vfs** support in CP only for external storage like SD cards (i.e. th...
The naming is confusing. CP has a vfs abstraction that it inherits from MicroPython. The underpinning is oofatfs, a light-weight fat fs implementation. Both external and internal flash CP filesystems use these. The ESP-IDF vfs component provides an abstraction for IDF tasks to access a filesystem or filesystem driver provided by another or the same IDF task. ESP-IDF is very much a small OS in its own right. It uses FreeRTOS as its kernel.
manic glacierBOT
manic glacierBOT
full plume
The naming is confusing. CP has a vfs abstraction that it inherits from MicroPyt...
Thanks for the explanation, that clear things up a bit. Most of the embedded C++ work I've done lately has been with the Arduino framework APIs, I haven't done much directly with the ESP-IDF apis or FreeRTOS yet.
full plume
<@329766224093249548> I've tracked down the component causing the double fault. ...
Not sure there's much I can effectively help with tracking down initialization (i.e. anything that happens before boot.py/code.py) issues without being able to run under a (presumably JTAG?) debugging environment. I have a rather old (https://www.olimex.com/Products/ARM/JTAG/ARM-USB-OCD/) JTAG interface, but I don't see any obvious matching configurations in the $OPEN_OCD_SCRIPTS/interfaces. I could theoretically get it working, as I've used it to debug ESP32 C++ firmware with PlatformIO, but as I suspected the JTAG pins aren't available on the Lolin S2/S3 mini boards. I have other S3 boards that provide access to the JTAG pins, so I could try those - they might reproduce the issue I'm seeing with the lolin_s3_mini. But that would probably be a non-trivial amount of fiddling to get everything set up, so...
@mortal kernel
- Would it make sense to first pull and build from the head of your circuitpython fork and see if that helps?
- Does that have the vfs mods (or are they hacks in the ESP-IDF code)?
- Any recommendations on a relatively affordable usb JTAG interface that works well with CircuitPython and ESP32-* controllers?
lone axle
it seems like readthedocs.org may be experiencing issues. I am seeing a default NGINX page instead of the normal site at the root URL.
tiny peak
<@849815081251766292>
- Would it make sense to first pull and build from the...
have you tried the built in USB JTAG interface?
mortal kernel
This is the JTAG setup I’m using. The board under test is an Adafruit Feather Huzzah ESP32. The probe is a J-link Mini. I made the small breakout board from a 10-pin cable, a couple of headers, and a miniature prototype board.
full plume
have you tried the [built in USB JTAG](https://docs.espressif.com/projects/esp-i...
ummm.... didn't even know about that.
full plume
This is the JTAG setup I’m using. The board under test is an Adafruit Feather Hu...
OTOH that looks familiar....
tiny peak
When I've done ESP32-S3 stuff with Zephyr, it was able to handle all the programming and debugging over the regular USB port
not sure exactly how the config worked under the hood, but I expect openocd has config for it, maybe?
full plume
not sure exactly how the config worked under the hood, but I expect openocd has ...
It looks like it does, but it doesn't appear to be in the ESP-IDF 5.3 or 5.4.1 installs for CircuitPython.
tulip sleet
@full plume We have other issues aplenty to debug if you don't want to get involved this deep in this particular chain. 🙂
full plume
https://github.com/orgs/micropython/discussions/11695 mentions starting openocd with ../esp32s3-builtin.cfg, but there isn't anything like that in $OPENOCD_SCRIPTS/interface
tiny peak
when I've had trouble with missing config files for openocd, the solution often involved changing my PATH to make sure that I was running the openocd binary from a specific toolchain, from a system package, or from a particular version of openocd that I installed locally. It's been my experience that openocd binaries will have hardcoded search paths for config files, and the config file I need may be somewhere else. Also, the openocd may be the wrong version (or not a particular vendor fork) and incompatible with the config file anyway.
full plume
<@375390851595108353> We have other issues aplenty to debug if you don't want to...
I've actually got a rather long history and mild affection for C++ initialization issues, and I know I'll eventually want the ability to run CircuitPython under a debugger, so it's less a matter of if than when
tulip sleet
got it. I think you could mock the lolin boards on a regular dev board with all the available pins, and see if you can duplicate the problem. I'm not sure why these problems should be lolin-specific
full plume
when I've had trouble with missing config files for openocd, the solution often ...
In this case, both the openocd binaries and support/interface config files are part of a dedicated install of the specified ESP-IDF toolchain version required by the CircuitPython espressif port.
tulip sleet
eightycc I think was trying openocd originally, with several roadblocks. Switching to J-Link helped, if i remember right.
full plume
So I'm pretty sure that's the one we should use, although finding a config file for the build in ESP32-S3 usb jtag support and adding it in might make sense.
hmm, looks like I was lookin in the wrong place and it might already be there, just under $OPENOCD_SCRIPTS/board/esp32s3-builtin.cfg instead of $OPENOCD_SCRIPTS/interface/..
full plume
I now have openocd connecting to an Freenove WROOM ESP32-S3 over USB/JTAG while running under WSL. It's a rather tedious process, but should be manageable by anyone who'd be trying to run CircuitPython under debug in the first place...
The down side is that this is one of those boards with dual USB-C connectors, only one of which shows up as a USB JTAG/serial debug unit.
I don't know how/if you can access the S3's built in USB-JTAG support on a board with a single USB like the Wemos/Lolin S2/S3 mini boards
full plume
eightycc I think was trying openocd originally, with several roadblocks. Switchi...
did he hit roadblocks before or after getting openocd working at least a little bit with the S3's built in USB-JTAG support?
slender iron
@mortal kernel look at the cmake file changes in the IDF around vfs. there are -u (IIRC) defines that are used to force linking in init code
full plume
@mortal kernel I've got an S3 N8R2 Devkit-C board that is at least initially connecting to openocd over USB-JTAG. I'm building your branch for espressif_esp32s3_devkitc_1_n8r2, hopefully that will let me step through the startup/initialization code.
tulip sleet
did he hit roadblocks before or after getting openocd working at least a little ...
I just suggested he use the J-Link and its software instead of openocd. I have found it to be more reliable. There's a fuse to set to be able to use JTAG on ESP
slender iron
https://github.com/adafruit/circuitpython/pull/10264 (gc selective collect) passed CI and is ready for review
full plume
JLINK looks nice, but I'm not ready to drop $500+ on a jtag interface at the moment.
tulip sleet
JLINK looks nice, but I'm not ready to drop $500+ on a jtag interface at the mom...
non-commercial price is $60:
manic glacierBOT
full plume
I have other C++ projects I'd want to use it for, which I might end up releasing under a commercial (or at least semi commercial dual) license someday. And while I'll probably be doing embedded development just for fun even after I'm retired, I currently still make my living as a developer. Feels a bit off to get a non-commercial license as a professional developer.
Although if I can't get openocd working, I might eventually consider picking up a non-commercial JLink license just for CircuitPython
mortal kernel
@tulip sleet The root cause of the double fault in #10191 appears to be that the ESP32 ROM was compiled with a version of newlib older than the one included in current versions of ESP-IDF. As a space optimization, ESP-IDF calls into a few ROM functions, notably __swsetup_r, which calls __swhatbuf_r via __smakebuf_r. Because the stat structure mismatches, stack corruption and ultimately a double fault result. A "SPIRAM cache workaround" is involved, I'm untangling that to get to the correct ESP-IDF configuration parameters to get ESP-IDF to stop using the older ROM routines.
tulip sleet
<@329766224093249548> The root cause of the double fault in #10191 appears to be...
aha! There must be some workaround setting -- I'm surprised it's not the default
Scott updated newlib in a recent PR, but more recently than we started seeing this issue.
looking for that...
mortal kernel
Was it related to a 32-bit time issue?
mortal kernel
This issue in ESP-IDF dates back to 2022. Depending on compiler options, the stack corruption may not have been causing a crash due to the arrangement of the stack frame.
tulip sleet
Is there an issue filed in their repo?
mortal kernel
Yes, it's the one I noted in our #10191 yesterday: https://github.com/espressif/esp-idf/issues/7980. It didn't quite make sense to me how a closed issue was still causing our crash until I started to pull apart the ROM routines with Ghidra this morning. Then I went back and looked at the actual changes to ESP-IDF and saw that it was dependent on configuration setting changes.
tulip sleet
[total aside: if you surround a URL in <...>, it suppresses displaying embeds
mortal kernel
Got it. That saves everyone else a bit of a flash when I quickly delete the embeds. Thanks!
lone axle
@slender iron are there any changes to your version of the boot animation beyond removing enumerate from the main loop? I'm started from the one here: https://github.com/adafruit/Fruit-Jam-OS/issues/1 and changed main loop to not use enumerate. I'm noticing that after it runs a few times successfully it will stop playing audio in a way that can be heard, but the while audio.playing: is staying True forever meaning the code never exits. I have it inside of code.py instead of boot.py and I restarted it with ctrl-c/ctrl-d, I get 2-4 playbacks with audio then silent only until I unplug / replug USB. I have the top link from S3 flashed currently on device.
mortal kernel
@tulip sleet Spotted where we started to pick up the ROM mismatch: The fix for #9486 manually adds esp32.rom.newlib-time.ld to the link for esp32, overriding the current newlib functions in question with their out of date ROM versions.
tulip sleet
mortal kernel
Yes. That's the one.
tulip sleet
I don't see an updated newlib (I see more variants), but I do see the REGISTRATION_FUNCTIONS addition there
mortal kernel
The newlib in question is an ESP-IDF component, so its versioning is out of our control.
tulip sleet
so newlib-foo was added for various foo, and it's one of those?
mortal kernel
Yes, in our ports/espressif/Makefile.
I've elided it, and getting a clean build. Now for a test...
mortal kernel
No, from here, around line 241:
ifeq ($(IDF_TARGET),esp32)
LDFLAGS += \
-Tesp32.rom.newlib-data.ld \
-Tesp32.rom.newlib-funcs.ld \
-Tesp32.rom.newlib-time.ld \
-Tesp32.rom.spiflash_legacy.ld
Removed the esp32.rom.newlib-time.ld
tulip sleet
I wonder what the utility of using the ROM functions is, except to save space.
is there a necessity for some of them?
mortal kernel
That's what I think. Sort of harkens back to patching Mac ROM routines to save space.
Although I would not recommend removing others without careful consideration...
@tulip sleet Yes! Now it boots. What a journey that was!
Looks like we have the same issue in esp32s2, and esp32c3.
Probably copy and paste.
slender iron
<@252717193496756235> are there any changes to your version of the boot animatio...
I'll post my boot.py shortly that I have backed up. May need to cut some traces to revive my fruit jam
The playback issue sounds like one I was seeing too
lone axle
I've done a lot of cleanup in the animation code, once I've got your latest one I'll merge this together and submit a PR to the repo with the new version that is cleaned up and had a few of the other final tweaks from the basecamp thread that hadn't made it into the repo yet.
lone axle
This one still uses enumerate() in the main loop. I've chagned that in mine to this:
for i in range(len(coordinator["steps"])):
step = coordinator["steps"][i]
is the target_frames_per_second necessary? It seems like using this sometimes causes the very last frame of the second blink sprite animation not to be rendered, so the eyes remain larger than the should be (the frame before the last one has larger eyes). With that removed and just using display.refresh() all of the animations look smooth to me, and I have never seen the same issue with the eyes remaining on that 2nd to last frame.
slender iron
Without target_frames_per_second you may actually run faster than you need
weird that you miss one last frame. is there a refresh call after it?
lone axle
yes, the code is the exact same except for removing target_frames_per_second argument. So if there weren't a refresh call after the sprite is updated I think it wouldn't be shown in either case.
it doesn't happen everytime. Anecdotally, it might be tied to the audio in some way. I feel like it's nearly 100% rate of occurance when the audio plays successfully. But occurs less frequently when the audio doesn't play, though I have still seen it several times without audio too.
slender iron
@lone axle can you assign it? I didn't think you could but that is the long term intention
lone axle
<@382939733107408897> can you assign it? I didn't think you could but that is th...
it seems to me like you can. with reinit_display.py like this:
import supervisor
from displayio import release_displays
import picodvi
import board
import framebufferio
SIZE = (640, 480)
# initialize display
release_displays()
fb = picodvi.Framebuffer(
SIZE[0],
SIZE[1],
clk_dp=board.CKP,
clk_dn=board.CKN,
red_dp=board.D0P,
red_dn=board.D0N,
green_dp=board.D1P,
green_dn=board.D1N,
blue_dp=board.D2P,
blue_dn=board.D2N,
color_depth=8,
)
display = framebufferio.FramebufferDisplay(fb)
supervisor.runtime.display = display
and the second file like this:
import supervisor
display = supervisor.runtime.display
print(f"isNone? {display is None}")
display.auto_refresh = False
print(f"size: {display.width},{display.height}")
I see False for is None and I do see the size printed.
slender iron
does it work across reloads?
lone axle
If I press ctrl-D after importing both of the above scripts it goes back to the animation and runs it successfully (it's in code.py). That code just accesses supervisor.runtime.display so it seems to if you mean the ctrl-C / ctrl-D kind of reload.
slender iron
cool! I didn't think we allowed setting it yet
slender iron
@tulip sleet would it make sense to define a CIRCUITPYTHON_SOURCE macro that we could use in the micropython header to limit the allocation APIs?
thinking about a way to "opt in" code to the "new" API
tulip sleet
thinking about a way to "opt in" code to the "new" API
That is interesting. There is already a CIRCUITPY macro which is always 1
slender iron
That is interesting. There is already a `CIRCUITPY` macro which is always `1`
That is set for all source code though. I'm trying to think of a way of having everything in shared-bindings/, shared-module/ and common-hal/ not be able to use m_malloc after I make it collect by default.
tulip sleet
oh, i see, that makes sense. so the idea of getting rid of allocators that don't specify collect/no-collect is appealing?
could that just be defined in py/circuitpy_mpconfig.h ? Is that included in any micropython-origin code?
slender iron
I think it is still implicit with py/mpconfig.h
I don't need to do it. With the build system (aka python) that zephyr uses, it'd be easy to change the cflags for those files
trickier to do in make
I think I will change the defaults to collect
that way I don't break ulab
tulip sleet
the old default is always collect, is that right? so the idea is to disallow unlabeled alloc macros/functions in code that's exclusively ours
tulip sleet
it could be more specific, like MICROPY_DISALLOW_UNSPECIFIED_ALLOCS
or ALLOW_UNSPECIFIED_ALLOCS or... best, i think, MICROPY_REQUIRE_SPECIFIED_ALLOCS.
maybe there's a better word than SPECIFIED. DISALLOW could be a double negative, confusing
slender iron
the challenge is how to mark the CP source for it
tulip sleet
short of adding an #include or a #define in all those source files, I don't see how.
or myabe just in the .h
#define would have to be guarded, so an include that does an #ifndef is a single-line solution
slender iron
the other approach I could see is just a python script that can be run to check it
more of a soft check
tulip sleet
a linting
slender iron
yup
tulip sleet
a pre-commit check
slender iron
doesn't even need to be that really
tulip sleet
i did already redo all the headers in the circuitpython-specific C files to be // This file is part of the CircuitPython project...
tulip sleet
I think I will change the defaults to collect
I think this is the better, safer, default
slender iron
yup, agreed
and I can put m_malloc_without_collect in strategic spots instead
I'm leaving m_malloc_items change because I like it 🙂
tulip sleet
I'm leaving `m_malloc_items` change because I like it 🙂
is items always collect?
slender iron
ya, because they are mp_obj_t
slender iron
@lone axle want a test build that does the gc optimization?
may not impact the animation because it shouldn't collect
jaunty juniper
has CircuitPythonLibrarians not been added to Adafruit_CircuitPython_Pathlib yet ? Or however that works ?
tulip sleet
has CircuitPythonLibrarians not been added to Adafruit_CircuitPython_Pathlib yet...
Fixed! added librarians as a collaborator
lone axle
<@382939733107408897> want a test build that does the gc optimization?
I stepped away for a bit. Sure if you've got one handy I'll try it out. If you're done or winding down for the day it's okay though, I could try to build it from the PR branch.
slender iron
check slack. I set it to jp there
blissful pollen
Just a FYI for now, Cooper and me have both started to notice a "clicking" in audio that seems to occur with the GC calls. He has been working on isolating and testing more. I'm going to try to look into it this weekend.
Interesting enough it does not occur is "something" (in my case a single midi note) is playing.
tulip sleet
@mortal kernel I know what's going wrong with TLS now. I had been spending a while looking for a bug or config setting in esp-idf, but there was no bug.
For some reason we have our own copy of bundle handling in lib/mbedtls_config. Jeff did this some time ago when refactoring the ssl code so it could be shared between Pico W and Espressif. But now that the bundle format has changed, the cert handling code in there is wrong. I'll have to see why that code was factored out into our own copy, instead of depending on the code in each SDK.
This also explains why, when I set logging inside those routines in ESP-IDF, the logging never got printed out. I thought there was something wrong with the logging setup, but it was just that code was never called.
mortal kernel
<@849815081251766292> I know what's going wrong with TLS now. I had been spendi...
OK, I think I follow what's going on. ESP-IDF is compiling its own mbedtls component, butports/espressif/Makefile is pulling in lib/mbedtls_config/crt_bundle.c.
tulip sleet
exactly
I had forgotten about mbedtls_config, but I did look at it relatively recently for some other reason.
for instance, there is a function pointer you set for the bundle attachment, and it's hooked into the mbetls_config
tomw I'll look at this and figure out what to do. it's late here now
mortal kernel
Here, too. I'll leave you with some good news: everything I'm trying with ESP-IDF 5.4.1, except for tls, is working.
tulip sleet
main thing is that the problem is now diagnosed, instead of being a head-scratcher
mortal kernel
Yes, it's good to take out the mystery part. Speaking of which, why are there merge conflict lines in lib/mbedtls_config/mbedtls_config_port.h?
tulip sleet
Yes, it's good to take out the mystery part. Speaking of which, why are there me...
sounds like a bug, and sounds like nothing else actually includes that. I'll take a look at that too.
mortal kernel
Something of more general interest, but related to CP builds. I've upgraded my development machine from an older Intel I5 10th gen 8 core (16 thread) box running Debian Bookworm to a Mac Mini M4 Pro using homebrew ports of the tools I was using on Linux. I expected build times to be faster, but they were noticeably slower by about 2x. I then installed Debian Bookworm in a Parallels virtual machine (allocated 6 cores and 16GB) and ran a few CP builds there. Surprisingly, the builds were not only faster than native MacOS builds, they were also about 2x faster than my older Intel box running on all 8 of its cores. So, even with virtualization in play, the M4 is significantly faster in this case.
jaunty juniper
I'm curious, does it properly use the available cores when running natively ?
(I have a MacMini 2018 😬 so i can't compare timings)
mortal kernel
It appears to looking at htop. I suspect it's homebrew related, possibly some AMD64 object code getting run on Rosetta.
I'm actually happy using virtualization for now. It has advantages. I can clone VMs for different build setups. Similar to Docker containers, but not as fussy to set up.
slender iron
@lone axle just went to test the keyboard with your guide an realized the example code doesn't setup the usb host
I guess the two boards you use have it on my default
lone axle
<@382939733107408897> just went to test the keyboard with your guide an realized...
I don't think I know what you mean setup the usb host
slender iron
# Initialize USB host power if available
if hasattr(board, "USB_HOST_POWER"):
usb_power = digitalio.DigitalInOut(board.USB_HOST_POWER)
usb_power.switch_to_output(value=True)
print("USB power on")
# Initialize USB host port with D10 and D11
usb_port = usb_host.Port(board.D11, board.D10)
Ah, no I don't think I've ever seen code like that. I haven't done that for any of the USB guides or examples that I have worked on I don't think.
slender iron
that's for working on a board that doesn't have designated pins
lone axle
Should I add it as commented out to the examples?
slender iron
maybe as a page to mirror into guides?
lone axle
A page detailing how to wire and initialize it for a board without the builtin pins / port?
slender iron
yup, if we don't have one already
lone axle
I will try to wire it up, I've only tested any of the examples on the 2 devices in the guide, plus fruitjam for some of them.
slender iron
👍 I'm trying with a rp2350 feather now
full plume
Something of more general interest, but related to CP builds. I've upgraded my d...
Roughly how long does a full (for one platform - ideally Espressif) CP build take on your system? Mine seem unusually slow compared to other builds. I'm wondering if it has something to do with the CP build running under WSL2 (Ubuntu) on a Windows 11 system.
tulip sleet
Roughly how long does a full (for one platform - ideally Espressif) CP build tak...
WSL filesystem access is pretty slow (WSL1 was worse)
are you doing make -j<n> ?
where n is available cores?
tulip sleet
Roughly how long does a full (for one platform - ideally Espressif) CP build tak...
metro esp32-s3 clean build is:
real 1m49.771s
user 8m19.294s
sys 3m29.030s
on my I7-8700, 6 cores
mortal kernel
On Debian Bookworm running in a 6 core 16GB virtual machine under MacOS on an M4 Mini:
time make BOARD=adafruit_feather_huzzah32 DEBUG=1 -j6
________________________________________________________
Executed in 34.52 secs fish external
usr time 116.30 secs 234.00 micros 116.30 secs
sys time 38.95 secs 408.00 micros 38.95 secs
So, that's about 34.5 seconds wall clock time.
(Did a clean first)
tulip sleet
@mortal kernel I have a fix for the TLS problem. I wrote a wrapper to use the espressif code and not the older copied code in lib/mbedtls_config. I removed that unused .h with the merge junk in it
it had appeared there during a merge and should have just been deleted
should I just push to your PR?
tulip sleet
On Debian Bookworm running in a 6 core 16GB virtual machine under MacOS on an M4...
that is with ULAB and camera code turned oof?
mortal kernel
Please push the PR. Things have gone a bit sideways with ESP32-C6.
No, that's tip of main unmodified.
On ESP32-C6 I'm seeing safe mode restarts.
full plume
On Debian Bookworm running in a 6 core 16GB virtual machine under MacOS on an M4...
wow... timing a build of lilygo_tdisplay_s3 to get an actual number, but it feels like my builds have been closer to 30 minutes than 30 seconds.
tulip sleet
how many cores and how much ram in your WSL instance?
and are you working in the local filesystem in WSL?
full plume
and that's running -j14 on an SMG Ryzen 9 7950X 16 core processor at 4.5GHz
slender iron
AMD 5950X w Arch Linux: ```
time make BOARD=adafruit_feather_huzzah32 DEBUG=1 -j32
Executed in 25.63 secs fish external
usr time 261.85 secs 614.00 micros 261.84 secs
sys time 168.48 secs 343.00 micros 168.48 secs
Honestly haven't done anything to configure WSL, it's just running defaults which has been fine for the few times I've used it before now
mortal kernel
wow... timing a build of lilygo_tdisplay_s3 to get an actual number, but it _f...
You might consider installing VBOX and running a linux distro under that. I've not heard anything positive re: WSL performance.
full plume
But I am seeing at least 8 cores running 50%+
full plume
You might consider installing VBOX and running a linux distro under that. I've n...
Yeah, I'm thinking VBOX might be the way to go.
mortal kernel
AMD 5950X w Arch Linux: ```
time make BOARD=adafruit_feather_huzzah32 DEBUG=1 -j...
Let me crank it up to 12 cores to see what it will do full throttle...
tulip sleet
@mortal kernel I pushed a commit
When I tried builds in WSL1, it was about 4x slower than native Linux.
mortal kernel
With all 12 cores and 32GB:
time make BOARD=adafruit_feather_huzzah32 DEBUG=1 -j12
________________________________________________________
Executed in 26.39 secs fish external
usr time 146.26 secs 295.00 micros 146.26 secs
sys time 54.96 secs 99.00 micros 54.96 secs
So the 5950X native with 32 threads is just a tad faster than 12 virtualized M4 cores. At least on this test.
full plume
Yep, something's definitely off. My build is about 15 minutes in and still running.
mortal kernel
<@849815081251766292> I pushed a commit
Short and sweet. Thank you, Dan!
tulip sleet
I was surprised it was so easy. I'm wondering if the lib/mbedtls_* stuff should move back to raspberrypi. The espressif port does not use the shared lib/mbedtls, and now doesn't use the other mbedtls stuff under lib any more.
the shared-module/ssl is still a good idea
mortal kernel
I was surprised it was so easy. I'm wondering if the lib/mbedtls_* stuff should ...
I think that it should, if only to avoid future confusion. Plus, Raspberry Pi really wants versions of lwip and mbedtls that track the Pico SDK. I haven't seen difficulties up to now, but I don't believe we're doing it quite right trying to use a "one size fits all" approach.
tulip sleet
i'll open an issue about this, then. This was all done several years ago and things have changed
full plume
I think that it should, if only to avoid future confusion. Plus, Raspberry Pi re...
FWIW, I'm generally a huge fan of portable single source builds. However, even I consider some areas best left to platform-specific configurations, and anything dealing with TLS or certificates usually falls into that category.
Oh, and my build is still going...
full plume
tulip sleet
so, is your build directory local to WSL, or is it something mapped in C:
full plume
yeah, it's mapped in C (under my windows home directory)
tulip sleet
I think that's going to be really slow, because it has to do filesystem translation. But the homedir in WSL is in the WSL filesystem, I think. That should be faster
i wrote this warning in the guide long ago:
Warning: Don't build in a shared folder (in /mnt/c). You'll probably have filename and line-ending problems.
I didn't consider slowness at the time. are you building in /mnt/c
WSL USB support has also not been great, though it might be better now
full plume
yeah, I'm going to try tweaking the .wslconfig settings and cloning / building under the local (WSL Ubuntu) file tree
@mortal kernel have you pushed your tls related changes yet (and if so, where) ?
tulip sleet
i pushed them to the PR, so they are in the branch that the PR references
blissful pollen
WSL USB support has also not been great, though it might be better now
It’s not really. I spent some time fighting it a while back. I use windows to actually connect / flash to usb if required.
slender iron
it is weird that 8 s2 builds are failing for every PR now
I bet it was #10281
the scheduler is bad at scheduling .h changes
for CI testing
full plume
Ok, this is more like it...
FWIW, the only changes I made were building out of the local Ubuntu file system and setting (creating) .wslconfig in my windows user home directory to
[wsl2]
memory=64GB
processors=14
swap=0GB
guiApplications=false
nestedVirtualization=false
So maybe I can hold off on VBOX at least for now.
full plume
It’s not really. I spent some time fighting it a while back. I use windows to ac...
Did you get it working at all? I've gotten to the point of successfully connecting to an ESP32-S3 Devkit C through openocd and the ESP32-S3 built-in USB JTAG support, but I haven't actuall tried running gdb yet.
Wondering how much further I want to push into that rabbit hole before giving up on trying to debug CP under WSL2
full plume
WSL USB support has also not been great, though it might be better now
Do you have a general idea how many developers (%) are building CP under WSL?
blissful pollen
Did you get it working at all? I've gotten to the point of successfully connec...
I think with openocd you can connect to an IP so I ran it on WSL but had another process on windows. That was over a year or two ago so I forget the details.
I never spent too long on it overall. WSL2 isn't great for USB support. There are some resources on forwarding it via IP but the documentation is sparse
tulip sleet
Do you have a general idea how many developers (%) are building CP under WSL?
not really, but a lot of people doing casual builds will use WSL, since it's really easy to set up and is Linux from the command line. I personally found VBOX to be more trouble to set up compared with VMWare Player. But my dev machine is native Linux. I don't use Windows unless I have to. I have several micro-form-factor machines (bought as off-lease refurbished). When i need to use windows, I use connect to a Windows box through NoMachine
blissful pollen
Do you have a general idea how many developers (%) are building CP under WSL?
I do use it as my primary (only) environment. The main frustration is the USB ports
tulip sleet
some people do dev on RPi's but they are very patient. RPI5 might not be too terrible
jaunty juniper
meanwhile, my 2018 Mac Mini 3 GHz Intel Core i5 6 cores takes 5:34 for
gmake BOARD=adafruit_feather_huzzah32 -j6 DEBUG=1
full plume
I do use it as my primary (only) environment. The main frustration is the USB po...
Yeah, I initially only used WSL for builds, then used UF2 to upload firmware and windows serial usb (typically Thonny) for REPL. But now I'm trying to get a gdb/JTAG/debugging setup for CP going.
blissful pollen
Yeah, I initially only used WSL for builds, then used UF2 to upload firmware an...
If I was smart I would have wrote down what I did... I am not smart. But I believe with openocd you can connect via TCP/IP ports. So I ran it on windows and then connected a matching process on WSL to that IP
tulip sleet
a used odler Optiplex MFF or similar can be had for $100-$200. I usually bought from dellrefurbished.com but they are very low on stock now, maybe due to panic buying. Lotsa laptops there, though. Also ebay etc.
full plume
@mortal kernel With firmware built from git clone https://github.com/eightycc/circuitpython.git (as of about twenty minutes ago), the lolin_s3_mini build of firmware.uf2 is now booting up and successfully providing USB REPL / CIRCUITPY drive. So whatever you did appears to have fixed that too.
mortal kernel
I put together a $500 Intel box for a friend a few weeks back. Here are the specs:
H610M S2H V2 Motherboard
Core i7-12700K 12-core
32GB 3200 DDR4 RAM
1TB Blue SN580 NVMe
CX650 650W (way overkill) ATX power supply
Core 1100 Mini-Tower Case
All new. Ran like a bandit, would make a great development box. Running Linux, of course.
full plume
My main dev system has traditionally been Linux, but for the last year or two it's been Windows. One of the main extra twists I deal with is that I typically run four or five 4K monitors, and keeping the video drivers / configuration for multi-monitor Linux systems up to date has been a bit of a pain over the years. Plus my local system has often just been a smart terminal for RDP/NoMachine remote access anyhow, although all my ESP32 embedded work has been local builds.
full plume
I put together a $500 Intel box for a friend a few weeks back. Here are the spec...
I've still got a few older Linux systems I could use. If I can't get a reasonably stable debug setup going under WSL, I might be tempted to try getting my local WSL CP build directory remote mounted on a modest Linux box and just use it for running GDB, keeping all the actual builds on my main system.
mortal kernel
I've still got a few older Linux systems I could use. If I can't get a reasona...
I'm interested in hearing whether you get JTAG over USB serial working, and if so how you did it. I tried, but gave up in favor of the J-link connected to JTAG pins.
full plume
But even before I have a working debug setup, now that my builds are going over 100x faster it's much easier for me to do some basic testing of the ESP5.4.1 branch on other S2/S3 targets (probably even some non-UF2 targets like vanilla ESP32, H and C variants. Beyond successfully booting up into REPL and file system access (+CIRCUITPY on supported builds), are there any recommended code.py + test module code to get a bit better test coverage?
full plume
I'm interested in hearing whether you get JTAG over USB serial working, and if s...
been keeping notes (not necessarily because I'm smarter, just lazy and been doing this long enough to learn which little extras payoff in the long term) so if it ends up working reasonably well I should be able to put together a decent writeup. And even if I can't get the USB / JTAG stuff reliable, I still might consolidate some pointers on just building CP under WSL (at least what's worked for me)
mortal kernel
But even before I have a working debug setup, now that my builds are going over...
The C variants are where the pain is right now.
If, by any chance, you've got ESP32-P4, that would be very helpful. I've got a note next to it in my notebook: "Unoobtainium"
jaunty juniper
wasn't there something at some point about the issue where the CDC input buffer is filled (by Mu generally) causing ctrl-C to be ignored (on SAMD21) ?
jaunty juniper
wasn't there something at some point about the issue where the CDC input buffer ...
ah yeah that https://github.com/adafruit/circuitpython/pull/7227
full plume
The C variants are where the pain is right now.
I have a lolin_c3_mini, lilygo_ttgo_t-oi-plus (C3), and a few others that don't yet have their own boards definition ( Lilygo T7-C6, generic C6 "SuperMini" ). I'll try getting those built and loaded, plus I still plan on doing more thorough testing on the lolin_s3_mini . I also have a TenStar Robot H2 SuperMini, which I might try with espressif_esp32h2_devkitm_1_n4
Looks like there are already ten supported (at least included in boards/* ) C6-based boards, so I should be able to find something worth trying for the two C6 controllers.
full plume
The C variants are where the pain is right now.
I don't have an ESP32-P4 yet, but it definitely looks interesting
round hazel
Hi is this where to ask questions when testing alpha/S3 builds?
lone axle
Hi is this where to ask questions when testing alpha/S3 builds?
This is mainly a channel for development, if your trying to make a build, or trying to ask about something planned or cutting edge yes here is good. If it's more about how to do something with an existing feature or device then #help-with-circuitpython is best.
manic glacierBOT
mortal kernel
@danh Where can I find the repo for code.circuitpython.org. I want to understand the protocol between the webpage and the USB serial device.
tulip sleet
@danh Where can I find the repo for `code.circuitpython.org`. I want to understa...
Melissa is the main person to work on this
mortal kernel
There's something coming over the serial connection from code.circuitpython.org when I hit the open button that's crashing just the ESP32-C6. That's with a commit that reads as if it's unrelated to serial com, although it does touch terminal. The feedback in the file directory panel looks like console output from the safe mode restart after the crash. I think I'll need to resort to my friend JTAG again.
tulip sleet
There's something coming over the serial connection from `code.circuitpython.org...
which commit is it?
mortal kernel
#10208
full plume
@mortal kernel just built and uploaded lolin_c3_mini. On connect (in Thonny) I see
mortal kernel
<@849815081251766292> just built and uploaded lolin_c3_mini. On connect (in T...
You're likely seeing the issue I'm working through now with ESP32-C6.
full plume
entering help("modules") at the REPL lists a bunch of modules but then appears to crash :
board json socketpool vectorio
builtins keypad ssl warnings
busdisplay locale storage watchdog
busio lvfontio struct wifi
canio math supervisor zlib
codeop max3421e synthio
Plus any modules on the filesystem
ESP-ROM:esp32c3-api1-20210207
Build:Feb 7 2021
rst:0xc (RTC_SW_CPU_RST),boot:0xd (SPI_FAST_FLASH_BOOT)
Saved PC:0x40382ab0
SPIWP:0xee
mode:DIO, clock div:1
load:0x3fcd5820,len:0x120
load:0x403cc710,len:0x900
load:0x403ce710,len:0x2178
entry 0x403cc710
Auto-reload is off.
Running in safe mode! Not running saved code.
You are in safe mode because:
CircuitPython core code crashed hard. Whoops!
Hard fault: memory access or instruction error.
Please file an issue with your program at github.com/adafruit/circuitpython/issues.
Press reset to exit safe mode.
Adafruit CircuitPython 10.0.0-alpha.2-37-gf9ab642fdd on 2025-04-25; Wemos Lolin C3 Mini with ESP32-C3FH4
>>>
Oh, that is very helpful. What you're seeing is the crash that's breaking tools that rely on the serial USB connection. Thank you for showing a simple means of reproducing it!
full plume
I'm currently at
james@peppy:~/git/circuitpython/ports/espressif$ git show
commit f9ab642fddc034f55f40a7cd8357374c6081d7b3 (HEAD -> main, origin/main, origin/HEAD)
Merge: 53a856b67a 28856dbbcb
Author: Scott Shawcroft <scott@adafruit.com>
Date: Wed Apr 23 13:07:55 2025 -0700
Merge pull request #10281 from rianadon/enumeration-bufsize
Increase CFG_TUH_ENUMERATION_BUFSIZE so that USB devices with long descriptors are recognized
@mortal kernel let me know when you'd like me to pull and rebuild lolin_c3_mini.
mortal kernel
<@849815081251766292> let me know when you'd like me to pull and rebuild lolin_c...
Will do. It might not be until later today or tomorrow.
tulip sleet
#10208
You could try building with CIRCUITPY_LVFONTIO = 0 to see if there's some bug introduced in the font lvfontio code. Also, supervisor/shared/display.c has some code reorganization; that possibly could be an issue
full plume
That's fine, I'm using the ESP-IDF 5.4.2 lolin_s2_mini build for now to work on my project. I have noticed one odd thing - when I try opening a file in Thonny from the "Circuitpython device" instead of CIRCUITPY drive I'm getting
everything looks like a directory, and you can't open the files.
mortal kernel
That's fine, I'm using the ESP-IDF 5.4.2 lolin_s2_mini build for now to work on ...
Interesting, that could be related. I'm going to follow up on Dan's suggestion, set CIRCUITPY_LVFONIO = 0 to see what that does. It does look like on ESP32 parts that don't crash, we're getting extraneous characters on USB serial.
full plume
I haven't been able to duplicate this with https://code.circuitpython.org/ , because I haven't been able to get it to use the "device" file access - it only lets me use CIRCUITPY. (Maybe this is normal behavior? I haven't used the online editor much).
tulip sleet
are you using it over wifi or over USB?
for boards where CIRCUITPY is available over USB, via MSC, it will do file read/writes. For boards like C3 without full native USB, it sends invisible commands to the REPL to read/write files
full plume
So far almost always over USB. I've tried WiFi just to check that it worked (although not with 5.4.1 builds yet)
full plume
Interesting, that could be related. I'm going to follow up on Dan's suggestion, ...
Sounds right - notice the "root" directory being shown as 🔳 A
mortal kernel
@tulip sleet CIRCUITPY_LVFONTIO = 0 didn't make a difference. I need to blow a fuse and strap to start using JTAG...
full plume
@mortal kernel I have espressif_esp32c6_devkitm_1_n4 running on a C6 "SuperMini". No CIRCUITPY ( AFAIK that's only supported on ESP32 S2 or S3? ) appears. When I try to open a file in https://code.circuitpython.org/, I get
(got a chuckle out of that)
mortal kernel
<@849815081251766292> I have **espressif_esp32c6_devkitm_1_n4** running on a C6 ...
Yes. Same bug. What you're seeing are the messages on USB serial from the re-boot into safe mode. And right, no CIRCUITPY MSC on the C6.
full plume
Thonny actually shows a normal looking directory, but attempting to load code.py fails.
mortal kernel
@tulip sleet Here's a backtrace from one failure instance that I've been unable to reproduce. I recall seeing message traffic about oddly truncated messages, but have had no luck finding a corresponding issue.
tulip sleet
this looks like some corrupted object, because the qstr it's trying to find seems to be out of range.
i think it's some second-order bug due to a memory smash, say,
mortal kernel
Yes, that's probably the case.
tulip sleet
these weird repl bugs remind me of https://github.com/adafruit/circuitpython/issues/8038, which we never figured out, but something really bad was happening.
I just reproduced 8038 in 9.2.7 and 10.0.0-alpha.2, so it's still there.
mortal kernel
The CP / Espressif debug story for RISCV has some gaps. CP's Espressif Makefile sets -DNDEBUG and -Os for RISCV targets. I don't know why it was setup like that. Also, I'm not getting any ESP-IDF tracing with DEBUG=1. And also, panics fly through reset without stopping, making trapping the fault problematic. Finally, the RISCV build of gdb doesn't support Python, so the FreeRTOS thread tools don't work.
mortal kernel
I'm going to take one more stab at trapping the fault in the handler with a hardware breakpoint. After that, I'll put it aside until Monday.
tulip sleet
The CP / Espressif debug story for RISCV has some gaps. CP's Espressif Makefile ...
-Os was needed, I think, because the RISC-V code is larger than with, say, -O2.
I have used gdb-multiarch successfully instead of the ARM version to get the Python support.
Thonny is doing something or other under the covers that is provoking this bug. Just typing that code into the REPL via a terminal program does not cause the problem. Since Thonny and code.circuitpython.org operate similarly, I'm wondering if these are all related. The SAMD21 port is easy to debug and much simpler than Espressif
mortal kernel
Interesting thought.
tulip sleet
But I don't know whether diagnosing that bug would lead to this bug or not.
and this bug was originally discovered because it was so bad that it caused an unprotected bootloader to get smashed (so something was writing flash)
If you try this on a SAMD21 board, update the bootloader from the matching update-bootloader*.uf2 in https://github.com/adafruit/uf2-samdx1/releases/tag/v3.16.0, not because you need a new bootloader, but because the bootloader updater is careful to make sure the bootloader is protected. There were some boards that shipped with unprotected bootloaders
I'm not sure whether #8038 fails on all SAMD21, or just ones with no external flash, etc.
but this is for later, you are on a roll right now
mortal kernel
This looks like it's a 10 blocking bug, so I'll make it my priority on Monday. I'll continue with the ESP32-C6 debug until I hit a wall, then I'll go for the SAMD21 bug.
tulip sleet
well, it's a very obscure bug, and most people don't hit it, so it's been long term. And most people don't use Thonny. I don't think this would fail on code.circuitpython.org because SAMD21 exposes CIRCUITPY, so USB is going to be the CIRCUITPY-supporting version, not the no-CIRCUITPY version
but i'll try it
I can't provoke the bug with code.circuitpython.org with a QT Py SAMD21, but not surprised
this bug you're seeing, I forget, is it provokable via Thonny too?
I need a build past alpha.2, right?
mortal kernel
You mean the SAMD21 bug. The ESP32-C6 bug blocks programming the part. I suppose one could code a setup.toml through the REPL and then switch to Web workflow, but as @full plume showed, even REPL isn't stable.
tulip sleet
REPL isn't stable even with a terminal program, is that right?
sorry I am asking you to summarize instead of going back to look
mortal kernel
Thonny will provoke it. The bug is in alpha.2. Right, REPL can fail, too.
I bisected it to pull #10208, but now I suspect that's not really responsible, it's just provoking a latent bug that's been there for a while...
tulip sleet
I just tried help("modules") on the espressif C6 DevKit board with 8MB flash, and it didn't crash with alpha.2. Do you see a crash with that on the C6 you're using? Which C6 is it
mortal kernel
James was seeing it on a C3. I wasn't able to get that failure on a C6 either.
I don't have a C3 board here, do you?
tulip sleet
i have a qt py c3 and a devkit C3
trying those
@full plume what were you building that you saw the help("modules") failure on the lolin C3 mini? Was it the tip of main or was it eightycc's PR? I'm trying to reproduce on other boards
mortal kernel
Trapped the panic! It's in exc_add_str. The stack's corrupt, but it gives me a place to work backwards from.
tulip sleet
I just reproduced the junk in the directory listing on the QT Py C3
mortal kernel
I've worked back to mp_obj_new_exception_msg_vlist, so we're handling a Python exception along the path to the panic. I'm going stop there for today.
manic glacierBOT
mortal kernel
@tulip sleet Good morning, Dan. I took a second look and made some progress trapping #10296. It fails in one of two ways, it seems to be about 50/50 which way it fails. Something I see that perhaps you can explain is that although the small program inducing the failure is injected through REPL, it is presented to the parse_compile_execute as a single vstr. Typing directly into REPL presents the same code as a series of separate vstr's. How is this difference in behavior triggered?
Also, an earlier problem I shot (double fault early in ESP-IDF initialization) was due to an incorrectly included .ld fragment. Thinking #10296 might have a similar root cause, I went back and looked into how ESP-IDF handles inclusion of .ld fragments. There is a single CMake file, ports/espressif/esp-idf/components/esp_rom/CMakeLists.txt that does all the heavy lifting for that process. Along with deciding which .ld fragments to include, it also provides patches for various ROM routines.
tulip sleet
it is probably sent to the REPL using "raw" mode in the REPL
ctrl-B or ctrl-E I forget. There is paste mode and there is raw mode
mortal kernel
Thank you, let me see if I can induce the bug from the command line.
tulip sleet
sorry, ctrl-A is raw mode
"raw REPL". ctrl-A to enter, ctrl-B to finish
ctrl-E is paste mode; it suppresses auto-indent, so you can paste multiple lines from an editor, etc.
mortal kernel
Back to ROM: CP does some of what the ESP-IDF CMake file does in its ports/espressif/Makefile, but not all and it doesn't entirely match. Also, CP is completely missing the patches (at least I can't find them).
tulip sleet
yes, certainly a lot easier than dealing with code.circuitpython.org or thonny
definitely worth fixing; the Makefile was snapshotted taken from ESP-IDF 3 or 4, and though we tried to update, I can believe there are missing things
mortal kernel
As you can see from the gdb output in the issue, I've been able to turn on full debugging for the C6 builds where there's enough flash space to handle it. That's helping.
tulip sleet
Another bug that may or may not be related to all this: https://github.com/adafruit/circuitpython/issues/10002
not related to serial stuff, but might be related to startup bugs you already fixed or will fix
mortal kernel
Shiny. I'm going to take a wild leap and fix the .ld fragments and patches for the C6 to see if it helps #10296.
(I honestly don't think parse_compile_execute is broken.)
tulip sleet
well, it could be some kind of buffer overflow: the accumulated raw REPL buffer is not allocated properly
mortal kernel
That's possible, too. I believe I've (almost) eliminated bad code gen by building with various -O settings and getting the same results.
tulip sleet
For instance, the SAMD21 bug occurs with a longer busio.UART() call; I didn't reproduce with a shorter call. I'll see if I can make it happen in the raw REPL.
mortal kernel
For instance, the SAMD21 bug occurs with a longer `busio.UART()` call; I didn't ...
I can now reproduce #10296 using raw REPL.Trapping and tracing through the USB serial code, I'm not seeing any buffer overflows. In usb_serial_jtag.c:_copy_out_of_fifo() there is a fixed sized buffer on the stack, but it is protected by a length ceiling check.There is UART driver patch code in ESP-IDF that CP is missing, so I'm going to look into that next.
One other thing that may be a clue: To reproduce the problem I have to run two small programs. The first is:
try:
import storage
print(storage.getmount("/").readonly)
except ImportError:
print(False)
I run this twice followed by 1 or more iterations of:
import os
import time
contents = os.listdir("/")
for item in contents:
result = os.stat("/" + item)
print(item, result[0], result[6], result[9])
Until the fault occurs.
The fault is a secondary effect of a spurious Unimplemented error that's getting thrown.
Here's the REPL of a failure:
Adafruit CircuitPython 10.0.0-alpha.2-31-gd5f89a1760-dirty on 2025-04-27; Adafruit Feather ESP32-C6 4MB Flash No PSRAM with ESP32C6
>>>
raw REPL; CTRL-B to exit
>OKFalse
>OKFalse
>OKsd 16384 0 946685558
settings.toml 32768 0 946685558
code.py 32768 22 946685560
lib 16384 0 946685560
boot_out.txt 32768 226 946685562
>OKsd 16384 0 946685558
settings.toml 32768 0 946685558
code.py 32768 22 946685560
lib 16384 0 946685560
ESP-ROM:esp32c6-20220919
Build:Sep 19 2022
rst:0xc (SW_CPU),boot:0xb (SPI_FAST_FLASH_BOOT)
Saved PC:0x4001975a
SPIWP:0xee
mode:DIO, clock div:2
load:0x40875730,len:0x128
load:0x4086c110,len:0xc34
load:0x4086e610,len:0x2458
entry 0x4086c110
Auto-reload is off.
Running in safe mode! Not running saved code.
You are in safe mode because:
CircuitPython core code crashed hard. Whoops!
Hard fault: memory access or instruction error.
Please file an issue with your program at github.com/adafruit/circuitpython/issues.
Press reset to exit safe mode.
is it actually necessary to use the raw REPL to provoke it, or if you used the regular REPL, would it also happen?
and suppose you defined a function to run the listdir, and then you called that function multiple times?
mortal kernel
James provoked it with regular REPL yesterday on a C3. I've been using raw, but I don't think that the method of getting the code into the interpreter is a factor. I've repeated the failure enough times that I can see it fail each time on a different random permutation of the iterator over the os.listdir result.
tulip sleet
I tried the listdir on a SAMD21 several times, no crash.
mortal kernel
Did you run the storage.getmount code I just mentioned?
mortal kernel
Yes. Which is unexplained, at least.
tulip sleet
does it happen if you don't include the try-catch in the first part?
mortal kernel
Didn't try. That's an interesting thought, let me try...
tulip sleet
and if it needs try-catch, then would any uncaught try-catch work? (it should be uncaught, since storage is present)
mortal kernel
Still fails w/o the try/catch.
Let me repeat w/o it altogether just to be sure...
It fails w/o the storage.getmount. My bad.
tulip sleet
on a C3 QT Py, I am not provoking the problem with an unguarded getmount() (twice), and then the listdir loop multiple times (using paste mode: ctrl-E)
mortal kernel
He provoked it with help("modules")
tulip sleet
ok, just got it to happen with the listdir on the first try, and nothing else, in raw mode, right after a hard reset
also provoked with ctrl-E paste mode first time
mortal kernel
It moves around and has moods.
tulip sleet
provoked leaving out the print()
provoked leaving out the import time
provoked first time every time after a hard reset in the above
import os
contents = os.listdir("/")
for item in contents:
pass
does NOT provoke
mortal kernel
Any idea what it might be? I've got very reliable JTAG up, so I'm open for testing ideas.
tulip sleet
i've gotta take my son to swim practice
not really! but look at os.stat code. Also if it's on the first iteration of the loop. and does it happen if you just call os.stat multiple times without the loop
could be some iterator setup issue
or some issue with os.stat result building or something
mortal kernel
All good thoughts. I'll let you know what I find. Thank you for your help on this. I know we'll nail it.
tulip sleet
also try it all on one line with semicolons:
import os; contents ...; for ...: print()
and try just assigning the result instead of printing it
or not even that, just calling os.stat
full plume
<@375390851595108353> what were you building that you saw the `help("modules")` ...
it was eightcc's CP fork
tulip sleet
@mortal kernel OK, here is a really simple reproducer:
Adafruit CircuitPython 10.0.0-alpha.2 on 2025-04-04; Adafruit QT Py ESP32C3 with ESP32-C3FN4
>>> [type ctrl-E]
paste mode; Ctrl-C to cancel, Ctrl-D to finish
=== for x in 2: pass
=== [type ctrl-D]
ESP-ROM:esp32c3-api1-20210207...
has to be in paste mode
notice there is an error here
but there doesn't have to be:
import os
for item in [1,2,3,4]: os.stat("/")
in paste mode will do it too. But the list has to be at least 4 items long
oops, I'm called away to play Sorry
mortal kernel
Yes, it doesn't take much. I can get the NotImplementedError: opcode failure with import os; os.listdir("/").
tulip sleet
I have an idea; tell you in a bit
tulip sleet
Yes, it doesn't take much. I can get the `NotImplementedError: opcode` failure w...
all these things have something to do with exceptions, including for loops, which raise StopIteration when finished
and the problem is on the RISC-V boards, so I am thinking there is something wrong with the nlr code py/nlr*.*
there are assembly language impls for most architectures, but not for riscv
ls nlr*
nlraarch64.c nlr.c nlr.h nlrmips.c nlrpowerpc.c nlrsetjmp.c nlrthumb.c nlrx64.c nlrx86.c nlrxtensa.c
Brilliant! That's a strong idea.
tulip sleet
the non-impl-specific code uses setjmp I think
I don't have time to debug this now, but I think it's a lead
mortal kernel
I'll take a look. The other threads I was pulling all lead nowhere. The patches are not an issue. The .ld fragments made no difference when I brought them up to date. I've learned a lot about how the interpreter works, but it hasn't gotten me any closer to solving this one.
tulip sleet
I'll take a look. The other threads I was pulling all lead nowhere. The patches ...
did 5.4.1 update the riscv gcc? Maybe there is some change there. Also worth checking out PR's and issues in micropython, because they did the same upgrade recently
they may have addressed this already
hmm, well, this manifests before 5.4.1, I forgot
but still, same idea, check upstream on this for riscv especially
mortal kernel
I'll look into any gcc changes. Just gonna say that.
manic glacierBOT
mortal kernel
I don't have time to debug this now, but I think it's a lead
Micropython implements native nlr for riscv. They're just getting started on ESP-IDF 5.4.1: https://github.com/micropython/micropython/pull/16760 . Nothing of substance in the PR yet.
@tulip sleet The PR adding native NLR for riscv is pretty much self contained. As an experiment, I'll add it to CP. The Micropython PR is here: https://github.com/micropython/micropython/pull/15094 .
lone sandalBOT
tulip sleet
@mortal kernel 👆 the simplest example is just an unassigned variable, like
paste mode; Ctrl-C to cancel, Ctrl-D to finish
=== x
=== [type ctrl-D here]
or 1/0 , which raises ZeroDivisionError
mortal kernel
I think your idea about nlr might be the answer. I'm pulling in the MicroPython riscv native support now...
tulip sleet
i could not reproduce on other serial-only boards like plain ESP32
(I added some stuff to the issue post)
either that or it is something about CIRCUITPY_ESP_USB_SERIAL_JTAG = 1 in particular. That is not set on the ESP32 boards
there are also some C3/C6 boards where CIRCUITPY_ESP_USB_SERIAL_JTAG = 0
notably espressif_esp32c3_devkitm_1_n4, which I have and will test for the crash (I think it did crash)
yup, it crashes too, with 10.0.0-alpha.2. This board has a separate USB-serial chip and does not use the native C3 USB-serial capability
mortal kernel
@tulip sleet I merged native riscv native nlr, made certain that MICROPY_NLR_SETJMP was forced to 0, built clean, flash erased, but the failure persists.
tulip sleet
ahhhhhhh 😦
mortal kernel
I heard that all the way here on the left coast.
I'm going out for my walk and a deep think. I'll be back shortly.
lone axle
It appears that the update-bundles workflow on adabot has failed the last few days https://github.com/adafruit/adabot/actions/runs/14700701652 I am looking into it to see if I can figure out why
slender iron
@mortal kernel @tulip sleet Did you try https://github.com/adafruit/circuitpython/pull/10291 ? That is a fix for thonny
mortal kernel
<@849815081251766292> <@329766224093249548> Did you try https://github.com/adafr...
Giving it a try now...
tulip sleet
<@849815081251766292> <@329766224093249548> Did you try https://github.com/adafr...
the bug we are looking it is replicable in 10.0.0 alpha.2, I think before the working-dir changes
I think we reduced #10296 to #10298
The bug you are fixing may be provoking #10296, but I think it's due to the for loop raising StopIteration
and not due to the filesystem
slender iron
is the idf 5.4.1 update in?
tulip sleet
you mean is it merged?
slender iron
yup. this bug seems like a distraction from the idf merge
tulip sleet
it originally looked like it wsa due to the merge. we can continue with the merge
slender iron
if it's been around but not many folks are hitting it, then I don't think its a super high priority
lone axle
<@&356864093652516868> the weekly meeting will occur here on discord in about 1 hour at the usual time 11am US pacific / 2pm US eastern. We look forward to hearing from everyone. You can add your notes to the doc ahead of time if you'd like: https://docs.google.com/document/d/1kT1f2pPZgEUXu0J9QgaDrGnV59SNDoVBhh-sEESoSoQ/edit?usp=sharing
mortal kernel
I just applied the commit Scott suggested, and so far I can't get a fail. I have no idea why this would be the case, but there it is.
I'm going to test with code.circuitpython.org now...
tulip sleet
you mean with just x in paste mode?
mortal kernel
@tulip sleet @slender iron No, that didn't fix it. I just hadn't tested it enough times.
tulip sleet
do you think the 5.4.1 merge is in good enough shape to merge? The C3/C6 problem is older than that, so it's not a blocker on the merge
mortal kernel
It's blocking tests of boards based on those parts, but otherwise I think it's good to go. I'll need to push the new 5.4.1 branch to the Adafruit ESP-IDF repo first, and then finalize the PR to CircuitPython so that it picks up that repo for ESP-IDF.
tulip sleet
could try to finish that off and then go back to the C3/C6 thing
not sure if I would do an alpha.3 tomw or not
full plume
Just FYI, when I copy firmware.uf2 to the UF2 loader drive (S2MINIBOOT after special boot, not CIRCUITPY) from my Windows 11 WSL mapped drive ( "\wsl.localhost\Ubuntu\home\james\git\circuitpython\ports\espressif\build-lolin_s2_mini\firmware.uf2" ) using Windows Explorer drag & drop, I get the following error :
However, it does appear to successfully install (boot_out.txt shows correct build date and it starts successfully), and if I copy the same firmware.uf2 to a normal Windows drive/directory, and then copy that file to S2MINIBOOT, I don't see the error. I suspect this is probably some weird interaction between Windows UF2 magic and WSL in general, as opposed to anything directly related to CP, but thought it worth mentioning.
tulip sleet
the drive goes away rather suddenly after you load something onto the BOOT drive, and it may be upset about that
windows an the BOOT drive is often a source of pain, especially with third-party drivers and utilities of various kinds. See https://learn.adafruit.com/welcome-to-circuitpython/troubleshooting#bootloader-boardnameboot-drive-not-present-2978448 and following, which you may have reviewed already. Not necessarily anything mentioned there, but there are unexpected nuisances
full plume
@mortal kernel OTOH, with that build (lolin_s2_mini, pulled from your fork, commit f9ab642fddc034f55f40a7cd8357374c6081d7b3 (HEAD -> main, origin/main, origin/HEAD) from April 23 ) some of my project code is failing :
File "code.py", line 6, in <module>
File "/lib/TerrainTronics/Demos/Cilgerran/SimpleTest.py", line 48, in demoMain
File "/lib/TerrainTronics/Demos/DemoBase.py", line 23, in run
File "/lib/LumensalisCP/Main/Manager.py", line 379, in run
File "asyncio/core.py", line 350, in run
File "asyncio/core.py", line 249, in create_task
AttributeError: 'TaskQueue' object has no attribute 'push_head'
Code done running.
The asyncio I'm running is from adafruit-circuitpython-bundle-9.x-mpy-20250307, do I need to get (or build) an updated adafruit-circuitpython-bundle?
crimson ferry
push_head was recently removed https://github.com/adafruit/Adafruit_CircuitPython_asyncio/pull/71
tulip sleet
<@849815081251766292> OTOH, with that build (lolin_s2_mini, pulled from your f...
You need the latest version. I removed push_head and some other deprecated methods. It's just push now.
jaunty juniper
you always want latest (or use circup)
tulip sleet
weekly meeting starting in three minutes in CircuitPython voice channel. See pinned messages for notes document
slender iron
Gotta learn when and how to use claude code just like any other tool.
lone axle
turbid radish
Yes, please pass good content to me through the week cpnews@adafruit.com
turbid radish
Thanks Dan!
lone axle
Thanks for hosting Dan, have a great week everyone.
tulip sleet
thanks all for attending!
crimson ferry
is the Google doc wiped every week, or is there some archive?
thorny jay
That was fast.
slender iron
is the Google doc wiped every week, or is there some archive?
archive is here: https://github.com/adafruit/adafruit-circuitpython-weekly-meeting
docs are left as-is with edit permissions revoked from the link
tulip sleet
is the Google doc wiped every week, or is there some archive?
they're also linked from the YouTube description
tulip sleet
Here is the notes document for next Monday’s CircuitPython Weekly Meeting. It is at the normal time of 11am Pacific / 2pm US Eastern here on Discord. Add your hug reports and status updates to the document before the meeting. If you are unable to attend but would still like to contribute, feel free to add your notes and we’ll read them off during the meeting. Hope to see you there! <@&356864093652516868>
https://docs.google.com/document/d/1doBXpUIa2VdBVaOj7NF-u8x41KzRPwHtfXAFTn63_2A/edit?usp=sharing
slender iron
tulip sleet
https://blog.adafruit.com/2025/04/28/improving-garbage-collection-time-in-circui...
this is great!
full plume
You need the latest version. I removed `push_head` and some other deprecated met...
Requiring upgrading to the 10.x bundle for CP 10.x firmware is certainly reasonable. Pruning deprecated methods is good, although that method was still in use in the 9.x bundle as of two months ago I might have to upgrade my CIRCUITPY lib/* "deployment process" (which is inevitable) sooner rather than later to deal with multiple bundle versions. But that's only a very minor inconvenience since I haven't officially released anything yet - mostly I just tend to be a bit obsessive about backwards compatibility.
jaunty juniper
Requiring upgrading to the 10.x bundle for CP 10.x firmware is certainly reasona...
the bundles are identical, the 9/10 difference is only the mpy format (which is also identical with 9/10)
the removed methods were deprecated in 9, they are removed in 10. Meaning the latest version of the library supports both 9 and 10. That makes the bundle backward compatible, not forward compatible (older versions of the bundle might not be compatible with latest CP, latest version of the bundle is compatible with currently supported versions of CP, which are 9 and 10).
note that "version" here refers to bundle releases. The latest release is 20250425 right now.
so:
adafruit-circuitpython-bundle-9.x-mpy-20250425.zip
adafruit-circuitpython-bundle-10.x-mpy-20250425.zip
Circup always uses the latest.
onyx hinge
huh TIL that python socket library can be used for CAN https://docs.python.org/3/library/socket.html#socket.AF_CAN https://www.kernel.org/doc/html/v4.17/networking/can.html -- for circuitpython we (I) just made a custom API.
fleet hollow
I'm working on developing a the definition for a board which features an SSD1306-compatible OLED. I'm able to get it working within a script using the following code:
spi = board.SPI()
dc = digitalio.DigitalInOut(board.OLED_DC)
reset = digitalio.DigitalInOut(board.OLED_RESET)
cs = digitalio.DigitalInOut(board.OLED_CS)
oled = adafruit_ssd1306.SSD1306_SPI(72, 40, spi, dc, reset, cs)
But I'm having trouble getting it to initialize as a busdisplay within board.c. Any advice as to how best to initialize the display?
stuck elbow
I'm working on developing a the definition for a board which features an SSD1306...
what's the problem exactly?
fleet hollow
I'd like the display to initialize automatically (ie: board.DISPLAY) like it does with the MacroPad RP2040 and other boards with built-in displays.
Right now, that doesn't work at all. 🤷
I probably need to study the display_init_sequence closer to make sure that's happening correctly, but I just wanted to probe to see if anyone had better luck with SSD1306 OLEDs within board definitions.
stuck elbow
I don't think there is anything special for the ssd1306, you just pass the same parameters as you would to the busdisplay call
fleet hollow
Okay. I'll keep playing around with it. 👍
stuck elbow
here is an example of a board.c for a board with an ssd1306 display: https://github.com/adafruit/circuitpython/blob/main/ports/espressif/boards/01space_lcd042_esp32c3/board.c
except that one uses i2c not spi
but the init sequence will be the same
fleet hollow
but the init sequence will be the same
I messed up one of the address registers. 🤦♂️ Thank you for pointing me in the right direction!
full plume
@mortal kernel I did a git pull from your fork and rebuilt lolin_s3_mini, and it's back to failing to start (no CIRCUITPY). Tried again after installing the new bootloader.bin and still fails. Tried installing using the online installer at https://circuitpython.org/board/lolin_s3_mini/ for 10.0.0-alpha2 and it also failed.
lone axle
@slender iron I'm working on FruitJam OS and thinking about ways we could store the icon and app title. Right now the title gets automatically set from the folder name, but that leads to long titles that can include other device names like "Metro" since the guides launched "disquised" as metro projects.
We currently automatically look for theapp/icon.bmp for its icon. I am thinking we could also check for theapp/apptitle.txt or similar, and if it exists use its contents for the title in the launcher. That way it can be configured to be just "Snake" or "Matrix" etc. without having to make different folder structures with more generic names inside the learn repo. Or would it makes sense to do something like metadata.json and it can contain the title and an icon filepath that way if an app wanted to define their icon as something other than icon.bmp specifically they could?
slender iron
<@252717193496756235> I'm working on FruitJam OS and thinking about ways we coul...
ya, I think json is more extensible
mortal kernel
@slender iron I've merged tip of main into my branch for the ESP-IDF 5.4.1 update and I'm seeing an unrelated CI zephr-cp failure: https://github.com/adafruit/circuitpython/pull/10267
tulip sleet
<@252717193496756235> I've merged tip of main into my branch for the ESP-IDF 5.4...
let me re-run that, it looks like a CI failure, maybe. Or you can rerun it. https://github.com/adafruit/circuitpython/actions/runs/14736412000/job/41363987479?pr=10267 choose "Re-run jobs", click for ddropdown, choose "Re-run failed jobs"
mortal kernel
let me re-run that, it looks like a CI failure, maybe. Or you can rerun it. <htt...
Re-running failed job...
mortal kernel
let me re-run that, it looks like a CI failure, maybe. Or you can rerun it. <htt...
Worked on the re-run. Now I know that with CI if at first you don't succeed...
Just for completeness, I'll be running some spot checks with the current build artifacts, then it should be good to go.
full plume
ya, I think json is more extensible
Probably obvious since similar things already happen with settings.toml , but IMHO "both" is even better. Use the same type of automatic naming already in place for defaults, but allow them to be overridden by a configuration file ( for which .json is often a great choice ).
cerulean fractal
I have a challenging board I want to make a port of circuitpython for. I've hit a roadblock porting it the traditional way -- it uses the W channel of LP5562 RGBW controller to control its LCD backlight. Zephyr has a driver for that controller. I guess what I'm asking is, how far along is the zephyr port? Is it currently capable of delivering all the bells and whistles that come with builds of circuitpython for LCD-equpped ESP boards?
tulip sleet
I have a [challenging board](https://docs.m5stack.com/en/core/AtomS3R) I want to...
The zephyr port builds but most functionality has not be hooked up yet. It will be a while. Can you make a separate driver library that does the right thing for brightness?
cerulean fractal
The zephyr port builds but most functionality has not be hooked up yet. It will ...
I could, it would take a lot of learning
tulip sleet
I could, it would take a lot of learning
I meant in Python, not C, though
so the LP5562 is just for the backlight?
cerulean fractal
Can a python library be hooked to do stuff independently from user code, like for the stuff that normally goes into board.c?
jaunty juniper
it could be used in user code to adjust the brightness
cerulean fractal
so the LP5562 is just for the backlight?
The funky part is that it's an RGBW controller split into RGB and W. There's an RGB LED and backlight. So the code to make it work would be just for this board.
So, a screen will be dark until user code is ran
jaunty juniper
if it's as simple as writing an I2C register to set the output it could be set to a default on boot
(maybe)
cerulean fractal
Alright, will try some "dumb" solutions. Ideally though, I was hoping I could produce a port I could upstream
jaunty juniper
a python library could be frozen into the board and provide the base access to the hardware
cerulean fractal
Is there such thing as a reverse binding, as in, can it be hooked into the C code somehow?
stuck elbow
you can run python functions from c code, but not inside interrupts
slender iron
Is there such thing as a reverse binding, as in, can it be hooked into the C cod...
what protocol do you talk to the rgbw controller?
jaunty juniper
it's an I2C chip
I found a driver there https://github.com/rickkas7/LP5562-RK it has a starting sequence that sets current levels and default PWM values and a bunch of things (it can apparently run a program that's a sequence of animations)
but then we'd only need that for the backlight
void LP5562::setW(uint8_t white) {
(void) writeRegister(REG_W_PWM, white);
}
Just do busio_i2c from board_init then
cerulean fractal
Yes, I found this example
slender iron
what do you want an example of?
tulip sleet
these files all do something with i2c:
ports/espressif/boards/lilygo_twatch_s3/board.c
ports/espressif/boards/m5stack_core2/board.c
ports/espressif/boards/m5stack_cores3/board.c
ports/espressif/boards/m5stack_stick_c/board.c
ports/espressif/boards/m5stack_stick_c_plus/board.c
@tulip sleet I'm hitting a new snag with ESP32-WROOM-32E, TLS, and ESP-IDF 5.4.1: -12288 error. Investigating...
Decodes as MBEDTLS_ERR_X509_FATAL_ERROR
tulip sleet
i will try a build
mortal kernel
The board is an Adafruit Feather Huzzah32.
@tulip sleet Despite the Huzzah32 being a 4M flash board, there is plenty of room for _bleio. I enabled it for testing, is there any reason I shouldn't leave it enabled in the PR?
tulip sleet
you can leave it enabled, but is there an ota1 partition?
is this the old HUZZAH32 or the ESP32 V2?
mortal kernel
The ESP part is labelled ESP32-WROOM-32E. There's no labeling that would indicate it's a V2. How would I tell?
tulip sleet
on the bottom of the board
mortal kernel
Just Huzzah32. The Adafruit part number on the packaging is 3405. I've had it for quite a while.
tulip sleet
this is the current partition table, so yes, you can leave it on:
# ESP-IDF Partition Table
# Name, Type, SubType, Offset, Size, Flags
nvs,data,nvs,0x9000,20K,
otadata,data,ota,0xe000,8K,
ota_0,app,ota_0,0x10000,2M,
user_fs,data,fat,0x210000,1984K,
but, I had trouble with BLE on, on the C3 board. It would bootloop
mortal kernel
Yes, I've noted that issue. There are a bunch of BLE related .ld fragments we're currently not including. Of course, the C3/C6 boards aren't working yet with 10.
full plume
<
tulip sleet
<@329766224093249548> I'm hitting a new snag with ESP32-WROOM-32E, TLS, and ESP-...
I am not seeing this error, either on HUZZAH32 or Feather ESP32 V2. I am running the attached file, with a populated settings.toml, and adafruit_connection_manager.mpy and adafruit_requests.mpy in lib/
I built from the latest commit in your issue-10191 branch
getting the libraries onto the board is a pain. I use the Thonny package manager.
mortal kernel
Weird. I'm running that exact Python code. The only differences are that I copied the libraries with code.circuitpython.org and enabled BLE.
tulip sleet
i would say disable BLE and try again
i couldn't get code.circuitpython.org to see the board for some reason :/
mortal kernel
I've had intermittent trouble, too. And not with just that board. Sometimes powering down and rebooting everything does the trick.
mortal kernel
@tulip sleet It was having _bleio enabled. With it disabled, no bug. I'll be leaving it off for now.
turbid radish
Submit your questions to Ladyada for Ask an Engineer https://discord.com/channels/327254708534116352/1280611628081221785/threads/1367150445193203752
slender iron
@mortal kernel what is the state of the idf update? I want to grab dev boards today I'll need to help fix it up.
mortal kernel
<@849815081251766292> what is the state of the idf update? I want to grab dev bo...
It’s good to go. I’m on the road , but will be back home in an hour. C3/C6 continue to have the interpreter crash, but it’s not related to 5.4.1.
lone axle
@slender iron I am thinking of making a helper function that offers an easy one-line API to initialize the display at a given size. As I'm working on the launcher I'm realizing that different apps leave the display in a state that the launcher is not expecting which can cause it to break.
I intend to do a pass through all of the learn project apps and work on some tweaks that will make their integration with the launcher go smoother. But ultimately the launcher can't really gurantee that the apps are going to leave the display in a state that it expects, so it will need logic to re-initialize it in case it was released. Realistically any app could need similar logic since they also cannot be certain that the display will be initialzied or in the mode they expect when they're launched. Making a request_display_config(height, width) or similar helper that contains all of the boiler plate to initialize it and set it onto the supervisor.runtime instead of having to copy/paste that into the launcher and other apps seems convenient to me. I am imagining it having hard-coded pins so it it would only work for FruitJam and Metro RP2350, or any future devices that share their pin naming.
Do you have any thoughts about this plan generally, or about where this code should live?
slender iron
<@252717193496756235> I am thinking of making a helper function that offers an e...
I think we'll want a fruit jam library. kinda like the portal libraries
it can init audio then too
slender iron
It’s good to go. I’m on the road , but will be back home in an hour. C3/C6 conti...
kk, will review it when I have time
gonna make sure I have esp devkits in my backpack for testing too
slender iron
@mortal kernel what's next on your list?
mortal kernel
The C3/C6 interpreter failure. Of course I'm open to anything else you think is a higher priority.
slender iron
The C3/C6 interpreter failure. Of course I'm open to anything else you think is ...
That's fine just pick a time cap for yourself. I'll try and look over the 10.x issues tomorrow to have a better feeling. I'm going to start working from there too. Email and reviews will be higher priority though.
mortal kernel
OK, how about the rest of today and if it's not resolved by tomorrow afternoon I'll move on to something else.
slender iron
k, sounds good. I'm hoping to have time tomorrow afternoon during nap. Definitely look at the 10.x issues for other things.
orchid basinBOT
mortal kernel
@slender iron I've found the code change that is causing the C3/C6 interpreter failure. I'm updating the issue (#10298) with the details. In a nutshell, it looks like a small change to parse_compile_execute is tickling a Risc-V code gen bug in gcc. I'm going to spend a bit more time looking at the generated code so that I can see if there's anything in gcc bug reports that matches.
I do have a change that moves a declaration/assignment out of the scope of a conditional that has the same effect as the failing code but does not provoke the error. I'll submit that as a PR.
onyx hinge
Advice Animal answers the question of what do we do after running cookiecutter, or how to we avoid telling people we support to run a sed script.
https://github.com/advice-animal/advice-animal
I wonder if this would have any application to circuitpython libs use of cookiecutter
fleet hollow
I'm looking to add these two boards, the Thumby and Thumby Color by TinyCircuits: https://github.com/adafruit/circuitpython/pull/10303. The original devices use the USB VID/PID of 0x2E8A/0x0005 for the standard MicroPython firmware. Should I go through the process of requesting a PID from the Raspberry Pi Foundation on TinyCircuits' behalf, or should I reach out to them to do that process? If the later is the case and they are not interested in pursuing this, does that halt the process of adding those devices to CircuitPython? Thanks!
slender iron
I'm looking to add these two boards, the Thumby and Thumby Color by TinyCircuits...
I'm not sure of the Raspberry Pi Trading (not the foundation) policy for third party requests. It would fit with pid.codes though
@mortal kernel Thanks for looking into that crash. Bummer it still happens with 5.4.1. I think it'd be best to do something else. Maybe more info will crop up later. I'll take a look at issues and let you know what would be good for you next
fleet hollow
I'm not sure of the Raspberry Pi Trading (not the foundation) policy for third p...
For now, I've sent out a message to the company for comment.
mortal kernel
<@849815081251766292> Thanks for looking into that crash. Bummer it still happen...
Since the Python interpreter crash was attributable to the gcc Risc-V code gen, it's likely that 5.4.1's change from riscv32-esp-elf-13.2.0_20240530 used by 5.3 to riscv32-esp-elf-14.2.0_20241119 is the cause of the crash with the workaround.
slender iron
Since the Python interpreter crash was attributable to the gcc Risc-V code gen, ...
I'm surprised that code gen is the culprit
don't we see the error without 5.4.1 too?
mortal kernel
I'm surprised that code gen is the
slender iron
@mortal kernel these other c3 bugs may be good to tackle since you are in that brain space
mortal kernel
<https://github.com/adafruit/circuitpython/issues?q=is%3Aissue%20state%3Aopen%20...
I don't have a C3 board on hand. Maybe I could get one over-nighted for Monday?
lone axle
@slender iron I've been working a little on the editor today. In doing so I noticed that with lvfontio.OnDiskFont if I use the font.bitmap from it on a TileGrid and then try to set tiles like tg[0,0] = 45 or similar it doesn't result in anything visible in the TileGrid. Doing the same thing with a TileGrid and terminalio.FONT does result in visible glyphs.
Is there some way to get the OnDiskFont to work directly with TileGrid? I have gotten it working with Terminal and hooked that up to the editor code, but the refresh / react to keyevent rate is quite slow, I'm brainstorming ways it might be able to be sped up by re-making the editor in different ways which led me to trying to use TileGrid with OnDiskFont.bitmap directly.
slender iron
I don't have a C3 board on hand. Maybe I could get one over-nighted for Monday?
Not that urgent but: https://a.co/d/f4Q9gIs
mortal kernel
Not that urgent but: <https://a.co/d/f4Q9gIs>
OK, I've got some other stuff in my Amazon basket. I'll take them up on their Prime 30 day offer and then cancel it immediately.
slender iron
could do digikey too if you want
theres no rush to do c3 though
there are other 10 issues you can do
slender iron
<@252717193496756235> I've been working a little on the editor today. In doing s...
OnDiskFont is trickier because it does reference tracking. I'd just use a normal font for now
I'd expect typing to be fast but scrolling slow
lone axle
Okay, I'll go back to default font and see how it does with tat. So far typing is quite slow. If I type "circuitpython" it takes on the order of 15-20 seconds for it to finish typing out each letter 1 by 1 on the display. It may be made worse by some of the things I tried to do to resolve it though, I'll back up a few steps with the default font and try to get a better baseline tomorrow.
mortal kernel
there are other 10 issues you can do
I'll have the C3 middle of next week. In the meantime, I'll pick up with the less comprehensive #9693 on Monday. I'll take on the remaining 10 issues from easiest to hardest then, too.
slender iron
Okay, I'll go back to default font and see how it does with tat. So far typing i...
that seems really bad. I feel like when I poked it it wasn't so slow. Maybe is doing a full redraw of something
loading the terminal at the start it over a second of time
slender iron
I've drafted 10.0.0-alpha.3 notes. Will release it tomorrow
manic glacierBOT
full plume
I don't have a C3 board on hand. Maybe I could get one over-nighted for Monday?
I have two C3 boards - a Lolin C3 Mini and a LilyGo Mini D1 Plus. I can at least do simple testing if you have anything you'd like me to try while you're waiting.
mortal kernel
I have two C3 boards - a Lolin C3 Mini and a LilyGo Mini D1 Plus. I can at leas...
Thanks! I’ll let you know, probably Monday.
lone axle
@slender iron I am back to default font now and the editor is much more responsive. I had also started converting the cursor to use TilePaletteMapper but am finding that is noticably slower than the existing "1 character label on top" cursor. Looking back into TPM I realize it causes the whole TileGrid to refresh with each change instead of just 1 tile. The easiest solution that comes to mind for me is to have TPM accept the TileGrid as an init argument, TileGrid already has mechanics inside of it for handling dirty area based on changes to single tiles, so TPM can make use of that on behalf of its TileGrid. a new tilegrid argument could replace width, and height arguments since they can be obtained from it.
I'm not sure if this is the best approach though, one bit that seems like it could get awkward is there would be a circular reference between them. TPM holds a reference to TileGrid and TileGrid holds a reference back to the TPM, code trying to follow them could go infinite. Do you have thoughts or ideas on how best to make TPM more effecient than causing a full refresh for each change?
slender iron
<@252717193496756235> I am back to default font now and the editor is much more ...
I think tying the two together is a good idea. The circular reference should be fine. It'll still get cleaned up if neither is referenced by something else.
lone axle
okay cool, I'll start working in that direction.
One more unrealted question, how hard do you think it would be to implement sys.argv for use with set_next_code_file()? In the short term I am intending to hack something together with temporary hidden files on CPSAVES that hold a list of arguments, but in the longer term it might be nice to share the CPython API for arguments.
slender iron
One more unrealted question, how hard do you think it would be to implement `sys...
Interesting! I hadn't thought of doing it that way. I don't think it'd be too hard. We could allocate a json encoded string with port_malloc and then parse it when reading it back into the new VM
lone axle
I'm looking at msgpack for the temp files, it should be slightly more compact than JSON if it matters.
slender iron
I'm looking at `msgpack` for the temp files, it should be slightly more compact ...
I doubt it does. Three characters "", per string isn't much. I'd expect json to be enabled on more boards than msgpack
jaunty juniper
there's 504 boards with json, 501 with msgpack, 500 with both
manic glacierBOT
candid sun
<@&356864093652516868> We'll have our weekly meeting in about 90 minutes from now in this text channel and in the circuitpython voice channel. Please take the time to add your notes in advance to the document: https://docs.google.com/document/d/1doBXpUIa2VdBVaOj7NF-u8x41KzRPwHtfXAFTn63_2A/edit?tab=t.0
slender iron
@mortal kernel is that crash because a pointer to collect is being stored in a register?
mortal kernel
<@849815081251766292> is that crash because a pointer to collect is being stored...
The pointer to the compiled function is temporarily held in a callee-saved register across the call to gc_collect. The disassembled code is in #10298.
slender iron
The pointer to the compiled function is temporarily held in a callee-saved regis...
We should make sure and collect those pointers. On other architectures we do
mortal kernel
We should make sure and collect those pointers. On other architectures we do
New issue?
mortal kernel
OK, I'll create one. For now, though, the fix I pushed works well for the RISC-V case that was biting us.
slender iron
That is small enough I may be able to pick it up too
jaunty juniper
wait for somebody to actually make a board called "pycache" (like a treasure box with a microcontroller in it or something)
slender iron
that's what I get for running the script locally 🙂
I removed it from the pr
along with crickit m0
lone axle
Thanks for hosting Liz, have a great week everyone.
candid sun
Here is the notes document for next Monday’s CircuitPython Weekly meeting. It is at the normal time of 11am Pacific / 2pm Eastern here on Discord. Everyone is encouraged to attend! Please add your hug reports and status updates even if you’ll be attending the meeting - it’s super helpful! If you are unable to attend but would still like to include updates, feel free to include them in the notes and we’ll read them off during the meeting. Hope to see you there! <@&356864093652516868>
https://docs.google.com/document/d/1SeffJXPObzSCRRPUpaYE8T5QIz8m8bc2UfMYpH9Rop0/edit?tab=t.0
slender iron
I'm working on improving our download logging. Here is the language breakdown of requests since this morning: ```en_US 128
en_GB 21
fr 15
de_DE 9
ru 6
it_IT 5
es 3
nl 2
tr 2
pl 1
ja 1
ID 1
Version breakdown: 9.2.7 168 10.0.0-alpha.4 23 10.0.0-alpha.2 2
slender iron
@candid sun I never PRed a fix to the TLV driver. It needs a tweak to how it reads registers
it needs to call write_then_readinto (iirc) instead of separate write and read
otherwise it won't do it right
short marsh
@slender iron Hi, I'm the developer who submitted this recent PR that you commented on: https://github.com/adafruit/circuitpython/pull/10309
This is my first attempt at trying to contribute to an open project on GitHub, so please forgive any mistakes I may be making in the process, I may need some handholding.
It still looks like my PR is not passing the automated tests (its still failing 6 checks). I'm not familiar with what these checks are, or what I changes I can make to ensure they pass. Can you give me some guidance here?
My code updates only touch three files, and the compiled version with the updates works as expected, so I'm not sure where all these checks would be failing based on my updates.
jaunty juniper
<@252717193496756235> Hi, I'm the developer who submitted this recent PR that yo...
one error is in the docs, try make html at the root of the repository to see (it's a little long). The docs are python docs inside C comments, there is at least a syntax error where you used "" instead of """ for the multiline dosctring.
And I'm not sure how you get it to compile locally ? There's a missing }, on line 156, also leading to everything after it being indentend
blissful pollen
<@252717193496756235> Hi, I'm the developer who submitted this recent PR that yo...
Hi, I've been doing a lot of audio work lately and can probably help guide you (as can others here). I probably won't have a chance to really look in detail for a couple days though. Offhand the doc is likely failing due to "" instead of """ on line 84 of the shared-bindings MixerVoice.
I think I see the other issue as well. I will try to add a review comment.
jaunty juniper
to find out, you can go to the checks tab or click on the failing checks and scroll through the output to see the errors, like this:
Traceback (most recent call last):
File "/home/runner/work/circuitpython/circuitpython/tools/extract_pyi.py", line 189, in convert_folder
tree = ast.parse(fragment)
File "/opt/hostedtoolcache/Python/3.13.3/x64/lib/python3.13/ast.py", line 54, in parse
return compile(source, filename, mode, flags,
_feature_version=feature_version, optimize=optimize)
File "<unknown>", line 25
"" Sets looping to False if sample is playing, allowing current looped
^^
that's the docs issue
and this:
../../shared-bindings/audiomixer/MixerVoice.c:159:7: error: expected ‘}’ before ‘{’ token
159 | {
-e See https://learn.adafruit.com/building-circuitpython; Adafruit Discord #circuitpython-dev
| ^
../../shared-bindings/audiomixer/MixerVoice.c:156:5: note: to match this ‘{’
156 | { MP_ROM_QSTR(MP_QSTR_end), MP_ROM_PTR(&audiomixer_mixervoice_end_obj)
| ^
../../shared-bindings/audiomixer/MixerVoice.c:163:84: error: expected ‘}’ before ‘;’ token
163 | { MP_ROM_QSTR(MP_QSTR_loop), MP_ROM_PTR(&audiomixer_mixervoice_loop_obj) }, };
| ^
that's the missing },
https://github.com/adafruit/circuitpython/actions/runs/14849585502/job/41690655663?pr=10309#step:7:44
those logs can be a bit difficult to navigate and read, they can be long and the actual issue is not at the end (like in this case the docs error is literally at the top, with an additional "Error: Failed to parse a Python stub from shared-bindings/audiomixer/MixerVoice.c" somewhere in the middle
slender iron
<@252717193496756235> Hi, I'm the developer who submitted this recent PR that yo...
No problem! You are at the right place for help. Thanks to mark and neradoc for their help!
short marsh
@blissful pollen @jaunty juniper Thank you for the information and suggested edits.
The compiling worked on my end, but then I ended up copy/pasting my edits into a newly created branch to attempt a PR. There were some errors in that copy/paste that missed the } - Iesson learned
short marsh
It looks like the suggested changes, and little reformatting has got it down from six to one failing checks: https://github.com/adafruit/circuitpython/pull/10309
- Failed: Build CI / docs (pull_request) - The error is showing
"Error reading SVG /home/runner/work/circuitpython/circuitpython/_build/doctrees/images/66d35c9c51e5252a3e72cb7306d23d236b7dca79/svg-badge.svg: XML parse error: Error domain 1 code 5 on line 1 column 2 of data: Extra content at the end of the document"
Not sure about this one, as my updates don't touch any SVG files
lone axle
It looks like the suggested changes, and little reformatting has got it down fro...
I do not think the SVG issue is related to your change. I'm looking into it now. It seems that file the error is referencing has something wrong with it
lone axle
@short marsh this issue ^ is created for what I think is the root cause of that SVG error. I don't think there is anything for you to do about it in your PR branch for now. We'll have to resolve that within the docs build somehow I believe.
ionic elk
has the flash utilization gone up with the latest circuitpython release?
my builds are 7kb over after rebasing
(on espressif)
mortal kernel
has the flash utilization gone up with the latest circuitpython release?
ESP-IDF 5.4.1 consumes more flash. Which parts are brimming over?
short marsh
<@692457741460963458> this issue ^ is created for what I think is the root cause...
For clarity, would this also be affecting anyone else trying to submit a PR, shouldn't all PRs be failing this if it is a larger issue?
lone axle
For clarity, would this also be affecting anyone else trying to submit a PR, sho...
I believe it can affect other PRs. Any PR that runs the Build CI / docs action will fail because of this issue I believe.
I think there is some logic that determines which action tasks run for the PR based on what files are changed, I'm not sure where it lives, but a PR could skip the docs build such as https://github.com/adafruit/circuitpython/pull/10315 in which case it wont cause the PR check itself to fail it seems.
turbid radish
Post your questions to Adafruit in the chat solicitation for tonight's Ask an Engineer broadcast https://discord.com/channels/327254708534116352/1369396567261315072
?showtimes
digital shoreBOT
Desk of Ladyada - Sunday Evening
JP's Product Pick of the Week - 4pm ET Tuesdays
3D Hangouts - 11am ET Wednesdays
Show & Tell - 7:30pm ET Wednesdays
Ask an Engineer - 8pm ET Wednesdays
John Park's Workshop - 4pm ET Thursdays
Deep Dive w/ Foamyguy - 5pm ET Fridays
FoamyGuy's CircuitPython Stream - 11am ET Saturdays
ionic elk
ESP-IDF 5.4.1 consumes more flash. Which parts are brimming over?
I'm no longer able to compile with debug mode to use the JTAG
I guess I will remove some modules while testing. Any suggestions on which ones will free up the most flash?
slender iron
ulab
mortal kernel
I'm no longer able to compile with debug mode to use the JTAG
Since you're rebuilding, the easiest thing to do is to change the flash partition map. I'm guessing you're targeting a 4M flash device.
CIRCUITPY_LEGACY_4MB_FLASH_LAYOUT = 0 in the mpconfigboard.mk will do the trick. Be sure to completely erase the flash, and if it's a uf2 capable device, flash the latest uf2 loader.
