#circuitpython-dev

1 messages · Page 54 of 1

devout jolt
#

Help me understand adafruit_httpserver. For this code that's from the examples:

import socketpool
import wifi
from adafruit_httpserver import Server, Request, FileResponse
pool = socketpool.SocketPool(wifi.radio)
server = Server(pool, "/static", debug=True)

@server.route("/home")
def home(request: Request):
    print("home route:", request)
    return FileResponse(request, "home.html", root_path="/www")

server.serve_forever(str(wifi.radio.ipv4_address))

When serving the request, the print() is not executed. Why could this be?
Also the root_path argument is not used. (e.g. the full file path the server tries to vend is /static/home.html). From what I can tell, server.py is constructing its own FileResponse instead of using my handler https://github.com/adafruit/Adafruit_CircuitPython_HTTPServer/blob/main/adafruit_httpserver/server.py#L317

In fact, it seems to be ignoring that @server.route(), falling back to the default mapping of route path -> constructed html filename

stuck elbow
devout jolt
#

In retrospect, that example is a little confusing. Is there a way to have a single route of "/home" match the paths ["/home", "/home/", "/home.html"]?

#

Answering my own question, looks like you can add multiple decorators to a function. So this would serve all three URL paths "/home", "/home/", and "/home.html"

@server.route("/home", append_slash=True)
@server.route("/home.html")
def home(request: Request): # ...
devout jolt
atomic summit
#

Yup, this is a question for the Adafruit CP devs , not one for me, sorry. I know Scott has done more on BLE recently I think and others have been working on bug fixes for it, but no idea of the roadmap for it for CP. You can always look at ESP-IDF as that's what everything is built off anyway.

ashen whale
brazen hatch
#

Yep, stm32 running out of flash.

#

The issue is the get_safe_mode() polls during each USB_ thing.

#

I don't know what to cut / reduce to get it to pass CI.

#

This board basically has no flash space, so maybe disable something like gifio or usb_midi?
I do think it's unlikely someone will need them on such a board.

tulip sleet
brazen hatch
#

Ran main vs pr compiles for firebeetle 2 (it's an s3 board), 534224-533904=320 bytes.

#

The only really big thing is the strings CIRCUITPY_USB_*, the whole purpose of the pr.

#

So I can't remove em.

#

They are pretty big, but self-explanatory.

CIRCUITPY_USB_MSC_DEFAULT
CIRCUITPY_USB_HID_DEFAULT
CIRCUITPY_USB_MIDI_DEFAULT
CIRCUITPY_USB_CDC_CONSOLE_DEFAULT
CIRCUITPY_USB_CDC_DATA_DEFAULT
#

And we need 72 bytes to fit. It's not like some rewording will get us to fit.

#

The only idea I have is maybe removing CIRCUITPY_USB_CDC_DATA_DEFAULT since nobody will use it.

#

But it would look weird in the code.

#

Having everything else done and leaving just that out.

tulip sleet
#

I think you could drop the CIRCUITPY_ prefix. I was also thinking of changing DEFAULT to ENABLE

#

as being more self-evident

#

I am still also wondering about this -- it's a second way to do something that is already available by editing a file (boot.py). In this case, it's editing a different file.

brazen hatch
#

If it's disabled in the toml, boot.py may still enable it, and that would sound weird to a beginner I think

tulip sleet
#

i get your thinking there

brazen hatch
#

and 1 * 5 chars isn't that much to hinder the meaning

tulip sleet
#

The whole feature could be toggleable for small boards

brazen hatch
#

Hmm, true, I guess.

#

#if CIRCUITPY_SETTINGS_CONFIGURABLE_ENDPOINTS?

tulip sleet
#

I wouldn't use the word "endpoints" here. That has a specific meaning for USB. They are USB devices (or, really, interfaces, but that's obscure.

#

CIRCUITPY_TOML_USB_DEFAULTS

brazen hatch
#

Sure, imma do that.

#

Thanks!

tulip sleet
brazen hatch
#

Fair enough, consistency is important.

manic glacierBOT
manic glacierBOT
#
dhalbert
[adafruit/circuitpython] New comment on issue #8693: Have usb drive, usb_hid and the rest of the things that can be enabled/disabled during boot be configurable from settings.toml

And it's hard for applications to just set it for the next boot in a clean manner, since there is no such ready-made api.

This could be done with micrcontroller.nvm, and tested in boot.py, or by SleepMemory (if applicable).

A higher level view on this:

We provided settings.toml, and before that .env, to provide settings that needed to be specified before any Python codee ran. This addition is a convenience, but is not really required. Code in boot.py already can do what thi...

manic glacierBOT
#
bill88t
[adafruit/circuitpython] New comment on issue #8693: Have usb drive, usb_hid and the rest of the things that can be enabled/disabled during boot be configurable from settings.toml

This could be done with micrcontroller.nvm, and tested in boot.py, or by SleepMemory (if applicable).

Now that I think about it, nvm could just work.
But it's something the user doesn't know how to toggle if the application breaks.
It's much less straightforward than a toml config.

So I'm still thinking about whether this is worth the code space and the increased documentation.

I am not sure either, this is why I opened an issue before a PR.

I know this is not necessarily ...

manic glacierBOT
#
RetiredWizard
[adafruit/circuitpython] New comment on pull request #8518: Better success using CIRCUITPY_FULL_BUILD=0

Well I've probably put this off long enough that it's gotten stale and would need to be tested across all ports again. The _FULL_BUILD option doesn't seem to be be significant anyway as the general rule is to turn off individual features rather than selecting the minimum build and then adding features back in.

I'm going to go ahead and close this and work on more interesting/possibly useful things.

manic glacierBOT
#
dhalbert
[adafruit/circuitpython] New comment on issue #8693: Have usb drive, usb_hid and the rest of the things that can be enabled/disabled during boot be configurable from settings.toml

Is this there future we are heading towards? Or are we just going to add as few things as possible in there?

A big advantage of boot.py is that it does not require extra firmware space to change settings that already have an API. Also, it can make settings decisions programmatically. The only reason we added .env/settings.toml was because we could not set those settings in boot.py. It was for necessity, not convenience.

One thing I have seen over the years is the tendenc...

manic glacierBOT
main furnace
#

CircuitPython on the small.

20231217_213249.jpg
empty salmon
#

Do all ports (boards) get built? Or only those on CircuitPython.org?
I am trying to decide when to submit a pull request for a board design. I will have production proofs of the board several months before I have “product” available.
I am trying to figure out when to submit the pull request to GitHub as well as when to submit the request to CircuitPython.org for including the board for end users to be able to get firmware.

stuck elbow
#

they are all built

#

(I should really finish it and get it to a fab, but no energy)

empty salmon
#

@stuck elbow - thanks. I could not find where the full set of built firmwares existed. Not a big deal. It just left me “not sure”.
This makes it easier for me to submit the pull request for the board with more lead time.

stuck elbow
tulip sleet
#

when new boards are added, an automatic PR updates circuitpython.org, adding skeletal entries. These are then filled in by the mfr or us.

#

we don't want to have builds on S3 that are not on circuitpython.org, unless they were deleted boards that were never produced.

pulsar sleet
#

How do I use py_gc.h in OnDiskBitmap.c to beable to get the free function. My code is

#include <stdlib.h>
...
#if MICROPY_ENABLE_GC
#include "py_gc.h"

#undef free
#define free gc_free
#endif
...
``` but I get a error saying that `py_gc.h` doesn't exist so do I use the stdlib free function or do I need to do something to use `py_gc`.
#

I got the code by looking at how malloc.c gets the free function

#

Fixed the error by doing py/gc.h

empty salmon
#

@tulip sleet …

we don't want to have builds on S3 that are not on circuitpython.org
Understood.

I will need to figure out the various lead times for

  1. Time from PR to integration into build
  2. Time from initial build to stable release
  3. Time from initial build to appearing on CircuitPython.org

The final step need to occur “slightly ahead” of the planned availability of the board.

tulip sleet
#

you can submit a Draft PR's to both repos

#

and we won't merge until you un-draft them

empty salmon
#

I will still need to figure out the lead times so things don’t diverge between “draft” and “un-draft”.

manic glacierBOT
#
robholmes
[adafruit/circuitpython] New comment on issue #5926: Add BLE GATT server support to espressif port

Looks like dynamic service support has been added to ESP nimble and is in IDF 5.1.2: espressif/esp-idf@94ad8f1

@tannewt I saw you managed to get IDF 5.1.2 into CircuitPython 9.0.0-alpha.6 :bow: so I went to give it a try and spotted that for the Seeed Xiao ESP32C3, it's now missing the _bleio module that was previously in 8.2.9 and is needed for BLE [](https://circuitpython.org/board/seeed_xiao_es...

tulip sleet
#

@lone axle thanks for the reviews. I did those because in at least some cases, the examples are published in Learn Guides. I helped someone a few days ago who was stuck trying to use the 9.x.x-only example on 8.x.x.

lone axle
#

You're welcome it's great to have them running on both while 8 is still stable. Thank you for adding compatibility to all of them.

orchid basinBOT
candid sun
manic glacierBOT
gilded cradle
#

sure

slender iron
#

I’ll be out or late today

lone axle
onyx hinge
slender iron
#

we decided to skip the next two meetings right? or shift one to jan 2nd"

tulip sleet
#

i am doing jan 2, tue

slender iron
#

kk, that's the next one I think

tulip sleet
#

that's right

slender iron
#

add, to the wrap up section of the notes

lone axle
#

Jeff I think you're reading your status updates instead of hug reports.

onyx hinge
#

oh boy brain on autopilot

#

muted now

slender iron
#

nice! no 8.2.x bugs!

#

🎮 👍

lone axle
onyx hinge
#

I did add one note to the PR about trying to understand how it interacts with SSL contexts. I don't overall understand how this fits together, so I can't give higher level feedback.

lone axle
#

Maybe SocketHandler could live outside of adafruit_requests.

onyx hinge
#

Can this 'new thing' act like a socketpool in its own right, just adding additional logic on top?

#

or the creation of this new object becomes purely internal to adafruit_requests or the other library that uses it

#

if adafruit_requests depends on it, it will need to be in the adafruit bundle and live under the adafruit namespace on github

#

In esp-idf-config we could increase the socket count from 8->16 but apparently 16 is an esp-idf maximum.

slender iron
#

thank you!

#

thanks for hosting Liz!

onyx hinge
#

thanks all!

gilded cradle
#

Thanks

errant grail
#

Thanks! Happy Holidays!

short tendon
#

Thanks!

wraith crow
#

Thanks all! Happy Holidays!!!

ember iris
#

Thanks all, best wishes to everyone over the holidays!

candid sun
#

thanks all!

lone axle
#

Thanks for hosting Liz! Hope everyone has a great week and Happy Holidays + New Years

tulip sleet
lone axle
#

Thank you. I must have clicked into them without completing or something. I was working from un-read emails from Github but these ones snuck into read without being odne.

candid sun
#

Here is the notes document for the next CircuitPython Weekly meeting in two weeks on Tuesday, January 2, 2024. It is at the normal time of 11am Pacific / 2pm Eastern here on Discord. Everyone is encouraged to attend! Please add your hug reports and status updates even if you’ll be attending the meeting - it’s super helpful! If you are unable to attend but would still like to include updates, feel free to include them in the notes and we’ll read them off during the meeting. Hope to see you there! <@&356864093652516868>
https://docs.google.com/document/d/1lh2RqzG9CWSjC6t06dRh8ScMLz2NbgN2WCurDAYCK-o/edit?usp=sharing

Google Docs
CircuitPython Weekly Meeting for Tuesday, January 2, 2024
candid sun
thorny jay
#

Oups, I have no clue what I had put (or not) it the meeting notes for today... I hope that was at least complete sentences, maybe some words about C6 and Danh help. Sure I forgot to thanks Anne for a year of newsletter and Paul for new episodes...

midnight ember
#

@lone axle was unable to update the notes doc since last Monday. Wasn't able to do hug reports either. I usually update it along the week at night but it just wasn't working. You would have been all over my hug report.

#

Even after years I still notice I'm the only person that updates their hug report and status updates occasionally during the week. Having the next notes doc available during the week is appreciated.

midnight ember
#

@short tendon I like the idea of an encompassing Session library that unifies requests, esp32spi, mqtt, and anything else that makes an outside request. The downside is likely a rather large library instead of smaller specialized ones. For user support it will definitely be a good thing. Can't tell you how many times I've run into situations of people wanting to make requests with esp32spi but trying to using wifi, socketpool, and other imports from other learn guide examples designed for the S2 or S3 and not the coprocessor. Having one library that everyone uses in the same way will go a long way towards making user support and learn guide examples consistent across the board. I'm all for it if that's the goal.

onyx hinge
#

Generated 3612 font libraries
hm have I maybe gone overboard at this point? thisisfinefire

errant grail
onyx hinge
onyx hinge
#

"ostrich sans dashed" just doesn't end up looking good. And some of the fonts end up cut off with bitmap label, this demo is with the old-fashioned label instead.

onyx hinge
#

there are also some duplicates, foo-webfont.ttf and foo.ttf, so zapped some of those. and trimmed the range of sizes. Now at just 2100 font libraries.

desert yarrow
#

Just saw this group... great that the traffic is open to see here

#

For circuitpython... what's the current status of USB Host? It looks like a lot of progress has been made on TinyUSB's side of things. Does that move the needle much for getting support in CircuitPython?

wraith crow
desert yarrow
#

@wraith crow - It looks like, as of right now at least, that @slender iron is leaning towards not supporting the native USB Host port, so it can be used for programming the board and focusing on just doing USB Host on the rp2040 through PIO. He mentioned that using the native USB would limit the functionality if it weren't available for programming the board, etc. But... I'd like to see if it's still possible. Any ideas?

wraith crow
#

Sorry, I've haven't been paying close attention to the USB Host plans. I was just thinking that if you were new to the discord group, maybe you weren't aware of the github information. I suspect one of the Adafruit folks or someone from the community more familar with the USB Host plans will respond. 😁

desert yarrow
#

Thanks for your help 🙂

slender iron
slender iron
#

If tinyusb support host on the native otg on rp2040 it wouldn’t be too hard to add. Getting it to be dynamic host or device would be trickier

desert yarrow
#

Ah, I see what you mean. It's more of a compile time setting in the tusb_config.h file, right? I guess that would lead to two different releases of cp, etc. I guess fundamentally, it's a problem with how TinyUSB operates, since there is no other reason why you'd need to "pick" one or the other (ie - device or host) mode up front.

#

USB Host support seems to be pretty difficult to come by... there's TinyUSB and I don't really know what else. Other projects like TinyGo or the Embedded Zig project and (maybe even) the embedded Rust project, etc. all seem to not really support USB Host.

There is a very interesting github repo that seems to implement it's own USB Host support from scratch. It does use PIO for bit twiddling, but the USB Host "guts" of the project seem to be a totally independent implementation from Alex. Here's a link:

https://github.com/ataradov/usb-sniffer-lite/tree/main/firmware

GitHub
usb-sniffer-lite/firmware at main · ataradov/usb-sniffer-lite

A simple USB sniffer based on Raspberry Pi RP2040. Contribute to ataradov/usb-sniffer-lite development by creating an account on GitHub.

#

The code there is super clean (imho)...

orchid basinBOT
orchid basinBOT
manic glacierBOT
#
BlitzCityDIY
[adafruit/circuitpython] Issue opened: #8730 MEMENTO does not have STEMMA_I2C() or microphone pin defined

CircuitPython version

Adafruit CircuitPython 9.0.0-alpha.6 on 2023-12-12; Adafruit Camera with ESP32S3

Code/REPL

>>> import board
>>> dir(board)
['__class__', '__name__', '__dict__', 'A0', 'A1', 'BATTERY_MONITOR', 'BUTTON', 'CAMERA_DATA', 'CAMERA_DATA2', 'CAMERA_DATA3', 'CAMERA_DATA4', 'CAMERA_DATA5', 'CAMERA_DATA6', 'CAMERA_DATA7', 'CAMERA_DATA8', 'CAMERA_DATA9', 'CAMERA_HREF', 'CAMERA_PCLK', 'CAMERA_PWDN', 'CAMERA_RESET', 'CAMERA_VSYNC', 'CAMERA_XCLK', ...
manic glacierBOT
#
BlitzCityDIY
[adafruit/circuitpython] New branch created: BlitzCityDIY-patch-1
onyx hinge
#

I don't suppose that anyone knows if it was intentional that the bundle zips are created without compression? This new font bundle is turning out ... big ... but compression decreases it by 93% 85%.

manic glacierBOT
tulip sleet
onyx hinge
#

apparently you have to request it, the default is no compression!

#

I briefly considered maybe it was to allow CP devices to access the zip file while not having zlib support or something, but I don't think we ever built tools for that

manic glacierBOT
tulip sleet
onyx hinge
#

OK, I'll put in a PR today to enable compression

main furnace
#

Are .mpy files compressible?

tulip sleet
#

probably about as compressible as any executable binary

#

I'd guess 30-50% at least

onyx hinge
#

30554502 -> 14637620 (-52%)

main furnace
#

Yeah, I'm getting the same.

#

Thanks for checking.

tulip sleet
#

the .mpy bundle zip?

short tendon
# midnight ember <@1152454697614069811> I like the idea of an encompassing Session library that u...

I'm happy to tackle this any way people feel is the best way. I felt doing this inside requests was best as if you are only doing something like mqtt, you won't run into the same issues. It probably should be its own lib, but either we'd have even more legacy/duplicated code or would break everyone's existing projects. If this was the regular Python world, we could just make breaking changes and version it.

tulip sleet
# short tendon I'm happy to tackle this any way people feel is the best way. I felt doing this ...

I think it would be good to factor out the common code (SocketHandler or whatever) to a common library that requests and mqtt can use. In general we don't want one all-purpose library that does HTTP, MQTT, etc. It's large and might not fit, and it's not how the CPython libraries are structured.

It is fine to make a major change; we will just make a major version bump. We can make the new common library be a prerequisite for the others.

short tendon
#

I also took time to double check what the real Python Request Session does:

rameters across requests. It also persists cookies across all requests made from the Session instance, and will use urllib3’s connection pooling. So if you’re making several requests to the same host, the underlying TCP connection will be reused, which can result in a significant performance increase (see HTTP persistent connection).
So SocketHandler is actually similar to urllib3's connection pooling.

short tendon
tulip sleet
#

exactly, we don't want duplicated code, because then we have to fix bugs or add features in two places

short tendon
#

Well you have it 😉

#

How do we come to next steps?

#

I'm here to help however I can

main furnace
tulip sleet
#

we cna change the name later too, not a really big deal. it will be Adafruit_CircuitPython_something

short tendon
tulip sleet
#

we can make you a collaborator on that particular repo

short tendon
#

I think the bigger question is, is Adafruit good with this direction.

#

And if it were me, I'd make it a Singleton, but there was a good point if someone needed 2 different ssl contexts...

tidal kiln
manic glacierBOT
#
BlitzCityDIY
[adafruit/circuitpython] New comment on pull request #8731: Add MIC and STEMMA_I2C to MEMENTO

tested the UF2 and both additions are working (i2c scan and analog in test). repl printout:

>>> import board
>>> dir(board)
['__class__', '__name__', 'A0', 'A1', 'BATTERY_MONITOR', 'BUTTON', 'CAMERA_DATA', 'CAMERA_DATA2', 'CAMERA_DATA3', 'CAMERA_DATA4', 'CAMERA_DATA5', 'CAMERA_DATA6', 'CAMERA_DATA7', 'CAMERA_DATA8', 'CAMERA_DATA9', 'CAMERA_HREF', 'CAMERA_PCLK', 'CAMERA_PWDN', 'CAMERA_RESET', 'CAMERA_VSYNC', 'CAMERA_XCLK', 'CARD_CS', 'DISPLAY', 'I2C', 'IRQ', 'MIC', 'MISO', 'MOSI', 'N...
stuck elbow
tulip sleet
tidal kiln
#

ah. ok. thanks!

tulip sleet
tidal kiln
#

cool. good person to be on it 🙂

prime flower
#

For those familiar with circup, I have a q. Where are the dependencies listed? Does it pull from requirements.txt?

tulip sleet
prime flower
tulip sleet
#

yes, requirements.txt. It can also use pyproject.toml but it only seems to use that for circup itself

prime flower
#

For libraries, should Circup install everything required for running the examples within the repo's examples/ ? Or for running the library/module itself?

tulip sleet
#

i am wondering whether the library's CI needs the example prereqs to run successfully. If you already have an example where an example prereq is not in requirements.txt and the CI runs successfully, then that is not true

#

I think it might be a bit of a judgment call. For instance, if the example uses some random I2C sensor, it is definitely not a library requirement that you have that lib installed

#

if the requirement is extremely common, then maybe it should be included. What's the specific case?

prime flower
tulip sleet
prime flower
#

That is one of them. It is missing a few imports after that, too.

manic glacierBOT
slender iron
tulip sleet
prime flower
tulip sleet
#

it may just not work

#

it seemed like some settings worked for some OS's and not for others.

slender iron
tulip sleet
#

or in some cases, it was really erratic (like, macOS): the OS seemed to give up scanning at some point

desert yarrow
manic glacierBOT
#
jepler
[adafruit/circuitpython] New comment on pull request #8723: Jpegdecoder enhancements

The updated test should cover all the related functionality:

  • reading from in-memory data or from an arbitrary object supporting the stream read protocol (disk files)
  • x,y offset into destination bitmap
  • x1,y1 - x2,y2 cropping of decoded jpeg data
  • color keying with skip_source_color, which works imperfectly due to the lossy nature of JPEG encoding

Note that this does not enable reading from a socketpool Socket object, because it does not appear to implement the stream read...

manic glacierBOT
celest marsh
short tendon
celest marsh
#

oh - your wanting to pull it out of Sessions completely, that does make sense from a microcontroller POV, would that remove it from being an explicit call out for shared sockets to something that is abstracted away?

pulsar sleet
#

I tried to fix some issues that I was having with memory but broke more stuff
I had bmp images rendering on my matrix portal m4 but the code would crash because of memory issues
I fixed those issues by recompiling circuitpython but telling displayio to make the palette once and reuse that. Then my images had weird colors so I put all my images in one big picture and set that to use one 256 color palette and cut that picture back into the individual ones and set their name.
When I put those images on the matrix portal the images didn't even resemble what the original image was. The palette that is generated is the same as the palette on the images (checked by a python script I made) so I don't know what could be causing the issues.
My modified circuit python Displayio/OnDiskBitmap.c file is uploaded.

📎 OnDiskBitmap.c
#

I can upload the code running on the matrixportal m4 and/or images if needed

short tendon
#

Do you have cases where you use a different ssl context?

celest marsh
#

I do have a few, but for those I would rather have a more stable socket pool and move the use case that requires a device certificate to be used onto it's own device, or if it's not a hard solution, make the ssl context inform/tag a socket as not being part of the shared pool

short tendon
#

Actually looking at the code, when I made this first attemp, it was to limit changes and only move files. In the end the Session can totally keep track of the ssl_context, and only in the case it's not passed in would it generate one...

So from:

socket = self._socket_handler.get_socket(host, port, proto, timeout=timeout)

to:

socket = self._socket_handler.get_socket(host, port, proto, timeout=timeout, ssl_context=self._ssl_context)
#

which would work the same in SocketHandler was a singleton

#

Looking, pulling out the shared socket handling code is ~1.5kb from mqtt and ~2kb from requests...

#

when compiled

pulsar sleet
# pulsar sleet I tried to fix some issues that I was having with memory but broke more stuff I ...

Continuing this.
In OnDiskBitmap.c in the get_pixel function it doesn't do anything with the palette. Some things use a bitmask but not when there is 1 byte per pixel

        uint32_t tmp = 0;
        uint8_t red;
        uint8_t green;
        uint8_t blue;
        if (bytes_per_pixel == 1) {
            uint8_t offset = (x % pixels_per_byte) * self->bits_per_pixel;
            uint8_t mask = (1 << self->bits_per_pixel) - 1;

            return (pixel_data >> ((8 - self->bits_per_pixel) - offset)) & mask;
        } else if (bytes_per_pixel == 2) {
            if (self->g_bitmask == 0x07e0) { // 565
                red = ((pixel_data & self->r_bitmask) >> 11);
                green = ((pixel_data & self->g_bitmask) >> 5);
                blue = ((pixel_data & self->b_bitmask) >> 0);
            } else { // 555
                red = ((pixel_data & self->r_bitmask) >> 10);
                green = ((pixel_data & self->g_bitmask) >> 4);
                blue = ((pixel_data & self->b_bitmask) >> 0);
            }
            tmp = (red << 19 | green << 10 | blue << 3);
            return tmp;
        } else if ((bytes_per_pixel == 4) && (self->bitfield_compressed)) {
            return pixel_data & 0x00FFFFFF;
        } else {
            return pixel_data;
        }
    }```
GitHub
circuitpython/shared-module/displayio/OnDiskBitmap.c at main · adaf...

CircuitPython - a Python implementation for teaching coding with microcontrollers - adafruit/circuitpython

onyx hinge
#

@pulsar sleet The pixel shader (such as a palette) is applied here: ```c
if (self->pixel_shader == mp_const_none) {
output_pixel.pixel = input_pixel.pixel;
} else if (mp_obj_is_type(self->pixel_shader, &displayio_palette_type)) {
displayio_palette_get_color(self->pixel_shader, colorspace, &input_pixel, &output_pixel);
} else if (mp_obj_is_type(self->pixel_shader, &displayio_colorconverter_type)) {
displayio_colorconverter_convert(self->pixel_shader, colorspace, &input_pixel, &output_pixel);

pulsar sleet
#

Ok thanks

onyx hinge
#

so say that the pixel at (3,9) is palette entry 37 -- common_hal_displayio_ondiskbitmap_get_pixel should return 37, and it would be turned into an actual color value by calling displayio_palette_get_color

#

as a corollary the right palette object has to be supplied as the pixel shader when constructing the TileGrid that will show the OnDiskBitmap.

manic glacierBOT
#
arduino12
[adafruit/circuitpython] Issue opened: #8733 `RGBMatrix` display showing white vertical lines when re-initialized on ESP32-S2

CircuitPython version

Adafruit CircuitPython 8.2.9 on 2023-12-06; S2Mini with ESP32S2-S2FN4R2

Code/REPL

import board
import displayio
from framebufferio import FramebufferDisplay
from rgbmatrix import RGBMatrix
from terminalio import FONT


def get_display():
    matrix = RGBMatrix(
        width=128,
        bit_depth=1,
        rgb_pins=[board.IO1, board.IO40, board.IO2, board.IO4, board.IO38, board.IO6],
        addr_pins=[board.IO8, board.IO...
short tendon
#

I had this thought about helping with deprecated methods/classes

I've seen things like this:

class Thing:
  def method():
    # Thing.method is deprecated, please use Thing.new_method instead.
    return new_method(False)
  
  def new_method(value):
    return value

We could do this:

try:
  from adafruit_deprecated import show_deprecated_warning
except ImportError:
  def show_deprecated_warning(warning):
    pass


class Thing:
  def method():
    show_deprecated_warning("Thing.method is deprecated, please use Thing.new_method instead.")
    return new_method(False)
  
  def new_method(value):
    return value

And then in code.py one could do:

import board
from adafruit_deprecated import set_deprecated_warning
from thin import Thing
import time

set_deprecated_warning(use_print=True, blink_led_pin=board.LED)

thing = Thing()

while True:
  thing.method()
  time.sleep(0.5)

and now every time I call thing.method() my LED would blink and would get a print statement in the REPL (could also use adafruit_logging)

tulip sleet
onyx hinge
#

thanks for the ping dan

#

we can merge that as is but doing it in the camera class might be better

tulip sleet
#

@tidal kiln can you change the PR to fix it in the constructor?

#

^^

tidal kiln
#

@tulip sleet @onyx hinge done

tulip sleet
#

thankyou!

onyx hinge
#

if anybody has the bandwidth, I wanted to talk about bundle & module naming for https://github.com/jepler/circuitpython-fonts/

Here's how things stand:

  • the repo is called circuitpython-fonts
  • the individual autogenerated libraries (which gosh I hope we don't release to pypi because there are ~2100) are named things like font-raleway-thin-6 for the Raleway font in the thin variant at a size of 6 pixels
  • the importable library is font_raleway_thin_6

Main question I guess is whether to have an "adafruit" in there somewhere. should the importable lib be adafruit_font_raleway_thin_6? Should the repo be adafruit-circuitpython-fonts? adafruit-circuitpython-font-bundle?

#

@tulip sleet @slender iron ^ naming help would be welcome

short tendon
onyx hinge
tulip sleet
tulip sleet
onyx hinge
#

somebody could probably code up a pure-Python warnings module that would sub in 8.x. I mean, it's basically print().

short tendon
#

I really liked the idea of being able to blink a LED. Being connected to the REPL to know can be hard sometimes.

#

If there's a need, happy to stub it out for 8.x

tulip sleet
short tendon
#

Fair

celest marsh
#

very inception-y feeling, add a deprecation warning to 7 to get ready to have it in 8, which would then be deprecated because 7 is now deprecated

tulip sleet
#

@onyx hinge I will make a pycamera release unless you are doing it this very minute

onyx hinge
#

I am not @tulip sleet

slender iron
pulsar sleet
#

what if you made it write a log somewere that can be accessed later. Maybe a file?

stuck elbow
#

it does that for things in boot.py, though

pulsar sleet
#

I know but maybe there is a way to have two file systems that when you hold some button or do something in the serial monitor to get access to that other filesystem that has the logs

tulip sleet
#

there are lots of things that might be logged, so logging just these warnings is a pretty special case. I'm not sure that's really needed

#

more than say, logging the most recent uncaught exception, etc.

#

the latter is a more general thing to log

#

but if you are sophisticated enough for that, then you're sophisticated enough to connect to the serial console and just see what is printed. However, another interesting case is when you're disconnected

pulsar sleet
#

I making a thing for my dad and was trying to add logging and that was a pain

stuck elbow
#

would it make sense to log uncaught exceptions to boot_out.txt when the filesystem is not mounted? would the flash wear be a problem?

tulip sleet
stuck elbow
#

maybe it could be disabled by default, but you could enable it for debugging?

tulip sleet
#

yes, or we could reserve a bit of non-fs flash for this.

stuck elbow
#

or create the file with the dump only when it doesn't already exist?

tulip sleet
#

yes, maybe last_exception.txt or similar

#

boot_out.txt is only rewritten if it's different. could be the same for last_exception.txt

stuck elbow
#

editors could even pick it up to display errors

crimson ferry
tulip sleet
stuck elbow
#

of course it could be confusing if it's stale, but maybe they could check the creation date

#

I wonder, if the file was read-only, and didn't grow in size, would it be possible to modify it by the microcontroller with the usb mounted without corrupting the filesystem?

tulip sleet
#

that's a lot of special-casing in the filesystem code, since we'd want to prevent its attributes from being set to r/w, etc.

#

maybe not a lot, but some

pulsar sleet
#

I think I might have found a source of my problems?
In Palette.c function displayio_palette_get_color I am printing the input and output and my output is 1, 65535. The 1 is the input (the palette index) and then it output 65535 which I assume is in rgb888 but the color should be 16121855. And 0 maps to 0 where the palette should map 0 to full white

tulip sleet
pulsar sleet
#

Apparently not. Maybe I don't know what the code is doing bit using pythons pil library on my pc and telling circuitpython to print the palette shows 24 bit colors

tulip sleet
#

i thought the OnDiskBitmap was RGB565

pulsar sleet
#

It looks like it just reads write the colors from the bmp file directly to the data from the palette data in the file so idk

tulip sleet
#

normally we use 565 to save space

pulsar sleet
#

weird

#

I think it is rgb888 because when the colors are black and white it sets color 1 (white) to 0xffffff

tulip sleet
#

i am talking about the bmp file and its corresponding palette. Are we talking about the same thing?

pulsar sleet
#

I was talking about the ondiskbitmap function but I think my files are using RGB888

tulip sleet
#

ok, I'm thinking about TFT displays, maybe. If you run the file command on one of the .bmp's you generated, what does it say?

pulsar sleet
#

NYJ.bmp: PC bitmap, Windows 98/2000 and newer format, 32 x 32 x 8, cbSize 2108, bits offset 1162

midnight ember
#

24-bit color images do work for some things like full background images. it's when you need something with transparency that the 8-bit indexed helps as you can select the specific index to maketransparent.

tulip sleet
#

ok, that seems like 8-bit indexed. But the mapper will convert to RGB888, I guess, for the matrix

midnight ember
#

it's been a while since i played with the higher color images. some 24-bit images get downsampled to 16-bit color.. i think.

tulip sleet
#

I am not really sure. Are you finding things that can be fixed?

midnight ember
#

most displays aren't big enough to really show that much of a difference in color depth. so 8-bit indexed is surprisingly still well colored for small displays... and it loads faster. now if you were to put it on a full sized monitor the difference between 8-bit and 24-bit would be night and day.

pulsar sleet
#

It looks like the process function (that I used to get these images to work) does do rgb 565 color but PIL's Image.getpalette()'s size is 3*256 so if PIL does convert rgb565 to rgb888 then that would make sence

midnight ember
#

i should clarify, most displays that will run from a microcontroller.

#

I use this to convert everything to 8-bit indexed https://online-converting.com/image/convert2bmp/ works fine but you don't get the choice of where the index color is in the palette. still have to open gimp to get the index for the color you want. just saying if you don't absolutely need to use PIL for image processing there are online alternatives.

tulip sleet
pulsar sleet
#

ya

tulip sleet
#

for this particular project

midnight ember
#

oh this is dev chat, thought it was circuit python channel. oops. i usually don't have much to contribute in this channel.

tulip sleet
#

np , you do have something to say 🙂

midnight ember
pulsar sleet
#

and even if the converting output is rgb 565 when giving index 0 the function returns 0 (black) when it should return white

tulip sleet
#

but... the image does display correctly, right?

pulsar sleet
#

No. It did until I made ondiskbitmap use one palette then the colors were weird but still showed the image. Then I changed the images to use the same palette and the images are worse

tulip sleet
#

each image is going to have its own special indexed palette because the logos have different colors

#

so you can't reuse the indexed palette successfully

#

because the indexes correspond to different colors in each image

#

e.g color 30 is blue in one and burnt orange in another

pulsar sleet
#

I know. Thats why in the first picture has weird colors. What I did then is combined every image together and run the process function on that big images. Then I cut that image into its indivisual images

#

and I got the 2nd image

tulip sleet
#

if you look at the big image on the regular screen, do the colors look OK?

pulsar sleet
#

yes

#

even the images that I put on the sd card (where I store images) is fine

tulip sleet
#

i don't have an explanation for why reusing the palette is somehow not working, but it sounds like some bug. Maybe the palette type is wrong somewhere

#

for instance if the palette got gc'd accidentally then there might be garbage there

pulsar sleet
#

maybe

#

there is a example that just shows a image and that works fine except randomly the palette's color count goes very high but that doesn't happen with my code

#

Is there a way to see if the palette gets gc'd

tulip sleet
#

so is this now mostly a programming exercise or are you trying to finish a Christmas gift or something like that?

pulsar sleet
#

its a chrismas gift

tulip sleet
#

I'm not sreally sure whether the palette is gc'd or not. I'd suggest printing out the palette and see if it changes, but you are already doing that?

#

would you consider getting different hw?

pulsar sleet
#

I need it at a somewhat working state so I could just download circuitpython 9 without my stuff and run it to show my dad the idea and I could fix it later but I would like it to be working before chrismas.

pulsar sleet
pulsar sleet
tulip sleet
#

the comment about compatible boards I think is for the Arduino library

#

another thing you might try is to cut down the logos to 24x24 or 16x16

#

but RAM is quite tight on the Matrix Portal

pulsar sleet
#

how important would this be. I want to do as much software stuff before getting new hardware

tulip sleet
#

Or just give the team names for now. Later when the Matrix Portal S3 comes back you can substitute

pulsar sleet
tulip sleet
#

you could use a single image and use TileGrid to fetch the section you want

pulsar sleet
#

The problem I have with that is I need to then have a big list of team names and where in the file they are

tulip sleet
#

that is the whole point of TileGrid.

#

The team names are in the json, I thought

pulsar sleet
#

if there is a better function to render just a image then I could use that

tulip sleet
#

i don't remember if the original program reads all the images and keeps them all. You could just read two at a time to display that particular ascore

pulsar sleet
pulsar sleet
tulip sleet
#

I"m saying just use a text label to print out the team name, no logo

pulsar sleet
#

I know

tulip sleet
#

no images whatsoever

pulsar sleet
#

I know

pulsar sleet
tulip sleet
#

The S3 Feather + the FeatherWing is like the S3 Matrix Portal. The latter is a single board. this is a combo

#

I'm not necessarily telling you to spend more money now. just brainstorming about what the higher-level alternatives are

pulsar sleet
#

I know

midnight ember
#

I've personally used the S2 Feather + Matrix Featherwing and S3 Feather + Matrix Featherwing... about 6 months before the Matrix Portal S3 even existed. It's definitely an acceptable substitutite. I have not attempted to use them on 12 matrix panels like with the Matrix Portal S3.

#

It will run 1 single panel plenty fine. The problem is you don't have the time between now and Christmas to get that HW.

pulsar sleet
#

Ya. And I need to run 4 64x32 panels

midnight ember
#

I can only vouge that the Matrix Portal S3 will run 4 panels... times 3.

#

I think a Feather S3 + Matrix featherwing will run 4 panels as long as you use an external power source.

tulip sleet
#

The Matrix FeatherWing is just a pin mapper. It has no active components (except for a voltage regulator)

#

the Matrix Portal S3 is the same kind of pin mapper (to the matrix connector)

midnight ember
#

Running 4 panels on the original Matrix Portal would be pushing it for that project.

#

The S3 has far more CPU & RAM. I think Dan's suggestion of just printing the team names for the time being would be a good temporary workaround.

pulsar sleet
#

I know but using different hardware doesn't fix this issue but it fixes the issue that I was having that cause my to break everything else that is not enough memory. So I think I could fix the issue I am having right now

#

It looks like in displayio_convert_color it acts as if the 8 bit number is in rgb332 and not indexed bmp. Maybe convert color doesn't get called if the bmp is indexed

#

unless the input gets converted to rgb888 from the palette already

tulip sleet
#

i don't know what the flow is, sorry

pulsar sleet
#

it looks like it is

tulip sleet
#

that would make sense to me in this case

pulsar sleet
#

what if the circuit python logo is causing the issue because before my program starts writing things it shows the prints on the screen and blinka

#

is there a way to disable that

midnight ember
#

This is an Adafruit ESP32-S3 Feather + RGB Matrix Featherwing. You don't need to wait for the Matrix Portal S3. You can get these components today, they're in stock.

pulsar sleet
#

I know

short tendon
crimson ferry
#

(at least some of) the SSLContext properties can be set dynamically in code.py, which can be very handy

#

i.e., using same SSLContext, but tweaking it for the specifics of the current request

#

I don't know if there is a material resource load with carrying around multiple contexts (as opposed to re-using one and tweaking the properties)

#

I guess it becomes a question of passing in a tweaked context (or one of several), or exposing the (singular) context for tweaking

onyx hinge
#

maybe you can, but as we try to move into an async world I would recommend against it. Anyway, I don't think an SSL context object is big, though it does have whatever certificate & key buffers you assign to it.

crimson ferry
#

recommend against re-using one context?

onyx hinge
#

I'd recommend against modifying a context

celest marsh
#

async does make things painful when you try to use singletons that change their behaviour based on who is currently interacting with them

short tendon
#

Holding them in the session isn't an issue, and now that I know that's the holder it's easy

celest marsh
#

you get odd shared memory errors

short tendon
#

Although all CP async is still just one thing at a time isn't it? And just sharing during yields?

onyx hinge
#

yes, that's async in a nutshell

#

*well, during awaits

short tendon
#

True

#

So shouldn't be an issue

onyx hinge
#

but ideally you'd be awaiting your socket activity too, so there'd be no guarantee about what else got interleaved while you're trying to establish a conection

#

having "establish a TCP (or SSL) connection" be async is still off in the distance, it doesn't work that way yet

short tendon
#

I will be happy to deal with those problems when we get there

#

I imagine anyone dealing with needing multiple ssl contexts aren't using a small chip and aren't first time developers so will be able to do a little extra setup and create a few extra sessions

pulsar sleet
#

I installed circuitpython 9 onto my matrixportal m4 and I still had issue so I think that the bug I am having is a bug with circuitpython

short tendon
#

So the real question is, if there is to be a lib, that handles sockets and connects them for you (to be used by requests, mqtt, etc), and helps keep them alive and releases them when they aren't being used and something needs one, what is it called? SocketHandler? SocketManager? Something else?

celest marsh
#

it's not managing the sockets tho, it feels more like it's managing the socket pool

short tendon
#

True... Although it manages them once they come out of the pool...

celest marsh
#

naming things is one of the three hardest tasks in computers, along with off by one errors

short tendon
#

50/50 someone would say that or the binary 10 equivalent....

celest marsh
#

LOL

#

honestly the name is going to be hidden behind the abstraction unless you are the few who need it, then you will go looking

#

so make it descriptive

crimson ferry
#

there's no equivalent superset in CPython that we could use (for naming)?

celest marsh
#

yea, I was wondering what urlib3 calls it

#

requests and httpx both use urlib3 IIRC

#

urlib3 calls them connection pools

#

yep, ConnectionPool is the class that urlib3 uses for this behaviour

short tendon
#

That is true, the other libs will import it. You just need to drag it to your lib folder

#

So we'll call it SocketNinja

#

Yeah, should probably go with ConnetionPool that's totally the same thing

celest marsh
#

thankfully anecdata stopped the slide down the naming rabbithole before we got to SocketNinja or PoolBoy2020

short tendon
#

Although they are for a single host, where this is for all of them...

#

But SocketNinja would be soo cool

celest marsh
#

for the vast majority of the use cases - most people are connecting to a single host

#

so maybe make your SocketNinja helper use this new ConnectionPool

short tendon
#

It's closer to urllib3.PoolManager

#

The manager is more what I'm trying to make (if everyone agrees). To manage the connection of the sockets, so we remove copy-pasta and make it easier for any other socket libraries

celest marsh
#

time for some "sleep on it and wait for the other timezones to read the conversation" pausing

crimson ferry
#

is this for any kind of sockets, or mainly TCP/Stream?

#

would NTP, for example, use this pool?

short tendon
#

Last I built anything for NTP, I used socket.SOCK_DGRAM, which I believe CP supports, so it could totally use it

crimson ferry
#

cool. there's also SOCK_RAW, not sure if anyone has used that

#

but in theory, it works

#

but all three kinds contribute to using up the limited sockets in the mcu

short tendon
#

But that's exactly my goal, if someone wants to write a CP NTP library, they can and not need to worry about getting a socket from the pool.

Everything works great if you are just doing one thing at a time (or just one thing), but as projects get bigger it gets harder to handle

crimson ferry
#

there is an NTP library (indeed using SOCK_DGRAM)

short tendon
#

For CP?

crimson ferry
#

adafruit_ntp

#

used to be for esp32spi, but got switched over for native wifi

short tendon
#

Ahh. Found it

#

I assume it works for the esp32spi too? With it's fake pool?

crimson ferry
#

nope

#

an old version still works for esp32spi, but not the current versions

#

presumably it could be written to work with either

short tendon
#

Why not?

#

I haven't had any issues using the fake pool

crimson ferry
#

don't know, maybe part was that esp.get_time() always works (and that's what the esp32spi version of the NTP library used, rather than a true UDP port 53 implementation)

#

(well, not always ...but after the chip has been connected to the network for about 15 seconds)

short tendon
#

Oh this one won't work. Some different sock methods. Well I'll add that to my list...

#

Do you know if the plan is to continue with the airlift? Since now it seems not everything works with it?

crimson ferry
#

any MCU without native wifi py not espressif and not raspberrypican benefit from Airlift

short tendon
#

Good to know. I'll spend some time finding differences

thorny jay
# short tendon Good to know. I'll spend some time finding differences

I am a bit lost in CP network (IP) stack but so far never had a good reason to dig as the high level API worked for me. So here is what I know:

  • ESP32 Nina FW / AirLift => that is the historical solution
  • Most Espressif board ESP32, S2, S3 and some of the new one
  • Pico-W
  • Wiznet5K Ethernet (now Wifi) stack (maybe pure python?)
  • ... (anything else?) ...

I would love to know what are the limitation tips and tricks for each, but I mostly don't.
Some have limited number of "Socket" or no SSL support (I am thinking W5500.
I don't know if all can do UDP or "RAW" and what you need for NTP.

manic glacierBOT
#
carson-coder
[adafruit/circuitpython] Issue opened: #8734 some Indexed BMP Images look corrupted

CircuitPython version

Adafruit CircuitPython 9.0.0-alpha.6 on 2023-12-12; Adafruit Matrix Portal M4 with samd51j19

Code/REPL

import board
import displayio
import framebufferio
import rgbmatrix

displayio.release_displays() # Release current display, we'll create our own

# Create RGB matrix object for a chain of four 64x32 matrices tiled into
# a single 128x64 pixel display -- two matrices across, two down, with the
# second row being flipped. widt...
crimson ferry
#

@thorny jay TLS is available on native wifi and Airlift. No TLS on WIZnet. Sockets are limited: chips/SDKs have resource limits, firmware build may limit further, and available socket count may vary by socket type on some platforms (e.g., raspberrypi https://github.com/adafruit/circuitpython/issues/7543). Yes, native wifi, Airlift, and WIZnet can all do UDP sockets (e.g., NTP).

manic glacierBOT
thorny jay
# crimson ferry <@681528999520108564> TLS is available on native wifi and Airlift. No TLS on WIZ...

I still need to check all the work done to unify API, to pool socket, to make HTTP server and the ethernet improvement frequently covered in streaming this year. I even have Wiznet hardware soldered but never used. Also better parsing of JSON, and the work done on updating PKI root certificate.
There is just too much interesting development by you and others, and so little time to play with everything.

manic glacierBOT
manic glacierBOT
#
carson-coder
[adafruit/circuitpython] New comment on issue #8734: some Indexed BMP Images look corrupted

BTW, identify -verbose NYJ.bmp and similar gives you all kind of info about the file, including its indexed colormap. identify is from ImageMagick.

When I run that I get

  Filename: NYJ.bmp
  Permissions: rwxrwxrwx
  Format: BMP (Microsoft Windows bitmap image)
  Class: PseudoClass
  Geometry: 32x32+0+0
  Resolution: 37.79x37.79
  Print size: 0.846785x0.846785
  Units: PixelsPerCentimeter
  Colorspace: sRGB
  Type: Palette
  Base type: Undefined
  Endiannes...
#
dhalbert
[adafruit/circuitpython] New comment on issue #8734: some Indexed BMP Images look corrupted

I reproduced your problem on a single panel with the Matrix Portal S3 (I can't find my Matrix Portal M4 at the moment).

I saw a similarly corrupted image. However, when I read the file into GIMP and then export it as .bmp again, with no changes, it displays properly:
NYJ1.bmp.zip
Try that.
The indexed palette as described by identify -verbose is exactly the same. However, there are these differences. The Compress...

manic glacierBOT
pulsar sleet
#

Fixed all of the issues with my program with lots of help from dhalbert. Now I kinda want to add support for compressed bitmaps but that requires me to get more hardware so idk

manic glacierBOT
#
michalpokusa
[adafruit/circuitpython] Issue opened: #8736 `MemoryError` with simple imports on Pico W on 9.0.0-alpha6

CircuitPython version

Adafruit CircuitPython 9.0.0-alpha.6 on 2023-12-12; Raspberry Pi Pico W with rp2040

Code/REPL

import gc

print("Before import", gc.mem_free())
try:
    import adafruit_datetime
    # from adafruit_datetime import datetime
except:
    pass
print("After import", gc.mem_free())

Behavior

Importing some libs on 8.2.9 works as expected, and allocates only a small portion of memory, the same code/imports on 9.0.0-alpha6 err...

solar whale
onyx hinge
# 

try:
    from typing import Sequence
except ImportError:
    pass```
The typing module exists on standard Python but not in CircuitPython. That's why the code is surrounded by a try/except block.
#

you sholdn't actually see an ImportError thrown from that line, it should be caught and ignored

solar whale
#

ah -- but the code stops

onyx hinge
#

what version of circuitpython?

#

what actually happens?

#

(repl output etc)

#

(this is supposed to be working and several people besides me are using it for the past couple o' days)

solar whale
# 
Adafruit CircuitPython 9.0.0-alpha.6 on 2023-12-12; Adafruit Camera with ESP32S3
>>> 
soft reboot

Auto-reload is on. Simply save files over USB to run them or enter REPL to disable.
code.py output:
Traceback (most recent call last):
  File "code.py", line 12, in <module>
  File "adafruit_pycamera/__init__.py", line 15, in <module>
ImportError: no module named 'adafruit_aw9523'

Code done running.

Press any key to enter the REPL. Use CTRL-D to reload.
onyx hinge
#

How did you install the pycamera library?

solar whale
#

oops -- the missing module was not reported on my Raspberry Pi -- it is on my Mac ....

#

I just copied it over

#

I missed the 2 dependencies - my bad...

onyx hinge
#

there had been a problem with some requirements not listed for circup but brent fixed that yesterday. manually copying, it's up to you.

solar whale
#

Thanks -- I'll re-install ... sorry for the false alarm

#

Someday I'll try circup...

#

@onyx hinge Yay! it is working!

#

Nice work!

manic glacierBOT
#
RetiredWizard
[adafruit/circuitpython] New comment on issue #8736: `MemoryError` with simple imports on Pico W on 9.0.0-alpha6

I'm thinking I'm missing something obvious, I can't seem to reproduce this.
my code.py

import gc

print("Before import", gc.mem_free())
try:
    import adafruit_datetime
    # from adafruit_datetime import datetime
except:
    pass
print("After import", gc.mem_free())

soft reboot

Auto-reload is on. Simply save files over USB to run them or enter REPL to disable.
code.py output:
Before import 130224
After import 130064

Code done running.
short tendon
#

What do people think about this approch for SocketHandler:

  1. Create new Adafruit_CircuitPython_Xxxxxxx lib that handles socket management
  2. In Adafruit_CircuitPython_Requests:
    a. Use Adafruit_CircuitPython_Xxxxxxx
    b. Remove _default_session, _FakeSSLSocket, set_socketand top levelrequest, get, post, etc.`
  3. In Adafruit_CircuitPython_MiniMQTT:
    a. Use Adafruit_CircuitPython_Xxxxxxx
    b. _FakeSSLSocket, set_socket and _get_connect_socket
manic glacierBOT
manic glacierBOT
tulip sleet
manic glacierBOT
#
michalpokusa
[adafruit/circuitpython] New comment on issue #8736: `MemoryError` with simple imports on Pico W on 9.0.0-alpha6

@deshipu Empty settings.toml.

@dhalbert I use .py version of adafruit_datetime.

@RetiredWizard are you sure you have adafruit_datetime in your /lib folder? I just removed everything from the board and got same results as you, but noticed I forgot to install the lib. Please try running:

import gc

print("Before import", gc.mem_free())
try:
    import adafruit_datetime
    # from adafruit_datetime import datetime
except Exception as e:
    print(e)
print("After i...
slender iron
stuck elbow
#

socketeer ;-)

short tendon
slender iron
#

yup

short tendon
#

Makes sense

slender iron
stuck elbow
#

oh no

#

it was a joke, please don't

short tendon
#

too late 😉

stuck elbow
#

use names that mean something

short tendon
#

Do you have an other ideas?

manic glacierBOT
stuck elbow
#

socket pool is already taken

manic glacierBOT
#
michalpokusa
[adafruit/circuitpython] New comment on issue #8736: `MemoryError` with simple imports on Pico W on 9.0.0-alpha6

The .py version has to be compiled, and that can cause fragmentation, even if the RAM eventually used is about the same.

I beleive this could also happen when trying to execute user code, not only when importing libraries.

Also, which is important, it used to work back in 8.2.9 and Pico W is a very popular board. I think it will happen to a lot of people when 9.0.0 comes out of alpha.

I guess one could avoid that by compiling user code, but if I am correct it is not a intended use ...

short tendon
#

Anything better than SocketManager or SocketTracker?

stuck elbow
#

maybe NetDevice to mirorr BusDevice?

slender iron
#

honestly I don't fully understand what it does. I haven't had time to look at the pr

short tendon
#

Just imagine all the socket specific code from requests.Session pulled out as it's own lib.

#

So keeping track of what sockets have been opened and not closing them unless something needs another and there isn't one free

stuck elbow
#

so NetSession?

manic glacierBOT
short tendon
# stuck elbow so NetSession?

It's handling the sockets, not the session. So kinda a mix between urllib3.connectionpool and urllib3.poolmanager (which CPython Requests uses for this)

tulip sleet
#

SocketManager sounds OK to me

stuck elbow
#

for me too

short tendon
#

Okay, I'll create forked branches for both Adafruit_CircuitPython_Requests and Adafruit_CircuitPython_MiniMQTT and a new local repo for Adafruit_CircuitPython_SocketManager and make the changes. Will ping here once they are ready to be looked at

#

It it looks generally good, then an offical repo can be created

slender iron
#

ConnectionPool could work too

tulip sleet
slender iron
#

since you are managing sockets that are connected already

short tendon
short tendon
stuck elbow
#

we already have a socketpool, so this could be confusing, though I admit I like the name

#

you could also frankenstein them both into ConnectionManager

#

but SocketManager is shorter

short tendon
#

I probably like ConnectionManager the best...

slender iron
#

that works for me

pulsar sleet
#

Where does circuitpython get the string methods / what would I do after adding a method in py/objstr.c. I am trying to add the str.title() function

stuck elbow
pulsar sleet
#

I know but I can't find the dict for strings. I think the dir is called mp_obj_str_locals_dict but I don't know where that is defined

onyx hinge
#

@tulip sleet possible to use normal validation and be rid of this string, maybe. TRANSLATION("color must be between 0x000000 and 0xffffff", 242)

"in use" messages, can any be coalesced without losing too much of importance? ```TRANSLATION("%q in use", 11)
TRANSLATION("A hardware interrupt channel is already in use", 53)
TRANSLATION("All channels in use", 54)
TRANSLATION("All event channels in use", 55)
TRANSLATION("All sync event channels in use", 56)
TRANSLATION("All timers for this pin are in use", 57)
TRANSLATION("All timers in use", 58)
TRANSLATION("Bus pin %d is already in use", 72)
TRANSLATION("Cannot vary frequency on a timer that is already in use", 83)
TRANSLATION("Clock unit in use", 85)
TRANSLATION("DAC already in use", 87)
TRANSLATION("EXTINT channel already in use", 96)
TRANSLATION("Serializer in use", 160)

tulip sleet
onyx hinge
#

TRANSLATION("Tile height must exactly divide bitmap height", 168)
TRANSLATION("Tile width must exactly divide bitmap width", 170)

#

OK, good luck

#

TRANSLATION("Slice and value different lengths.", 161)
TRANSLATION("buffer slices must be of equal length", 215)

tulip sleet
#

the refactoring really mostly works if there are qstrs available, which there aren't in some cases. I consolidated some GCLK messages. Also some NotImplemented ones which were redundant, just used the arg name

#

each one of those is only a few bytes, unfortunately

#

recompiling with gcc13 is a much bigger improvement, but not prepared to go there for 8.2.x yet

onyx hinge
# 
TRANSLATION("can't convert %s to float", 223)
TRANSLATION("can't convert %s to int", 224)
TRANSLATION("can't convert NaN to int", 226)
TRANSLATION("can't convert inf to int", 227)
#

(I'm looking in main fwiw)

tulip sleet
#

when you do some of these you have to use the _varg call, which is larger, and passing the args, and sometimes there is no net gain, or worse

stuck elbow
#

I bet you could shorten some of the translations if you knew Chinese...

tulip sleet
#

yes, wish it were French

#

Maybe I should just delete a few translations

#

since this is just for 8.2.x

#

possible to use normal validation and be rid of this string, maybe. TRANSLATION("color must be between 0x000000 and 0xffffff", 242)

#

@onyx hinge I started to do the color value one, but the hex value is pretty meaningful, so I back off on that

onyx hinge
#

true

short tendon
#

Quick question, I see tox used by both Adafruit_CircuitPython_Requests and Adafruit_CircuitPython_MiniMQTT for tests, and targeting different Python versions 3.8/3.9. Should I:
a) Use tox in the new project
b) Can I standardize to a version (I would pick 3.11)

slender iron
manic glacierBOT
stuck elbow
#

@tulip sleet I wonder if it would be possible to have pseudo-files in circuitpython, that appear in the filesystem table, but that have physical size 0, and when you try to open them, you get dynamically generated content, like the files on the uf2 filesystem

tulip sleet
stuck elbow
#

yeah, something like that

#

and have the last exception in it, and also move the boot_out.txt to it

tulip sleet
#

that could be a separate "filesystem", like .frozen

stuck elbow
#

well, you want it available over msc

tulip sleet
#

hmm, yes

#

we could look for a prefix like /proc/ and then go from there.

#

But is this better than supervisor.last_exception? etc.

#

boot_out.txt is a real file but is limited in size, though just for safety

#

you could use MSC or you could use talking to the REPL to get the info

stuck elbow
#

I'm thinking that "can you paste your status.txt here" is easier than "can you download and install putty, connect with it to the serial console, and see if there are any errors"?

#

and by making it a pseudofile we dodge the flash damage issue

tulip sleet
#

maybe there is a direct connection: /supervisor/runtime/last_exception is exactly the result of calling supervisor.runtime.last_exception

stuck elbow
#

I would prefer to just have one text file with the version and exception at the top level, personally

#

for ease of support

tulip sleet
#

i am just a little worried about adding another way to do something that is already available

#

but from a support point of view, it might be really valuable

stuck elbow
#

I mean it would take that exception from supervisor.runtime.last_exception of course

#

internally

tulip sleet
#

I think this would be worth bringing up in an issue or in the weeds, i think it's a valueable idea. The boot_out.txt buffer could be in RAM instead of flash. That would make it safer, and it could also note when CIRCUITPY is not available, which right now is a safe mode

#

I wonder if it not being persistent is an issue. I'm not sure -- are there cases where what is in boot_out.txt needs to survive a hard reset?

manic glacierBOT
stuck elbow
#

but it gets overwritten on boot anyways

tulip sleet
#

yes, boot_out.txt is always written, unless there's no filesystem. (Actually the write is suppressed if the message is identical, to save flash wear.)

stuck elbow
#

right, so it's not really persistent

tulip sleet
#

not sure how many files we want. Can it all be subsubmed into a single STATUS.TXT?

stuck elbow
#

or even reuse the boot_out.txt, since it's already there

#

a single file would be preferable in my opinion

tulip sleet
#

ya know, I don't think this is really going to work, because the MSC connection is block-oriented

stuck elbow
#

virtual blocks?

tulip sleet
#

or reserved blocks, as you first mentioned

stuck elbow
#

make the file entry point to a block that doesn't exist physically, and handle it as a special case

tulip sleet
#

we have to put hooks in oofatfs, I think

#

if we had multiple luns working it would be easier, but that may not ever work

mortal kernel
tulip sleet
#

or we could have a second pseudo-MSC device, maybe, not sure about endpoints

stuck elbow
#

I know this is hacky, but I would give a lot to have an easy way to check exceptions on the pewpew devices during workshops

#

it could also contain stuff like the current ip address

tulip sleet
#

i kind of want to encourage people to use the REPL -- I think it's very convenient for learning. Mu makes that much easier than putty, but yes, you have to have some easy way. I haven't seen a usable browser-based terminal program yet

stuck elbow
#

or even a qr code with an url to the web interface

tulip sleet
#

there is very little room left on the small boards

stuck elbow
#

but that would be all dynamically generated

#

that's the whole point

tulip sleet
#

i meant code space to implement this

#

or maybe you meant something else

stuck elbow
#

well, we already have internal code to get all of that information

#

as you pointed out with the last_exception

#

and we could remove the current boot_out.txt code

#

maybe that would make some room

tulip sleet
#

that is about five lines of code

#

in main.c

stuck elbow
#

so conditional for the larger boards only?

tulip sleet
#

@mortal kernel are you suggesting that we reserve some pseudo-clusters in the FS?

tulip sleet
mortal kernel
stuck elbow
#

so one physical device, but two partitions?

mortal kernel
#

Yes, we already present a partitioned device with one filesystem.

#

Related: Have we seen corruption at mount time? Tracing I've done leads me to think the boot.txt writes might not be safe across an automatic reload.

tulip sleet
#

boot_out.txt is only written after a hard reset, before USB is enabled

#

its original purpose was stdout for boot.py

mortal kernel
#

That's good, just wondered.

#

A second filesystem could be faked with very little flash. The whole thing could be 1 or 2KB with the VBR and FAT entirely faked.

tulip sleet
#

being read-only makes it pretty simple

mortal kernel
#

Absolutely, read only.

tulip sleet
#

i think this is very intriguing and we can cogitate on this, maybe open an issue or just talk about it here and in the next meeting (which is not until Jan 2, though)

#

i will mention it internally

mortal kernel
#

It would be possible to just present the NVRAM area as a single binary file in the fake filesystem.

manic glacierBOT
#
deshipu
[adafruit/circuitpython] Issue opened: #8737 Fake filesystem or a pseudo-file for status information

This is a loose idea for now, but I will dump what we have so far, so it can be further discussed.
The crux is to have either a pseudo-file, or a whole fake filesystem on the CircuitPython's USB MSC disk, to make available dynamically-generated status information from the running CircuitPython.

It would replace the boot_out.txt file, but because it's not physically written to the flash and does not wear the flash storage or consume extra space on the device, it could contain a variety o...

stuck elbow
#

I summarized what I understood from our discussion ^^

slender iron
tulip sleet
#

and maybe a fatal flaw

manic glacierBOT
tulip sleet
#

i'm not sure if we tried this already or not

stuck elbow
#

in the worst case we can ask people to reset

tulip sleet
#

then we lose the data, though

#

I think the stackoverflow provides the solution

stuck elbow
#

most of the data doesn't change after startup

#

the exception is the big one

#

on a side note, I wonder if marking the media as removable could help with the Sonoma bug?

tulip sleet
short tendon
#

What would people assume would happen if you don't use Session and just do:

import adafruit_requests
response = adafruit_requests.get("http://wifitest.adafruit.com/testwifi/index.html")

That it would:

  1. Raise an Exception
  2. Attempt to create a Session using socketpool.SocketPool(wifi.radio)
  3. Something else?

Right now it would error, since _default_session is set via set_socket

stuck elbow
manic glacierBOT
#
eightycc
[adafruit/circuitpython] New comment on issue #8737: Fake filesystem or a pseudo-file for status information

I'm fond of the fake filesystem approach for these reasons:

  • Does not depend on flash filesystem integrity. A corrupted or non-existent flash filesystem would not affect its ability to retrieve info from the fake filesystem.
  • No complex interaction with existing flash filesystem metadata such as the VBR, FATs, and directories.
  • No modifications to oofatfs are necessary to implement the fake filesystem.
  • CP code writes to a fake filesystem files are simply stores into backing RAM buf...
tulip sleet
#

or will Session support be transparent

short tendon
#

no that code will work exactly like it did before

#

the legacy ones that created the _FakeSSLContext to make a Session would need to change

tulip sleet
#

it would be kinda neat if you could just do adafruit_requests.get(), would need to see if wifi and radio existed and were connected

#

mostly I have never seen _FakeSSLContext in normal coding

short tendon
#

it's mostly used for ESP32spi

#

which can use Session too

manic glacierBOT
tulip sleet
short tendon
#

I have this in my test files:

try:
  import socketpool
  import wifi
  import ssl
  pool = socketpool.SocketPool(wifi.radio)
  esp32 = False
except:
  import adafruit_esp32spi.adafruit_esp32spi_socket as pool
  from adafruit_esp32spi import adafruit_esp32spi
  esp32 = True

The only thing missing is the call to pool.set_interface(esp) after the call to esp = adafruit_esp32spi.ESP_SPIcontrol(spi, esp32_cs, esp32_ready, esp32_reset)

tulip sleet
#

i'm in an internal meeting, so time-sharing on this

short tendon
#

In theory adafruit_requests.get() could try to import socketpool and then adafruit_esp32spi_socket and use the one it gets, or raise. Would need to play with it

#

Meaning there might be a world that all you need to do is import requests...

tulip sleet
#

not sure if you want automate the wifi.radio.connect()

#

as well

short tendon
#

that's true

tulip sleet
#

that is a slow process, maybe better to keep that explicit for now

short tendon
#

although could see something in the toml file to auto connect...

#

True, but I think we could not need the import for ssl, and socketpool

tulip sleet
#

you could defer these new features for now until the basic refactoring is done, though keeping it on the stove is good to make sure your API will work for that

short tendon
#

100%

#

I like small PRs

#

But I still need something for the adafruit_requests.get(). We could also have adafruit_requests.create_session(pool) (which would replace adafruit_requests.set_socket(pool)

manic glacierBOT
lone sandalBOT
manic glacierBOT
#
bill88t
[adafruit/circuitpython] New comment on issue #8737: Fake filesystem or a pseudo-file for status information

I like the idea a lot too, but there are some things to consider.

First of all, while it would be nice to have this in ram, a well written app could just implement most if not all of this functionality in boot.py, or in another file and check if a write needs to be performed.
Unless there is an error, the ip and stuff would probably be the same every time, so no writes.
The only constantly changing thing is status, but you could say, "if there are no errors, it's running".
`boot.p...

manic glacierBOT
#
anecdata
[adafruit/circuitpython] New comment on issue #8737: Fake filesystem or a pseudo-file for status information

I also like the idea of collecting traceback and potentially other debug info, particularly when the USB serial console is unavailable (battery projects, etc.) I think it should be non-volatile though, rather than RAM, so it persists across safe modes, or soft or hard resets, or unexpected losses of power.

Related to #8704, #7490, and #1054.

We have nvm (flash) and alarm.sleep_memory (RAM) for relatively small amounts of data, though as bytes only. Note that alarm.sleep_memory is ...

manic glacierBOT
#
ide
[adafruit/circuitpython] New comment on issue #8449: OSError: [Errno 5] Input/output: macOS Sonoma is delaying writes on small filesystems

/etc/fstab works on Sonoma 14.2.1 for a Raspberry Pi Pico W running CircuitPython 8.2.9. Get the volume UUID from diskutil and then run sudo vifs to add:

UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx none msdos rw,sync,auto

The key is the sync option (this blog helped). Now, when plugging in the Pico you should get:

$ mount | grep CIRCUITPY
/dev/disk4s1 on /Volumes/CIRCUITPY (msd...
manic glacierBOT
#
Freyr86
[adafruit/circuitpython] Issue opened: #8739 SPI bus management T-DECK Lilygo

I am working on programming the Lilygo T-Deck. I have chosen CircuitPython because of its flexibility, which is welcome for conducting tests.

In this module, there are two systems that operate on the SPI bus: the screen (ST7789) and a LORA transceiver (SX1262).

During the boot of CircuitPython on this board, the REPL activates on the screen, so the SPI is initialized for the screen. If I want to send data to the LORA module, I need to have the name of the object managing the SPI bus.

...

manic glacierBOT
manic glacierBOT
#
Freyr86
[adafruit/circuitpython] New comment on issue #8739: SPI bus management T-DECK Lilygo

It seems that the board.SPI() function is apparently working. I will have to investigate further because I have other issues in the library I'm using. In any case, a big thank you. I've been stuck on this issue for a while.

Regarding the board.DISPLAY.display_bus.spi_bus, it is not compatible with the board. I mention it here for those who might have the same issue.

Thanks, @deshipu

manic glacierBOT
manic glacierBOT
#
dhalbert
[adafruit/circuitpython] Issue opened: #8740 rotaryio works poorly on Espressif

found by user: https://forums.adafruit.com/viewtopic.php?t=206898

Reproduced on Feather ESP32-S2 with 7.3.3, 8.2.9, and 9.0.0-alpha.6. Also reproduced on Feather ESP32-S3 with 8.2.9.

With ANO encoder: counts OK in one direction, misses many counts and sometimes goes wrong way in other direction.
With regular encoder: many extra counts in either direction.

Tested with SAMD21 and RP2040 Feathers: both en...

manic glacierBOT
#
rsmets
[adafruit/circuitpython] New comment on issue #7409: CIRCUITPY doesn't mount properly when there are multiples of a device type on USB macOS 13.1

I am running into this same issue on mac os 13.4.1. Flashing nrf52840s directly from factory. However, given I using a new version of Circuit Python, 8.2.9, I feel this should be resolved, right?

@anecdata you solution

Closing, fixed by storage.erase_filesystem() with unique volume_uuid. See https://github.com/adafruit/circuitpython/pull/7410.
doesn't fully make sense looking at the other issue thread #7410. Would you mind shedding some light on it?

Much appreciated for the help!

cedar veldt
tulip sleet
manic glacierBOT
#
schaefer01
[adafruit/circuitpython] Issue opened: #8741 Crash into the HardFault_Handler, bricks rp2040

CircuitPython version

8.2.4. on rp2040

Code/REPL

I have 2 versions of my program/app, one that leads to the hard fault and an earlier version that does not. I can send both files to you but I also did a delta and the difference is an import of adafruit_ntp that has a May 2023 date on it

Behavior

Auto-reload is off.
Running in safe mode! Not running saved code.
You are in safe mode because:
CircuitPython core code crashed hard. Whoops!
Hard fa...

#
jepler
[adafruit/circuitpython] New comment on issue #8740: rotaryio works poorly on Espressif

I wrote a test for rotary encoders that just loops back a digital output signal to the input pins.

My test device was a Metro ESP32-S2 running 8.2.9.

I agree that it looks like the encoder is not counting correctly.

<details>
<summary>Test program></summary>

from rotaryio import IncrementalEncoder
from digitalio import DigitalInOut
import board

class QuadratureOut:
    ph_a = [0, 1, 1, 0]
    ph_b = [0, 0, 1, 1]

    def __init__(self, a, b):
        self.a = Dig...
#
bablokb
[adafruit/circuitpython] Issue opened: #8742 Inconsistent rendering of border pixels with vectorio

CircuitPython version

Adafruit CircuitPython 8.2.9 on 2023-12-06; Raspberry Pi Pico with rp2040

Code/REPL

import time
import board
import displayio
import busio
import adafruit_ssd1675

from adafruit_bitmap_font import bitmap_font
from adafruit_display_text import label as label
from vectorio import Rectangle
from terminalio import FONT as font

LOREM_IPSUM = (("Lorem ipsum "*3)+"\n")*11

def show_text(display):
  view = displayio.Group()
  s...
short tendon
midnight ember
#

in the middle of a project, will take a look later

manic glacierBOT
short tendon
manic glacierBOT
#
schaefer01
[adafruit/circuitpython] Issue opened: #8743 bricked rp2040, tried the given steps to recover but no recovery

CircuitPython version

adafruit 8.2.4 on rp2040

Code/REPL

See #8741 for steps to reproduce (summary: invoke an out-of-date adafruit_ntp library)

Behavior

the rp2040 circuitpy goes away, whatever you put into RPI-RP2 does not return with a CIRCUITPY drive

Description

tried flash_nuke.uf2, and it did not result in creating a CIRCUITPY drive on reboot or power cycle.

I followed flash_nuke.uf2 by Feather_RP2040.uf2, this too, did not result ...

#
schaefer01
[adafruit/circuitpython] New comment on issue #8741: Crash into the HardFault_Handler, bricks rp2040

Ooops, this worked until I cycled power. Then the same problem as before:

.]0;üêçDone | 8.2.4.\Auto-reload is off.
Running in safe mode! Not running saved code.

You are in safe mode because:
CircuitPython core code crashed hard. Whoops!
Hard fault: memory access or instruction error.
Please file an issue with your program at github.com/adafruit/circuitpython/issues.
Press reset to exit safe mode.

Press any key to enter the REPL. Use CTRL-D to reload.
soft reboot

#
schaefer01
[adafruit/circuitpython] New comment on issue #8741: Crash into the HardFault_Handler, bricks rp2040

Crashed another, so it is not NTP. There has to be a different difference between the two files that causes the soft reboot. I'm making a copy of each and renaming the .py files to .txt and attaching. The first one is "safe" which doesn't reboot, the second is "remagen" because it is a version of code too far.
Also I can now recover due to help I received in problem #8743
mqtt_ether-safe.txt
[mqtt_ether-remagen...

short tendon
#

Both requests and minimqtt have a _FakeSSLSocket , does it make sense to move that into adafruit_connectionmanager as adafruit_connectionmanager.create_generic_ssl_context()? I would put it in adafruit_esp32spi, but assume there are other libs that would need it as well

manic glacierBOT
#
schaefer01
[adafruit/circuitpython] New comment on issue #8741: Crash into the HardFault_Handler due to adafruit_ntp

I can replicate the code crashed problem. It is a combination of two issues:

  1. import and use of ntp
  2. looping on oled button press the adafruit button library .update() call
    If I comment out both kinds of references, the problem goes away. If I use either or both, the problem comes back. I'm attaching the file that works that is closes to remagen but with imports and button update
    [mqtt_ether-no-ntp.txt](https://github.com/adafruit/circuitpython/files/13745680/mqtt_ether-no-ntp.txt...
midnight ember
#

Who made Mu because I need to buy them a beer. Turns out every code session is stored deep in appdata/local/python/mu. Just recovered code after a board died because of Mu. ❤️

tulip sleet
candid sun
#

that's one of my favorite features. it's saved me a couple of times

manic glacierBOT
tulip sleet
#

@onyx hinge are you thinking of working on the rotaryio issue? I am inclined to forget about PCNT and use interrupts, so any number of encoders can be set up.

#

I could work on it in the next few days

random junco
manic glacierBOT
onyx hinge
#

@midnight ember I think that ntoll is a key person behind mu. they have a github sponsors. https://github.com/ntoll

GitHub
ntoll - Overview

Music, philosophy, teaching, writing & computing. Just like this bio: concise, honest and full of useful information. Everything I say is false... ;-) - ntoll

midnight ember
#

Not the kind of guide i was expecting to write. Makes sense to document the ordeal.

#

Been at it since 10pm last night trying to resurrect that feather. I’m mentally exhausted.

manic glacierBOT
manic glacierBOT
#
jeremyamoore
[adafruit/circuitpython] New comment on issue #8449: OSError: [Errno 5] Input/output: macOS Sonoma is delaying writes on small filesystems

Tried the "vifs" work around with a new Memento board (ESP32-S3) and also seems to work.

In the Sonoma release notes, there is this "new feature" - wonder if this is having some impact when mounted as async:

The implementations of the exfat and msdos file systems on macOS have changed; these file systems are now provided by services running in user-space instead of by kernel extensions. If the application has explicit checks or support for either the exfat or msdos file systems, validat...

manic glacierBOT
#
jepler
[adafruit/circuitpython] New comment on issue #8740: rotaryio works poorly on Espressif

well I'm puzzled. setting a pull up explicitly makes no difference. source shows says a pull up is configured anyway when setting up pcnt with esp-idf calls. and implementing my own "fake open bus" mode via .switch_to_input() for a True output fails in the same way (so it doesn't seem to be the a problem of the open bus mode on esp32s2, which we've had before on other platforms)

In fake open-bus mode I can read back the bus value as high (True) so I know there's really a pull there.

I s...

onyx hinge
#

yes it's still simulated

manic glacierBOT
onyx hinge
#

the great thing about quadrature is that bounce doesn't actually matter, at least as long as only one of the A or B switches is undergoing contact bounce at any given time. When the switch settles, the value will be right; and before it's settled it can only be off by 1 count (1/4 electrical revolution) (and /2 or /4 mode with hysteresis fixes that) If both switches are bouncing at the same time, yes, you can't keep count. It's true that incorrect quadrature counting algorithms will often have their wrong counting be exacerbated by contact bounce, but fundamentally that's because they're incorrect implementations of quadrature decoding.

The fact that all the valid quadrature state transitions works correctly for me in push-pull output mode but isn't doing so in open bus mode makes me think it must be something about the GPIO pad configuration, but .. it looks right.

tulip sleet
#

there is a "glitch remover" flag in the PCNT config, but I haven't looked at that

onyx hinge
#

but if all valid quadrature transitions count correctly then it must be right 🙂

tulip sleet
#

in any case, I think usin PCNT is not a great choice because it's limited to one encoder (right?). It's been on my distant radar to not use this peripheral. We made a similar decision about not using the quadrature periph on SAMd51

#

i thought maybe it was something that changed recently (could have been pin reset state, for instance), but it's bad in 7.3.3 also

#

maybe it's good in 6.x

onyx hinge
#

I do think that if the PCNT peripheral is working like it's documented, the implementation of quadrature counting is correct but obscure. It counts correctly on all 8 valid quadrature transitions in isolation when driven push-pull, anyway.

#

so my money's on it being some kind of GPIO set up mistake

#

but that's not to say we shouldn't do the pin change interrupt thing, it does offer additional flexibility

tulip sleet
#

I fixed something about this years ago if you look at "blame"

#

maybe my fix was wrong, but I did think I tested it.

#

but if that was in 6.x, it could have broken since then

#

well, this was not even in 6.x

onyx hinge
#

also esp-idf seems to have deprecated the pcnt APIs we call, they may have added their own resource allocation layer or something? and it goes by handles now.

#

also some of the comments are wrong imo

-        .pos_mode = PCNT_COUNT_DEC,      // Count up on the positive edge
-        .neg_mode = PCNT_COUNT_INC,      // Keep the counter value on the negative edge
+        .pos_mode = PCNT_COUNT_DEC,      // Count down on the positive edge
+        .neg_mode = PCNT_COUNT_INC,      // Count up on negative edge
tulip sleet
#

that would mess up a weak pull-up, I think

#

retesting with d11 and d12 with a newer version

#

classic rotary encoder is fine. Let me try ANO

#

smooth with ANO now. Maybe I had a wiring problem before. I will close this issue with a caveat, and respond to the user.

onyx hinge
#

encoders are apparently easy to wire subtly wrong

tulip sleet
#

i was just breadboarding, maybe have a had loose connection

onyx hinge
#

AFK for the afternoon. thanks for looking into it Dan!

tulip sleet
#

thanks for the testing. what a goose chase

manic glacierBOT
onyx hinge
#

I'm glad that I now understand how the use of PCNT is supposed to be working.

tulip sleet
manic glacierBOT
crimson ferry
#

What's the rule of thumb when requesting PIDs (from Espressif, for example)... I've seen typically 3 PIDs are requested: 1 for CircuitPython, 1 for bootloader, and 1 for Arduino. #8744, for example, requested 1 PID from Espressif (btw that PR isn't merged yet and conflicts with another open PID PR... I can comment on the CP PR if desired)

tulip sleet
crimson ferry
tulip sleet
#

I thought Espressif assigned the PID's

crimson ferry
#

it's... interesting, you're supposed to pick the next available in the sequence

#

do we have PIDs in a learn guide or doc somewhere... we could include some of the requirements, naming, etc

tulip sleet
manic glacierBOT
#
dhalbert
[adafruit/circuitpython] New branch created: revert-8744-flipper-wifi-dev-pid
crimson ferry
#

I commented on the learn guide

manic glacierBOT
#
dhalbert
[adafruit/circuitpython] New comment on pull request #8746: Ad Led_IR pin name to ATOM Lite

@sctv is this what you had in mind?

@carson-coder two things:

  1. If you do this against 8.2.x instead of main then it can get into the next 8.2.x release. You would make the change off a branch that is branched from 8.x.2. The board is available in 8.2.x. Then we will merge it to main when we catch up main to 8.2.x.
  2. You made this change on your main branch. It's generally better to create a new branch off the dev branch (in this case, either main or 8.2.x). Otherwise...
manic glacierBOT
manic glacierBOT
manic glacierBOT
manic glacierBOT
short tendon
#

Hello! I had an idea on requests and sessions and keeping things like adafruit_requests.get around. And even making them better...

Define Sassion as so:

_global_session = None  # pylint: disable=invalid-name

class Session:
    def __init__(
        self,
        ...
        set_global_session: bool = True,
    ) -> None:
        ...
        if set_global_session:
            global _global_session
            _global_session = self

and the get is:


def get(url: str, **kw):
    """Send HTTP GET request"""
    return _global_session.request("GET", url, **kw)

And now you can do:

adafruit_requests.Session(pool, ssl_context)
response = adafruit_requests.get("https://www.adafruit.com/api/quotes.php")

The other benefit is that in a seperate file you can just do:

import adafruit_requests

def test():
  print(adafruit_requests.get("https://www.adafruit.com/api/quotes.php").text)

And it will work. Thoughts?

midnight ember
#

haven't had time to sit down and try out your example. from the looks of how long my helper project is taking it'll be a few days too.

#

will the get have a place to add headers and content type? that is fairly important for working with most online API's.

celest marsh
#

the way get() is defined with **kw then you can pass in other parameters and they will flow thru the downstream call

#

the only thing I can see after a quick look is that I would "harden" the way you implemented get() by having a check if _global_session is None when doing return _global_session.request("GET", url, **kw)

short tendon
#

It's how the legacy ones are defined currently. My main goal is to get ConnectionManager done, with the fewest changes. Then start building up

midnight ember
#

being able to check for if_connected status is important too yup

#

ConnectionManager is a good name. 👍

celest marsh
#

to be sure my thoughts are more "oh, I see this" items, I am fully appreciating the work to make this both useful, functional and pythonic - great job

short tendon
#

It's a good note. I personally would love to get coverage in these libs to 100% and have good exceptions. That way when we help in #help-with-circuitpython we see specific errors and know wha't wrong

celest marsh
#

indeed

#

that urge, and the conversation about having a virtual filesystem or file to log debug messages, I see as really helping folks debug and also a boon to those trying to help folks. The ability to say "set CP_DEBUG_LOG=1" and check this file and then get some non-obvious logging help is important

#

when I'm coding my own projects, every time I have a "what is this doing print() call" I wrap them in if _debug: so they become living documentation

short tendon
#

the other thing is there is adafruit_logger. If the libs tried to import that by default and then logged if it was there, we could also just say Hey! add this to you lib folder and show us the logs

midnight ember
#

yup I've been doing the same. instead of removing all the print statements that i had to do along the way just wrap them in If_debug just in case I need them again someday. especially for online api's that change seemingly annually.

celest marsh
#

yep, getting that to be something that can easily enable debug logs, especially if adafruit_logger can inherite a destination device to send it's data towards

short tendon
short tendon
#

Is there a way to make a real ssl_context when using something like an esp32spi?

midnight ember
#

that would be buried in esp32spi somewhere. unlike requests when you use esp32spi the connection part is mostly done for you by the library including exception handlers. the difference in using esp32spi vs the way you use wifi and socketpool with an esp32-s2/s3 is still quite far apart.

lone axle
# short tendon the other thing is there is `adafruit_logger`. If the libs tried to import that ...

it is slightly trickier than this in most cases because the device will likely be mounted as read only (the default behavior) so the log file cannot be written unless an SDCard or similar is available to write to, or the drive has been explicitly mounted to writable from boot.py which then means the PC it's plugged into can't write to it so editing code files and libraries becomes more challenging.

#

But I am definitely in favor of the overall general idea of the libraries logging what they can if it's available and us compiling the easiest instructions to enable that for users that are having trouble.

lone axle
short tendon
midnight ember
#

i've been using esp32-s3 for so long that when i recently had to use an airlift and esp32spi there are some stark differences in syntax. would love to see a library that incorporates everything so only 1 library is needed regardless of what online project you're trying to do or what board you're using. no idea if that's a realistic goal to merge everything like that. from a usability standpoint it would make learn guides, examples, and docs for connection initiation far more consistent.

short tendon
#

That's part of my goal

lone axle
#

Yeah cookiecutter is a good place to start. We can get the try/catch fallback syntax nailed down there so that by the time it's making it to other librariers we have some consistency.

short tendon
#

To get them as close as possible

short tendon
lone axle
#

Thanks for all of your work on it 🙂

short tendon
#

I also have a tox file that I would love to see get in there - with tests, lint, coverage and docs

midnight ember
#

I know foamyguy started on making esp32spi code to act more like wifi & socketpool a couple months ago. that was a nice stream. it's come a long way but still a long way to go.

#

a unifying library and syntax i'm all for. when looking up examples for online projects half of them are esp32spi and half are wifi & socketpool. then you have users asking for help with projects that have an esp32-s3 trying to use esp32spi example code. i think it'll help cut down the amount of support issues for online projects too.

lone axle
#

Dan did the heavy lifting of what I was working on. He worked on equalizing the socket API to the extent possible. I fixed a few of the libraries built on top of it that had come to rely on non-standard socket behavior.

midnight ember
#

everything is built on top of something with open source. the continual improvements because of people like.... well everyone that usually hangs out in /dev chat is really what makes circuit python better year after year.

short tendon
#

This is what I'm using for a quick test. For general http requests, works on both:

from os import getenv
import board
import busio
from digitalio import DigitalInOut
import adafruit_requests
import adafruit_connectionmanager

try:
  import adafruit_esp32spi.adafruit_esp32spi_socket as pool
  from adafruit_esp32spi import adafruit_esp32spi
  esp32 = True
except:
  import socketpool
  import wifi
  import ssl
  pool = socketpool.SocketPool(wifi.radio)
  esp32 = False

wifi_ssid = getenv("CIRCUITPY_WIFI_SSID")
wifi_password = getenv("CIRCUITPY_WIFI_PASSWORD")

url = "https://www.adafruit.com/api/quotes.php"

print("Connecting to WiFi")
if esp32:
  esp32_cs = DigitalInOut(board.D13)
  esp32_ready = DigitalInOut(board.D11)
  esp32_reset = DigitalInOut(board.D12)
  spi = busio.SPI(board.SCK, board.MOSI, board.MISO)
  esp = adafruit_esp32spi.ESP_SPIcontrol(spi, esp32_cs, esp32_ready, esp32_reset)

  ssl_context = adafruit_connectionmanager.create_fake_ssl_context(pool, esp)

  while not esp.is_connected:
    try:
      esp.connect_AP(wifi_ssid, wifi_password)
    except OSError as e:
      print(f"could not connect to AP, retrying: {e}")
      continue

else:
  ssl_context=ssl.create_default_context()

  while not wifi.radio.connected:
    wifi.radio.connect(wifi_ssid, wifi_password)

requests = adafruit_requests.Session(pool, ssl_context)
response = requests.get(url)
print(response.json())
midnight ember
#

I like it, wrap it up into a library that's easy to use so both esp32spi and requests use the same syntax.

short tendon
#

With the same comments for different pins

midnight ember
#

i'd like to see it where all that is needed regardless if it's esp32spi or socketpool is something like ```py
while not wifi.is_connected:
try:
wifi.connect(wifi_ssid, wifi_password)
except OSError as e:
print(f"could not connect to AP, retrying: {e}")
continue
response = requests.get(url)

short tendon
#

Right now I'm getting close to the point that there's little more I'll do until all the smart people in this room say they like what I'm doing. Cause it either get's adopted or I'll just have forks for my projects

midnight ember
#

also one issue i ran into with esp32spi.Control is the try/except around connection never produces an exception handler you can use because it's got one internally... which attempts to connect to wifi forever. so i found it pointless to use a try/except with it. it was not helpful as i couldn't do what i wanted with an exception as it would never trigger.

onyx hinge
#

uh oh / argh. circuitpython (main branch, espressif) can't connect to my own website. It happens to have a letsencrypt certificate but idk if that's the reason or not.

#

I think it's because I have configured that site to support TLS 1.3 only. TLS 1.3 support is optional, default N in esp-idf.

        config MBEDTLS_SSL_PROTO_TLS1_3
            bool "Support TLS 1.3 protocol"
            depends on MBEDTLS_TLS_ENABLED && MBEDTLS_SSL_KEEP_PEER_CERTIFICATE && !MBEDTLS_DYNAMIC_BUFFER
            select MBEDTLS_HKDF_C
            default n
#

jpeg image HTTP streamed & decoded with circuitpython 🎉

short tendon
onyx hinge
#

I connected to a different server that also uses a letsencrypt certificate. I'm reasonably confident it's the TLS 1.3 thing.

stuck elbow
short tendon
#

So don't set it as pseudo private?

#

And then be specific you want to set it?

stuck elbow
#

less magic = good

#

just document in the docs that the global is there

manic glacierBOT
#
jepler
[adafruit/circuitpython] New comment on pull request #8723: Jpegdecoder enhancements

This is now able to stream-decode a JPEG. However, in my test it was not particularly fast.

  +0.0000: connecting to wifi
  +3.0908: fetching image
 +16.0797: status 200
 +17.0609: Image size is 196x200
 +17.0610: Decoding image
 +19.0146: Decoding complete

<details>
<summary>CircuitPython code for demo on Memento / PyCamera</summary>

import io
import os
import ssl
import time

import displayio
import adafruit_requests
import board
import jpegio
import...
short tendon
stuck elbow
orchid cliff
short tendon
manic glacierBOT
short tendon
manic glacierBOT
onyx hinge
#

can't figure out why my website doesn't support TLS 1.2. 😧

tulip sleet
onyx hinge
#

It's self-hosted apache

#

debian apache configuration is a bit of a soup but it looks like this is the configuration that should be active: SSLProtocol TLSv1.2 TLSv1.3

tulip sleet
#

I see SSLProtocol -all +TLSv1.2 +TLSv1.3 in a tutorial. don't know if the + signs are significant

short tendon
tulip sleet
short tendon
#

@tulip sleet are you willing to peek at the ConnetionManager code and see if it generally feels right?

tulip sleet
# short tendon <@329766224093249548> are you willing to peek at the `ConnetionManager` code and...

I really am not an expert on this at all. The people who have been helping here use it a lot more than me. You could submit draft PR's to the adafruit libraries, and say to use your new library as well. You could also transfer your new library, owned by you to us. It would be nice to have its contents be a PR that could be reviewed. That could be done by starting with an empty cookie cutter and then you could PR against that.

onyx hinge
#

@tulip sleet thanks. I was just sure my configuration was right, because I had a line that listed 1.2 and 1.3 both. but apparently it is wrong.

SSLProtocol             TLSv1.2 TLSv1.3   # Wrong, enables only 1.3
SSLProtocol             -all +TLSv1.2 +TLSv1.3 # correct, enables both
tulip sleet
#

@Justin or you could just refer the draft PR's to your library for now, and people can submit GitHub comments on that

tulip sleet
onyx hinge
#

@tulip sleet yeah it sure fooled me.

short tendon
tulip sleet
short tendon
#

I can also redo my repo and commit the cookiecutter first and open a PR to that and mention it in the other two draft ones

tulip sleet
manic glacierBOT
short tendon
#

Cool. Will make 3 PRs later today

#

Is there anyone I can tag in GH so they can look when they have time?

manic glacierBOT
tulip sleet
short tendon
#

Oh I do

#

you should see the tox file I added 😉

#

I made sure, docs, lint and tests all pass

tulip sleet
#

maybe adadfruit_connectionmanager should be temporarily be marked as mock to avoid the errors.

short tendon
#

Tests all pass locally. Not sure why it can't pull it down.

#

mocking it out needed a lot of code to get all the tests to pass with all the different socket cases those two libs have

tulip sleet
short tendon
#

but a pip install -r requirements should handle a public GH repo...

tulip sleet
#

i don't know, need to look in the CI log

#

yes, looks like it

# 
2023-12-22T17:28:55.4431322Z Collecting Adafruit-Circuitpython-ConnectionManager@ git+https://github.com/justmobilize/Adafruit_CircuitPython_ConnectionManager (from -r requirements.txt (line 6))
2023-12-22T17:28:55.4434810Z   Cloning https://github.com/justmobilize/Adafruit_CircuitPython_ConnectionManager to /tmp/pip-install-9z_e5nq8/adafruit-circuitpython-connectionmanager_0625f9b6fd1a4a8b9f6c7478cd04f6cb
2023-12-22T17:28:55.4462708Z   Running command git clone --filter=blob:none --quiet https://github.com/justmobilize/Adafruit_CircuitPython_ConnectionManager /tmp/pip-install-9z_e5nq8/adafruit-circuitpython-connectionmanager_0625f9b6fd1a4a8b9f6c7478cd04f6cb
2023-12-22T17:28:56.2101433Z   Resolved https://github.com/justmobilize/Adafruit_CircuitPython_ConnectionManager to commit 8cc811efe6396cc4215a9857b2abcc5a9348bd73
#

ImportError: cannot import name 'get_connection_manager' from 'adafruit_connectionmanager'

#

hmm

#

btw, trivial naming thing, but maybe connection_manager everywhere instead of adafruit_connectionmanager, due to get_connection_manager, etc.

#

I'd prefer separate words

#

if it's camel-cased in mixed cased, then can change to _ in all-lower-case

#

example Adafruit_CircuitPython_BusDevice -> adafruit_bus_device

manic glacierBOT
#
sctv
[adafruit/circuitpython] New comment on pull request #8747: Add Led_IR pin name to ATOM Lite

@dhalbert I have downloaded the artifact adafruit-circuitpython-m5stack_atom_lite-en_US-20231221-6a0c8ac.bin to my computer, but having trouble flashing it. Using https://espressif.github.io/esptool-js/ generates a warning: Image file at 0x1000 doesn't look like an image file, so not changing any flash settings.
@carson-coder The uf2 would be useful, but this vESP32-PICO-D4 doesn't turn up as a drive.

short tendon
tulip sleet
short tendon
#

man, I'm an idiot

#

Since I re-did the repo main was just CookieCutter, needed the branch name in the requirements....

#

@tulip sleet renamed from adafruit_connectionmanager to adafruit_connection_manager and all tests passing in the other repos

mortal kernel
#

I'm cleaning up event handling in my rp2040 _bleio and have a question about memory management. My code is modeled on the nrf implementation that uses a mixture of static and dynamically (on the heap) allocated objects to represent registered event callbacks. So, the list of callbacks will contain a mixture of both. In one case, nrf resets the list by setting its root to NULL, and in another it selectively removes only event handler objects on the heap. I'm duplicating this behavior, but I would like to understand why.

mortal kernel
#

From ports/nrf/bluetooth/ble_drv.c:

void ble_drv_reset() {
    // Linked list items will be gc'd.
    MP_STATE_VM(ble_drv_evt_handler_entries) = NULL;
    sd_flash_operation_status = SD_FLASH_OPERATION_DONE;
}

void ble_drv_remove_heap_handlers(void) {
    ble_drv_evt_handler_entry_t *it = MP_STATE_VM(ble_drv_evt_handler_entries);
    while (it != NULL) {
        // If the param is on the heap, then delete the handler.
        if (gc_ptr_on_heap(it->param)) {
            ble_drv_remove_event_handler(it->func, it->param);
        }
        it = it->next;
    }
}
short tendon
#

Since it would be for global use, would also want things like addHandler, removeHandler and setLevel in there too

manic glacierBOT
#
romkey
[adafruit/circuitpython] New comment on issue #8449: OSError: [Errno 5] Input/output: macOS Sonoma is delaying writes on small filesystems

Using fstab this way does not solve the problem for me on macOS Sonoma 14.3beta. Haven't had a chance to try it on 14.2.1 yet.

I tried sync, noasync and both of them with freshly erased CIRCUITPY drives on a Matrix Portal M4, all combinations still had the problem.

I think the effective workaround is unmounting without ejecting, then mounting again, and "noasync" was a red herring because to set that you had to do the umount/mount sequence. That sequence works for me without changing an...

manic glacierBOT
cedar veldt
#

SleepMemory survives reset on RP2040, and I think I read somewhere earlier that it doesn't on espressif, so I was wondering why that is.

manic glacierBOT
#
eightycc
[adafruit/circuitpython] New comment on issue #8449: OSError: [Errno 5] Input/output: macOS Sonoma is delaying writes on small filesystems

@romkey I'm perplexed, too. But, I have a theory. Earlier @jeremyamoore noted that Sonoma had moved fat filesystems from kernel to user space. I'd missed that entirely, so all my reading of the kext code never turned up a plausible bug. It's possible that the automatic mount is using the new user space filesystem code while a command line mount is pulling in the older kernel extension (kext). Now to devise an experiment to see if that's the case.

quasi coral
#

Hello from Tokyo all

manic glacierBOT
#
eightycc
[adafruit/circuitpython] New comment on issue #8449: OSError: [Errno 5] Input/output: macOS Sonoma is delaying writes on small filesystems

My theory panned out, the reason that behavior differs after a manual mount of the filesystem is that it brings in kernel space kext com.apple.filesystems.msdosfs which then gets used instead of the new user space driver.

The older kext does not delay meta data writes while the newer user space driver does. Apple has two entirely different fat filesystem drivers, which one gets used depends on the mechanism used to mount the filesystem.

cedar veldt
#

Hope they don't "fix" it to use the new driver all the time without actually fixing the new driver 😅

manic glacierBOT
manic glacierBOT
manic glacierBOT
manic glacierBOT
manic glacierBOT
manic glacierBOT
#
300bps
[adafruit/circuitpython] Issue opened: #8750 socketpool.SocketPool().socket: Add ability to specify protocol for raw sockets to more closely align with CPython.

First, CircuitPython has great networking support! It is amazing how much you can do with it. While I'm aware of wifi.radio.ping, I've recently been attempting to manually create an ICMP ping as a RAW packet and have run into a bit of a speed bump. I think I see a fairly simple possible solution that I'll describe below. Note that I'm using CircuitPython 9 on a RPi Pico.

The issue:
To create an ICMP packet, in addition to specifying a socket type of SOCK_RAW, you also need to be...

manic glacierBOT
manic glacierBOT
#
bablokb
[adafruit/circuitpython] New comment on issue #8742: Inconsistent rendering of border pixels with vectorio

Below are five images with varying parameters. The first, third and fifth are the expected result. But for the third and fifth I have to tweak the values so the result is correct.

If you take a closer look at the second image, you can see that the border on left is missing and the border on the right is two pixels wide. In the fourth image, the bottom border is missing and the top border is two pixels wide.

  • adafruit_display_shapes.rect.Rect: rotation=270, x=0,y=0 and x=1,y=1:
    ![di...
manic glacierBOT
manic glacierBOT
manic glacierBOT
ornate breach
#

has anyone investigated the internal watchdog expiration on ESP32-S3 for audiobusio I2SOut? I'm curious if the reason the expiration is happening, is because of timers not properly being released. And maybe it's something specific to how the ESP32-S3 handles multiple timers.

#

One thing I noticed is that when it's by itself, it's a bit hit and miss if the watchdog expires. If I remove the loop=True bit from the play(sample) call, it doesn't seem to fail.

manic glacierBOT
manic glacierBOT
mortal kernel
#

@danh Wondering if you are available for a brief consult regarding BLE events? If not, it can wait until after the holidays.

keen eagle
#

@mortal kernel I won't pollute github with this but if you enjoy those movies and haven't checked it out yet, you might enjoy Monarch on Apple's streaming service (sorry for being massively off-topic 🙂 )

manic glacierBOT
tulip sleet
mortal kernel
pulsar sleet
#

How do I build circuitpython for the esp32_devkitc_1_n8. I tried the command
make BOARD=espressif_esp32s3_devkitc_1_n8
But I keep getting this error

  The C compiler

    "/usr/bin/xtensa-esp32s3-elf-gcc"

  is not able to compile a simple test program.

  It fails with the following output:

    Change Dir: '/media/carson/2TB-Drive/circuitpython/ports/espressif/build-espressif_esp32s3_devkitc_1_n8/esp-idf/CMakeFiles/CMakeScratch/TryCompile-yoYWrm'
    
    Run Build Command(s): /usr/bin/ninja -v cmTC_1801f
    [1/2] /usr/bin/xtensa-esp32s3-elf-gcc   -mlongcalls -o CMakeFiles/cmTC_1801f.dir/testCCompiler.c.obj -c /media/carson/2TB-Drive/circuitpython/ports/espressif/build-espressif_esp32s3_devkitc_1_n8/esp-idf/CMakeFiles/CMakeScratch/TryCompile-yoYWrm/testCCompiler.c
    FAILED: CMakeFiles/cmTC_1801f.dir/testCCompiler.c.obj 
    /usr/bin/xtensa-esp32s3-elf-gcc   -mlongcalls -o CMakeFiles/cmTC_1801f.dir/testCCompiler.c.obj -c /media/carson/2TB-Drive/circuitpython/ports/espressif/build-espressif_esp32s3_devkitc_1_n8/esp-idf/CMakeFiles/CMakeScratch/TryCompile-yoYWrm/testCCompiler.c
    xtensa-esp32s3-elf-gcc: error trying to exec 'cc1': execvp: No such file or directory
    ninja: build stopped: subcommand failed.
    
    

  

  CMake will not be able to correctly generate this project.
Call Stack (most recent call first):
  esp-idf/tools/cmake/project.cmake:296 (__project)
  CMakeLists.txt:19 (project)```
spare jacinth
#

did you setup ESP's tooling?

pulsar sleet
#

no. How do I do that

spare jacinth
#

try and do . esp-idf/export.sh from ports/espressif and it probably guides you what/how to do

#

mainly installing a Python virtualenv and some toolchains + export env variables

pulsar sleet
#

what is a IDF_PATH

spare jacinth
#

one of those env variables that ESP's tooling uses to set-up the build system and/or build (the folder where to find some stuff, likely)

#

anyway, given you are apparently not too familiar with this... do you really need to do a custom build instead of downloading a bin/uf2?

pulsar sleet
spare jacinth
#

welp. as said, trying to run the shell script (export.sh) to get the environment variables set (done with the .) should tell you something like "hey, ESP stuff not installed/found, do it like this: ..."

pulsar sleet
#

I am getting this error

If you have set IDF_PATH manually, check if the path is correct.```  I am assuming I need to download something that I am trying to figure out what to install
spare jacinth
#

IDF_PATH should be pointing to the esp-idf folder on circuitpython/ports/espressif, IIRC

#

does your .bashrc define it/have you tried using ESP-IDF yourself in the past?

pulsar sleet
#

I looked online it it just said to set IDF_PATH to ~/esp/esp-idf but I can change it

stuck elbow
#

that's what you have it set to

#

you forgot to run the install.sh script from esp-idf

pulsar sleet
#

I fixed it by adding the . in the command. I thought that it was a typo but I have it working now

spare jacinth
#

welp, that would do it 🤣

#

afaik you dont need to install anything nor configure env variables beforehand

#

just let it guide you through

pulsar sleet
#

after running idf.py I get this error
```home/carson/.espressif/tools/xtensa-esp32-elf/esp-2021r2-patch5-8.4.0/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: esp-idf/freertos/libfreertos.a(port_common.c.obj):(.literal.main_task+0x1c): undefined reference to app_main' /home/carson/.espressif/tools/xtensa-esp32-elf/esp-2021r2-patch5-8.4.0/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: esp-idf/freertos/libfreertos.a(port_common.c.obj): in function main_task':
/media/carson/2TB-Drive/circuitpython/ports/espressif/esp-idf/components/freertos/port/port_common.c:135: undefined reference to `app_main'
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.
ninja failed with exit code 1

spare jacinth
#

you dont have to run idf.py manually, tho

#

that kind of looks like you tried to manually call the compiler (using idf.py)

pulsar sleet
#

I ran . esp-idf/export.sh and it said

Go to the project directory and run:

  idf.py build```
spare jacinth
#

yeah, but CPy does (or might be doing) a lot of stuff on top of the idf.py call

#

you still use make

#

you use the . esp-idf/export.sh to setup environment such that you can build, and then use make (just like you attempted before)

manic glacierBOT
pulsar sleet
#

Ok. How do I install dependences because I already ran make fetch-all-submodules but I am getting File "/media/carson/2TB-Drive/circuitpython/ports/espressif/../../tools/gen_web_workflow_static.py", line 7, in <module> import minify_html ModuleNotFoundError: No module named 'minify_html' do I need to run make fetch-all-submodules (or port submodules) again

spare jacinth
#

that's a python error, you need to pip install ... the module

#

pip install -r circuitpython/requirements.txt (or whatever the file was named) probably installs everything

#

if still failing, just replace the ... with the module's name

#

repeat with any module that may error, until you succeed 🤣

pulsar sleet
spare jacinth
#

it doesnt tell you to run it

pulsar sleet
#

I am not running it but it doesn't tell me to do anything else

#

but running make wont work

spare jacinth