#circuitpython-dev

1 messages · Page 58 of 1

solar whale
#

and the code ```
import binascii
import io
import busio
import os
import ssl
import time
import board
import socketpool
import wifi
import adafruit_requests
from adafruit_io.adafruit_io import IO_HTTP, AdafruitIO_RequestError

aio_username = os.getenv('AIO_USERNAME')
aio_key = os.getenv('AIO_KEY')
i2c_bus=busio.I2C(scl=board.SCL,sda=board.SDA)

pool = socketpool.SocketPool(wifi.radio)

requests = adafruit_requests.Session(pool, ssl.create_default_context())

Initialize an Adafruit IO HTTP API object

io = IO_HTTP(os.getenv("AIO_USERNAME"), os.getenv("AIO_KEY"), requests)

Adafruit IO feed configuration

try:
# Get the 'camera' feed from Adafruit IO
feed_camera = io.get_feed("esp32s3eye")
except AdafruitIO_RequestError:
# If no 'camera' feed exists, create one
feed_camera = io.create_new_feed("esp32s3eye")
print("got feeds")

tulip sleet
#

thanks - I misunderstood and thought it was at beta.0

#

I could do a bisect

solar whale
#

YEs -- I can also try in the morning, but feel free to look into it.

#

some unnecessary code above -- quick hack

#

fails at feed_camera = io.get_feed("esp32s3eye")

#

This really confused me today when I was testing the ov2460 camera since once it worked, I saw these errors....but I verified this also occurs on my memento and a feather esp32s3(4mbflash/2mbytepsram) I have not tried other MCUs ... yet.

#

I need to go now, but will be happy to dig into it in the morning if you have not found anything.

manic glacierBOT
#
eightycc
[adafruit/circuitpython] New comment on issue #8449: OSError: [Errno 5] Input/output: macOS Sonoma is delaying writes on small filesystems

@romkey and @dahlbert I can confirm that 14.4 beta 2 fixes this issue. Tracing shows meta-data updates now follow a data write within a few hundred milliseconds. Like this:

flash_write_blocks: time(ms)=406324, dev_block_num=   42, num_blocks=    1, data
block 41 (data) : cluster 7.0
0000: 94 25 f6 76 cb 55 6d ea be ee 93 af 52 69 71 bf .%.v.Um.....Riq.
0010: f9 64 fe 8f d9 8f e2 9e 0f 0d 00 05 95 fa d1 89 .d..............
0020: ce 52 85 be 35 78 0f c2 a3 18 e5 af ef 8c d3 b4 .R..5...
manic glacierBOT
short tendon
manic glacierBOT
#
czei
[adafruit/circuitpython] New comment on issue #8449: OSError: [Errno 5] Input/output: macOS Sonoma is delaying writes on small filesystems

Thanks for doing the testing!  We all really appreciate it.

————————————
Michael Czeiszperger
Greensboro, NC
@.***
On Feb 11, 2024 at 9:27 PM -0500, eightycc @.***>, wrote:

@romkey and @dahlbert I can confirm that 14.4 beta 2 fixes this issue. Tracing shows meta-data updates now follow a data write within a few hundred milliseconds. Like this:
flash_write_blocks: time(ms)=406324, dev_block_num= 42, num_blocks= 1, data
block 41 (data) : cluster 7.0
0000: 9...

manic glacierBOT
#
henrygab
[adafruit/circuitpython] New comment on issue #8903: Broken link to UF2 binary on CircuitPython.org

I appreciate the spoiler-avoidance. I'm not sure what was not public. A simple web search for the next numerical version gives multiple pages of information. Also, emails to subscribers already went out, with shipping information.

I will defer to your greater knowledge. Perhaps it was because I said only "reviewing" instead of "reviewing instructions for"?

Either way, thanks for the quick fix!

manic glacierBOT
manic glacierBOT
manic glacierBOT
manic glacierBOT
#
jerryneedell
[adafruit/circuitpython] Issue opened: #8907 Repeated Socket Failures with recent builds

CircuitPython version

This issue first occurs with this build and is present on all later builds.
Adafruit CircuitPython 9.0.0-beta.0-19-g7ba62bb1cb on 2024-02-07; Adafruit Feather ESP32S3 4MB Flash 2MB PSRAM with ESP32S3

which was from this commit
https://github.com/adafruit/circuitpython/commit/7ba62bb1cb6a52eec7d91d4b48ce3063d0502de7

Code/REPL

import binascii
import io
import busio
import os
import ssl
import time
import board
import socketpo...
#
jerryneedell
[adafruit/circuitpython] New comment on issue #8907: Repeated Socket Failures with recent builds

I also tried using a feather_esp32s2 and it fails the same way with the build reported above, but it also fails with memory errors for the previous build...


10.0.0.107 | REPL | 9.0.0-beta.0-19-g7ba62bb1cbreload.
Adafruit CircuitPython 9.0.0-beta.0-19-g7ba62bb1cb on 2024-02-07; Adafruit Feather ESP32S2 with ESP32S2
>>> 
>>> import aio
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "aio.py", line 29, in <module>
  File "adafruit_io/adafruit_io.py"...
manic glacierBOT
#
jerryneedell
[adafruit/circuitpython] New comment on issue #8907: Repeated Socket Failures with recent builds

The feather esp32s3 NoPSRAM behaves the same as the esp32s3 with PSRAM above.


10.0.0.202 | REPL | 9.0.0-beta.0-19-g7ba62bb1cbreload.
Adafruit CircuitPython 9.0.0-beta.0-19-g7ba62bb1cb on 2024-02-07; Adafruit Feather ESP32S3 No PSRAM with ESP32S3
>>> import aio
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "aio.py", line 29, in <module>
  File "adafruit_io/adafruit_io.py", line 748, in get_feed
  File "adafruit_io/adafruit_io.py", line 578, in...
manic glacierBOT
#
agustinmista
[adafruit/circuitpython] New comment on issue #7322: audio glitches when I2C or SPI display is updated

Hey,

I'm experiencing the same issue as @aalary: a big tradeoff between glitches, sampling rate and latency.

My current setup is:

  • CircuitPython: Adafruit CircuitPython 8.2.9 on 2023-12-06; Raspberry Pi Pico with rp2040
  • Board: Raspberry Pi PICO
  • RP2040 frequency: 240_000_000
  • I2C frequency: 1_000_000
  • Synthio sampling rate: 44_100
  • Audiomixer buffer size: 4096

This seems to work fine so far, although I would like to reduce the buffer size a bit further to avoid latenc...

manic glacierBOT
#
justmobilize
[adafruit/circuitpython] New comment on issue #8907: Repeated Socket Failures with recent builds

Tweaked test code:

import binascii
import io
import busio
import os
import ssl
import time
import board
import socketpool
import wifi
import adafruit_requests
from adafruit_io.adafruit_io import IO_HTTP, AdafruitIO_RequestError

ssid = os.getenv("CIRCUITPY_WIFI_SSID")
password = os.getenv("CIRCUITPY_WIFI_PASSWORD")
aio_username = os.getenv('AIO_USERNAME')
aio_key = os.getenv('AIO_KEY')

print("Connecting to WiFi...")
while not wifi.radio.connected:
  wifi.radio.co...
midnight ember
#

@short tendon In this weeks notes did you mean to misspell ConnectionManager or is it actually called ConnetionManager?

manic glacierBOT
short tendon
manic glacierBOT
#
RetiredWizard
[adafruit/circuitpython] New comment on pull request #8906: Pimoroni DV Base: Fix i2s audio pin assignments

It's odd that there are pre-commit changes in files you didn't modify. Someone will need to determine why these changes are being made by pre-commit now.

If it helps, I cloned the repo against main instead of 8.2.x and before making any changes ran pre-commit and also got the same 70 formatting changes.

This is the first time I'm building after re-installing my OS and remaking the build environment, but I don't seem to be having any issues creating working UF2 images.

manic glacierBOT
tulip sleet
proven garnet
#

Hey @tulip sleet, following up on that circuitpython_typing mpy bundle issue - unless I misunderstand, can't we just add an ignore-bundles arg for the unified action? I don't think the underlying circuitpython-build-tools needs to be updated since we can just use the --ignore option for the build bundles script like the GH release action does.

#

I commented on the issue, but it sounds like that's what you were saying, but the change should be done to workflows-circuitpython-libs then, and not the build tools I think.

tulip sleet
#

and right now we are not using ignore on the bundle part

proven garnet
#

Sweet I can patch that, thanks!

#

Oh, apparently we're up to Node 20 now for actions too given the deprecation notice, neato.

tulip sleet
proven garnet
#

I think I added a script to adabot for checking build statuses, I think it was useful in finding libraries like that since I could find actions that failed.

#

I can recheck that

manic glacierBOT
#
jerryneedell
[adafruit/circuitpython] New comment on pull request #8901: add OV2640 support to sdkconfig for esp32s3_eye build

for example, I just built the esp32s3_metro and in the generated sdkconfig it has

# CONFIG_OV7670_SUPPORT is not set
CONFIG_OV7725_SUPPORT=y
# CONFIG_NT99141_SUPPORT is not set
# CONFIG_OV2640_SUPPORT is not set
CONFIG_OV3660_SUPPORT=y
CONFIG_OV5640_SUPPORT=y

but the sdkconfig in boards/adafruit_metro_esp32s3 does not have ny camera specified. It would probably be good to get bot the OV5640 and OV2640 here as well

#
bradanlane
[adafruit/circuitpython] Issue opened: #8908 Add error handling for misspelled kwargs in adafruit_display_text.label

The parameters for the adafruit_display_text.label are anchor_point and anchored_position. This is confusing and results in frequent user errors. The error is not caught by Python which is difficult for a user to debug.

I don't know if there is a generalized solution for catching this type of error.

A solution for this specific case would be to add anchor_position as a property and raise an error on it.

#
jerryneedell
[adafruit/circuitpython] New comment on pull request #8901: add OV2640 support to sdkconfig for esp32s3_eye build

That probably makes sense. It is confusing. Where does OV5640 get enabled? It seems like it is globally enabled but I don't see it in there.

It is on by default in the kconfig: https://github.com/espressif/esp32-camera/blob/master/Kconfig#L40

With 9.x we only include non-default settings in our kconfigs.

but in line 26 https://github.com/espressif/esp32-camera/blob/master/Kconfig#L26 OV2640 is also enabled, but we don't get it without specifying it...

solar whale
#

Do you want to talk about the "socket retry" issue with esp32s3 "in the weeds" today?

slender iron
solar whale
#

It has not been a need for anyone elso so I can build my own if I want it...camera support, that is

austere acorn
#

just lurking, all good 🙂

slender iron
#

☝️ Discussion about kconfig "# FOO is not set" meaning something

slender iron
lone axle
midnight ember
#

There's been a big uptick in support issues concerning Sonoma. Nice to hear this might be fixed finally.

lone axle
midnight ember
#

Always nice to see new authors contributing. Nice to see Kattni's name pop up again too. 🤗

#

Had to look up US100. It's a USB to Serial converter library.

short tendon
#

would pypi also be automated tests?

midnight ember
#

Are there stats for board downloads from circuitpython.org? That would be interesting to hear about weekly.

lone axle
midnight ember
#

@solar whale 🤗

proven garnet
midnight ember
#

Always enjoy hearing about displayio land stuff you're working on.

#

🐱

#

Good to see everyone excited about the ConnectionManager changes. Great stuff Justin.

#

64-bit TFT?

gilded cradle
#

@midnight ember no, TFT on 64-bit Linux

crimson ferry
#

re: ESP32-S3 boards with 4MB flash ...this will become more acute over time ...how many people use OTA? ...consider for CP 10 to remove OTA by default, and expand both firmware and user filesystem?

short tendon
#

Yes, the S2 bug looks fixed in the latest, but not in the released beta

slender iron
short tendon
#

I have a big stack of boards if I can test anything

midnight ember
#

I'd be interested in seeing you work more with iMX in a deep dive.

slender iron
midnight ember
#

Would work still be progressing for BLE with ESP32-S3 with 16 or 8mb flash?

short tendon
#

can we build the extra ones only on official release?

midnight ember
#

It'll be a caveat for 4mb or less boards. I don't think it's a big deal as long as someone knows they might be able to do it with 16 or 8mb esp32-s3 metro or feathers.

lone axle
#

I didn't know OTA updates were a possibility.

midnight ember
#

wait this is just for OTA?

#

yeah same didn't even know about it.

solar whale
#

Is there a guide for using OTA?

wraith crow
#

I think going forward flash is only going to get bigger so I wouldn't cut things to support the smaller boards

solar whale
#

I think once it gets used, it will be liked.

midnight ember
#

I think getting SD Card support will get more people interested in web workflow too.

#

and now that's pretty close..

crimson ferry
#

good news is 4MB on S3 should be limited to shipped stock... Mini and Wroom-2 all have 8 or more (I think)

midnight ember
crimson ferry
#

oh

midnight ember
#

Right tool for the right job. Right board for the right project. Anyone using a 4mb ESP32-S3 can't reasonably expect it to run everything an 8mb version could.

short tendon
#

I always grab from the the learn guide or github to play with the py files

lone axle
#

plus1 from me at least for this idea of including .py files. I often find myself wanting to edit libraries for troubleshooting so being able to grab the complete set for a project will save me some steps.

midnight ember
#

Can it be moved to an amazon S3 bucket like nightly builds of UF2's?

#

It can be helpful to also offer the PY's, alpha builds for example required the PY files for a little bit there and would throw an error if you used the MPY.

#

I misunderstood the issue. :/

lone axle
#

Thanks for hosting Dan. Hope everyone has a great week!

gilded cradle
#

Thanks everyone

ember iris
#

Thanks everyone, have a great week all!

mortal kernel
#

Thank you all, good meeting.

ember iris
#

If nothing else, on the BLE enable/disable choice--if disabling it reduces the surface area of bugs/complexity when it comes to getting more features stable, then for that reason alone it'd be worth disabling for a while just to make sure it's easy to know what works, and why something else doesn't work while trying to sort out a lot of different problems at once

manic glacierBOT
#
justmobilize
[adafruit/circuitpython] New comment on issue #8907: Repeated Socket Failures with recent builds

Extra info, added some logging. This is on a FeatherS3 - ESP32-S3 Development Board by Unexpected Maker

Adafruit CircuitPython 9.0.0-beta.0-24-g52c3cf6448 on 2024-02-09; FeatherS3 with ESP32S3
>>> 
soft reboot

Auto-reload is on. Simply save files over USB to run them or enter REPL to disable.
code.py output:
Connecting to WiFi...
https://api.github.com
_get_socket: start
_get_socket: try: 1
_get_socket: set timeout: 300
_get_socket: socket.connect - OSError: Failed SSL h...
manic glacierBOT
#
jerryneedell
[adafruit/circuitpython] New comment on issue #8907: Repeated Socket Failures with recent builds

SO this is getting strange. The example I have been using to test access to AIO has been failing with the Repeated Socket failures reliable witht the latest builds BUT I just found taht if I change the name of the AIO feed I am trying to access or create, the it works fine:

Adafruit CircuitPython 9.0.0-beta.0-22-g58b1ce46f1 on 2024-02-09; Adafruit Metro ESP32S3 with ESP32S3
>>> 
>>> import aio
feed exists
got feeds
>>>

here is the code

 import binascii
import io
im...
slender iron
#

@solar whale does it break with other long feed names?

#

maybe it is a length issue

#

@tulip sleet are you looking at jerry's issue? I'm going to

solar whale
slender iron
#

maybe some ssl negotiation or internal memory fragmentation

solar whale
#

IT seems to work OK for the simple test but more complex code --using the camera -- fails -- so may be memory related. Still trying to understand the feed name differences.

#

ooh -- I have an ide -- it may depend on if the feed has history enabled.... For the camera tests it is disabled.... this may be an AIO API issue.

slender iron
#

it is totally possible that the camera is using internal memory

solar whale
#

nope -- not due to history being disabled....

slender iron
#

the last test you posted works ok for me

solar whale
#

And it works fine for me. Try changing the feed anem to esp32s3eye

#

ha -- if I delete the feed esp32s3eye and then run my test, it works ok. I'll update the issue.

tulip sleet
short tendon
#

@slender iron There is still some sort of issue. I've tried a bunch of s3's and none of them can connect to https://api.github.com

slender iron
manic glacierBOT
tulip sleet
#

if there was an accidental reversion of the certificates submodule, that might have happened. But doesn't seem likely

short tendon
#

on the ticket, but:

import os
import ssl
import socketpool
import wifi
import adafruit_requests

ssid = os.getenv("CIRCUITPY_WIFI_SSID")
password = os.getenv("CIRCUITPY_WIFI_PASSWORD")

print("Connecting to WiFi...")
while not wifi.radio.connected:
  wifi.radio.connect(ssid, password)

pool = socketpool.SocketPool(wifi.radio)
ssl_context = ssl.create_default_context()

requests = adafruit_requests.Session(pool, ssl_context)

url = "https://api.github.com"
print(url)
r = requests.get(url)
manic glacierBOT
solar whale
#

@slender iron changing the order in which I do things with AIO and the camera also seems to have a big impact -- that sounds like memory to me.

tulip sleet
#

Here is the notes document for next Tuesday's CircuitPython Weekly Meeting. The meeting is delayed one day due to a US Monday holiday. It is at the normal time of 11am Pacific / 2pm US Eastern here on Discord. Add your hug reports and status updates to the document before the meeting. If you are unable to attend but would still like to contribute, feel free to add your notes and we’ll read them off during the meeting. Hope to see you there! <@&356864093652516868>
https://docs.google.com/document/d/1x3O7daDGGkd0NgEyHz0p9OKz8tCtKB4gli7iCisYtmI/edit?usp=sharing

tulip sleet
slender iron
solar whale
#

latest

slender iron
#

looks at how camera allocates

solar whale
#

I have to go offline -- I'll get back into this tomorrow.... fun stuff 😉

slender iron
solar whale
#

@slender iron just to clarify— the camera alone is working fine on the esp32s3_eye. I only have issues if I also try to access AIO.

#

I’m trying to send the image to AIO.

#

This is the code I am trying to use for camera and AIO.

📎 code.py
manic glacierBOT
#
RetiredWizard
[adafruit/circuitpython] New comment on pull request #8906: Pimoroni DV Base: Fix i2s audio pin assignments

The DOCS build check failed :frowning:

The log warnings I spotted:

circuitpython-stubs/memorymap/__init__.pyi
[WARN] Missing argument type: start on line 26
[WARN] Missing argument type: length on line 26

circuitpython-stubs/synthio/__init__.pyi
[WARN] Missing return type: __init__ on line 129
[WARN] Missing return type: retrigger on line 166
[WARN] Missing return type: __init__ on line 234
[WARN] Missing argument type: once on line 137
[WARN] Missing argument type...
manic glacierBOT
#
rgov
[adafruit/circuitpython] New comment on issue #8902: Support @micropython.native on RP2040

Maybe we should turn it off for now.

I understand that advanced features like native code support is not really aligned with CircuitPython's goals, but I am hoping to compile C modules to loadable .mpy extensions without maintaining a fork of the CircuitPython. This is a really nice feature—as long as you don't need to access the MCU's SDK. I don't actually want to use the decorator (and indeed it probably is a bit of a foot-gun for people who apply it too prodigiously because fast.)

onyx hinge
#

@lone axle so excited about your new guide

lone axle
# onyx hinge <@382939733107408897> so excited about your new guide

Thanks for sharing that editor initially. That ended up "just working" with the keyboard more than I had imagined it would. It's great to have a "bite sized" thing like this that builds from the same concepts as more complex terminal GUIs. I'm only part way thru the original tutorial but it's opened my eyes to a lot about why some of the utilities I use work the way they do. I hope to finish out that tutorial and attempt a small CPython project of my own to apply the knowledge to.

manic glacierBOT
manic glacierBOT
manic glacierBOT
#
RetiredWizard
[adafruit/circuitpython] New comment on issue #8690: Web workflow interrupts running code.py and hangs after uploading a file

I tested using a Feather ESP32S3 4/2 and the artifact from #8909. I was not able to cause any web workflow page hangs, however the code.py file did stop running for 1-2 minutes when the initial web workflow page was displayed. Accessing the file browser screen would start the code.py program for a few seconds and it would then stop again. Performing file I/O (uploading, renaming, deleting files) would start the code.py program running until I returned (using the browser back button) to the we...

#
RetiredWizard
[adafruit/circuitpython] New comment on issue #8690: Web workflow interrupts running code.py and hangs after uploading a file

When the test code was actually placed in a code.py file accessing the web workflow page didn't interrupt it's execution until a file was uploaded via web workflow which caused an auto-reload. After the auto-reload, the code.py program started running again but then if the web workflow welcome page was accessed the code.py program would hang and the behavior described in my prior post was observed.

manic glacierBOT
manic glacierBOT
#
jepler
[adafruit/circuitpython] New review comment on pull request #8900: Fix bugs exposed by posix port

Yup, this is to match cpython's behavior:

>>> random.getrandbits(-1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: number of bits must be non-negative
>>> random.getrandbits(0)
0
>>> random.getrandbits(31)
496699869

I think the "< 0" test matches standard python better than "== 0" or "<= 0". This case is specifically tested in the micropython test suite though we did not run it on our shared-bindings implementation until my work on t...

manic glacierBOT
manic glacierBOT
#
jerryneedell
[adafruit/circuitpython] New comment on issue #8907: Repeated Socket Failures with recent builds

After further testing, it appears that the "repeated socket failures" errors are misleading and not reproducible with a simple test case. I am able to reliably send data to AIO from the esp32s3_eye and metro_esp32s3. The issues arise when attempting to send camera images to AIO and this need further review. I am closing this issue and will reopen an issue regarding sending images to AIO when I have a better example.

manic glacierBOT
#
jerryneedell
[adafruit/circuitpython] New comment on issue #8907: Repeated Socket Failures with recent builds

I am re-opening this since I have found that there is definitely an issue starting with build https://github.com/adafruit/circuitpython/commit/7ba62bb1cb6a52eec7d91d4b48ce3063d0502de7
that is still present

I have been working with the MEMENTo Doorbell guide code https://learn.adafruit.com/diy-iot-doorbell-camera-with-memento/code and it works fine with commit https://github.com/adafruit/circuitpython/commit/09fcc5997f2359761c57bf68aeb89684ad9acf42 but fails with anything newer.

the...

manic glacierBOT
#
justmobilize
[adafruit/circuitpython] Issue opened: #8910 Unable to connect to api.github.com

CircuitPython version

Most recent 9.0.0 build

Code/REPL

import os
import ssl
import time
import board
import socketpool
import wifi
import adafruit_requests

print("------------------------------")
print(os.uname())

ssid = os.getenv("CIRCUITPY_WIFI_SSID")
password = os.getenv("CIRCUITPY_WIFI_PASSWORD")

print("Connecting to WiFi...")
while not wifi.radio.connected:
  wifi.radio.connect(ssid, password)

pool = socketpool.SocketPool(wifi.ra...
solar whale
#

@onyx hinge I am trying to use some camera breakouts (OV5640) on a metro_esp32s3 and I often get timeouts on the cam.take() call. I've tried adjusting the timeout time, but that does not seem to help. I do have the camera on 3 inch jumper wires. I'm just curious if you have run into issues with cam.take() returning None and if you think the wire length could be an issue. The camera is detected. cam.sensor_name returns ov5640. I'm just looking for directions to troubleshoot.

onyx hinge
solar whale
#

jpeg

onyx hinge
# solar whale jpeg

JPEG mode is looking for some specific bit patterns. if those get garbled on the wires then you'll get a None instead of an image.

#

I don't think that with the current code you can directly change the camera data rate

solar whale
#

hmmm -- good to know -- I'll try rgb mode to see if that responds differently.

onyx hinge
#

if you capture RGB images yo're likely to see this same effect appear as noise

#

but I don't think you should get None images

solar whale
#

at least that would point me in a known direction. I was trying to understand the None repsonses.

onyx hinge
#

and of course you could build some esp-idf or arduino software and see if it behaves any differently

solar whale
#

with RGB I get "better" responses, but still many Nones, juts not as many as with JPEG

onyx hinge
#

huh.

#

how does the data look, are you displaying it?

solar whale
#

No - not yet -- still need to get a display connected... I was trying to send them to AIO, but that has opened a whole new can of worms (and issues) . I'll try to improve my wiring and also se if I get tsame issues on the integrated boards -- memento or esp32s3_eye.

#

Thansk for the tips -- welcome back!

onyx hinge
#

thanks

#

I'm relatively sure we don't see the "none" return value frequently on memento, somebody would be reporting that.

manic glacierBOT
#
eightycc
[adafruit/circuitpython] New comment on issue #8449: OSError: [Errno 5] Input/output: macOS Sonoma is delaying writes on small filesystems

Is everyone else seeing that it is slow? I haven't tried it yet.

Yes. Writing a file of all zeroes like this:

rabeles@Bobs-MacBook-Air raspberrypi % time dd if=/dev/zero of=/Volumes/CIRCUITPY/test.dat bs=512 count=16 
16+0 records in
16+0 records out
8192 bytes transferred in 21.585040 secs (380 bytes/sec)
dd if=/dev/zero of=/Volumes/CIRCUITPY/test.dat bs=512 count=16  0.00s user 0.01s system 0% cpu 21.595 total

This works out to ~2.5 seconds per 1K written. Looks like w...

onyx hinge
# 
                            if (frame_buffer_event->len != cam_obj->fb_size) {
                                cam_obj->frames[frame_pos].en = 1;
                                ESP_LOGE(TAG, "FB-SIZE: %u != %u", frame_buffer_event->len, (unsigned) cam_obj->fb_size);
``` in RGB mode If the frame doesn't contain the expected amount of data you will also get a distinctive FB-SIZE message on the debug UART, if using a debug build
#

noise on the H & V sync lines would make a frame be the wrong size

solar whale
#

for normal build, should RGB mode ignore the noise?

onyx hinge
#

no, the noise would affect a normal build

#

but with a debug build, besides getting a None frame you'd get this message on the debug UART

#

if a false VSYNC is seen by the microcontroller (or if one is missed) then the amount of data will be wrong. This would print as an FB-SIZE error in debug mode, and would get you a None frame regardless of debug mode

solar whale
#

OK -- thanks.

onyx hinge
#

similar for if HSYNC/HVALID has noise, either extra pixel data can be recorded or pixel data can be lost

solar whale
#

I have not tried Debug builds -- where is the Debug UART output -- or, better, can you point me to what I need to read to use it.

#

BTW -- no Nones on MEMENTO! That's reasssuring.

onyx hinge
#

you have to make a debug build on your own. There are two pins pre-defined on the esp32-s3 for the debug UART, but I think this is often the standard TX pin

#

and you have to disable things (I usually disable wifi & ulab) to successfully enable debug builds.

#

I haven't done it lately so my memory's not super accurate

solar whale
#

First I'll try my tests on the integrated boards and try to shorten my wires... low hanging fruit!

onyx hinge
solar whale
#

Thanks!

manic glacierBOT
#
tannewt
[adafruit/circuitpython] New comment on issue #8910: Unable to connect to api.github.com

Here is the MBEDTLS log. I'm not sure why it is failing. Maybe outgoing buffer size?

I (265484) mbedtls: ssl_tls.c:3931 => handshake

I (265484) mbedtls: ssl_msg.c:2370 => flush output

I (265484) mbedtls: ssl_msg.c:2379 <= flush output

I (265484) mbedtls: ssl_tls.c:3850 client state: MBEDTLS_SSL_HELLO_REQUEST

I (265494) mbedtls: ssl_msg.c:2370 => flush output

I (265504) mbedtls: ssl_msg.c:2379 <= flush output

I (265504) mbedtls: ssl_tls.c:3850 client state: MBEDTLS_SS...
#
jepler
[adafruit/circuitpython] New comment on issue #8910: Unable to connect to api.github.com

Just a note that "%zu" is intended to print a value of type size_t but this is not supported and so just "zu" is printed.

"I (265634) mbedtls: ssl_msg.c:2312 in_left: zu, nb_want: zu"

This problem almost certainly stems from the hacky way I connect esp-idf's debug printing to use circuitpython's printf implementation:

#if CIRCUITPY_CONSOLE_UART
static int vprintf_adapter(const char *fmt, va_list ap) {
    return mp_vprintf(&mp_plat_print, fmt, ap);
}   

void port_serial_ear...
#
justmobilize
[adafruit/circuitpython] New comment on issue #8910: Unable to connect to api.github.com

I updated the test list to:

urls = [
  "http://wifitest.adafruit.com/testwifi/index.html",
  "http://www.adafruit.com/api/quotes.php",
  "https://api.github.com",
  "https://api.thingspeak.com",
  "https://opensky-network.org",
  "https://api.fitbit.com/",
  "https://discord.com/api",
  "https://id.twitch.tv",
  "https://api.twitter.com/"
  "https://youtube.googleapis.com",
]

And github is the only one that fails

manic glacierBOT
slender iron
#

merge needed help on the cli

wraith crow
#

@slender iron I was going to submit a second pimoroni I2S fix against main, it's running into the same black formatting changes, do you want me to go ahead and submit it, or is something else in the works to update main?

slender iron
wraith crow
#

Cool, I'll wait then 😄

slender iron
#

it needs to be done manually unfortunately

wraith crow
#

Ah, thus the comment just above, thanks!!!

manic glacierBOT
#
tannewt
[adafruit/circuitpython] New comment on issue #8910: Unable to connect to api.github.com

I'm at a loss. It looks like the server is resetting the connection after we send the client key:

I (23072) mbedtls: ssl_msg.c:2379 <= flush output

I (23082) mbedtls: ssl_tls.c:3852 client state: MBEDTLS_SSL_SERVER_HELLO_DONE

I (23082) mbedtls: ssl_tls12_client.c:2686 => parse server hello done

I (23092) mbedtls: ssl_msg.c:4134 => read record

I (23102) mbedtls: ssl_msg.c:4202 reuse previously read message

I (23102) mbedtls: ssl_msg.c:4206 <= read record

I (23112) mbe...
short tendon
manic glacierBOT
manic glacierBOT
#
RetiredWizard
[adafruit/circuitpython] New review comment on pull request #8816: Initial working m5stack cardputer setup

According to the cardputer schematic GP41 is the BCLK (I2S_BIT_CLOCK), GP42 is SDATA (I2S_DATA) and GP43 is LRCLK (I2S_WORD_SELECT). Renamaing these pins accordingly might make sense:

// I2S Audio
{ MP_ROM_QSTR(MP_QSTR_I2S_BIT_CLOCK), MP_ROM_PTR(&pin_GPIO41) },
{ MP_ROM_QSTR(MP_QSTR_IS2_DATA), MP_ROM_PTR(&pin_GPIO42) },
{ MP_ROM_QSTR(MP_QSTR_I2S_WORD_SELECT), MP_ROM_PTR(&pin_GPIO43) },

Although, I believe aduiobusio won't run if the Bit Clock and Word Select pins are ...

manic glacierBOT
manic glacierBOT
mortal kernel
#

@danh I'm iterating over some interesting device/filesystem/size permutations now. I can see so far that USB sticks are affected and that filesystem size is a factor. I'm looking around 8M to see if I can identify where the performance drop is. While I'm under the hood, is there anything specific you'd like checked out?

manic glacierBOT
tulip sleet
#

We keep not being sure if there's an 8MB break-point here. Also besides the slow writes, there are the write errors, which may be a manifestation of the same problem, or something else

mortal kernel
#

Here's another bug for you: I now have to plug USB MSC devices in twice to get them to mount.

tulip sleet
#

If slowness is reproducible on a disk image, then that's easily reportable

#

any USB MSC or just small ones?

mortal kernel
#

Reproducible on a USB stick. The underlying size of the stick is not an issue, I'm using a native 16G stick.

#

The reason I'm looking at 8M is that it is a number we said to them in some of the reports.

tulip sleet
#

I'd think MANY people would see the stick problem, hmm

#

did your instrumentation see a difference >8MB in metadata write delays?

mortal kernel
#

At any rate, I'll binary search my way to the right number. If the number is small then not many people will see the problem.

mortal kernel
tulip sleet
#

thanks for all this testing! I looked at the apple OSS repos, and there are recent changes, but not in the ones I linked to in the issue comments

mortal kernel
#

The OSS repos generally get updated after the release. There's a sanitation process they go through first to prevent trade secrets from leaking.

manic glacierBOT
tulip sleet
#

@slender iron do you think #8913 might fix the api.github.com issue too? I was just starting to work on that.

#

i'll test it after the build

slender iron
#

I gave up on the GitHub issue because it looks like the GitHub server is ending the connection

tulip sleet
#

api.github.com had trouble before with a cert problem but that could be a red herring

#

it uses an elliptical intermediate cert and an RSA root cert

#

anyway, I'll try a few things

#

the root cert is def in the build now

slender iron
#

I posted the log I found. Maybe I disabled the rsa portion

tulip sleet
#

most certs are RSA, so I think more would break

#

but I will check the mbedtls settings too

short tendon
#

Are there other sites that have a similar cert chain we could check to see if we have the same issue?

manic glacierBOT
tulip sleet
short tendon
#

If there's anything I can do to help test, please let me know.

#

Do you have a test suite you run on certain boards before a release?

onyx hinge
#

https://badssl.com/ has a suite of variously configured https servers, many of which are "should fail" but you can also connect to some that only work with specific cryptographic elements (e.g., ecc256 and nothing else)

short tendon
#

Oh that's pretty cool. Going to add an example test in ConnectionManager for that

tulip sleet
#

badssl did not have a test for the particular issue we had with Let's Encrypt certs, which have a slightly complicated failover scheme to handle a certain cert

short tendon
#

I'll for sure add others to the test

#

It'll be nice, since all that needs to happen with CM is the socket connect

#

Also random question. Is there a goal for CP9 general release?

manic glacierBOT
tulip sleet
short tendon
manic glacierBOT
tulip sleet
#

I think this missing prototypes problem would be worth filing as an issue (and maybe a PR) in https://github.com/bluekitchen/btstack. I looked at some recent issues and some were closed recently to fix new compiler warnings, for instance.

manic glacierBOT
#
eightycc
[adafruit/circuitpython] New comment on issue #8449: OSError: [Errno 5] Input/output: macOS Sonoma is delaying writes on small filesystems

It is possible to reproduce poor write performance using a USB stick. All of these test were performed with a 16G stick. Test writes 512 blocks of 512 bytes of zeroes.

dd if=/dev/zero of=/Volumes/NONAME/test.dat bs=512 count=512

Performance jumps ~400x when the filesystem size exceeds 1G.

Filesystem Size Filesystem Time(Sec.)
4M FAT12 26.981
8M FAT12 28.780
15M FAT16 29.019
16M FAT16 29.139
32M FAT16 29.669
64M FAT16 29.414
128M FAT16 50.138
mortal kernel
manic glacierBOT
#
eightycc
[adafruit/circuitpython] New comment on issue #8449: OSError: [Errno 5] Input/output: macOS Sonoma is delaying writes on small filesystems

For convenience I made the test sticks on Linux. Should be possible to do the same with Disk Utilities on MacOS.

Partitioned with fdisk creating a primary partition of the desired size, for example:

rabeles@ub2004:~$ sudo fdisk /dev/sdd

Welcome to fdisk (util-linux 2.37.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Command (m for help): d
Selected partition 1
Partition 1 has been deleted.

Command ...
mortal kernel
#

@danh Since the Sonoma behavior changes at the 1G line, it won't be so easy to obtain trace data. To do it, I'll need to fake a 1G+ filesystem. If you think it's worth the time, let me know and I'll do it.

tulip sleet
mortal kernel
#

BTW, I think the 1G line was chosen so that the majority of USB storage devices would not be affected by the "fix".

tulip sleet
#

so maybe the >1GB have long delays, hmm, could be.

mortal kernel
#

Just so. Also, it might still be possible to get correct behavior with the re-mount workaround. I have not tried it.

manic glacierBOT
manic glacierBOT
manic glacierBOT
manic glacierBOT
manic glacierBOT
#
robertlipe
[adafruit/circuitpython] New comment on issue #8449: OSError: [Errno 5] Input/output: macOS Sonoma is delaying writes on small filesystems

Seems unlikely since this performance regression was introduced in an os
beta that hasnt been released and is in DOS-era FSes, and mostly in small
ones at that.

The bugs in this issue are exclusively in removable DOS filesystems.

On Tue, Feb 13, 2024, 11:23 PM Luke Hamburg @.***>
wrote:

Sorry if this is wildly off topic, but I'm fighting a demon of a bug over
at syncthing/syncthing#9371
https://github.com/syncthing/syncthing/issues/9371 and just wondering
if thi...

stuck elbow
#

are you going to keep spamming all the channels?

vital matrix
midnight ember
# 
try:
    import alarm
except ImportError:
    print ("This board doesn't support the alarm module")
    pass
try:
    import audiobusio
except ImportError:
    print ("This board doesn't support the audiobusio module")
    pass
``` etc...
vital matrix
#

So, I was trying to figure out more if there's some intent to be able to run everything in that directory via a script (i.e., like how run-tests.py works)

manic glacierBOT
#
FoamyGuy
[adafruit/circuitpython] Pull request opened: #8916 Blit swap bytes

This adds a new argument swap_bytes to bitmaptools.blit() which lets the caller indicate they want to perform a byte-swap on a 565 color value.

This allows you to blit from a source rgb888 Bitmap that came from a .bmp file (which will be using a ColorConverter to get into 565 colorspace) into a destination Bitmap that was populated by JpegDecoder from a .jpg file (which uses 565_Swapped colorspace) and have the resulting combined image look correct.

More generally it means if you h...

midnight ember
#

@lone axle This sounds confusing blit from a source rgb888 Bitmap that came from a .bmp file (which will be using a ColorConverter to get into 565 colorspace) into a destination Bitmap that was populated by JpegDecoder from a .jpg file It says from twice, is the source a .bmp or .jpg? Here it sounds like the source is both due to the word from used twice.

lone axle
#

There are two image files. One .jpg and one .bmp. The data from the .bmp is acting as the Source for the blit, and the data from the .jpeg is acting as the destination for the blit.

lone axle
# vital matrix So, I was trying to figure out more if there's some intent to be able to run eve...

I don't really have any specific prior knowledge of those tests, but it looks to me like they are intended to be run more manually.

Theoretically you could add a layer on top that uses exec() to run through multiple of them, or perhaps if you added __init__.py in them then you could use import in a higher level script to launch multiples. It might be tricky though, some of them look to have boot.py components too. I haven't looked through them but it's possible there could be different tests that require different configurations in boot.py in which case it would not be possible to easily go from one to the other automatically.

midnight ember
#

If it's a type of 2 file merge then I would recommend using merge somewhere in the description. A color conversion suggests one image.

lone axle
midnight ember
#

I think of blit as a brute force bit stream like with a matrix panel. Perhaps my understanding of the word blit is incorrect? I've never thought of blit as a merge. Interesting.

lone axle
#

I do think it's a complex thing to discuss though also because blit uses concepts of source and destination Bitmaps. But in the case I'm working from one of the Bitmap objects came from a Jpeg. And also I think of both of those image files as "sources" for their respective Bitmap objects, even though only one of those Bitmaps is a "Source" for the purpose of the blit and the other is a "Destination"

#

I would not count on my own knowledge about it's meaning. Tbh within bitmaptools and I think maybe pygame are the only places I've run into the term blit and I don't think I've ever looked up a definition of the word outside of the contexts of those two.

midnight ember
#

Sounds good, you'd know far better than me. I take it this is camera image conversion stuff? I haven't gotten into the memento yet.

short tendon
#

https://en.wikipedia.org/wiki/Bit_blit give a good definition...

Bit blit

Bit blit (also written BITBLT, BIT BLT, BitBLT, Bit BLT, Bit Blt etc., which stands for bit block transfer) is a data operation commonly used in computer graphics in which several bitmaps are combined into one using a boolean function.The operation involves at least two bitmaps: a "source" (or "foreground") and a "destination" (or "background"),...

midnight ember
#

Thank you. My understanding of the term was definitely incorrect.

short tendon
#

It's also one of those terms that get's misused. I needed to skim most of the artical to make sure it was mostly right...

manic glacierBOT
#
jepler
[adafruit/circuitpython] New comment on pull request #8916: Blit swap bytes

You mention rgb888, but the code you've written doesn't seem to convert from rgb888 to rgb565, whether swapped or not. It does convert color numbers of the form 0xPPQQRRSS to 0xPPQQSSRR, which would change between rgb565 and rgb565_swapped (in which case P and Q are both 0). If the value is actually an rgb888 value, then it would convert it to an rbg888 value.

Perhaps instead of the explicit bit operations you can just use __builtin_bswap16, which we use elsewhere. This can use the ARM i...

mortal kernel
#

@danh I like @romkey 's idea for splitting off the new Sonoma performance bug as a new issue. What do you think?

manic glacierBOT
tulip sleet
mortal kernel
tulip sleet
#

I was thinking about their Feedback.

#

I will check my own Feedback to see if there's any response

mortal kernel
#

Way back when I asked folks to reference my feedback in their reports.

tulip sleet
#

The current issue is so long it's hard to find anything. Would you be willing to start a new one? A mostly copy/paste from your data table post would be a fine start, maybe with a paragraph from the previous post touched up a bit

#

I looked and my Feedback is still Open

mortal kernel
#

@danh I'm ready to open a new CP issue.

tulip sleet
#

I was already thinking of a writeup of how to do some instrumentation on the new issue:

  1. your CircuitPython instrumentation
  2. WIreshark monitoring a USB stick
  3. Beagle traces. I have a Beagle, but not with the fancier software. I think we'd have to write/find some trace analysis software for MSC packets
#
  1. macOS verbose logging. I have a vague memory we could monitor all the writes, but I may be misremembering
#

But we can start with Feedbacks that just point out the obvious problem as noted in your elapsed time table

#

my mac mini is updating to beta 3 right now

mortal kernel
#

Personally, I'm trying to balance this issue with my work on BLE. I'd love to see this issue resolved, but it is costing me quite a bit of time that might be better spent elsewhere on CP. On the other hand, instrumentation and tracing with a Beagle are very interesting to me.

manic glacierBOT
tulip sleet
#

I assume you don't have a Beagle. I have not used Wireshark for USB on macOS but hope it is possible.

mortal kernel
#

No Beagle here, just rabbits. To collect the raw data for Wireshark we need a hardware device like the Beagle. Internal MacOS USB tracing looks like it died several releases ago. Building from what I already have, a modified CP faking a >1G filesystem could be the path of least resistance.

slender iron
slender iron
manic glacierBOT
tulip sleet
#

@slender iron @onyx hinge do you think I should make an 8.2.10 today or tomorrow? There are about 10 changes. Shouldn't take long.

onyx hinge
#

sounds good to me!

manic glacierBOT
slender iron
tulip sleet
#

Release in process.

manic glacierBOT
#
dhalbert
[adafruit/circuitpython] New tag created: 8.2.10
orchid basinBOT
manic glacierBOT
vital matrix
orchid basinBOT
short tendon
#

@slender iron and @onyx hinge, if you are currious, on both 8.2.9 and the most recent 9.0.0 I ran through the baddsll endpoints and got this:

Testing: Certificate Validation (High Risk)
 expired:443 - passed
 wrong.host:443 - passed
 self-signed:443 - passed
 untrusted-root:443 - passed

Testing: Interception Certificates (High Risk)
 superfish:443 - passed
 edellroot:443 - passed
 dsdtestprovider:443 - passed
 preact-cli:443 - passed
 webpack-dev-server:443 - passed

Testing: Broken Cryptography (Medium Risk)
 rc4:443 - passed
 rc4-md5:443 - passed
 dh480:443 - passed
 dh512:443 - passed
 dh1024:443 - passed
 null:443 - passed

Testing: Legacy Cryptography (Moderate Risk)
 tls-v1-0:1010 - passed
 tls-v1-1:1011 - passed
 cbc:443 - error - success:maybe | fail:yes | exc:None
 3des:443 - passed
 dh2048:443 - passed

Testing: Domain Security Policies
 revoked:443 - error - success:maybe | fail:yes | exc:None
 pinning-test:443 - error - success:maybe | fail:yes | exc:None
 no-sct:443 - passed

Testing: Secure (Uncommon)
 1000-sans:443 - error - success:yes | fail:maybe | exc:Error connecting socket: Failed SSL handshake
 10000-sans:443 - error - success:yes | fail:maybe | exc:Error connecting socket: Failed SSL handshake
 sha384:443 - passed
 sha512:443 - passed
 rsa8192:443 - error - success:yes | fail:maybe | exc:Error connecting socket: Failed SSL handshake
 no-subject:443 - error - success:yes | fail:maybe | exc:Error connecting socket: Failed SSL handshake
 no-common-name:443 - error - success:yes | fail:maybe | exc:Error connecting socket: Failed SSL handshake
 incomplete-chain:443 - passed

Testing: Secure (Common)
 tls-v1-2:1012 - passed
 sha256:443 - passed
 rsa2048:443 - passed
 ecc256:443 - passed
 ecc384:443 - passed
 extended-validation:443 - passed
 mozilla-modern:443 - passed

All the unmatched ones are maybes for them.

tulip sleet
short tendon
#

Nope, at least not for those...

orchid basinBOT
manic glacierBOT
#
eightycc
[adafruit/circuitpython] Issue opened: #8918 Input/Output: MacOS Sonoma 14.4b3 CIRCUITPY Slow Write Performance

CircuitPython version

All CP version affected.

Code/REPL

n/a

Behavior

Writes to the CIRCUITPY filesystem when mounted by USB MSC on MacOS Sonoma 14.4 beta 2 and above are 400 to 800 times slower than with previous releases of MacOS.

Description

Previously, MacOS Sonoma 14.0 introduced a regression in its MSDOS filesystem support that caused a number of issues that are documented in issue #8449. The root cause of the issue was in Sonoma...

tulip sleet
manic glacierBOT
short tendon
#

@tulip sleet, something that might be worth looking at.
Below is a timing of connecting to the host (just a socket connect, no data fetched) between 8.2 and 9.0.
9.0 is about 2x slower in connecting...

8.2.9

 - Common hosts
   - api.fitbit.com:443 took 0.69 seconds | passed
   - api.github.com:443 took 1.43 seconds | passed
   - api.thingspeak.com:443 took 1.20 seconds | passed
   - api.twitter.com:443 took 0.67 seconds | passed
   - discord.com:443 took 3.75 seconds | passed
   - id.twitch.tv:443 took 0.55 seconds | passed
   - oauth2.googleapis.com:443 took 0.87 seconds | passed
   - opensky-network.org:443 took 1.64 seconds | passed
   - www.adafruit.com:443 took 0.63 seconds | passed
   - www.googleapis.com:443 took 1.17 seconds | passed
   - youtube.googleapis.com:443 took 0.90 seconds | passed

 - Known problem hosts
   - valid-isrgrootx2.letsencrypt.org:443 took 1.94 seconds | passed

Current 9.0.0 build

 - Common hosts
   - api.fitbit.com:443 took 1.66 seconds | passed
   - api.github.com:443 took 11.11 seconds | error - success:yes, fail:no, exc:Error connecting socket: Failed SSL handshake
   - api.thingspeak.com:443 took 1.17 seconds | passed
   - api.twitter.com:443 took 0.72 seconds | passed
   - discord.com:443 took 7.47 seconds | passed
   - id.twitch.tv:443 took 3.10 seconds | passed
   - oauth2.googleapis.com:443 took 3.20 seconds | passed
   - opensky-network.org:443 took 1.69 seconds | passed
   - www.adafruit.com:443 took 3.01 seconds | passed
   - www.googleapis.com:443 took 3.19 seconds | passed
   - youtube.googleapis.com:443 took 3.22 seconds | passed

 - Known problem hosts
   - valid-isrgrootx2.letsencrypt.org:443 took 18.00 seconds | passed
tulip sleet
#

which board are you using? (interested in how much PSRAM, etc.)

short tendon
#

UM FeatherS3

tulip sleet
#

Could you go ahead and open an issue about this: "Slower https access on 9.0.0", or something like that? Thanks.

short tendon
#

Happy to. Will also add the test code in a comment (note: it does use ConnectionManager to make it easy to get the socket...)

#

lunch first

solar whale
#

The current build for the ESP32S2-Kaluga_1.3 does not have anything in the sdkconfig and as built supports the OV5640 but not the OV2460. If I wanted to enable OV2640, do I just have to add it via the sdkconfig like we did for the esp32s3-eye or is there something different about the esp32s2 -- If I do populate the sdkcongig file for an esp32s2 do iI have to add a line like this ```# LWIP

CONFIG_LWIP_LOCAL_HOSTNAME="espressif-esp32s3"

end of LWIP

tulip sleet
solar whale
#

Thanks, I'll try adding the OV2640 support.

#

Just stumbled across an new issue with cameras. With an OV5640 on the Kaluga I am getting an "espidf.IDFError: Generic Failure" sometimes when initializing the camera. I'm working on a repeatable trigger, but as with other cases, it seems to depend on what else is being done... Does the "Generic Failure" suggest it is memory related or anything else... This is on the ESP32S2 Kaluga. I have not seen this on an esp32s3...yet.

#

This is on the lastest build

tulip sleet
#

@onyx hinge I am not familiar with the camera errors at all. Does this ring a bell? ^^

solar whale
#

When I first ran into it, it was only for the RGB Pixel format, not for JPEG, but now I am getting it for JPEG as well but I have added accessing AIO to the code so lots more going on.

onyx hinge
#

it means something returned ESP_FAIL. The source code has to be investigated to find out what those things are.

#

there are 20 places in esp-camera that explicitly return ESP_FAIL

solar whale
#

OK -- I'll look into it. Thanks

onyx hinge
#

as before, a debug build may show additional information

onyx hinge
# solar whale OK -- I'll look into it. Thanks

a couple more notes:

  • JTAG pins are shared with the camera; enabling JTAG hoses the camera. Enabling DEBUG may automatically enable JTAG
  • the camera may not have a reset connection, if not make sure to power cycle (not just reset button or software reset) after anything goes wrong
  • there may be some other peripheral or GPIO that is shared
  • try without wifi and see if it makes a difference
solar whale
#

Thanks for the tips, Jeff. The Kaluga has it's share of quirks. I am only using it now because it has the nice camera conector. BTW -- I'm quite convinced that the issue I was having with the failed captures yesterday was due to the long wires. ( ~3 inches) That prompted me to dig out the kaluga but it is present some new challenges....

manic glacierBOT
#
justmobilize
[adafruit/circuitpython] Issue opened: #8919 Slower https access on 9.0.0

CircuitPython version

Adafruit CircuitPython 9.0.0-beta.0-32-g2e29772eb2 on 2024-02-14; FeatherS3 with ESP32S3
From:
2024-02-14T19:32:55.000Z
adafruit-circuitpython-unexpectedmaker_feathers3-en_US-20240214-no-branch-2e29772.uf2

Code/REPL

# SPDX-FileCopyrightText: Copyright (c) 2024 Justin Myers
#
# SPDX-License-Identifier: Unlicense

import time
import wifi

import adafruit_connection_manager


adafruit_groups = [
    {
        "heading": "Co...
slender iron
tulip sleet
onyx hinge
#

fine by me

solar whale
manic glacierBOT
manic glacierBOT
manic glacierBOT
manic glacierBOT
#
tannewt
[adafruit/circuitpython] Pull request opened: #8920 Improve neopixel on ESP

Greedily grab as much RMT memory as we can. It blocks other RMT transmit channels but we only use it temporarily anyway. The more we can grab, the fewer interrupts are needed to keep the transmit going.

Flickers may still happen due to file system writes but most of the time the animation just pauses.

On ESP32, move CircuitPython to the second core. This helps NeoPixel by moving the RMT interrupt to the second core as well.

When testing ESP32 I noticed that settings.toml writes won't...

manic glacierBOT
#
bill88t
[adafruit/circuitpython] New comment on issue #8443: idf v5.1 `listen`/`accept` race condition ends up losing clients

@tannewt Oh I'm still experiencing this issue on the daily since I rely on ftp for non-usb boards.

It has become rarer for some reason.
Not sure when, since I've not been tracking master all that well (usually my main board's version is a week old).

My workarounds still are inadequate, and only work 30% of the time.
Still it should be noted, not a single of these errors had occurred on 8.x.

manic glacierBOT
manic glacierBOT
manic glacierBOT
manic glacierBOT
#
dhalbert
[adafruit/circuitpython] Pull request opened: #8924 esp-idf-config/sdkconfig.defaults: Fix TLS access to api.github.com
  • Fixes #8910

Both of these were not set (and therefore not turned on), and it caused TLS access to api.github.com to not work at all.

# CONFIG_MBEDTLS_ECP_NIST_OPTIM is not set
# CONFIG_MBEDTLS_ECP_FIXED_POINT_OPTIM is not set

Turning on either one fixes the problem. Turning on either CONFIG_MBEDTLS_ECP_NIST_OPTIM is smaller (+2.2kB vs +6.9kB) and faster (1.6 seconds vs 7.6 seconds, at least some of the time).

I don't understand why some "OPTIM" must be turned on for i...

tulip sleet
#

@onyx hinge this is in mpconfigport.mk on espressif:

CIRCUITPY_GIFIO ?= $(CIRCUITPY_ESPCAMERA)
CIRCUITPY_JPEGIO ?= $(CIRCUITPY_ESPCAMERA)
CIRCUITPY_QRIO ?= $(CIRCUITPY_ESPCAMERA)

Is there a reason other than space that you turned off GIFIO and JPEGIO on non-camera boards? A user requested them on for TFT boards. QRIO I can understand

onyx hinge
tulip sleet
#

well, when I turned on gifio, it didn't fit, but we are turning off _bleio on 4MB flash boards now. So I'm just trying to make it consistent. displayio makes sense

#

so that makes sense, I'll make it dependent on displayio, and I'm doing the _bleio narrowing at the same time

onyx hinge
#

Thank you!

manic glacierBOT
manic glacierBOT
manic glacierBOT
manic glacierBOT
manic glacierBOT
#
CDarius
[adafruit/circuitpython] New review comment on pull request #8816: Initial working m5stack cardputer setup 
uint8_t display_init_sequence[] = {
    // SWRESET and Delay 140ms
    0x01, 0 | DELAY, 140,
    // SLPOUT and Delay 10ms
    0x11, 0 | DELAY, 10,
    // COLMOD 65k colors and 16 bit 5-6-5
    0x3A, 1, 0x55,
    // INVON Iiversion on
    0x21, 0,
    // NORON normal operation (full update)
    0x13, 0,
    // MADCTL columns RTL, page/column reverse order
    0x36, 1, 0x60,
    // RAMCTRL color word little endian
    0xB0, 2, 0x00, 0xF8,
    // DIPON display on
 ...
manic glacierBOT
manic glacierBOT
#
CDarius
[adafruit/circuitpython] New review comment on pull request #8816: Initial working m5stack cardputer setup

Cardputer display backlight is provided by the StampS3:

image

I did some test on the brightness and found out that with a PWM frequency of 50kHz any brightness value less then 1.0 make the screen completely dark. For example if the brightness is set to 0.9 the display goes completely black. I guess that this strange behavior is caused by the SGM2578

![image](https://github.com/adafruit/...

manic glacierBOT
manic glacierBOT
manic glacierBOT
#
justmobilize
[adafruit/circuitpython] New comment on issue #8919: Slower https access on 9.0.0

Running 5 tests on the most recent build of 9.0.0-beta.0 I got:

Running Adafruit

 - Common hosts
   - api.fitbit.com:443 took 0.70 seconds | passed
   - api.github.com:443 took 1.14 seconds | passed
   - api.thingspeak.com:443 took 0.90 seconds | passed
   - api.twitter.com:443 took 0.55 seconds | passed
   - discord.com:443 took 1.02 seconds | passed
   - id.twitch.tv:443 took 0.51 seconds | passed
   - oauth2.googleapis.com:443 took 0.72 seconds | passed
   - opensky-...
onyx hinge
#

well that's weird. latest-ish build on memento, I can type into the terminal and I see the output on the LCD but it's not echoed back to the computer.

#

I've triggered this a couple of times, but it's not consistent. It seems to just happen randomly.

#

It's happening at c84c066 as well so it's nothing to do with the PR I'm working on

#

I haven't reproduced it on another computer so I'll write it off as something odd on this laptop (which I don't want to reboot right this second, I've got a multi hour test of an unrelated thing running)

short tendon
tulip sleet
#

that is used in a "random art picture" Learn Guide

short tendon
tulip sleet
manic glacierBOT
#
jepler
[adafruit/circuitpython] New comment on pull request #8808: Change sys.version to return the whole port identification.

Try with:

diff --git a/py/makeversionhdr.py b/py/makeversionhdr.py
index 430b9bef4b..edcb994b4f 100644
--- a/py/makeversionhdr.py
+++ b/py/makeversionhdr.py
@@ -119,7 +119,7 @@ def make_version_header(repo_path, filename):
 #define MICROPY_VERSION_STRING "%s"
 // Combined version as a 32-bit number for convenience
 #define MICROPY_VERSION (MICROPY_VERSION_MAJOR << 16 | MICROPY_VERSION_MINOR << 8 | MICROPY_VERSION_MICRO)
-#define MICROPY_FULL_VERSION_INFO "Adafruit CircuitPyt...
mortal kernel
#

@danh I followed up with a PR to bluekitchen/btstack for the missing prototypes. They merged it immediately into their development branch.
I've been very favorably impressed with BTstack and wonder if it would be worthwhile for Adafruit to request a license that would allow us to use it more widely in CP. For example, it could make it much easier to bring devices/ble_hci/common-hal/_bleio up to par.

manic glacierBOT
solar whale
#

@onyx hinge my new OV featherwing works!! OV5640 connected to a metro esp32s3. Sending images to AIO…. Now I can easily swap OV5640 and OV2640 boards. The shorter connections seem to work fine.

IMG_6045.jpg
onyx hinge
manic glacierBOT
#
dhalbert
[adafruit/circuitpython] New comment on pull request #8924: esp-idf-config/sdkconfig.defaults: Fix TLS access to api.github.com; remove BLE from 4MB ESP32-S3

so the theory is, github gets tired of waiting for key exchange without the optimized implementation of the algorithm?

That could be. I looked at the code and I don't see that's it non-functional without the optimizations.

I think we want the optimizations anyway because it's quite slow without them in 9.0.0 compared with 8.2.x, enough to provoke #8919.

tulip sleet
manic glacierBOT
short tendon
#

@tulip sleet I ran through what I could easily find. Some are just bad now, but this is the one that errors: aeroapi.flightaware.com with a Failed SSL handshake - and it's both in 8.2.9 and most current 9.0.0

manic glacierBOT
tulip sleet
short tendon
#

Yes. I can not find a build it works in

manic glacierBOT
mortal kernel
tulip sleet
#

@slender iron I should have waited for your review for #8924, in case you had anything to say about the BLE enabling/disabling refactoring.

tulip sleet
#

bluez is central only, I think

slender iron
#

esp-nimble is my pick

#

though zephyr will have the most active development

mortal kernel
#

I checked, and you're right about bluez being central only. I see a few projects that have added peripheral with bluez. esp-nimble is new to me, but on first glance it looks very promising.

short tendon
tulip sleet
mortal kernel
#

Does zephyr incorporate a branch of nimble?

slender iron
#

no, it is a separate stack

#

but nordic is all in on zephyr

#

nimble originated with mynewt but it's lost adoption to zephyr

#

espressif is moving to nimble from bluedroid and maintains their own fork of nimble

#

28 open 9.0.0 issues!

#

almost under a page

solar whale
#

Just a quick follow-up to some of the issues I was reporting yesterday with the cameras. I have had no problems with OV5640 and OV2640 cameras on the esp32s3 builds (memento, metro, feather 4mflash 2M PSRAM) I had to make custom builds for the metro and feather to enable the espcamera and add OV2640 support)... On the esp32s2 boards (kaluga 1.3 and feather esp32s2) espcamera support is there, but using the same code as on the esp32s3's I run into ESPIDF Memory errors and other issues like the ESP General Failure. I'll move on to trying to create coherent issues for this, but I just wanted to give a quick update and see if anyone had any thoughts about the different results with esp32s2 vs s3.

manic glacierBOT
#
tannewt
[adafruit/circuitpython] New comment on issue #8637: displayio Label text Y position ignored after first refresh

This works on 8.2.8 and 00824ad12229af15541d1431cf31f216e8e3587d on a MagTag. It may have been a library bug. I'm using display text 3.0.6.

Full test code:

import displayio
import board
import terminalio
import time

from adafruit_display_text import label

display = board.DISPLAY

INK_WIDTH = display.width
INK_HEIGHT = display.height

INK_WHITE = 0xffffff
INK_BLACK = 0x000000

display_group = displayio.Group()
background_bitmap = displayio.Bitmap(INK_WIDTH, INK...
manic glacierBOT
#
bill88t
[adafruit/circuitpython] New comment on issue #8443: idf v5.1 `listen`/`accept` race condition ends up losing clients

I prepared an example that relies on no libraries and I had even weirder behaviour:

CP code.py:

import wifi; from socketpool import SocketPool; from supervisor import reload
if not wifi.radio.connected:
    print("No wifi, reloading!"); reload()
print("IP: " + str(wifi.radio.ipv4_address) + "\nSetting up wifi for socket receives")
pool = SocketPool(wifi.radio)
rx_buf = bytearray(64)
def reset_rx_buf():
    for i in range(64):
        rx_buf[i] = 0
while True:
    print("S...
#
justmobilize
[adafruit/circuitpython] Issue opened: #8925 SSL error connecting to aeroapi.flightaware.com

CircuitPython version

Adafruit CircuitPython 9.0.0-beta.0-33-g264e68fe16 on 2024-02-15; FeatherS3 with ESP32S3
from `Merge pull request #8924 from dhalbert/tls-nist-optim`

Code/REPL

import os
import socketpool
import ssl
import wifi
import adafruit_requests

wifi_ssid = os.getenv("CIRCUITPY_WIFI_SSID")
wifi_password = os.getenv("CIRCUITPY_WIFI_PASSWORD")

while not wifi.radio.connected:
    wifi.radio.connect(wifi_ssid, wifi_password)

requests...
onyx hinge
#

I went down that rabbit hole for some reason ^

#

I think requests.head is broken.

print("HEAD")
r = requests.head("http://wifitest.adafruit.com/testwifi/index.html", timeout=3)
r.close()    
print("done")

This fails ```HEAD
Traceback (most recent call last):
File "code.py", line 41, in <module>
File "adafruit_requests.py", line 313, in close
File "adafruit_requests.py", line 301, in _throw_away
File "adafruit_requests.py", line 203, in _recv_into
OSError: [Errno 116] ETIMEDOUT

.. I think because a HEAD request may return a 'Content-Length' header but doesn't ever actually have any content..
short tendon
#

If you create an issue and tag me, I can get to it next week

onyx hinge
#

cool, appreciated!

manic glacierBOT
slender iron
manic glacierBOT
slender iron
solar whale
slender iron
solar whale
#

good...

manic glacierBOT
#
RetiredWizard
[adafruit/circuitpython] New comment on pull request #8816: Initial working m5stack cardputer setup

I tried to implement I2S audio in this branch, but I haven't gotten it working yet:

Unfortunately, I pulled the trigger on purchasing my Cardputer from the manufacturer rather than a local distributor without thinking first so I won't have one to test for a month or two. I suspect that I2S Audio out will work without your PDMIn updates though. I assume you're trying to get the Mic to work using the PDMIn module?

import synthio
import audiobusio
import board
import time

...
#
jepler
[adafruit/circuitpython] New comment on pull request #8926: mbedtls: move to shared-module

good call. The old Espressif code sets a 120s socket timeout(!) during ssl connect, overriding the regular timeout:

void common_hal_ssl_sslsocket_connect(ssl_sslsocket_obj_t *self,
    const char *host, size_t hostlen, uint32_t port) {
...
        // Connection successful, set the timeout on the underlying socket. We can't rely on 
the IDF
        // to do it because the config structure is only used for TLS connections. Generally,
 we
        // shouldn't hit this timeout becaus...
#
jepler
[adafruit/circuitpython] New comment on pull request #8926: mbedtls: move to shared-module

I see that get_socket in connectionmanager defaults to a timeout of 1s. This timeout may simply be too short, since it was never being honored on espressif before this PR.

That said I'm still a bit surprised; I would have said that this timeout is about the time until the low-level socket can be read/written, not the time before an SSL-encrypted byte can be read/written. why would a 1s timeout be problematic during the SSL handshake unless it really takes 1s before the server is sendi...

manic glacierBOT
manic glacierBOT
short tendon
#

Question for you all. Is there a reason for libraries we go:

  • root
  • frozen
  • lib

If there is a fix in a library and people follow normal guides and put it into lib, if it's frozen, the fix wouldn't be used...

slender iron
#

I think it was to prevent unintended RAM use

#

since using from lib will use RAM

#

but frozen uses less

short tendon
#

Gotcha. Makes sense, but also might add confusion.

manic glacierBOT
tulip sleet
manic glacierBOT
short tendon
tulip sleet
#

Circuit Playground Express and other SAMD21 boards don't have much RAM

#

and those are still popular boards

manic glacierBOT
short tendon
#

And if you had requests in lib, the running code would have crashed because not enough memory?

manic glacierBOT
tulip sleet
# short tendon And if you had requests in lib, the running code would have crashed because not ...

That would be a definite possibility. We had it the other way, and people would copy libraries onto CPX that were frozen. Then their programs would fail. It caused a lot of support traffic. The case of needing to override frozen libraries was a lot less common, so we switch the order. We still have to explain how to do that, but in most cases the frozen libraries are fine. It's confusing either way, but the current way works out better for most people.

#

MicroPython also has the same order, I believe.

short tendon
#

Sounds good. I'll leave well enough alone... I may try to swap that and see what happens next time I'm bored... 😉

#

Feels with the TLS fixes it's almost time for beta.1...

tulip sleet
#

yes, I'll probably do it within the next few days

short tendon
#

I'm excited, cause right now I have to go to S3 (AWS) to get a good UF2

manic glacierBOT
tulip sleet
short tendon
#

So, it was found out that the espressif code had a crazy hard coded timeout. I've now made sure to also test on a Pico W. As far as I'm concerned, I feel it's good to merge (both to get rid of the timeout and to make all boards more similar)

manic glacierBOT
short tendon
#

I also love it because what was a ssl handshake issue is now 4 specific errors

manic glacierBOT
short tendon
#

sooooo close to ConnectionManager v1

manic glacierBOT
manic glacierBOT
manic glacierBOT
#
michalpokusa
[adafruit/circuitpython] New comment on issue #8268: HTTPS Server works on raspberrypi but not on espressif

Does anyone have working test code for HTTPS Server for espressif with this PR? I'm getting RuntimeError: Invalid socket for TLS using code from https://github.com/ide/circuitpython-https-server updated for latest adafruit_httpserver. (I may be doing something wrong)

i.e., how do we know this is completed?

I also started trying HTTPS with adafruit_httpserver on ESP32-S2 TFT Feather. I get exactly the same error, specifically it gets raised by Server._sock.accept() in `...

manic glacierBOT
#
jepler
[adafruit/circuitpython] New review comment on pull request #8923: displayio: add, check "in rom" flag for groups

I tried this suggestion, but it was worse for code size. Instead, I renamed check_rom and in_rom (I'll push that in a moment)

I think this is because checking the bit is just an instruction or 2, while checking the gc size is some pointer dereferencing and a function call. Still, I was surprised by the amount of code growth, about 22 bytes per check_readonly call (there are 7), so I may have done something wrong. The check_readonly function was inlined on the pico w build so that m...

manic glacierBOT
#
anecdata
[adafruit/circuitpython] New comment on issue #8268: HTTPS Server works on raspberrypi but not on espressif

client side (essentially same output for ESP32-S3 and Pico W:

% curl -v --insecure https://192.168.6.243
*   Trying 192.168.6.243:443...
* Connected to 192.168.6.243 (192.168.6.243) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* Recv failure: Connection reset by peer
* LibreSSL/3.3.6: error:02FFF036:system library:func(4095):Connection reset by peer
* Closing connection
curl: (35) Recv failure: Connection reset by peer
lone sandalBOT
manic glacierBOT
manic glacierBOT
tulip sleet
manic glacierBOT
onyx hinge
#

I think we'll get down to 15 if we can merge 4 PRs that are in process

#

If I was in list trimming mode I'd consider striking https://github.com/adafruit/circuitpython/issues/8829 (it's a new feature) and https://github.com/adafruit/circuitpython/issues/8824 (it's an upstream bug)

GitHub
High duty cycle connectable directed advertising · Issue #8829 · ad...

The minimum advertising interval is 20ms. I want to use a smaller interval time. Is there any way to support for high duty cycle connected direct advertising?

GitHub
hard fault on serial disconnect on MacroPad RP2040 · Issue #8824 · ...

CircuitPython version Adafruit CircuitPython 9.0.0-alpha.6-33-g9f016796e0-dirty on 2024-01-22; Adafruit Macropad RP2040 with rp2040 Code/REPL # N/A; also happens with freshly reformatted CIRCUITPY ...

manic glacierBOT
slender iron
#

we can have thach look at 8824 when he's back

#

agree that 8829 can be punted to 9.x.x

manic glacierBOT
tulip sleet
#

@lone axle also of interest to you, or if you have time, could you take a look? 👆

manic glacierBOT
slender iron
#

17 9.0.0 open!

#

@tulip sleet @onyx hinge want to meet today to talk over the remaining bugs?

tulip sleet
#

sure - we are going out for a walk at 3pm ET

slender iron
#

jeff may be on a walk already

onyx hinge
#

I'm around, but wasn't hearing notifications @tulip sleet @slender iron

#

now is fine for me

tulip sleet
#

can be in one minute

slender iron
#

works for me

#

we're in amelia

manic glacierBOT
#
bill88t
[adafruit/circuitpython] New comment on pull request #8808: Change sys.version to return the whole port identification.

Oh whoops, sorry for the late response, I'm having exams currently, so I'm not very active.
Thanks Jepler for the fix.

Adafruit CircuitPython 9.0.0-alpha.6-32-gc7701709ef on 2024-02-16; DFRobot FireBeetle 2 ESP32-S3 with ESP32S3
>>> import sys
>>> sys.implementation
(name='circuitpython', version=(9, 0, 0), _machine='DFRobot FireBeetle 2 ESP32-S3 with ESP32S3', _mpy=262)
>>> sys.version
'3.4.0; Adafruit CircuitPython 9.0.0-alpha.6-32-gc7701709ef on 2024-02-16; DFRobot FireBeetle...
manic glacierBOT
tulip sleet
#

testers: PR merge build filenames in S3 ("Absolute Newest") now include the name of the branch they were merged to and the PR number. This was previously in PR build artifacts, but is now also in the S3 filenames.

image.png
short tendon
#

Oh that's so awesome!!!

tulip sleet
#

this should make it easier to track down what a particular build is for and what might be causing an issue if you are doing a "bisect" with S3 builds

#

also distinguishing between 8.2.x and main builds

#

The "no-branch" there was a previous failed attempt at this

short tendon
#

does main rebuild after a merge?

crimson ferry
#

what's the diff in the S3 builds between no-branch and main?

#

(looking for absolute latest with all the mbedtls goodies, but for raspberrypi, for which there was no build in the PR)

tulip sleet
#

no-branch is a bug: it is either main or 8.2.x. It was a backstop name when the code tried to find out the branch and couldn't

crimson ferry
#

so those builds are not to be trusted?

#

oh, oops sorry, just saw your post above

tulip sleet
#

no, nothing wrong with the builds. Before I first tried fixing the filenamesnames, there was no branch name in the filename. Then I added code to include the branch name. it worked on the artifact builds (in the zip's), but not in the merge builds. I had to fix that. The info available for a "pull_request" build vs a "push" build (the latter is when a PR is merged) is in different places

#

debugging CI .yml is tedious. I debugged this by making a PR to my own fork repo a number of times, and then merging it.

#

We even have someone here in discord who works for a company that has a GitHub Actions CI debugger

manic glacierBOT
#
dhalbert
[adafruit/circuitpython] Pull request opened: #8936 Use pre-commit/action@v3 instead of @v3.0.0

Fixes this warning which is appearing when running pre-commit.yml:

pre-commit
Node.js 16 actions are deprecated. Please update the following actions to use Node.js 20: actions/cache@v3. For more information see: https://github.blog/changelog/2023-09-22-github-actions-transitioning-from-node-16-to-node-20/.

pre-commit/actions@v3.0.1 updates to Node.js 20. Let's do @v3 to pick up further bug fixes automatically.

manic glacierBOT
short tendon
#

Man! was so hoping CM would get in there.... Next time

tulip sleet
#

it's only beta.1 🙂

short tendon
#

I know. Will that SSL fix get in beta.1?

manic glacierBOT
#
tannewt
[adafruit/circuitpython] New comment on issue #8443: idf v5.1 `listen`/`accept` race condition ends up losing clients

So I don't think there is necessarily a bug in CP here. The example code has a couple bugs from its intention I think.

  1. connection2 = socket.accept() is meant to be connection2 = socket2.accept(). That way it gets the correct data.
  2. socket2 is never closed in code.py. socket.bind() currently allows port reuse which makes this leak not obvious. As a consequence, the total number of sockets gets filled up with each loop.

I don't think bind() should allow reuse implicitly ...

slender iron
#

@plucky tulip I'm thinking about making bind() not allow port reuse automatically. will that work ok for httpserver?

#

I'm adding the ability to request it explicitly though

plucky tulip
slender iron
#

ya, only if a socket was just closed (or accidentally left open)

plucky tulip
#

But after a board reset, the sockets will reset too, wouldn't they? We could modify the lib to explicitly set the reuse on CP, therefor the behaviour would stay the same. Is there any negative aspect of using reuse that coudl affect httpserver?

slender iron
#

hard reset will reset everything. a reload should close sockets but that takes time

#

close doesn't happen immediately because there may be inflight packets on their way

#

allowing reuse risks getting those inflight packets from the previous bind. that's fine for a server though I think

plucky tulip
#

Sound like the exact problem that was mentioned not so long ago in CPython with adafruit_httpserver, the solution seemed to be setting the reuse on the socket, so imo doing the same in CircuitPython wouldn't be much different.

slender iron
manic glacierBOT
#
justmobilize
[adafruit/circuitpython] New comment on pull request #8935: lib/certificates: update to include ssl.com RSA cert

tested on Adafruit CircuitPython 9.0.0-beta.0-47-ga0f907e71a on 2024-02-16; FeatherS3 with ESP32S3 with all the known hosts in the learning guides:

Running Adafruit

 - API hosts
   - api.coindesk.com:443 took 2.51 seconds | passed
   - api.covidtracking.com:443 took 4.99 seconds | passed
   - api.developer.lifx.com:443 took 1.04 seconds | passed
   - api.fitbit.com:443 took 0.50 seconds | passed
   - api.github.com:443 took 2.94 seconds | passed
   - api.hackaday.io:443 too...
manic glacierBOT
#
mMerlin
[adafruit/circuitpython] Issue opened: #8939 super().__new__() different inheriting from object vs custom class

CircuitPython version

Adafruit CircuitPython 8.2.9 on 2023-12-06; Adafruit PyPortal with samd51j20

Code/REPL

class ChildSingleton(TopSingleton):
    """Inherit from singleton (another singleton)"""
    # do not explicitly override (shadow) class variable from parent class
    def __new__(cls, *args, **kwargs):
        if TopSingleton.instance is None:
            if sys.implementation.name == 'cpython':
                super().__new__(TopSingleton, *a...
#
dhalbert
[adafruit/circuitpython] New comment on issue #8939: super().__new__() different inheriting from object vs custom class

I am seeing the the same behavior in MicroPython:

icroPython v1.22.1 on 2024-01-05; PYBv1.1 with STM32F405RG
Type "help()" for more information.
>>> from bug_test_shadow_attributes import bug_main
>>> bug_main()
bug_main: start for (name='micropython', version=(1, 22, 1, ''), _machine='PYBv1.1 with STM32F405RG', _mpy=7686)
ChildSingleton.__new__: instance <class 'NoneType'>
ChildSingleton.__new__: instance <class 'ChildSingleton'>
ChildSingleton.instance_check: TopSingleton.ins...
tulip sleet
#

@slender iron I am afk for a while now but will see about finishing off the rest of the PR's and getting things ready for a release. Or could be Monday; but we're in a good space now

short tendon
#

@tulip sleet thanks for working through all the cert issues I found this week

manic glacierBOT
#
michalpokusa
[adafruit/circuitpython] New comment on issue #8268: HTTPS Server works on raspberrypi but not on espressif

@jepler I also tested on my MatrixPortal S3, works very fast, I also tested with Websockets and wss://.

But I was unable to make it work with ESP32-S2 Feather TFT, I can start the server but when trying to connect I get an empty MemoryError:. Seems strange considering that both have 2MB of RAM, although MatrixPortal has SRAM and Feather has PSRAM, would that make any difference?

#
bill88t
[adafruit/circuitpython] New comment on pull request #8940: Require explicit socket port reuse 
>>> import wifi
>>> import socketpool
>>> pool = socketpool.SocketPool(wifi.radio)
>>> socket = pool.socket(pool.AF_INET, pool.SOCK_STREAM)
>>> socket.bind(("0.0.0.0", 20))
>>> socket2 = pool.socket(pool.AF_INET, pool.SOCK_STREAM)
>>> socket.bind(("0.0.0.0", 21))
>>> socket2.close()
>>> socket2 = pool.socket(pool.AF_INET, pool.SOCK_STREAM)
>>> socket.bind(("0.0.0.0", 21))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OSError: [Errno 112] EADDRINUSE
...
#
anecdata
[adafruit/circuitpython] New comment on issue #8268: HTTPS Server works on raspberrypi but not on espressif

I get the same on QT Py ESP32-S3 (N4R2), no dual http/https servers (just https):

server:

Traceback (most recent call last):
  File "adafruit_httpserver/server.py", line 404, in poll
MemoryError: 
Traceback (most recent call last):
  File "code.py", line 61, in <module>
  File "adafruit_httpserver/server.py", line 450, in poll
  File "adafruit_httpserver/server.py", line 404, in poll
MemoryError:

client:

$ curl -iL -v --insecure https://192.168.6.180?HTTPS
*   ...
brazen hatch
# manic glacier

Please someone tell me this is a bug and not intended behaviour. Cause if this is how it should be, a lot of web applications cannot be implemented at all.

#

Shouldn't .close() unbind?

#

If not, shouldn't there be a .unbind()??

manic glacierBOT
short tendon
#

Asking because I'm thinking about it, but not expecting a response over the weekend....

Is there a reason Adafruit doesn't squash PRs to simplify the history?

slender iron
#

CPython does this too

slender iron
slender iron
short tendon
slender iron
short tendon
#

It's ready when you are (will also comment on it)

slender iron
#

I'm done for the week so I'll look next week

short tendon
#

Not rush. Have a great weekend

brazen hatch
brazen hatch
#

But yea SO_REUSEADDR so it's fine.

manic glacierBOT
manic glacierBOT
#
sperok
[adafruit/circuitpython] New comment on issue #8918: Input/Output: MacOS Sonoma 14.4b3 CIRCUITPY Slow Write Performance

The code attached will automatically remount a CIRCUITPY device as synchronous, likely minimizing risk of data corruption and delayed writes. It is human readable XML, uncommented. Install once, runs unattended, persists across Mac reboots and unlimited insertions/removals on any number of devices.

This is a launch daemon property list configured to watch /Volumes for change events without polling. An embedded bash script examines all mounted CIRCUITPY devices on change. Devices mounted ...

manic glacierBOT
manic glacierBOT
#
jerryneedell
[adafruit/circuitpython] Issue opened: #8942 espcamera ESPIDF Generic Failure and Memory Error on ESP32S2

CircuitPython version

Adafruit CircuitPython 9.0.0-beta.0-52-g6c1e34e957 on 2024-02-17; Adafruit Feather ESP32S2 with ESP32S2

Code/REPL

import board
from digitalio import DigitalInOut
import time
import espcamera

cam = espcamera.Camera(
    data_pins=[board.D6,board.A5,board.D9,board.A4,board.D10,board.A3,board.D11,board.A2],
    external_clock_pin=board.D12,
    pixel_clock_pin=board.A1,
    vsync_pin=board.A0,
    href_pin=board.D13,
    pixel...
manic glacierBOT
#
jerryneedell
[adafruit/circuitpython] New comment on issue #8942: espcamera ESPIDF Generic Failure and Memory Error on ESP32S2

I am trying to make a build for the feather esp32s2 with imagecapture instead of esp32camera, but it is not building. What am I missing?


jerryneedell@Mac-mini espressif % cat boards/adafruit_feather_esp32s2/mpconfigboard.mk
USB_VID = 0x239A
USB_PID = 0x80EC
USB_PRODUCT = "Adafruit Feather ESP32S2"
USB_MANUFACTURER = "Adafruit"

IDF_TARGET = esp32s2

CIRCUITPY_ESP_FLASH_MODE = qio
CIRCUITPY_ESP_FLASH_FREQ = 80m
CIRCUITPY_ESP_FLASH_SIZE = 4MB

CIRCUITPY_ESP_PSRAM_SIZE = 2M...
manic glacierBOT
#
jepler
[adafruit/circuitpython] New comment on issue #8942: espcamera ESPIDF Generic Failure and Memory Error on ESP32S2

That code has been removed. Scott removed part of it in 3b159317240a1ff2b56101d1c92aec4ab2d9f454 while I removed the rest in efed3c975897069943c7e409fd48670dc17e0a37. So you could travel back in time to efed3c975897069943c7e409fd48670dc17e0a37 with git to have that module possible to build, maybe -- it hadn't been used in a long time (not since the idf5 update), so I don't even know whether it would build before the related files were removed.

#
jerryneedell
[adafruit/circuitpython] New comment on issue #8942: espcamera ESPIDF Generic Failure and Memory Error on ESP32S2

I think the real question is whether there is a bug causing the problems with espcamera on the esp32s2 or if esp32s2 simply can’t handle espcamera. If that is the case, do we just document that and move on or is it worth trying to revert to something lime imagecapture. There does not appear to have been a great demand for it so I expect there are more important issues to resolve at this time.

manic glacierBOT
lone sandalBOT
manic glacierBOT
#
eightycc
[adafruit/circuitpython] New comment on issue #8918: Input/Output: MacOS Sonoma 14.4b3 CIRCUITPY Slow Write Performance

The excellent workaround for #8449 by @sperok will work with Sonoma 14.3 and earlier. The reason that the workaround worked was that re-mounting the device caused the older MSDOS filesystem kext to be loaded.

Unfortunately, Apple has changed mount behavior in Sonoma 14.4 beta 4 so that it continues to use the faulty user space MSDOS filesystem driver across a re-mount.

manic glacierBOT
#
eightycc
[adafruit/circuitpython] New comment on issue #8918: Input/Output: MacOS Sonoma 14.4b3 CIRCUITPY Slow Write Performance

wouldn't breaking "synchronous,noasync" behavior qualify as a bug?

Of course, but that's not exactly what's going on.

Apropos #8449 mount flags are a red herring. I tested various combinations of flags on a remount, and found they had no effect. I later found that this was because re-mounting with any combination of flags caused Sonoma to pull in older kext code that worked properly vs. Sonoma's new user space driver. "Properly" in this context is that it did not delay metadata writes...

manic glacierBOT
#
bill88t
[adafruit/circuitpython] Issue opened: #8943 ESP32 missing `SOL_SOCKET` and SO_REUSEADDR`

CircuitPython version

Master

Code/REPL

import wifi
import socketpool
pool = socketpool.SocketPool(wifi.radio)
socket = pool.socket(pool.AF_INET, pool.SOCK_STREAM)
socket.setsockopt(pool.SOL_SOCKET, pool.SO_REUSEADDR, 1)

Behavior

Traceback (most recent call last):
File "", line 1, in
AttributeError: 'SocketPool' object has no attribute 'SOL_SOCKET'

Description

No response

Additional information

No response

lone axle
#

Are there any known issues with MEMENTO during file writes from the PC? I'm having a very hard time copying files onto the device. Partway through copying the device resets and then when it comes up there is a hard fault message

CircuitPython core code crashed hard. Whoops!
Hard fault: memory access or instruction error.
Please file an issue with your program at github.com/adafruit/circuitpython/issues.
Press reset to exit safe mode.

In many cases the device was sitting in the REPL not running code.py at the time.

tulip sleet
lone axle
#

will do, it doesn't seem like the specific files make a difference. Sometimes it was just copying libs from a project zip downloaded from bundle. Another time it was editing boot.py and another it was making arbitrary changes in an unused file called other.py

mortal kernel
#

@danh FYI: Looks like we lost the ability to work around new Sonoma performance issue. See #8918.

lone axle
#

actually, I take that back. It happens for me consistently if I paste the display text library and it seems to always fail on the same file from my OSs perspective

image.png
tulip sleet
tulip sleet
lone axle
#

I have done that once but I'll try a few more times on various versions and see if it stays consistent.

tulip sleet
#

also try copying from, say, a Linux system or an RPi

lone axle
#

Everything has been copied from a Linux system so far (my primary PC is Ubuntu). I can try a RasPi too though I have one handy

mortal kernel
#

@danh @lone axle Anything I can do to lend a hand?

lone axle
mortal kernel
lone axle
#

For me it freezes ever other run after it prints sdcard init @ 2.834 but before mount vfs @ 2.995. The other times it seems to run successfully and can save photos.

#

Okay no worries. I'll re-try later with beta.1 and if I'm still getting the same consistent trouble I'll make an issue with the details.

manic glacierBOT
mortal kernel
#

A while back I thought I saw support added for accessing an SD card through USB MSC in addition to the built-in flash. Is this true, or am I making like an AI and hallucinating it? It could be very helpful for some debug I'm doing.

manic glacierBOT
#
dhalbert
[adafruit/circuitpython] New tag created: 9.0.0-beta.1
tulip sleet
orchid basinBOT
manic glacierBOT
wraith crow
#

@lone axle I finally pulled out my Memento and loaded it up. I didn't have any trouble getting the CircuitPython bundle from the learn guide copied over to the Memento after which I took a quick test photo which seemed to work fine as well. ~I did see a Web Workflow issue that I thought was at least mostly resolved but I was able to get the SD card to show up through web workflow and copy the image off by making sure no program was running on the Memento before attempting to access the web workflow pages. Of course that means you need to manually mount the SD card from the REPL as well. I've got to run out for a couple hours but I'll play around with it later this evening and maybe open an issue on the web workflow if I can pin it down well enough.~ seems resolved with Beta 1

manic glacierBOT
#
eightycc
[adafruit/circuitpython] New comment on pull request #6555: Multiple luns

Could it be dynamic in that case?

It could be dynamic in the same sense as adding/changing/deleting partitions with fdisk if you imagine fdisk running inside CP. You'd need to remove the single LUN from the host OS and reinsert it in order to ensure that the host OS sees the changes. It could be messy.

Would the partitions for unmounted filesystems be visible?

No. We'd elide all but existing filesystems from the fake MBR.

This approach would work best where the additional S...

orchid basinBOT
midnight ember
#

I loaded up my memento with beta 0 to help FoamyGuy troubleshoot. I used the project bundle from the camera learn guide. Could not replicate the issue with Mu on Windows 10. I took some pictures and soft reloaded about 20 times without issue. Hopefully just an SD Card issue. I did have to reformat my SD Card on first use even though I pulled it directly from a TFT Featherwing where it was working fine. The Memento refused to detect the card. Reformatted with SD Formatter and everything was fine. My Memento came preformatted with ```py
TinyUF2 Bootloader 0.16.0-27-g3d79e6c - tinyusb (0.15.0-566-gf1e006d09)
Model: Adafruit PyCamera
Board-ID: ESP32S3-Camera-revC
Date: Oct 27 2023
Flash Size: 0x002C0000 bytes

tulip sleet
tulip sleet
#

@onyx hinge @slender iron @lone axle I passed on a report above that the adabot reports_cron.yml appeared not to be running properly, because the latest reports were missing from the S3 bucket. It turned out the reports were running just fine, but there were so many files in there that the javascript file lister could not list them all properly in order. If you scrolled down you could see the missing reports. I deleted the 2022 reports and moved the 2023 reports to a subdir, and now they show up in proper order.

short tendon
#

So excited! Under 9.x - adafruit_minimqtt + adafruit_requests + adafruit_connection_manager is only 1 yes 1 extra byte!! (although how long that will stay true, nobody knows...)

wraith crow
#

Just flashed the Memento with 9.0.0 Beta 1 and Web Workflow is working fine with the camera demo from the learn guide running.

vital matrix
#

Idle question from looking through the pure-python libraries - in what situations does adafruit_bitbangio come in handy, compared to the C-based bitbangio implementation? (I've been naively assuming anything that supports digitalio will definitely include bitbangio anyway, but maybe that's not the case/other reasons to use the pure-python version?)

midnight ember
wraith crow
#

I'm sure one of the Adafruit folks will come back with a definitive answer but I think the pure python version of sdcard pre-dates the C-based version scardio and it's maintained because some of the smaller boards don't have the sdcardio module enabled (it won't fit). May not apply to bitbangio though 😄

vital matrix
wraith crow
#

From the Docs site, it doesn't look like the bitbangio c-module is included on the samd21 boards

midnight ember
#

many adafruit libraries are helper abstractions. doesn't necessarily mean it's the best way for your use case. adafruit is beginner centric so helper libraries are often geared towards beginner use. if you have a history of python and/or C development then you might want a more direct route and sometimes that means not using a helper library.

#

or cherry picking functions from helper libraries to reduce overhead. samd21 is a great example as it doesn't have a lot of overhead so you will have to highly optimize vs a samd51 or other board with more cpu & ram.

short tendon
manic glacierBOT
#
sperok
[adafruit/circuitpython] New comment on issue #8918: Input/Output: MacOS Sonoma 14.4b3 CIRCUITPY Slow Write Performance

Thanks for the detailed explanation. I've now gone back through #8449 and read the entire history, but remain unclear on the resolution of #8449 by 14.4. If Apple "re-write all metadata (FAT tables and root directory in our case) after every few data block writes" doesn't there remain a vulnerable window for auto-reload? After write of data block and before metadata? or is default auto-reload timer sufficient?

manic glacierBOT
tulip sleet
lone axle
#

@wraith crow and @midnight ember I'm doing some more testing on the MEMENTO and think I've narrowed the source of some of my problems further. I did swap SD cards and it resolved my "every other run freezes" issue. And I can now seemingly use the camera successfully over and over.

I went to get back toward opening my .bmp file and blitting it with the resulting photos but then I started having the hard fauld while writing files problem return. I've now noticed that if I have my frame_rgb888.bmp file on the CIRCUITPY drive it seems to cause the major problems. It will either fail to paste that file and hard crash. Or I've seen it succeed to paste this file but then if I start pasting other files or making changes to other files it hard crashes on one of the writes.

If either of you have a moment can you try pasting this file onto your device and see if it successfully pastes and whether you start having trouble afterward? FWIW I seem to be able to get back to normal afterward by deleting that file and then doing storage.erase_filesystem() in the REPL.

#

(the first image was one of the corrupted ones. The current one here is the real one)

#

My theory right now is maybe any file that is "too large" on the drive causes this trouble. I don't think there is anything particularly special about this image, but it is by far the largest file on the drive

midnight ember
#

Can you share the code.py because I'm just using the default one from the guide.

#

Pretty sure I don't have the border code if that's what the image is for.

lone axle
#

but TBH I don't think the code matters. I'll test without it. I'm pretty sure as soon as the large image is on the drive it starts going sideways without codepy doing anything.

midnight ember
#

My windows file data says the file is 24-bit bmp. Is that intentional?

short tendon
#

@lone axle so you are saying putting a large file on the drive causes issues? I have a memento and am happy to test

lone axle
#

any additional testing appreciated. I even just tested with this dummy file

📎 dummy.txt
lone axle
short tendon
#

Will play with it when I get home (if it hasn't been figured out already)

lone axle
#

Thank you. 👍

#

I did also confirm just now. without the camera code present at all I can still get the hard fault. So I'm pretty certain the specific code in code.py or anything about the camera specifically isn't related.

midnight ember
#

Put the file on the Memento, took a picture, hit file save in Mu a couple times. Still good.

#

Default code.py I can't use the buttons though. ok, select, dpad, they do nothing. Unsure if that's supposed to happen it's my first time using it. Only thing that works is off/on, reset, and cheese.

lone axle
#

I haven't tried any of those either, I don't recall seeing them mentioned in the code, they're probably not used in the basic example.

#

Thank you for testing. I'll try copying the file from an rpi and see if I can find a windows computer to test with. Maybe it's something about my computer copying large files that it's having trouble with.

midnight ember
lone axle
#

Sometimes pasting the first copy of dummy.txt succeeds but so far attempting to paste a second copy then causes a hardfault reliably.

#

I did see the same thing when copying that file from the raspberry pi just now as well.

wraith crow
#

As soon as I start a paste or copy of the file, the Memento does an auto-reload. After several copy attempts I did get a hard fault. I'm going to try and turn off the auto-reload function and see if it behaves better

short tendon
#

Haven't had a chance to test, but would be interesting if you did it in the REPL. Just created a big file to see what happens

tulip sleet
wraith crow
#

ubuntu

manic glacierBOT
#
FoamyGuy
[adafruit/circuitpython] Issue opened: #8944 Large Files on MEMENTO Causes Device Reset & Eventual Hard Crash

CircuitPython version

Adafruit CircuitPython 9.0.0-beta.1 on 2024-02-17; Adafruit Camera with ESP32S3

Code/REPL

# Doesn't seem to matter
print("Hello World!")

Behavior

I've noticed that putting files of around 900kb in size on the MEMENTO drive seems to cause problems for the device. Originally I noticed this with a .bmp file that is 900kb I can provide that if it's though to be important, but I've since replicated the same issue using this "dum...

lone axle
#

I will reset my drive and try from windows

tulip sleet
#

i have reproduced this with dummy.txt on Ubuntu

lone axle
#

Thank you (everyone) for trying. I'm to see it's reproducable at least.

manic glacierBOT
wraith crow
#

is placing supervisor.runtime.autoreload=False in a boot.py the correct way to turn off auto-reload?

tulip sleet
#

yes

wraith crow
#

okay, that doesn't help 😄

lone axle
#

reproduced on windows. 1st paste of the file was successful, but within a second or two of attempted to paste a second copy I got same symptom.

manic glacierBOT
#
FoamyGuy
[adafruit/circuitpython] New comment on issue #8944: Large Files on MEMENTO Causes Device Reset & Eventual Hard Crash

I have reproduced the same issue when the host OS pasting the file is Win 10 as well.

For testing that I did storage.erase_filesystem() first to start with a blank slate.

Then plugged in the MEMENTO to Win10 PC and copied the dummy.txt file to the drive

First paste of that file was successful. Renaming the first copy of the file was successful.

Then when I attempt to paste a new copy of the file the device resets with the same hard fault message as posted above.

tulip sleet
#

I cannot reproduce on Metro ESP32-S3 with beta.1. I can reproduce on MEMENTO with alpha.2.

#

and later

lone axle
#

I had been using the ESP32-S3 TFT prior to switching to the MEMENTO and I never saw it on that device (or any other devices that I can recall)

manic glacierBOT
lone axle
#

I don't see it anywhere on the silkscreen so I'm guessing no, but is the DBG pin exposed on the MEMENTO somehow?

tulip sleet
#

now which is JP2? ...

manic glacierBOT
tulip sleet
#

I also could not reproduce on a Feather ESP32-S3 Reverse TFT, which uses the same ESP32-S3-MINI-1 N4R2 module.

#

edited my issue post to include that info

manic glacierBOT
#
davs2rt
[adafruit/circuitpython] Issue opened: #8945 UART: unable to set value of Pull-up/down on defined pins, especially needed on RX

CircuitPython version

Adafruit CircuitPython 8.2.9 on 2023-12-06; Seeed Studio XIAO ESP32C3 with ESP32-C3FN4 

digitalio allows setting pin.pull values, however using busio.UART (and possibly others) the RX pin seems to be pulled up.

In my use case, the circuit driving RX is weak, and I'm not getting solid low voltages, causing data corruption.  I'd like to see the default on inputs be configurable, or at least not pulled up or down, i.e tri-stated or pin.pull=None.
This ...
lone axle
# tulip sleet You mean the UART?

On the Feather S3 TFT and I think a few other S3 feathers there is a pin down at the end labeled DBG. Where you can listen for more data during hard crashes with a debug build.

Is that moved to the uart pins for memento?

image.png
tulip sleet
#

yes, that's just the TXD0 pin, which is available on the MEMENTO too

#

uh oh, FLASH_SDKCONFIG = esp-idf-config/sdkconfig-4MB-1ota.defaults in mpconfigboard.mk for the Camera. That file does not actually exist! I'll do another build

manic glacierBOT
#
FoamyGuy
[adafruit/circuitpython] New comment on issue #8944: Large Files on MEMENTO Causes Device Reset & Eventual Hard Crash

I was also able to reproduce without the host OS involved in the copying at all by setting storage to writable and using code like this in the REPL:

>>> f = open("dummy.txt", "w")
>>> row = "AAAA AAAA AAAA AAAA\n"
>>> f.write(row*45*1000)
900000
>>> f.close()
>>> f = open("dummy2.txt", "w")
>>> f.write(row*45*1000)
Disconnected

It reset to the same hardfault message during the second write call.

tulip sleet
#

The FLASH_SDKCONFIG setting is irrelevant. It's not actually used anywhere. But the partitions set up by the combined.bin when the bootloader is flashedmay not match what the build is expecting.

#

I erased flash and loaded a .bin version of beta.1, and it does not crash

wraith crow
#

Is there a new beta.1 or are you just adding the erase step before writing the bin?

tulip sleet
#

the tinyuf2 partitition table and the one I think I am using are identical. Still more to check.

manic glacierBOT
wraith crow
#

@danh Just for kicks, I built an image with your first observation fixed (FLASH_SDKCONFIG = esp-idf-config/sdkconfig-flash-4MB-1ota.defaults) and it doesn't seem to be crashing, I've copied the 900kb file over, renamed it and then copied it again so two copies, with no hard crash

tulip sleet
wraith crow
#

Nope, a UF2

tulip sleet
#

so you commented it out, or you changed it?

#

what change did you make?

wraith crow
#

I thought I remembered some script digging in and pulling the value out to build the partition with, but I'm not really sure

tulip sleet
#

what is the size of cpy? df -h

wraith crow
#
  • FLASH_SDKCONFIG = esp-idf-config/sdkconfig-4MB-1ota.defaults changed to:
#
  • FLASH_SDKCONFIG = esp-idf-config/sdkconfig-flash-4MB-1ota.defaults
#

2.6MB (2,580,480) : size of UF2

#

/dev/sde1 2.3M 1.8M 537K 78% /media/retiredwizard/CIRCUITPY

tulip sleet
#

see, the thing is, FLASH_SDKCONFIG isn't mentioned anywhere. It's now FLASH_SIZE_SDKCONFIG

#

if you grep (I use ag), that's the only mention in any file

wraith crow
#

Well, I didn't put the demo bundle on, so maybe I'll do that and see if the crash comes back

tulip sleet
#

i think a possible issue may be that we're filling up CIRCUITPY with the second copy and it doesn't react well. But there is also a possible issue with dualbank maybe expecting two partitions (not sure about that).

#

my CIRCUITPY after building "correctly" is 3.7MB

wraith crow
#

A year or two ago, I was looking at the partition tables, never figured them out but I thought there was a weird script in the build process that didn't use the variable just went in and found the filename by searching the mpconfigboard.mk file. I could be totally off, but thats why I went ahead with the build

tulip sleet
#

tools/update_sdkconfig.py does look inside, but that's to create the sdkconfig statically.

#

Scott cleaned all this up a few months ago.

wraith crow
#

I'll go ahead and do an install of the learn bundle and see if I can reproduce the crash again....

tulip sleet
#

But i am find remnants, like this case. This is actually the only board that specifies a nonstandard partitition table in mpconfigboard.mk

#

I will probably stop soon and sleep on this.

#

and it's not clear to me that the fatfs filesystem stuff is using any info about the partitition table. I think it's just looking for the user filesystem partition and using it.

#

there is no actual specification of CIRCUITPY size for espressif; sometimes there is on other ports (especially when it's internal flash that's shared with the firmware0

wraith crow
#

as with the PicoW?

tulip sleet
# 
halbert@cod:~/Downloads$ cp dummy.txt /media/halbert/CIRCUITPY/
halbert@cod:~/Downloads$ df -h /media/halbert/CIRCUITPY/
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb1       3.7M  885K  2.9M  24% /media/halbert/CIRCUITPY
tulip sleet
#

ok, now I'm apparently getting corruption:

halbert@cod:~/Downloads$ df -h /media/halbert/CIRCUITPY/
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb1       3.7M  143K  3.6M   4% /media/halbert/CIRCUITPY
halbert@cod:~/Downloads$ ls -lh dummy.txt
-rw-rw-r-- 1 halbert halbert 879K Feb 17 19:56 dummy.txt
halbert@cod:~/Downloads$ cp dummy.txt /media/halbert/CIRCUITPY/
halbert@cod:~/Downloads$ sync
halbert@cod:~/Downloads$ df -h /media/halbert/CIRCUITPY/
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb1       3.7M  885K  2.9M  24% /media/halbert/CIRCUITPY
halbert@cod:~/Downloads$ cp dummy.txt /media/halbert/CIRCUITPY/
cp: cannot create regular file '/media/halbert/CIRCUITPY/dummy.txt': Read-only file system
#

ok, good night, I'm not going to keep myself awake with this now. need to unwind before sleeping

#

also I did some unrelated cleanup of some obsolete settings I found while making other changes

wraith crow
#

good night, I'll stare at your changes for a bit but I'll probably call it soon as well....

manic glacierBOT
#
dhalbert
[adafruit/circuitpython] New comment on issue #8944: Large Files on MEMENTO Causes Device Reset & Eventual Hard Crash

I corrected the MEMENTO build mpconfigboard.mk to use FLASH_SIZE_SDKCONFIG instead of FLASH_SDKCONFIG, and setting CIRCUITPY_DUALBANK = 0. See https://github.com/dhalbert/circuitpython/tree/memento-storage-crash.

Now, starting from a clean filesystem, I copy the file once, and then when copying again, the filesystem has been set to read-only due to an error during the first copy. var/log/syslog shows:

Feb 17 23:37:20 cod udisksd[819]: Mounted /dev/sdb1 at /media/halbert/CIR...
orchid basinBOT
#
michalpokusa
[adafruit/circuitpython-org] Pull request opened: #1328 Refactor of search function from regex to string based, massive speedup in searchbar performance

Current searching and filtering login implementation uses regexes, which leads to performance issues. If someone would type quickly "matrixportal", the search bar would hang on e.g. 8th character for a moment and then continue to render.

Additionally, current regex makes it that "bit matrix" and "matrix bit" produce different result, which i believe is not intended.

Overall, this change speed up filtering (in my case, on my computer) from 400-700ms to 2-5ms and produces same results no...

manic glacierBOT
#
michalpokusa
[adafruit/circuitpython] Issue opened: #8947 `Error: Failure to bind` on ESP when using HTTPS with `adafruit_httpserver`

CircuitPython version

Adafruit CircuitPython 9.0.0-beta.1 on 2024-02-17; Adafruit Feather ESP32-S2 TFT with ESP32S2
Adafruit CircuitPython 9.0.0-beta.1 on 2024-02-17; Adafruit MatrixPortal S3 with ESP32S3

Code/REPL

import ssl
import socketpool
import wifi

from adafruit_httpserver import Server, Request, Response


class TLSServerSocketPool:
    def __init__(self, pool, ssl_context):
        self._pool = pool
        self._ssl_context = ssl_conte...
manic glacierBOT
orchid basinBOT
manic glacierBOT
manic glacierBOT
#
RetiredWizard
[adafruit/circuitpython] New comment on issue #8948: Enhancement: Support built-in keyboard in terminal on LilyGO T-Deck.

This was discussed in #8684 but the original poster closed it. Tannewt did make a comment suggesting it might be in the future though, so maybe this issue (or a re-opened #8684) could be marked long-term.

The python scripts from tdeck_repl mentioned in #8684 (or PyDOS) both require a serial terminal to initially set up but once installed get you a standalone python T-Deck device.

short tendon
crimson ferry
#

@short tendon seems sensible, I think it's similar to what requests did before (though I get confused often when trying to read the library). I'm no expert though... I know some basics on how to use sockets in CP, but not much about the next layer down or how we compare to broader CPython context

short tendon
#

It is similar, but the code did some strange things (would try once if nothing had been opened before, try five times if it had). Just trying to think if there would be an exception that would be worth trying again regardless

crimson ferry
#

there's a scenario I encounter often, but it may not be common... on many devices, I disconnect wifi between network accesses, so requests always fails on the first try (sometimes espidf: MemoryError even though there is tons of esp-idf and heap, or TLS error trying to reuse the old socket I think), then re-establishes socket with TLS on the second or so try, takes a little longer

#

(OSError: Unhandled ESP TLS error 78 0 8018 -78 in _send)

#

(espidf.MemoryError: in _get_socket)

#

ultimately it almost always works eventually (wouldn't even know it was happening but I run an instrumented version of adafruit_requests)

manic glacierBOT
short tendon
#

How have you changed Requests to get better errors?

crimson ferry
#

not really, mostly just added print statements, incl. printing exceptions that get passed

#

i think the reason it’s done is that in the typical case, the same API(s) are accessed repeatedly, so the socket is kept around to avoid the extra time of getting a new socket from scratch

short tendon
#

I did tweak the error returned to be:

raise RuntimeError(f"Error connecting socket: {last_exc}") from last_exc

So you still get the double stack trace, but the error is also there right on the bottom

manic glacierBOT
tulip sleet
#

@wraith crow hi, how are you determining the CIRCUITPY size? I think there are two problems here: 1. MEMENTO should have been built with CIRCUITPY_DUALBANK=0, and 2. os.statvfs() and friends is returning bad values, esp for filesystem size

wraith crow
#

df -h from host, but I think I'm also running into some fs corruptions which confuses things, I just erase_flash'd and am starting again from scratch 😄

tulip sleet
#

another thing is that the special partition arrangement for MEMENTO grows the firmware partition, NOT the CIRCUITPY partition. I don't even think we need the larger firmware partition. So that should probably just go away, though it's hardwired into the UF2 bootloader :/

#

then we can rely on the automatic use of the OTA1 partition for making a larger CIRCUITPY

#

my df -h says 3.7MB, which is just wrong

#

Also the cluster size is 1kB, which I think is wrong. It looks like there are bad calcuations in extmod/vfs_fat.c

# 
Adafruit CircuitPython 9.0.0-beta.1-1-g3871899860-dirty on 2024-02-18; Adafruit Camera with ESP32S3
>>> import os
>>> os.statvfs('/')
(1024, 1024, 3753, 3610, 3610, 0, 0, 0, 0, 255)
#

same results when querying from the host:

Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.statvfs('/media/halbert/CIRCUITPY')
os.statvfs_result(f_bsize=1024, f_frsize=1024, f_blocks=3753, f_bfree=3610, f_bavail=3610, f_files=0, f_ffree=0, f_favail=0, f_flag=4102, f_namemax=1530)
#

so the host is using bad info when deciding how much to write

#

3753 is way too big

#

it should be 960 or so

#

that is the actual partition size

#

or 1920 if using 512-byte clusters

#

@mortal kernel relying on your FAT knowledge. The statvfs values above for bsize and frsize might be wrong? Should they be 512 each?

#

these statvfs values may have been wrong forever

#

3753/2 is about right for total clusters (blocks) for a 960kB filesystem

#

@wraith crow, yes, I started from a clean flash, reloaded the bootloader, etc. And after that I did storage.erase_filesystem() each time

wraith crow
#

I'll try and catch up/reproduce what you're getting but after my erase_flash/start over the firmware is no longer finding the circuitpy drive and booting to safe mode.

mortal kernel
wraith crow
#

Are you using Adafruit_MEMENTO_Factory_Reset.bin to put the bootloader back on?

tulip sleet
#

@onyx hinge do you remember why we went to a no-OTA partition scheme on MEMENTO? I thought it was to get a larger CIRCUITPY (a long time ago), but it doubles firmware space, not CIRCUITPY.

onyx hinge
#

no it was to get firmware space

wraith crow
#

Since for OTA to work you need a second copy isn't "no-OTA" effectively the same as "1-OTA"?

onyx hinge
# 
switch camera to use the no-ota partition scheme

this gives us a comfortable few hundred kB flash space left for the app

1398912 bytes used,  763776 bytes free in flash firmware space out of 2162688 bytes (2112.0kB).

(commit message on c516870f89ffa82b33e52ade0e96baff5e31cc85)

tulip sleet
#

so I made them consistent with tinyuf2

mortal kernel
tulip sleet
#

so what's OK, 1024, or should be 512?

#

for example a MicroPython PyBoard reports 512 for both values

#

for its filesystem

mortal kernel
#

oofatfs makes the call re: cluster size in mkfs. The numbers it picks can be janky for boundary cases, but nothing breaks.

tulip sleet
#

@onyx hinge current en_US build is:

1286048 bytes used, 1597536 bytes free in flash firmware space out of 2883584 bytes (2816.0kB).
tulip sleet
# mortal kernel `oofatfs` makes the call re: cluster size in `mkfs`. The numbers it picks can be...

This is code in extmod/vfs_fat.c:

    t->items[0] = MP_OBJ_NEW_SMALL_INT(fatfs->csize * SECSIZE(fatfs)); // f_bsize
    t->items[1] = t->items[0]; // f_frsize
    t->items[2] = MP_OBJ_NEW_SMALL_INT((fatfs->n_fatent - 2)); // f_blocks
    t->items[3] = MP_OBJ_NEW_SMALL_INT(nclst); // f_bfree
    t->items[4] = t->items[3]; // f_bavail
    t->items[5] = MP_OBJ_NEW_SMALL_INT(0); // f_files
    t->items[6] = MP_OBJ_NEW_SMALL_INT(0); // f_ffree
    t->items[7] = MP_OBJ_NEW_SMALL_INT(0); // f_favail
    t->items[8] = MP_OBJ_NEW_SMALL_INT(0); // f_flags
    t->items[9] = MP_OBJ_NEW_SMALL_INT(FF_MAX_LFN); // f_namemax
#

csize I guess is cluster size, but it may be multiplying by two. f_blocks would then appear to be wrong

mortal kernel
#

Just the code I was looking at. fatfs->csize is number of physical blocks (sectors) in a cluster.

tulip sleet
#

ok, I have it backwards

#

the filesystem is not 3.7MB, not even close (as reported by df -h, which uses statvfs(), I assume

mortal kernel
#

Ah, now that's interesting. If the filesystem size is getting misreported by ioctl when oofatfs creates it, then that could happen.

#

The filesystem would behave normally until its physical area is exceeded as the FAT fills, and then bad things will happen.

tulip sleet
#

yes, it is getting misreported up to the host when a statvfs happens. Same (wrong, I think calculation) is in ports/espressif/supervisor/internal_flash.c:

STATIC DWORD fatfs_bytes(void) {
    fs_user_mount_t *fs_mount = filesystem_circuitpy();
    FATFS *fatfs = &fs_mount->fatfs;
    return (fatfs->csize * SECSIZE(fatfs)) * (fatfs->n_fatent - 2);
}
#

that is used only in supervisor_flash_update_extended(), though

#

compare with flash_ioctl() in supervisor/shared/flash.c, which uses flash_get_block_count() for IOCTL_BLOCK_COUNT

mortal kernel
#

There's some messy code in oofatfs that decides whether to use the MBR or ioctl calls to determine the partition and therefore the filesystem size.

tulip sleet
#

all this inconsistency might explain multiple storage problems we have been seeing.

mortal kernel
#

I'm pretty familiar with the code involved. What would I need to reproduce it here?

tulip sleet
#

I think you don't have a MEMENTO board, right?

mortal kernel
#

Don't have one. I do have some ESP boards.