Building CircuitPython on Local Machine

[crimson flare]

Thanks @compact bison , @oak elk, and @marsh finch for helping me build CircuitPython on my local machine. The Adafruit build guides, especially the Building CircuitPython (https://learn.adafruit.com/building-circuitpython) and Espressif Builds (https://learn.adafruit.com/building-circuitpython/espressif-build), have been incredibly helpful in explaining each step of the process. With your guidance, I was able to identify and resolve two issues in my recent builds, summarized below.

Environment:
I am using Ubuntu 24.04LTS with python 3.12.3, git 2.49.0, arm-gnu-toolchain-14.2.rel1-x86_64-arm-none-eabi, ninja-build 1.11.1, cmake 3.28.3, and C compiler GNU 13.2.0.
I tested my builds in version 10.0.0-Alpha.7, 10.0.0-Alpha.2, 9.2.8, 9.2.7, and 9.2.4 for my Adafruit boards which includes 1. Feather_esp32_v2, 2. Feather_esp32s3_4mbflash_2mbpsram, 3. Feather_esp32s3_tft, and 4. Qtpy_esp32s3_4mbflash_2mbpsram.

Issue #1: The undesirable '--upgrade' flag?
I followed the Espressif Builds guide meticulously, which meant including the '--upgrade' flag when installing Python packages in the ESP-IDF venv. This worked at first because the submodules (e.g., lib_esp32, lib_esp32c3_family) and tools (e.g., setuptools, cryptography, click) were in sync with the latest release. However, when I ran the same build commands weeks later, I encountered errors due to version mismatches. The '--upgrade' flag was pulling newer versions of tools and submodules that were incompatible with the release I was building. I noticed Todbot’s build instructions omit the '--upgrade' flag. After removing it from my workflow, all build errors disappeared.

Suggestion: Could you review the Espressif Builds guide to clarify whether the '--upgrade' flag is necessary in this step?

Issue #2: Removing Submodules When Switching Versions
When building an older version, I followed the Building CircuitPython guide and used git checkout to switch to a specific release--for example, git checkout 9.2.4. I was familiar with the 'remove-all-submodules' and 'make fetch-port-submodules' commands described in the 'Fetch Submodules' section. However, I didn't realize removing submodules was necessary when switching between versions. I assumed the git checkout to a specific version and then running the 'make fetch-port-submodules' command again was sufficient, especially since there were no errors when fetching the espressif submodules. Previously, I had occasionally built older versions without issues—likely because I started from a fresh clone of circuitpython and checked out to an older release directly, unknowingly avoided having to clean up submodules. Thanks to DanH who alerted me to remove all submodules at the top level. Now, I understand the need to remove all submodules for different build versions. After adding 'make remove-all-submodules' to my build process, I've been able to build any version of CircuitPython without any issue.

Suggestion: Would you consider updating the "Fetch Submodules" section of the Building CircuitPython guide to emphasize the importance of removing all submodules when switching to different build versions? I understand there’s no make remove-port-submodules, as noted in the guide.

Thank you very much!

These are my workflows:

---

Building the Latest CircuitPython Release:

1. git clone https://github.com/adafruit/circuitpython.git
2. cd circuitpython
3. make fetch-tags
4. source ./.venv/bin/activate
5. pip3 install --upgrade -r requirements-dev.txt
6. deactivate
7. cd ports/espressif
8. make fetch-port-submodules
9. ./esp-idf/install.sh
10. source ./esp-idf/export.sh
11. pip3 install -r ../../requirements-dev.txt
12. make BOARD=adafruit_feather_esp32s3_4mbflash_2mbpsram

---

How to Switch and Build a Different CircuitPython Version:

1. cd circuitpython
2. make fetch-tags
3. git status # (check current build version)
4. git checkout 9.2.4 # (example 9.2.4)
5. git status # (check that it has been switched)
6. make remove-all-submodules
7. source ./.venv/bin/activate
8. pip3 install --upgrade -r requirements-dev.txt
9. deactivate
10. cd ports/espressif
11. make fetch-port-submodules
12. ./esp-idf/install.sh
13. source ./esp-idf/export.sh
14. pip3 install -r ../../requirements-dev.txt
15. make BOARD=adafruit_feather_esp32s3_4mbflash_2mbpsram

---

Building CircuitPython for Different Boards:

1. cd circuitpython/ports/espressif
2. source ./esp-idf/export.sh
3. make BOARD=adafruit_feather_esp32s3_tft
4. make BOARD=adafruit_qtpy_esp32s3_4mbflash_2mbpsram

[marsh finch]

Thanks for all these notes! I will be updating the guide. There are so many things one must get right.