Hello, there was a problem, the creation of artifacts never ends, the assembly itself takes a very long time, this has never happened before
#Release creation can't finish, long wait
long badge
long badge
wary oriole
Can you re-run with -v and share the output?
stoic cobalt
Thank you for filing.
The "Creating Artifacts..." step is uploading files to Shorebird's servers.
I wonder if it's having difficulty reaching shorebird's servers?
There is a lot more we could do here I'm sure.
long badge
There is access to the server, it seems that the server has started to sag when loading artifacts, 50 minutes for the release is somehow too much, I tried from different servers
Full logs, see long operations create artifact
wary oriole
There is access to the server, it seems that the server has started to sag when ...
What’s your download speed? We can take a closer look tomorrow 👍
long badge
long badge
The question is - is it just me or do everyone’s artifacts take a long time to load, with the release patch taking just as long?
long badge
What’s your download speed? We can take a closer look tomorrow 👍
What should I do, wait for a solution, I’m not the only one who has a problem with artifacts, it takes 1-1.5 hours to post any release or patch, it’s very tiring ?
wary oriole
What should I do, wait for a solution, I’m not the only one who has a problem wi...
One thing you could try in the meantime is to use a vpn and see if that improves the times. The storage bucket is us central atm
long badge
The problem is not in the network, direct access to the bucket occurs instantly, turning on the VPN does not fix the problem, it feels like the code is spinning in circles or there are constant retracements over the network
maybe it's not downloading in the right place, without a filter, I noticed my traffic is growing to 4-6 MB per second, the artifact takes up a couple of megabytes
long badge
Yes
bronze oriole
Let me try on my windows machine (we all use macs)
Which version of windows do you use?
long badge
11
bronze oriole
Hrmm, I'm not seeing this (albeit with a very simple app)
$ shorebird patch android
✓ Downloading patch... (0.1s)
✓ Extracting patch... (0.1s)
✓ Fetching apps (1.1s)
✓ Fetching releases (0.2s)
Which release would you like to patch?
Which release would you like to patch? 1.0.0+4
✓ Fetching aab artifact (0.2s)
✓ Downloading aab (2.1s)
✓ Downloading patch... (31ms)
✓ Extracting patch... (45ms)
✓ Building patch with Flutter 3.24.3 (5a6dc546e8) (6.6s)
✓ Verifying patch can be applied to release (28ms)
✓ Fetching release artifacts (0.6s)
✓ Downloading release artifacts (1.1s)
✓ Creating patch artifacts (1.5s)
🚀 Ready to publish a new patch!
📱 App: flutter_three_two_two_oh (1fd35586-2fe3-4983-9191-04c8cfa4491a)
📦 Release Version: 1.0.0+4
🕹️ Platform: android [arm32 (130 B), arm64 (122 B), x86_64 (122 B)]
🟢 Track: Production
Would you like to continue? (y/N) Yes
✓ Creating patch (1.9s)
✓ Uploading artifacts (1.3s)
✓ Fetching channels (0.1s)
✓ Promoting patch to stable (0.2s)
✅ Published Patch 1!
It doesn't make a ton of sense that Dart would be substantially slower than curl (we aren't doing anything fancy)
Your release artifacts aren't exactly huge either (I'm seeing ~12MB)
The aabs are a bit larger (~57MB) but I still wouldn't expect that to be causing this
stoic cobalt
It must be hitting some exception and then not failing out correctly.
stoic cobalt
Full logs, see long operations create artifact
Looking now.
Wow, yeah 2024-10-08T11:02:24.074018 [HTTP] POST https://api.shorebird.dev/api/v1/apps/35e8245d-7939-44fa-9dab-b6a8e5344bca/releases/108680/artifacts is what took forever.
Wonder why 🤔
I think that's this line? https://github.com/shorebirdtech/shorebird/blob/a1dfc7d6bb8a2c425998eb5decf36969ba814752/packages/shorebird_cli/lib/src/code_push_client_wrapper.dart#L496
Amusingly aabPath isn't actualy an aab I don't think?
bronze oriole
https://github.com/shorebirdtech/shorebird/blob/stable/packages/shorebird_code_push_client/lib/src/code_push_client.dart#L174-L224 is probably the more useful part
stoic cobalt
At least this isn't an aab: ```
2024-10-08T11:02:23.922086 Creating artifact for D:\GIT\Raptor\GIT\sekundant-mp\sekundant\build\app\intermediates\stripped_native_libs\release\stripReleaseDebugSymbols\out\lib\armeabi-v7a\libapp.so
pressumably it was the http.send which took forever? https://github.com/shorebirdtech/shorebird/blob/stable/packages/shorebird_code_push_client/lib/src/code_push_client.dart#L216C1-L217C1
bronze oriole
The download seems to be the slow operation in the initial screenshot
fwiw
Er, not initial, but this one
Which I believe is pulling an AAB
Which I think is ultimately calling https://github.com/shorebirdtech/shorebird/blob/stable/packages/shorebird_cli/lib/src/artifact_manager.dart#L55-L84
stoic cobalt
I'm reading line 79302 in their log.
Which seems to result in: ```
⠦ Creating artifacts... (3243.1s)
2024-10-08T11:56:27.092179
✓ Creating artifacts (3243.3s)
Either way, seems like networking issues of some form?
bronze oriole
Yep
@long badge I'm curious if there's somehow an http package issue. Would you be ok uninstalling and reinstalling shorebird to see if that fixes the issue?
stoic cobalt
Thanks for your help!
lilac linden
Here I am with the same situation...
bronze oriole
I'm noticing a commonality: this seems to be happening when working on the D:\ drive. Where do you have shorebird installed?
lilac linden
I'm noticing a commonality: this seems to be happening when working on the D:\ d...
C:
I just ran the script that is in the doc
wary oriole
Can you re-run with the --verbose flag and share the output?
lilac linden
✓ Building patch with Flutter 3.22.3 (19f36dba93) (114.0s)
✓ Verifying patch can be applied to release (0.2s)
[HTTP] GET https://api.shorebird.dev/api/v1/apps/7e2d79e3-ecb0-45e9-bc0d-2ec58b4248ca/releases/90630/artifacts?arch=arm&platform=android
⠏ Fetching release artifacts... (3.8s)[HTTP] GET https://api.shorebird.dev/api/v1/apps/7e2d79e3-ecb0-45e9-bc0d-2ec58bd
⠙ Fetching release artifacts... (6.5s)[HTTP] GET https://api.shorebird.dev/api/v1/apps/7e2d79e3-ecb0-45e9-bc0d-2ec58bd
✓ Fetching release artifacts (8.3s)
[HTTP] GET https://storage.googleapis.com/shorebird_release_artifacts/7e2d79e3-ecb0-45e9-bc0d-2ec58b4248ca%2Fandroid%2Farm%2F90630%2Flibapp.so?X-Goog-Algorithm=GOOG4-RSA-SHA256&X-Goog-Credential=patch-artifacts-storage-bucket%40code-push-prod.iam.gserviceaccount.com%2F20241010%2Fauto%2Fstorage%2Fgoog4_request&X-Goog-Date=20241010T183445Z&X-Goog-Expires=3600&X-Goog-SignedHeaders=host&X-Goog-Signature=6391b2f6be948198e9124ab4c927b5fd18ba4495922908c645a21b563e28e86bd86ceb58fd9b59728dc01baedd867e5866ecb9dd09489854f18f86f3369ffc25fe5786dde288272fe373efca0a45ea62f662e7e0cca595b4a9771fd68e90404dbd2dc747fcca04952340c697ecd1b96bd8b40f3c94d031eab398653ca6c5b9d7ac64f1a3875dbb9ef1e409dc716de4cdd3229c5e6f75076b65f380e214c3d7199f6474eb4daccafacd5bba42b1b820195d6ec30db29eb84eb35e77c5a3959cb022e5ff31912d245f14a63d6d48b95119776668c10fb45097891974c78319c55fb34088ebb17444e58f15ff1b7e3a2ef8c35e5d9d30c97be03e6d72f5b99d8c43
⠧ Downloading release artifacts... (31.3s)
all output
bronze oriole
That looks like it's still running? I'm not seeing the 400 Bad Request failure you included in your screenshot
lilac linden
timeout happened just now
long badge
Yep
Reinstall not work. Open url in browser, storage error, status 200, after a while the file is downloaded, processing and retry is necessary or the signature algorithm itself must be changed https://github.com/shorebirdtech/shorebird/blob/stable/packages/shorebird_cli/lib/src/artifact_manager.dart#L55-L84
long badge
crystal umbraBOT
long badge
after 3000 -4000 sec it completes successfully
lilac linden
Any feedback on this situation?
wary oriole
We’re looking into it still and will report back when we have an update. Apologies for the inconvenience!
stoic cobalt
Our current theory is that there is some sort of strangeness on GCP's side (since we haven't changed anything related to this in a very long time), but we're investigating. Thank you for the report and patience.
crystal umbraBOT
Any feedback on this situation?
crystal umbraBOT
Our current theory is that there is some sort of strangeness on GCP's side (sinc...
barren narwhal
有结果了吗？
long badge
Have you found a pattern for anyone experiencing this behavior or is it chaotic? Maybe you need to update from a specific region?
stoic cobalt
We believe something changed on GCP's side.
Felix is investigating right now.
It seems to be that GCP storage uploads are suddenly hanging for unknown reasons.
But we'll know more soon.
bronze oriole
https://github.com/shorebirdtech/shorebird/issues/2532 Filed this for tracking
bronze oriole
We just released a new version of the shorebird CLI with some improved diagnostics. When you have a second, could you upgrade and run shorebird doctor -v?
lilac linden
bronze oriole
Interesting! So your upload speed appears to be pretty good, but you're seeing the incredibly long wait times?
lilac linden
Yes, it is usually around 4000 seconds and fails returning a 400
wary oriole
Yes, it is usually around 4000 seconds and fails returning a 400
Can you share the full error?
lilac linden
all output
@wary oriole I'm still running shorebird patch android --verbose with the new version
I'm configuring WSL2 to try to run through it
long badge
In the new version the behavior has not changed, loading artifacts to the computer remains the same long, on average the publication of a patch is 1-3 hours, variable. Windows only behavior
Once again, the file via the link downloads quickly, in the browser, via the code it takes a very long time, there is a problem with signatures
It is interesting that the publication ends successfully, you have to wait a long time
lilac linden
all output new version
wary oriole
Can you upgrade to v1.4.2 and run shorebird doctor -v and share the output?
We just shipped some improvements that will measure download speed in addition to upload speed.
stoic cobalt
Presumably something else is going on if the test upload is working 🤔
wary oriole
Presumably something else is going on if the test upload is working 🤔
According to some of the logs above it’s the release artifact download that’s hanging. I’d be curious to have folks re-run shorebird doctor -v using v1.4.2
long badge
Windows
Linux
Windows After restarting the system, the situation has not changed.
After executing the script, you can delete the file, which means the system does not hold it
stoic cobalt
Thanks. That’s just a bug in our windows handling. We probably failed to close the file before deleting (which Linux won’t care about but windows will).
wary oriole
Yeah seems like a windows bug on our end. We’ll get a fix out on Monday
bronze oriole
The shorebird doctor issue should be fixed in the latest release
long badge
Thanks, everything works, release, patch can't download artifacts like before
quasi halo
@long badge few people in #💬│general said using git bash worked for them
bronze oriole
We just release 1.4.4, which shows download progress percentage while patching
I'd be curious to know those having this problem see
bronze oriole
Also, this has been fixed
bleak valley
I'd be curious to know those having this problem see
still the same. too slow in powershell
bronze oriole
So it looks like the download is happening, just suuuuuper slowly. Great. I've been able to reproduce on my windows laptop and hope to get to the bottom of this today
bleak valley
after 7500s, 89% downloaded in step 1/3 and just stopped downloading after that and i checked nothing changed and download was 89% in 8000s
bronze oriole
One thing I found that helped, which is absolutely not a long-term solution, is opening Task Manager, finding the dart.exe process that is shorebird, and changing its priority from normal to realtime
bleak valley
bleak valley
One thing I found that helped, which is absolutely not a long-term solution, is ...
also checked this too. dart.exe downloads like heart beat. in average, every 2 min download speed raise to 30mb/s and down to 0mb/s in a sec
and windows antimaleware exe was active and used cpu for all time
bleak valley
also checked this too. dart.exe downloads like heart beat. in average, every 2 m...
this is how downloading
bronze oriole
Disabling real-time protection also seems to help quite a bit, but that's definitely not a real solution
stoic cobalt
Is our current theory that this is just a windows bug?
bronze oriole
I believe so
bronze oriole
I don't think any of us have seen it in our regular development workflows, and given the number of people who seem to be running into it, I'd suspect we would have by now if it affected macs
frozen ibex
I have the same issue, it gets stuck at Uploading artifacts ... while doing shorebird release ios
Here is the result of shorebird doctor
Shorebird 1.4.13 • [email protected]:shorebirdtech/shorebird.git
Flutter 3.24.5 • revision e8206cf2e76e3a4e0c8e062c20bb563cae39a1f1
Engine • revision 9621213e02eade3f7e66d460688a313e2f6bac6eURL Reachability
✓ https://api.shorebird.dev OK (1.3s)
✓ https://console.shorebird.dev OK (1.0s)
✓ https://oauth2.googleapis.com OK (0.6s)
✓ https://storage.googleapis.com OK (0.6s)
✓ https://cdn.shorebird.cloud OK (0.4s)✓ Shorebird is up-to-date (2.9s)
✓ Flutter install is correct (5.3s)
✓ AndroidManifest.xml files contain INTERNET permission (0.3s)
✓ shorebird.yaml found in pubspec.yaml assets (21ms)No issues detected!
I using macOS
bronze oriole
This isn't the same issue (or at least, it has a different cause) if you're on mac
What does shorebird doctor -v show you? We run a basic speed test as part of that
magic bridge
this seems to still be an issue, I recently noticed that in my github actions workflow, building a patch can easily take over 10 minutes:
✓ Building patch with Flutter 3.29.0 (2e1bcd18e2) (640.7s)
The runner is using ubuntu-latest
wary oriole
Can you share the output of shorebird doctor -v in your CI? Also can you re-run the workflow with the -v flag and share the output?
stoic cobalt
I'm slightly lost. Sounds like this thread was originally the windows-defender-causes-slow-downloads issue, but is now builds-are-slow-on-github?
wary oriole
yeah this thread is a bit all over the place
stoic cobalt
Anyway, we're happy to help. We just get a bit overwhelmed with questions coming in in all directions at times. 🙂