RAM-based storage in GitHub Actions | PyPA | Page 1

lament forum Dec 25, 2024, 7:32 PM

#

I was curious what GitHub Copilot can do, so after some googling around, I threw in a text prompt on the web UI and... it gave me something that I don't trust but looks fine as a starting point -- I don't know how I feel about it downloading a random binary from the internet but that can be validated with a checksum... This is probably fine for getting started for a prototyping-ish run.

name: Create RAM Disk on Windows

on: [push]

jobs:
  create-ramdisk:
    runs-on: windows-latest

    steps:
    - name: Checkout repository
      uses: actions/checkout@v3

    - name: Set up RAM disk
      run: |
        # Download and install ImDisk
        wget https://mirrors.kernel.org/sourceforge/imdisk/ImDiskTk-x64.zip -O imdisk.zip
        Expand-Archive -Path imdisk.zip -DestinationPath imdisk
        Start-Process -FilePath ".\imdisk\install.bat" -ArgumentList "/S" -Wait

        # Create a 1GB RAM disk
        Start-Process -FilePath "imdisk\imdisk.exe" -ArgumentList "-a -t vm -s 1G -m Z: -p \"/fs:ntfs /q /y\"" -Wait

    - name: Verify RAM disk
      run: |
        fsutil volume diskfree Z:

    - name: Use RAM disk 
      run: |
        # Example: Copy repository files to RAM disk
        xcopy . Z:\ /E /H /C /I

    - name: Cleanup RAM disk
      if: always()
      run: |
        # Remove the RAM disk
        Start-Process -FilePath "imdisk\imdisk.exe" -ArgumentList "-D -m Z:" -Wait

gritty mulch Dec 25, 2024, 7:33 PM

#

see, I refuse to even use AI in my personal life.

lament forum Dec 25, 2024, 7:33 PM

#

Well, you didn't! 😂

gritty mulch Dec 25, 2024, 7:33 PM

#

This is not at all dodgy :P I'll take a look, but I am a little wary of depending on an even more complex script or a 3rd-party action/script.

#

It better like make the runtimes 3 minutes 🏎️

lament forum Dec 25, 2024, 7:34 PM

#

gritty mulch It better like make the runtimes 3 minutes 🏎️

Yea... I'd love that but I doubt that.

#

IIRC, filesystem overhead on Windows is a lot higher.

gritty mulch Dec 25, 2024, 7:35 PM

#

I'm being tongue in check, haha

lament forum Dec 25, 2024, 7:36 PM

#

FWIW, I also have a somewhat "ugh" feeling about this. Not helped by the fact that the code block uses checkout v3.

gritty mulch Dec 25, 2024, 7:37 PM

#

lament forum Yea... I'd love that but I doubt that.

It'd be amazing to have a CI turnaround time of 10 minutes, but even 15 minutes would be nice.

lament forum Dec 25, 2024, 7:37 PM

#

Yea.

gritty mulch Dec 25, 2024, 7:37 PM

#

We're sitting at 18-20 AFAIK.

lament forum Dec 25, 2024, 7:39 PM

#

I deal with pip-like CI times at work (and worse) and one of my favourite improvements is always making things quicker... Or when I hop on a project where I'm the primary maintainer and the test suite takes 12s to run 17295 tests or so.

gritty mulch Dec 25, 2024, 7:40 PM

#

lament forum FWIW, I also have a somewhat "ugh" feeling about this. Not helped by the fact th...

I'm going to try using the old ramdisk script we used to have, see how it blows up and then work backwards from that.

lament forum Dec 25, 2024, 7:40 PM

#

Honestly, I'm very happy we're at 20 minutes on pip and not the 60+ minutes we had when we were spread across Travis and Azure and GHA. 🫠

lament forum Dec 25, 2024, 7:41 PM

#

gritty mulch I'm going to try using the old ramdisk script we used to have, see how it blows ...

IIRC, it was basically a no-op by the time we removed it but I don't remember all the details. It'll be in the history under tools + there's probably some context from a past me in the PR.

#

(I hope so)

gritty mulch Dec 25, 2024, 7:42 PM

#

like no-op as in it didn't help CI times? or it stopped working entirely?

#

The context is here: https://github.com/pypa/pip/pull/12115

lament forum Dec 25, 2024, 7:42 PM

#

gritty mulch like no-op as in it didn't help CI times? or it stopped working entirely?

The first one, likely due to the latter that I tried investigating before going "I'm way out of my depth here".

#

IIRC

gritty mulch Dec 25, 2024, 7:43 PM

#

there seems to be very little difference in runtimes between the two, suggesting that the RAM disk isn't giving us much benefit. Given the amount of time (and computing resource) wasted in restarting CI jobs that fail in the "Create a RAM disk" step, I think it's better to just remove it.
from Paul

gritty mulch Dec 25, 2024, 7:43 PM

#

lament forum The first one, likely due to the latter that I tried investigating before going ...

That's my concern as well. The dev drive powershell script was already pushing my windows/PS knowledge to the limit, and any sort of ramdisk implementation would definitely be out of my wheelhouse.

lament forum Dec 25, 2024, 7:44 PM

#

gritty mulch > there seems to be very little difference in runtimes between the two, suggesti...

Aye, I misremembered! It wasn't me!

gritty mulch Dec 25, 2024, 7:44 PM

#

It's worth a shot, but I don't want to introduce the maintenance liability unless it's even faster.

young coyote Dec 25, 2024, 7:44 PM

#

I'll try and find the PR later, but it was turned off because it was causing issues

lament forum Dec 25, 2024, 7:45 PM

#

young coyote I'll try and find the PR later, but it was turned off because it was causing iss...

The PR is linked above.

#

But that... tracks. I think we had a meaningful speedups.

#

And, then it stopped working in some cases and stuff.

gritty mulch Dec 25, 2024, 7:48 PM

#

lament forum Honestly, I'm very happy we're at 20 minutes on pip and not the 60+ minutes we h...

Oh yeah, I've read those old issues. An hour sounds awful ducky_australia

lament forum Dec 25, 2024, 7:48 PM

#

2020 was an interesting time

gritty mulch Dec 25, 2024, 7:51 PM

#

Creating the ramdisk already takes 100 seconds, ouch. https://github.com/pypa/pip/actions/runs/12496073071/job/34867446618

lament forum Dec 25, 2024, 7:52 PM

#

Aye, I remember now. Github had started installing the server that we're installing at the top.

#

IIRC, that was the thing that took like a minute.

gritty mulch Dec 25, 2024, 7:53 PM

#

Ah, like it was pre-installed before?

lament forum Dec 25, 2024, 7:53 PM

#

We'd asked for it to be installed in the images by default.

#

And they removed it, if I had to guess.

lament forum Dec 25, 2024, 7:53 PM

#

lament forum We'd asked for it to be installed in the images by default.

They added that at the time.

gritty mulch Dec 25, 2024, 7:54 PM

#

A little off-topic, but the next windows and ubuntu images also remove svn so that'll be fun to deal with.

lament forum Dec 25, 2024, 7:54 PM

#

Yea, pip's VCS support situation is annoying.

gritty mulch Dec 25, 2024, 7:54 PM

#

(I'd tried to use Windows Server 2025 to see whether its more modern Dev Drive implementation would improve the times, it did not.)

lament forum Dec 25, 2024, 7:54 PM

#

Most of our CVEs are around the VCS stuff.

#

Which, me no likely.

#

But that's off topic as you said.

lament forum Dec 25, 2024, 7:56 PM

#

gritty mulch (I'd tried to use Windows Server 2025 to see whether its more modern Dev Drive i...

Optimistic, but a good try!

gritty mulch Dec 25, 2024, 7:57 PM

#

While I wait for the CI run to finish, the other ideas I had for speeding up the test suite was to

comb through the existing tests to identify redundant work that can be removed (like unnecessary wheel building)
caching HTTP requests globally (although honestly this is more important locally) where possible

lament forum Dec 25, 2024, 7:57 PM

#

gritty mulch While I wait for the CI run to finish, the other ideas I had for speeding up the...

I like the sound of it.

gritty mulch Dec 25, 2024, 7:58 PM

#

^ for the 2nd one. I was writing a test that needed to invoke a PEP 517 build and it was awfully slow even compared to the same test on the CLI. It turns out downloading setuptools ate 2 seconds.

lament forum Dec 25, 2024, 7:58 PM

#

Honestly, the test suite would benefit a lot from being fewer end to end tests but the ship sailed on that about 8+ years ago.

gritty mulch Dec 25, 2024, 7:59 PM

#

I have no appetite to rewrite pip's entire test suite 😅

lament forum Dec 25, 2024, 8:00 PM

#

gritty mulch I have no appetite to rewrite pip's entire test suite 😅

Sad. 😂

#

What do you mean you like being sane!?

#

thinks about all the "whyyyy" sounds I've made looking at the test suite code

gritty mulch Dec 25, 2024, 8:01 PM

#

I'd also like to prototype switching the provisioning mechanism for isolated environments to install deps in-process, as I suspect the major performance penalty from build isolation is a nontrival factor in slow tests.

lament forum Dec 25, 2024, 8:01 PM

#

Yep.

gritty mulch Dec 25, 2024, 8:01 PM

#

Of course, actually writing a good patch to do that is out my wheelhouse, but I'd like to prototype it at least.

lament forum Dec 25, 2024, 8:02 PM

#

The build isolation situation as a whole is like... a debt that we need to pay down interest and principal on.

gritty mulch Dec 25, 2024, 8:02 PM

#

And the interest is 30 APR.

lament forum Dec 25, 2024, 8:02 PM

#

gritty mulch Of course, actually writing a good patch to do that is out my wheelhouse, but I'...

Happy to chat about all things around that area!

gritty mulch Dec 25, 2024, 8:03 PM

#

I appreciate it!

#

Honestly, I feel partially disincentived to work on major changes like that though because I don't think we have enough review capacity for any of that.

lament forum Dec 25, 2024, 8:04 PM

#

I spent a lot of time on this a few years ago, before losing steam because the immediate environment around me wasn't conducive to OSS work and I had to start pretending to be an adult for reals. 😅

lament forum Dec 25, 2024, 8:04 PM

#

gritty mulch Honestly, I feel partially disincentived to work on major changes like that thou...

This is good to know, and I'm honestly not sure how to solve this issue at this point.

gritty mulch Dec 25, 2024, 8:05 PM

#

I know I'm supposed to be part of the solution, but there is so much historical context and codebase expertise that you need in order to review major PRs, so it'll be a good while before I feel comfortable reviewing such PRs.

#

This is why I stopped contributing to mypyc. TBF I was also out of my depth as I don't work on compilers in my own free time, but none of my work was receiving attention.

lament forum Dec 25, 2024, 8:07 PM

#

gritty mulch I know I'm supposed to be part of the solution, but there is so much historical ...

Yup yup! I'll try to set aside more time for OSS stuff next few months as a whole (I like the "set 3 month goals" thingie from some video by CGPGrey about theme system).

#

I hope that'll help!

gritty mulch Dec 25, 2024, 8:08 PM

#

I don't have any plans to stop contributing to pip, but I'm focusing on the parts where I'm not as reliant on others.

lament forum Dec 25, 2024, 8:08 PM

#

lament forum Yup yup! I'll try to set aside more time for OSS stuff next few months as a whol...

I have a lot of loose ends that need tying up, aside from the regular old "deal with the slow trickle of things".

gritty mulch Dec 25, 2024, 8:09 PM

#

lament forum Yup yup! I'll try to set aside more time for OSS stuff next few months as a whol...

Frankly, I'd prefer that we dedicate some time so @tiny yacht's thing with us is not wasted, but either way, sounds good!

lament forum Dec 25, 2024, 8:09 PM

#

That's another reason I'm motivated to make more time for this stuff. :)

#

Plus, I have a PEP that I'm supposed to be writing down that I'll throw up sometime after the holiday season.

#

Which is gonna be a fun time. 🙃

gritty mulch Dec 25, 2024, 8:10 PM

#

oof

#

I'm very glad that I don't participate in the standards discussion. I'm happy to stick in our own smaller pip world, where we generally don't have active fires in our discussions...?

lament forum Dec 25, 2024, 8:11 PM

#

Oh, this isn't gonna be standards.

lament forum Dec 25, 2024, 8:12 PM

#

gritty mulch I'm very glad that I don't participate in the standards discussion. I'm happy to...

Yea, though to be fair, some of us aren't around enough to have a fight/flame war. 😂

young coyote Dec 25, 2024, 8:13 PM

#

Standards discussions can be so tiring 😔, though I am going to try and put a PEP proposal together in January

lament forum Dec 25, 2024, 8:14 PM

#

young coyote Standards discussions can be so tiring 😔, though I am going to try and put a PE...

Yea, it's tricky to balance the needs of a lot of people, while trying to set a foundation to build upon in the future.

gritty mulch Dec 25, 2024, 8:14 PM

#

To bring it back on-topic (or off-topic since the topic changed?). The ramdisk seems to place in the middle: https://github.com/pypa/pip/actions/runs/12496073071/job/34867446550 test suite duration wise, and once you factor in the ramdisk creation time, it's not very competitive.

#

[3.13 (1)] mean: 0:11:15 min: 0:10:29
[3.13 (2)] mean: 0:10:39 min: 0:09:55
[3.8 (1)] mean: 0:17:37 min: 0:16:40
[3.8 (2)] mean: 0:14:59 min: 0:14:04

for reference, these are the averages from the last 50 runs on main.

lament forum Dec 25, 2024, 8:16 PM

#

Thanks for trying the old script again to validate that! It's probably worth trying imdisk since it was a consistent suggestion on a bunch of forums.

#

But, it's also 100% OK if you say that you don't wanna spend time on this. 😅

#

This was genuinely a "ooh, maybe this will help" that it seems did not help.

gritty mulch Dec 25, 2024, 8:18 PM

#

well I had a lot of those too

gritty mulch Dec 25, 2024, 8:18 PM

#

lament forum But, it's also 100% OK if you say that you don't wanna spend time on this. 😅

I think I'd like to PR the current improvements I have, and then we can investigate better ramdisk implementations later.

lament forum Dec 25, 2024, 8:19 PM

#

Sounds good to me!

gritty mulch Dec 25, 2024, 8:35 PM

#

And because I was bored, here's a scatter plot of the last 50 CI runs on main.

#

I really should add caching to this script, but I'll deal with that later.

#

I'm not sure why there is a giant gap given that there should be weekly CI runs, but /shrug.

#

Annnd the last 100 runs.

[3.8 (1)] mean: 0:17:35 min: 0:16:38
[3.8 (2)] mean: 0:15:00 min: 0:13:46
[3.13 (1)] mean: 0:11:15 min: 0:09:57
[3.13 (2)] mean: 0:10:44 min: 0:09:22

#

It'd be neat to add this to https://ichard26.github.io/ghstats/pip/ now I think about it

#

I'll add that to the "absolutely lowest priority to-do, maybe will be done at some point, probably not" list.

tiny yacht Dec 25, 2024, 9:11 PM

#

gritty mulch Frankly, I'd prefer that we dedicate some time so <@334562134274998273>'s thing ...

I appreciate that but I think I am mostly blocked on the actual purpose of pip and the current architecture https://github.com/pypa/pip/issues/13111

#

internally I've basically marked that as not going to happen (please correct me if you disagree)

gritty mulch Dec 27, 2024, 2:02 AM

#

tiny yacht I appreciate that but I think I am mostly blocked on the actual purpose of pip a...

yea, that seems about right

#RAM-based storage in GitHub Actions