#Failed to allocate guard page when spawning many threads

18 messages · Page 1 of 1 (latest)

balmy vector
#

I am attempting to allocate a large number of threads on an Ubuntu 22.04 system, but I am running into issues with spawning threads. I included the full error below, but the core of it seems to be failed to set up alternative stack guard page: Cannot allocate memory (os error 12). This is somewhat annoying since there should be plenty of address space on this x64 machine.

To try to diagnose the problem, I created this example which simply creates a ton of threads then exits. The error above was created from this code.

use std::thread::spawn;
use std::sync::Barrier;
use std::sync::Arc;

pub fn main() {
        let num_threads = 20000;  // Works up to about 16k threads
        let barrier = Arc::new(Barrier::new(num_threads));
        let mut threads = Vec::new();

        for _ in 0..num_threads {
                let barrier_clone = barrier.clone();
                threads.push(spawn(move || {
                        barrier_clone.wait();
                }));
        }

        for handle in threads {
                handle.join().unwrap();
        }

        println!("No issues!");
}

I ran this on an Ubuntu 22.04 r6g.medium AWS EC2 instance (1 vCPU, 8 GiB Memory).

#

The full output is a bit mangled:

thread 'thread 'thread '<unnamed>thread '' panicked at '<unnamed>failed to set up alternative stack guard page: Cannot allocate memory (os error 12)' panicked at '', thread 'failed to set up alternative stack guard page: Cannot allocate memory (os error 12)library/std/src/sys/unix/stack_overflow.rs<unnamed>', :' panicked at 'library/std/src/sys/unix/stack_overflow.rsthread '144failed to set up alternative stack guard page: Cannot allocate memory (os error 12)::', 144<unnamed>13library/std/src/sys/unix/stack_overflow.rs:thread '
' panicked at ':13failed to set up alternative stack guard page: Cannot allocate memory (os error 12)note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'fatal runtime error: <unnamed>144
', library/std/src/sys/unix/stack_overflow.rsfatal runtime error: :' panicked at ':13failed to set up alternative stack guard page: Cannot allocate memory (os error 12)<unnamed>144
', ' panicked at ':fatal runtime error: thread 'thread 'library/std/src/sys/unix/stack_overflow.rsfailed to set up alternative stack guard page: Cannot allocate memory (os error 12)13failed to initiate panic, error <unnamed>:
', 2015020144' panicked at '144library/std/src/sys/unix/stack_overflow.rsfatal runtime error:
failed to set up alternative stack guard page: Cannot allocate memory (os error 12)thread ':Aborted (core dumped)
#

Maybe I have the wrong idea, but I'm imagining each thread only needs a couple pages of memory within the x64 address space, but guard pages would not occupy any physical memory.

#

Running top with 16k threads running also appears to indicate there is plenty of memory left available:

top - 06:19:15 up  1:02,  3 users,  load average: 0.15, 20.59, 28.77
Tasks: 128 total,   1 running, 127 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.0 us,  1.4 sy,  0.0 ni, 98.6 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :   7818.8 total,   4190.8 free,    794.5 used,   2833.5 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.   6816.7 avail Mem
elfin turret
#

someone else hitting this on HN
https://news.ycombinator.com/item?id=25261494

nzentzis

I just tested this on my Linux workstation.2000 threads does nothing - everything's still responsive, and the process is shown as using 0% of the CPU.16,000 threads uses ~30% of a core, with a ~136MB RSS. The system still handles it fine, though, and everything stays responsive.At 20,000 the program panics when spawning threads, with the message...

iron lodge
balmy vector
iron lodge
#

It defaults to 65530 on most systems

iron lodge
#

assuming root

#

or vm.max_map_count=262144 in /etc/sysctl.conf plus sudo sysctl -p for a persistent one

#

I think systemd machine may have /etc/sysctl.conf named differently

balmy vector
# iron lodge `sysctl -w vm.max_map_count=262144 ` to temporarily increase it

That fixed that issue for 20k threads, but if I were to increase it to 21k I get the following error. I found this https://stackoverflow.com/q/62409492/5987669 which addresses the issue, but I am unsure how to apply their solution to Rust.

thread 'main' panicked at 'failed to spawn thread: Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }', /rustc/e092d0b6b43f2de967af0887873151bb1c0b18d3/library/std/src/thread/mod.rs:652:29
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
iron lodge
#

You can also set RUST_MIN_STACK when building to set the global size

#

You are really pushing the kernel to the limit with this many threads

balmy vector
balmy vector