Fixing UB in random crates | Rust Programming Language Community | Page 4

tough leaf Jun 14, 2022, 1:31 PM

#

i'll check that out

manic tangle Jun 14, 2022, 1:31 PM

#

in Drop?

tough leaf Jun 14, 2022, 1:32 PM

#

i'll make a PR

#

easier to show you there

#

you're allocating a Box<[Entry]> and then making a *mut [Entry] and then throwing away the length by casting it to *mut Entry

#

so when you Box::from_raw the *mut Entry, the layout is wrong unless the thread.bucket_size is 1

manic tangle Jun 14, 2022, 1:34 PM

#

but in drop it does ptr::from_raw_part_mut

#

oh wait

tough leaf Jun 14, 2022, 1:34 PM

#

this isn't in drop

manic tangle Jun 14, 2022, 1:34 PM

#

is this in the race to allocate

tough leaf Jun 14, 2022, 1:34 PM

#

yes

manic tangle Jun 14, 2022, 1:34 PM

#

that... would be my fault

tough leaf Jun 14, 2022, 1:34 PM

#

in get_or

#

does the pinned miri version not catch this?

#

hmmm

manic tangle Jun 14, 2022, 1:35 PM

#

okay it's not released yet in thread-local I don't think

#

why doesn't ci catch this though ferrisThonk

tough leaf Jun 14, 2022, 1:37 PM

#

fuck if i know

#

does old miri just not check this?

manic tangle Jun 14, 2022, 1:38 PM

#

miri from 6 months ago ferrisThonk

haughty mica Jun 14, 2022, 1:41 PM

#

thread-local is sketch

manic tangle Jun 14, 2022, 1:41 PM

#

how

haughty mica Jun 14, 2022, 1:41 PM

#

Oh that's just old version never mind

manic tangle Jun 14, 2022, 1:42 PM

#

the cve?

tough leaf Jun 14, 2022, 1:42 PM

#

yeah old miri just doesn't notice this

#

for some reason

haughty mica Jun 14, 2022, 1:42 PM

#

manic tangle the cve?

No, some crates use a version before 1.0

#

Old Miri does not have weak memory emulation or preemption

#

There's probably a futex bug in new Miri

manic tangle Jun 14, 2022, 1:43 PM

#

haughty mica Old Miri does not have weak memory emulation or preemption

this is dealloc with the wrong layout

#

oh wait

haughty mica Jun 14, 2022, 1:44 PM

#

Old Miri definitely checks for that, if it hits that code path

#

But just saying, the weak memory stuff literally has more code coverage

manic tangle Jun 14, 2022, 1:44 PM

#

that'a true the cas probably never fails without weak memory

#

or preemption

#

preemption not weak memory

tough leaf Jun 14, 2022, 1:45 PM

#

ohhh right

#

yes

#

that makes sense

haughty mica Jun 14, 2022, 1:45 PM

#

-Zmiri-preemption-rate=0

#

It's in the readme

manic tangle Jun 14, 2022, 1:45 PM

#

your pr fails cargo fmt lol

haughty mica Jun 14, 2022, 1:46 PM

#

The usual

#

Sorry to see you're not a sigma format on save enjoyer

tough leaf Jun 14, 2022, 1:47 PM

#

fuck

#

oh god what is that formatting

#

-   let to_free = std::ptr::slice_from_raw_parts_mut(new_bucket, thread.bucket_size);
+   let to_free =
+       std::ptr::slice_from_raw_parts_mut(new_bucket, thread.bucket_size);

haughty mica Jun 14, 2022, 1:48 PM

#

That's normal when you toggle the setting for it

tough leaf Jun 14, 2022, 1:48 PM

#

let _ = Box::from_raw(std::ptr::slice_from_raw_parts_mut(
    new_bucket,
    thread.bucket_size,
));

#

okay that's reasonable

manic tangle Jun 14, 2022, 1:49 PM

#

also remove std:: ferrisBorrowCheck

tough leaf Jun 14, 2022, 1:49 PM

#

oh is ptr imported

#

yes

#

there

manic tangle Jun 14, 2022, 1:50 PM

#

thanks

#

should I enable miri strict-provenance?

haughty mica Jun 14, 2022, 1:50 PM

#

If you want

#

I think it's mostly tag-raw-pointers with worse errors

#

That's partly my fault

tough leaf Jun 14, 2022, 2:32 PM

#

hmmmm
strict provenance fails
because it's saying something isn't a valid pointer

it's a union

#

is this
doing a ptr to int transmute?

manic tangle Jun 14, 2022, 2:37 PM

#

🤫

#

it might be doing a ptr to int transmute but only for comparisons

#

the invalid pointer issue is a different bug that I am fixing

west violet Jun 14, 2022, 2:58 PM

#

I've been thinking, maybe you should let contributors PR in miri results to a git repo that the website runs off of

#

Then you could automate the running part and have it produce a file/archive of the relevant results that they can then pr

manic tangle Jun 14, 2022, 3:09 PM

#

okay now miri is saying some weird things and I'm not sure if it's a weak memory emulation bug or not

haughty mica Jun 14, 2022, 4:28 PM

#

west violet I've been thinking, maybe you should let contributors PR in miri results to a gi...

This makes it challenging to verify that the results are accurate, that they're run with a correct config, etc

haughty mica Jun 14, 2022, 10:10 PM

#

Yay I found the UB in heapless that 0xide punted on fixing

#

Woooo

#

Go me

#

Yay

ruby jacinth Jun 14, 2022, 10:11 PM

#

u detectowed the ub ferrisHeartEyes

haughty mica Jun 15, 2022, 12:02 AM

#

I love how you can debug some of these just from the backtrace in gdb

#

#0  core::slice::raw::from_raw_parts::{closure#0}<u32> ()
    at /home/ben/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/intrinsics.rs:2009
#1  0x00005568b59e8e69 in core::ops::function::FnOnce::call_once<core::slice::raw::from_raw_parts::{closure_env#0}<u32>, ()> ()
    at /home/ben/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:248
#2  0x00005568b5a1baa7 in core::intrinsics::const_eval_select<(), fn(), core::slice::raw::from_raw_parts::{closure_env#0}<u32>, ()> (arg=(), _called_in_const=0x0, called_at_rt=...)
    at /home/ben/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/intrinsics.rs:2375
#3  0x00005568b591cf42 in core::slice::raw::from_raw_parts<u32> (data=0x7f25b0006593, len=0)
    at /home/ben/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/slice/raw.rs:93
#4  0x00005568b42c1e46 in parquet::util::hash_util::crc32_hash (bytes=..., seed=0) at src/util/hash_util.rs:115

#

This one is a 9.8 CVE ferrisClueless

#

Once again, easy from the backtrace

#0  core::slice::index::{impl#2}::get_unchecked::{closure#0}<u8> () at src/intrinsics.rs:2009
#1  0x00005623ca330885 in core::ops::function::FnOnce::call_once<core::slice::index::{impl#2}::get_unchecked::{closure_env#0}<u8>, ()> () at src/ops/function.rs:248
#2  0x00005623ca348e87 in core::intrinsics::const_eval_select<(), fn(), core::slice::index::{impl#2}::get_unchecked::{closure_env#0}<u8>, ()> (arg=(), _called_in_const=0x7fa8975fb620, called_at_rt=...) at src/intrinsics.rs:2375
#3  0x00005623ca357f37 in core::slice::index::{impl#2}::get_unchecked<u8> (self=0, slice=...) at src/slice/index.rs:225
#4  0x00005623ca358327 in core::slice::{impl#0}::get_unchecked<u8, usize> (self=..., index=0) at src/slice/mod.rs:405
#5  0x00005623ca1b882c in encode_unicode::utf8_char::Utf8Char::from_slice_start_unchecked (src=...) at src/utf8_char.rs:450

#

Wait a second actually maybe no

#

Should have trusted myself, I was right, this test is just some incredible facepalm material

#

#3  0x000055ef3792c877 in core::intrinsics::copy_nonoverlapping<u64> (src=0x7fbc74000d39, dst=0x7fbc7a9fd588, count=1)

facepalm

golden summit Jun 15, 2022, 12:15 AM

#

hey saethlin are you gonna test on windows after threading is merged, or are you gonna wait for file io too?

haughty mica Jun 15, 2022, 12:15 AM

#

Test on windows as a host?

#

Or as a target?

golden summit Jun 15, 2022, 12:15 AM

#

target

#

it would be nice to have something like the shim wishlist issue

haughty mica Jun 15, 2022, 12:16 AM

#

I'll do anything

golden summit Jun 15, 2022, 12:16 AM

#

but for windows

haughty mica Jun 15, 2022, 12:16 AM

#

Oh I see the angle

#

For sure

#

I'll run some stuff tomorrow

#

#3  0x0000557dc42fae22 in core::slice::raw::from_raw_parts_mut<u8> (data=0x0, len=0)

You absolutely do not love to see it

#

#3  0x00005572598b101c in core::intrinsics::copy<wasmer_vm::vmcontext::VMFunctionImport> (src=0x8, dst=0x7f5df8001b34, count=0)

#

wat

pastel lily Jun 15, 2022, 12:27 AM

#

Copying a zst “allocated” at dangling to stack?

haughty mica Jun 15, 2022, 12:28 AM

#

Copy 0 of them though? thonk

#

That's not a number of bytes that's a number of instances of T

#

Hit another cryptography library sigh

#

Oh this one is just an actual SIGILL, it's not one of my debug assertions firing

#

I've found one or two crates with bindings that seem to be just broken because they build but SIGILL or SIGSEGV

#

idk what's up with that tbh

hybrid forge Jun 16, 2022, 5:33 PM

#

Hello, is there any reason someone would use partially unitialised arrays over ArrayVec beside the extra dependency? Because that's what I see a lot in crates

tough leaf Jun 16, 2022, 5:34 PM

#

hybrid forge Hello, is there any reason someone would use partially unitialised arrays over `...

probably the extra dependency, yeah

#

what crates are doing this im gonna melobonk them

hybrid forge Jun 16, 2022, 5:36 PM

#

It's a weak reason to introduce unsafe code tbh, especially given the fact arrayvec is already in their dependency tree in most cases. I wonder if it wouldn't be better to point at it in rust docs too?

#

For some embed-targeted crates it's understandable (to avoid the dep) but that is seldom

haughty mica Jun 16, 2022, 6:37 PM

#

Where are you seeing this? An example would help

tough leaf Jun 16, 2022, 8:25 PM

#

i'm tempted to push for more mem::uninit panics tbh

now that http has been fixed

thorny venture Jun 16, 2022, 8:27 PM

#

Do ittt

thorny venture Jun 16, 2022, 8:28 PM

#

hybrid forge Hello, is there any reason someone would use partially unitialised arrays over `...

Someone should really try to get some array vec, or some generic storage trait into std so that people don't need to worry about dependencies

tough leaf Jun 16, 2022, 8:28 PM

#

generic storage
oh hi

#

it's you

#

:D

thorny venture Jun 16, 2022, 8:28 PM

#

Well, I first wrote an RFC for an array vec

#

I only became a storage shill after the controversy

#

https://github.com/rust-lang/rfcs/pull/3131#issuecomment-897084808

tough leaf Jun 16, 2022, 8:32 PM

#

one thing that might be a good idea is to submit API compatible versions of old crates

#

API and ideally MSRV

#

like, crossbeam-queue 0.1.2 is fucked

#

but if you run cargo update, you don't get a newer version

#

yeah so

game plan

go through the historical crater run and look at all failures
for all root causes, try and get a semver compatible but fixed crate released
fire off a crater run with the stricter checks and hope there's not 400 regressions

#

if a cargo update fixes it, I don't care

#

maybe straight up say "this panic seems to come from crossbeam-queue 0.1.2, you can run cargo update -p crossbeam-queue to update to a fixef version"

#

in the panic

haughty mica Jun 16, 2022, 9:31 PM

#

Phew

#

Good luck with that plan

#

As an aside I'm amazed how many people took my UAF fixes

tough leaf Jun 16, 2022, 10:00 PM

#

struct Inner<T> {
    bytes: [u8; 64],
                                                                       
    /// `[T; 0]` ensures alignment is at least that of `T`.
    /// `PhantomData<T>` signals that `CachePadded<T>` contains a `T`.
    _marker: ([T; 0], PhantomData<T>),
}
                                                                       
impl<T> Inner<T> {
    fn new(t: T) -> Inner<T> {
        assert!(mem::size_of::<T>() <= mem::size_of::<Self>());
        assert!(mem::align_of::<T>() <= mem::align_of::<Self>());
                                                                       
        unsafe {
            let mut inner: Self = mem::uninitialized();
            let p: *mut T = &mut *inner;
            ptr::write(p, t);
            inner
        }
    }
}

#

aaaaaaaa

#

why is this not just holding a T

#

to be fixed size?

haughty mica Jun 16, 2022, 10:02 PM

#

Where is this code?

#

crossbeam?

tough leaf Jun 16, 2022, 10:02 PM

#

crossbeam-utils

haughty mica Jun 16, 2022, 10:02 PM

#

Oh I see the problem

#

Also wtf

tough leaf Jun 16, 2022, 10:03 PM

#

https://github.com/crossbeam-rs/crossbeam-utils/blob/v0.2.2/src/cache_padded.rs

#

they have a MSRV of 1.12

#

at this point in time

#

oh right

#

it needs to be 64 bytes long

#

in newer versions of rust you can just repr(align) it

#

well

#

i could create Inner by zero initing bytes

#

that's still unsound if you have padding bytes in T

#

but honestly it's the best i can do
hmmm

#

this is the full list of regressions filtered by unique types

that's.... a lot less than i was expecting

📎 message.txt

#

so it looks like four root causes for zero init, 3 of them being crossbeam

haughty mica Jun 16, 2022, 10:21 PM

#

That does not mean your plan is easy

tough leaf Jun 16, 2022, 10:22 PM

#

oh no it's not
but it's doable

west violet Jun 16, 2022, 10:22 PM

#

tough leaf it needs to be 64 bytes long

struct Inner<T> {
    inner: T,
    padding: [u8; 64 - size_of::<T>()],
}

?

haughty mica Jun 16, 2022, 10:23 PM

#

Requires const fn

#

msrv too low

west violet Jun 16, 2022, 10:23 PM

#

Ahhh

tough leaf Jun 16, 2022, 10:23 PM

#

yeah the MSRV is far too low for that
the correct solution is just #[repr(align = 64)]

haughty mica Jun 16, 2022, 10:25 PM

#

I now know of 3 crates which pass overlapping ranges to copy_nonoverlapping

west violet Jun 16, 2022, 10:25 PM

#

lmao

tough leaf Jun 16, 2022, 10:26 PM

#

oh also would an abort in unreachable_unchecked be of any value or in practice is that just an abort anyways

haughty mica Jun 16, 2022, 10:27 PM

#

That would be useful

#

No need to take any chances

#

It's definitely a ud2 in debug mode already but why risk it

tough leaf Jun 16, 2022, 10:27 PM

#

that'd just be assert_unsafe_precondition!(false); i assume

haughty mica Jun 16, 2022, 10:28 PM

#

Eh

#

Or just call abort

tough leaf Jun 16, 2022, 10:36 PM

#

fair, yeah

haughty mica Jun 16, 2022, 10:53 PM

#

Lot of this stuff just hurts my brain

#

At least I've categorized all the SIGILL crates now

#

# use of abomonation
abomonation/0.7.3
abomonation_derive/0.5.0
differential-dataflow/0.12.0

# wild get-unchecked in a Map implementation?
heapless/0.7.13

# copy_nonoverlapping called on overlapping ranges
dmsort/1.0.1
lightws/0.6.4
symbolic-debuginfo/8.7.3 # dmsort

# null slice
rustler_sys/2.2.0
filebuffer/0.4.0
multiboot/0.8.0
spectrum-analyzer/1.2.4
rusty_v8/0.32.1
plotters/0.3.1
plotters-bitmap/0.3.1
tract-core/0.17.0
tract-hir/0.17.0
tract-linalg/0.17.0

# get_unchecked into an empty slice
compression/0.1.5
fallible_collections/0.4.4
garando_syntax/0.1.0
mp4parse/0.12.0
quickersort/3.0.1
slice-deque/0.3.0
stun_codec/0.2.0
stun_codec_blazh/0.1.13
swc_ecma_transforms_compat/0.104.1
swc_ecma_transforms_optimization/0.130.0
swc_ecma_transforms_testing/0.91.0
typed-index-collections/3.0.3
rav1e/0.5.1
secp256k1-plus/0.5.7 # arrayvec 0.3
polars/0.22.8
encode_unicode/0.3.6 # test-only and... why

# out-of-bounds get_unchecked, index = len
chess/3.2.0
grin_secp256k1zkp/0.7.11

# slice::from_raw_parts with misaligned pointer
commitlog/0.2.0
lua_bind_hash/1.0.1
malloc_buf/1.0.0
pelite/0.9.0
safe-transmute/0.11.2
parquet/16.0.0
datafusion/9.0.0

# Misaligned pointer passed to copy/copy_nonoverlapping
nanoserde/0.1.30
wasmer/2.3.0
wasmer-middlewares/2.3.0
as-ffi-bindings/0.2.4

# Requires a bunch of libraries, someone else can take this
opencv/0.65.0

west violet Jun 16, 2022, 11:33 PM

#

haughty mica It's definitely a ud2 in debug mode already but why risk it

It’s only a ud2 if it can’t be eliminated

haughty mica Jun 16, 2022, 11:33 PM

#

debug mode usually means no optimization

west violet Jun 16, 2022, 11:34 PM

#

haughty mica ``` # use of abomonation abomonation/0.7.3 abomonation_derive/0.5.0 differential...

timely and maybe some of the associated crates (eg timely_communication) also use abomination iirc

haughty mica Jun 16, 2022, 11:34 PM

#

I didn't hit them

#

They... Do not have much test coverage

#

That's probably why

west violet Jun 16, 2022, 11:35 PM

#

This is true

haughty mica Jun 16, 2022, 11:35 PM

#

It's very interesting that I only found one wild OOB slice access

west violet Jun 16, 2022, 11:36 PM

#

I’d imagine that’s something pretty easy to find in most scenarios

#

That is, it actually manifests as a bug

haughty mica Jun 16, 2022, 11:36 PM

#

slice access

#

The memory access would be valid if they didn't go through get_unchecked

west violet Jun 16, 2022, 11:37 PM

#

Huh?

haughty mica Jun 16, 2022, 11:37 PM

#

get_unchecked is UB to index out of bounds, even if there is memory out there

west violet Jun 16, 2022, 11:38 PM

#

Oh I see

haughty mica Jun 16, 2022, 11:38 PM

#

Some codebases do vec[vec.len()..].get_unchecked(0) to get the first uninit element for example

haughty mica Jun 17, 2022, 12:14 AM

#

Oh font-kit is a servo crate. Nice

#

You know what, I should just try building servo

#

I'm sure Firefox would be much more exciting

pastel lily Jun 17, 2022, 12:19 AM

#

hahah

haughty mica Jun 17, 2022, 12:22 AM

#

diff --git a/rust-toolchain b/rust-toolchain
index abcacd9bd1..bf867e0ae5 100644
--- a/rust-toolchain
+++ b/rust-toolchain
@@ -1 +1 @@
-nightly-2022-04-11
+nightly

#

That toolchain predates the debug assertions

tender nimbus Jun 17, 2022, 4:04 AM

#

haughty mica ``` # use of abomonation abomonation/0.7.3 abomonation_derive/0.5.0 differential...

I like how you put abomination into its own category

haughty mica Jun 17, 2022, 4:06 AM

#

🤷

tender nimbus Jun 17, 2022, 4:06 AM

#

corro

hybrid forge Jun 17, 2022, 2:19 PM

#

tough leaf <https://github.com/crossbeam-rs/crossbeam-utils/blob/v0.2.2/src/cache_padded.rs...

Since the repo has been archived for 4 years not sure if they will even accept the patchs

tough leaf Jun 17, 2022, 2:35 PM

#

also https://github.com/rust-lang/rust/issues/66151

GitHub

Tracking issue for panics in mem::uninitialized/zeroed · Issue #661...

With #66059, mem::uninitialized and mem::zeroed dynamically detect some misuses (such as mem::zeroed::<&T>() or mem::uninitialized::<bool>()) and panic instead o...

#

added a list of all the bad crates there

west violet Jun 17, 2022, 2:51 PM

#

hybrid forge Since the repo has been archived for 4 years not sure if they will even accept t...

iirc it’s been brought into crossbeam proper

haughty mica Jun 17, 2022, 4:19 PM

#

The issue with all of these is convincing maintainers to release a patch for an ancient version

west violet Jun 17, 2022, 4:19 PM

#

Or yank them

haughty mica Jun 17, 2022, 4:20 PM

#

That's even less of a fix

west violet Jun 17, 2022, 4:20 PM

#

Fair

knotty oar Jun 17, 2022, 4:28 PM

#

only supported versions though?

haughty mica Jun 17, 2022, 4:31 PM

#

What does supported mean?

knotty oar Jun 17, 2022, 4:35 PM

#

depends on what the author defines - i nmost cases just the latest major version (and all its minor ones)

haughty mica Jun 17, 2022, 4:39 PM

#

Yeah so that's my concern with 522's plan

tender nimbus Jun 17, 2022, 4:50 PM

#

"we want to break those crates" could be a convincing argument for a maintainer to backport (even more so if we deliver branches with the backports)

OTOH, "don't" could be the response to such a request

tough leaf Jun 17, 2022, 4:53 PM

#

true
i'll let llvm updates break the code instead

west violet Jun 17, 2022, 4:58 PM

#

Writes pass targeting this code

tough leaf Jun 17, 2022, 5:02 PM

#

honestly depending how aggressive llvm (and us) gets with noundef i wouldn't be surprised if even http's use breaks

#

(now patched, but it's been UB for like.... ever)

tender nimbus Jun 17, 2022, 5:13 PM

#

only took a few pings to get it fixed ferrisballSweat

haughty mica Jun 17, 2022, 6:01 PM

#

Get what fixed?

#

http?

thorny venture Jun 17, 2022, 6:03 PM

#

Rust

tender nimbus Jun 17, 2022, 6:08 PM

#

haughty mica `http`?

yes

tender nimbus Jun 17, 2022, 6:08 PM

#

thorny venture Rust

that needs a few more pings

#

vvv like this one

#

@thorny venture fix rust

thorny venture Jun 17, 2022, 6:09 PM

#

Done

thorny venture Jun 17, 2022, 6:09 PM

#

tender nimbus yes

Can I make it as audited and safe then? ferrisPeekPing

tender nimbus Jun 17, 2022, 6:10 PM

#

https safety could still be improved by the 3 open PRs but at least it's not about soundness anymore

haughty mica Jun 17, 2022, 6:33 PM

#

Oh really?

#

Doesn't it still do & -> &mut

tender nimbus Jun 17, 2022, 6:39 PM

#

ferrisThink

haughty mica Jun 17, 2022, 6:42 PM

#

It has a single Iter type which powers both shared and mutable iteration. You can't do that soundly.

tender nimbus Jun 17, 2022, 6:43 PM

#

uh

#

for the header map?

#

that one has IterMut, Iter and IntoIter

#

according to git blame, for years

haughty mica Jun 17, 2022, 6:49 PM

#

#[derive(Debug)]
pub struct Iter<'a, T> {
    inner: IterMut<'a, T>,
}

#

That cannot work

tender nimbus Jun 17, 2022, 6:51 PM

#

oohh

#

ferrisballSweat

#

is there a fix pr for this ws well

haughty mica Jun 17, 2022, 6:51 PM

#

I do not recall

#

The fix is pretty easy if you want to code it up, it just looks like awful programming unless you understand SB

#

It's really just copy+paste another definition of IterMut over Iter then duplicate all its methods
Possibly a macro would be nice

#

I definitely implemented this at least once just to convince myself that was the problem

#

Because you never know if you understand a bug until you've fixed it, you know

thorny venture Jun 17, 2022, 6:53 PM

#

Why not IterPtr then turn into &mut or & respectively

tough leaf Jun 17, 2022, 6:54 PM

#

you can't just clueless yourself

thorny venture Jun 17, 2022, 6:54 PM

#

Fr though I'm saddened by how complex the iterator trait is for efficiency

#

Specialisation pls

pastel lily Jun 17, 2022, 6:57 PM

#

ferrisClueless

tender nimbus Jun 17, 2022, 6:57 PM

#

ferrisClueless

pastel lily Jun 17, 2022, 6:57 PM

#

you can always clueless yourself

tender nimbus Jun 17, 2022, 6:57 PM

#

thorny venture Specialisation pls

while we're at it, why not just add lifetime specialization as-is
I've heard it's useful

#

ferrisClueless

haughty mica Jun 17, 2022, 7:38 PM

#

golden summit it would be nice to have something like the shim wishlist issue

Updated original comment

#

It feels like somehow the Windows stuff is running slower. That doesn't make any sense

#

Maybe it's just that I'm back at the top of the list where things have more tests

golden summit Jun 17, 2022, 7:39 PM

#

huh.

#

createthread aint even on the list

haughty mica Jun 17, 2022, 7:40 PM

#

The list goes on

#

Oh

golden summit Jun 17, 2022, 7:40 PM

#

theres a lot of file system stuff

haughty mica Jun 17, 2022, 7:40 PM

#

I run things with --jobs=1 --test-threads=1

golden summit Jun 17, 2022, 7:40 PM

#

GetFullPathNameW is painful to implement, the last time i tried it

haughty mica Jun 17, 2022, 7:41 PM

#

And I think somehow Miri ends up running tests serially anyway

#

I should really raise the parallelism to 2 or something, to make it easier to find races

golden summit Jun 17, 2022, 7:41 PM

#

golden summit GetFullPathNameW is painful to implement, the last time i tried it

because of the \\?\ stuff

#

if i remember right

#

CreateIoCompletionPort is probably tokio

#

i think it would be funny tho if i got async file io in miri on windows before linux

haughty mica Jun 17, 2022, 7:43 PM

#

Yeah tokio just dominates everything

golden summit Jun 17, 2022, 7:44 PM

#

but also that sounds incredibly painful to implement

haughty mica Jun 17, 2022, 7:45 PM

#

Yes

#

Could you just return an error from the case you don't want to handle?

golden summit Jun 17, 2022, 7:57 PM

#

oh yeah i remembered why GetFullPathNameW is up there so high

#

std calls it every time you need to use a path for anything

#

so all file io stuff is being covered up by it

#

except CreateIoCompletionPort

#

somehow

#

well thats probably being manually done by tokio

#

oh you dont actually need a file for CreateIoCompletionPort

west violet Jun 17, 2022, 8:11 PM

#

Yah, ports are io objects, they're not files

haughty mica Jun 18, 2022, 12:12 AM

#

3284 counts
(  1)     1684 (51.3%, 51.3%):  GetFullPathNameW
(  2)      633 (19.3%, 70.6%):  CreateIoCompletionPort
(  3)      304 ( 9.3%, 79.8%):  GetFileInformationByHandleEx
(  4)      195 ( 5.9%, 85.7%):  GetModuleFileNameW
(  5)       60 ( 1.8%, 87.6%):  WSAStartup
(  6)       46 ( 1.4%, 89.0%):  SleepConditionVariableSRW
(  7)       27 ( 0.8%, 89.8%):  CompareStringOrdinal
(  8)       25 ( 0.8%, 90.6%):  GetTickCount64
(  9)       19 ( 0.6%, 91.1%):  CreateFileW
( 10)       18 ( 0.5%, 91.7%):  AcquireCredentialsHandleA
( 11)       17 ( 0.5%, 92.2%):  Sleep
( 12)       15 ( 0.5%, 92.7%):  CreateNamedPipeW
( 13)       13 ( 0.4%, 93.1%):  CryptAcquireContextW
( 14)       13 ( 0.4%, 93.5%):  WakeAllConditionVariable
( 15)       13 ( 0.4%, 93.8%):  log1p
( 16)       12 ( 0.4%, 94.2%):  CreatePipe
( 17)       11 ( 0.3%, 94.5%):  CreateSemaphoreA
( 18)       10 ( 0.3%, 94.9%):  FileTimeToSystemTime
( 19)       10 ( 0.3%, 95.2%):  SHGetKnownFolderPath
( 20)       10 ( 0.3%, 95.5%):  SetThreadErrorMode
( 21)       10 ( 0.3%, 95.8%):  zx_vmo_create
( 22)        8 ( 0.2%, 96.0%):  GetCurrentProcess
( 23)        8 ( 0.2%, 96.3%):  GetModuleHandleA
( 24)        7 ( 0.2%, 96.5%):  GetProcessHeap
( 25)        7 ( 0.2%, 96.7%):  LocalAlloc
( 26)        7 ( 0.2%, 96.9%):  SystemTimeToTzSpecificLocalTime
( 27)        5 ( 0.2%, 97.0%):  AddVectoredExceptionHandler
( 28)        5 ( 0.2%, 97.2%):  GetComputerNameExW
( 29)        5 ( 0.2%, 97.4%):  llvm.x86.sse.cmp.ps
( 30)        5 ( 0.2%, 97.5%):  signal
( 31)        5 ( 0.2%, 97.7%):  zx_channel_create
( 32)        4 ( 0.1%, 97.8%):  CertCreateCertificateContext
( 33)        4 ( 0.1%, 97.9%):  CertOpenStore
( 34)        4 ( 0.1%, 98.0%):  GetAdaptersAddresses
( 35)        4 ( 0.1%, 98.1%):  GetBinaryTypeW
( 36)        4 ( 0.1%, 98.3%):  llvm.x86.sse.sqrt.ss
( 37)        4 ( 0.1%, 98.4%):  zx_cprng_draw
( 38)        4 ( 0.1%, 98.5%):  zx_time_get
( 39)        3 ( 0.1%, 98.6%):  llvm.x86.aesni.aeskeygenassist
( 40)        3 ( 0.1%, 98.7%):  zx_event_create
( 41)        3 ( 0.1%, 98.8%):  zx_port_create
( 42)        2 ( 0.1%, 98.8%):  CreateFileA
( 43)        2 ( 0.1%, 98.9%):  FormatMessageW
( 44)        2 ( 0.1%, 99.0%):  MultiByteToWideChar
( 45)        2 ( 0.1%, 99.0%):  NotifyIpInterfaceChange
( 46)        2 ( 0.1%, 99.1%):  clang_createIndex
( 47)        2 ( 0.1%, 99.1%):  objc_getClass
( 48)        2 ( 0.1%, 99.2%):  zx_ticks_get
( 49)        2 ( 0.1%, 99.3%):  zx_timer_create
( 50)        1 ( 0.0%, 99.3%):  CloseHandle
( 51)        1 ( 0.0%, 99.3%):  CryptStringToBinaryA
( 52)        1 ( 0.0%, 99.4%):  FreeConsole
( 53)        1 ( 0.0%, 99.4%):  GetCurrentProcessId
( 54)        1 ( 0.0%, 99.4%):  GetLogicalProcessorInformation
( 55)        1 ( 0.0%, 99.5%):  GetModuleHandleExW
( 56)        1 ( 0.0%, 99.5%):  GetNativeSystemInfo
( 57)        1 ( 0.0%, 99.5%):  GetUserDefaultLocaleName
( 58)        1 ( 0.0%, 99.5%):  LoadLibraryW
( 59)        1 ( 0.0%, 99.6%):  OpenClipboard
( 60)        1 ( 0.0%, 99.6%):  PFXImportCertStore
( 61)        1 ( 0.0%, 99.6%):  PQinitSSL
( 62)        1 ( 0.0%, 99.7%):  RegOpenKeyExW
( 63)        1 ( 0.0%, 99.7%):  _getmaxstdio
( 64)        1 ( 0.0%, 99.7%):  llvm.x86.sse42.pcmpestri128
( 65)        1 ( 0.0%, 99.8%):  objc_allocateClassPair
( 66)        1 ( 0.0%, 99.8%):  objc_allocateProtocol
( 67)        1 ( 0.0%, 99.8%):  objc_getClassList
( 68)        1 ( 0.0%, 99.8%):  sel_registerName
( 69)        1 ( 0.0%, 99.9%):  strerror
( 70)        1 ( 0.0%, 99.9%):  zx_cprng_add_entropy
( 71)        1 ( 0.0%, 99.9%):  zx_eventpair_create
( 72)        1 ( 0.0%,100.0%):  zx_fifo_create
( 73)        1 ( 0.0%,100.0%):  zx_socket_create

#

@golden summit see above, got some SetThreadErrorMode but no CreateThread

#

That list is 2,000 crates

#

I'm going to rerun with --jobs=2 --test-threads=2 to see if things change

golden summit Jun 18, 2022, 12:13 AM

#

i think SetThreadErrorMode can be a noop

haughty mica Jun 18, 2022, 12:14 AM

#

I just wonder if that's blocking CreateThread?

#

Only 10 calls though

#

Oh

#

It's from the crates libloading and dlopen

#

Never mind

golden summit Jun 18, 2022, 12:14 AM

#

id expect most threads to go through std

#

how the fuck is it calling GetFileInformationByHandleEx

#

theres no handles to call it on

haughty mica Jun 18, 2022, 12:15 AM

#

Here's the crates that do this:

arboard/2.1.1
arrayref/0.3.6
atty/0.2.14
authenticator/0.3.1
base32/0.4.0
bollard/0.12.0
bv/0.11.1
colored/2.0.0
env_logger/0.9.0
indicatif/0.17.0-rc.11
jsonpath_lib/0.3.0
jsonrpc-client-transports/18.0.0
maxminddb/0.23.0
mdbook/0.4.18
multipart/0.18.0
pretty/0.11.3
pretty-lint/0.1.1
radix_fmt/1.0.0
serde-xml-rs/0.5.1
solana-frozen-abi/1.10.25
structopt/0.3.26
trust-dns-resolver/0.21.2
yamux/0.10.1

#

env_logger seems like a good place to look

#

The call is in atty

#

     Running tests/init-twice-retains-filter.rs (target/miri/x86_64-pc-windows-msvc/debug/deps/init_twice_retains_filter-511f5a5722b421d8.exe)
thread 'main' panicked at 'unsupported Miri functionality: can't call foreign function: GetModuleFileNameW', /root/.rustup/toolchains/miri/lib/rustlib/src/rust/library/std/src/sys/windows/os.rs:239:28
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
     Running tests/log-in-log.rs (target/miri/x86_64-pc-windows-msvc/debug/deps/log_in_log-aaa47fd10b166f41.exe)
thread 'main' panicked at 'unsupported Miri functionality: can't call foreign function: GetFileInformationByHandleEx', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/atty-0.2.14/src/lib.rs:132:15
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
     Running tests/log_tls_dtors.rs (target/miri/x86_64-pc-windows-msvc/debug/deps/log_tls_dtors-a50580c8219b890c.exe)
thread 'main' panicked at 'unsupported Miri functionality: can't call foreign function: GetFileInformationByHandleEx', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/atty-0.2.14/src/lib.rs:132:15
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
     Running tests/regexp_filter.rs (target/miri/x86_64-pc-windows-msvc/debug/deps/regexp_filter-165a09d07488d5d8.exe)
thread 'main' panicked at 'unsupported Miri functionality: can't call foreign function: GetModuleFileNameW', /root/.rustup/toolchains/miri/lib/rustlib/src/rust/library/std/src/sys/windows/os.rs:239:28

golden summit Jun 18, 2022, 12:16 AM

#

oh hmm

#

its probably calling it on stderr/stdout

haughty mica Jun 18, 2022, 12:16 AM

#

That sounds about right

#

Rerunning everything, emergency grocery run, I'll be back later

haughty mica Jun 18, 2022, 1:04 AM

#

Still no CreateThread, 381 crates in

#

OH

#

Wrong search

#

╰ ➤ rg unsupported | cut -d' ' -f2- | rev | cut -d' ' -f2- | rev | counts
632 counts
(  1)      626 (99.1%, 99.1%): 'main' panicked at 'unsupported Miri functionality: can't create threads on Windows',
(  2)        3 ( 0.5%, 99.5%): 'main' panicked at 'unsupported Miri functionality: can't call foreign function: GetModuleFileNameW',
(  3)        2 ( 0.3%, 99.8%): 'main' panicked at 'unsupported Miri functionality: can't call foreign function: GetFileInformationByHandleEx',
(  4)        1 ( 0.2%,100.0%): 'main' panicked at 'unsupported Miri functionality: can't call foreign function: GetFullPathNameW',```

#

There's your answer

golden summit Jun 18, 2022, 2:54 AM

#

oh yeah thats right

#

theres a special error for it

haughty mica Jun 18, 2022, 3:26 AM

#

"cache everything, in num_cpus build directories" turns out to be exceedingly bad

#

Ran myself out of disk space in like 15 minutes

west violet Jun 18, 2022, 5:53 PM

#

@haughty mica Have you heard of CodeQL any? https://codeql.github.com/

#

Because I think we should explore it for finding ub in the ecosystem, it's especially nice since it can potentially catch the end consumers of things that aren't on github, the actual binaries being used by people

tough leaf Jun 18, 2022, 5:57 PM

#

west violet <@176135688666742784> Have you heard of CodeQL any? <https://codeql.github.com/>

it's basically a nicer clippy, no?

west violet Jun 18, 2022, 5:57 PM

#

That's doing a pretty big disservice

tough leaf Jun 18, 2022, 5:57 PM

#

and by nicer i mean propritary

west violet Jun 18, 2022, 5:57 PM

#

CodeQL is a program database, basically

#

And you can write queries against that program database to find pretty much anything

#

It's also not really proprietary, it's oss

tough leaf Jun 18, 2022, 6:02 PM

#

west violet It's also not really proprietary, it's oss

i've not seen a single OSS license that put restrictions on how you're allowed to use the code

west violet Jun 18, 2022, 6:02 PM

#

It's MIT?

tough leaf Jun 18, 2022, 6:03 PM

#

is it?

west violet Jun 18, 2022, 6:03 PM

#

Yah

#

https://github.com/github/codeql/blob/main/LICENSE

GitHub

codeql/LICENSE at main · github/codeql

CodeQL: the libraries and queries that power security researchers around the world, as well as code scanning in GitHub Advanced Security (code scanning), LGTM.com, and LGTM Enterprise - codeql/LICE...

tough leaf Jun 18, 2022, 6:03 PM

#

so i can run it on closed source code?

#

despite their ToS saying you can't

west violet Jun 18, 2022, 6:03 PM

#

I mean, it's MIT

#

I don't see any tos on the repo

tender nimbus Jun 18, 2022, 6:04 PM

#

i guess you can't use their hosted database thing for everything

#

but if you do it all yourself, it does look like its fine

tough leaf Jun 18, 2022, 6:04 PM

#

You can create CodeQL databases yourself for any project that's under an OSI-approved open source license.

#

hmmm

tender nimbus Jun 18, 2022, 6:04 PM

#

ferrisballSweat

tough leaf Jun 18, 2022, 6:04 PM

#

the thick plottens...

west violet Jun 18, 2022, 6:05 PM

#

That's besides the point though

tough leaf Jun 18, 2022, 6:05 PM

#

The CodeQL CLI (including the CodeQL engine) is hosted in a different repository and is licensed separately

#

yep
open source frontend talking to a closed source core

#

well, not open source, idk if it's closed

west violet Jun 18, 2022, 6:05 PM

#

Weird

#

REGARDLESS

#

It could be really useful in the search for ub

tender nimbus Jun 18, 2022, 6:06 PM

#

if it supported rust ferrisClueless

west violet Jun 18, 2022, 6:06 PM

#

Yah, that's the sucky part

tender nimbus Jun 18, 2022, 6:07 PM

#

well, they actually use some rust ferrisBut
but don't support querying it

west violet Jun 18, 2022, 6:07 PM

#

But like, it's got dataflow analysis and stuff, that could be amazingly useful

haughty mica Jun 18, 2022, 6:07 PM

#

west violet Because I think we should explore it for finding ub in the ecosystem, it's espec...

That's static analysis and I'm yet to be convinced this is a particularly worthwhile effort

tough leaf Jun 18, 2022, 6:07 PM

#

in any case
it would be good if clippy had the power to do stuff like this

west violet Jun 18, 2022, 6:08 PM

#

"Woah there cowboy, this branch is 100% reachable but you put an unreachable_unchecked there!"

tender nimbus Jun 18, 2022, 6:08 PM

#

haughty mica That's static analysis and I'm yet to be convinced this is a particularly worthw...

it could lint on the common pitfalls I guess like .as_ptr on slices to mutate and so on

haughty mica Jun 18, 2022, 6:08 PM

#

Yeah but... so could clippy

tender nimbus Jun 18, 2022, 6:08 PM

#

yes

haughty mica Jun 18, 2022, 6:08 PM

#

A lot of the static patterns I'm aware of are incredibly blatant, everything else is subtle

#

Put it this way: I'll be convinced this is worthwhile when I stop finding things with Miri

#

Dynamic analysis is great because you can do it without false positives. Static analysis has the opposite problem, it can both miss things and have false positives. The singular advantage is that you don't need test cases.

tender nimbus Jun 18, 2022, 6:12 PM

#

miri is obviously better
but codeql wouldn't need extensive test coverage and doesn't care about miri being slow
but yeah, as long as codeql is not a thing with rust, it's not getting anywhere anyways

haughty mica Jun 18, 2022, 6:12 PM

#

There's some research group doing static analysis looking for security issues in the ecosystem

#

I mostly am aware of them because they submitted a broken clippy lint, but I have seen them do correct things elsewhere

tender nimbus Jun 18, 2022, 6:13 PM

#

ferrisBut what did the lint do

haughty mica Jun 18, 2022, 6:13 PM

#

It was something with transmute

#

It flagged a lot of valid code as UB

tough leaf Jun 18, 2022, 6:30 PM

#

https://github.com/rust-lang/rust-clippy/issues/7706

GitHub

Vec> transmute to Vec · Issue #7706 · rust-lang/rust-clippy

Lint name: unsound_collection_transmute I tried this code: unsafe fn foo(v: Vec<std::mem::MaybeUninit<u8>>) -> Vec<u8> { std::mem::transmute(v) } I ...

#

wait what

#

this is literally UB isn't it

#

well

#

depending on unspecified layout

west violet Jun 18, 2022, 6:31 PM

#

Indeed

tender nimbus Jun 18, 2022, 6:32 PM

#

what

#

why

#

did they "fix" it

tough leaf Jun 18, 2022, 6:33 PM

#

tender nimbus did they "fix" it

the old code checked the abi

west violet Jun 18, 2022, 6:33 PM

#

Someone should try to break it with -Z randomize-layout

tough leaf Jun 18, 2022, 6:33 PM

#

and the abi of MaybeUninit<u32> is Union

#

and the abi of u32 is "integer"

#

ABI's different, therefore lint

#

the current code checks the align of the abi only

#

https://doc.rust-lang.org/nightly/nightly-rustc/rustc_target/abi/enum.Abi.html is the ABI being talked about

Abi in rustc_target::abi - Rust

Describes how values of the type are passed by target ABIs, in terms of categories of C types there are ABI rules for.

#

uhhhh

#

maybe

#

yes

#

the old code also would have linted on a bool to u8 transmute

#

because those have different ABIs

#

specifically, the valid_range

tender nimbus Jun 18, 2022, 6:35 PM

#

oh, yeah that is stupid

tough leaf Jun 18, 2022, 6:35 PM

#

regardless

#

the layout of vec is unspecified

#

soooo

tough leaf Jun 18, 2022, 6:35 PM

#

west violet Someone should try to break it with `-Z randomize-layout`

honestly just add a field to vec

#

hmm

#

can you make a field's size depend on T

#

well

#

typeid is not const is it

#

wait shit it is

#

?play

#![feature(generic_const_exprs, core_intrinsics)]
struct FuckYouVec<T> where [(); (std::intrinsics::type_id::<T>() % 8) as usize]: {
  a: T,
  lol: [u8; (std::intrinsics::type_id::<T>() % 8) as usize],
}

small apexBOT Jun 18, 2022, 6:36 PM

#

warning: the feature `generic_const_exprs` is incomplete and may not be safe to use and/or cause compiler crashes
 --> src/main.rs:1:12
  |
1 | #![feature(generic_const_exprs, core_intrinsics)]
  |            ^^^^^^^^^^^^^^^^^^^
  |
  = note: `#[warn(incomplete_features)]` on by default
  = note: see issue #76560 <https://github.com/rust-lang/rust/issues/76560> for more information

error: `std::intrinsics::type_id` is not yet stable as a const fn
 --> src/main.rs:3:34
  |
3 | struct FuckYouVec<T> where [(); (std::intrinsics::type_id::<T>() % 8) as usize]: {
  |                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |
  = help: add `#![feature(const_type_id)]` to the crate attributes to enable

error[E0310]: the parameter type `T` may not live long enough
 --> src/main.rs:3:34
  |
3 | struct FuckYouVec<T> where [(); (std::intrinsics::type_id::<T>() % 8) as usize]: {
  |                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...so that the type `T` will meet its required lifetime bounds
  |
help: consider adding an explicit lifetime bound...
  |
3 | struct FuckYouVec<T: 'static> where [(); (std::intrinsics::type_id::<T>() % 8) as usize]: {
  |                    +++++++++

error: `std::intrinsics::type_id` is not yet stable as a const fn
 --> src/main.rs:5:14
  |
5 |   lol: [u8; (std::intrinsics::type_id::<T>() % 8) as usize],
  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |
  = help: add `#![feature(const_type_id)]` to the crate attributes to enable

error[E0310]: the parameter type `T` may not live long enough
 --> src/main.rs:5:14
  |
5 |   lol: [u8; (std::intrinsics::type_id::<T>() % 8) as usize],
  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ...so that the type `T` will meet its required lifetime bounds
  |
help: consider adding an explicit lifetime bound...
  |```Output too large. Playground link: <https://play.rust-lang.org/?version=nightly&mode=debug&edition=2021&gist=5316b77b0f364001e128d17266e07eb8>

tough leaf Jun 18, 2022, 6:37 PM

#

gah

west violet Jun 18, 2022, 6:38 PM

#

Actually MaybeUninit is transparent so like, this may be ok?

tough leaf Jun 18, 2022, 6:38 PM

#

uhhhhh

tender nimbus Jun 18, 2022, 6:38 PM

#

according to the current rules, it's not ok
but, honestly, it should be

#

you should be allowed to transmute vecs

west violet Jun 18, 2022, 6:38 PM

#

Vec is repr(Rust)

tough leaf Jun 18, 2022, 6:39 PM

#

tbh i Kinda agree but also kinda not

#

vec being special like this would be kinda weird

tender nimbus Jun 18, 2022, 6:39 PM

#

vec tells you that its layout is two usizes and a pointer, and if T is repr(transparent) then like... it will literally never ever break

#

on the other hand making weird exceptions is weird

haughty mica Jun 18, 2022, 6:40 PM

#

test pool::tests::grow ... error: Undefined Behavior: dereferencing pointer failed: alloc53 has size 31, so pointer to 136 bytes starting at offset 330947 is out-of-bounds
   --> /home/ben/.rustup/toolchains/miri/lib/rustlib/src/rust/library/core/src/ptr/non_null.rs:381:18
    |
381 |         unsafe { &*self.as_ptr() }
    |                  ^^^^^^^^^^^^^^^ dereferencing pointer failed: alloc53 has size 31, so pointer to 136 bytes starting at offset 330947 is out-of-bounds
    |
    = help: this indicates a bug in the program: it performed an invalid operation, and caused Undefined Behavior
    = help: see https://doc.rust-lang.org/nightly/reference/behavior-considered-undefined.html for further information

#

Wow that uh sounds bad

tender nimbus Jun 18, 2022, 6:41 PM

#

uhm, crate?

haughty mica Jun 18, 2022, 6:41 PM

#

heapless

tough leaf Jun 18, 2022, 6:41 PM

#

again

haughty mica Jun 18, 2022, 6:41 PM

#

No I'm just actually running Miri on it

#

I punted on it because of problems like this before

tender nimbus Jun 18, 2022, 6:42 PM

#

Undefined Behavior: type validation failed at .value: encountered uninitialized bytes, but expected initialized plain (non-pointer) bytes
  --> target/miri/x86_64-unknown-linux-gnu/debug/build/if_rust_version-679cd1ad5881e595/out/generated.rs:114:33
   |
10 |     let mut foo: u32 = unsafe { ::std::mem::MaybeUninit::uninit().assume_init() };
   |                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ type validation failed at .value: encountered uninitialized bytes, but expected initialized plain (non-pointer) bytes

wow, they used MaybeUninit, progress!

tough leaf Jun 18, 2022, 6:42 PM

#

lmaooooo

haughty mica Jun 18, 2022, 6:42 PM

#

Hey now

tough leaf Jun 18, 2022, 6:42 PM

#

honestly if you're gonna do that i'd rather people just use mem::uninitialized

#

at least then i can panic

haughty mica Jun 18, 2022, 6:42 PM

#

Isn't that code supposed to be like that though?

tough leaf Jun 18, 2022, 6:43 PM

#

what, UB?

haughty mica Jun 18, 2022, 6:43 PM

#

Yes

#

I think this is like a generated test to see if things compile/run

tender nimbus Jun 18, 2022, 6:44 PM

#

it's if_rust_version

tough leaf Jun 18, 2022, 6:44 PM

#

fn test_uninit() {
    unsafe fn init_bool(x: *mut bool) {
        std::ptr::write(x, true);
    }

    let b: bool = {
        if_rust_version! { >= 1.36 {
            let mut x = std::mem::MaybeUninit::<bool>::uninit();
            unsafe { init_bool(x.as_mut_ptr()); }
            unsafe { x.assume_init() }
        } else {
            let mut x : bool = unsafe { std::mem::uninitialized() };
            unsafe { init_bool(&mut x as *mut bool) }
            x
        }}
    };
    assert_eq!(b, true);
}

#

they have this test

#

which is correct (if your rust version is higher than 1.36)

#

and i think literally panics if it's not

#

?eval unsafe { std::mem::uninitialized::<bool>() }

small apexBOT Jun 18, 2022, 6:44 PM

#

thread 'main' panicked at 'attempted to leave type `bool` uninitialized, which is invalid', src/main.rs:2:10
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

tough leaf Jun 18, 2022, 6:44 PM

#

wait

#

if you're lower than 1.36

#

you wouldn't have that panic

#

lmao

tender nimbus Jun 18, 2022, 6:45 PM

#

it's just a doc example

#

# use if_rust_version::if_rust_version;
if_rust_version!{ >= 1.36 { use std::mem::MaybeUninit; }}
// ...
if_rust_version!{ < 1.36 {
    let mut foo: u32 = unsafe { ::std::mem::uninitialized() };
} else {
    let mut foo: u32 = unsafe { ::std::mem::MaybeUninit::uninit().assume_init() };
}}

but an, uhm, bad one

tough leaf Jun 18, 2022, 6:45 PM

#

brb having UB in your tests make them pass even though your code doesn't work

tender nimbus Jun 18, 2022, 6:45 PM

#

this is a terrible doc example honestly

tough leaf Jun 18, 2022, 6:46 PM

#

yes

#

honestly

#

if you use mem::uninitialized your code is sus

tender nimbus Jun 18, 2022, 6:46 PM

#

but i cant be bothered to pr it

proper belfry Jun 18, 2022, 6:47 PM

#

tender nimbus ```rust # use if_rust_version::if_rust_version; if_rust_version!{ >= 1.36 { use ...

this is actually so funny lmao

#

like, “if your Rust version is 1.36, cause UB”

#

“but if your Rust version is above 1.36, we can take advantage of those new features to…also cause UB!”

thorny venture Jun 18, 2022, 6:49 PM

#

There should be a crate that offers maybeuninit APIs that can toggle it's underlying data

#

either it's backed on std::mem::maybeuninit, or it just uses UB

#

Also, if it has the slice/array APIs I'd use it

tender nimbus Jun 18, 2022, 6:50 PM

#

thorny venture There should be a crate that offers maybeuninit APIs that can toggle it's underl...

why

#

what is missing for MU

tough leaf Jun 18, 2022, 6:51 PM

#

the ability to be UB on older versions

thorny venture Jun 18, 2022, 6:51 PM

#

Yeah specifically the msrv support based on a build script to auto enable features

tender nimbus Jun 18, 2022, 6:52 PM

#

ok but
who cares about pre-MU versions at this point

tough leaf Jun 18, 2022, 6:52 PM

#

rt

thorny venture Jun 18, 2022, 6:52 PM

#

There's always someone

tender nimbus Jun 18, 2022, 6:52 PM

#

that shit's 3 years old and almost every other crates has higher MSRV

thorny venture Jun 18, 2022, 6:52 PM

#

Generic array still exists

tough leaf Jun 18, 2022, 6:53 PM

#

debian oldoldstable is apparently 1.41

#

that's beyond MaybeUninit

tender nimbus Jun 18, 2022, 6:53 PM

#

thorny venture Generic array still exists

that's mostly because people don't care about updating their code to const generics, not because they use old compilers

tough leaf Jun 18, 2022, 6:53 PM

#

if fuckin debian stable has it

#

so does everyone else

thorny venture Jun 18, 2022, 6:53 PM

#

I still haven't actually managed to port rust crypto to const generics rip

tough leaf Jun 18, 2022, 6:53 PM

#

thorny venture Generic array still exists

typenum :fear:

tender nimbus Jun 18, 2022, 6:53 PM

#

tough leaf if fuckin debian stable has it

debian stable ferrisClueless

#

I've heard they're getting ready for rust 1.0 soon, I'm excited! though it's a bit annoying, their latest upgrade forced me to use Box instead of ~, that was a huge change

thorny venture Jun 18, 2022, 6:54 PM

#

Wow rust really is verbose

tender nimbus Jun 18, 2022, 6:55 PM

#

to be honest the ~ worked perfectly fine, i don't get why they've even upgraded

#

like, i was programming just fine

thorny venture Jun 18, 2022, 6:55 PM

#

Should just be heap noalias T

tender nimbus Jun 18, 2022, 6:55 PM

#

but then - boom- rust upgrade and everything was broken

tender nimbus Jun 18, 2022, 6:55 PM

#

thorny venture Should just be `heap noalias T`

this would avoid so much box ub

#

|r| &**r as *const _ as *mut Context

oh come on

#

please at least read the nomicon
even the nomicon says that this is bad

#

thread_local!(
    /// each thread has it's own generator context stack
    static ROOT_CONTEXT: Box<Context> = {
        let mut root = Box::new(Context::new());
        let p = &mut *root as *mut _;
        root.parent = p; // init top to current
        root
    }
);

ugh

#

yeah that's just cursed, not gonna touch that

#

(generator)

#

Undefined Behavior: type validation failed at .value[0]: encountered uninitialized bytes
   --> /root/.cargo/registry/src/github.com-1ecc6299db9ec823/solana-sdk-1.10.24/src/packet.rs:101:28
    |
101 |             data: unsafe { std::mem::MaybeUninit::uninit().assume_init() },
    |                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ type validation failed at .value[0]: encountered uninitialized bytes

lmao solana

#

 unsafe { mem::transmute::<&SpannedExpr<Id>, &SpannedExpr<Id>>(&*self.expr) }

the fuck???

#

uhhhhhh, the borrow checker is not happy removing some of those transmutes

#

it's a fucking abstract syntax tree what are they doing ferrisSob

haughty mica Jun 18, 2022, 7:18 PM

#

Where?

tender nimbus Jun 18, 2022, 7:19 PM

#

gluon

haughty mica Jun 18, 2022, 7:20 PM

#

Nice

thorny venture Jun 18, 2022, 8:04 PM

#

tender nimbus ```rust unsafe { mem::transmute::<&SpannedExpr<Id>, &SpannedExpr<Id>>(&*self.ex...

Lifetime's don't match up? No problem

tender nimbus Jun 18, 2022, 8:05 PM

#

although the type says something about Owned* so it's probably fine perhaps maybe idk didn't investigate closer

pastel lily Jun 18, 2022, 9:21 PM

#

tender nimbus I've heard they're getting ready for rust 1.0 soon, I'm excited! though it's a b...

That change was before 1.0 :p

#

Somewhere before 0.11

haughty mica Jun 19, 2022, 3:38 PM

#

dmsort released with all my fixes

haughty mica Jun 19, 2022, 4:31 PM

#

Yoooooo tract passes a null pointer to memcpy with size 0

#

That's an old C bug

tender nimbus Jun 19, 2022, 4:32 PM

#

ferrisballSweat

pastel lily Jun 19, 2022, 4:37 PM

#

ferrisballSweat

#

actually what does C say about that

tough leaf Jun 19, 2022, 4:38 PM

#

uhhhhh

#

even if the size is 0 the pointers still need to be valid according to Someone Online

#

https://groups.google.com/g/comp.lang.c/c/LmTrGXN22Sw

pastel lily Jun 19, 2022, 4:53 PM

#

lol

tender nimbus Jun 19, 2022, 5:11 PM

#

Yeah that's ub ferrisballSweat

west violet Jun 19, 2022, 5:42 PM

#

Yah, zero is always an invalid pointer

tough leaf Jun 19, 2022, 5:44 PM

#

except for free and realloc

#

:)

#

free(null) is allowed

west violet Jun 19, 2022, 6:27 PM

#

Thanks C™️

haughty mica Jun 21, 2022, 1:52 PM

#

Try to fix UB in a crate

#

Add a SIGSEGV

#

Try to run tests with ASan

#

Find different UB

tender nimbus Jun 21, 2022, 2:07 PM

#

ferrisballSweat

west violet Jun 21, 2022, 2:10 PM

#

Wow Saethlin, dbsp absolutely thrashes miri

haughty mica Jun 21, 2022, 2:23 PM

#

Which part though

west violet Jun 21, 2022, 2:34 PM

#

This one was the latest, it's smaller which may be helpful
https://github.com/vmware/database-stream-processor/blob/f86557c080c925cefa4b781a80ae5c06ce7046c4/src/trace/ord/merge_batcher/tests.rs#L32-L41

#

If I increased the numbers here it OOM'd on CI and stuff https://github.com/vmware/database-stream-processor/blob/f86557c080c925cefa4b781a80ae5c06ce7046c4/src/trace/ord/merge_batcher/tests.rs#L14-L16

haughty mica Jun 21, 2022, 2:35 PM

#

To be clear, "this test causes Miri to timeout/consume all the memory" is not interesting in itself.
If it's doing this for a new reason, that's interesting

west violet Jun 21, 2022, 2:35 PM

#

Ah gotcha

haughty mica Jun 21, 2022, 9:06 PM

#

Ok new problem

#

Analysing: /home/ben/miri-tools

└── 100.00% [164.13 GB] ── miri-tools
   ├── 85.00% [139.51 GB] ── shared-target
   ├── 7.47% [12.26 GB] ── logs
   ├── 6.74% [11.07 GB] ── cache
   └── 0.76% [1.25 GB] ── target

#

Tried just doing everything in a shared target dir, you know, for caching and speed

#

While it is definitely more efficient, this disk usage is intolerable

haughty mica Jun 21, 2022, 11:10 PM

#

Oh dear the codebase also dies under MSan, before it dies under ASan

hybrid forge Jun 23, 2022, 3:41 PM

#

thorny venture <https://github.com/rust-lang/rfcs/pull/3131#issuecomment-897084808>

Sorry for being off-topic but I couldn't find the generic storage RFC that is mentioned there, do you know which one is it? I don't think this is https://github.com/rust-lang/rfcs/blob/master/text/1398-kinds-of-allocators.md

thorny venture Jun 23, 2022, 7:58 PM

#

I don't know if there is a storage rfc just yet

thorny venture Jun 23, 2022, 8:02 PM

#

hybrid forge Sorry for being off-topic but I couldn't find the generic storage RFC that is me...

Core concept: Storage trait defines how to get a contiguous region of memory. Can be implemented in different ways, one for heap allocation, one for arrays etc. This allows vec to use trait for backing storage and for free get a built in ArrayVec and HeapVec and Cursor API in the same vec impl

#

Some people extend it further to generic dyn storage so you can have unsized stuff on the stack but it has align issues

hybrid forge Jun 23, 2022, 8:08 PM

#

I see. I wonder if that kind of trait would be generic enough so that bitvec for example could be rewritten with it

thorny venture Jun 23, 2022, 8:56 PM

#

Unlikely. Bitvec is very specific

west violet Jun 23, 2022, 8:56 PM

#

I don't see why bitvec would care tbh

#

It has memory, it's happy

thorny venture Jun 23, 2022, 8:59 PM

#

I think they mean a bit storage trait that auto impls bitvec with regular vec

#

You definitely can use storage with your own bitvec wrapper, but you need to implement the bit twiddling in the vec and not the storage

west violet Jun 23, 2022, 9:01 PM

#

You mean like std::vector<bool>?

thorny venture Jun 23, 2022, 9:01 PM

#

Ja

west violet Jun 23, 2022, 9:01 PM

#

Please no

thorny venture Jun 23, 2022, 9:01 PM

#

Well, that's why you have bitvec and not Vec<bool>

west violet Jun 23, 2022, 9:02 PM

#

Oh yah, totally agree

#

I mean, you can probably implement vector<bool> given specialization, but the allocator is fairly unrelated to that

tough leaf Jun 23, 2022, 9:03 PM

#

vector<bool> feels like a good idea in a language that is higher level than C++ or rust is

west violet Jun 23, 2022, 9:03 PM

#

It doesn't matter how you get your memory, just how you interact with it

tough leaf Jun 23, 2022, 9:03 PM

#

like, a pure FP language can do it just fine if you have no concept of references

pastel lily Jun 23, 2022, 9:08 PM

#

yeah

haughty mica Jun 23, 2022, 11:13 PM

#

So I'm running ASan just plain old ASan and overriding with -Copt-level=0 and my results are ~50% coroutine libraries that ASan doesn't understand and ~50% merkle tree implementations that recurse and blow the stack

haughty mica Jun 24, 2022, 12:05 AM

#

https://gitlab.com/hjiayz/const_std_vec/-/blob/main/src/lib.rs

GitLab

src/lib.rs · main · ppc / Const Std Vec · GitLab

GitLab.com

tender nimbus Jun 24, 2022, 4:21 AM

#

the fuck 😵‍💫

#

this is like UB on every level

pastel lily Jun 24, 2022, 4:25 AM

#

what

#

this ever worked?!?!

#

like yea it's UB but I would have assumed this also just... wouldn't produce the expected output

stable arrow Jun 24, 2022, 4:31 AM

#

if you never mutate the vec the capacity is almost never read … ?

pastel lily Jun 24, 2022, 4:32 AM

#

ah

#

I guess

tender nimbus Jun 24, 2022, 4:38 AM

#

Don't forget about the fact that this is extremely UB (hopefully with a crash?) when you init this with more than 0 elements ferrisForgor

#

also was Vec::new not const when this was written?

haughty mica Jun 24, 2022, 4:47 AM

#

There's another crate like this which was written because the author learned about transmute before once_cell

haughty mica Jun 24, 2022, 10:19 PM

#

I just found an off-by-one error that absolutely takes the cake

#

Fortunately the crate isn't too big, only about a million downloads

west phoenix Jun 24, 2022, 10:21 PM

#

Ooooh

haughty mica Jun 24, 2022, 10:33 PM

#

https://github.com/gtk-rs/gtk-rs-core/pull/720/files

diff --git a/glib/src/gstring.rs b/glib/src/gstring.rs
index 9890a05fe50..241f0283f38 100644
--- a/glib/src/gstring.rs
+++ b/glib/src/gstring.rs
@@ -700,7 +700,7 @@ impl From<&str> for GString {
         unsafe {
             // No check for valid UTF-8 here
             let copy = ffi::g_malloc(s.len() + 1) as *mut c_char;
-            ptr::copy_nonoverlapping(s.as_ptr() as *const c_char, copy, s.len() + 1);
+            ptr::copy_nonoverlapping(s.as_ptr() as *const c_char, copy, s.len());
             ptr::write(copy.add(s.len()), 0);
 
             GString(Inner::Foreign {

#

I'm now running ASan on crates, just... all crates...
And most of my ASan reports are https://github.com/rust-lang/rust/issues/98454

#

I have 29 reports, and I think 17 of them are that bug

#

facepalm Mozilla why

#

https://github.com/mozilla/lmdb-rs/blob/946167603dd6806f3733e18f01a89cee21888468/lmdb-sys/tests/simple.rs#L20-L24

macro_rules! str {
    ($expr:expr) => {
        ::CString::new($expr).unwrap().as_ptr()
    };
}

pastel lily Jun 24, 2022, 11:15 PM

#

what

pastel lily Jun 24, 2022, 11:15 PM

#

haughty mica <https://github.com/gtk-rs/gtk-rs-core/pull/720/files> ```diff diff --git a/glib...

why

#

how

#

what

#

I know we all do off by 1

#

but this one is... something

stable arrow Jun 24, 2022, 11:17 PM

#

that reads to me as someone thinking "and copy the existing \0" (except that there is no such thing, for &str)

#

I notice they're also not checking for nulls inside the string, but that's a semantic problem rather than memory unsafety

pastel lily Jun 24, 2022, 11:22 PM

#

ohhh

#

that could make sense for someone coming from C

#

does strlen include the \0 normally (I don't think so?)

tough leaf Jun 24, 2022, 11:39 PM

#

no it does not

haughty mica Jun 24, 2022, 11:54 PM

#

stable arrow I notice they're also not checking for nulls inside the string, but that's a sem...

The line directly after that seems to refute this

#

If the + 1 copies the null, why does the code then write one

stable arrow Jun 25, 2022, 12:03 AM

#

I don't mean to claim it's self-consistent code, just that I can imagine how someone might write a + 1 out of habit

haughty mica Jun 25, 2022, 12:06 AM

#

Haha okay

tender nimbus Jun 25, 2022, 7:08 AM

#

what null terminators do to sane programmers

knotty oar Jun 25, 2022, 8:59 AM

#

lol

tender nimbus Jun 25, 2022, 9:02 AM

#

ah yes, "c programmers\0"

ruby jacinth Jun 25, 2022, 9:19 AM

#

tender nimbus ah yes, "c programmers\0"

No u mean str!("c programmers")

haughty mica Jun 25, 2022, 3:52 PM

#

53:==4409==ERROR: AddressSanitizer: requested allocation size 0xffffffffffffffff (0x800 after adjustments for alignment, red zones etc.) exceeds maximum supported size of 0x10000000000 (thread T61 (tests::read_to_))

lmao

#

Of course it's actually from fallible-collections

#

That crate is so sus

pastel lily Jun 25, 2022, 3:54 PM

#

corro

tender nimbus Jun 25, 2022, 3:55 PM

#

        let ptr = alloc(layout)?.as_ptr() as *mut T;
        unsafe {
            core::ptr::write(ptr, t);

#

oh nvm that's legal

#

it's a NonNull

#

i was already scared

haughty mica Jun 25, 2022, 3:57 PM

#

Yeah it's technically fine because the allocator just fails, but it's extra amusing because the size ASan reports wraps

tender nimbus Jun 25, 2022, 3:57 PM

#

test vec::tests::try_clone_oom ... error: Undefined Behavior: invalid metadata in wide pointer: slice is bigger than largest supported object lmao

pastel lily Jun 25, 2022, 3:59 PM

#

oh no

haughty mica Jun 25, 2022, 3:59 PM

#

// FIXME: EXPLAIN SAFETY

tender nimbus Jun 25, 2022, 3:59 PM

#

(they from_raw_parts usize::MAX)

haughty mica Jun 25, 2022, 3:59 PM

#

This is what I mean when I say safety comments are dubious

tender nimbus Jun 25, 2022, 3:59 PM

#

oh

haughty mica Jun 25, 2022, 4:00 PM

#

I'm just not sure anyone wrote some unsafe code, it was wrong, and they went to type out the Safety comment are were like "oh wait yeah I can't do this actually"

tender nimbus Jun 25, 2022, 4:00 PM

#

ignoring that issue, at least the tests pass disable-sb ferrisClueless

#

default miri is ok as well

haughty mica Jun 25, 2022, 4:01 PM

#

Btw some fraction of the invalid pointer offset Miri errors are buffer overflows

#

Miri won't report a buffer overflow because people do invalid offsets first

tender nimbus Jun 25, 2022, 4:02 PM

#

i think everyone can imagine what raw pointer tagging miri thinks about this

haughty mica Jun 25, 2022, 4:03 PM

#

Also the invalid offsets are definite UB in any case

#

Irrespective of aliasing model, because of GEPI

tough leaf Jun 25, 2022, 4:04 PM

#

haughty mica I'm just not sure anyone wrote some unsafe code, it was wrong, and they went to ...

// SAFETY: it's not, but i don't care.

haughty mica Jun 25, 2022, 4:20 PM

#

// TODO: document why this is ok

Narrator: it is not ok

tender nimbus Jun 25, 2022, 4:25 PM

#

ferrisballSweat

#

now i see why this is still a todo

#

it's simply impossible

haughty mica Jun 25, 2022, 5:06 PM

#

Omg this is just a bad memmem implementation

#

Aaaaaaaa

#

Just use the good crate

tough leaf Jun 25, 2022, 5:07 PM

#

haughty mica Just use the good crate

evil C coders be like: i will use dependencies and not re-invent the wheel myself

haughty mica Jun 25, 2022, 5:08 PM

#

I've actually heard C coders insist that SIMD buffer overflows don't count

tough leaf Jun 25, 2022, 5:11 PM

#

well, you literally can't SIMDify strlen if you don't say "welll..... if we read it but don't use it, it's fine"

#

that being said i think it's usually written in asm

#

where stuff like that is fine

#

ish

haughty mica Jun 25, 2022, 5:12 PM

#

Uhhhhh

#

Yeah

#

So about that

#

The one in glibc is not

tender nimbus Jun 25, 2022, 5:13 PM

#

i mean, it's glibc, they are allowed to do these cursed things if they're very careful ferrisClueless

haughty mica Jun 25, 2022, 6:28 PM

#

wtf is chewing up all my CPU
Oh I'm running an FFT library with ASan

#

At least it's not a solana crate, I'll just renice this instead of kiling it

thorny venture Jun 25, 2022, 6:44 PM

#

Who's fft library?

#

I can't remember if I even published mine so it's ok 😅

haughty mica Jun 25, 2022, 6:48 PM

#

ark-poly

#

Not to worry, you'll get a PR if one of your crates has a problem

tough leaf Jun 25, 2022, 6:50 PM

#

#

this is what my github homepage is

#

you forking repos

haughty mica Jun 25, 2022, 6:52 PM

#

blob3cevil

#

I've exited the popular crates and there's a lot more UB down here

#

Same story every time

tough leaf Jun 25, 2022, 6:53 PM

#

i'm having another pop at sentry's crates

#

for fun

#

i don't think they have too many problems directly it's just their deps

#

and those have been largely fixed to my knowledge

#

pop as in fuzzing

#

not UB

haughty mica Jun 25, 2022, 6:54 PM

#

Yeah sentry is all about processing untrusted data isn't it

tough leaf Jun 25, 2022, 6:54 PM

#

yes

#

yes it is

#

untrusted data from processes who just crashed

haughty mica Jun 25, 2022, 6:55 PM

#

Do they run sentry on sentry?

tough leaf Jun 25, 2022, 6:55 PM

#

presumably

haughty mica Jun 25, 2022, 6:56 PM

#

Don't count on it

#

Customers specifically try to run our network sensors out-of-band, and we've occasionally told them that this is a bad idea because the sensor won't monitor itself

#

Btw --cpu-shares slaps

#

Computer is now very usable with all the cores pinned by various containers

tough leaf Jun 25, 2022, 6:57 PM

#

oh look
they have a subtract with overflow in a parser

#

that doesn't count

#

i'll run in release to get more real issues

pastel lily Jun 25, 2022, 6:58 PM

#

though that should be changed to use an explicit method or type

#

because it might not be the intent that it overflows

tough leaf Jun 25, 2022, 6:58 PM

#

oh true

#

but "hey i found a debug assertion" is not as interesting as "hey i found a segfault"

pastel lily Jun 25, 2022, 6:59 PM

#

yeah

tough leaf Jun 25, 2022, 6:59 PM

#

which i did, once

in C++ code

pastel lily Jun 25, 2022, 6:59 PM

#

but now you have a value of several billion that’s presumably not meant to be several billion

tough leaf Jun 25, 2022, 6:59 PM

#

breakpad really living up to its name

#

sure

pastel lily Jun 25, 2022, 7:00 PM

#

oh yea and you got called awesome in a blog post ferrisOwO

tough leaf Jun 25, 2022, 7:00 PM

#

yes

haughty mica Jun 25, 2022, 7:40 PM

#

I'm now asking questions like "How did this ever work"

#

Computers are wild

west violet Jun 25, 2022, 8:02 PM

#

Too bad miri isn't more performant, fuzzing under miri could be amazing

#

Plus if you got really creative you could probably take advantage of miri knowing what's going on to give yourself SAGE-style path guided fuzzing

tough leaf Jun 25, 2022, 8:03 PM

#

west violet Plus if you got really creative you could probably take advantage of miri knowin...

hmmmmmmm

#

yeah

west violet Jun 25, 2022, 8:04 PM

#

Still sad sage isn't public

tough leaf Jun 25, 2022, 8:04 PM

#

you won't get fast

#

so you need to optimise for making the most of every execution you get

tender nimbus Jun 25, 2022, 8:05 PM

#

haughty mica I'm now asking questions like "How did this ever work"

what is it doing

haughty mica Jun 25, 2022, 8:05 PM

#

I was misreading it, the buffer overflow is not nearly as large as I thought at first

#

But it does overflow by like 24 bytes

west violet Jun 25, 2022, 8:05 PM

#

Wowza

tender nimbus Jun 25, 2022, 8:05 PM

#

only a little overflow
we can take that

#

ferrisClueless

haughty mica Jun 25, 2022, 8:06 PM

#

It's a SIMD loop and the loop bound is wrong

#

But I looked at the tests and this library really needs fuzzing

tough leaf Jun 25, 2022, 8:06 PM

#

HI

#

IM HERE

#

WHAT'D YOU NEED

tender nimbus Jun 25, 2022, 8:07 PM

#

one fuzzing, please

haughty mica Jun 25, 2022, 8:07 PM

#

https://github.com/aws/dcv-color-primitives

GitHub

GitHub - aws/dcv-color-primitives: DCV Color Primitives Library

DCV Color Primitives Library. Contribute to aws/dcv-color-primitives development by creating an account on GitHub.

tough leaf Jun 25, 2022, 8:07 PM

#

okay :3

haughty mica Jun 25, 2022, 8:07 PM

#

Image conversion

tough leaf Jun 25, 2022, 8:07 PM

#

oooh

#

i wonder if they have a security.md :(

#

they don't

#

cool

haughty mica Jun 25, 2022, 8:07 PM

#

I have yet to find one

west violet Jun 25, 2022, 8:07 PM

#

AWS ACE time ferrisVibe

tough leaf Jun 25, 2022, 8:07 PM

#

i've not once found ACE through a fuzzer

#

in rust

west violet Jun 25, 2022, 8:07 PM

#

That's fair, it's a little harder in rust

haughty mica Jun 25, 2022, 8:08 PM

#

I'm mostly suspicious of how this handles odd sizes/strides. The color transformation is just basic bit twiddling

#

Oh wait they do have a security policy lmao

tough leaf Jun 25, 2022, 8:16 PM

#

i don't care

#

:)

haughty mica Jun 25, 2022, 8:17 PM

#

Yeah uh so this link takes me to a big page and I can't figure out what to do from here 😂

#

Okay going for the generic AWS one

tough leaf Jun 25, 2022, 8:18 PM

#

You can not use any other library function (also in other threads) while the initialization is in progress. Failure to do so result in undefined behaviour

#

> on a safe function

#

static mut GLOBAL_STATE: GlobalState = GlobalState

#

there you go

#

that'll do it

west phoenix Jun 25, 2022, 8:20 PM

#

I was literally about to post that lmao

haughty mica Jun 25, 2022, 8:20 PM

#

Yeah I saw that

#

I was trying to figure out how to do that in a fuzzer and was like naw

tough leaf Jun 25, 2022, 8:20 PM

#

once_cell in a static

#

well

#

once

#

static INIT_ONCE: std::sync::Once = std::sync::Once::new();
...

INIT_ONCE.call_once(|| {
    dcp::initialize();
});

#

the fuzzer itself isn't multi-threaded so this seems fine

haughty mica Jun 25, 2022, 8:22 PM

#

Ah

tough leaf Jun 25, 2022, 8:22 PM

#

also i think maybe libfuzzer has a init callback

#

since i'm assuming C++ code also wants to do setup code

haughty mica Jun 25, 2022, 8:26 PM

#

I sent AWS a message. If they don't get back to me in 24 hours, PR goes up.

haughty mica Jun 25, 2022, 9:37 PM

#

                        vertices: [[180, 200], [260, 200], [260, 150], [180, 150]].as_mut_ptr(),

#

I'm deep enough that I'm now seeing jetscii's users

#

😩

pastel lily Jun 25, 2022, 10:53 PM

#

tough leaf > You can not use any other library function (also in other threads) while the i...

what the fuck

tender nimbus Jun 25, 2022, 10:57 PM

#

pastel lily what the fuck

it will be fine if you're careful

pastel lily Jun 25, 2022, 10:57 PM

#

ferrisClueless

#

who wouldn't be careful

tender nimbus Jun 25, 2022, 10:58 PM

#

i would be very careful

haughty mica Jun 26, 2022, 5:00 AM

#

Somewhere in this file is a write which is a stack buffer overflow

#

https://docs.rs/wdg-base32/0.6.1/src/wdg_base32/include/decode/decode.rs.html#1-78

decode.rs.html -- source

Source to the Rust file .cargo/registry/src/github.com-1ecc6299db9ec823/wdg-base32-0.6.1/src/include/decode/decode.rs.

#

chefskiss

#

I will not be debugging this

#

Holy shit this other crate is mostly use after frees

#

facepalm

#

Every other line is CString::new().as_ptr()

#

Ok I made it sound worse than it is, they only did the pattern 37 times

tender nimbus Jun 26, 2022, 7:01 AM

#

why do people write unsafe code

golden summit Jun 26, 2022, 7:03 AM

#

haughty mica Ok I made it sound worse than it is, they only did the pattern 37 times

only??

tender nimbus Jun 26, 2022, 7:08 AM

#

golden summit *only*??

could be worse

#

for example, 38 times

golden summit Jun 26, 2022, 7:11 AM

#

or god forbid 39 times

knotty oar Jun 26, 2022, 7:23 AM

#

oh damn can't imagine 42 times, that would be the way of life

tender nimbus Jun 26, 2022, 9:36 AM

#

yoooo #960647106056581150 message @haughty mica ferrisFlushed

#

(on the eyre community discord)

#

tough leaf Jun 26, 2022, 10:04 AM

#

tender nimbus

yeah we were chatting the other day and I sent it to her

tender nimbus Jun 26, 2022, 10:04 AM

#

ferrisOwO

#

good

knotty oar Jun 26, 2022, 10:09 AM

#

lol

thorny venture Jun 26, 2022, 11:35 AM

#

when the behaviour is uwu

tender nimbus Jun 26, 2022, 11:37 AM

#

uwundefined behaviour

haughty mica Jun 26, 2022, 2:47 PM

#

Very cool

#

Also smallvec released a new version so they can be off the site soon

tender nimbus Jun 26, 2022, 2:52 PM

#

ferrisOwO

thorny venture Jun 26, 2022, 2:54 PM

#

I wonder if I can get small vec + storage to work ferrisBigBrain

#

(safely owo)

haughty mica Jun 26, 2022, 3:04 PM

#

You mean tinyvec?

tame jewel Jun 26, 2022, 4:43 PM

#

tough leaf > You can not use any other library function (also in other threads) while the i...

Yeah, I've seen that and discarded the whole crate as a C-ism port..Oops, should have realized its actually being used

#

And then started to write an alternative anyways ferrisHmm

tough leaf Jun 26, 2022, 4:47 PM

#

Lazy in the stdlib is nightly only 😭

#

i'll tell them to use once_cell or lazy_static then I guess

tame jewel Jun 26, 2022, 4:53 PM

#

Indeed. Did it pass miri fuzzing yet? The whole unsafe business with pointers seems very nonlocal … and hardly motivated.
No bounds anywhere, all slices immediately cast to raw pointers on function entry. The simd modules do a lot of bounds-check logic themselves, instead of being shared and passing some sort of wrapper. Tbh, one too many complexities to seriously take a look at soundness.

tough leaf Jun 26, 2022, 4:54 PM

#

i don't think it would run under miri because SIMD?

tame jewel Jun 26, 2022, 4:54 PM

#

There's even a custom bswap(i32) in there

#

Jeez.

tough leaf Jun 26, 2022, 4:54 PM

#

oh, it also uses cpuid

tame jewel Jun 26, 2022, 4:55 PM

#

And it's unsafe.

tough leaf Jun 26, 2022, 4:55 PM

#

and miri does not support cpuid

tame jewel Jun 26, 2022, 4:55 PM

#

Yep, not quite yet, wasn't there work-in-progress? I don't quite know where it's at

tough leaf Jun 26, 2022, 4:55 PM

#

anyways it's corporate code i don't work for free :)

#

except when i do (see: sentry)

tame jewel Jun 26, 2022, 4:56 PM

#

Ah, portable simd is implemented but none of the arch specific versions

tame jewel Jun 26, 2022, 4:57 PM

#

tough leaf anyways it's corporate code i don't work for free :)

Just help out with my crate instead, it's not corporate ferrisPlead

tough leaf Jun 26, 2022, 4:57 PM

#

tame jewel Just help out with my crate instead, it's not corporate <:ferrisPlead:8574408458...

half the image-rs crates can OOM
boom, found your bugs

#

oh should i fix them too

#

:D

tame jewel Jun 26, 2022, 4:57 PM

#

A little update on this: the issue of supporting SIMD in MIR-based Rust verification tools came up at the Rust Verification Workshop at ETAPS, and there is a lot of interest in finding a shared solution. One avenue we are considering is to have a pure Rust implementation of those intrinsics; tools consuming MIR or LLVM IR should be able to compile that implementation to the required IR and replace intrinsic calls with that IR.

_Originally posted by @RalfJung in https://github.com/rust-lang/miri/issues/662#issuecomment-817122692_

GitHub

SIMD instructions are not supported · Issue #662 · rust-lang/miri

constant evaluation error: tried to call a function with ABI RustIntrinsic using caller ABI PlatformIntrinsic --> …/nightly-2019-03-15-x86_64-apple-darwin/lib/rustlib/src/rust/src/libcore/.....

tough leaf Jun 26, 2022, 4:58 PM

#

honestly i'm not that happy with the state of preallocation OOMs

#

and like

#

i don't know a sound way to fix it

tame jewel Jun 26, 2022, 4:58 PM

#

tough leaf half the image-rs crates can OOM boom, found your bugs

Right, except tiff and then we get complaints you can't open a 32GB image without setting custom limits ferrisUpsideDown

tough leaf Jun 26, 2022, 4:58 PM

#

haha

#

yeah idk

#

i wonder if just taking a generic allocator that you use for all allocations then implementing limits as an allocator with a cap would be good

#

you'd need falliable allocation then though

#

on all data structures you want to use

tame jewel Jun 26, 2022, 5:01 PM

#

Also I was more referring to new stuff, color buffers and color management. Decoders and Encoders are a totally different deal.
Well, slowly. What recently came up: we really need a unified way to have core-io that's SemVer compatible with std-io..
So that crates can offer std-io as a feature with no SemVer incompatibility from adding the feature.
And we'd also really like having optional-Seek-readers.

tame jewel Jun 26, 2022, 5:02 PM

#

tough leaf i wonder if just taking a generic allocator that you use for all allocations the...

Not really, I don't think customizing data structures over an allocator (as in, the same trait/concept for different data structures) is that good of a solution.

#

There's many details with specifics of how the allocator is used, depending on data structure and you don't want that in the interface

#

Or all the interfaces to have all the details

tough leaf Jun 26, 2022, 5:03 PM

#

wait hm?

#

not sure what you mean

tame jewel Jun 26, 2022, 5:04 PM

#

Well, sometimes you allocate as an optimization

tough leaf Jun 26, 2022, 5:04 PM

#

do you mean the Allocator trait is bad

tame jewel Jun 26, 2022, 5:04 PM

#

It's suboptimal to implement that policy via an allocator, what I'm saying.

tough leaf Jun 26, 2022, 5:04 PM

#

oh hm

#

ig

tame jewel Jun 26, 2022, 5:05 PM

#

There's contextual information you'll want to have, or share with other parts, and those shouldn't all be owned by the allocator handle type, and writing an unsafe wrapper around the allocator (because the allocator trait is unsafe) is a rather unfortunate too

#

What about concurrent allocation? Do you then use atomics to track actual use or is a rough policy (up to a constant factor, or a constant overhead) good enough?
Performance implications of that? How to you even express 'allocate n bytes, but really if you can't then m bytes are enough, just please never block any code, and also I'd like to store the result there because some work-stealing task elsewhere is going to take that and finish the rest you couldn't allocate with worse performance afterwards instead`'.
Nah, it's omplicated enough to work that out in a dedicated unit of code without be constrainted by the allocator interface.

#

The trait is actually in a fine place. If anything, I'm question if we should not decouple stabilizing the new allocator trait/concept from stabilizing the type parameters on standard containers more. Because providing a mechanism of alignment-changing-realloc is so much more relevant than being able to customize all std-containers with arbitrary policies. If you don't like the policy choices that std can offer then you'll write your own containers and this is fine. Tbf, the fact that C++ containers are the most relevant comparison make me worried that this ends up copying too much without having the same reasons.
In the end, Linux and other projects will code their own containers and allocator handles anyways. (And be it just to interoperate with vtable-based C libraries which, btw, mostly work by taking three simple method pointers alloc,realloc,dealloc.) The most relevant question is then: which shared abstractions can we offer so that programming these is as safe and sound as possible. Anything else doesn't have the same priority in my eyes.

tender nimbus Jun 27, 2022, 8:35 PM

#

@haughty mica do you plan on using permissive provenance in the next run once it becomes stable?

haughty mica Jun 27, 2022, 8:36 PM

#

I could use it now

#

Answer is... maybe?

#

It's worth a try

tender nimbus Jun 27, 2022, 8:38 PM

#

I think having such a run would be useful

haughty mica Jun 27, 2022, 8:38 PM

#

Actually this would be a great way to find stuff

tender nimbus Jun 27, 2022, 8:39 PM

#

Not displaying the trivially fine ptr2int "UB" while still catching interesting cases around pointer misuse

haughty mica Jun 27, 2022, 8:40 PM

#

Well the thing is that previously, ptr2int was actually UB

#

SB with Untagged does not permit -Zmutable-noalias, which was on by default

tender nimbus Jun 27, 2022, 8:41 PM

#

I meant int2ptr ferrisballSweat

haughty mica Jun 27, 2022, 8:41 PM

#

ptr2int2ptr actually

#

Yeah

tender nimbus Jun 27, 2022, 8:41 PM

#

The kind of thing no provenance model would dare to break

#

Well, no "real model" in the tower of weakenings

#

strict provenance very much breaks it ferrisOwO

tough leaf Jun 27, 2022, 8:44 PM

#

tender nimbus The kind of thing no provenance model would dare to break

honestly there's not a lot of code that can't pass strict provenance

#

the two examples are XOR linked lists and FFI

tender nimbus Jun 27, 2022, 8:45 PM

#

I'm not talking about theoretical but actual code

#

Which could totally be rewritten, but doesn't have to

tough leaf Jun 27, 2022, 8:46 PM

#

okay, what's actual code that doesn't follow it and can't

#

well

#

AtomicPtr needs manipulation ops

#

which it's getting

haughty mica Jun 27, 2022, 8:47 PM

#

XOR linked lists are dumb and people should stop bringing them up

#

You can always invent some diabolical use case that breaks any reasonable model. The reality is that they are incredibly niche, and if you can't implement XOR linked lists in Rust that is a fine price to pay for a reasonable memory model.

tough leaf Jun 27, 2022, 8:48 PM

#

agreed

haughty mica Jun 27, 2022, 8:48 PM

#

FFI is a lot more interesting

tough leaf Jun 27, 2022, 8:49 PM

#

you'd need a "this for FFI purposes is a (int, long, whatever) but for opt purposes is a pointer" ig

haughty mica Jun 27, 2022, 8:50 PM

#

FFI is the boundary of the Rust AM so I really don't know if you can just fib and use your platform knowledge to keep everyone happy, though that's more a question for #932319394724479037

#

Ok got 4840 crates in the queue

haughty mica Jun 27, 2022, 9:24 PM

#

@tender nimbus https://github.com/rust-lang/miri/issues/2133#issuecomment-1167889063

GitHub

The plan for provenance · Issue #2133 · rust-lang/miri

#2059 is nearing completion, so I figured it would make sense to lay down my current plan for the future of provenance in Miri. My idea is that, by default, we will work in "permissive&...

#

int2ptr is still unsound because angelic nondet isn't checkable

#

I'm going to use the default mode, and probably come up with some other kind of categorization

haughty mica Jun 27, 2022, 9:47 PM

#

lmao found some legit issue in crossbeam-epoch

#

I like this

haughty mica Jun 27, 2022, 10:38 PM

#

I think a large fraction of crates are graduating to timeouts ugh

#

Maybe I should version the bucket

haughty mica Jun 27, 2022, 11:51 PM

#

Ralf is forking concurrency stuff again

#

10/16 wildcard errors so far are string_cache

tender nimbus Jun 28, 2022, 4:16 AM

#

ferrisballSweat

west violet Jun 28, 2022, 4:04 PM

#

If you want freebies on UB take a gander at https://www.mmtk.io/

#

It's a cool project so I'd really like to see it sound-ified tbh

haughty mica Jun 28, 2022, 4:26 PM

#

https://miri.saethlin.dev/ub?crate=mmtk&version=0.12.0

#

Not a lot of tests

haughty mica Jun 28, 2022, 9:23 PM

#

1,000 crates left to run in the previously-ub list

#

I'm rerunning all the previously UB crates with new Miri

tame jewel Jun 28, 2022, 9:30 PM

#

rayon 1.5.3
UB: deallocating while item is protected: [SharedReadOnly for <412516> (call 112693)] (rayon-core-1.9.3), deallocating while item is protected: [SharedReadOnly for <4513936> (call 1177075)] (rayon-core-1.9.3), deallocating while item is protected: [SharedReadOnly for <716428> (call 189531)] (rayon-core-1.9.3)
Okay, what's that about

tough leaf Jun 28, 2022, 9:46 PM

#

is this the atomic bomb

#

in Arc

haughty mica Jun 28, 2022, 10:05 PM

#

Could be

#

Oh wait no

#

Rayon does the same pattern as Arc, which I think was made valid recently

#

Updates like that are tricky to do

#

I have built in a way to rerun just crates with new versions but it's so hard to know if the validity of a crate has changed due to changes in the model

#

Miri's weak memory emulation uncovered a lot of data races that were previously undetected, and also I'm not running things with more that one thread so that will probably uncover more too

#

Just a few minutes left now ablobhungry

#

I think I'm misattributing a lot of UB to once_cell

#

histogram over kind of UB
uninitialized memory
  773 ##################################################
SB-use-outside-provenance
  719 ##############################################
null pointer dereference
  695 ############################################
SB-invalidation
  608 #######################################
type validation failed: encountered uninitialized raw pointer
  254 ################
&->&mut
  152 #########
misaligned pointer dereference
  144 #########
dereferencing pointer failed: [hex] is not a valid pointer
  132 ########
SB-uncategorized
  100 ######
invalid pointer offset
   74 ####

#

That's a lot of use outside provenance now

#

I almost wonder if that's incorrect because the wildcard diagnostics are incomplete

#

Of 68 crates which report wildcard errors, 32 are due to string_cache

haughty mica Jun 28, 2022, 11:34 PM

#

Oh exposed tags need to still be valid

#

Oh dear me

#

R I P string_cache I will work on something else

haughty mica Jun 29, 2022, 12:54 AM

#

L M A O

I just realized that because I never delete stuff from the bucket, build logs for old crates stick around

haughty mica Jun 29, 2022, 1:05 AM

#

tame jewel > rayon 1.5.3 > UB: deallocating while item is protected: [SharedReadOnly for <4...

I think this is a legitimate report, and it is a bug. Someone should report this, I'm out for the night

tame jewel Jun 29, 2022, 7:00 AM

#

Huh. I can't reproduce it locally. Maybe it requires a very recent miri …

#

Or it's the --test-threads=2 argument

haughty mica Jun 29, 2022, 12:50 PM

#

I can repro with just isolation off, on the latest nightly

tame jewel Jun 29, 2022, 1:27 PM

#

It doesn't always happen at the exact same test for me.. Which one is it for single threaded tests?

#

That is, is it deterministic then? I really hope so

#

Anyways: https://github.com/rayon-rs/rayon/issues/952

GitHub

Miri detects undefined behavior in the test suite · Issue #952 · ra...

(Found while scrolling through https://miri.saethlin.dev/ub?crate=rayon&version=1.5.3) Undefined Behavior: deallocating while item is protected: [SharedReadOnly for <2273653> ...

haughty mica Jun 29, 2022, 2:42 PM

#

I would not count on it being deterministic

#

You need to disable isolation for num_cpus to work

#

Why did you set -Zmiri-backtrace=full?

tame jewel Jun 29, 2022, 2:54 PM

#

haughty mica Why did you set `-Zmiri-backtrace=full`?

I was curious what additional information it might provide. For some reason I missed the test case name on your site initially :)

haughty mica Jun 29, 2022, 2:56 PM

#

Ah. It's supposed to be the analogue of RUST_BACKTRACE=full

#

It's meant to prune away frames which are just noise unless you're debugging the Rust runtime. If it's pruning away anything you care about that would be very important to know

haughty mica Jun 29, 2022, 2:57 PM

#

tame jewel I was curious what additional information it might provide. For some reason I mi...

Well, the scroll is set to be the UB report line, not the one right above it 🤔 maybe I should adjust that

tame jewel Jun 29, 2022, 2:57 PM

#

Hm, yes, might be a quick and subtle ux win

#

Is there some way to find the line number corresponding to the SharedReadOnly tags?

#

I.e. get miri to print how those came to be on the stack in the first place

haughty mica Jun 29, 2022, 2:58 PM

#

What report are you looking at?

tame jewel Jun 29, 2022, 2:58 PM

#

The one from rayon. I'm wondering if we can find out which reference is still alive when the stack frame is deallocated

haughty mica Jun 29, 2022, 2:59 PM

#

OH

#

Yeah

#

:/

#

I punted on implementing something for protector errors, but if you're rerunning locally you can do a MIRIFLAGS=-Zmiri-track-pointer-tag=9230543 but you'll end up with an FnEntry retag so things might look a bit odd

tame jewel Jun 29, 2022, 3:32 PM

#

   --> rayon-core/src/latch.rs:372:5
    |
372 | /     fn set(&self) {
373 | |         L::set(self);
374 | |     }
    | |_____^ created tag 2782933
    |
    = note: inside `<&latch::LockLatch as latch::Latch>::set` at rayon-core/src/latch.rs:372:5
note: inside `<job::StackJob<&latch::LockLatch, [closure@rayon-core/src/registry.rs:465:17: 469:18], ((), ())> as job::Job>::execute` at rayon-core/src/job.rs:114:9
   --> rayon-core/src/job.rs:114:9
    |
114 |         this.latch.set();
    |         ^^^^^^^^^^^^^^^^

Is there maybe a worker thread alive, still waiting before on a mutex? (in rayon-core/src/latch.rs:LockLatch::lock it tries to lock a mutex).

haughty mica Jun 29, 2022, 3:40 PM

#

I'm hesitant to answer that confidently

#

I suspect this is the same problem as Arc had

#

Strange... I feel like this whole things should end up being retagged SRW because it's all atomics

west violet Jun 29, 2022, 4:29 PM

#

@haughty mica since there’s no perf degradation in my miri pr is it good?

haughty mica Jun 29, 2022, 4:39 PM

#

No (I'd rather discuss this elsewhere, because this is not about fixing UB)

tame jewel Jun 29, 2022, 7:39 PM

#

haughty mica Strange... I feel like this whole things should end up being retagged SRW becaus...

Well the cell around an atomic is tagged SharedReadOnly

haughty mica Jun 29, 2022, 7:42 PM

#

tame jewel Well the cell around an atomic is tagged SharedReadOnly

Oh dear

#

So this needs either a fix in rayon-core or the full fix Ralf suggested

#

https://github.com/rust-lang/rust/issues/55005#issuecomment-1166611797

GitHub

Arc::drop has a (potentially) dangling shared ref · Issue #55005 · ...

Discovered by @Amanieu on IRLO. Quoting their report: Arc::drop contains this code: if self.inner().strong.fetch_sub(1, Release) != 1 { return; } Once the current thread (Thread A) has decremented ...

tame jewel Jun 29, 2022, 8:07 PM

#

Hm okay, finally understood the relation. Makes sense. What if Drop of the lock guard would be guaranteed to occur like a tail-call, through the whole call chain?
Then the order would be first removing the stack protectors or parents, then actually making the atomic subtract visible. Well, seems hard to solve in a decently safe manner, without turning every parameter into *const _..

haughty mica Jun 29, 2022, 8:09 PM

#

Yes that is exactly it

tame jewel Jun 29, 2022, 8:09 PM

#

But it's still confusing why it would only show up for that test, not for all other tests.

haughty mica Jun 29, 2022, 8:10 PM

#

If you comment that test out, do the others after it all pass?

tame jewel Jun 29, 2022, 8:12 PM

#

Many other's do, yes. join_context_second leaks memory but works

#

There's a large chunk of tests that show as already passed in your site too.
What's the condition for another thread having taken a task, and why is it so consistent in that case

haughty mica Jun 29, 2022, 9:10 PM

#

I will offer up that in the Arc example linked above, one needed hundreds of iterations to get the timing that actually hits the bug.

I wonder if setting -Zmiri-preemption-rate=1 or some other very high value makes these more likely to hit

#

Or I wonder if there's some other way to make this more likely

tough leaf Jun 30, 2022, 4:41 PM

#

simd-json's tests get SIGILLed when you build-std

#

hmmmm

#

unsafe {
    input_buffer
        .as_mut_slice()
        .get_unchecked_mut(..len)
        .clone_from_slice(input);
    *(input_buffer.get_unchecked_mut(len)) = 0;
    input_buffer.set_len(len);
};

hmmmm

#

yes this is a get_unchecked_mut out of bounds

knotty oar Jun 30, 2022, 4:42 PM

#

lol

tough leaf Jun 30, 2022, 4:45 PM

#

now, what's the least bad way to do this....

#

there's MaybeUninit::write_slice but that's unstable

#

pointer methods it is

tender nimbus Jun 30, 2022, 4:47 PM

#

a little bit of ptr::copy_nonoverlapping maybe

tough leaf Jun 30, 2022, 4:48 PM

#

yep

tough leaf Jun 30, 2022, 6:00 PM

#

oh no this seems like the kind of program that Really depends on indexing out of bounds

#

and their CI seems fucked rn

#

fu

#

n

tender nimbus Jun 30, 2022, 6:14 PM

#

simd-json is very cursed

#

but the maintainer is open to fixes

tough leaf Jun 30, 2022, 6:15 PM

#

oh i know
there's just

a lot to fix

tender nimbus Jun 30, 2022, 6:16 PM

#

A Whole Lot

#

And Miri can't help ferrisballSweat

tough leaf Jun 30, 2022, 6:17 PM

#

might add a compiler option for verbose errors on failure of a unsafe precondition

#

idk

#

so it just panics

#

with a nice error message

tender nimbus Jun 30, 2022, 6:18 PM

#

ferrisOwO

#

Would be good

haughty mica Jun 30, 2022, 6:50 PM

#

Miri could help

#

It's just simd 4head

tough leaf Jun 30, 2022, 6:53 PM

#

that and no one's going to be running a webserver in prod under miri, but you might run at least some under debug assertions

tender nimbus Jun 30, 2022, 8:34 PM

#

yeah one would run a webserver in Miri because it can't even work ferrisSob

tough leaf Jun 30, 2022, 10:32 PM

#

tender nimbus but the maintainer is open to fixes

i went through and made every single get_unchecked actually just a normal index, if you have debug assertions enabled
guess how many tests pass

haughty mica Jun 30, 2022, 10:33 PM

#

None

tough leaf Jun 30, 2022, 10:33 PM

#

the correct answer is none, yes

#

this feels like a thing the stdlib could do better at

haughty mica Jun 30, 2022, 10:33 PM

#

Please complain to libs

#

I'd be happy to improve the debug assertion output but I would want a way to pick which version is in use

#

I have no idea what that would be

tough leaf Jun 30, 2022, 10:34 PM

#

well, we have #[rustc_inherit_overflow_checks] don't we

#

i've got no idea how that works

haughty mica Jun 30, 2022, 10:35 PM

#

No I mean the abort version is faster and I want to be able to select that

tough leaf Jun 30, 2022, 10:35 PM

#

oh

haughty mica Jun 30, 2022, 10:35 PM

#

But we should have a way to emit something helpful too

tough leaf Jun 30, 2022, 10:35 PM

#

i mean right now it could be a feature flag on core

haughty mica Jun 30, 2022, 10:35 PM

#

The UX is brutal but yes.

tough leaf Jun 30, 2022, 10:36 PM

#

well, if you're building std it's not much worse

#

but yes

haughty mica Jun 30, 2022, 10:36 PM

#

Just don't ask for too much help in the errors

#

You can have a stringify of the expression that's false and a panic but that's probably it, unless we do every one individually

tough leaf Jun 30, 2022, 10:38 PM

#

the way i imagine it, it would be a SIGILL with no formatting under stock settings (or maybe under a "immediate_abort" setting, idk), and a debug_assert showing as much info as we have and that would be useful
so yes that would include string formatting imo

#

like, if you go to read from an unaligned address, we should probably mention the address

haughty mica Jun 30, 2022, 10:39 PM

#

Argh so you want a case by case check

tough leaf Jun 30, 2022, 10:40 PM

#

wdym case by case
every function needs hand-written debug assertions specifically for that function, no?

haughty mica Jun 30, 2022, 10:40 PM

#

But not case by case formatting

tough leaf Jun 30, 2022, 10:40 PM

#

isn't case by case formatting what debug_assert does

#

is this for perf reasons or maintenance reasons

haughty mica Jun 30, 2022, 10:41 PM

#

https://github.com/rust-lang/rust/blob/7425fb293f510a6f138e82a963a3bc599a5b9e1c/library/core/src/intrinsics.rs#L2444

GitHub

rust/intrinsics.rs at 7425fb293f510a6f138e82a963a3bc599a5b9e1c · ru...

Empowering everyone to build reliable and efficient software. - rust/intrinsics.rs at 7425fb293f510a6f138e82a963a3bc599a5b9e1c · rust-lang/rust

#

Me being lazy reasons

tough leaf Jun 30, 2022, 10:42 PM

#

ahhh

#

so the issue is you'd need 3 assertions there

#

instead of just the one

haughty mica Jun 30, 2022, 10:42 PM

#

Yup

tough leaf Jun 30, 2022, 10:42 PM

#

ah right

haughty mica Jun 30, 2022, 10:42 PM

#

Really you need 5

#

And many pointer APIs need null + aligned

tough leaf Jun 30, 2022, 10:45 PM

#

those assertions also don't check that src/dest + sizeof(T)*count doesn't overflow
but i Highly Highly Highly doubt any crate is doing that, seeing as that is going to segfault

#

but yeah

#

i think macros for common cases would be good
like

pub const unsafe fn copy_nonoverlapping<T>(src: *const T, dst: *mut T, count: usize) {
    extern "rust-intrinsic" {
        #[rustc_const_stable(feature = "const_intrinsic_copy", since = "1.63.0")]
        pub fn copy_nonoverlapping<T>(src: *const T, dst: *mut T, count: usize);
    }

    // SAFETY: the safety contract for `copy_nonoverlapping` must be
    // upheld by the caller.
    unsafe {
        debug_assert_aligned_non_null!(src, dest);
        debug_assert_nonoverlapping!(src, dest, count);

        copy_nonoverlapping(src, dst, count)
    }
}

#

maybe don't make debug_assert_aligned_non_null variadic that seems maybe confusing

#

but this is both easier to read imo, as well as would easily allow verbose formatting on failure

#

and then one-off debug assertions can either be moved to a new macro or have the definition of debug_assert changed to have a abort() depending on the feature flag

#

since we already have quite a few one-off debug assertions already for stuff like Option::unwrap_unchecked

#

those probably aren't as hot of a path as copy_nonoverlapping though

haughty mica Jul 3, 2022, 6:57 PM

#

Lmaoooo my SB optimizations make more crates appear as UB because their tests don't time out anymore

pastel lily Jul 3, 2022, 6:57 PM

#

lmao

tough leaf Jul 3, 2022, 7:02 PM

#

stonks

#

memory sanitizer has no way to go "hey, tell me if any of the bytes in this value are uninit" :(
it doesn't even seem to care about noundef metadata??

west violet Jul 4, 2022, 1:28 PM

#

Ok, I actually thought up another use for CodeQL with rust, one that could actually be super valuable

#

It could act as a replacement for crater in some circumstances

#

It can answer questions like "how many people call X", "how much does this pattern get used", etc.

#

Even stuff like "most ignored lint" and whatnot

knotty oar Jul 4, 2022, 1:34 PM

#

sounds interesting

haughty mica Jul 4, 2022, 3:46 PM

#

Wait doesn't Google do this with C++ internally

tender nimbus Jul 4, 2022, 4:01 PM

#

Meta would not be able to do this since they have no idea where there code is apparently ferrisBut

west violet Jul 4, 2022, 4:04 PM

#

haughty mica Wait doesn't Google do this with C++ internally

Most big companies probably do

#

Code databases aren't exactly new tech

haughty mica Jul 4, 2022, 4:05 PM

#

Yeah I'm sure Facebook does not have such a tool

#

I know that Google has a good way to do queries and rewrites of theirs, based on semantic analysis

#

You can't get hung up on generics, for example

tough leaf Jul 6, 2022, 2:32 PM

#

does llvm really just refuse to let you enable msan checks on noundef loads
is that not implemented, rather

#

im upsetti

tender nimbus Jul 6, 2022, 2:32 PM

#

ferrisPensive

tough leaf Jul 6, 2022, 2:32 PM

#

i did ask in the llvm discord

#

basically "heyyyyy is there a way we can do this"

#

because like

#

it would be Very Good

tender nimbus Jul 6, 2022, 2:33 PM

#

implement it ferrisClueless

tough leaf Jul 6, 2022, 2:33 PM

#

no, llvm is C++

#

if i did my MIR sanitizer

#

then that could do it

#

by knowing what bits of every value are init

#

and touching them

#

(well, branching on their value)

#

-fsanitize-address-param-retval

#Fixing UB in random crates