Dictionaries

[mellow kiln]

Discussions related to dictionaries

@hardy haven
Discussion about https://github.com/typst/typst/issues/2394

[hardy haven]

Hi everyone! I would love to hear some of the community's opinions on this matter ^^
in short, this proposes being able to pass arguments as fun(dict.nest.key = val) to set nested dictionaries. I believe this will allow for nicer use of some APIs and simplify set-rule folding logic.
For more details see the original issue—and thanks for tuning in :)

[zealous sluice]

I'm still unsure what the exact rules for folding s overriding would be

[hardy haven]

wdym overriding?

This puts the 'burden' of deciding when to fold completely on the call sites rather than the content being styled (except in special cases like strokes)

[zealous sluice]

well the example you specified was a little confusing for me

#set box(stroke.thickness: 5pt)
#box[This is a box with a thick stroke]

#set box(stroke.color: green)
#box[This is a box with a thick, green stroke]

#set box(stroke: (thickness: 1pt))
#box[This is a box with a thin, _black_ stroke]

at least in a real doc i can see this quickly becoming confusing

would it only fold when the path sytnax is used?

and shorthands?

[hardy haven]

Essentially, in the third set rule, you're specifying explicitly the stroke argument, rather than a field of it

So when you specify a field you override the field but not the whole argument

[zealous sluice]

mhm

[hardy haven]

when you specify an argument then you override the whole thing, including previously-specified fields

Again, this is all very similar to how toml does it, except that anything that would be an error in toml instead just resets as little as possible to make it no longer an error

[zealous sluice]

hmm

I'm torn

it seems helpful, but I don't have much of a need for I personally as I rarely adjsut styles in such a granular fashion, so I'm not sure how much of this is needed in documents

perhaps someone like @devout oyster or @honest estuary has more of an opinion considering they work on packages that have lots of styling options

ah pardon the ping i forgot @slient

[hardy haven]

now you've summoned them to cast their opinions into the flames, I can't complain

[devout oyster]

Well

Nix does this so it's not alien to me, but I'm not sure regarding its need in Typst, it's usually more useful when it's too nested

Also I'm not sure regarding having this affect fold mechanics, perhaps it should be a purely syntactical construct

Otherwise I think I'm mostly neutral

[remote nimbus]

some of these packages people are writing would greatly benefit from having this, as a sneakily way to ~just~ get going..

i've never seen it before.. but it makes sense, of course...

I kind of support it...............

[honest estuary]

I also like this idea. In cetz you often have nested dictionaries of depth 3+ when setting styles.

[meager herald]

If it's a purely syntactical construct (so that packages don't need to change their code in any way) then I like it, especially with the cetz use case mentioned above, though I personally haven't found the lack of this feature a nuisance.
It might complicate the parser, though, so I think there's a trade-off I'm not sure which end has more weight.

[naive finch]

It seems like many of these benefits can be obtained by a helper function... Once get rules are established, it will also solve the issue of what style options to override vs. merge., leaving not much of a fundamental need. So I'm inclined to say it's not worth the additional dedicated syntax

(As a casual observer, not a maintainer)

?r

#let hierarchical-merge(a, b) = {
  for (key, value) in b {
    if key not in a {
      a.insert(key, value)
    } else {
      let a-value = a.at(key)
      if type(a-value) == dictionary and type(value) == dictionary {
        a.insert(key, hierarchical-merge(a-value, value))
      } else {
        a.insert(key, value)
      }
    }
  }
  a
}

#let nested-from-dots(..keys-values) = {
  keys-values = keys-values.pos()
  let out = (:)
  for index in range(keys-values.len(), step: 2) {
    let key = keys-values.at(index)
    let value = keys-values.at(index + 1)
    for part in key.split(".").rev() {
      value = ((part): value)
    }
    out = hierarchical-merge(out, value)
  }
  out
}

#nested-from-dots("a.b.c", 5, "a.b.d", 7, "a.f", 6, "g", 9)

[hardy haven]

Package authors certainly will need to make no changes—the only logic that would change is the folding behaviour of set rules, in a way that should very minimally affect existing code

[mellow kiln]

Dictionaries

[meager herald]

Okay, dictionaries.

The first question is, (1) should it support other data types for keys? If the conclusion is no, then the rest can be skipped.

If the answer to (1) is yes, then (2) what other data types? Note not all types have Hash/Eq implemented. Maybe we should first support types that do? I think it's sensible to skip those that don't (at least for now), and it's not surprising to users, who use Typst for typesetting works, not general programming.

Notably, if (1)'s answer is yes, I think integers should definitely be supported, since users (including package authors) may wish to store IDs, "enum"-like integers, etc. to a dictionary. The IDs are not necessarily contiguous - like the paper currency bills 1,5,10,20,50,100 - so arrays are not good fits here.

(as for floats NaN, python hashes it to 0 until a recent change in 3.10. But I think the usage of typesetting doesn't need NaN that much so hashing all NaN to 0 seems acceptable to me.)

[mellow kiln]

the problem is also that NaN != NaN so independently of how it is hashed, it could not be looked up. I guess if we want to support float keys, we could still error on storing NaN.

[meager herald]

Yeah error on NaN sounds good to me as well

so.. do we have a consensus here or need more deliberation? @mellow kiln (i’m fine either way). If consensus-ed, is PR 3344 (that adds integers) a good first PR, or do you wish to see a PR that supports more types?

(boarding a plane now so might lose connection in a few min, forgot if this airline has free wifi..)

[mellow kiln]

I think we're getting closer, but we haven't quite reached consesus yet. In particular, I would like some more input from @devout oyster who was in favor of a separate map type. Moreover, there is the question of which types exactly should be supported as keys.

[meager herald]

On the proposal of a separate “map” type: though not absolutely against it, I’m not comfortable with a separate new “map” type because from a language design point of view, it feels unnecessarily redundant (as mentioned in a review comment on that pr). Sure, string-keyed dicts are indeed good fits for manipulating function arguments, yet a more versatile dict doesn’t break this usage (and 1. we can add a method to let users check keys’ homogeneity if so desired 2. we can add checks in typst internal to ensure the arguments spreaded from dicts are string-keyed).

[mellow kiln]

I agree with you, I just want to hear the other perspective too.

[meager herald]

(plane about to take off so throwing in a few thoughts) assuming we agreed on making the existing dict more versatile (am okay with disagreement on this point! no pressure 🙂 ).. on more data types as keys: if we agree on continuing with PR 3344, does it offer a good foundation that can be extended to more data types, or a new design is needed?

[devout oyster]

im still not sure

it sounds like we're intentionally giving up on performance with those checks

and , in the future , when we have type annotations and stuff, we'd need a separate generic argument for dict keys or something

i'd prefer if this complexity were opt-in

[meager herald]

to address the perf issue, how about simply not checking the non-string keys in arguments? Unexpected named argument keys will result in an “unexpected argument” error anyway.

[devout oyster]

that means we would have to allow the arguments type to take Values as keys, that doesnt seem to make much sense

[lethal pilot]

As a side note, should we add a dictionary-from-pairs constructor?

@limber bay pointed out you can do #for (k, v) in a.pairs() {((k): v)}, although that’s not obvious.

I’m drafting a PR but don’t know which syntax we’d prefer:

1️⃣ dictionary(a.pairs()) == a
2️⃣ dictionary(..a.pairs()) == a
3️⃣ dictionary.from-pairs(a.pairs()) == a
4️⃣ something else/don’t add

[devout oyster]

what's the current syntax for the constructor?

[lethal pilot]

No constructor apart from literals

[devout oyster]

hmmm

.from-pairs sounds pretty explicit

maybe that's better

[hardy haven]

Coming back to this, I think currently with call syntax mirroring array/dict syntax, there is a strong association between dictionary and arguments. If we support non-string keys, should we also support them in arguments? What would be the type of args.named()? What happens when you use the spread operator on a dict with non-string keys?

[smoky glen]

Hello to anyone who's still in here. How are dictionaries compared to arrays performance-wise in Typst, especially on a larger scale?

[frozen belfry]

Well arrays are vectors, and dictionaries are ordered hashmaps (currently using the IndexMap library). These objects have quite different performance characteristics in general. If you’re asking at a fundamental CS level, then you’ll be better off with general resources on the internet around the big-O characteristics of hashmaps and vectors.

But if you’re asking about how Typst uses them, then the more important thing is what your particular use case is. It’s very likely that whatever operations you’re doing with them in Typst will be massively overshadowed by the performance costs of layout and the overhead of Typst as a dynamic language in general.

But if you just want a bit of chatting about the performance characteristics, then give me something a bit more specific to go off of and I can likely oblige :)
(or maybe @placid stream would want to chime in)

[smoky glen]

Yes, I was only interested in Typst, as your first paragraph is already throwing me off :) I should have mentioned something about the use cases, but let's say it doesn't matter. I had a feeling that with array currently having many more methods at its disposal, picking dictionary would almost always result in the need to create your own hacky methods like changing insertion order, which I guess would be slower than the built-in ones that array for example has.