Parsing an index operator with nibble

[fossil peak]

Heyo!
I've started working on implementing a CEL (Common Expression Language) lib in Gleam, using nibble for parsing and would like some help. It's gone quite well so far but I'm now struggling to add the "member index" syntax, variable[index]. I did somehow manage to get a (seemingly) working ternary operator working condition ? true_branch : false_branch but I've spent too much time now on getting anything at all working to parse the index operator. The expression parsing function can be found here. My parsing knowledge is very fresh so please do let me know what things I should do differently! 🙏

I've tried to add some kind of intermediate state like I did with the ternary operator but I never seem to successfully "close" it with the attempted postfix right square bracket. Any ideas or pointers would be greatly appreciated!
(Moved from #general)

[terse sentinel]

pub fn index(config) {
  use expr <- do(pratt.subexpression(config, 0))
  use _ <- do(token(LBracket))
  use idx <- do(pratt.subexpression(config, 0))
  use _ <- do(token(RBracket))

  return(Index(expr, idx))
}

i wouldnt treat it as an operator personally

[fossil peak]

Should I place that in the one_of: of the pratt.expression? That seemingly causes the infinite left-recursion again?

[fossil peak]

Sorry for bothering you @terse sentinel but I've really tried to get the indexop and ternary parsing working without any ugly hacking with infix operators but I haven't been able to. I would so much appreciate some more direction. Thank you 🙏

I assume this is not the right way to go about it but I can't come up with anything else that makes sense really?

  pratt.expression(
    one_of: [
      atom_expr_parser,
      ident_parser,
      parens_parser,
      list_parser,
      map_parser,
      pratt.prefix(8, nibble.token(lexer.ExclamationMark), not),
      pratt.prefix(8, nibble.token(lexer.Minus), unary_sub),
      member_index_parser,
    ],
    and_then: [
      pratt.infix_left(9, nibble.token(lexer.Dot), member_attribute),
      pratt.infix_left(7, nibble.token(lexer.Star), mul),
// ...
      pratt.infix_left(3, nibble.token(lexer.Or), or),
      pratt.infix_right(
        2,
    ],
    dropping: nibble.succeed(Nil),
  )
}

fn member_index_parser(conf) -> Parser(Expression, lexer.Token, Context) {
  use expr <- nibble.do_in(InSubExpr, pratt.sub_expression(conf, 0))
  use _ <- nibble.do(nibble.token(lexer.LeftSquare))
  use index <- nibble.do_in(InSubExpr, pratt.sub_expression(conf, 0))
  use _ <- nibble.do(nibble.token(lexer.RightSquare))

  nibble.return(Member(expr, Index(index)))
}

[terse sentinel]

theres nothing magic about this stuff you just have to keep breaking things down into parts that compose. you have a list of parsers there that neatly model "leaf nodes in an expression tree", all the stuff in one_of.

array index is <leaf node> [ <expresion> ] so...

one_of: fn(config) {
  // try all the normal boring stuff
  use leaf <- nibble.do(nibble.one_of([
    atom_expr_parser(config),
    ident_parser(config),
    ...
  ]))
  // see if what follows is a `[`
  use lbracket <- nibble.do(nibble.optional(nibble.token(LBracket)))

  case lbracket {
    // if it wasnt, just return the expr
    option.None -> nibble.return(leaf)
    // otherwise proceed with parsing member index
    option.Some(_) -> {
      use index <- nibble.do(pratt.sub_expression(config, 0))
      use _ <- nibble.token(RBracket)

      nibble.return(Member(leaf, Index(index)))
    }
  }
}

[fossil peak]

Aaah, of coure... That makes more sense. I tried breaking it down within the one_of list but then I kept running into the same wall. Of course I need to construct it differently! Thanks so much. I think I should be able to figure it out from here!

[terse sentinel]

dunno how much help it'd be seeing it in not-gleam but here im doing a similar thing

[fossil peak]

I think it's finally clicked! I've gotten everything parsing in a not-too-hacky-I-think-way, finally. Thanks a bunch for the help!

[terse sentinel]

wicked!

[fossil peak]

@terse sentinel One thing I've ran into a couple of times is that the parser "successfully" finishes (doesn't produce any errors) but the resulting expression doesn't contain stuff corresponding to all the lexed tokens. I'm currently trying to parse false ? 'hmm' : [1, 2, 3, 4].filter(x, x % 2 == 0) == [2, 4] ? 'yes' : 'no' but it parses false ? 'hmm' : [1, 2, 3, 4].filter(x, x % 2 == 0) == [2, 4] successfully and the rest/outer/rightmost ternary gets silently dropped. Is there any way to debug this nicely? I'm assuming something with the associativity is wrong but I'm not sure how to approach debugging here? Sorry if these questions are more about general parsing challenges than about nibble, if you have a good resource to point me at I'm all ears 🙏

[terse sentinel]

nibble has an eof parser that you can use to guarantee you're at the end of the input

there is also a way to inspect a parser when it runs

i know nibble doesnt have the most documentation but consulting the api is good to do too 🙂
https://hexdocs.pm/nibble/1.1.1/nibble.html#inspect