#Creating an XML parser. How should I deal with Unicode?

1 messages · Page 1 of 1 (latest)

wanton stream
#

I am using a unicode Utf8 Iterator. At first I was using the nextCodepointSlice method but I realized I need to "peek" the next character. So I wonder if I should use peek and advance the iterator afterwards or should I just use nextCodepointSlice?

# 
    const State = struct {
        tokens: std.ArrayList(Token) = undefined,
        fn tokenize(self: *State, bytes: []u8) !void {
            // Assume ASCII encoding
            // Convert into codepoint then determine if the codepoint is within range.
            _ = self;
            var it = std.unicode.Utf8Iterator{
                .bytes = bytes,
                .i = 0,
            };
            while (it.i < it.bytes.len) {
                const peek = it.peek(peekLen);
                const isAscii = switch (peek.len) {
                    1 .. peekLen => true,
                    else => false,
                };
                if (isAscii) {
                    if (peek.len == 1) {

                    } else if ()
                }
                const code = try std.unicode.utf8Decode(slice);
                if (!isValidCodepoint(code)) {
                    return error.invalidCodepoint;
                }
            }
    };
#

I think the code before I did this change was cleaner.

graceful quiver
wanton stream
#

Alright thanks.

#

I'll check after I give it some more thought.

# 
            while (it.nextCodepointSlice()) |slice| {
                const bytes = it.peek(1);
                if (bytes.len == 1 and slice.len == 1) {
                    // ASCII
                }
            }
#

I like this. Seems more sensible

#

Well I cannot peek early like this. Or else I might skip some ascii characters.

#

i could put the slice and the peek into a buffer and compare it the the characters i want

reef fiber
#

Might be simpler to iterate by codepoints (it.nextCodepoint) rather than slices and run the parser one character behind, so except for the first iteration, the iterator is running at your peek position and your parser is running on the previous character. To advance, shift the lookahead character to the parser's character and iterate. Keep a little bit of logic to handle the end of sequence, by filling the lookahead with the invalid unicode character as a sentinel.

#

isAscii is then just char <=127

reef fiber
#

Doing it this way, the focus is on your parser and anlyzing incoming characters, rather than navigating variable-length utf8 sequences.

wanton stream
#

Do you use a hashmap?

#

What is the extras feild in Document?

#

Ah I found it. Didnt know you can import variables like that

wanton stream
# 
pub fn eatRangeM(ore: *Parser, comptime from: u21, comptime to: u21) !?u21 {
    const from_len = comptime std.unicode.utf8CodepointSequenceLength(from) catch unreachable;
    const to_len = comptime std.unicode.utf8CodepointSequenceLength(to) catch unreachable;
    const amt = @max(from_len, to_len);
    try ore.peekAmt(amt) orelse return null;
    const len = std.unicode.utf8ByteSequenceLength(ore.buf[0]) catch return null;
    if (amt != len) return null;
    const mcp = std.unicode.utf8Decode(ore.buf[0..amt]) catch return null;
    if (mcp >= from and mcp <= to) {
        defer ore.shiftLAmt(len);
        return @intCast(mcp);
    }
    return null;
}
#

Why do this line?

if (amt != len) return null;
#

Doesn't this exclude the codepoints which require less bytes?

reef fiber
#

This complex logistics around length of byte sequences bolsters my argument that you're better off iterating code points, rather than sequences. It's unnecessary complexity.

wanton stream
#

Maybe they have a test that shows this function in action.

graceful quiver
#

sorry i didnt pitch in more, was on a flight today

#

if there's still any questions about it im happy to answer

wanton stream
#

@graceful quiver just the question I posted about amt != len

graceful quiver
#

that check is saying that if the byte length of the next available codepoint isnt the same as the codepoint being asked for then we know its not it

wanton stream
#

Why get the max byte length of the start and end range though?

graceful quiver
#

there's likely a very subtle bug here if from_len and to_len arent the same

#

you're right in that it should check both if that assumption isnt true

wanton stream
#

Neat. Just checking because it didn't make sense to me. Otherwise reading your code has been helpfull.

fading spire
#

UTF8 is made in such away that you can make apart ascii characters by just looking at individual bytes. Bytes with the highest bit being 0 can't be part of a multibyte sequence. If you just need to process ascii bytes, you don't really need an iterator

halcyon tusk
#

indeed, given that all the defined keywords and tokens in xml are contained entirely within ASCII encoding, you can effectively just go off of direct ASCII comparisons, and defer unicode verification to a later step

#

if you're outside a tag, just seek until the next & or <

#

if you're inside a tag, seek to the next non-whitespace character