Syntax Errors Result in Assertions Being Ignored

[kevinastone]

I've run into this a few times where either an actual parser bug or a typo has caused the assertion to fail parsing. Because this syntax format uses existing comment delimiters, it must assume anything that doesn't parse is still valid syntax in the original grammar and pass it through unchanged.

This leads to confusion from either: assertion tests being treated as source grammar, or worse, tests being ignored since they're just irrelevant comments, creating a false sense of assurance.

Figure out how to determine when the line is an assertion even if it has parsing errors and complain loudly rather than silently ignoring.

[bd82]

Basically you need a heuristic to distinguish between "irrelevant input" and "malformed" input.

I can think of a few things may be combined to create such a heuristic.

• Lexer - Percentage of the line columns that make up valid tokens.
  • The lexer is quite simple, so that might be used instead of detect "irrelevant input" instead of detecting "malformed input"
• Parser - The beginning of the assertion is valid.
  • For example if you have gotten past the position rule while parsing an assertion, you can mark
    a flag that indicates that it "looks like an assertion" even if that input later failed in the scope rule.
• Parser - Expand grammar definition to include common mistakes.
  • Your grammar only allows spaces as whitespace, Lets assume users commonly use tabs by mistake, you can expand the definition of whitespace to include tabs, parse successfully and warn the user .
  • another example is the whitespace between positions and scopes It does not look like it is required for any dis-ambiguity, just for readability. you can make it optional and warn if it is missing.
  • Generally expanding the grammar for common mistakes will allow you to successfully parse some malformed inputs.
• Parser - Enable automatic error recovery, build an AST and investigate the AST's structure.
  • This is the complex but also potentially the "strongest" approach.
  • If you build an intermediate representation (AST) of the Assertion, which using the automatic error recovery may be only partially complete, you can ask "questions" with much finer granularity about which parts exist and which parts are missing in the AST to decide between malformed or invalid input.
• This is really just a generalization of the number 2 option suggested above.

[kevinastone]

@bd82 Yeah, I was thinking of just throwing different errors if the parser at least found a position token (using a boolean). Then the parser can either ignore if no position was found or complain if it looked partially valid.