Assign location to tokens

[kant2002]

• That allow implement parsers which indirectly span across multiple files, like C/C++

Related: #34 (comment)

[kant2002]

This PR can be considered draft in a sense that it has flaws and require discussion. Also tired.

• SourceFile as abstraction too much tied to TextReader. Lexers need direct access to TextReader, but SourceFile diagnostics machinery require buffered access where precalculated lines. Possble highligt issue in SourceFile implementation
• I have to reimplement Equals and GetHashCode for Token. That also hints on too much coupling between concepts.

Would like to start talking about directions.

[kant2002]

Also I did not attempt to make API without breaking changes. Have to know opinion on the matter, and is breaking changes possible

[kant2002]

I come to conclusion that features of SourceFile which trigger inconsistent behaviour is overkill. Would be good if SourceFile would no longer be derived from TextWriter, and for backward compatibility there either other class with similar functionality, or make breaking change.

[kant2002]

Will try to summon @LPeter1997 to discuss things

[LPeter1997]

Sorry for the super-delayed reply. I'll try to get to this in a week or so. I'm sadly working on other things and I have many-many things left here to tidy up.

[kant2002]

@ForNeVeR I manage to change PR to the state where it does not have source breaking changes, and only binary one. What do you think, should I go further and make it binary compatible (which ideally proper way)

[ForNeVeR]

I'd say that Yoakke shouldn't care about binary compatibility for now.