Location tracking, with partial streaming

[dignifiedquire]

I am considering switching the parsing in https://github.com/rpgp/rpgp to a different parsing library (currently using nom) as I have to do a lot of work to make streaming parsing work, and continue to run into bugs. Additionally I need to start attaching location information (offset + length) to individual pieces so I can build hexdump style views for the parsed trees.

While experimenting with winnow I am trying out sth like this, but unfortunately this explodes with partial streams need to handle ErrMode::Incomplete

type Stream<'i> = Partial<Located<&'i Bytes>>;
type LocatedSlice<'a> = (&'a [u8], std::ops::Range<usize>);

fn parser<'s>(input: &mut Stream<'s>) -> PResult<(LocatedSlice<'s>, LocatedSlice<'s>)> {
    separated_pair(alpha1.with_span(), ',', alpha1.with_span()).parse_next(input)
}

fn main() {
    let mut consumed_parser = parser.with_span();
    let res = consumed_parser.parse(Partial::new(Located::new("abcd,ab".into())));
    dbg!(&res);
}

My main questions are

• can this be made to work in the way think about it?
• is there a way to make the parsers generic over the usage of Located? Most use cases don't need the location tracking, and so ideally I would be able to write the parsers generic, and only enable it when necessary.

[epage]

Parser::parse is exclusively meant for complete input, rather than partial input. If there are times that your partial input is complete, you can specify that with complete.

To parse partial input, you will need to use Parser::parse_next. If you are wanting to get some of the location information out on the error case, you can look at the implementation of Parser::parse to see how that is done.

is there a way to make the parsers generic over the usage of Located? Most use cases don't need the location tracking, and so ideally I would be able to write the parsers generic, and only enable it when necessary.

Yes, 99% of winnows parsers are generic and don't care if you are using Located or not. Just make their input a I: Stream. Some other bounds might be needed, depending on what parsers you call. You'd likely want to create a supertrait with a blanket impl to simplify your bounds (we at least simplified this by merging a lot of Input traits into Stream).