About input formats for training and inference

[wen020]

Anygpt is trained only with the Next Token Prediction task.
Take text to image as an example，Is the training input speech tokens text tokens image tokens music tokens?
I want to know the input formats for training and inference.
training input ：<sos> speech tokens <eos> text tokens <soi> image tokens <eoi> <som> music tokens,
training label ：speech tokens <eos> text tokens <soi> image tokens <eoi> <som> music tokens <eom>. Is my understanding correct about training input and label?

[JunZhan2000]

No, the content of the training depends on what data and what task we use. For example, for text to image conversion, the data will only contain text tokens image tokens . This is just an example. The actual template used can be found in the paper.

[JunZhan2000]

Hello, we provide some training data samples and related descriptions, please refer to https://github.com/OpenMOSS/AnyGPT?tab=readme-ov-file#pretraining-and-sft