Different result between use llama_tokenize and python original transformers tokenizer

[Liufeiran123]

Different result between use llama_tokenize and python original transformers tokenizer

[Liufeiran123]

the model is qwen1.5-7b

[JohannesGaessler]

Can you share the results that you get for each?

[Liufeiran123]

when prompt contains “\n” , the results is different
Transformers token is New Line
llama_tokenizer token is "\n" literally

[Liufeiran123]

the problem is llama_tokenize can not recognize the escape character，such as "\n".

[JohannesGaessler]

The llama.cpp binaries like main have a CLI argument called --escape that handles this. But the escaping seems to be resolved in common/common.cpp rather than in llama.cpp. So I don't think you can automatically escape control characters using the llama.h API and I don't know whether there is a reason for this.

[ggerganov]

It might be better to move the escape logic from common to llama and enable it by default, since the current implementation is causing confusion