Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Could multiple Chinese characters be counted as a single token?


Possibly. Telephone (电话) is electricity/electronic (电) + talking/speech (话).

In Japanese there's the Japanese possessive ('no') which can also be a modifier/qualifier in text like 男の子 (boy, literally "man of child") and 女の子 (girl, literally "woman of child"), so there are sequences of Chinese characters (possibly in combination with Japanese) that could be a single token like character sequences in the Latin script.

I've found https://digitalorientalist.com/2025/02/04/to-merge-or-not-to... with some information/analysis of this.




Consider applying for YC's Fall 2026 batch! Applications are open till July 27.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: