This is exactly what we need! Will try: https://github.com/ClickHouse/ClickHouse...

		zX41ZdbW 5 days ago \| parent \| context \| favorite \| on: GigaToken: ~1000x faster Language model tokenizati... This is exactly what we need! Will try: https://github.com/ClickHouse/ClickHouse/issues/108247 It will be nicer if the README focuses more on per-core performance. About the actual algorithm - will something like matching in a perfect hash table help?
		help