From Cheese (they don't have a HN account, so I'm posting for them):
Each weight is a FP16 float which is 2 Bytes worth of data, you have 70B tokens, so the total amount of data the weights take up is 140GB then you have a couple extra GBs for the context.
Then to figure out the theoretical tokens per second you just divide the amount of memory bandwidth, 5300GB/s in MI300X's case, by the amount of data that the tokens and context take up so 5300/142 which is about 37 tokens per second.
You have to get those weights from the RAM to the floating point unit. The bandwidth here is the rate at which you can do that.
The weights are not really reused. Which means they are never in registers, or in L1/L2/L3 caches. They are always in VRAM and always need to be loaded back in again.
However, if you are batching multiple separate inputs you can reuse each weight on ech input, in which case you may not be entirely bandwidth bound and this analysis breaks down a bit. Basically you can't produce a single stream of tokens any faster than this rate, but you can produce more than one stream of tokens at this rate.
Each weight is a FP16 float which is 2 Bytes worth of data, you have 70B tokens, so the total amount of data the weights take up is 140GB then you have a couple extra GBs for the context.
Then to figure out the theoretical tokens per second you just divide the amount of memory bandwidth, 5300GB/s in MI300X's case, by the amount of data that the tokens and context take up so 5300/142 which is about 37 tokens per second.