Apple has run Muse Spark Open, an 816-billion-parameter model from Meta with 42 billion parameters active at a time, across four Mac Studios and measured over 36 tokens a second. Apple’s machine learning researchers say the model “cannot fit on a single machine”, and their figures put four M5 Ultra machines splitting it by width at 36.73 tokens a second at an 8,192-token context, against 15.74 for two machines in a pipeline.
The banner Apple published with the research post on October 15, alongside the MLX release the benchmarks were run with. Image: Apple
Memory is why it takes four. Muse Spark Open “would require more than 1.6TB of RAM in its native precision (BF16) for the model parameters alone”,
