If China manages to scale their hardware production, this could finally be “their day in the sun” where they clearly surpass the West in the way the West has outshone them for 100 years. MoE has its place, but a MoDE where each E is itself a dense model would be more powerful still, mostly you need the silicon gates and power to drive them.
If China manages to scale their hardware production, this could finally be “their day in the sun” where they clearly surpass the West in the way the West has outshone them for 100 years. MoE has its place, but a MoDE where each E is itself a dense model would be more powerful still, mostly you need the silicon gates and power to drive them.
Three Gorges makes hydro-power, right? 22MW -> 100 TWh per year, just from that one structure, I bet that will run at least one AGI… https://www.fredgao.com/p/deepseeks-liang-wenfeng-breaks-his