The article from Anthropic describes how a small agentic model~15-50gb can be distributed on an inference server hosted across a network of 10-20 IoT devices like smart-TVs, which can have up to 4gb RAM–higher end IoT appliances can also have 1-4GB RAM to hold a meaningful shard of a 30B model.
The article from Anthropic describes how a small agentic model~15-50gb can be distributed on an inference server hosted across a network of 10-20 IoT devices like smart-TVs, which can have up to 4gb RAM–higher end IoT appliances can also have 1-4GB RAM to hold a meaningful shard of a 30B model.