sanitation@lemmy.today to Technology@lemmy.worldEnglish · 2 days agoIf open weight models are the future, U.S. AI companies are going to have a hard timewww.fastcompany.comexternal-linkmessage-square88linkfedilinkarrow-up1280arrow-down13
arrow-up1277arrow-down1external-linkIf open weight models are the future, U.S. AI companies are going to have a hard timewww.fastcompany.comsanitation@lemmy.today to Technology@lemmy.worldEnglish · 2 days agomessage-square88linkfedilink
minus-squarebrucethemoose@lemmy.worldlinkfedilinkEnglisharrow-up2·1 day agoYou want this one: https://huggingface.co/turboderp/Qwen3.6-27B-exl3_3.30bpw/tree/main Or maybe the 3.5bpw one if you don’t mind less context, or 3bpw if you need more: https://huggingface.co/turboderp/Qwen3.6-27B-exl3 For faster inference at the cost of a little more VRAM usage: https://huggingface.co/turboderp/Qwen3.6-27B-DFlash-exl3 And you run those in: https://github.com/theroyallab/tabbyAPI And FYI, if you have 64GB of RAM or more, you might consider hybrid inference instead.
minus-squareAsafum@lemmy.worldlinkfedilinkEnglisharrow-up2·1 day agoThank you for all the links! Much appreciated :)
You want this one:
https://huggingface.co/turboderp/Qwen3.6-27B-exl3_3.30bpw/tree/main
Or maybe the 3.5bpw one if you don’t mind less context, or 3bpw if you need more:
https://huggingface.co/turboderp/Qwen3.6-27B-exl3
For faster inference at the cost of a little more VRAM usage:
https://huggingface.co/turboderp/Qwen3.6-27B-DFlash-exl3
And you run those in:
https://github.com/theroyallab/tabbyAPI
And FYI, if you have 64GB of RAM or more, you might consider hybrid inference instead.
Thank you for all the links! Much appreciated :)