sanitation@lemmy.today to Technology@lemmy.worldEnglish · 2 days agoIf open weight models are the future, U.S. AI companies are going to have a hard timewww.fastcompany.comexternal-linkmessage-square88linkfedilinkarrow-up1274arrow-down13
arrow-up1271arrow-down1external-linkIf open weight models are the future, U.S. AI companies are going to have a hard timewww.fastcompany.comsanitation@lemmy.today to Technology@lemmy.worldEnglish · 2 days agomessage-square88linkfedilink
minus-squareAsafum@lemmy.worldlinkfedilinkEnglisharrow-up1·23 hours agoThanks for the tip! I do have a 5080 so I’ll have to look up exllamav3
minus-squarebrucethemoose@lemmy.worldlinkfedilinkEnglisharrow-up2·21 hours agoYou want this one: https://huggingface.co/turboderp/Qwen3.6-27B-exl3_3.30bpw/tree/main Or maybe the 3.5bpw one if you don’t mind less context, or 3bpw if you need more: https://huggingface.co/turboderp/Qwen3.6-27B-exl3 For faster inference at the cost of a little more VRAM usage: https://huggingface.co/turboderp/Qwen3.6-27B-DFlash-exl3 And you run those in: https://github.com/theroyallab/tabbyAPI And FYI, if you have 64GB of RAM or more, you might consider hybrid inference instead.
minus-squareAsafum@lemmy.worldlinkfedilinkEnglisharrow-up2·21 hours agoThank you for all the links! Much appreciated :)
Thanks for the tip! I do have a 5080 so I’ll have to look up exllamav3
You want this one:
https://huggingface.co/turboderp/Qwen3.6-27B-exl3_3.30bpw/tree/main
Or maybe the 3.5bpw one if you don’t mind less context, or 3bpw if you need more:
https://huggingface.co/turboderp/Qwen3.6-27B-exl3
For faster inference at the cost of a little more VRAM usage:
https://huggingface.co/turboderp/Qwen3.6-27B-DFlash-exl3
And you run those in:
https://github.com/theroyallab/tabbyAPI
And FYI, if you have 64GB of RAM or more, you might consider hybrid inference instead.
Thank you for all the links! Much appreciated :)