• gabelstapler@feddit.org
    link
    fedilink
    English
    arrow-up
    33
    arrow-down
    1
    ·
    1 day ago

    I’m wondering what training data they are using. The same pirated and scraped data the others are using as well?

    • Artisian@lemmy.world
      link
      fedilink
      English
      arrow-up
      8
      ·
      1 day ago

      They could use self-play and existing ethical models, like Apertus.

      That said, they don’t disclose afaict. Haven’t trained it, so they probably don’t know themselves.

        • Artisian@lemmy.world
          link
          fedilink
          English
          arrow-up
          1
          ·
          17 hours ago

          I suspect no, if they haven’t actually trained a model yet?

          Presumably you mean after they actually train the thing. Then I assume yes, but I’m unfamiliar with the law.

      • Echo Dot@feddit.uk
        link
        fedilink
        English
        arrow-up
        5
        ·
        15 hours ago

        Isn’t parameters just the number of virtual dials that can be changed, essentially the number of neurons, I don’t think it’s got anything to do with the actual amount of data.

        After all distilling works (somehow), by taking all the knowledge of the large AI with a lot of parameters and putting it into a smaller AI with fewer parameters. The amount of data hasn’t changed only the number of parameters. Presumably the inverse is also true, you can have a lot of parameters but not a lot of data.

  • Sibbo@sopuli.xyz
    link
    fedilink
    English
    arrow-up
    10
    ·
    2 days ago

    Cool! How much does a home server cost that runs it at a reasonable tokens per second?

    • ramble81@lemmy.zip
      link
      fedilink
      English
      arrow-up
      15
      arrow-down
      1
      ·
      1 day ago

      If you want a more reasonable European developed open-weight model, consider looking at Ministral. Developed by the French, and it’s a slightly smaller version of their Mistral full. Ministral 3 - 27B runs just fine on a 12GB card.

    • gnutrino@programming.dev
      link
      fedilink
      English
      arrow-up
      18
      ·
      2 days ago

      Running EUROPA on in-house hardware would take a lot of memory, even with the usual compression to 4 bits per parameter. At that level, its weights alone would take up at least 476 GB by our math, plus working memory. That puts it in the same range as Muse Spark Open, Meta’s open model. That model has 816 billion parameters and runs on two Apple Mac Studio desktops with 512 GB of memory each.

      Although it’s worth noting they haven’t actually trained the thing yet so we’d have to wait to get the actual weights to know more.

    • OhneHose@feddit.org
      link
      fedilink
      English
      arrow-up
      6
      arrow-down
      1
      ·
      2 days ago

      Judging from the size of other frontier models on hf, you won’t host it.

      The best/biggest you can squeeze on 64gb ram rn is ~70-80B in a Q4 Quant. Going lower with quantization will usually leave you with notable degraded performance.

        • OhneHose@feddit.org
          link
          fedilink
          English
          arrow-up
          3
          ·
          1 day ago

          Depends on your hardware. Apples unified memory system can give near complete access to the ram for their GPU/npu.

          Apart from that you don’t need a single bit of vram/gpu to host a model. Then can be run completely on CPU+ram. But that comes with a significant processing penalty.

          On top of that there’s new “engines” which can run a LLM from disk storage +system ram/vram, yet again with lots of speed penalities, since you chunk the model into small pieces and load them into ram accordingly. This will also cripple the lifetime of ssds with the sheer amount of read/write cycles.

          Then there’s also dedicated systems or extension cards which can be used as accelerator.

          None the less we speak about a minimum of 5000€'s for a system which can run llms and (lightweight) games simultaneously.

        • B0rax@feddit.org
          link
          fedilink
          English
          arrow-up
          6
          ·
          1 day ago

          You can also get systems with unified memory (from amd for example, or from Apple), which will be available* up to 512gb of ram.

          *that means, if you pay the equivalent of a middle sized new car.

    • Skipcast@lemmy.world
      link
      fedilink
      English
      arrow-up
      3
      ·
      2 days ago

      Domyn has not yet released EUROPA’s weights, the files that hold its parameters and let anyone run it on their own hardware. It plans to publish them with details of its training methods when EUROPA comes out. The post gives no date. In June, the outlet Declic Media reported that Domyn was aiming for a release “within the year.”

      Guess we’ll have to wait and see