Abstract page for arXiv paper 2604.03136: StoryScope: Investigating idiosyncrasies in AI fiction

  • AnAmericanPotato@programming.dev
    link
    fedilink
    English
    arrow-up
    84
    ·
    4 months ago

    93.2% macro-F1 for human vs. AI detection and 68.4% macro-F1 for six-way authorship attribution

    This isn’t what I’d call “reliable”.

    I’m also not impressed with their methodology, which is heavily based on Gemini.

    To generate mirrored AI stories, we reverse-engineer writing prompts from each human story by prompting Gemini 2.5 Flash (Gemini Team, 2025a) to infer the underlying premise

    That’s too many steps removed from anything you’d encounter in the wild. They’re not even testing against human-prompted output.

    And then they use Gemini again to analyze all the stories. Relying on proprietary cloud models for the core of your analysis is like building on sand.

      • Kevin@programming.dev
        link
        fedilink
        English
        arrow-up
        47
        ·
        4 months ago

        Percentages are somewhat misleading here. That’s around 2 in 25, enough to falsely accuse a student or 2 in a fairly standard classroom for example.

        I remember somebody mentioning a quote along the lines of “A 95% success rate sounds good, until you become the 1 in 20 that it kills”

          • Holytimes@sh.itjust.worksdeleted by creator
            link
            fedilink
            English
            arrow-up
            10
            ·
            4 months ago

            Getting your degree or being kicked out and banned from your college and still requires to pay back the loans. IS life or death.

            It literally can determine the quality of life of a person for the next 30-40 years. If not forever. This is NOT something to take lightly.

              • doben@lemmy.ml
                link
                fedilink
                English
                arrow-up
                7
                arrow-down
                1
                ·
                4 months ago

                Holy the ignorant self-centered anecdote. Too long, no relevance, do not recommend. I‘d probably give it a failing grade.

              • luciferofastora@feddit.org
                link
                fedilink
                English
                arrow-up
                1
                ·
                4 months ago

                Okay so what I’m taking away from this is that you were lucky to find gainful employment without proof of a degree and didn’t have any debt to pay off? Because the context here is people having all the debt and none of the credentials required for jobs where they make enough to pay off that debt. If your card is declined, how you feel about it doesn’t matter.

                  • luciferofastora@feddit.org
                    link
                    fedilink
                    English
                    arrow-up
                    1
                    ·
                    4 months ago

                    I had student loans, luckily they only amounted to 4-5 months’ gross income - lucky because scholarships stepped up and I only had to take a loan for one year instead of all six.

                    Still more than you should have had (education shouldn’t put you in debt imo), but definitely far less than other people have. Particularly people studying in the last few years. .com is long gone, prices have gone up and the margin between net income and cost of living is slim for many people.

                    That’s my whole point: you shouldn’t downplay the impact that being thrown out can have if you’ve got $50k in student loan debt. If you cheated, sure, that’s on you. But if you were falsely accused by a detector that turns up false positives for one or two students per class, on average, that just sucks like an open window in space.

      • Damarus@feddit.org
        link
        fedilink
        English
        arrow-up
        16
        arrow-down
        1
        ·
        4 months ago

        That’s not good enough. Just a thought experiment: Only 7% of humans died because their work got flagged incorrectly

        • Allah@piefed.worldOP
          link
          fedilink
          English
          arrow-up
          2
          arrow-down
          37
          ·
          4 months ago

          this is just spherical cow fallacy combined with purity culture slop

          no methodology is perfect, it just needs to be practically valid, using ai != killing people for using ai

          unless you are for killing people who use AI, which wouldn’t surprise me one bit about Anti-AI horde

          • darkmarx@lemmy.world
            link
            fedilink
            English
            arrow-up
            27
            ·
            4 months ago

            Tell that to the student who gets expelled because they’re writing was falsely identified as AI and then takes their own life.

            Tell that to the person falsely identified with facial recognition and is arrested and loses their job.

            Using AI to detect AI is as bad as using AI for any important decision.

            This isn’t hypothetical. It happens. People will lose their livelihood or die because of it. 7 in 100 is not accurate enough when any penalty is applied to detection.