“Finding” is also rather generous, going by numbers from Anthropic and from the Linux kernel, things like Mythos are wrong 95% of the time.
They’re much easier to run though, compared to more reliable tools. So even being wrong 95% of the time wins out if you’ve got 100x more people running them.
A 95% false positive rate isn’t actually that big a deal if you have a good way to test the claims. I wonder if you could set it up CtF-style, where you don’t get notified until the LLM can write to a specific file on the target box.
“Finding” is also rather generous, going by numbers from Anthropic and from the Linux kernel, things like Mythos are wrong 95% of the time.
They’re much easier to run though, compared to more reliable tools. So even being wrong 95% of the time wins out if you’ve got 100x more people running them.
A 95% false positive rate isn’t actually that big a deal if you have a good way to test the claims. I wonder if you could set it up CtF-style, where you don’t get notified until the LLM can write to a specific file on the target box.