If AI actually adds value, it should be trivial to demonstrate that value-add in a way that passes scientific rigor.
The underlying problem is that we don’t have a good way to measure code value. Software quality is most closely coporable to a weird combination of scientific paper, mechanical diagnostic, and toy instructuon. And we don’t have good ways to measure those, either.
Note that the headline is misleading – stanford apparently trainded an AI model to “rate code” in a way that agreed with some of their staff and then ran that on a bunch of projects. The “good at simple and new, bad at complex and old” matches my intuition, but isn’t really a stronger test than counting minutes spent in a project or dollars spent on programming with or without AI.
And all AI output is slop. It’s just that for some things slop is good enough.
~Which really should be an argument more for discarding those things than boiling oceans to generate more of them, but capitalism loves doing wasteful things~
If AI actually adds value, it should be trivial to demonstrate that value-add in a way that passes scientific rigor.
The underlying problem is that we don’t have a good way to measure code value. Software quality is most closely coporable to a weird combination of scientific paper, mechanical diagnostic, and toy instructuon. And we don’t have good ways to measure those, either.
There was apparently one study from Stanford:
https://medium.com/@manusf08/does-ai-really-boost-developer-productivity-a-stanford-study-of-100-000-devs-has-answers-4f64c64ebe97
Note that the headline is misleading – stanford apparently trainded an AI model to “rate code” in a way that agreed with some of their staff and then ran that on a bunch of projects. The “good at simple and new, bad at complex and old” matches my intuition, but isn’t really a stronger test than counting minutes spent in a project or dollars spent on programming with or without AI.
And all AI output is slop. It’s just that for some things slop is good enough.
~Which really should be an argument more for discarding those things than boiling oceans to generate more of them, but capitalism loves doing wasteful things~