“Teach” the LLM the proper procedure for counting the Rs and tell it to commit that to a file (or as a tool call). Then from there it will suceed 100% of the time.
It doesn’t matter if it’s true learning or not. When you teach an employee to do something you give them the procedure and have them do it. If AI can be fed a procedure too, it will do it just fine even if it’s not in its true training. Does it matter if it’s consuming the procedure as prompt context everytime? Hell a lot of humans do read the procedure every time when it’s complex enough.
What the example shows is that you cannot “teach” the llm how to count the letter R, because LLMs don’t work that way.
The AI company would have to solve the problem another way, let’s say by recognizing that the user is asking for letter-counting, and pass that off to a different kind of algorithm that can count letters.
Yes, and that’s called a harness, which everyone uses these days. The harness increases perceived intelligence (or accuracy) by absurd amounts. You can “teach” (or the equivalent of) LLMs anything with a custom harness.
Well the idea is you don’t need to code it yourself, you can have it do it for you. Sure it’ll have bugs the first few times, but humans also make mistakes the until they get the hang of it.
The difference i guess is that humans are capable of learning and producing better code as they become more expert at it.
The big tech companies are surely trying to improve AI with these “harnesses” as you call them, and you can try vibe coding them yourself.
But it seems to me like spending a lot of time adding features to a technology to try to catch edge cases, but the edge cases will never end, and you’ll never be able to use it for anything except rough approximations or bullshit
Because right now LLMs (and potentially other forms of AI) are the only technology capable of doing it. Humans can too, but are much slower and more expensive.
Counterpoint:
“Teach” the LLM the proper procedure for counting the Rs and tell it to commit that to a file (or as a tool call). Then from there it will suceed 100% of the time.
It doesn’t matter if it’s true learning or not. When you teach an employee to do something you give them the procedure and have them do it. If AI can be fed a procedure too, it will do it just fine even if it’s not in its true training. Does it matter if it’s consuming the procedure as prompt context everytime? Hell a lot of humans do read the procedure every time when it’s complex enough.
What the example shows is that you cannot “teach” the llm how to count the letter R, because LLMs don’t work that way.
The AI company would have to solve the problem another way, let’s say by recognizing that the user is asking for letter-counting, and pass that off to a different kind of algorithm that can count letters.
Yes, and that’s called a harness, which everyone uses these days. The harness increases perceived intelligence (or accuracy) by absurd amounts. You can “teach” (or the equivalent of) LLMs anything with a custom harness.
How is it teaching? Sounds like programming to me.
Well the idea is you don’t need to code it yourself, you can have it do it for you. Sure it’ll have bugs the first few times, but humans also make mistakes the until they get the hang of it.
The difference i guess is that humans are capable of learning and producing better code as they become more expert at it.
The big tech companies are surely trying to improve AI with these “harnesses” as you call them, and you can try vibe coding them yourself.
But it seems to me like spending a lot of time adding features to a technology to try to catch edge cases, but the edge cases will never end, and you’ll never be able to use it for anything except rough approximations or bullshit
yes lets have the llm be a blurry frontend for a bunch of invisible proprietary harness programs. that surely is a valuable human endeavor
Because right now LLMs (and potentially other forms of AI) are the only technology capable of doing it. Humans can too, but are much slower and more expensive.