The only thing that worries me is that we don’t actually know how the human brain works. There’s an entire school of psychological theory and practice - really its oldest truly scientific branch, that holds that human intelligence actually does work a lot like an LLM. The hard core behaviorists believed that literally all human behavior was just a really complex version of Pavlov’s dogs. It sounds absurd, but they had good arguments. In principle even very complex behaviors can be the result of reinforcement and conditioning.

https://en.wikipedia.org/wiki/Behaviorism