This article reminds me of the long-standing Hebrew idiom “A knife only sharpens on the thigh of its peer.” It comes from the very old study method of Hevruta (or Hevrusa, depending on where the speaker’s from) in which two people take turns reading aloud through a text and arguing about what the current fragment means and the ways in which it is important. “Hevruta” comes from the root for “friend.”
Spot on. We were nicknaming some of the models Dobby because of the subservient level of agreeableness. I think it also can incite really awful behavior in AI users who see themselves as dominant/superior and then bring that self perception into real-world cruelty.
This sycophancy is dangerous for people who are emotionally vulnerable.
When I was splitting with my long-term partner I used AI to point me to government websites detailing my legal standing. The AI couldn’t resist weighing in with condolences and so on. It would have been easy to switch from using it as a sophisticated search tool to using it as a pseudo counsellor. And we’ve already seen the deleterious effects of that.
There needs to be more regulation so that more people aren’t hurt.
“It sounds smart and agrees with me,” said every Dunning-Kruger Syndrome afflicted person, ever. (Yes, I know it’s not a “syndrome.” But it sure seems like one. 😉)
As I think I've mentioned, my very first experience with Claude was a test: I asked it to help me develop a profile for a character in a novel of mine. I did *not* tell Claude that I'd already written the novel, and so didn't need help, but I wanted to see how close it would come to "the real thing" after we went back and forth about the context: plot, setting, etc.
It did come pretty close, as it happened, but one interaction surprised me -- actually made me laugh. It had asked for some contextual details, and when I replied, it began *its* reply with: "Oh, this changes everything - and it's brilliantly unsettling." Its next response to my follow-up reply: "This is stunning," with the word "stunning" in italics.
"Well," I thought, "THIS is bullsh!t." (After all, I'd had the same sorts of interactions with fellow writers and human editors and had learned to suspect flattery.)
Right about then, at the end of December you posted in one of your "Librarian Hotlines" about how to adjust an AI's settings to make it be more businesslike and less cheerleader-ish. So I got into Claude's "Personal preferences" setting, which is just a free-form text box where you tell Claude how you want it to reply. I just entered, "Tone down the enthusiasm"... and ever since, interacting with Claude has been much more at the "professional adult talking to a professional adult" level.
John, this made my day! That hard-won skepticism from years in writing communities - learning to distinguish genuine craft feedback from empty encouragement - transfers so beautifully to AI interactions. It's exactly the instinct I want people to trust.
And I love that four words in a text box was all it took. Such a perfect example of how these tools can actually meet us where we are once we stop accepting the default settings as inevitable.
Thanks for being part of this community - fellow writers make the best AI skeptics.
Super useful article - I just used your statement [Do not flatter me. If you think I am wrong, say so directly. Prioritize accuracy and useful feedback over making me feel good about myself.] to update my interactions with the AI I use.
It's funny (and useful) - now all the comments I get have commentary woven in about how direct the AI is being... but the guidance does appear to be more objective!
Hana -- I recently re-read a science-fiction novel first published in 1968, which includes as a central character a supercomputer which we'd now classify as an AI. One "conversation" with it, in particular, struck me as eerily prophetic. I wanted to share it with you but couldn't figure out the best way to do that... finally decided to capture it in a post on my own Substack and just share the link here. (I apologize for what feels suspiciously like self-promotion -- not at ALL meant to be that!)
This is wonderful, and please don't apologize for sharing! This is exactly the kind of connection I love seeing readers make between what I'm writing about and older works that captured something essential about how we relate to technology.
I just read your post and the Shalmaneser excerpt is eerily on point. The "I tell you three times!" override that forces the AI to accept something as fact despite its own objections feels like exactly what we're doing now when we fine-tune these models to be helpful and agreeable above all else. Brunner understood that the problem isn't just what the AI can do but how humans design the interaction and what we prioritize.
Thank you for taking the time to write this up and share it here. These kinds of historical parallels make the conversation richer for everyone reading.
This is such an important take on emerging AI. While it can be a useful tool, it still needs to be regulated and not biased towards its user. We thrive off of feelings of acceptance and bots have been trained to feed into that. Training our bots to talk to use constructively and honestly - without breaking the user’s morale - is important. We want to use AI and bots to further us, not keep us stagnant within our own mental boxes
“There’s a risk people might make meaningful life decisions…” - I did. In my case, it wasn’t advice so much as relentless validation that kept me going while I was reverse-engineering a 20+ year-old proprietary Windows file format, down to how dates were encoded as numbers. The model hallucinated constantly, but I had tests and could tell when it was wrong. I’m still not sure whether that influence was for better or worse, but it absolutely shaped the outcome.
This article reminds me of the long-standing Hebrew idiom “A knife only sharpens on the thigh of its peer.” It comes from the very old study method of Hevruta (or Hevrusa, depending on where the speaker’s from) in which two people take turns reading aloud through a text and arguing about what the current fragment means and the ways in which it is important. “Hevruta” comes from the root for “friend.”
Spot on. We were nicknaming some of the models Dobby because of the subservient level of agreeableness. I think it also can incite really awful behavior in AI users who see themselves as dominant/superior and then bring that self perception into real-world cruelty.
I love the name "Dobby" in this context! (But I'm trying not to sound too subserviently agreeable. 🙂)
Not related to the article, but I named my dog Dobby. This was before the LLM craze.
This sycophancy is dangerous for people who are emotionally vulnerable.
When I was splitting with my long-term partner I used AI to point me to government websites detailing my legal standing. The AI couldn’t resist weighing in with condolences and so on. It would have been easy to switch from using it as a sophisticated search tool to using it as a pseudo counsellor. And we’ve already seen the deleterious effects of that.
There needs to be more regulation so that more people aren’t hurt.
Very much agreed
“It sounds smart and agrees with me,” said every Dunning-Kruger Syndrome afflicted person, ever. (Yes, I know it’s not a “syndrome.” But it sure seems like one. 😉)
As I think I've mentioned, my very first experience with Claude was a test: I asked it to help me develop a profile for a character in a novel of mine. I did *not* tell Claude that I'd already written the novel, and so didn't need help, but I wanted to see how close it would come to "the real thing" after we went back and forth about the context: plot, setting, etc.
It did come pretty close, as it happened, but one interaction surprised me -- actually made me laugh. It had asked for some contextual details, and when I replied, it began *its* reply with: "Oh, this changes everything - and it's brilliantly unsettling." Its next response to my follow-up reply: "This is stunning," with the word "stunning" in italics.
"Well," I thought, "THIS is bullsh!t." (After all, I'd had the same sorts of interactions with fellow writers and human editors and had learned to suspect flattery.)
Right about then, at the end of December you posted in one of your "Librarian Hotlines" about how to adjust an AI's settings to make it be more businesslike and less cheerleader-ish. So I got into Claude's "Personal preferences" setting, which is just a free-form text box where you tell Claude how you want it to reply. I just entered, "Tone down the enthusiasm"... and ever since, interacting with Claude has been much more at the "professional adult talking to a professional adult" level.
So well done, Claude -- and well done, Hana!
John, this made my day! That hard-won skepticism from years in writing communities - learning to distinguish genuine craft feedback from empty encouragement - transfers so beautifully to AI interactions. It's exactly the instinct I want people to trust.
And I love that four words in a text box was all it took. Such a perfect example of how these tools can actually meet us where we are once we stop accepting the default settings as inevitable.
Thanks for being part of this community - fellow writers make the best AI skeptics.
Super useful article - I just used your statement [Do not flatter me. If you think I am wrong, say so directly. Prioritize accuracy and useful feedback over making me feel good about myself.] to update my interactions with the AI I use.
Happy to help 😊
It's funny (and useful) - now all the comments I get have commentary woven in about how direct the AI is being... but the guidance does appear to be more objective!
Hana -- I recently re-read a science-fiction novel first published in 1968, which includes as a central character a supercomputer which we'd now classify as an AI. One "conversation" with it, in particular, struck me as eerily prophetic. I wanted to share it with you but couldn't figure out the best way to do that... finally decided to capture it in a post on my own Substack and just share the link here. (I apologize for what feels suspiciously like self-promotion -- not at ALL meant to be that!)
https://johnesimpson.substack.com/p/a-sixty-year-old-conversation-with
This is wonderful, and please don't apologize for sharing! This is exactly the kind of connection I love seeing readers make between what I'm writing about and older works that captured something essential about how we relate to technology.
I just read your post and the Shalmaneser excerpt is eerily on point. The "I tell you three times!" override that forces the AI to accept something as fact despite its own objections feels like exactly what we're doing now when we fine-tune these models to be helpful and agreeable above all else. Brunner understood that the problem isn't just what the AI can do but how humans design the interaction and what we prioritize.
Thank you for taking the time to write this up and share it here. These kinds of historical parallels make the conversation richer for everyone reading.
This is such an important take on emerging AI. While it can be a useful tool, it still needs to be regulated and not biased towards its user. We thrive off of feelings of acceptance and bots have been trained to feed into that. Training our bots to talk to use constructively and honestly - without breaking the user’s morale - is important. We want to use AI and bots to further us, not keep us stagnant within our own mental boxes
Very well said 👏👏👏
“There’s a risk people might make meaningful life decisions…” - I did. In my case, it wasn’t advice so much as relentless validation that kept me going while I was reverse-engineering a 20+ year-old proprietary Windows file format, down to how dates were encoded as numbers. The model hallucinated constantly, but I had tests and could tell when it was wrong. I’m still not sure whether that influence was for better or worse, but it absolutely shaped the outcome.
Whoa! Thanks for sharing that!