Chat Room Dispatches: Intelligence Not Included

Context

Below is a short message I wrote (~3 months ago) to an online discussion board about LLMs among a community of (very non-technical) writers and book enthusiasts, and it tries to clarify by way of example and analogy what kinds of things LLMs are, why they are frustratingly bad at what it they are marketed/hyped/feared for, but are good at (comparatively mundane yet very useful) tasks that nobody ever talks about. Also, there was some discussion about Godel Escher Bach by Hofstader, which is why I mention it at the end.

Because this is basically a copy-paste of my response, it is conversational in tone, annoyingly overuses internet-slang, and is only slightly edited, but certainly not polished nor very rigorously researched. While I am technical (software engineer), I am not an ML engineer. I deliberately avoided talking about LLMs in software engineering because of the context, but may do a follow up in the future. I am posting this here as a kind of snapshot of my thoughts (and, by implication, the general thoughts and sentiments that I was replying to) on the technology circa late 2024 (before the latest, and in my humble opinion very good, DeepSeek news). A lot has already changed since then, and we will see how much of this changes by late 2025.

The Message

FWIW, my $0.02, most AI (LLMs in particular) is embarrassingly bad at most of the things that the AI companies are marketing it for (i.e. terrible at writing, terrible at coding, not great at reasoning, terrible at critique of writing, terrible at finding mistakes in code, good at a few other things, but can easily get confused if you give it a "bad" question and have to start the conversation from scratch). However, there are things that it is great at, but that the AI companies do not want to market it for. A few examples:

It is a phenomenal reverse dictionary (i.e. which English words mean "of a specific but unspecified character, quality, or degree"). It not only works for English, but also for Esperanto (i.e. which Esperanto words mean "of a specific but unspecified character, quality, or degree"), as well as my own obscure native language. This is a huge time-saver when learning languages (normal dictionaries won't cut it, and bi-lingual dictionaries are limited, if they are available at all). Even if you are just using a language you are fluent in, a reverse-dictionary-prompt can help you find words and usages, and can also help you find "dark spots" in the language's lexicon.

It is also good at metaphors – as we've seen – but not great, and can get confused if the subject is obscure or not widely talked about. And it sometimes comes up with "lazy" metaphors. A lot of its metaphors are superficial, but a lot of them are not.

It was able to accurately describe the difference between two similar books by the same author (the amazon and goodreads pages had the same description for two similar looking books with similar sounding titles – they were not the same, but rather companion-books (the LLM overcame this obscurity even though nobody on/at amazon or goodreads or $obscureUniversityPublishingHouse could)).

It is able to recommend books on topics, both obscure and unobscure (better than Google by a mile, and a fine complement to Kagi).

It is excellent for getting an idea of what the broad internet-consensus is on a topic. It is also able to do this for a linguistic region (i.e. asking the question in $obscureLanguage will provide the consensus-view of the speakers in that language, not English). It does not always work, but it's better than nothing.

In short, it's an analytical tool – a telescope for language – but it is being marketed as a synthetical tool, which (on the one hand) scares people whose livelihood and calling it is to creatively synthesize belles-lettres and other artifacts, and (on the other hand) disappoints everyone who thinks that they can finally become a one-man/woman garage-kubrick by paying $20 a month, and turning off their brain (that last part is the problem – these tools require a dialectical mindset, because you are basically talking to a holocron of the entire internet, a kind of artificial being that can finish your sentences for you, but has absolutely no concept of time and causality and consciousness (or that it even is any more than your car understands that it is (which is not to say that machines (of any kind) do not have souls))).

The AI companies do not understand[1] how to market the thing they have created, which betrays the fact that they do not understand/care who appreciates their product the most (it turns out, this demographic also happens to be unprofitable).

Worst thing about Gen-AI, IMHO, is that it makes the spam-problem much less tractable.[2]

Here is an author using custom ML/AI tools to synthesize sentences and thoughtfully select them (in 2017): link 

Backref for those who are not familiar with "Garage Kubrick": link (AI brings the "Garage" part closer, but the individual writer has to bring the "Kubrick" part on their own).

Hofstader is wonderful. He co-wrote some books with Dennet, and also started a project about analogy-making (called copycat), which is the subject of Melanie Mitchell's (one of his research students, IIRC) book "Analogy-Making as Perception", which you might enjoy if you enjoyed GEB (it's written for a technical audience, but is still accessible). It's amusing (if one reads the book) that all of the AI tech we use today was thought out in the 70s and 80s, and it just took 40 to 50 years for the hardware to catch up, and for the internet to fill up with our writings (minus a few details like what NN-hyperparameters were best for which tasks).

[1]: Or maybe they do. Calling an LLM a very sophisticated, first of its kind analytical tool is much more boring than calling it a magic genie – it also implies that one might have to do quite a bit of thinking in the process of using it and shaping its outputs, and that is a hard sell for people who are already mentally overwhelmed by various familiar demands.

[2]: I do wonder if technological progress in computing is, to a non-trivial degree, driven by the (figurative, of course) arms-race between spammer and spammee.

Show Comments