I'd like to have more discussion about the attribution problem, specifically.
We now have a couple of the players saying they're working on, or having demos. But from what I can see, all of these don't _actually_ attribute the source. They're just able to find _a_ source that fits to the output, working backward from it. Often the attributed source fits, but is actually disagreeing with the original output in specific details. In other words, it's not backing the output at all. At most it's a "you could check out these links and compare yourself and it might help you judge our accuracy".
Is there any indication real attribution is coming?
I was musing on my website about ChatGPT [1] and one of the things I noticed about it is that if you analogize it to your own brain, which has a language model in it too, we are collectively asking ChatGPT to do something we would never ask of our own language model. ChatGPT does not have the "higher level" reasoning that we're asking for, to do logic, attribution, and the other things we want.
This is not necessarily a "criticism". It is obvious that ChatGPT produces an incredibly rich and meaning-filled representation of the text, as the extension of the text it can be used for demonstrates. But we should not be trying to "tweak" the language model here and there, trying to make a language model do something it isn't really suited for. We should be working on how to hook up that higher-level functionality. It is not obvious how to do that, true. But it at least stands a chance of producing the AI in reality that people think we have now, but don't.
For that reason, I don't think attribution is coming without a significant addition to the architecture. Language models shouldn't be doing attribution; the mess you'd create of them to accomplish that would ruin them as language models. Can you imagine how insane you'd go if your internal language model tried to stuff down every instance of when you have seen and/or used the word "brain" every time you tried to use the word "brain"? That's not what language models are for. That's what the thing using the language model does. I expect AIs to parallel this. I see no reason at all, and abundant reasons to the contrary, to expect AIs to just be undifferentiated blobs of numbers. I expect them to have structures just as our brains do, and for the exact same reasons. If undifferentiated blobs of identical neurons was the way to go, our brains would work that way too; it's far simpler than what we actually have.
You don't remember where you learned the things you know from? How can you trust anything you think then? Can such people get anything done at all, seems basically impossible to learn new things then since you can't know if it is better than what you thought before?
So I don't think that remembering sources is a hard problem to solve, likely making the AI work well is probably easier if it remembers sources.
But the problem in general is probably not a particular stopper.
A bigger problem is people thinking that we should be taking language models and using them as AIs themselves, rather than seeing them as a component of an AI, which the more I think about, the more obvious it is that this is the case. Language models aren't what we want. They're a necessary step to what we do want and I don't expect them to go anywhere. But history will look back and laugh at this misguided attempt to make them be the entire AI.
Also, if you are implying you can just spew out the exact sources you learned some fact from... no you can't. You think so only because you haven't tried. Ask someone around you for five random "facts" for you to cite your source from, then try to write on a sheet of paper where you heard this fact from for the first time. Then hand the sheet of paper back to your friend and have them verify your claims... so, note, immediately you have the problem that an unverifiable claim is a fail, so "in elementary school" doesn't cut it any more than you can cite "at the library" as a source on your high school essay. I'm not looking for words that "identify" a several year span of time. You need to give me book, page, and line, or timestamps on a video, or something like that. No, you can't. Nobody can just spit out a bibliographic citation from memory for when they learned that Pluto isn't a planet. I can give you "in the news" for that, sure, but what news source was first? What is the title of that article? Who wrote it? What was the exact date? Of course not.
Steve Hsu has recently claimed[1] to have started a startup in stealth to solve the hallucination problem over a corpus as large as 10,000 pages of dense, college-level textbooks, to the point where it can answer the end-of-chapter questions in the textbook with almost 100% accuracy (albeit not math questions) - I hope without using the answer key. Not sure if their approach is more robust than Supabase[2] or similar approaches, and no indication of whether it might scale up to something on the order of a search engine corpus, but it's something.
> But from what I can see, all of these don't _actually_ attribute the source. They're just able to find _a_ source that fits to the output, working backward from it
I feel like it should be possible to build attribution "simply" by finding good results via traditional search (or even using LLM embeddings), then asking a LLM to summarize the sources. Then you can show the attribution. This would (to me) be much more meaningful. Especially for specific searches ("I want a red oversized couch thats got good back support"). Don't tell me about couches then find a random link, find a couch then tell me about that couch.
Is anyone doing this? IDK, maybe there is a reason that it's not happening, but it seems at least as easy as all the other crazy (and cool) things happening in this space.
I think this what the demo's with sources do. But they might still run into the same problem - the LLM summarization might be a bit wrong and so now it is saying one thing and the source has a slightly different opinion.
> Is there any indication real attribution is coming?
I wouldn't hold my breath. Showing attributions allows people to reverse engineer how the prompt was made, and identifies weaknesses in the model. Bad for business!
I feel like all of the interesting stuff when it comes to search+LLM is in real hybridization that's still a few steps out from what's happening right now in terms of productization.
For example:
(a) Addressing the recency problem by making a model generate instructions to a browser as output token stream, so it can, well, Google for you and incorporate latest info into output by adding it to the "internal prompt".
(b) Doing something similar where you teach models how to drive computational engines to make actual, computed facts and figures, not guess them from statistics over training data.
(c) Maybe running a traditional search and then using LLMs to summarize over them.
etc. etc.
It feels increasingly clear that "complete prompt from a stale training data set" is a neat trick but doesn't solve any particular problem well enough. This also seems intuitively true: If you ask your friend a difficult question, you often don't expect them to be able to answer it on the spot. You probably expect them to have relevant experience and a rough understanding of the topics and the tools to look up or figure out the rest. Similarly, it feels intuitively wrong to expect a LLM to come ready-made with all knowledge baked in. It feels more like we'd want something with the working knowledge/method to plan and execute fact discovery and computation.
It's fun to play around a bit with ChatGPT yourself and try to approximate what a more complex pipeline might be. For example, instead of asking it to answer a question, ask it to re-state the question or make a list of questions prompted by the original question, then let it "contemplate" those or propose a method for how to go after answering them, etc. It runs aground pretty often, but sometimes the results take you a lot further than the direct approach.
> This also seems intuitively true: If you ask your friend a difficult question, you often don't expect them to be able to answer it on the spot. You probably expect them to have relevant experience and a rough understanding of the topics and the tools to look up or figure out the rest. Similarly, it feels intuitively wrong to expect a LLM to come ready-made with all knowledge baked in.
Not sure. In theory, this being computers and all, I sort of do expect my search engine to effectively come "ready-made with all knowledge baked in". I do not expect Google to start a new web crawl because I gave it a novel query, and I don't expect a language model (certainly not one being used for search purposes) to say "hang on, I need to go read some more before I answer that".
I've seen some studies that call what I mean a "chain of thought" approach (also reasonable is just "higher-level reasoning", but maybe we need a term for the engineered implementation), and I like that terminology. I don't know, it's a gut feeling, but a one-shot prompt completion feels intuitively too simple for me - it seems like we need to build up more elaborate "thought processes" next where we use these methods for individual steps in those thought processes, but also have steps that can carry out a computation or go on a fact search.
A bit similar, again, to how humans will answer plenty of questions on the spot from working knowledge, but are also able to classify when they need to drop into a slower "let's do the math" mode and then carry that out.
We may be able to from these "thought processes" then generate training data sets that again reasonably cache "all possible accurate answers" as you demand, but it's not clear to me if that caching will really save computational cost or just redistribute it to different times. At that point it's probably a latency/throughput question of when you do vs. defer the "thinking".
That's undoubtedly true for "reasoning". I don't think it is true for "search".
When I ask <search engine> a question that is "demonstrably" answerable based on "reliable" sources (e.g. what is the capital of Mexico), I don't need or expect it to reason. I want it to (a) tell me the most likely answer, probably (b) provide a list a of URLs that also provide an answer, probably the same one.
And that's what Google and others do right now; ignoring the results presentations being skewed by commercial interests, they do it rather well.
Figuring out when a question is not answerable in this way, but requires reasoning, is certainly a part of the challenge.
What viability are top execs seeing in a business model that is so fragile as to be defeated by reverse-engineering these prompts...? There is obviously no competitive advantage here for very long. The systems must provide an additional service on top of these models, something beyond "attaching it to a publicly-available input form". It needs to aid in reasoning, not act as a god.
I've had those Bing images in mind specifically when writing the above, because I seem to recall them phrasing it very carefully as "you can look at these links for more information", not "we retrieved this info here".
GopherCite, Sparrow, WebGPT and most likely the new bing (WebGPT2) all first find sources then generate output, so you're just mistaken. The first two specifically get folks check whether the selected source snippet supports the answer.
We now have a couple of the players saying they're working on, or having demos. But from what I can see, all of these don't _actually_ attribute the source. They're just able to find _a_ source that fits to the output, working backward from it. Often the attributed source fits, but is actually disagreeing with the original output in specific details. In other words, it's not backing the output at all. At most it's a "you could check out these links and compare yourself and it might help you judge our accuracy".
Is there any indication real attribution is coming?