o1 says "I’m not aware of a widely recognized or universally known product called “CurrentKey Stats,” so it may not be a mainstream or highly publicized app. There are a few possibilities, though:" and then some guesses.
o3-mini is similar: "I couldn’t find any widely recognized information about an app specifically called CurrentKey Stats. It’s possible that:"
When I turn on its shiny new search capability, it correctly identifies and summarizes the app.
o3-mini-high, search turned off, asks for clarification.
Which means the default free 4o is quietly doing retrieval-augmented generation behind the scenes. I thought o3-mini would search if it didn't know the topic, but I might be misremembering.
Fascinating. I would have thought the fancier models would have definitely known about it. The app is six years old, has ~120 ratings globally with ~4.5 stars, with several articles written about it etc.. It should be pretty widely available in the training data
o1 says "I’m not aware of a widely recognized or universally known product called “CurrentKey Stats,” so it may not be a mainstream or highly publicized app. There are a few possibilities, though:" and then some guesses.
o3-mini is similar: "I couldn’t find any widely recognized information about an app specifically called CurrentKey Stats. It’s possible that:"
When I turn on its shiny new search capability, it correctly identifies and summarizes the app.
o3-mini-high, search turned off, asks for clarification.
Which means the default free 4o is quietly doing retrieval-augmented generation behind the scenes. I thought o3-mini would search if it didn't know the topic, but I might be misremembering.