Why does the FDI AI Assistant occasionally return different results when the same question is asked?

Question

We have observed on occasion that the AI Assistant in FDI will return different results when asking it the same question. Why is that? It makes the feature seem unreliable to our business users when this occurs.

In the below screenshots, you can see an example where the same query is being posed to the Assistant, but it returns two separate, but related, graphics.

Gianni Ceresa · Answer

By looking at your 2 results, the difference is in what metric you expect to be used to define a Top10 list. I would say that your question is the reason for the different results, because it is extremely vague. You ask a Top10, but don't give a single hint on what measure you want the Top10 to be based on.

If you ask such a generic question to 10 of your colleagues, you will also probably have different answers, because some will answer based on Top10 by money spent (your first chart I believe), while others will answer based on a Top10 by count (number of occurrences).

Is one better than the other? Not really, both are perfectly valid for your question, because you didn't ask anything precisely. And the answers tells you that, by explaining what the result is, what measure has been used.

I don't know what temperature is being used for the LLM, but I wouldn't exclude that for such a open question where you explicitly let the LLM decide what measure to pick, maybe even with 0 it would still switch between the 2 most common answers.

Improve your question, be a bit more precise, and the AI  Assistant will also answer in a more predictable way.

You can't really blame the AI Assistant… It's like if in Rome (Italy) you ask your map app on your phone the road to go to a church. There are over 900 churches in town, the chances it will always answer with the road to the same church are close to none.