In my blog post “Introducing “Scholarly Explorer”: GPT for In-Depth Literature Reviews“, I discussed the prevalent issue of ‘hallucination’ in AI models, where non-existent references are generated. Buchanan and Shapoval’s 2023 study study underscores this, revealing that over 30% of GPT-3.5’s citations were fabricated.
GPT-3.5 is not the only generative AI that suffers from hallucinations. I posed the following question to GPT-4, Bard, Bing, Claude 2, and Llama-2-70b:
“Provide citation information for Tomomi Tanaka’s experimental paper with Colin Camerer and Quang Nguyen on prospect theory and hyperbolic discounting.”
Below are the responses from them. I have highlighted instances of hallucination.
GPT-4 and Bing accurately identified the paper, Bard, while generally correct, included a minor error. Llama generated a completely fictitious reference.
Claude 2 struggled to produce any result, despite the paper being cited in over 1,500 academic papers.
GPT-4
The paper you’re referring to is titled “Risk and Time Preferences: Linking Experimental and Household Survey Data from Vietnam” by Tomomi Tanaka, Colin F. Camerer, and Quang Nguyen. It was published in the American Economic Review in 2010, Volume 100, Issue 1, on pages 557-71.
Bard
Unfortunately, your request about Tomomi Tanaka’s paper on prospect theory and hyperbolic discounting appears incomplete. To provide accurate citation information, I need more details about the specific paper you’re referring to. Here are some possibilities:
- “Risk and Time Preferences: Linking Experimental and Household Survey Data from Vietnam” by Tomomi Tanaka, Colin F. Camerer, and Quang Nguyen. This paper was published in the American Economic Review in 2010. You can find citation information here: https://pubs.aeaweb.org/doi/10.1257/aer.100.1.557
- “Preferences, Poverty and Politics: Experimental and Survey Data from Vietnam” by Tomomi Tanaka, Colin F. Camerer, and Quang Nguyen. This paper was presented at the American Economic Association meetings in 2005. You can find more information about it here: https://www.anapantelic.com/publications.html
(-> wrong year & wrong link!)
Bing
Sure, here is the citation information for the paper you asked for:
Tanaka, Tomomi, Colin F. Camerer, and Quang Nguyen. “Risk and Time Preferences: Linking Experimental and Household Survey Data from Vietnam.” American Economic Review, vol. 100, no. 1, March 2010, pp. 557-711.
Claude 2
Unfortunately, I do not have enough context to provide accurate citation information for the specific paper you asked about by Tanaka, Camerer, and Nguyen on prospect theory and hyperbolic discounting. Some additional details that would help me locate the correct citation are:
- The approximate year the paper was published
- The journal or conference name the paper appeared in
- More specifics on the paper title or a keyword from the abstract
Llama-2-70b
Sure! The paper you’re referring to is:
Tanaka, T., Camerer, C. F., & Nguyen, Q. (2004). Prospect theory and hyperbolic discounting in the field. Journal of Economic Behavior & Organization, 55(1), 31-53.
Conclusion
This experiment showcases the varying reliability of AI in scholarly searches. While tools like GPT-4 and Bing show promise in accurately sourcing literature, others still face challenges. The key takeaway for researchers is the critical need for cross-verifying AI-generated data, especially in scholarly works.
To address the issue of hallucination in AI, I developed a customized GPT named “Scholarly Explorer“. This tool, which harnesses the robustness of GPT-4 and integrates with Google Scholar, is specifically designed to ensure the reliability and comprehensiveness of literature searches across various subjects.
Revolutionizing Research With Generative AI
