Jan 12 2025 38 mins 7
Current text generators, such as ChatGPT, are highly unreliable, difficult to use effectively, unable to do many things we might want them to, and extremely expensive to develop and run. These defects are inherent in their underlying technology. Quite different methods could plausibly remedy all these defects. Would that be good, or bad?
https://betterwithout.ai/better-text-generators
John McCarthy’s paper “Programs with common sense”: http://www-formal.stanford.edu/jmc/mcc59/mcc59.html
Harry Frankfurt, "On Bullshit": https://www.amazon.com/dp/B001EQ4OJW/?tag=meaningness-20
Petroni et al., “Language Models as Knowledge Bases?": https://aclanthology.org/D19-1250/
Gwern Branwen, “The Scaling Hypothesis”: gwern.net/scaling-hypothesis
Rich Sutton’s “Bitter Lesson”: www.incompleteideas.net/IncIdeas/BitterLesson.html
Guu et al.’s “Retrieval augmented language model pre-training” (REALM): http://proceedings.mlr.press/v119/guu20a/guu20a.pdf
Borgeaud et al.’s “Improving language models by retrieving from trillions of tokens” (RETRO): https://arxiv.org/pdf/2112.04426.pdf
Izacard et al., “Few-shot Learning with Retrieval Augmented Language Models”: https://arxiv.org/pdf/2208.03299.pdf
Chirag Shah and Emily M. Bender, “Situating Search”: https://dl.acm.org/doi/10.1145/3498366.3505816
David Chapman's original version of the proposal he puts forth in this episode: twitter.com/Meaningness/status/1576195630891819008
Lan et al. “Copy Is All You Need”: https://arxiv.org/abs/2307.06962
Mitchell A. Gordon’s “RETRO Is Blazingly Fast”: https://mitchgordon.me/ml/2022/07/01/retro-is-blazing.html
Min et al.’s “Silo Language Models”: https://arxiv.org/pdf/2308.04430.pdf
W. Daniel Hillis, The Connection Machine, 1986: https://www.amazon.com/dp/0262081571/?tag=meaningness-20
Ouyang et al., “Training language models to follow instructions with human feedback”: https://arxiv.org/abs/2203.02155
Ronen Eldan and Yuanzhi Li, “TinyStories: How Small Can Language Models Be and Still Speak Coherent English?”: https://arxiv.org/pdf/2305.07759.pdf
Li et al., “Textbooks Are All You Need II: phi-1.5 technical report”: https://arxiv.org/abs/2309.05463
Henderson et al., “Foundation Models and Fair Use”: https://arxiv.org/abs/2303.15715
Authors Guild v. Google: https://en.wikipedia.org/wiki/Authors_Guild%2C_Inc._v._Google%2C_Inc.
Abhishek Nagaraj and Imke Reimers, “Digitization and the Market for Physical Works: Evidence from the Google Books Project”: https://www.aeaweb.org/articles?id=10.1257/pol.20210702
