How Small Can Language Models Be and Still Speak Coherent English?
www.bulaev.net
This morning I found research named "TinyStories: How Small Can Language Models Be and Still Speak Coherent English?". It was done in May of 2023. It is pretty interesting. TinyStories is a unique synthetic dataset comprised of short, easy-to-understand stories crafted by GPT-3.5 and GPT-4. These stories contain only words familiar to 3 to 4-year-old children, providing a different angle to test the capabilities of Small Language Models (SLMs). Although SLMs are much more compact than the latest models, the study shows that when trained with TinyStories, these SLMs can produce fluent, coherent, and diverse multi-paragraph stories demonstrating near-perfect grammar and reasoning skills.
How Small Can Language Models Be and Still Speak Coherent English?
How Small Can Language Models Be and Still…
How Small Can Language Models Be and Still Speak Coherent English?
This morning I found research named "TinyStories: How Small Can Language Models Be and Still Speak Coherent English?". It was done in May of 2023. It is pretty interesting. TinyStories is a unique synthetic dataset comprised of short, easy-to-understand stories crafted by GPT-3.5 and GPT-4. These stories contain only words familiar to 3 to 4-year-old children, providing a different angle to test the capabilities of Small Language Models (SLMs). Although SLMs are much more compact than the latest models, the study shows that when trained with TinyStories, these SLMs can produce fluent, coherent, and diverse multi-paragraph stories demonstrating near-perfect grammar and reasoning skills.