News!
- February 2017: The first Story Cloze Test shared task is now concluded, you can read the summary paper here! Visit the Shared Task page for more details.
- December 2016: ROCStories Winter 2017 release with 52,666 new stories is now out! Get access to the dataset below.
- November 2016: Story Cloze Test is now the shared task for LSDSemâ17. Register by 15 Dec! To learn more please visit the Shared Task page.
- October 2016: Have a system on the Story Cloze Test? Now you can submit your results on CodaLab to help the community keep up with the SOTA!
Please fill in this form to get access to the dataset, which is free to everyone. The latest release includes 98,159 ROCStories and 3,744 Story Cloze Test instances. If you do not receive the email within a few minutes, please feel free to contact us! Stay tuned for the upcoming Story Cloze Test sets.
âStory Cloze Testâ is a new commonsense reasoning framework for evaluating story understanding, story generation, and script learning. This test requires a system to choose the correct ending to a four-sentence story. We propose the Story Cloze Test to replace the state-of-the-art for evaluating narrative structure learning, the âNarrative Cloze Testâ (Chambers & Jurafsky, 2008).
To enable the Story Cloze Test, we created a new corpus of five-sentence commonsense stories, âROCStoriesâ. This corpus is unique in two ways: (1) it captures a rich set of causal and temporal commonsense relations between daily events, and (2) it is a high quality collection of everyday life stories that can also be used for story generation.
Examples
Story Cloze Test
| Context | Right Ending | Wrong Ending |
|---|---|---|
| Karen was assigned a roommate her first year of college. Her roommate asked her to go to a nearby city for a concert. Karen agreed happily. The show was absolutely exhilarating. | Karen became good friends with her roommate. | Karen hated her roommate. |
| Jim got his first credit card in college. He didnât have a job so he bought everything on his card. After he graduated he amounted a $10,000 debt. Jim realized that he was foolish to spend so much money. | Jim decided to devise a plan for repayment. | Jim decided to open another credit card. |
| Gina misplaced her phone at her grandparents. It wasnât anywhere in the living room. She realized she was in the car before. She grabbed her dadâs keys and ran outside. | She found her phone in the car. | She didnât want her phone anymore. |
ROCStories
| Title | Five-sentence Story |
|---|---|
| The Test | Jennifer has a big exam tomorrow. She got so stressed, she pulled an all-nighter. She went into class the next day, weary as can be. Her teacher stated that the test is postponed for next week. Jennifer felt bittersweet about it. |
| The Hurricane | Morgan and her family lived in Florida. They heard a hurricane was coming. They decided to evacuate to a relativeâs house. They arrived and learned from the news that it was a terrible storm. They felt lucky they had evacuated when they did. |
| Spaghetti Sauce | Tina made spaghetti for her boyfriend. It took a lot of work, but she was very proud. Her boyfriend ate the whole plate and said it was good. Tina tried it herself, and realized it was disgusting. She was touched that he pretended it was good to spare her feelings. |
Main Papers
LSDSem 2017 Shared Task: The Story Cloze Test In Proceedings of the EACL Workshop on Linking Models of Lexical, Sentential and Discourse-level Semantics (LSDSem), 2017
Relevant Papers and Resources
Any questions? Please contact: me [at] nasrin [dot] info