What is the MOTES?

MOTESMost educators and parents share the goal of fostering their students’ capacity to think originally. How, then, do educators know whether their instructional practices positively impact students’ development of original thinking? This is where the measurement of original thinking comes in. 

For several decades, psychologists and educators have focused on measuring original thinking in both adults and children. However, assessments of original thinking have had limited applicability in the school context due to the time-intensive manual scoring procedures and high costs. As a result, fewer children are assessed on their original thinking compared to other psychological attributes such as intelligence: a situation that has led directly to under-representation of historically marginalized minority students and low-SES learners in gifted and talented programs. 

In this project, we aim to develop a rapidly scorable, freely accessible, reliable and valid measure of original thinking for elementary school students which we call MOTES. We intend MOTES to be a widely applied as a universal screening tool for gifted and talented programs in U.S. schools. The MOTES is comprised of four divergent thinking tasks that will be scored using text-mining methodology, meaning that no human reader will be required to score the test. Throughout the course of this project, we will psychometrically evaluate the reliability, validity, and fairness of MOTES scores across demographic groups.

Meet the Team


More About Us

We are excited to provide the MOTES, as well as the text-mining technology used to score it, free to the research community via this website when it is ready. For inquiries about using the MOTES, please contact us via email (motes.info@gmail.com) or check this website for updates on its availability.

Why does creativity matter?

In a fast-changing world with unforeseeable challenges and problems, a major goal of education should be to prepare young learners for the future (Craft, 2011; Treffinger et al., 2012). A recent analysis of 702 occupations indicated that 47% of total employment in the United States is at risk because of automatization (Frey & Osborne, 2017). According to the report by the Institute for the Future (2017), most jobs that today’s children will apply for in the future do not yet exist. Therefore, it is important to design our educational structure and curriculum according to this picture and priority should be given to skills and capabilities that will make future generations ready for the challenges of tomorrow. The framework of 21st Century Skills (Partnership for 21st Century Skills, 2006) was developed to reflect such a future-oriented perspective. Becoming “future-proof” requires a number of skills including creativity, problem-solving, cognitive flexibility and adaptability, and innovative thinking. Parallel to this, LinkedIn Learning analyzed the skills needed in companies by using an Economic Graph that represents the entire data on LinkedIn and found that creativity came out to be the most needed skill (Petrone, 2019). This technical analysis echoes World Economic Forum’s (2016) report that featured creativity and related abilities (i.e., complex problem solving, cognitive flexibility) as most needed skills in 2020 and survey results of IBM’s (2010) research with 1,500 Chief Executive Officers from 60 countries and 33 industries around the world.

Those findings and reports are far from a surprise. There are original thoughts and ideas behind every novelty and progress. Every new technology we cherish starts with an original idea. Decades of scholarship has been dedicated to the theoretical foundation of this crucial construct. Creativity is often defined as the production of novel/original and useful/effective ideas or solutions (Stein, 1953; Runco & Jaeger, 2012). Based on The U.S. Patent Office’s criteria, some (e.g., Boden, 2004; Bruner, 1962; Simonton, 2012; 2018) added “surprising” or “non-obvious” as a third component. Clearly, creativity is multi-faceted and the magnitude of the relevance of each of those components is pertinent to an accurate understanding of the concept. Acar et al. (2017) found that among those three criteria and the criteria of aesthetics and elegance, originality was the strongest predictor of creativity followed by surprise and value, respectively. Additionally, surprise was also more related to originality than usefulness. Those findings are parallel to those of Diedrich, Benedek, Jauk, and Neubauer (2015) who tested the usefulness of bi-partite definition on idea evaluation. They found that both novelty and usefulness were significant predictors of creativity but novelty was a better predictor than usefulness. Evidently, originality is the core aspect of creativity. It is the only component that most scholars indicate as a necessary condition of creativity (Rothenberg & Hausman, 1976; Runco, 1988). This is why we focus on this particular cognitive skill in the MOTES project.


Craft, A. (2011). Creativity and education futures: Learning in a digital age. England: Trentham Books.

Treffinger, D. J., Schoonover, P. F., & Selby, E. C. (2012). Educating for creativity and innovation. Waco, TX: Prufrock Press.

Frey, C. B., & Osborne, M. A. (2017). The future of employment: How susceptible are jobs to computerisation? Technological Forecasting and Social Change, 114, 254–280. https://doi.org/10.1016/j.techfore.2016.08.019

IBM (2010). Capitalising on complexity: Insights from the global chief executive officer (CEO) study. Portsmouth: UK: IBM United Kingdom Limited.

Runco, M. A. (1988). Creativity research: Originality, utility, and integration. Creativity Research Journal, 1, 1–7. https://doi.org/10.1080/10400418809534283

Rothenberg, A., & Hausman, C. R. (1976). The creativity question. Durham, NC: Duke University Press.

Diedrich, J., Benedek, M., Jauk, E., & Neubauer, A. C. (2015). Are creative ideas novel and useful? Psychology of Aesthetics, Creativity, and the Arts, 9(1), 35–40. https://doi.org/10.1037/a0038688

Bruner, J. S. (1962). The conditions of creativity. In H. E. Gruber, G. Terrell, & M. Wertheimer (Eds.), Contemporary approaches to creative thinking: A symposium held at the University of Colorado (pp. 1–30). New York, NY: Atherton Press.

Boden, M. A. (2004). The creative mind: Myths and mechanisms (2nd ed.). New York, NY: Routledge.

World Economic Forum (2016). The future of jobs: Employment, skills and workforce strategy for the fourth industrial revolution. http://www3.weforum.org/docs/WEF_Future_of_Jobs.pdf

Partnership for 21st Century Skills. (2006). A state leader’s action guide to 21st century skills: A new vision for education. Tucson, AZ: Partnership for 21st Century Skills.

Petrone, P. (2019). Why creativity is the most important skill in the world. https://www.linkedin.com/business/learning/blog/top-skills-and-courses/why-creativity-is-the-most-important-skill-in-the-world

Stein, M. I. (1953). Creativity and culture. The Journal of Psychology, 36(2), 311-322. https://doi.org/10.1080/00223980.1953.9712897

Runco, M. A., & Jaeger, G. J. (2012). The standard definition of creativity. Creativity Research Journal, 24(1), 92-96. https://doi.org/10.1080/10400419.2012.650092

How does text mining work?

This scoring system utilizes a massive corpus of 840 billion words scraped from the Internet, in order to represent the general structure of the English language. So, the model knows what words tend to be typically associated with the prompt on a divergent thinking task, and it can identify words that are less commonly associated.

In order to quantify the degree of Originality for a particular response on a DT task, the model represents the prompt and the response as vectors in semantic space. Then, by calculating the cosine of the angle between the vectors, it calculates the latent correlation among the ideas. Below is a figure illustrating what this might look like for the AUT prompt “shovel” and two responses: “dig a hole” and “fling tennis balls for a dog to chase”. The angles in this figure correspond to those that would be outputted by the GloVe 840B text-mining model, which this freeware is based on. Please visit The Creativity Open Scoring Website for more information.