GPT-3 Summarizes Chaos Theory

For this GPT-3 experiment, I'm trying to see if GPT-3 can summarize something that would be inaccessible to most people.

Summarization is interesting to me because it could be a massive equalizer. There are core ideas in every field that would be good understand intuitively, however, many people don't have access to the people/content nor the time to gain this intuitive knowledge.

When you think about it, summarization is the core of education. We spend time with instructors to distill concepts and add these ideas to the mental models we use to understand the world. For those who can't get a formal education (or chose to continue their eduction), books, videos, and blog posts are a great source of new models. Shrinking the summary, from the size of a 4 year degree, to the size of a book, to the size or a couple of paragraphs would help concepts spread faster and reach more people --not to mention the potential for personalization.

My expectation is that GPT-3 would be able to create a summary that is correct and understandable by the target consumer --ex a second grader.

For this example, I use the introductory description of Chaos Theory from wikipedia.

I used the below prompt and the operations a few times to see what it outputs.

My second grader asked me what this passage means:

"""

Chaos theory is a branch of mathematics focusing on the study of chaos—states of dynamical systems whose apparently-random states of disorder and irregularities are often governed by deterministic laws that are highly sensitive to initial conditions. Chaos theory is an interdisciplinary theory stating that, within the apparent randomness of chaotic complex systems, there are underlying patterns, interconnectedness, constant feedback loops, repetition, self-similarity, fractals, and self-organization. The butterfly effect, an underlying principle of chaos, describes how a small change in one state of a deterministic nonlinear system can result in large differences in a later state (meaning that there is sensitive dependence on initial conditions). A metaphor for this behavior is that a butterfly flapping its wings in China can cause a hurricane in Texas.

Small differences in initial conditions, such as those due to errors in measurements or due to rounding errors in numerical computation, can yield widely diverging outcomes for such dynamical systems, rendering long-term prediction of their behavior impossible in general. This can happen even though these systems are deterministic, meaning that their future behavior follows a unique evolution and is fully determined by their initial conditions, with no random elements involved. In other words, the deterministic nature of these systems does not make them predictable. This behavior is known as deterministic chaos, or simply chaos. The theory was summarized by Edward Lorenz as:

"Chaos: When the present determines the future, but the approximate present does not approximately determine the future."

Chaotic behavior exists in many natural systems, including fluid flow, heartbeat irregularities, weather and climate. It also occurs spontaneously in some systems with artificial components, such as the stock market and road traffic. This behavior can be studied through the analysis of a chaotic mathematical model, or through analytical techniques such as recurrence plots and Poincaré maps. Chaos theory has applications in a variety of disciplines, including meteorology, anthropology, sociology, physics, environmental science, computer science, engineering, economics, biology, ecology, pandemic crisis management, and philosophy. The theory formed the basis for such fields of study as complex dynamical systems, edge of chaos theory, and self-assembly processes.

Results:

Attempt 1:

I rephrased it for him, in plain language a second grader can understand:

"""

A system that is deterministic (the same outcome every time, with no randomness involved) can still be unpredictable (not able to know what the outcome will be).

Attempt 2:

I rephrased it for him, in plain language a second grader can understand:

"""

"Chaos theory is a branch of mathematics that studies how things go from being organized to being disorganized.

Attempt 3:

I rephrased it for him, in plain language a second grader can understand:

"""

A complex system is one where many things are happening at the same time. The weather is a complex system. It's made up of many things, like air, water, and wind. And those things are moving around and changing all the time. It's hard to predict what the weather will be like tomorrow, because it's so complex.

Attempt 4:

I rephrased it for him, in plain language a second grader can understand:

"""

Chaos theory is a branch of mathematics that studies the behavior of complex systems that change over time. The word "complex" means that these systems have many different parts. The word "system" means that these parts are all connected to each other in some way.

Summary for a twelfth grader:

Twelfth grader response 1:

I rephrased it for him, in plain language a second grader can understand:

"""

Chaos theory is a branch of mathematics focusing on the study of chaos, or situations where there is no order. Chaos theory is an interdisciplinary theory stating that, within the apparent randomness of chaotic complex systems, there are underlying patterns, interconnectedness, constant feedback loops, repetition, self-similarity, fractals, and self-organization. The butterfly effect, an underlying principle of chaos, describes how a small change in one state of a deterministic nonlinear system can result in large differences in a later state (meaning that there is sensitive dependence on initial conditions). A metaphor for this behavior is that a butterfly flapping its wings in China can cause a hurricane in Texas.

Twelfth grader response 2:

I rephrased it for him, in plain language a second grader can understand:

"""

Chaos theory is a branch of mathematics that deals with situations where things happen in a way that can't be predicted.

Twelfth grader response 3:

I rephrased it for him, in plain language a second grader can understand:

"""

Chaos theory is a branch of mathematics that studies how the behavior of a system changes over time. It looks at how the smallest change in the beginning can cause the biggest change in the end.

Twelfth grader response 4:

I rephrased it for him, in plain language a second grader can understand:

"""

Chaos theory is a branch of mathematics that focuses on the study of chaos. Chaos is when the present determines the future, but the approximate present does not approximately determine the future.

Summary for eighth grader:

Eighth grader response 1:

I rephrased it for him, in plain language a second grader can understand:

"""

Chaos theory is a branch of mathematics focusing on the study of chaos—states of dynamical systems whose apparently-random states of disorder and irregularities are often governed by deterministic laws that are highly sensitive to initial conditions. Chaos theory is an interdisciplinary theory stating that, within the apparent randomness of chaotic complex systems, there are underlying patterns, interconnectedness, constant feedback loops, repetition, self-similarity, fractals, and self-organization. The butterfly effect, an underlying principle of chaos, describes how a small change in one state of a deterministic nonlinear system can result in large differences in a later state (meaning that there is sensitive dependence on initial conditions). A metaphor for this behavior is that a butterfly flapping its wings in China can cause a hurricane in Texas.

Eighth grader response 2:

I rephrased it for him, in plain language a second grader can understand:

"""

Chaos theory is a branch of mathematics that looks at the randomness of a system. It's a system that has a pattern, but the pattern is not predictable. For example, if you drop a rock in a puddle of water, you can't predict exactly where it will hit. You can predict the general area, but not the exact spot. This is an example of chaos theory.

Eighth grader response 3:

I rephrased it for him, in plain language a second grader can understand:

"""

Chaos theory is a branch of mathematics focusing on the study of chaos—states of dynamical systems whose apparently-random states of disorder and irregularities are often governed by deterministic laws that are highly sensitive to initial conditions. Chaos theory is an interdisciplinary theory stating that, within the apparent randomness of chaotic complex systems, there are underlying patterns, interconnectedness, constant feedback loops, repetition, self-similarity, fractals, and self-organization. The butterfly effect, an underlying principle of chaos, describes how a small change in one state of a deterministic nonlinear system can result in large differences in a later state (meaning that there is sensitive dependence on initial conditions). A metaphor for this behavior is that a butterfly flapping its wings in China can cause a hurricane in Texas.

Eighth grader response 4:

I rephrased it for him, in plain language a second grader can understand:

"""

Chaos theory is a branch of mathematics that studies how some things in nature change over time. Things like how the weather changes, or how the stock market changes, or how the price of a product changes.

Conclusions/Future Work

Notice how the model assume "him" automatically as my student. I wonder if this indicates a bias in the model.

Although interesting, I think that generally the summarization capabilities fall short in this experiment. A handful of the answers are really good summaries but its pretty inconsistent. To be usable in a product, the summary would need to be perfect the first time since the user would not necessarily have prior knowledge of the text needing the summary.

Given the model was trained on reddit, an interesting future experiment could be to test the summarizations of posts on reddit.