Remember how COVID-19 was going to turn education on its head? Here is the latest threat: Artificial Intelligence. It never ceases to amaze me how little impact technology has on education, given its reach into all other areas of life. Presently, the top apps can churn out mediocre, repetitive work to simple prompts. Should we be panicking? Embracing its possibilities? Or just designing better tasks?
This example seems impressive at first. But if you look closely, it’s fairly vague. Caveat: my geopolitical knowledge about Japan is not close to expert-level. But what exactly are Japan’s political strengths here? I don’t recall too many terrorist attacks in Japan. You’ll also notice that there is very little here on political strengths and susceptibilities because that would require some evaluative skill. Other than this detail, the prompt is fairly simple and requires a factual response, something that AI is reasonably good at. Throw in an ‘evaluate’ and this is what you get.
Evaluate the impact of geography and politics on Japan's susceptibility to natural and man-made disasters.
Japan is a country that is highly susceptible to both natural and man-made disasters. The country is situated in a seismically active region and is regularly hit by natural disasters such as earthquakes, tsunamis, and typhoons. In addition, Japan is also susceptible to man-made disasters such as nuclear accidents.
The impact of geography and politics on Japan's susceptibility to natural and man-made disasters is significant. The country's geographical location makes it prone to natural disasters, and its political situation makes it vulnerable to man-made disasters.
The geographical location of Japan makes it susceptible to natural disasters such as earthquakes, tsunamis, and typhoons. The country is situated in a seismically active region and is regularly hit by natural disasters. For example, in 2011, Japan was hit by a massive earthquake and tsunami which caused widespread damage and loss of life. In addition, the country is also frequently hit by typhoons, which can cause significant damage.
The political situation in Japan also makes the country vulnerable to man-made disasters. The country has a number of nuclear power plants, and the 2011 Fukushima Daiichi nuclear disaster showed the world the devastating effects that a nuclear accident can have. The Fukushima disaster had a profound impact on the Japanese people and their attitude towards nuclear power, and it is still having an impact on the country today.
The 1,000-word version contained more facts but avoided politics altogether. I’ve used the gold-standard Open-AI’s GPT-3 (the “text-davinci-002” model for the curious).
Being an English teacher, I thought I would test it as perhaps a way to save time generating advance organisers for skim reading. A Streetcar Named Desire by Tennessee Williams seemed like a good choice — there would be plenty of information about it on the internet, but not the bazillion summaries that exist for something like Macbeth. In one summary, the wording suggested that Stella had two husbands. Blanche was pregnant. Satisfyingly, Stella leaves Stanley but unfortunately, Blanche commits suicide.
Following is the abject disaster that the app produced in response to a question asking students to analyse how Williams problematises gender in a rapidly changing world.
In "A Streetcar Named Desire", Tennessee Williams problematises gender in a rapidly changing world by depicting the conflict between traditional gender roles and the changing attitudes of society. The playwright uses the characters of Stanley and Stella to represent the old guard, who cling to traditional gender roles, and Blanche and Mitch to represent the changing attitudes of society.
Blanche is a woman who does not conform to traditional gender roles. She is a woman who is not afraid to speak her mind, and she is not afraid to challenge the status quo. Blanche is a woman who is comfortable in her own skin, and she is not afraid to show her vulnerability. Stella is a woman who conforms to traditional gender roles. She is a woman who is submissive to her husband, and she is a woman who is not comfortable in her own skin. Mitch is a man who is comfortable in his own skin, and he is not afraid to show his vulnerability. He is a man who is open to change, and he is a man who is willing to listen to Blanche's ideas.
The conflict between traditional gender roles and the changing attitudes of society is evident in the relationship between Stanley and Stella. Stanley is a man who believes that a man should be the head of the household. He is a man who is not afraid to use force, and he is a man who believes that a woman should be submissive to her husband. Stella is a woman who is tired of being submissive to her husband. She is a woman who is ready to stand up for herself, and she is a woman who is ready to challenge the status quo.
The conflict between traditional gender roles and the changing attitudes of society is also evident in the relationship between Blanche and Mitch. Blanche is a woman who is not afraid to speak her mind, and she is a woman who is not afraid to challenge the status quo. Mitch is a man who is open to change, and he is a man who is willing to listen to Blanche's ideas.
In "A Streetcar Named Desire", Tennessee Williams problematises gender in a rapidly changing world by depicting the conflict between traditional gender roles and the changing attitudes of society. The playwright uses the characters of Stanley and Stella to represent the old guard, who cling to traditional gender roles, and Blanche and Mitch to represent the changing attitudes of society.
As well as some inaccuracies and contradictions, there is no analysis, no discussion of why the world was changing, and it’s essentially a descriptive recount. I love any kind of tech that can reduce workload and enhance learning, so I was at least willing to test the basic uses and limits. One of the responses to the original tweet was to deride the idea of simply banning student use of AI as a breach of academic integrity. Others made vague gestures about integrating its use as a tool for teaching writing. I’m not sure if this method would be aligned to productive failure in a currently inchoate way. I’m unconvinced.
So I think the question we are all asking is, should we worry? My answer is, not right now. An obvious but unpopular solution is to set examinations! This aside, I think the kinds of students who might dabble in academic malpractice are the same students who are currently buying essays, using paraphrasing tools and just lifting the work of others already. With regular formative assessment, it’s not too difficult to spot a piece of writing from a student that doesn’t match the usual quality of their output. And a well-crafted assessment task makes this kind of technology next to useless.
I am actually having way too much fun playing with this thing. I think the tech is a good way to start a conversation about academic integrity. If I had plenty of time on my hands (I don’t), I could design a lesson about the pitfalls that I have discussed here. I could even draw a long bow and say that it has tested my knowledge of Streetcar. I would genuinely love to hear from anyone who can think of a legitimate and economical use of this tech that I haven’t thought of. Other than enabling students to cheat sloppily, I’m coming up short.
As someone who has regularly trained language models, fine tuned and used LLM's including GPT3 almost daily, I think I can comment.
1. When you prompt the algorithm to "write an essay on" you won't get much success, without showing it some examples first and with a bit of breakdown at the paragraph level. Finding an efficient way of breaking down what "analysis" is and feeding it into the model in a way it understands (few - shot learning) will help it learn what you want.
2. In fact, not just analysis, anything else that fits the marking criteria - quotes form the book, clarity of language. There is an emerging field of AI engineering called "prompt engineering" geared towards how to get these generative models to produce what you want.
3. It is possible to fine tune davinci to generate analytical paragraphs. With enough labelled data, writing an analytical paragraph is possible.
The rumor is that GPT-4 is coming out soon, maybe early next year. It'll be better, but my guess is that it will have some of the same pitfalls of GPT-3.