More Evidence that AI Excels at Generating Bullshit

29Apr23

I continue to find it mind-boggling that people seem to think that a good use of AI tools like ChatGPT is to gather accurate information. Any cursory reading on the topic should explain that ChatGPT is designed to predict language in response to a prompt, not actually present information based on “knowledge” or “intelligence.” I don’t pretend to understand how it really works, but it’s pretty clear that it’s not a reliable alternative to a search engine.

And yet, people keep using it that way. I had an odd experience with it this week in marking student essays: the prompt was to analyze a specific episode of TV (that I provided access to) using two concepts we’d covered in class. One student’s essay was well-written and the analysis was strong, demonstrating a good understanding of the concepts – but the specific moments in the episode they referred to were flat-out wrong, describing moments that never happened (and the essay didn’t discuss what actually did happen). When I met with them to inquire about this, they admitted being pressed for time and instead of watching the (30 minute) episode, they had ChatGPT produce a summary of it for them to analyze – which was completely inaccurate! So the writing and analysis was all the student’s, but they used AI to avoid having to watch TV and hence analyzed an episode that doesn’t exist!

One lesson: never take a short-cut to avoid watching TV! Another more broadly applicable lesson: don’t use AI instead of a search engine! There are many online summaries of this episode that are actually accurate (and still would be much worse than watching the episode), but ChatGPT didn’t look at those. Instead it generated a bunch of words and sentences that sound like they could be feasible accounts of what happened in the episode.

This got me wondering whether ChatGPT could summarize some films and television episodes effectively, and when it would just generate bullshit. As in my last post, where I showed how it invented details about films to analyze racial representation, there might be some instances when it succeeds in generating a feasible summary. Is there a pattern to what would films or episodes work better or worse to summarize? Experiment to follow…

Let’s start with an easy one:

summarize the plot of the film PSYCHO
The film “Psycho” directed by Alfred Hitchcock follows the story of Marion Crane, a young woman who steals money from her employer to start a new life with her boyfriend. While on the run, she stops at the Bates Motel, where she meets the proprietor, Norman Bates, who seems to have an unhealthy attachment to his mother. During her stay, Marion is brutally murdered in the shower by an unseen assailant. The investigation into her disappearance leads her sister and boyfriend to the Bates Motel, where they uncover the shocking truth about Norman’s relationship with his mother and his split personality disorder. In the film’s climactic scene, Norman’s true nature is revealed, and he is institutionalized for his crimes. The film is known for its iconic score, shocking twists, and the groundbreaking use of violence and psychological horror in cinema.

I’m not sure who the sister’s boyfriend refers to, but overall not horrible. [Update: it was pointed out to me that “her sister and boyfriend” means Marion’s sister and Marion’s boyfriend, while I thought it was the sister’s boyfriend – ChatGPT needs a little lesson on phrasing with possessive pronouns!] How about a more recent but well-known film?

summarize the plot of the film PARASITE
The film “Parasite” directed by Bong Joon-ho follows the story of a poor family, the Kims, who live in a cramped basement apartment in South Korea. The Kims gradually infiltrate the lives of the wealthy Park family by conning their way into becoming their employees. Through a series of deceitful acts, the Kims are able to secure employment for each member of their family in the Park household, but their secret schemes are threatened when an unexpected visitor arrives. The situation quickly spirals out of control, leading to a catastrophic event that exposes the stark class differences in Korean society. The film is known for its biting social commentary, subtle symbolism, and masterful storytelling, which earned it critical acclaim and multiple awards, including the Palme d’Or at the Cannes Film Festival and Best Picture at the Academy Awards.

Pretty solid, although I’m sure you can find a lot more detail via a web search.

summarize the plot of the film ADAPTATION.
The film “Adaptation” directed by Spike Jonze follows the story of Charlie Kaufman, a screenwriter struggling to adapt a non-fiction book about orchid hunting into a Hollywood screenplay. As Charlie becomes increasingly anxious and paranoid about his ability to adapt the book, he begins to incorporate his own personal experiences and anxieties into the script, blurring the line between reality and fiction. Along the way, Charlie becomes involved with Susan Orlean, the author of the book he’s adapting, and her husband John Laroche, an eccentric orchid hunter. As Charlie delves deeper into their lives, he becomes enmeshed in a web of deception and obsession, and must confront his own insecurities and flaws as a writer and a person. The film is known for its metafictional storytelling, its exploration of creative block and self-doubt, and its exceptional performances by Nicolas Cage, Meryl Streep, and Chris Cooper, who won an Academy Award for his supporting role as John Laroche.

So if I saw a student refer to John Laroche as Susan Orlean’s husband, I’d be pretty suspect! Clearly ChatGPT generates incidental misinformation in the process of stringing together likely words and phrases. What about a more obscure film?

summarize the plot of the film THE SOUVENIR part 2
“The Souvenir Part 2” is the sequel to the film “The Souvenir” directed by Joanna Hogg. The film continues the story of Julie, a film student who is now trying to move on from her tumultuous relationship with the older and drug-addicted Anthony. She is still studying at film school and becomes involved with an actor named Tom, who offers her a different perspective on life and art. As she tries to find her own voice as a filmmaker, Julie must also navigate the difficult relationships with her family, including her mother who is battling cancer, and her new lover, who has his own secrets and insecurities. Throughout the film, Julie reflects on the lessons she learned from her past relationship with Anthony and tries to apply them to her present and future. The film is known for its intimate and nuanced portrayal of the complexities of human relationships and the struggles of young artists trying to find their place in the world.

Pretty much half of what’s described above is completely and utterly untrue, but sounds plausible – in other words, bullshit. (But it’s a great film!)

How about TV episodes, where there’s less written specifically around the episode and more generally about the show itself? I tried a bunch of episodes that I show in my Television & American Culture course:

summarize the plot of the ALL IN THE FAMILY episode “Sammy’s visit”:
In the “Sammy’s Visit” episode of All in the Family, Sammy Davis Jr. pays a visit to the Bunker household. Archie Bunker, the patriarch of the family, is initially resistant to the idea of having a black man in his home, but he eventually warms up to Sammy after he sings a song for him. However, the rest of the family is not as accepting of Sammy, particularly Archie’s son-in-law, Mike, who is appalled when Sammy kisses him on the cheek as a greeting. In the end, the family comes together to take a group photo with Sammy before he departs.

Again, a mix of real info and utter bullshit, and in this case, it violates the core conflict of the episode.

summarize the plot of the I LOVE LUCY episode called “Lucy Does a TV Commercial”
In the “Lucy Does a TV Commercial” episode of I Love Lucy, Lucy is offered the chance to appear in a TV commercial for “Vitameatavegamin,” a health tonic that contains alcohol. Lucy struggles with the taste of the tonic and becomes increasingly intoxicated as she films the commercial, leading to hilarious results. Meanwhile, her husband Ricky, who is directing the commercial, becomes frustrated with Lucy’s antics and has to find a way to finish the shoot despite her drunkenness. In the end, the commercial is a success, and Lucy becomes a local celebrity for her memorable performance.

Starts strong, then veers off into total bullshit.

summarize the plot of the 30 ROCK episode called “Jactor”
In the “Jactor” episode of 30 Rock, Tracy Jordan, one of the stars of the show within the show, wants to break out of his comedic typecast and win an Academy Award for a serious drama. However, his attempt at serious acting is terrible, and Liz Lemon, the show’s head writer, and Jenna Maroney, another actor on the show, must find a way to convince Tracy to stick to comedy. Meanwhile, Jack Donaghy, the network executive, tries to avoid his visiting mother by having her stay with Liz, which causes tension between the two. Ultimately, Tracy realizes that he is better suited for comedy and returns to his roots, while Liz and Jack make amends.

Completely wrong for this episode, but it sounds like it could be a feasible 30 Rock plot.

summarize the plot of the PARKS & RECREATION episode called “Pawnee Zoo”
In the “Pawnee Zoo” episode of Parks & Recreation, Leslie Knope, the deputy director of the Parks Department, plans a publicity stunt for the local zoo to boost Pawnee’s public image. However, her plan backfires when she accidentally falls into the pit of the exhibit and becomes trapped with a local television news anchor. Meanwhile, Andy Dwyer, a Parks Department employee, adopts a live cougar, which he mistakenly believes is a cat, and brings it to the office. Chaos ensues as the Parks Department tries to wrangle the dangerous animal, leading to a tense confrontation with animal control. In the end, Leslie is rescued from the pit, and Andy’s cougar is returned to the zoo, but not before causing more chaos. The episode is known for its humor, physical comedy, and the introduction of the recurring character, Chris Traeger.

I mean, that sounds really fun, but never happened on the series! And now an all-time pedagogical classic:

summarize the plot of the FATHER KNOWS BEST episode “Betty, Girl Engineer”
In the “Betty, Girl Engineer” episode of Father Knows Best, Betty, the oldest daughter of the Anderson family, becomes interested in pursuing a career in engineering after a visit to her father’s workplace. Despite initial skepticism from her family and guidance counselor, Betty perseveres and works hard to excel in her math and science classes. However, when she enters a city-wide engineering contest, she faces discrimination and hostility from her male competitors, who refuse to take her seriously. Despite the obstacles, Betty’s determination and intelligence shine through, and she ultimately wins the contest and earns the respect of her peers. The episode is known for its progressive message about gender roles and the importance of pursuing one’s dreams, as well as its realistic portrayal of the challenges faced by women in male-dominated fields.

If only…

I tried a bunch of famous TV episodes, hoping to find one whose summary contained no bullshit – a task that is much harder than for films. The closest I could find:

summarize the plot of the SOPRANOS episode “College”
In the Sopranos episode “College”, Tony takes his daughter Meadow on a tour of colleges in Maine, but on the way, he discovers that a former mob associate, Fabian Petrulio, has relocated to the area under the Witness Protection Program. Tony sees an opportunity to take revenge on Petrulio for testifying against him, and decides to track him down. Meanwhile, Meadow explores the college campus and spends time with her father, unaware of the danger he is putting them in. As Tony becomes more obsessed with finding Petrulio, he faces unexpected challenges and moral dilemmas that test his loyalty to the mafia and his relationship with his daughter. The episode is known for its tense atmosphere, complex character development, and its exploration of the blurred line between family and organized crime.

Nothing is outright wrong in this one, but it completely omits the subplot with Carmella and Father Phil, which is pretty essential to the episode.

In short, do not ask ChatGPT to summarize things, as it will likely offer a combination of information and bullshit in varying degrees. And most importantly, if your professor asks you to watch TV, do it!

Filed under: Academia, digital humanities, Film, Teaching, Technology, Television   |  2 Comments
Tags: AI, bullshit, ChatGPT