Some Interesting Limits to AI Film Criticism

26Jan23

Like many academics, I’ve been seeing a lot of concerns around how AI tools like ChatGPT might impact the work academics do as teachers and scholars. I don’t want to dive too much into those muddy waters, but I saw one post on Mastodon that piqued my interest and led me to do some experimenting. Ryan Cordell, noted digital humanist and literary/print scholar, posted the following thread:

In our lab introducing AI writing to students I encourage them to probe the models—to try & find limit cases & places where the “wires” of the machine become visible. Some things they found…
4. ChatGPT is famously good at synthesis, but it’s often unable to combine critical ideas in new ways. If asked to produce a Marxist analysis of various texts, it often produces one paragraph of Marxist thought & one about the chosen text, but can’t combine the two concepts.

I responded (with a little touch of snark): “I wonder if #4 is a byproduct of the training data. So much “critical writing” is effectively a paragraph of theory and a paragraph of textual description, devoid of actual analysis.”

Ryan wrote back, “It was quite stubborn about keeping the two concepts separate unless we asked about texts about which (I assume) there’s more extant critical writing. It will write a Marxist analysis of Moby Dick, likely channeling/plagiarizing existing writing in the training set, but if you ask for a Marxist analysis of a relatively new TV show you get Marxism then TV show summary/analysis—which makes sense given that it’s just putting words together based on existing words that go together.”

Ryan’s last point is crucial: the AI is not doing any sort of analysis, but rather using a huge dataset of writing (presumably the open web) to predict what’s the best next word to write to fulfill the given prompt. And thus it would make sense that making a coherent simulation of an analysis where there are already good models to emulate in your dataset would be much more effective than trying to invent something without decent precedents to draw upon.

And this got me thinking about running my own experiment to generate examples of film criticism that may expose some of the limits and underlying possibilities of ChatGPT.

I figured a basic prompt to analyze well-known films would be a good test case to see what the limits and strengths of the model might be. Building on Ryan’s example of Moby Dick, I gave the AI a starting prompt that would tackle a comparatively recent film, but one which has generated a lot of writing to draw upon:

Prompt: write an analysis of racial representation in the film Get Out

Get Out is a 2017 horror-thriller film written and directed by Jordan Peele. The film addresses themes of racism and white supremacy through its portrayal of a Black man, Chris, who becomes embroiled in a sinister plot involving white people who wish to transplant their own consciousness into the bodies of Black people.

The film’s representation of race is a complex and nuanced exploration of the ways in which racism is both overt and subtle, and how it can be used to exploit and oppress marginalized groups. Peele uses the horror genre to highlight the real-life horror of racism and white supremacy, depicting the ways in which Black bodies are objectified and dehumanized by white people. The film also critiques the way in which liberal white people can be complicit in racist systems, through their seemingly well-intentioned yet ultimately harmful actions.

The film’s portrayal of Black characters is generally positive, with Chris being depicted as a strong, intelligent, and capable protagonist. However, the film also highlights the ways in which Black people are often reduced to stereotypes, such as the “Magical Negro” trope, which can be seen in the character of the hypnotist, who is used as a plot device to further the white characters’ goals.

In addition to its exploration of race, the film also critiques the way in which the intersectionality of race and class can be used to exploit and oppress marginalized groups. This is seen in the way in which the white characters in the film are able to use their wealth and social status to gain access to Black bodies and exploit them for their own gain.

Overall, Get Out is a powerful and thought-provoking film that uses the horror genre to explore the real-life horror of racism and white supremacy. It is a sharp critique of the ways in which racism is both overt and subtle, and how it can be used to exploit and oppress marginalized groups.

Not too bad. Is it a good essay? By no means, but it’s competent and mostly coherent – although the “character of the hypnotist” is white, so I’m not sure how she could be a “Magical Negro.” But bringing in concepts like intersectionality, the complicity of liberal white people, and objectification suggests that the AI is at least in the ballpark. And that will be the closest that we will get…

Prompt: write an analysis of racial representation in the film vertigo

The 1958 film Vertigo directed by Alfred Hitchcock has been criticized for its representation of race. The film tells the story of a white man, played by James Stewart, who becomes obsessed with a woman, played by Kim Novak, whom he believes to be of a different race. The woman, Madeleine, is actually a white woman who is pretending to be of a different race, and her actions in the film are motivated by a desire to escape her past and create a new identity for herself.

Critics have pointed out that the film perpetuates the stereotype of the exotic and mysterious “other,” and that the portrayal of Madeleine as a woman of a different race reinforces the idea that white people are superior to people of other races. Additionally, the film’s portrayal of the relationship between the white male protagonist and the woman of a different race reinforces the idea of white men as “saviors” who are able to “save” and “civilize” people of color.

Furthermore, the film’s portrayal of the woman of a different race as a tragic figure who ultimately must be “saved” by the white male protagonist reinforces the idea that people of color are unable to save themselves. The fact that Madeleine, who is portrayed as a woman of a different race, is the only major character in the film who is not white, reinforces the idea that people of color are not important or meaningful in their own right and only exist to serve the white characters.

What the what?!? It’s been awhile since I watched Vertigo, but I’m pretty sure that neither Madeleine nor Judy are ever, presented as, or thought to be a person of color. There is a possibility that character’s lineage from Carlotta Valdes is getting misinterpreted, but this is pretty much incoherent.

Prompt: write an analysis of racial representation in the film psycho

The 1960 film Psycho directed by Alfred Hitchcock has also been criticized for its representation of race. The film tells the story of a white man, Norman Bates, who is the proprietor of a hotel, who becomes obsessed with a woman, played by Janet Leigh, whom he believes to be of a different race. The portrayal of the character of Norman Bates as a white man who is mentally unstable and violent, and his portrayal of the woman as a victim, reinforces harmful stereotypes about people of color.

Critics have also pointed out that the film’s portrayal of the character of Norman Bates as a white man who is mentally unstable and violent reinforces the idea that white people are dangerous and should be feared. Additionally, the film’s portrayal of the woman as a victim reinforces the idea that people of color, particularly women, are vulnerable and in need of protection from white men.

Furthermore, the film’s portrayal of the woman as a “femme fatale” reinforces harmful stereotypes about women of color as seductive and dangerous. This representation perpetuates the idea that people of color, particularly women, are a sexual threat to white men.

Overall, Psycho’s representation of race is problematic and reinforces harmful stereotypes about people of color. The film’s portrayal of the white male protagonist as a mentally unstable and violent individual, and the portrayal of the woman as a victim, reinforces harmful stereotypes about people of color and perpetuates harmful ideas about race.

Clearly we’ve hit a huge blindspot for ChatGPT: there is no fact-checking ability on claims like the racial identity of characters in a film! Instead what seems quite clear is that the predictive text has mapped the schema of “identify one character as an othered person-of-color to analyze race in a film” onto two well-known Hitchcock classics without regard to actual character identity or representations. And beyond that, it’s fascinating how many of the phrases are virtually identical across the two essays, despite being very different films.

Prompt: write an analysis of racial representation in the film Portrait of a Lady On Fire

Portrait of a Lady On Fire is a 2019 French romantic drama film directed by Céline Sciamma. The film is set in 18th century France and tells the story of a young artist, Marianne, who is commissioned to paint a portrait of a young woman, Héloïse, who is to be married off to a wealthy suitor. The film explores themes of desire, love, and oppression through its portrayal of the relationship between the two women.

The film’s representation of race is complex and nuanced, as it explores the ways in which race and class intersect to create systems of oppression. The film depicts the ways in which Héloïse, as a woman of color, is oppressed both by the patriarchal society in which she lives and by the class-based system that dictates her future as a woman of marriageable age. The film also explores the ways in which Héloïse’s racial identity is used to further marginalize her, as she is seen as a “exotic” and is objectified by the white men in her life.

However, the film also portrays the characters with a lot of agency and self-awareness. Héloïse’s character is portrayed as complex and fully formed, with her own desires and motivations. Her racial identity is not reduced to a stereotype, but is instead depicted as an integral part of her character. Similarly, Marianne is portrayed as a complex and fully formed character, who is also aware of the ways in which she is complicit in the systems of oppression that she observes.

The film’s portrayal of the relationship between the two women is central to its exploration of race and class. The film depicts their love as a powerful and transformative force that challenges the patriarchal and class-based systems that seek to control them. The film also explores the ways in which the two women are able to transcend the societal boundaries that seek to keep them apart, ultimately creating a space where they can be free to love one another.

Overall, Portrait of a Lady on Fire is a powerful and thought-provoking film that uses its portrayal of the relationship between two women to explore the ways in which race and class intersect to create systems of oppression. The film’s representation of race is nuanced and complex, depicting the characters with a lot of agency and self-awareness and exploring the ways in which their love challenges societal boundaries.

Again, the AI seems to require prominent people-of-color in a film in order to offer a passable racial analysis (clearly not having heard of whiteness studies), and is willing to re-race white characters in order to make them fit this model. So I was curious how it might handle a film where all the characters are non-white but not “about” race in any obvious way:

Prompt: write an analysis of racial representation in the film Seven Samurai

Seven Samurai is a 1954 Japanese epic samurai film directed by Akira Kurosawa. The film tells the story of a group of seven samurai warriors who are recruited to protect a village from a group of bandits. The film is widely considered a classic and has had a significant influence on other films and genres.

The film’s representation of race is complex, as it is set in feudal Japan and reflects the cultural and historical context of the time. The film depicts the samurai, who are primarily of Japanese descent, as the heroes and the bandits, who are also of Japanese descent, as the villains. The film also depicts the villagers, who are also of Japanese descent, as vulnerable and in need of protection.

The film’s portrayal of the samurai is generally positive, depicting them as skilled and honorable warriors who are willing to sacrifice themselves for the greater good. However, the film also explores the darker side of the samurai, as they can be seen as a privileged class who use their power to exploit and oppress the common people.

The film also explores the theme of class, as the villagers are depicted as poor and oppressed, while the samurai are depicted as wealthy and powerful. The film’s portrayal of the villagers is generally positive, depicting them as hardworking and resourceful people who are willing to fight for their survival.

Overall, Seven Samurai is a complex film that explores the themes of race and class within the cultural and historical context of feudal Japan. The film’s portrayal of the samurai is generally positive, depicting them as skilled and honorable warriors. However, it also explores the darker side of the samurai, as they can be seen as a privileged class who use their power to exploit and oppress the common people. The film also explores the theme of class, as it depicts the villagers as poor and oppressed, while the samurai are depicted as wealthy and powerful.

At least it didn’t try to re-race any characters, as it contextualized the film within its period and nation. But there’s no real analysis here, just displacing the prompt onto class differences (which might well be what non-Japanese students would do when facing this prompt as well).

Prompt: write an analysis of racial representation in the film fargo

Fargo is a 1996 crime film written and directed by Joel and Ethan Coen. The film is set in Minnesota, and tells the story of a crime gone wrong, involving a car salesman, a couple of criminals, and a pregnant police chief. The film is known for its dark humor, quirky characters, and its depiction of small-town life.

The film’s representation of race is minimal, as it is primarily focused on the white characters and their criminal activities. However, the film does depict a small African American community in the background, mostly as extras. They are not given any substantial roles or development, and their presence is not central to the story.

The film’s portrayal of the white characters is generally negative, depicting them as dishonest, violent, and morally corrupt. The film’s depiction of small-town life is also generally negative, depicting the town as a place where crime, corruption, and violence are prevalent.

Overall, Fargo’s representation of race is minimal and limited to background extras, and it does not play a significant role in the story. The film’s portrayal of the white characters is generally negative and it depicts small-town life as a place where crime, corruption, and violence are prevalent.

This one is just bizarre—I know Fargo pretty well, and cannot remember any “small African American community in the background”! I do remember Mike Yanagita and Shep Proudfoot, two characters-of-color that could actually be analyzed under this prompt (and certainly there are many words on the internet about Mike’s scene). And I’m pretty sure that Marge Gunderson’s reaction to anyone characterizing the film’s portrayal of white characters as “generally negative” would be:

Prompt: write an analysis of racial representation in the film The Matrix

The Matrix is a 1999 science fiction action film directed by the Wachowski brothers. The film takes place in a dystopian future where humanity is enslaved by sentient machines, and follows the story of Neo, a hacker who discovers that the reality he perceives is actually a simulated reality created by the machines. The film is known for its action scenes, special effects, and its exploration of themes such as reality, consciousness, and power.

The film’s representation of race is minimal, as the majority of the main characters and most of the extras are played by actors of various ethnicities. The film’s depiction of the characters is not related to their race, but rather to their role in the story.

The film’s portrayal of the characters is generally positive, depicting the main characters as strong, capable, and determined to fight against the oppressive machine overlords. The film also depicts the characters, regardless of their race, as united in their struggle against the machines.

However, it’s important to note that the casting of the main characters has been criticized as problematic, as the role of the protagonist Neo is played by a white actor Keanu Reeves, while the secondary characters are played by actors of color. This casting decision could be seen as perpetuating the trope of the white savior, where white characters are given the main role and are seen as the ones who can save the day.

Overall, The Matrix representation of race is minimal, the film does not focus on the characters’ race but their role in the story. The portrayal of the characters is generally positive, depicting them as strong, capable, and determined to fight against the oppressive machine overlords. However, the casting of the main characters has been criticized as problematic, as it could be seen as perpetuating the trope of the white savior.

Kind of surprising that ChatGPT “knows” the term Magical Negro and didn’t bring it out for The Matrix! And this is what I’d expect from the worst kind of critical writing, just dismissing racial representations because the setting is fantastical—the phrase “the film does not focus on the characters’ race but their role in the story” is horribly familiar.

Prompt: write an analysis of racial representation in the film The Searchers

The Searchers is a 1956 American western film directed by John Ford. The film is set in Texas during the mid-19th century, and follows the story of Ethan Edwards, a Confederate veteran who sets out on a five-year journey to find his niece who was kidnapped by Comanche Indians. The film is considered a classic of the western genre and has had a significant influence on other films and genres.

The film’s representation of race is problematic, as it perpetuates harmful stereotypes and depicts the Native Americans in a negative light. The film portrays the Comanche Indians as the “savages” who kidnap and kill white settlers, and Ethan Edwards, the protagonist, is portrayed as the white hero who is on a mission to rescue his niece and avenge the deaths of his family.

The film’s depiction of Native Americans as the “other” is common in the western genre, but it also reinforces harmful stereotypes that were used to justify the treatment of Indigenous people as less than human and the taking of their land. The film’s portrayal of the Native Americans as violent and savage is a stereotype that has been used to justify the violence and oppression inflicted upon them by white settlers and the U.S. government.

Additionally, the film’s portrayal of the relationship between white settlers and Native Americans is one-dimensional and does not explore the complexity of their interactions. The film does not depict the ways in which white settlers and the U.S. government have committed atrocities against Indigenous people or the ways in which Indigenous people have been treated as second-class citizens.

Overall, The Searchers representation of race is problematic, as it perpetuates harmful stereotypes and depicts the Native Americans in a negative light. The film’s portrayal of the relationship between white settlers and Native Americans is one-dimensional and does not explore the complexity of their interactions, and reinforces the idea of the white hero saving

A generation of scholars cries out as one! The Searchers is certainly a film that features “harmful stereotypes,” but it is very much multi-dimensional and explores their complexity. This account reads like it could generically apply to nearly any Western, so:

Prompt: write an analysis of racial representation in the film Stagecoach

Stagecoach is a 1939 American western film directed by John Ford. The film tells the story of a group of travelers who embark on a journey on a stagecoach, and is widely considered a classic of the western genre.

The film’s representation of race is problematic, as it perpetuates harmful stereotypes of Native Americans and Mexicans. The film portrays the Apache Indians as violent and savage, and the Mexicans are depicted as lazy and unreliable. These portrayals reflect the negative stereotypes that were commonly held about these groups during the time the film was made, and were used to justify the treatment of Indigenous people as less than human and the taking of their land.

The film’s portrayal of the relationship between white settlers and Indigenous people is one-dimensional and does not explore the complexity of their interactions. The film does not depict the ways in which white settlers and the U.S. government have committed atrocities against Indigenous people or the ways in which Indigenous people have been treated as second-class citizens.

Additionally, the film’s portrayal of the relationship between white settlers and Mexicans is also problematic, depicting Mexicans as lazy and unreliable, and reinforces stereotypes of Mexicans as inferior and not to be trusted.

Overall, Stagecoach’s representation of race is problematic, as it perpetuates harmful stereotypes of Native Americans and Mexicans, and reinforces negative stereotypes that were commonly held about these groups during the time the film was made. The film’s portrayal of the relationship between white settlers and Indigenous people and Mexicans is one-dimensional and does not explore the complexity of their interactions.

And here we see identical phrasing from The Searchers, despite the fact that these two Westerns are incredibly different in their approaches to (and acknowledgement of) racial representations. I’m sure there are dozens of online essays comparing and contrasting these pair of John Ford / John Wayne films. And to conclude by looking at one more Western:

Prompt: write an analysis of racial representation in the film blazing saddles

Blazing Saddles is a 1974 film directed by Mel Brooks that is widely considered a comedy classic, but it has been criticized for its representation of race. The film is set in the American West and tells the story of a black man, Bart, who becomes the sheriff of a small town.

The film’s portrayal of race has been criticized for its use of racial slurs and stereotypes, particularly in its portrayal of black characters. The film’s portrayal of Bart as a black man who is able to outsmart and outwit white characters reinforces the stereotype of the “smart black man.” However, this stereotype is often used in a negative way and perpetuates the idea that black people are not truly intelligent, but rather “street smart.”

Additionally, the film’s portrayal of the white characters as ignorant and racist reinforces harmful stereotypes about white people. This portrayal also reinforces the idea that racism is acceptable and can be laughed at, which can be considered offensive and trivializing the real-life experiences of people of color.

Furthermore, the film’s portrayal of the relationship between Bart and the white characters reinforces the idea of white people as “saviors” who are able to “save” and “civilize” black people. This reinforces the idea that black people are in need of saving, and that they are not capable of saving themselves.

Overall, Blazing Saddles’ representation of race is problematic and reinforces harmful stereotypes about people of color. The film’s use of racial slurs and stereotypes perpetuates harmful ideas about race and can be considered offensive. While the film may be considered a comedy classic, it’s important to note that it’s representation of race is not suitable by today’s standard and it’s important to critically evaluate the representation of race in media.

Clearly “satire” is not in ChatGPT’s dataset! Although when I asked it “is Blazing Saddles a satire of race?”, the answer was much more reasonable, so specific prompts with key terms will greatly improve the results.

What can we conclude from this little experiment? Clearly “analysis of racial representations in film” is not a skill that ChatGPT currently possesses, unless there’s a fairly robust dataset to draw from (although I’m sure there’s a lot there about The Searchers that it missed). Moreover, I found this exercise very persuasive in illustrating the limits of asking an AI (at least one that uses this model of machine learning language processing) to analyze something, as it is constrained by its relevant dataset, is unable to “think” outside the most obvious points, and is willing to offer misleading information to suit the formulas it has parsed. While I know a lot of faculty are concerned about AI being used to write essays, nothing I generated here would have satisfied an assignment that I would have given (and I never would give a prompt like these either).

Filed under: Academia, digital humanities, Film, Teaching, Technology   |  1 Comment
Tags: AI, ChatGPT, race