PhD Student Takes a Data-driven Look at Art

When graduate student Sarah Reiff Conell enrolled in her first Pitt digital humanities course, it didn’t take her long to become hooked on using digital methods to solve historical problems.

A history of art and architecture PhD student in the Kenneth P. Dietrich School of Arts and Sciences, she set out to map patterns of geographic locations of cults that worshipped Christ’s holy blood in Medieval Europe. She first used an Excel spreadsheet, then moved on to Google Fusion Tables to visualize the points on the map and potential clusters. As everything came into focus, she had a “clarifying moment” as patterns emerged that had literally been obscured by layers of paper maps and colorful push pins.

They uncovered a noticeable gap in one region with no relics of that type. “That prompts new research questions,” said Conell. “What caused this gap? Were the relics destroyed in the early modern period and erased from the historical record?”

In another project, Conell and fellow Pitt graduate student Clarisse Fava-Piz researched information in an accounting book from the 1600s that listed the sculptors working for King Louis XIV of France. Using Excel, they first tracked the names of the sculptors, how much they were paid, if and how they collaborated and other topics. Then they used RAWGraphs to visualize the information.

“Our visualizations offered a fascinating viewpoint on artistic production that was difficult to envision when entries were scattered across hundreds of pages,” she said.

While these projects might seem obscure to non-experts, the methods and analysis Conell has honed recently made their own history as part of a first-ever project with the National Gallery of Art.

She and other data scientists and art historians were invited to analyze and visualize part of its massive permanent collection using data-driven methods.

The resulting two-day Datathon, as it was called, allowed the teams to present their findings, which are designed to help the gallery better understand its art, the breadth and scope of its collections and how they are exhibited.

Conell worked with Golan Levin and Lingdong Huang from Carnegie Mellon University’s STUDIO for Creative Inquiry, along with CMU Digital Humanities Developer Matt Lincoln. Their project used a software called Inception V3 Neural Network, which was trained to look for similar features in photos. The software organized thousands of paintings from the gallery according to visual similarity. Then, recognizable image types were clustered together—portraits, nudes, landscapes, still lifes and so on—to paint a picture of the collection. Using this method shakes up the way that artworks are organized, allowing art historians to see them with fresh eyes.

“This is just a visual comparison of the collections, which can only go so far,” noted Conell, a native of Seguin, Texas, who also received her MA in art history at Pitt in 2017. “But it points us to opportunities for new research and to ask new questions.”

Another grouping that caught the team’s eye in the analysis was the juxtaposition of Freydal, The Book of Jousts and Tournament of Emperor Maximilian I, next to comics by American cartoonist Robert Crumb.

“Why are these images shown near each other?” asked Conell. “Is it the dynamic movement? The use of color? Maybe it’s pointing to something like whimsy. It begs for more research.”

Conell says she’s encouraged the gallery opened its collections to data-driven work, because there has been critique over the years that its collection is not as diverse as the nation itself.

“I hope this is part of a larger interest toward transparencies in museums and opening up the dialogue beyond the people working within the institution,” she said.

She says there is a lot to be gained from computational methods, but only if subject area specialists are meaningfully engaged in the research process—either as individual researchers or collaborators within a team.

“Computers can’t give us answers,” she said. “They can offer new ways to approach longstanding questions, and some of the best results point us back to the archive.”