When we read literature and look at images on pages or screens, we are usually trying to extract information from words, symbols and figures. We don’t often think about the physical processes involved in this or even the material objects with which we are interacting: we seem to see through the paper pages and the illuminated computer screens, barely registering differences between media as we work out the messages being transmitted through them. Yet the material properties of different media have all sorts of subtle effects on our reading and viewing experiences. For example, the way that the cover of a book feels or looks can determine whether we pick it up and enjoy reading it, and the brightness and speed of electronic images can determine whether we stay on one site or click our way to another.

In the first phase of explorations, the Dundee project team approached the question of materiality in poetry, setting the following specific research questions:

  1. To what extent does the medium in which something is viewed (on screen vs. in a book) influence the process of perception; the way parts are related to the whole, in what order and time-span they are perceived?
  2. Does materiality affect interpretative responses, and if yes, in what ways? Does it affect the participants’ evaluative judgements? Does it externalise, stabilise, dictate and thus limit ambiguities, or does it indeed lead to ambiguity, by adding additional and complex layers of meaning and tools of expression?
  3. How does spatial and material information get translated into words? How is material information verbalised?

This phase of research consisted of four different experimental sessions, combining different methodologies. In this phase we attempted to target the question of how, when art works combine textual and visual elements, the modes of attention specific to reading text and viewing images interact and modify each other. To approach this question, four works were chosen which were seen to combine textual and visual elements in different ways.

Map showing where CRs fixated on the screen while reading 'Words of Wisdom' by Jim Carruth

In the eye-tracking session, each CR saw two works that combined visual and textual elements in distinct ways: this was to give us the opportunity to compare how their reading patterns and modes of attention changed across these different modes of combination.

There were 10 co-researchers involved, in two groups: group A did a slightly different process to group B.

1. Group A was taken to a laboratory, where they were fitted with a polhemus device, which measures the co-ordinates of the body. There were given pages in sequence, from a folio by Edwin Morgan called Nuspeak. They were asked to verbalise their thoughts as they looked at it, and at the end of the session they were given time to write up an interpretation of the piece. They were told to describe the piece, say how it works, say what they thought the author was aiming to do, say whether they found it interesting and why, and which parts they liked in particular.

2. Both Group A and Group B were brought into the eye-tracking lab for the second session. They were told they would see a sequence of pages and that they should take as much time as they wished to view them. Afterwards they were asked to write answers to the following questions: Do you like them? What, if anything, do you find interesting about them? Do you notice any patterns? How important are the text and images, relatively speaking? Do you think the work is more visual or more textual? Their tracking was reviewed by the researcher and any patterns or idiosyncrasies noted. A combination of videos and fixation maps gathered from their session were then shown to the CR, and they discussed how they had behaved during the reading and what they thought of the process/material.

3. Both Group A and Group B were brought into a video-recording studio for the third session. Each participant was fitted with a Polhemus machine, and positioned next to a table, on which was a black cloth.


Under the cloth was the materials they were to look at. When the polhemus was started, they took out the material and were given a few minutes to look at it before being asked to rate the work on a scale of 1 -10, as to how much they liked it. They were then asked to describe what they were looking at, to say what they looked at first, comment on whether they thought there was a meaning in the way a text appears in book/material and whether they thought there was a meaning in the visual substance of the book/material. Finally they were asked whether they thought the text and the visual material mutually enhanced each other. This was then repeated with another three sets of material. The first two sets, were the materials they had seen on screen in the second session, the last two sets were completely new material.

4. Group A was shown Morgan’s Nuspeak for a second time, and asked to comment on how their perception of it had changed (or not) as a result of the experimental process. Group B, who had not seen the work before, was shown the same piece and asked to rate it according to how much they liked it. They were then given 30 minutes to write an interpretation of it. This was the longest that any group would spend with any work in the phase, and we arranged it in this way with the idea that this length of time would encourage CRs to consider the work in depth.

What did we find?