Materiality
When we read literature and look at images on pages or screens, we are usually trying to extract information from words, symbols and figures. We don’t often think about the physical processes involved in this or even the material objects with which we are interacting: we seem to see through the paper pages and the illuminated computer screens, barely registering differences between media as we work out the messages being transmitted through them. Yet the material properties of different media have all sorts of subtle effects on our reading and viewing experiences. For example, the way that the cover of a book feels or looks can determine whether we pick it up and enjoy reading it, and the brightness and speed of electronic images can determine whether we stay on one site or click our way to another.
In the first phase of explorations, the Dundee project team approached the question of materiality in poetry, setting the following specific research questions:
- To what extent does the medium in which something is viewed (on screen vs. in a book) influence the process of perception; the way parts are related to the whole, in what order and time-span they are perceived?
- Does materiality affect interpretative responses, and if yes, in what ways? Does it affect the participants’ evaluative judgements? Does it externalise, stabilise, dictate and thus limit ambiguities, or does it indeed lead to ambiguity, by adding additional and complex layers of meaning and tools of expression?
- How does spatial and material information get translated into words? How is material information verbalised?
This phase of research consisted of four different experimental sessions, combining different methodologies. In this phase we attempted to target the question of how, when art works combine textual and visual elements, the modes of attention specific to reading text and viewing images interact and modify each other. To approach this question, four works were chosen which were seen to combine textual and visual elements in different ways.
In the eye-tracking session, each CR saw two works that combined visual and textual elements in distinct ways: this was to give us the opportunity to compare how their reading patterns and modes of attention changed across these different modes of combination.
There were 10 co-researchers involved, in two groups: group A did a slightly different process to group B.
1. Group A was taken to a laboratory, where they were fitted with a polhemus device, which measures the co-ordinates of the body. There were given pages in sequence, from a folio by Edwin Morgan called Nuspeak. They were asked to verbalise their thoughts as they looked at it, and at the end of the session they were given time to write up an interpretation of the piece. They were told to describe the piece, say how it works, say what they thought the author was aiming to do, say whether they found it interesting and why, and which parts they liked in particular.
2. Both Group A and Group B were brought into the eye-tracking lab for the second session. They were told they would see a sequence of pages and that they should take as much time as they wished to view them. Afterwards they were asked to write answers to the following questions: Do you like them? What, if anything, do you find interesting about them? Do you notice any patterns? How important are the text and images, relatively speaking? Do you think the work is more visual or more textual? Their tracking was reviewed by the researcher and any patterns or idiosyncrasies noted. A combination of videos and fixation maps gathered from their session were then shown to the CR, and they discussed how they had behaved during the reading and what they thought of the process/material.
3. Both Group A and Group B were brought into a video-recording studio for the third session. Each participant was fitted with a Polhemus machine, and positioned next to a table, on which was a black cloth.
Under the cloth was the materials they were to look at. When the polhemus was started, they took out the material and were given a few minutes to look at it before being asked to rate the work on a scale of 1 -10, as to how much they liked it. They were then asked to describe what they were looking at, to say what they looked at first, comment on whether they thought there was a meaning in the way a text appears in book/material and whether they thought there was a meaning in the visual substance of the book/material. Finally they were asked whether they thought the text and the visual material mutually enhanced each other. This was then repeated with another three sets of material. The first two sets, were the materials they had seen on screen in the second session, the last two sets were completely new material.
4. Group A was shown Morgan’s Nuspeak for a second time, and asked to comment on how their perception of it had changed (or not) as a result of the experimental process. Group B, who had not seen the work before, was shown the same piece and asked to rate it according to how much they liked it. They were then given 30 minutes to write an interpretation of it. This was the longest that any group would spend with any work in the phase, and we arranged it in this way with the idea that this length of time would encourage CRs to consider the work in depth.
What did we find?
- Works that used text in an imagistic rather than linguistic fashion were judged to be more visual than textual. However, irrespective of layout or surround, if the text could be read in a habitual pattern of left to right and top to bottom, the work was judged to be textual. This demonstrates how deeply engrained our reading habits are and how they impact on aesthetic experience and judgement.
- Works that had been judged as predominantly visual were often discussed in terms of challenges and puzzles: seven of the ten CRs commented that they felt there was something to ‘get’ (as in a joke) or to work out, although they felt unable to say what that was. In contrast, work that had been judged as textual was evidently felt to be easier to access and comprehend. This raises further questions about the assumptions that underpin how we respond, or try to respond, to different modes of presentation. What is it about these ‘visual’ works that makes the viewers/readers feel ‘challenged’?
- There was no significant difference in the average liking scores of these works; those that were challenging were not liked any more or less than those that were considered more accessible. This raises the question of how liking is related to reward, in the sense of the positive feeling generated by mastering complex work or ideas, and what the relationship between taste and education is.
- The CRs rated works in which they felt the text and image mutually enhanced one another more highly than works which they felt were ‘fragmented’ or disjunctive. For one CR, who gave both works 10 out of 10, the idea of how text and image worked together was pivotal to her discussion of each work and something that she explored in detail. Three CR gave disjointedness or disruption as reasons for responding negatively to the works. These results might open out questions about harmony and likability: is perceived harmony a stronger factor for liking than reward? Can people actively like disjunction and disharmony?