Usability Testing: Art Museum Web Sites

Download 172.09 Kb.
Size172.09 Kb.
  1   2   3

Usability Testing: Art Museum Web Sites

Woo, Jeongwon

A research project for the course

EDC 385G: Designs and Strategies for New Media

Spring 2006

Table of Contents


The Websites………………………………………………………………3
Data collection…………………………………………………………….8
Related Links……………………………………………….………….…38
Appendices (from A to G)………………………………………………..39

Generally, museums exist to display extensive collections such as works of art, scientific artifacts and historic objects. People can appreciate and experience valuable exhibits by visiting various museums. Especially, thanks to the advancement of World Wide Web, we also can appreciate exhibits in a virtual museum at any time without visiting the museum actually. Thus, most museums nowadays employ both actual museums and virtual museums. So we can take full advantage of museum as vast source for education and a means to enrich the culture.
Art museum is the exemplar of classical museums and the most familiar type of museums.

This research paper is to test the usability of art museum web sites. Most popular and famous three art museum web sites were selected and tested by two users. Tasks and procedures for usability testing were developed refer to Nielson (2000)’s Methodology report and Rubin (1994)’s usability handbook. Participants were required to clear both specific and open-ended tasks after answering pre-test questionnaire. Also, there was a short interview and survey about the websites. The recommendations for each site were made based on results from usability testing. This study will provide useful information to improve virtual museum or museum web sites. In addition, it will give some ideas for the people who plan to develop online museums.

The Websites
The three websites studied in this paper were selected based on the following criteria:

- Popularity and reputation of actual museums

- Abundance of art collections

- Art museums in the U.S.

- Famous to the general public

- Have virtual museums(able to see the collections through the website)

1) The Metropolitan Museum of Art
The Metropolitan Museum of Art is the biggest art museum in the U.S. and located at New York City. It has almost over 3,000,000 collections including Egyptian art, Greek art, Medieval and European paintings, American paintings, ancient artifacts and so on. It holds over 30 exhibits a year and 5,500,000 visitors from all over the world. It surely works as the center of art museum in the U.S. The purpose of museum is to cultivate the quality of life through arts.

2) The Museum of Modern Art
The Museum of Modern Art (so-called MOMA) is also located at New York City. As the name explains, it was established in 1929 to exhibit paintings, sculptures and prints as well as photographs, architecture and commercial designs which were made after 1880’s. MOMA has over 20,000 collections and be famous for collection of Picasso, Rousseau and Matisse. MOMA intends to enlighten visual art through modern art.

3) Museum of Fine Arts, Boston
Museum of Fine Arts at Boston is the next comprehensive art museum after Metropolitan Museum of Art. MFA has exhibits divided by 7 themes: Egyptian Art, Ancient Art, Asian Paintings, European Paintings, Prints, Dyeing and Weaving and American Art. Especially, it is famous for Asian collections like quality arts of Indian, China, Korea and Japan. It also has subsidiary art school (School of the Museum of Fine Arts) to educate artist and provide various certificate and continuing education programs.

The Participants: Two participants were selected to perform the usability test of the three art museum web sites. Their interests and background information was collected through pre-test questionnaire (see the Appendix B). They were selected because they ordinarily showed interest about works of art and artists. The information is as follows:
Lee, S:

- 38 years old, single, male

- Background: Ph. D. student in Curriculum & Instruction Department of University of Texas at Austin

- Computer experience: He usually uses the Internet at home and spends approximately 4 hours a day. He uses Internet for checking e-mail, researching, shopping, entertaining, information and blogging. He has managed his blog for almost 2 years and regularly updated it.

- Interest about art: His level of interest about fine art is very high. Even though he had no chance to visit museums in U.S., he enjoys visiting museums. When he was in Korea, he often visited Korean National Museum of Contemporary Art. He also visits online art museums frequently like Rijksmuseum in Amsterdam, Museum of Modern Art, and The National Gallery in London. He likes Netherlands painters and their paintings of 17C. He utilizes these pictures with his essays in his blog. He knows well about his area of interest.
Kim, S:

- 28 years old, single, female

- Background: Ph. D. student in Neuroscience of University of Texas at Austin

- Computer experience: She uses the Internet both at work and home. She spends almost 3 hours a day to check e-mail, research, entertaining and get information. She hardly shops by online.

- Interest about art: She has medium level of interest about fine art. But she enjoys visiting museums and has some knowledge about art and artists. The most impressive museum she ever visited was the Guggenheim Museum in New York City because of the beautiful structure of it. Also, she actually visited Museum of Modern Art (MoMA) twice. She hardly visits art museum web sites. She likes modern artists such as Monet, Gogh, Kandinsky and Klimt.
The Tasks: The tasks were developed based on Nielsen’s Methodology (Nielsen et al., 2000). And also, after considering the general use of art museum web sites, specific tasks were designed. For example, one of the common tasks for three sites was to find a specific painting among their collections. The tasks are divided into three parts: TaskⅠ(general impression), TaskⅡ(specific tasks), and Task Ⅲ(open-ended tasks).

The tasks given to each user were the following (To see more detailed tasks, see the Appendices).

TaskⅠ- General impression: This task was to get the general first impression of the web sites. The questions are described in a similar way as in Nielsen’s methodology. Minor adjustments were made to the questions to reflect information that users would typically want to find out from an art museum website.

Task I

a) What do you think this website is about?

b) Who do you think is its target audience?

c) If it is the first visit of this web site, what would you say about that?

d) At first glance, do you think this site could provide you with any information of your interest?

Task - Specific tasks: Participants are required to perform specific tasks at each web sites. The tasks are the most common functions for the art museum web sites. Some changes are made according to each websites but the basic requirements for the tasks are similar.

Task II

Metropolitan Museum of Art

Museum of Modern art

Boston Museum of Fine Art

• Find the painting ‘Boating’ of Édouard Manet from its collections and get the information about when the painting was drawn. Zoom the image.

• Find the painting ‘Les Demoiselles d'Avignon’ of Pablo Picasso from its collections and get the information about when the painting was drawn.

• Find the painting ‘Postman Joseph Roulin’ of Vincent Van Gogh from its collections and get the information about when the painting was drawn. Then, listen to the audio file explaining the painting. Zoom the image.

• Appreciate the exhibition ‘Samuel Palmer: Vision and Landscape’ paintings with audio guide.

• Send an E-card to a friend with the painting you’ve found in previous task.

• Send an E-card to a friend with the painting you’ve found in previous task.

• Find the educational material: ‘How Van Gogh made his mark’

• Appreciate the exhibition ‘Edvard Munch: The Modern Life of the Soul’ paintings with audio tour.

• Find out how many exhibitions are held now.

• Find the information about Metropolitan museum’s admission fee, directions to go there, open hours written in Korean.

• Find the information about MOMA’s admission fee, directions to go there, open hours.

• Find the information about Boston Museum’s admission fee, directions to go there, open hours.

Task - Open-ended tasks: This type of task is to find out what is going on at the websites and get the insights we hardly see (Neilson et al. 2000). In this task, users are allowed to explore the web sites freely based on their own interests.

Task III

  1. Buy some fine art product(e.g. posters, panels, books, jewelry…) you want at the Museum’s online store

  2. Explore and find more about your favorite artist or art

Data Collection
Participant Background Questionnaire: After getting signature of consent form from the participants, each participant was given with a pre-test questionnaire. The questionnaire is divided into three parts: demographics, computer experience and interest about art. The first part of the questionnaire is about age, occupation, and the filed of study now. Second part is about usage of the Internet such as Internet access place, time of usage per day, and the purpose of using the Internet. The last part is to find out the participant’s usual interest about fine art. It has questions about interest level about fine art, museum visit experience, and online art museum experience (See the Appendix B).
Task Construction: The three tasks are made by researcher. The first task is to know the first impression about the art museum website. In this task, participants had 2-3 minutes to explore the sites and then answered the questions: what do you think this website is about? , who do you think its target audience?

Task Ⅱ is composed by specific tasks and open-ended tasks for each site. Then, there was a simple survey about the easiness during performing the tasks. Specific tasks are a) find a specific painting from the museum’s collection using its search function and zoom the image if possible, b) send an E-card to a friend with the painting found in previous task (Not for the Metropolitan Museum), c) Appreciate special exhibition held in the Museum now with audio guide (Not for the Boston Museum of Art), and d) Find the information about admission fee, directions to go there and open hours. The paintings are different for each museum and some tasks are replaced by similar one because one museum doesn’t provide some functions which other two museums provide.

Task Ⅲ is open-ended tasks. It is to buy fine art products such as posters, panels, books, jewelry and so on. Then, participants can explore freely according to their favorite artists or areas of interest.

Short survey is about the easiness of use which is related to perform the tasks. For example, the data about the easiness of searching a specific painting and quality of the images seen on the screen were gathered. The data was collected by 7 point Likert scale (See Appendix C, D, E, and F).

Art Museum Web site Usability Questionnaire: When all the tasks are performed, participants are required to complete the overall usability questionnaire. The questionnaire has 27 questions divided by 4 sections: overall impression, information design, interaction design, and interface design. The data was also estimated by 7 point Likert scale (See Appendix G).
Usability testing execution: During the process of task execution, researcher sat beside the participant and took notes of their behavior and actions when participants attempted to complete tasks. Their reactions during performing the tasks were observed and recorded by the researcher. If the participants had problems about the task, researcher gave some hint and recorded the problem. Moreover, their comments and recommendations about websites were documented. The average usability testing hour was about one and half hour per a participant.
Testing Facilities: The usability tests were conducted at the researcher’s home with personal desktop computer. Thus, the testing environment was not influenced by distracter factors like other people or noises. The web browser used for the test was Internet Explorer 6.0 operated by Windows xp. The setting of computer was Pentium 4 processor and the Internet speed was 100M bps. The monitor was 19’’ flat LCD monitor and resolution was 1280x1024.

Findings from the task execution of two participants are discussed below. The results are organized by each art museum web site and the three tasks.
1) The Metropolitan Museum of Art (

Task Ⅰ- The first impression of the Met museum website was that it seems very formal and traditional. It seems to have many information and lots of collections of art. However, one participant said that it is too informative or educative and thus boring. Both participants reported that the target audience of this site is the adult and general public. The index page is too overwhelming by small texts. The collection and exhibition are not clearly divided. And it may confuse the users. The web site seems to have the information we need but it is not neatly organized.

Task Ⅱ- The first task is to find ‘boating’ of Édourad Manet from the museum’s collection and get information about that painting. Both of the participants had no problem finding that picture using the search function. If the user input the title of painting or the artist name, the search engine shows all the items which has that words and organizes the result according to the categories: works of art, special exhibition, membership, miscellaneous, and met store.

Zooming the image was satisfying for both users. Because Met Museum provides advanced zooming function, it is easy to look closely the whole painting. Met Museum’s image quality got the highest score in the questionnaire.

The second task is to appreciate special exhibition ‘Samuel Palmer: Visions and Landscape’ with audio guide. Both participants succeeded doing this task but they were confused by the audio guide function between ‘download the audio file’ and ‘stream the full audio program’. Because the menus for exhibition are basically just text only menus and links, users are easy to be confused. Moreover, the website does not provide the paintings of special exhibition and audio explanation in combined way. Compared to MoMA’s online exhibition, it looks like the graphic and audio do not match.

The third task is to find the special material made by the Met Museum: How Van Gogh made his mark. As we can see form its first page, the Met Museum concerns about educational programs for children and students. Thus, the museum prepared various flash movie clips about some artist or certain kind of art. The materials are under the ‘Explore and Learn’ menu. But one of the participants didn’t find that information and needed help. However, both participants showed satisfaction about making that kind of educational material.

The final task is to find the visitor’s information such as admission fee, directions, and open hours. It was easy for both users because that ‘visitor information’ was one of the main menus. Especially, the Met museum provides 10 different languages on visitor’s information page. The participants showed satisfaction with its consideration of foreign users. And the information was very detailed for actual visitors.

Task Ⅲ- In the open-ended tasks, shopping was the main task. The male participant bought a book and the female participant bought a magnetic after looking around the online store. The female participant showed interests about shopping function of museum web site but she said that the ‘Met Store’ is not attractive enough compared to other two museum online stores. They provide shopping status bar for the customers.

When exploring the website, the male participant said that the timeline and collection matching was very impressive. He told that it’s a good idea to provide a separate submenu ‘Timeline of Art history’. Also, he reported that it was very informative and useful both for general public and students.

Although the Met museum expressed that it is famous for kid-friendly art museum. It was pointed out that it actually not that friendly. Because it only has the list of materials which already exist in the museum web site, the kid-museum does not have its own purpose and mission. Although one of the participants said the Met Museum is too educative, it’s not clear that the special features were made for kids or for adults.

The Met Museum’s permanent collection is organized by its own categories. But the search function is too simple considering its vast amount of collections.

And in the survey about the easiness of usage, the average point for easiness of search function was 4.5 which was same with MOMA but lower than Boston Museum. About the quality of images, the point was 5.5 which was top with the Boston Museum. The overall easiness to appreciate museum collection/exhibitions was 5.5. And the easiness of shopping was 6.5 and easiness of navigation was 5. Compared to other two museums, the results say that Met museum has advantage in quality of images, easiness of appreciation, and the easiness of shopping.

2) Modern Museum of Art (
Task Ⅰ- The first impression of MoMA was that it was neat and simple. However, one of the participants said that the first page of MoMA has too much information which is basically texts with links. Perceived target audience was general public and adults. One participant said that the menus are not intuitive and obtrusive. The menus were designed like text although they are actually buttons which changes color when mouse rolls over.

Task Ⅱ- The first task of finding the painting ‘Les Demoiselles d’Avignon’ of Pablo Picasso was successful for both participants. However, the participants had problems with searching. If one enters ‘Avignon’ in the search box at the top, it shows many results related to the word ‘Avignon’ and it also organized search result into categories similar to Met Museum: The collection, exhibitions, press, online store and other. The main problem is that the first searching result was not the actual painting but the conservation of the painting ‘Les Demoiselles d’Avignon’. Only by using the searching function in the ‘collection’ menu, participants were able to find that painting. One participant was trying to find that painting from ‘paintings and sculpture’ menu, but the web site shows all the thumbnails of pictures, it was impossible to find specific painting in that. Even though it provides collection highlight at each areas, users said that it was a little difficult.

There are two versions of search function, but users do not feel that both of them are easy to use.

As you can see above, the wrong search result is the link to the conservation of ‘Les Demoiselles d’Avignon’ not the painting itself. Users can get the right result only from searching ‘browse and search the online collection’ menu.

In addition, MoMA does not provide enlarge or zoom function, so it was impossible to look the painting closely.

The second task was to send an E-card to a friend with the painting ‘Les Demoiselles d’Avignon’ of Pablo Picasso. Both participants said that it was a good function but the buttons were confusing. Because combining the word ‘preview & send’ into one button, participants hesitated click that button. One participant pointed out that generally the ‘preview’ and ‘send’ buttons are separated.

The third task was to appreciate exhibition ‘Edvard Munch: The Modern Life of the Soul’ with the audio tour. The reaction about online exhibition with voice was positive for both participants. If one clicks ‘full program’, it shows a pop-up window which has the exhibition paintings and related audio guide. However, they expressed confusion among ‘full program’, ‘Flash 7 or higher’, and ‘download audio clips’. The reason is that they are too closely located in the same page, and they are same in color, font, and size. And one participant said that it was inappropriate to have main menus and supplementary menu such as downloading specific programs at the same location.

The final task was to find the information about admission fee, directions and open hours of MoMA. It provides this visitor information in 5 different languages.

Task Ⅲ- At the shopping task, one participant confused between online store and MoMA stores. After finding the right page, both participants had no special problems with shopping. However, both users complained that they can not get out of the store directly and go back to MoMA main page. Instead of making link to MoMA homepage at the left top, they provide just small button ‘’ at the left bottom. And it opens a new window.

Exploring the website, one participant noticed that MoMA provides various educational programs. Unlike Met Kid Museum, it has some materials made by Flash which children can learn about modern art/artists with fun. Also, it has many adult programs, international programs, and teacher programs. Participants expressed affirmative reaction about these menus but they reported that overall navigation is too complex.

MoMA’s collection page is not very different from other museum’s collection page. But it provides viewing options such as single object, browse, list and text only. But users doubt that function is necessary.

The average point for easiness of search function was 4.5 which was same with Met Museum but lower than Boston Museum. About the quality of images, the point was the lowest 2.5 because it does not provide zooming function. The overall easiness of appreciating museum collection/exhibitions was 4.5 and it was the lowest point. They responded the search function is uncomfortable. And the easiness of shopping was 3.5 and easiness of navigation was also 3.5. It was hard to move between museum website and online store site. And the buttons and menus are not clear. Compared to other two museums, MoMA showed the lowest scores in all aspects in the easiness of using.

3) Museum of Fine Art at Boston (
Task Ⅰ- The first impression about Boston Fine Art Museum was that it was attractive and neat. One participant said that the main page looks simple and menus are clearly categorized. He said the accessibility to the information what he needs was the highest among the three museum sites (only 6 menus). The other participant also expressed positive reactions about the overall impression. Although the index page does not seem to have too much information, the menus are very natural and intuitive.

Task Ⅱ- The first task was to find ‘Postman Joseph Roulin’ of Vincent Van Gogh from its collections and then listen to the audio file explaining that painting. One participant failed once because she confused between ‘site search’ and ‘collection search’, but she completed the task. One participant said that he was very satisfied with their categories of collections and menus.

About the audio guide, one participant didn’t find it easily because the sound icon was at the bottom of the whole page. Also, the ‘enlarge function’ of the whole painting is restricted to 800x800 pixels. Instead, it provides zoomed version of some partials of the painting.

Same as the MoMA website, Boston Museum also provides E-card function. Both participants had no problem performing this task. Yet, they said that send button was not explicit and too small. Also, they can not preview the final version of E-card before sending it.

Third task was to find out how many exhibitions are held now in Boston Museum. This task was different from the other two museum sites because Boston Museum does not provide online exhibition with audio guide. It just presents the introduction about exhibition. The participants had no problem to find that information.

The final task was to find Boston Museum’s admission fee, directions, and open hours. Like other museum web site, Boston Museum provides this information by main menu. So participants had no problem completing this task. It also provides this information in 7 different languages.

Task Ⅲ- In this open-ended task, participants bought book and magnetic each. Both of them had no problem in shopping these items. However, it found out that once the customer checks out, it is hard to come back to museum website because there are no buttons or links.

And during exploring the Boston Museum website, it was found that they also provide special features about fine art history or specific themes. However, unlike other two museum site, Boston Museum provides rather static information basically made by html files. They has ‘online tour’ menu, but it was just the collection of pictures.

The average point for easiness of search function was 7 point which was the highest and perfect score. About the quality of images, the point was 5.5 and it was the same with Met Museum. The overall easiness of appreciating museum collection/exhibitions was 5 point. And the easiness of shopping was 5.5 and easiness of navigation was also 7 point. Compared to other two museums, Boston Museum showed the most satisfactory results in all aspects of the easiness of using.

In this part, the usability of each site will be discussed under the perspective of information, interface, and interface design. Data was collected from the “Art Museum Web site Usability Questionnaire” (See Appendix G) which was implemented after performing the tasks.
The Questionnaire collected data according to overall impression, information design, interaction design, and interface design. And 7 point Likert scale was used. The results are as follows:

Met Museum


Boston Museum

Overall Impression




Information Design




Interaction Design




Interface Design








Download 172.09 Kb.

Share with your friends:
  1   2   3

The database is protected by copyright © 2022
send message

    Main page