We see the need for a system that creates an augmented social environment where devices enable one to capture information about a social gathering (e.g. pictures, audio recordings, written thoughts and expression). The information would be presented in a manner that enables users to access it as well as manipulate the content. This should allow people to recall aspects of the event and promote opportunities for sharing.

These ideas have been explored and implemented into our prototype system - the Memory Collage. The goal of this system is to provide novel means by which information in one's focus and periphery at a social gathering can be stored as a memory. It provides the viewer context through recorded sounds and written annotations. We want the users to have memories pertaining to what they have experienced directly, what has been experienced by the group, as well as have the users obtain an awareness of previously unknown activities.

Downloads

Overview

This project was a four month endeavor for a graduate course in Human Interface Technologies taught by Sidney Fels. We envisioned a system with two main functions:

• To augment a social environment using various technologies capable of extracting "memorable" information from a party event.
• To later present the captured information in a personalizable collage for further browsing and sharing.

With the four months, a prototype system was built and tested at the Memory Collage party. Testing consisted of observing user interaction with various devices in the social gathering environment, questionnaires on invasion of privacy and obtrusiveness of the devices, and several sessions observing user interaction with the Memory Collage application followed by interviews.

The findings of our study indicated that audio clips do indeed enhance memory recall of an event. Other findings were also prominent in our experience, in particular:

• Information capture devices used in the social environment can change existing methods of interaction. If this is undesired, the devices should not attempt to make or rely on changes in the goals and actions of the guests.
• Devices should be flexible to variations in the environment and to variations in how the devices themselves may be used.

We believe that a second iteration of our system should involve more intelligent devices, capable of understand the interactions between people and the event space, objects symbolic to the party, and other guests at the party. Possibilities are discussed in the future work section below, but prior to that, the two system parts shall be summarized.

Information Capture

Information formats chosen to create a holistic memory of an event captured were: images, sound, discernable events, and writings. There were three primary methods by which our project augmented an event and captured information in those formats. A digital camera was modified so that it automatically records audio annotations from before and after a picture is taken. In doing so we explore both the usefulness of the pairing of photographs with audio, and examine the effectiveness of using recorded audio for framing the photographs' context. Microphones were placed around in the environment to record ambient and localized sounds. Touch sensors were also used to record the execution of "events" which could be defined by the users of the system (ex. something funny happened, someone said something really interesting). Papers and markers around the environment were used to capture guest writings. A tablet interface to a PC was also available.

The Memory Collage

After the various channels of event data have been collected, the collage generator processed the data into a succinctly viewable format. The photographs taken and the written artefacts were organically scattered in the collage. Users were able to interact with the collage by manipulating the photos, moving them to fit an arrangement of their choosing, as well as changing the size of each photo using the mouse's scroll wheel. The images were thumb nailed and arranged in a manner approximately related to the temporal proximity in which they were taken. The audio playback component of the collage fall into two categories: general ambient sounds, and annotation sounds from the photos. When a specific photo was moused-over, audio icons is available to play back the appropriate sound clip that was recorded at the same time index during the party associated with the photo. Implementation of the Memory Collage Generator was done using Java 2d and Java Sound written as a stand-alone application. Facilities for performing image display and sound playback are readily available with Java compared to using other potential development platforms such as OpenGL or DirectX. Scripting methods with TCL were also considered, though the weak support for sound playback with TCL lead us back to the first choice for Java as the primary development environment.

Future Work

From the findings discussed in the project overview, we see interesting research opportunities in the Memory Collage project:

• Exploring nonobstrusive methods of sensing the party environment such as augmenting socially symbolic objects or using a location detection system.
• Researching the ambient detection of interesting events in the social environment using intelligent systems.
• Exploring the development of an ontology for social conversations - how might they be understood and processed automatically.
• Comparing the use of a collage metaphor for photo browsing and sharing applications with other like applications.

Publications

Contact Information