“Lifelogging” and neurophysiological computing: Will we forget how to forget?
One of the most famous examples of reminiscence includes a
madeleine dipped in tea, which lead to almost 3,000 pages of recollection by
the narrator in the beginning of Marcel Proust's novel In Search of Lost Time,
and we have all experienced these sensory triggers to a particular memory. Remembering
the past helps us to re-examine our lives, make choices, and share personal
accomplishments. We often use external devices to help us remember
events big and small, and with advances in technology, we often record and make
plans using a variety of digital devices such as iPhones, Microsoft’s Outlook,
and even smart watches. We have the capability to store a lifetime of data with
these advanced technologies, and with the advent of Facebook, Twitter, “selfies”,
and blogs it has become routine for many people to document their lives on a
daily basis in a digital form, a practice that has been referred to as
“lifelogging.” The outcome of documenting activities digitally are human
digital memories (HDM), which have been defined as “a combination
of many types of media, audio, video, images, and many texts of textual content [1].
The concept of recording and then later having the ability to review certain documents was first proposed by Dr. Vannevar Bush in 1945 when he described the “Memex” (a combination of “memory” and index”) in an issue of Atlantic Monthly [2]. As described in the article, a Memex was “a device in which an individual stores all his books, records, and communications, and which is mechanized so that it may be consulted with exceeding speed and flexibility. It is an enlarged intimate supplement to his memory.”The device would look like a desk where documents were either recorded via microfilm or photography.
Since that time, many
similar devices have been developed, but a revolutionary advance was seen with
Microsoft’s SenseCam a wearable camera with a wide-angle lens and multiple
sensors, including an infrared sensor to detect the presence of other people.
The camera takes a photo every 30 seconds, resulting in up to 2,500 photos a
day and is capable of storing 30,000 images in total. Photos can then uploaded
to a computer and viewed later using a Microsoft application [3]. SenseCam was developed as a memory aid and there has been over 50 research institutions
that have used the device in a variety of studies involving memory and behavior [4]. Notably the SenseCam has shown promising results in studies where it
was used as memory aid for a child with anterograde amnesia [5] and with adult patients that were
suffering from amnesia [3]. Aside from the medical
purpose that a camera such as SenseCam
could potentially serve, “lifelogging” has become more socially acceptable as
we live in a digital age where Facebook posts and Twitter feeds are consumed
constantly, and “selfies”
are a regular occurrence at most events.
However, our memories and our experiences are made up of more
than static images. Our memories are composed of sensory information, such as
temperature and smells, and especially emotions and physiological signals. The
next step of HDM would involve going beyond just digital images, and instead
would include physiological information that has been captured with sensors.
This type of information could allow for a more vivid recall and potentially
could remind us how we felt at any point in time [4]. Wearable systems that
incorporate sensors which collect data for future review have been the focus of
many researchers. One example of this type of system is the “Physiological Sensor Suite (PSS),” which
collects electrocardiogram (ECG), electromyogram (EMG), electrooculogram (EOG)
and through-hair electroencephalogram (EEG) that is then sent wirelessly to a
data sensor [6]. Although the sensors of this
suite did not require a gel to be applied on the skin first, the most
effortless way to use sensors to record our daily activities would be smart
fabrics, such as “smart shirts” that combine textiles and wireless sensors
to monitor heart rate, angle of inclination, body temperature, and location [7]. Shirts of this type were originally
designed for a hospital setting as a noninvasive method to monitor patients,
but in the age of personal computing, technologies like this could potentially
be used outside of a hospital.
Of course, these types of sensors that record physiological data such as heart rate or body temperature would be essentially meaningless alone; it would be difficult to reconstruct any memory, even a simple memory, based on physiological data since it is so ambiguous. However, if physiological data were supplemented with more data, such as a photo of the user, the location where the data was recorded, and the temperature, these clues together may help to trigger certain memories or parts of memories. One example of a system that works to combine data from a variety of sources is the AffectAura, an emotional prosthetic where data is collected from devices such as a microphone, Microsoft’s Kinect, and a webcam to predict emotional states such as engagement, valence, and arousal. Six participants were recorded over 4 days, and based on the data collected, users were able to reconstruct stories about their days [8]. There are still multiple challenges associated with creating accurate systems that could reveal emotional aspects of a memory, especially when systems and devices are created that go beyond only capturing photos. However, sensors will only become smaller and smaller in the coming years, and as a society we have a great interest in recording events for cultural reasons, so it is not unreasonable that one point, most people will participate in the act of “lifelogging” and the creation of HDM.
It is important to note though that physiological data or even photos can only act as triggers to a memory, as there is no method to actually capture an exact “memory” or a “thought.” Additionally, not only would an HDM need to incorporate data from a variety of sensors and then correctly corroborate and translate this data into meaningful information (a significant challenge), but an entire lifetime of memories would need be recorded for an accurate reflection and then a database that is searchable would also be required.
If however we could accurately record and then disseminate data that could compose a memory, does this documentation act as crutch? Reminiscing and sharing personal stories with families and friends is a basic human experience that acts as way to connect with others. If instead of memorializing a lost relative through stories or laughing with friends over a childhood experience, we could just push “play” on a device, how would that change us? We already live in a society where any question can be answered with a quick Google search on a phone requiring no discussion between two people, but how would our interactions with others change if we could just “Google” how a past experience played out or made us feel? (MIT professor Dr. Sherry Turkle has been studying how technology impacts people for over 15 years and has written numerous articles and books on the topics, and given this recent interesting TED talk on the subject). Devices that use information from HDM are meant to help us with the reminiscence process, but what if these devices are actually making us lose that ability, or at the very least, fundamentally altering the memory process?
Additionally, just as remembering is central to our existence, so is forgetting. Just because at some point in the future we may be able to document a person’s entire lifetime with a wealth of data, including physiological data, should we? Just this year, researchers from the University of Basel discovered the musashi protein, a protein that appears to inhibit molecules that stabilize synaptic connections. These connections are important for the development of memories, and based on this discovery, it appears that forgetting is an active biological process. The biological processes behind remembering and forgetting appear to work together, and forgetting is not just a passive process [9]; we most likely forget for a reason, even if more research is necessary to discover why.
References
1. Kelly L. The Information Retrieval Challenge of Human Digital Memories. Proceedings of the 1st BCS IRSG Conference on Future Directions in Information Access [Internet]. Swinton, UK, UK: British Computer Society; 2007 [cited 2014 Aug 19]. p. 17–17. Available from: http://dl.acm.org/citation.cfm?id=2227895.2227913
2.Bush V. As We May Think. The Atlantic [Internet]. 1945 Jul [cited 2014 Aug 19]; Available from: http://www.theatlantic.com/magazine/archive/1945/07/as-we-may-think/303881/
3. Hodges S, Williams L, Berry E, Izadi S, Srinivasan J, Butler A, Smyth G, Kapur N, Wood E. SenseCam: A Retrospective Memory Aid. UbiComp 2006 4206: 177-193.
4. Fairclough, S.H., and Gilleade, K. (2014). Capturing Human Digital Memories for Assisting Memory Recall In Advances in Physiological Computing, S.H. Fairclough, and K. Gilleade, eds.(Springer London), pp. 211-234.
Want to cite this post?
Strong, K. (2014). “Lifelogging” and neurophysiological computing: Will we forget how to forget? The Neuroethics Blog. Retrieved on , from http://www.theneuroethicsblog.com/2014/08/lifelogging-and-neurophysiological.html
The concept of recording and then later having the ability to review certain documents was first proposed by Dr. Vannevar Bush in 1945 when he described the “Memex” (a combination of “memory” and index”) in an issue of Atlantic Monthly [2]. As described in the article, a Memex was “a device in which an individual stores all his books, records, and communications, and which is mechanized so that it may be consulted with exceeding speed and flexibility. It is an enlarged intimate supplement to his memory.”The device would look like a desk where documents were either recorded via microfilm or photography.
 |
| From u-tx.net |
 |
| The SenseCam. From microsoft.com |
 |
| An example of a "smart shirt". From howstuffworks.com |
Of course, these types of sensors that record physiological data such as heart rate or body temperature would be essentially meaningless alone; it would be difficult to reconstruct any memory, even a simple memory, based on physiological data since it is so ambiguous. However, if physiological data were supplemented with more data, such as a photo of the user, the location where the data was recorded, and the temperature, these clues together may help to trigger certain memories or parts of memories. One example of a system that works to combine data from a variety of sources is the AffectAura, an emotional prosthetic where data is collected from devices such as a microphone, Microsoft’s Kinect, and a webcam to predict emotional states such as engagement, valence, and arousal. Six participants were recorded over 4 days, and based on the data collected, users were able to reconstruct stories about their days [8]. There are still multiple challenges associated with creating accurate systems that could reveal emotional aspects of a memory, especially when systems and devices are created that go beyond only capturing photos. However, sensors will only become smaller and smaller in the coming years, and as a society we have a great interest in recording events for cultural reasons, so it is not unreasonable that one point, most people will participate in the act of “lifelogging” and the creation of HDM.
It is important to note though that physiological data or even photos can only act as triggers to a memory, as there is no method to actually capture an exact “memory” or a “thought.” Additionally, not only would an HDM need to incorporate data from a variety of sensors and then correctly corroborate and translate this data into meaningful information (a significant challenge), but an entire lifetime of memories would need be recorded for an accurate reflection and then a database that is searchable would also be required.
If however we could accurately record and then disseminate data that could compose a memory, does this documentation act as crutch? Reminiscing and sharing personal stories with families and friends is a basic human experience that acts as way to connect with others. If instead of memorializing a lost relative through stories or laughing with friends over a childhood experience, we could just push “play” on a device, how would that change us? We already live in a society where any question can be answered with a quick Google search on a phone requiring no discussion between two people, but how would our interactions with others change if we could just “Google” how a past experience played out or made us feel? (MIT professor Dr. Sherry Turkle has been studying how technology impacts people for over 15 years and has written numerous articles and books on the topics, and given this recent interesting TED talk on the subject). Devices that use information from HDM are meant to help us with the reminiscence process, but what if these devices are actually making us lose that ability, or at the very least, fundamentally altering the memory process?
Additionally, just as remembering is central to our existence, so is forgetting. Just because at some point in the future we may be able to document a person’s entire lifetime with a wealth of data, including physiological data, should we? Just this year, researchers from the University of Basel discovered the musashi protein, a protein that appears to inhibit molecules that stabilize synaptic connections. These connections are important for the development of memories, and based on this discovery, it appears that forgetting is an active biological process. The biological processes behind remembering and forgetting appear to work together, and forgetting is not just a passive process [9]; we most likely forget for a reason, even if more research is necessary to discover why.
References
1. Kelly L. The Information Retrieval Challenge of Human Digital Memories. Proceedings of the 1st BCS IRSG Conference on Future Directions in Information Access [Internet]. Swinton, UK, UK: British Computer Society; 2007 [cited 2014 Aug 19]. p. 17–17. Available from: http://dl.acm.org/citation.cfm?id=2227895.2227913
2.Bush V. As We May Think. The Atlantic [Internet]. 1945 Jul [cited 2014 Aug 19]; Available from: http://www.theatlantic.com/magazine/archive/1945/07/as-we-may-think/303881/
3. Hodges S, Williams L, Berry E, Izadi S, Srinivasan J, Butler A, Smyth G, Kapur N, Wood E. SenseCam: A Retrospective Memory Aid. UbiComp 2006 4206: 177-193.
4. Fairclough, S.H., and Gilleade, K. (2014). Capturing Human Digital Memories for Assisting Memory Recall In Advances in Physiological Computing, S.H. Fairclough, and K. Gilleade, eds.(Springer London), pp. 211-234.
5. Pauly-Takacs K,
Moulin CJA, Estlin EJ. SenseCam as a rehabilitation tool in a child with
anterograde amnesia. Mem Hove Engl.
2011, 19(7): 705–12.
6. Matthews R,
McDonald NJ, Hervieux P, Turner PJ, Steindorf MA. A wearable physiological
sensor suite for unobtrusive monitoring of physiological and cognitive state. Conf Proc IEEE Eng Med Biol Soc 2007, 2007:
5276–81.
7. López G, Custodio
V, Moreno JI. LOBIN: E-textile and wireless-sensor-network-based platform for
healthcare monitoring in future hospital environments. IEEE Trans Inf Technol Biomed Publ 2010, 14(6): 1446–58.
8. Mcduff D, Karlson
A, Kapoor A, Roseway A, Czerwinski M. AffectAura: An Intelligent System forEmotional Memory.
9. Hadziselimovic N,
Vukojevic V, Peter F, Milnik A, Fastenrath M, Fenyves BG, et al. Forgetting Is
Regulated via Musashi-Mediated Translational Control of the Arp2/3 Complex. Cell, 2014, 156(6): 1153–66.
Want to cite this post?
Strong, K. (2014). “Lifelogging” and neurophysiological computing: Will we forget how to forget? The Neuroethics Blog. Retrieved on , from http://www.theneuroethicsblog.com/2014/08/lifelogging-and-neurophysiological.html
