A.B.S.T.R.A.C.T.
ANALYSIS OF BEHAVIOR AND SITUATION FOR MENTAL REPRESENTATION ASSESSMENT AND COGNITIVE ACTIVITY MODELLING
Objective: An instrumented analysis of human activity
The ABSTRACT project is a research project in cognitive sciences and covers both the fields of cognitive ergonomics and computer science.
It is intended to address a nead for means to better understand a human activity from a computer recording of this activity.
Our objectives are objectives of ergonomics, in its general meaning of a "science of human activity". The goals are broad: facilitating an activity, predicting it, preventing undesired acts to happen, etc. To do so, we are developing a methodology and a tool that facilitates the "discovery of knowledge from traces of activity".
Theoretical framework
In human sciences: Exploratory Sequential Dated Analysis
From a human science point of view, we recognize ourselves in the research area that Penelope Sanderson and Carolanne Fischer have named "Exploratory Sequential Data Analysis" (ESDA) in their article of 1994: "Exploratory Sequential Data Analysis: Foundations".
This article defines the objectives of the ESDA as "Analyzing sequential data with a quest for their meaning in relation to some research or design questions". It is indeed an exploratory process which aims at "Looking at data to see what it seems to say" rather than a process of confirmation of assumption by statistical validation or invalidation.
In computer science: "Knowledge Discovery"
From the computer science and knowledge engineering point of view, we recognize ourselves in the field of "Knowledge Discovery" as Usama Fayyad presents it in his article of 1996: "From Data Mining to Knowledge Discovery in Database".
Beyond the problem of "pattern finding" and "machine learning" through algorithms of "dated mining", "Knowledge Discovery" questions the epistemological problem of the meaning of the discovered "patterns".
Figure 1: Steps that compose the KDD process - Fayyad 1996.
We insist on the need for "keeping the man in the loop" of the "knowlege discovery" cycle such as Fayyad presents it (figure 1). We propose an approach made up of successive levels of abstraction, continuously confronted with the interpretation of the ergonomist. The knowledge is not built inside the software but inside "the head" of the ergonomist in interaction with the software. Progressively, with this interaction, the knowledge is however capitalized in the software under the form of a step by step improvement of the inference and visualization rules.
Methodological principle: A process of abstraction
The raw data
We begin with an observation of the activity which can take multiple forms: directly by the ergonomist, video cameras or microphone recording, sensor recording of numerical data, software "logs", etc.
In our example, the data is collected with an instrumented vehicle during car driving experimentations. The data is obtained from cameras, sensors (speed, steering angle, pedal use, telemeters, GPS positioning, eyetracker), and from interviews with the driver.
In any case, the observation must produce a set of dated data, i.e. a set where each piece of data is attached to a moment of the activity identified by a "Time-Code".
This dated data constitutes our raw data. Its choice and its form is not neutral, it is based on our initial knowledge of the activity we are studying, and on our research goals.
The pre-processing
This step of pre-processing consists of low level processing such as sensor calibration, "noise" filtering, or elimination of the uninteresting variables.
The collected trace
The collected trace is the result of a "discretization" of the data. For instance, we extract the noticeable points of the analogical curves (Thresholds, local Minimums and Maximums, inflection points) (figure2); or the triggering events of areas of interest from the eye-tracker (figure 3); or we convert raw software "logs" into significant events.
Figure 2: Discretization of the analogical data.
Figure 3: Eye-tracker area of interest.
This discretization produces a succession of events to which are attached numerical or textual properties. We call this succession of events the "collected trace". It constitutes the starting point of the various symbolic traces that we are producing.
This collected trace must be validated by the ergonomist in order to ensure that the events it contains are meaningful. The adjustment of the various parameters used for producing these events is an important step. To carry out this validation, we have inplemented a software tool which enables the ergonomist to play it in Excel in synchronisation with the video. When the video is played, the trace is automatically scrolled in Excel (figure 4). Each line represents an event; column 1 contains its Time-Code in seconds, column 4 contains its type, the following columns contain its properties.
Figure 4: Checking the collected trace with Excel.
The video is not entirely encoded into symbols and remains exploited by the ergonomist in parallel with the symbolic traces.
The analyzed traces
The collected trace is then converted into an RDF graph to enable its enrichment by more abstract symbols (figure 5). The circles represent the events of the collected trace, the triangles and the squares represent the more abstract events, and the arrows represent the inference relations.
Figure 5: Infering abstract descriptors.
The inference rules are written as SPARQL queries. The ABSTRACT tool makes it possible to build these queries in a semi-graphic way, i.e. the ergonomist interactively produces skeletons of queries, and then he manually completes them.
Parallel to the creation of the queries, the ergonomist defines the symbols in an ontology. We provide him with the Protégé ontology editor to let him define the semantic and visual properties of the symbols he creates. This ontology is exploited by the SPARQL inference engine and by modules which graphically display the traces.
The User interface
Figure 6 shows a screenshot of the user interface for trace analysis.
Figure 6: ABSTRACT user interface.
This interface offers the following functionalities: (1,6) select a trace, (2) view general information about this trace, (3) view its content under different formats, (4) transform it by defining and applying rules of transformation, (5) define desired symbols in an ontology, (7) choose different visualisation modules, (8) go to a specific time code.
The visualisation module (10) shows the whole trace with the higher level symbols, the (9) shows a zoom into an interval of 10 seconds. The user can click a symbol to shows its properties (11). The user can drag the trace horizontaly or synchronize it with the video (12).
Example 1
Figure 7 shows the visualization of a typical "motorway lane change schema" as it is produced by ABSTRACT. The "Start-thinking" symbol corresponds to the moment when the driver declares that he starts considering changing lane in the self-confrontation interview. The "Button" is a signal from the experimenter recorded during the course for indexing every lane change, and the "Lane-crossing" is the moment where the left front wheel crosses the lane, manually encoded from the video. All other symbols are automatically inferred from the sensor data. The labels are not part of the Abstract visualisation, they were added in the figure.
Figure 7: Motorway lane change schema.
In this display, the "x" position of the events is given by their time-code. The ergonomist is free to define the shapes of symbols, their colors and their "y" position. This figure uses a display model where the lower level symbols are the dots at the bottom. The middle level symbols are placed around the middle horizontal axis. What concerns the left of the vehicle is placed above this axis and what concerns the right is placed below it. Triangles oriented to the right concern something "frontward" (e.g. "Look ahead"), triangles oriented to the left concern something "backward" (e.g. "Left mirror glance"). The lines are the relations of inference from lower level symbols to higher level symbols. The upper part of the display is used for the highest level symbols which are not concerned by this left/right rule. For instance the "Decision to change lane" symbol which could be infered from the coocurence of acceleration and left mirror gaze.
Exemple 2
Figure 8 shows the terrorist activity in the Republic of Ireland between 1970 and 2007. The data comes from the Global Terrorism Database (GTD).
Figure 8: Terrorist activity in the Republic of Ireland.
The visualisation on the top represents a zoom over a 100 day interval. The visualization on the bottom represents the 37 years.
In these visualizations, the main icon is associated with the field "WEAPON_TYPE". The three main weapon types are represented: "Firearms" (gun), "Explosive" (star) et "Incendiary" (flame). A second icon representing a corpse outline is appended when the "ATTACK" field is equal to "assassination".
The "y" position is associated with the field "PERPETRATOR". That is, the principal terrorist groups are represented each on a distinct line. Loyalist groups are represented above the central axis. Rebublican groups are represented below the central axis.
In this exemple, the ontology is hosted online under the form of a Semantic-Media-Wiki. The tip window of an event gives the links to the different classes defined in the ontology. The "GTD_ID" field gives a link to the event description in the Global Terrorism Database.
The user of ABSTRACT defines these visualizations in a XSL style-sheet that exploits the parameters set in the ontology.
Demonstration
A simpler version of ABSTRACT is available online for your free usage. It is called ABSTRACT-LITE. A full documentation is also provided.
Download
The full version of ABSTACT is available for download from the Liris SVN repository at this address. This repository is read only. if you want to contribute to the development, please contact the authors.
Publications
Supporting Activity Modeling from Activity Traces. Olivier L. Georgeon, Alain Mille, Thierry Bellet, Benoit Mathern, Frank E. Ritter (2012). Expert Systems, 29 (3), 261-275. doi: 10.1111/j.1468-0394.2011.00584.x.
Early-Stage Vision of Composite Scenes for Spatial Learning and Navigation. In the proceedings of the First Joint IEEE Conference on Development and Learning and on Epigenetic Robotics (ICDL-EPIROB 2011). Olivier L. Georgeon, James B. Marshall, Pierre-Yves R. Ronot. Frankfurt (24-27 August 2011).
A Comparative study of Exploratory Sequential Data Analysis tools. 2010. Sowmyalatha Srinivasmurthy. Technical report. The Pennsylvania State University.
Other publications on CiteUlike.
