Evaluation of Learning Technologies in Higher Education

article submission for


Journal of

International Forum of Educational Technologies & Society

And

IEEE Learning technologies Task Force


ISSN 1436-4522








Multimedia evaluation

by semiotics approach







Patrick BENAZET

Doctorant

Sémiotique et communication
Université de Perpignan
Sous la direction du Pr. Robert MARTY

patrick.benazet@libertysurf.fr


 

Abstract

Since the multimedia resources were introduced on the educational market, the experts endeavoured to analyze quality of it. When we examine the various existing methods, we are brought to raise that the criteria taken into account in the evaluation stick primarily to the ergonomic, technical aspects and with the simplicity of implementation of the educational software. The cognitive aspects are seldom considered. It however seems essential to have tools which make it possible to measure the capacity that has a system of multimedia training to produce knowledge beyond its simple aesthetic aspect. Basing us on the semiotics of Charles Sanders Peirce, we project here making of a tool for semiotic assistance to the evaluation of educational multimedia.


Table of contents


      1 - Semiotics multimedia definition
      2       - Interactives situations in educational multimedia
      3       - Volition according Peirce’s theory
      4 -       Triadic analyze of interactives situations
      5 -       Cognitive impacts of the interactivity
      6 -       How existant evaluating methodologies consider cognitives process
      7 -       Using Peirce’s semiotics to elaborate an evaluation method

       


1 - Semiotics multimedia definition


The framework of our study supposes that we give, initially, a definition to educational multi-media in order to delimiting contours of it. This phase allows us, for the continuation of our study, to exactly determine the field on which carries the evaluation. We will integrate into the educational multimedia term, any resource or any device using multimedia technologies at ends of training. For that we must first of all determine the definition of the multimedia word itself. We will consider that a resource is a multimedia one when it calls simultaneously upon different sensory registers and that it generates an interactivity between the learner and the artefact, made up in the majority of the cases of a data-processing device. The interactivity is thus fundamental and constitutes the specificity which distinguishes the multimedia from the audio-visual. Thus a no-interactive audio-visual resource, even if it is numerical and presented on a CD-Rom or in line on the Internet, like a video clip for example, cannot really be regarded as multimedia resource within the meaning of our study. The plurality of the sensory registers used thus does not make multimedia a specific device on the cognitive level, not more than audio-visual or the cinema, even the opera which were the subject of studies by syncretic approaches.

The fact that the learner influences directly its course of training and thus the processes of knowledge acquisition makes interactivity a completely particular situation that the evaluation must take into account specifically concerning the cognitive processes of the multimedia resources within the framework of the multimediatized trainings. Consequently, we can consider the multimedia for its cognitive capacity through the setting in scene of the situations of interactivity. In that, we approach the multimedia resources within the framework of a semiotic system which we define by supporting us on the definitions that Charles Sanders Peirce gave of the signi. A multimedia resource is comparable with a sign, complex, general and pragmatic. We will say that a multimedia sign is a syncretic sign due to the plurality of the sensorial registers witch compose it.

To locate it in the scientific theory, we point out that its structure is established on the basis of a triadic system which puts in scene three instances by the set of successive determinations. Any sign perceived refers to its object, thus the sign is determined by its object but the sign refers also to the interpretant so that the interpretant is determined by the object through the sign. This succession of determinations constitutes a triadic relational structure on which we will be based in the continuation of our study.

 

 

2 - Interactives situations in educational multimedia

 

The characteristic of the educational multimedia resources thus comes from the interactivity. We describe the situation of interactivity likely a situation of communication. The difference between the interactivity and a situation of communication comes owing to the fact that the user don’t communicate with a human but with a technological device, an artefact. However just like a classic educational situation of communication, the situation of interactivity is contextualizedii.

The communication and information sciences, while being based on work of the school of Palo Altoiii, consider seven contexts constituting the general context of the situation of communication. Those contexts, taken in the real communication, are the spacial context in which have place the communication, the physical sensory context which interfere in the communication in put concerned the different sensory register, the temporal context which locate the moment where the communication have place, the context of position of the actors of the communication, the relational social context, the cultural context of reference made up of standard known by the actors of the communication, and the context of identity which is constituted of the intentions of the actorsiv.

Within a multimedia framework, the context is essentially fixed by the software. I.e. by the program that the designer of the educational resource introduced into the computer. To constitute this context, we have a very reduced space which represents the screen of the computer and the loudspeakers. To govern the exchanges of the situation of interactivity we have the input devices, particularly the keyboard and the mouse. It is to say that the restriction of certain contexts, due to the technological constraints, restricts the context of the situation of interactivity comparatively to the context of communication. The situation of interactivity is thus in that a situation of communication contextualy restricted accompanied with an artefact.

The concept of context is thus to consider as in a situation of communication except that the elements of the various contexts which influence the communication and thus on the significance which emerges from it are reconstituted technologically and perceived through the numeric picture and sound produced by the computer. The intervention of the learner is as for it in an intentional register in volitive matter. We thus raise three instances in the situation of interactivity : the visual one, the sound one and the volitive one. The situation is based on dyadics relationships between the three instances. Thus the following cases are observed. We will note them by taking “V” for visual, “S” for sound and “v” for volitive and note the determination by “à”.


1) Visual leads to sound. This case is observed when the attention of the user is drawn by a visual sign identified like calling upon an audio resource. For example an image of loudspeaker or the logo of a mark exploiting a mode of processing of the sound like RealAudio. (VàS)


2) Volition is induced by the visual one each time someone meets a zone of the display which calls to click or to position the mouse. They are in particular the hypertext links generally materialized by underlined text, the sensitive images located by the form which takes the pointer of the mouse (a hand pointing a finger) or animated images containing a slogan such as " do not click here " or " click to know more" etc. (Vàv)


3) Volition acts on the visual one when the screen display is modified on the intervention of the user. For example when the user positions the mouse upon a zone changing this one aspect, according to the rollback technique largely used on the Web, or more traditionally during the striking of a text to the keyboard. (vàV)


4) The sound one is started by volition when the movement of the mouse causes the emission of an audio sign or when the user starts a sound while clicking on a zone of the screen. (vàS)


5) Volition is determined by the sound one whenever an audio message calls to click or to intervene with the keyboard or when an aural beep indicates an incorrect handling. This type of sound is generally indexical, a very short beep for example or symbolic like a car warning. (Sàv)


6) The visual one is determined by the sound one when a display followed upon an aural beep. This practice is met in particular in the exercises where the score obtained is displayed after the emission of a sound whose nature locates the result on a scale of values. (SàV)

 


3 - Volition according Peirce’s theory



“What is volitive relates to the will, volition is an act by which the will is determined with something”. Such is the lexical definition of volition. But Charles Sanders Peirce approached this process by introducing a distance between will and volition. This act which is volition is the result of a process with a psychological origin and whose last moment is of a muscular naturev. They are in fact a chain orders placed to the nervous cells by the brain, transmitted then to the muscular cells taking the form of discharge.

Thus, if my hand strongly tightens the cord which I hold, at the end of which is suspended a fragile package, it is because I judged that there was a risk of slip and consequently I tightened the cord to prevent it from slipping. The feeling of risk made me tighten the cord or more exactly made that my fingers contracted so that my hand tightens the cord. Does this act translate for as much a will of my share to tighten the cord whereas I know that I am likely to see my hand burned if I tighten too much? Volition is not, according to Peirce, always desired. He advances the concepts of active volition and passive volitionvi. Volition can be undergone.

Figure 1 : the contraction of the hand (volition) is determined by the fragility of the package (feeling), and the awareness of risk (cognition).


To take again the history of the cord, the act originates in a feeling, of risk. The fragile character of the package induces a risk because I bring it closer to the slipperiness of the cord. At this time, a cognition is produced. Peirce defines this circumstance by the triad feeling-volition-cognition. For him, volition is dual. There is in volition duality of the agent on the patient, of the effort on the resistance, the effort on inhibition, the action on the individual and the external objects : in our example, the fingers which tighten on the cord. If I tightened the cord without wanting to do it, it is that the contraction of it my fingers was controlled whereas I did not wish it. The desire implies a tendency to volition but it is noted that if the man does what he wants, it regularly sometimes happens to him to make things in spite of his desire.


Volition leads then so that a desire is dissatisfied even annoyed. For as much the desire is not always at the origin of volition. It is necessary to distinguish the intentional volition, which could be the different term to indicate the will of passive volition, involuntary. It is what attempts to do Peirce by quoting the marked difference between intentional cramp and the crispation due to a great fear. However, these two modes of volition must be generally classified together as modes of " double conscience ". A feeling of sorrow generates an active volition whereas a feeling of pleasure results rather in a passive volition. Thus, volition without desire occurs in reality and the difference between will and volition is well marked.


Figure 2 : the triadic relation between feeling, cognition and volition according to Peircevii is observed when using an educational multimedia resource.


So, with the difference of the lexical definition raised in the dictionaries, volition would be rather according to Peirce an act started by the feeling and conditioned by cognition. One then observes indeed the triadic phenomenon evoked by Peirce in which take part the three instances feeling-volition-cognition.

 

 

4 - Triadic analyze of interactives situations

 

The situation of interactivity breaks up into a succession of actions between the human and the computer. If we consider the situation at one unspecified time of the human-machine dialogue, the starting point can be indifferently the action of the human or the machine one. We will consider here only the action of the human, the machine one concerns the software. The dyads seen previously constitute occurrences of actions which can follow another one to give place to a situation of interactivity. By associating them, we obtain triplets which constitute a relational structure with three instances.



The whole of these combinations gives us six triplets which constitute the six possible situations of interactivity.


Situation 1: the visual one is determined by the sound one and the volitive one is determined by the visual one. ((SàV) à v)

It is the case where a sound determines a volitive intervention by the means of a visual display. An application of this configuration is raised in certain teaching software where a cry of animal is emitted whereas several images of animals are displayed. The image in connection with the cry must then be clicked. In this case, we are in the presence of a first sound sign (the cry of the animal) which determines the visual sign that the user must select (the image of the corresponding animal) to act above by means of the mouse, i.e. a volition.


Situation 2 : the visual one is determined by the volition and the sound one is determined by the visual one. ((vàV)àS)

In this case, the volitive intervention determines the sound level through visual display. It is a practice became current with the appearance of Dynamic HTML. The over flight of a zone by the mouse pointer causes a sound level whereas the zone flown over changes. We meet this configuration in the documents of the Web in particular on pages of choice or in many teaching software.


Situation 3: the volitive one is determined by the visual one and the sound one is determined by volitive ((Vàv)àS)

In this case, a sound level is determined by the visual display through the volitive intervention. I.e. that the visual display leads the user to act and at this time a sound level occurs. This case is frequently encountered in the various recreative or teaching software like on the Web. This situation is different from the preceding one because the visual display does not change.


Situation 4 : the volitive one is determined by the sound one and the visual one is determined by the volitive one. ((Sàv) àV)

In this case, a visual display is caused by a volitive intervention of the user based on a sound level. It is thus the sound level which conditions the volitive intervention which starts the visual display. Let us take the example of a software of musical education which presents several buttons to the screen carrying of the names of musical instruments. A sound of instrument is emitted, it calls the user to click on one of the buttons which starts the display of visual symbolizing an answer right or false. The visual display is thus conditioned here by the validity of the volitive intervention.


Situation 5 : the sound one is determined by the volitive one and the visual one is determined by the sound one. ((vàS) àV)

In this case, a visual display is determined by a volitive intervention through a sound level. This configuration can appear similar to the preceding one but this time it is the sound level which determines the visual display. This situation is met in many cases on the Web. It is very near from the situation 2, only the order of the determinations sound, visual is reversed. However the two modes of synchronization get different effects.


Situation 6 : the sound one is determined by the visual one and the volitive one is determined by the sound one. ((VàS) àv)

Here, the visual display determines a volitive intervention through a sound level. This situation is met in the games of memorizing such as " memory quizz ". A series of different visual elements are displayed successively causing each time a particular sound level. On the whole of the display, only one of the visual elements caused a sound correspondent with the display. The player then owes to click on the good element. This configuration approaches triad 1.


Figure 3 : the triadic structure of the situation of interactivity highlights successive determinations of sound, visual and volition.


The starting point of those relational structures is the perception of a sign and the arrival point is the action of the user on the machine. The action being the result, is started by means of a media and puts in scene a volition. There is thus, at a certain time of the situation of interactivity, union of two of the instances by the action of the third one that lead us to advance that this relational structure is triadic. It is not besides without resembling that which Peirce described by feeling-volition-cognition.



5 - Cognitive impacts of the interactivity


The situation of interactivity, such as we define it, is a situation of communication which places the human actor in front of a second actor who is the artefact. It is thus significant to regard this communication situation as a global system interfering two actors with different nature and included in a complex context determined by the software architecture on the one hand and the volitive attitude of the user, in our study the learner, on the other hand. It is to say that our problems of the multimedia resources evaluation is apprehended in a systemic way and that it relates to the contextualisation of the situation of interactivity.

This contextualisation includes the seven contexts that the communication and information sciences releasedviii, in the software, whole except for the behavioral part which concerns the volition and which constitutes the context of construction of the quality of the relation. This multimedia context, which retranscribes the contextual whole of the situation of interactivity, is established by the global sign perceived by the learner. This sign is complex and refers to various sensory registers.

We present it as a triadic multimodal sign as we analyzed previously. There, the situation of interactivity finds its base with cross semiotics and information and communication sciences. We approach the sign by his architecture such as the peircian semiotics allows us to do it, like an hypersign made up of several signs present at a given moment. By basing us on the grammar of the sign and more particularly on the lattice of the classes of signsix, we can peel this hypersign which contextualise the situation of interactivity to extract from it the contexts returned by the signs which constitute it.

By this method we can analyze the nature of the sign and deduce the constitution of the various communication contexts, named processes by some authors, which influence the production of the significance. We can thus determine which cognitive path the learner must borrow to reach the knowledge by the analysis of the classes of signs and thus to evaluate the cognitive impact of the situation of interactivity. It is in this, that semiotics brings cognitive dimension in our approach.




6 - How existant evaluating methodologies consider cognitives process


As of the appearance of the first educational multimedia ressources, in the Eighties, the scientifics were challenged on the quality of the products. We can observe that to evaluate the educational multimedia implies that at least two aspects are considered : the multimedia resource intrinsically and cognitive strategy adopted for the course of the training. Many steps of development of methods have been led and gave place to various tools. The significant number of undertaken work states clearly that the evaluation of multimedia is a constant and widespread concern. Broadly the multimedia evaluation methods elaborated up to now take the shape of a form or a grid which it is advisable to inform simultaneously with the test of the evaluated product. These grids take a paper form or sometimes have been the subject of a software development which makes it possible to produce statistical datax.

One observes in all the cases the taking into account of the ergonomic aspects on the one hand and didactic quality on the other hand. The analysis of these two registers is made according to a number of variable criteria of one method to the other and in all cases by an empirical approach. The references employed result from the bases describing the mechanisms of education or principles of design and are based on specific concepts with the trainings, generaly speaking. In all the cases a step of training is evaluated whose support have changed compared to a traditional training in the presence of a teacher. The specificity of educational multimedia seldom appears in the methods of analysis, in these rare cases, the simultaneous processing of the medias exploited by the computer becomes determining and the criteria of evaluation are enriched of them.

One then considers the device of formation either a textbook but really like a didactic unit integrating the teaching resource and the framing of learning using interactivity. If we tried to constitute a grid of evaluation which covers the whole of the fields treated by each existing method we could distinguish the following axes: objectives and contents of the formation, quality of the interactivity, attraction of the product, teaching strategy, data-processing reliability. These five axes including each one a certain number of criteria, are approached according to models resulting primarily from the approaches of Piaget and Gagné.

A great number of evaluation tools thus exist and remain despite everything, the business of specialists. They are interested primarily by the act of teaching, the cognitive impact is seldom measured and in those rare cases according to an empirical mode. It appears of this fact the need for being interested by the cognitive aspects by a theorized method which integrates the analysis of the conditions in which the significance is produced. For that, triadic semiotics constitutes a real scientific base.


7 - Using Peirce’s semiotics to elaborate an evaluation method


The situation of interactivity constitutes the context of knowledge production in the course of a teaching using a multimedia resource. It is thus this situation retranscribed by the multimedia components that it is advisable to analyze to evaluate its cognitive load. We consider the situation of interactivity according to the six triadic structures seen previously.

The semiotics of Charles Sanders Peirce teaches us that cognition occurs by the relational processxi which puts in scene the perceived sign, the object which it represents in the mind of the interpreter and the social habitus of this one, by the set of successive determinations and by considering the three modes to be, that are the firstness which is the domain of the quality, of the feeling, the secondness which represents the domain of the fact and the thirdness which represents the domain of the social law, of the concept.


The quality of the perceived sign is thus of primary importance in the production of significance, which Peirce calls the semiosis. To build our educational multimedia evaluation methodology, we are based consequently on the grammar of the sign, formalized by the lattice of the classes of signsxii which enables us to diagnose the structure of the sign. The lattice of the classes of signs gives us a diagrammatic vision of the combinative grammar of the elements of the categorical analysis of the sign. The argument located in top of the diagram contains all the classes of sign located below, by traversing the lattice from the top to the bottom and by analyzing the classes of signs of each level, we obtain the cognitive course which the sign imposes to arrive at the production of the significance. It is not besides necessary that the lattice is complete.





The cognitive power of the situation of interactivity depends on the entirety level of the sign structure. However, the multimedia sign is an hypersign made up of several signs in relation, constituting the syncretic context.



Figure 4 : the lattice of classes of signs gives us the grammar of the sign. All classes are encapsulated into the sign. The arrows signify that a class is contained into another. For example, a rhematic indexical legisign is contain into the rhematic symbol.


In addition the multimedia signxiii is not a real sign such as perceived in the nature, it is a replicated sign, which numerically reconstitutes an element of the field of reality. We are in the presence of a system which connects signs concerned with varied sensory registers, visual: text, fixed image, animated image, or sound : sound effects, music, various noises. It is thus by a systemic approach of this semiotic unit and in particular by the analysis of the relations which bind the various signs constitutive of the hypersign that we reconstitute the semiotic structure of the situation of interactivity and that we deduce the nature of the volition in reference to the double conscience exposed by Peircexiv and so the cognitive capacity of this one.


In substance, our step relates to an analysis of the lattice of classes of signs of each sign which constitutes the multimedia context, supplemented by an analysis of the relations between the various lattices. The study of the complementarity of the various lattices enables us to constitute the total lattice of the situation of interactivity. This approach enables us to deduce the entirety from the total sign through the 10 existing classes of signs and the borrowed cognitive course, described in the paths to the knowledge, which indicates the way to us in which knowledge is produced.

 


A case of study


We present here an extract of the evaluation of the educational multimedia software called ADIBOU published by the Company COKTEL which addresses to children from 3 to 7 years and which approaches calculation, the writing and the reading.

Initially we note that the application is structured in 5 parts: A clip of starting, a user guide, a user profile personalization module, environment of starting, and finally exercises. The interactivity really only begins with the third part because the two first are passive, the user does not have to intervene. It is noted however that the context of training is fixed right now as of the first part. One is accommodated by a small character (Adibou) introduced by a short cartoon illustrated by a music background. The small character makes a wink, it is equipped with coloured clothes and presents a very relaxed attitude, the bottom of the screen is blurred and represents a countryside a little fantastic. The music as for it is involving, one perceives there primarily an accordion and a guitar. The context of training is composed here by the complementarity of sound one and the visual one. It comes out from it a feeling of dynamism and relaxation. The learning child is from the start put in confidence.

Maintaining our self on the fourth part (environment of starting) which is really interactive. Let us recall however that the learning child had beforehand the occasion to personalize his user profile by carrying out a judicious mask to represent him and by specifying his name and his date of birth, all that in an environment requiring only graphic objects displacements.

The fourth part is thus the environment of starting. One discovers there a scene which is held in a very green field where runs a quiet river of a perfect blue. A house is on the right part, it has the shape of a chubby face capped with one cap to the bent visor. The sky is of a blue azure, a personified tree opens a big mouth in its trunk. The character (Adibou) enters in scene and presents himself, it speaks while alternating his gestures: he puts his hands in the pockets of his trousers then he rubs its belly. Its voice is piercing and very childish it invites the child " to learn while having fun at the paradise of the children ". In the decoration are a certain number of objects: a watering-can, a horn of jalopy, a traffic sign etc.

If after a time out the child did not click to select an activity, the character makes enter in scene his dog which is sad and cries because the child " did not build to him of niche ". If at the end of an additional time the child still haven’t click, then a nice monster arrives frightening which presses on the horn then sets out of the scene again.

This succession of calls to click constitutes as many signs which have all as an argument " to learn while having fun " but whose respective lattices vary. They also constitute a variation of context, which Bateson calls the " realignment ".

In synthesis of our evaluation, (we pass here the detail of analysis of lattice and of contextual component for some reason of length of drafting), we can say that the context successive envisaged in the situation of interactivity himself base successively on the feeling of confidence, of pleasure, then of emotion and finally of fear. It is to say that the major registers of the feeling are present. The learning child is able to test all the judicious possible sentimental forms to cause his volition. Its choice will thus be done at a time or at another, but it will be made in an unquestionable way. Its volition will be passive or active according to the nature of the feeling, the double conscience is thus treated. Concerning the value of the hypersign, it consists of signs complementary to sensory nature either visual, or sound, in all the cases both are complementary in the total lattice. The iconicity is very strong with a completely graphic environment, the symbolic system is very present as well in the sound one as in the visual one and the indexes which are sound or visual are positioned also perfectly. The conclusion of our evaluation shows that this situation of interactivity has a strong cognitive load. Consequently we have the assurance which this situation will lead the learner to a knowledge whatever its field of interpretant.

This methodology can be applied to any form of educational multimedia resource and to all the levels of training, of the youth at the university. It makes it possible to release a precise vision of the cognitive capacity of the software.

 


Patrick BENAZET

August 2001

 

References


BATESON,Gregory – « Vers une écologie de l’esprit » - Le Seuil - ISBN 2-02-025767-X

CROZAT,Stéphane, HU,Olivier, TRIGANO,Philippe – « EMPI (Evaluation du Multimédia Pédagogique Interactif) »

MARC,Edmond et PICARD,Dominique – « L’école de Palo Alto » - RETZ/HER – 10085494 – BSBM 80

MARTY,Claude et Robert– « 99 réponses sur la sémiotique »

MARTY, Robert, « 76 définitions du signe relevées dans les écrits de C.S. PEIRCE »

MUCCHIELLI, Alex – « Les sciences de l’information et de la communication »

MUCCHIELLI, Alex – « Théorie des processus de la communication »

PEIRCE, C.S. – « Collected papers »

 

i C S. PEIRCE gave many definitions of sign, Robert MARTY raised 76 produced by the american philosopher between 1865 and 1911 - Robert MARTY, “ 76 définitions du signe relevées dans les écrits de C.S. PEIRCE “
ii Gregory BATESON – « Vers une écologie de l’esprit »
iii Edmond MARC et Dominique PICARD – « L’école de Palo Alto » 
iv
Alex MUCCHIELLI – « Les sciences de l’information et de la communication »
v
C.S. PEIRCE – volition at CP 3.331
vi C.S. PEIRCE – intentional volition at CP 3.376
vii C.S. PEIRCE – the triad feeling-cognition-volition at CP 3.332
viii Alex MUCCHIELLI – Théorie des processus de la communication
ix Claude et Robert MARTY – 99 réponses sur la sémiotique
x Stéphane CROZAT, Olivier HU, Philippe TRIGANO - EMPI (Evaluation du Multimédia Pédagogique Interactif)
xi C.S. PEIRCE – semiosis at CP 3.484
xii Claude et Robert MARTY – 99 réponses sur la sémiotique
xiii cf. 1 semiotics multimedia definition
xiv C.S. PEIRCE – double conscience at CP 3.334