This paper has been published on the Journal of New Music Research, Vol. 25 (1996), pp.212-230.

Formalization of Computer Music Interaction Through a Semiotic Approach

Fernando Iazzetta
Laboratório de Linguagens Sonoras
PUC-SP - Communication and Semiotics
R. Ministro Godoy, 969, sala 416-C
São Paulo - SP - 05015-901 - BRAZIL
e-mail: iazzetta@exatas.pucsp.br

Abstract: Musical activity involves, in its essence, interactive processes in all of its levels: composition, performance and listening. This paper is an attempt to understand human/computer interaction and the way it works in the field of computer music performance. We first present interaction as a communication process and introduce some concepts of the semiotic theory established by the American philosopher C. S. Peirce. Peirce's triadic categories are presented and used to classify interactive music systems. Then, we proceed to analyze different aspects of music interaction in relation to Peirce's semiotics.

1- INTRODUCTION


Musical activity involves, in its essence, interactive processes in all of its levels: composition, performance and listening. It shows different characteristics if the interactive agents include only musicians (as in an orchestral performance), only machines (as in a music box), or both (as in interactive computer music systems). Our interest is focused on musician/machine interaction, and in particular, in the case of those systems that make use of computer technology.

This paper aims at analyzing human/computer interaction and the way it works in the field of music activity. For this purpose, we first introduce interaction as a kind of communication process and then we formalize music interaction through a semiotic perspective.

2 - INTERACTION AS A COMMUNICATION PROCESS


Speeches, texts and works of art arise from a particular organization of signs in a specific context. They communicate or become meaningful when someone is able (even without being completely conscious of it) to perceive, at some level, this occurrence of organization. Although the idea of communication is an essential characteristic of living things, many disciplines, such as cybernetics, information theory, artificial intelligence, and cognitive sciences, have successfully applied this concept to contexts that include non-living agents, especially in the field of computers and other automatic machines.

Music interaction, too, can be seen as a communicative process. We understand communication as the message exchange between a sender and a receiver. As a result of this process, the message affects the receiver in a certain way by means of its signification. A message is a sign or a string of signs. After being formulated by the sender the message is encoded and transmitted through a channel and finally decoded by the receiver. Code and channel are supposed to be shared, totally or in part, by all the agents involved in a communication process. The entire process is enclosed by another component, the context, which plays an important role in the way the message is formulated and understood. Although this model may look static, the context forces the communication system to behave in a dynamic way (Figure 1).

One can clearly notice a direction in the communication process which goes from the sender's formulation of message to the receiver's understanding. This semiotic flow arises from the movement of cause (formulation) and effect (understanding) of the signs (message) that are being transmitted. Interaction is a particular case of communication where the semiosic process is not unidirectional but can virtually happen in and from the direction of each involved agent. That is, to each message that is received, a new response is elaborated by the receiver and sent back to the sender of the message. More importantly, this response will influence the sender's further actions. It establishes a process that goes beyond the duality cause/effect, and creates a continuous growing of signs. This question of directness is extremely important because it implies that the agents are constantly exchanging their roles of sender and receiver, and are, therefore equally affected by the whole communication process.


Figure 1: General diagram of communication.

The study of the communication process and its agents is part of the subject of semiotics, or general theory of signs. Although each of the components in the communication process (sender, receiver, message, code, channel, and context) is important to the semiotic investigation, its focal point remains on the action of signs, or semiosis, that makes possible this process.

The subject matter of semiotics, it is often credited, is the exchange of any messages whatsoever -- in a word, communication. To this must at once be added that semiotics is also focally concerned with the study of signification (Sebeok, 1991: 13) .

As a communication process involving such things as representation, signification, and interpretation, music interaction can be analyzed with the help of semiotic theories. Thus, a semiotic view of interaction in a musical context can help composers, performers, listeners, and soft/hardware designers in the development of interactive music systems.

3 - THE TRIADIC THOUGHT


Semiotics, as it is used here, is related to the "doctrine of signs" established by Charles Sanders Peirce, instead of the concept of semiology originated from Ferdinand Saussure's (Saussure, 1949) thought. While semiology takes linguistics as its foundation, Peirce's semiotics has its more general and abstract point of departure in philosophy. According to Peirce, semiotics is only another name for Logic (CP 2.227) .

Peirce's theory is based on triadic relations that can be related to any and all phenomena. Despite the complexity of relations that this triadic thought involves, for the purpose of this work, one can summarize these logical categories as follows :

The first of Peirce's categories is related to the "being of positive qualitative possibility" (CP 1.418). It comprises the qualities of a phenomenon, like redness or goodness. They are not the phenomenon itself, but once there is a phenomenon there is such a quality. They are units, mere possibility of being. Peirce calls this first category firstness.

These vague and potential qualities merge into one another (not without losing their particularities) to become a more general phenomenon, the actual facts. It is related to the second Piercean category and comprises our experience and all things that happen in time and space. In Peirce's terminology it is called secondness is and it does not represent a mere possibility, but existence.

Peirce's cosmology also encompasses the idea that things evolve from chance and freedom (firstness) to regularities (secondness). Regularity and existence tend to become habit, law. This process leads to what Peirce calls thirdness, the category of generality, process and thought. Any general phenomenon like a thought, a law, or a concept, is not a quality for a quality is "eternal, independent of time and of any realization" neither is it a fact because, since it is general, it refers to "all possible things, and not merely to those which happen to exist" (CP 1.420).

It is important to notice that firstness, secondness and thirdness are not discrete categories since they occur as a continuos flow, qualities generating facts, and facts generating habits or laws. The following table summarizes the triadic relations among the three Peircean categories:

Table 1: Relations among Firstness, Secondness and Thirdness.

4- DYNAMICS OF INTERACTION


Interaction is a reflexive process. The actions performed by each interactive agent do not only determine the system's responses, but are also influenced by those responses. Interaction works as a functional loop where each action acquires its meaning in function of other co-related actions. It implies a constant exercise of adaptation to contextual situations. Each involved agent must assimilate and accommodate itself to the environment. The way the agents react and adapt themselves to the context will determine the kind of interaction they will perform. Basically we can distinguish three general kinds of interaction:

1 - Competitive Interaction: The agents do not act as partners and may not share the same goal. Generally, in this situation, the success of one agent's action, implies the failure of another. This situation can be observed in different contexts like economics, evolution, and board games like chess.

2 - Cooperative Interaction: The success of one agent in reaching a goal implies the success of other agents. The cooperative agents share the same goals and act in collaboration over a period of time. Examples could be a teaching situation or the enactment of a theater play. Most of this paper is concerned with this kind of interaction.

3 - Symbiotic Interaction: The agents have different goals but they act in association and share the same context. The nature provides many examples of species that live in symbiotic interaction. For Example, some cellulose-eating animals, like the horse and the cow, have bacteria living in their stomachs that help in digesting their food. In exchange, the bacteria receive their food from them.

It is worth noting that an interactive system that clearly fits into one of the above categories in relation to its global behavior, can have its subsystems acting locally in a different way. For example, a big band performing jazz can be seen as a good example of cooperative interaction in a global level while the confrontation between two musicians in a solo-improvising section can be locally characterized as a competitive interaction.

From the view that interaction is a reflexive kind of communication process, it is possible to say that most of the communication processes involving multiple agents (be it living beings or machines) involve some level of interaction. Therefore, what should be the basic point in the study of interaction is not the delimitation of a boundary between interactive and non-interactive processes, but the analysis of interaction in terms of degree over a large and continuous range.

In one direction of this continuum, interaction tends to be developed in a steady manner which is extremely restricted by the system's configuration and exhibits low capacity of adaptation to context transformations. This kind of static interaction is typical of situations where the goals and actions to be performed are strictly defined and likely characteristic of machine-based systems. In the other direction, the interaction tends to be sensitive and adaptive to context fluctuations that may demand reasonably elaborated cognitive skills from the involved agents. It leads to a more dynamic interaction that is characteristic of situations which involve human or other natural agents and whose goals cannot be completely defined in advance.

Dynamic interaction does not occur only in human context, nor is static interaction restricted to machine systems. In fact, human interactive processes can become closer to machine interactive processes if the human agents have to work under extremely rigid ruled contexts (people making hamburgers in a McDonald's kitchen, for example). In the same way, machines can simulate human behavior, as in the case of systems where some artificial intelligence techniques are applied (for instance, a computer running a neural network based program).

Table 2: According to its characteristics interactive processes can tend to be dynamic or static.

The differences among systems that involve only humans, only machines, or that comprise the action of both, might be understood in terms of degree instead of opposition. Most of their characteristics are not exclusive but, rather, complementary. Thus, they may be seen as tendencies to one or another aspect in a continuum rather than opposite categories. In our specific case, we are particularly interested in human-computer interaction and the way these systems can be designed to exhibit the desirable characteristics that can be found in machine or human interactive systems. Table 2 summarizes some characteristics of dynamic and static interaction.

5 - HUMAN-COMPUTER INTERACTION


Interaction occurs in a communication context when multiple agents are able to perform or take part in one action over one object. For instance, during a dialogue two individuals (agents) express themselves by formulating their opinion (action) in the form of speech. It is important to notice that not only the agents are affected during this process but also the subject of discussion (object) undergoes transformation as the agents reflect upon it.

Generally speaking, a very simple interactive computer music system presupposes the existence of at least two agents: the human and the computer. To achieve a fully cooperative and dynamic interaction these agents may act in accordance to cause a change in the present or future states of a certain object (Figure 2).


Figure 2: Human-computer interactive system.

The agents may know about two aspects of the system:

1- State of the object: An interactive system must allow the agents to have access to information about the state of the object(s) they are acting upon. It is especially important to permit dialogue between the agents concerning which actions they will decide to take. This information is not only related to the present state of the object(s). The agents may also have a 'memory' of previous states and predict further states. In order to achieve a high degree of interaction, each action should be based on knowledge about previous states of the object and should influence the actions that may happen in the future (feedback).

Both the human and the computer agents have specific capabilities when working on a task; these capabilities may not be overstepped. Most of the time, the computer seeks for a specific type of information and will not perform operations in a desirable way when fed with unexpected data. Warnings and error messages are common tools the machine uses to prevent these situations. The agents' capabilities determine the kind of action they are able to perform as well as the kind of knowledge they must have access. For example, in a system where a computer adds a fifth to each note of a melody it receives as an input, the computer does not need to know much about the music that is going on. All it needs to know is when each note is played, its velocity, its pitch, and when it ends. On the other hand, if the computer provides a harmonic basis to this melody, it is fundamental that the computer has access to a wider knowledge about the object states. Since each chord in a harmonic context is strongly related to previous chords as well as to the possible chords that could come later, the computer must be able to access a memory of past events, characterize the context where these events have happened, and predict future states.

2- Behavior of the system: Agents in an interactive computer music system may also have knowledge of the global system behavior. They must coordinate their actions in a way that they can act as partners instead of unrelated agents. Thus, it is necessary to establish a "common ground" (Laurel, 1993) , a sign-space shared by all the involved agents during their interaction. This common ground includes a mutual knowledge (usually incomplete) as well as beliefs and suppositions about the components and the processes involved in the interaction. One can say that common ground in a interactive music system involves knowledge about:

1) the representational context of the interactive process;
2) the object(s) the agents are acting on;
3) the actions other agents can perform;
4) the actions each agent expects from the other agents; In general, this mutual knowledge cannot be seen as a finite set of pre-given information since interactive systems behave in a dynamic way due to the successive recursions and transformations which occur during interaction. For this reason, one can say that a common ground is not only necessary to establish a cooperative interaction but is also a product of the interaction.

Different agents - for instance, human and computer - have different knowledge about the system, and they also might use different signs and representational contexts to work on the same object, even if they share a common ground. In fact, each agent has only partial access to the system's information. In a musical interactive system, man and computer can be working on the same set of relations among pitches for example, but they will process this information through different representational contexts.

The internal operations executed by a computer are related to sign contexts which can exhibit a highly developed grammatical structure but any apparent connection to meaningful things. "A digital computer [...] operates only on the physical form of the symbols it computes; it has no access to their semantic value" (Varela, Thompson, & Rosch, 1991: 41) . A sequence of bits can be used to represent virtually any object. The semiotic connection between computational symbols and the objects these symbols stand for, is made through software and interfaces. Alternatively, the user operates with signs that point to their objects and relations among objects in a much more direct way. These operations include not only the grammatical level (structure and organization) but also higher semiotic levels that involve meaning, though, and contextualization (language) .

In a interactive music system, despite the difference between computer's symbolic language and performer's gestures, both computer and performer may refer to the same kind of sound object. To share information as well as actions over this object, it is necessary that there exists a kind of agent of mediation. Thus, they must have an efficient interface to mediate their interaction. The interface creates representations of the objects and context involved in the process. Moreover, the interface can be able to show object and context changes and thus it can lead to more elaborated representations about the behavior of the system as a whole or about some part of it. Figure 3 shows a more complete diagram of a human-computer interactive system.


Figure 3: Human-computer interactive system.

6 - INTERACTION AS SEMIOSIS


Interactive systems can be thought of as semiotic machines. They have a large capacity for sign generation, and signs tend to generate semiosis, the process of action of signs. According to Peirce, a sign generates another sign that generates another sign in a continuous chain. Here we should take a look at Peirce's sign definition:

A sign or representamen, is something which stands to somebody for something in some respect or capacity. It addresses somebody, that it creates in the mind of that person an equivalent sign, or perhaps a more developed sign. I call the interpretant of the first sign. The sign stands for something, its object. It stands for the object, not in every respect, but in reference to a sort of idea, which I have sometimes called the ground of the representamen (CP 2.228)

This triadic relation between sign (first), object (second) and interpretant (third) can be represented thus:


Figure 4: The triadic relations among sign, object and interpretant.

But the interpretant is itself a sign that also stands for an object (its previous sign or the relation between its previous sign and object) and generates another interpretant that can again become a new sign. Unless this process of creation stops for some reason, it may continue in two directions: it can evolve into new and even more complex signs or, when the fonts of novelty are used up, the interpretant tends to orbit around the same kind of signs.

In creation, and especially in artistic creation, the 'language management' to maintain the equilibrium between the generation of new ideas and the stabilization around an area of attraction is decisive. The emergence of a poem or a symphony appears from successive interactions in a sign space at different hierarchical levels. One way of viewing the work of a writer or a composer is that they create contexts where signs can grow. For example, a simple melodic fragment that is repeated insistently loses its interest after a few repetitions. It works as a rupture in the semiosic chain. To maintain its growth a sign needs to be put against other signs. Of course, one can consider cases where the exhaustive repetition serves as fount of novelty, as in the minimalist music. However, in these cases the interest lays much more in the processes that are going on under the repetitive structures or in the relations between the repeated material and the contextual transformations than in the recurrent elements by themselves. A simple, but splendid, example of this situation is drawn on a single phrase in Gertrude Stein's book, The World is Round (Stein, 1986) : "ROSE IS A ROSE IS A ROSE". Here, the author explores the potentiality of rhythm and sonority of the repeated words and, at the same time, draws attention to the multiplicity of meanings embodied in the recursive phrase.

In semiotic terms, one can say that the traditional composer's role in western music is to manage in actual pieces the development of this process of semiosis. But in this traditional sense, once the composer's work ends and the piece is finished, it does not mean that this particular piece loses its sign-generative capacity: as a complex system of signs a piece of music is always pointing up to new interpretants. It is only the composer who may lose the possibility of interfering in the process. After being created, an art-work gets a kind of semiotic autonomy. It is like a ship on the sea: if it is consistent and the environment is adequate, it navigates through the semiotic space; but if it is weak or the context is hostile, it cannot go much farther and, unless a big change takes place, it will probably sink.

Music, like any other sign or system of signs, can only survive by remaining open to the flow of signification. An interactive system can be seen as an auxiliary medium that enables the use of contextual information as growth material. Unlike traditional totally pre-composed music, interactive music can make use of the internal and external processes that are going on during the performance as part of the elements which react in the musical composition. Emphasizing the interactive role in music also represents a shift from the emphasis on compositional processes which has been imposed by electroacoustic music some forty years ago, to the performance .

7 - INTERACTIVE MUSIC SYSTEMS


Cooperative interaction has always been present in music and its analysis may be related both to traditional forms of music and to computer music. In the first case, the concept of interaction is closely associated to the choices the player or the singer can take during the performance. In this context, the role of interaction can range from very expressive, as in the case of improvisational music, to very slight, as in the music which the performer must follow the directions previously established by a score. In the case of computer music indeed, the concept of interaction acquires a new role that can be analyzed in the light of semiotics theories.

As Robert Rowe has said, "Interactive computer music systems are those whose behavior changes in response to musical input" (Rowe, 1993: 1) . To reach this kind of responsiveness it is necessary to provide the system with an efficient algorithm to respond

to a performer in a complex, not entirely predictable way, adding information to what a performer specifies and providing cues to the performer for further actions (Chadabe, 1989: 144) .

In this sense, the interactive system would be able to: - Interpret the performer's actions as partials controls for the music.
- Generate control for those aspects of the music not controlled by the performer.
- Direct the synthesizer in generating sounds (Idem). Music interaction operates in a continuum and is a question of degree. Thus, it is hard to define any strict boundary between interactive and non-interactive systems. One can only attempt to define the level of interaction in a specific system. As we have seen, it may tend to be dynamic when the performers can significantly interfere in the music context and the computer can deal with different kinds of input; or it may tend to be static when the performer's actions are highly pre-defined and the computer is not able to respond to many different kinds of situations.

Another way of viewing how interactive a system is may be achieved by thinking of a continuum that "could be evaluated by three variables: frequency (how often one could interact); range (how many choices are available); significance (how much the choices really affected matters)" (Laurel, 1993: 20) . The level of interaction based on these variables does not necessarily show a linear behavior where the increase of the values, would proportionally result in an increase in the system's level of interaction. Rather, the system's behavior is more likely a nonlinear one. In fact there are many ways to arrange these variables to reach a considerable level of interaction. It is not necessary to have a system where frequency, range and significance show high values to get highly interactive responses. As Brenda Laurel points out, to achieve a significant level of interaction one must also "feel [one]self to be participating in the ongoing action of representation" (Laurel, 1993: 21) . To interfere in the process may not be enough to make the performer a participating agent. Interaction is highly context-dependent and if the system cannot respond to significant aspects of each particular music state, it will be hard to feel that an interactive process is going on.

As an example, we can think of an interactive system which allows a musician to play some pitches on a MIDI keyboard to which the computer responds with some sort of "accompaniment". Although the computer's output can be very dependent on the pitches produced by the performer, the system's response can sound completely random and non-interactive, depending on the established rules (which can, of course, be random, as well). Let's call this program "InAtt" (Interaction-Attempt). To each MIDI-note InAtt receives from the keyboard, it generates a new note based on some pre-established transforming function. Let's say that InAtt also takes the MIDI-velocity values of each note to calculate a proportionally inverted output delay. In this case, soft pitches will lead to longer delay responses than louder pitches. Figure 5 shows a possible implementation of this idea in a MAX environment. The MAX object named 'table A' implements the transforming function and the patch 'p invert' determines the response delay.


Figure 5: Detail of an implementation of InAtt in a MAX environment.

The evaluation of InAtt's interaction level can lead to two quite opposite values depending on the music context. In a first case, a composition can demand the performer to play some fast sequences of pitches chosen from a diatonic scale in a very soft way. Because InAtt produces long delays to soft sounds, the computer will respond to the performer's input with a considerable delay making it difficult to perceive any connection between the performer's stimulus and the computer's response. Although the result is completely dependent on both performer's and computer's actions, the system's response cannot be considered as a satisfactory interactive process. The output response will sound completely random and the relation between performer's and computer's actions will be very weak. To frame an interactive situation, the computer should be able to respond immediately to the performer's actions in a way that those actions influence the future development of the music process.

If we consider a second case where the performer has to play loud pitches taking a long delay between each note, the result would be completely different. Each note played by the performer will be immediately followed by a note played by the computer. The performer's input would work as a kind of appogiatura to the computer notes, and the relation between the performer's actions and the computer's response would be quite clear. If compared to the previous example, in this context InAtt shows a much more interactive behavior (Table 3).



Table 3: The level of interaction in a system can vary according to the performed actions and context.

8 - A TRIADIC VIEW


Interactive computer music systems involve three different semiotic levels: musical agents, structure, and organization. These three levels would correspond to the general logic categories, firstness, secondness, and thirdness, established by C. S. Peirce (see section 3).

The first level of an interactive system comprises the individual elements which compose the system's universe. It is much like the concept of agents introduced by Marvin Minsky in his book The Society of Mind (1986) . Elements or agents hold specific characteristics and qualities and are able to perform defined actions. This level is related to the elements that compose the system, but not to the relations that exist among those elements. Therefore, it does not give any information about how the system works. It only reflects the system's possibilities. The second level, or structure emerges from the relation among those elements or agents and encompasses the interaction between parts of the system and its interfaces. It reflects the system's actuality. Organization, as the third level, "signifies those relations that must be present in order for something to exist" (Maturana & Varela, 1987: 42) . The term organization is taken here in a broad sense of general laws which guide the creation, use, and understanding of signs. It is related to the way agents manipulate the signs that take part in the interaction and involves the rules that operate on the system in different hierarchical levels. Operating on the system's elements and its relations, this level implies the notion of grammar and language and, thus, reflects the system's potentialities.

As an example, we can define these three levels for the previous program InAtt. The structure is composed by two agents, a performer and a computer/software agent. They both perform distinct roles in the interactive process. Besides being able to play different notes on a MIDI-keyboard the performer can also make decisions about these notes concerning pitch, velocity, and tempo. He also listens to and evaluates the computer's responses. The computer uses the performer's action to produce its own response. It processes the performer's input through a pre-defined program and, thus, generates its own output.

The system has an interface to connect the agents and to transform their actions into sound objects. This interface is composed by a MIDI-keyboard and MIDI connections. This interface seeks for two different kinds of actions -- the performer's key strokes on the keyboard and digital signals coming from the computer -- and it is able to correlate both of them in the same representational basis.

The structural level involves the relations between the agents and their actions. This level is designed by a composer that uses the system's agents to manage the occurrence of real actions. In this example, the structure is quite simple. The performer's input is used as raw material by the computer whose program executes a series of transformations over this material and, finally, outputs a new response (see Figure 5). Both performer's input and computer's response are translated in sound by the interface.

Organization, as the third level, concerns the system's language and is, in a musical sense, closely related to composition and music style. It comprises a set of rules that will guide the agents' behavior. These rules are fixed regarding the system's possibilities (agents) and internal relations (structure). The organization is taken in terms of a general law which governs the system's performance. In terms of our musical example, it could be a set of constraints about the actions to be executed. These constraints can be specific (play a major C# chord, loudly and shortly) or general (the length of the piece is five minutes) but, in any case, it is only by their whole effect that music arises as a creative work.

As in Peirce's triadic categories, these three levels exhibit a hierarchical configuration where each of them embodies the previous one. The agents reflect a mere possibility of the system. As isolated unites they can be seen as a set of qualities which are independent of any actualization in time. In Peircean terms, they belong to the firstness. Structure arises from the relations among these elements and it does not reflect the system's possibilities, but the way the system actually works. It implies causal reactions such as "the action A by the agent x produces the response B by the agent y". Based on concrete actions that happen in time, it belongs to the secondness. Finally, these actualities make it possible the emergence of entities semiotically more elaborated. It is related to the field of thoughts and ideas and, although they can potentially happen at a specific time, they do not depend upon any particular realization to exist: for example, a music composition embodies a certain structure and organization into itself even if it has never been played. Its general and a-temporal properties reflect its thirdness character (Table 4).

Table 4: Relationship between levels of interaction is a system and Peirce's triadic categories.

9 - INTERACTIVE FUNCTIONS



Each interactive music system will tend to concentrate its procedures on operations that can be focused in the generation, transformation, or interpretation of music data according to which kind of action they perform. These procedures can be defined by three functions: 1) the detection of context and of the material the system receives; 2) the processing applied to this material; and, 3) the elaboration of responses. These three functions are well depicted in a very similar way by Robert Rowe as part of three interactive stages: sensing stage, processing stage, and response stage (Rowe, 1993: 9). Most of the systems which have been developed up to now involve, in some way, all these three instances, but in general their approach is inclined towards one of them.

Thus, generative systems are related to the development of creative processes and are based on the agents' performance. They operate over structural material, detecting and exploring situations which are nothing more than possibilities of development. This process occurs in most improvisational music. In jazz, for example, the musician is given melodic and harmonic structures from which he may create original solos. The musician has to detect rhythmic and harmonic situations that will support the creation of solos. The input of interactive music systems which emphasize generative capacities consists of 'guidelines' instead of strict scores. The performers (humans or computers) enjoy some autonomy of action and can develop their own skills with certain independence from the composer's directions. In this situation the agents are presented to some kind of structure (a formula, a diagram) which can be a harmonic sequence or a rhythmic pattern, but not to the data to feed this structure. Instead of being coded in some kind of score, this data is conceived by the performers through a generative process of creation.

In another way, a transformative system is directed, not to creation of new material, but to the re-elaboration of previously conceived music or music fragments. In this case, the system operates over structured material. It receives not only formulas but also the data that feeds these formulas. In other words, it manipulates existent music data pre-established in such different ways as traditional scores, midi files or recorded sound data. In fact, in this instance, the system is not strictly creating newness, but distinguishing and transforming previously elaborated music data. Score followers programs are basically concentrated on this procedure. They keep track of input information and, based in previously stored data, they modify, link, or add new information to the original data.

Finally, in interpretative systems, the main goal is neither to originate new data nor to modify pre-existent music fragments. This kind of system aims to establish general relations among music structures by identifying or organizing music patterns which will rule the music discourse. In the interpretative level the system recognizes more elaborated structures which can be found by comparison against previously stored patterns or by detecting some kind of pattern recurrence in the music. As in a transformative system, it must be fed with some musical information but, as in a generative system it can also originate entirely new music data. Being able to detect and operate over some general aspects that govern a certain piece of music, these systems frequently are based on artificial intelligence techniques and their behavior exhibit adaptive or evolutionary characteristics. Generative, transformative and interpretative instances are closely related to the semiotic triads drawn in Table 4. Generative stage in concentrated on the development of the agent's skills and deals with non-temporal information (formulas, diagrams) to create musical response. Transformative stage deals with existent music material that is temporally organized. Finally, interpretative stage works by detecting the occurrence of events which can be taken as general laws which underlie some aspects of the music, independently of their occurrence at any specific time in a piece of music (Table 5). Most interactive music systems currently being developed comprise these three stages, but they tend to emphasize one of them in their processes.

interaction Generative Transformative Interpretative
material to process formula data pattern
kind of action detects musical context process music material elaborates music responses
operational level music agents music structure music organization
semiotic correspondence firstness secondness thirdness
Table 5: Functions in interactive music systems and their relation to semiotic categories.

10 - CONCLUSION


Since computer was introduced as a musical tool some decades ago, one can observe a tendency towards the production of works which do not require the collaboration of performers. However, in the last 10 years the development of new music technologies and efficient interfaces, has brought the collaborations between performers and computer systems as an important issue. Through interaction, the use of computers and other digital technologies reintroduces some important musical characteristics which have been relegated to a secondary plan during early stages of electroacoustic music. These characteristics are related to improvisation, contextual adaptation, emphasis on performance, and the approximation between composition, performance and listening processes.

The study and comprehension of music interaction is a valuable point in the development o music systems which involve the use of new technologies. In this sense, a semiotic approach can lead to a better understanding about the way interaction works and how it would be possible to explore the full potentialities of interactive systems.

Interactive music systems must operate in a full semiotic range in order to create conditions for the development of musical creativeness and stimulate sign growth. The composer of interactive music may pay attention not only to traditional subjects of music composition, but also to the expansive semiotic capabilities embodied in interactive processes. The dynamics brought to music by interaction is not a consequence of changes occurred in the context of performance or by the introduction of new musical elements. Indeed, this dynamic is much more related to the formulation of actions (composition and system design) than to the actions (performance) themselves. Interactive music systems might be conceived to freely permit the arousal of music signs, their creative connections and limitless expansion.

Computer music research has been guided towards a technical approach of music problems since its very beginning. In this paper we tried to raise some theoretical matters which are usually put aside by the computer music community. We hope that this attempt to formalize interactive music processes can help composers, performers, and designers of computer music systems in the generation of their musical ideas. Indeed, we believe that a better understanding of interaction would lead to the establishment of a vigorous relationship process between man and machine in the artistic field.

Acknowledgments:


This paper was written during my visit to CNMAT (Center for New Music and Audio Technologies), University of California, Berkeley. Part of the research was made possible by a grant from CNPq, Brazil.

References: