There are currently a wide variety of diverse methods of entering music into a computer in a format suitable for musical analysis and/or performance. Conventional music notation has been found to be less than ideal for music input and representation, hence a variety of music input languages have been developed. This chapter of the thesis looks at existing approaches to music notation and input languages and also compares between the features of different designs and the restrictions due to paradigms employed in the designs. A review of music performance systems employing one or more of the above input languages will also be covered. Whenever convenient, a comparison between the Rubato system and the system being reviewed will also be made.
This review will cover existing music input languages, music generation and performing languages, as well as musical programming languages that implements musical data types and operations and allow a representation of time in a musical composition. The languages surveyed will be presented here in a nondeterministic order, although some effort have been made to present languages related to one another in sequence for easy comparison.
Early programs developed for computer sound and music synthesis, generation and performance often define a musical note as the "specification of an acoustic event". This is mainly due to the history of acoustic experimentations in electronic music moving into the digital computer domain. The interest on timbre was a primary impetus in the growth of electronic music and, later on, computer music synthesis. The advantage of specifying music as a collection of acoustic parameters that is synthesized into waveforms is the freedom of expression beyond that of conventional musical instruments. However, every detail of a music performance had to be specified in painstaking detail.
The first, and most influential, of these family of programs were a suite of programs, named Music I to Music V, developed under Max Mathews et al at AT&T Bell Laboratories. These programs spawned a whole tree of descendants with similar names such as Music 4B, Music 4BF[7], Music 10 etc.[8] I will simply take Music V as a role model and refer to the whole class of programs for the remainder of this essay as MUSIC N.
The Music V environment is a fast general purpose computer with mass storage and digital-to-analogue converters. Music V takes a set of synthesis algorithms and note specifications as input, and generates waveform samples for the entire musical score onto tape. An auxiliary program then reads the samples off the tape and sends them across to the digital-analogue converters, hence producing music.
Synthesis algorithms are specified by interconnecting unit generator modules resulting in an instrument. A series of note statements containing a list of expressions specifying the note parameters are than parsed by Music V and converted into waveforms using the instruments defined previously.
Music V processes its input in three passes: a parameter conversion pass, a note sorting pass, and the actual synthesis pass. Instruments can be reentrant (i.e. the same instrument may be playing more than one note), a feature new to Music V and not present in Music IV.
In pass I, the score is read and data statements are interpreted into operations. Statements are free form in that they are terminated by semicolons and more than one statement may coexist on the same line. The first field is the operation code, the second the action time that specifies when the operation is to be done. Further fields are specific to the operation type. Fields are separated by white space or commas.
The operation code is a three letter mnemonic which is converted to a numeric equivalent.
After pass I, a temporary file is created which is directed on to pass II. Each statement is then sorted by action time in ascending order and a metronome function is applied to change the time scale.
Finally, pass III computes the actual acoustic samples by organizing unit generators into instruments and playing the instruments[9].
MUSIC N's complete lack of structure in the event (note) specifications allowed new approaches to composition. However, the acoustical model can sometimes constrain musical composition. Transcribing from CMN to MUSIC N is possible but is quite tedious for humans (A graphical interface to MUSIC IV has been done[10]). MUSIC N does not operate in real time, and often a 1 minute composition may take up to 20 minutes of computation. Performance interpretation, missing from CMN, is very difficult to code on MUSIC N, and the implicit assumption of notes as static entities that cannot vary once instantiated is not valid for certain types of music.
MUSPEC (J. P. Citron, MUSPEC pp. 97 - 111 [11]) is a high-level tool in musical composition. It may also be used as a music input language. The paradigms employed in MUSPEC are musicological rather than numerical in nature, but MUSPEC is not a true music input language. Input data into MUSPEC is only casually related to musical notation. Instead, the user is expected to think in terms of musical ideas such as pitches in tone systems, chord structures and rhythms.
MUSPEC grew out of research on the use of computers in aural pattern recognition.[12] As a result, MUSPEC views a musical composition as "... a voiced harmonic continuity and its subsequent melodization."[11]
The output of MUSPEC is a printed listing of notes and time durations, but not in musical notation. There are currently no means of performing (realizing) MUSPEC output as MUSPEC does not feature supporting hardware for musical performance. The only way of "listening" to MUSPEC output is to transcribe the results into conventional music notation for instrumental performance, or into a computer music performance system.
MUSPEC has been used to generate and compare the musical characteristics of persistent lines of chemical substances, phases of seismic disturbances, electrocardiograms, stellar luminosity plots and other phenomenological recordings[11]. In fact, the name MUSPEC is derived from the projected use of the program as a "MUsical SPECtroscope".
MUSPEC input is divided into two blocks: a declarative block and a 'executable' block.
This is the declarative section of the input. Some possible declarations are:
This is the pitch system of the music, given in user-defined symbols. Note that MUSPEC translates the symbols into numbers for internal processing and reconverts the resultant numbers back into symbols for output. The actual significance of the symbols as pitches in some musical scale is ignored by MUSPEC itself. An example of a tonal system declaration is:
TONSYS C DF D EF E F GF G AF A BF B
which defines the twelve-tone scale.
This is the allowed set of root tones upon which chords may be constructed. The entries are numeric offsets to the symbols named in the TONSYS statement.
ROOTS 1 3 5 6 8 10 12
CYCLES 5 2 3 5 2 5
gives the allowed root progressions according to the root tone scale in ROOTS.
Structures are numbers which enumerate intervals in the TONSYS scale separating consecutive notes in the structure. For example,
STRCTR 4 3 3
specifies a seventh chord on allowable root tones. Structures are used for harmonic composition as well as melodization.
Voicing statements specifies the harmonic continuity of the music and control the voice leading between chords. The entry of a particular voicing position specifies the structure note of the next chord which the note in the current structure component must move to. An example is
VOICNG 1 3 4 2
The first chord is voiced using either CHORD1 or CHORDP. For example,
CHORD1 1 3 4 2
states that the initial reference choird is ordered root, third, fourth and second structure tone. CHORDP will cause structures and root tones to be "phased" in calculations.
Basic Rhythmic Group specifies and overall or "macrorhythmic" control:
BSCGRP 2,6/1,4/1,3,2
establishes 2 units of time with a maximum of 6 attacks, 1 unit of time with a chord change (as indicated by the minus sign) and maximum 4 attacks, and 1 unit of time with a maximum of 3 attacks and a minimum of 2 attacks.
"Microrhythmic" attack patterns are specified as relative duration groups, i.e. duration is an integer with respect to a minimum value. A negative duration value indicates a rest:
RELDUR 3 1 2 2 -1 3 2 2
A MELODY statement is a string of integers representing notes selected from interval structures.
The second data block may contain any set of numbers at all, whether abstractly chosen or purposely contrived. This block begins with a line with the word LINES starting in column one. The numbers in the line data are used to select material from the musical data. First, a root tone cycle is selected from available CYCLE scales, followed by voicing choice, macro- and microrhythmic selection and melody. Hence, each set of numbers of a line triggers a set of selections and corresponds to the 'executable' section of a programming language.
One of the reasons behind the implementation of MUSPEC was to allow composers and arrangers to communicate musical thoughts in a relatively high level to the computer, and in this respect it is fairly successful. MUSPEC employs a different set of paradigms from CMN hence converting from musical scores into MUSPEC input requires a bit of thought. The biggest disadvantage of MUSPEC (which it shares with some other music input languages) is the lack of ability for the computer to 'play' music written in MUSPEC, but it is not inconceivable to develop a music performance system that plays MUSPEC output. MUSPEC lacks essential musical attributes such as timbre and loudness, and until these features are added to the language it will remain primarily a compositional tool.
MIR (Musical Information Retrieval) is a language developed in order to assist musicologists in "... answering internal-evidential questions about ... (musical scores)". (M. Kassler, MIR - A Simple Programming Language for Musical Information Retrieval, pp. 299 - 327[11]) Hence, it is primarily an analysis tool rather than a music input language. In theory, any query relating to the symbols of musical notation such as notes, rests, clefs and so on can be formulated as an MIR program.
MIR is written as a macro-language of FAP (FORTRAN Assembly Program) and individual MIR instructions can be viewed as direct commands to a machine that performs specific operations on musical data. In this respect, MIR programs can be likened to the assembly language instructions for the Rubato virtual machine. At any given time, this virtual machine can hold at most one note and all its related attributes. This is called the current note.
The following is representative of the type of data pertaining to a note:
In addition, a note also contains attributes such as dynamic mark values, timbre, title, author, publisher of composition and so on.
MIR instructions consists of three fields: location (or symbolic address), operation (or instruction type), and variable (object acted on by operation). Hence MIR programs are very similar in spirit and style to assembly language programs on computers. There are instructions such as TOSECT, TOMEAS, TOLYNE, TONOTE etc. which directs processing to a certain position in the composition, arithmetic instructions, logical instructions and output instructions that allow queries to be made on a musical composition.
MIR is not a music input language, and hence does not feature a means whereby music is transcribed into the system (this can be done by a suitable music input language).[13] The companion music input language to MIR is called IML, which is not discussed here.[14] It is interesting however in that it treats a musical composition as an information database on which inferences may be made and queries can be formulated. A conversion program exists that converts the MIR database to a MUSIC IV program which allows "proof-hearing" of the contents of MIR scores and greatly enhance the usefulness of MIR. The biggest advantage of MIR is its programmability and flexibility, its biggest disadvantage its cumbersome assembly syntax.
PILE (P. Berg, PILE - A Language for Sound Synthesis, Computer Music Journal, 3(1): 30-41 (1979), reprinted[15]) is a real-time, direct sound synthesis language on the DEC PDP-15 computer. It produces several layers of sounds simultaneously in real-time. It differs from the MUSIC N species of programs in that PILE instructions are based on machine operations rather than an acoustical model. Hence PILE instructions typically represent "structured" assembly languages rather than MUSIC N streams and in this respect PILE is more closely related to MIR and the Rubato virtual machine rather than MUSIC N.
Each PILE instruction contains the name of an operation on one line and further information on subsequent lines. Instructions are terminated by semicolons and must begin with white space, in order to differentiate instructions from labels (another convention borrowed from the assembly language world). Music is generated from manipulating the contents of an "accumulator" and then sending it to a hardware "channel".
Besides the usual conditional, logical, arithmetic and branch instructions, there are instructions to manipulate 'lists', the accumulator and external devices.
PILE can be a flexible means of generating music using a low-level machine operations based method. Berg explains (pp. 169-172[15]), in a hierarchical example, how PILE can be used for FM synthesis [16].
A set of tools for the Structured Sound Synthesis Project (SSSP) at the University of Toronto has been developed which uses interactive computer graphics to allow a composer to edit musical scores in a highly interactive manner. (W. Buxton et al, The Evolution of the SSSP Score-Editing Tools, Computer Music Journal, 3(4): 14-25 (1979), reprinted[15]) It operates on a DEC PDP-11/45 running under the UNIX system[17] with vector-drawing graphics displays and digitizing tablets. Musical output is produced through the SSSP Digital Synthesizer (W. Buxton et al, "An Introduction to the SSSP Digital Synthesizer" [15])
Ludwig[18] is a graphical score editor based on conventional music notation. A graphical representation of the score is displayed in conventional music notation (or a subset thereof) and the current display "window" into the music score can be scrolled in the horizontal direction. Editing is performed through the digitizing tablet and editing operations are based on recording tape technology in that the composer can perform a splice and mix of the score which corresponds to the relevant tape recorder operations. In order to eliminate typing as much as possible, timbral colours and filenames can be selected by pointing at the appropriate menu item on the screen through the digitizing tablet.
Scored is very similar to Ludwig but employs an alternative display paradigm: music is displayed in "piano-roll notation" rather than using conventional music notation symbols. The internal data structure of Scored is similar to Ludwig, the same information is essentially displayed in a different form.
This is yet another representation of the same data structures, but the approach used here is alphanumeric rather than graphical. The primary reason for the development of sced (W. Buxton, A tutorial introduction to SCED[19]) seems to be to allow editing of scores on alphanumeric terminals.
A new idea was introduced in sced which was the concept of scope. Ludwig and Scored are essentially "scopeless" editors; the composer either deals with a note by itself or the score as a whole. Sced however allows a group of notes to be manipulated as a whole as a result of the editor interface which resembles the UNIX text editor ed(1).
This is the cumulative result of previous work in designing the above editors. Scriva (W. Buxton, A tutorial introduction to SCRIVA[19]) is a graphical score editor that can view a common data structure in one of four different notation schemes: conventional music notation, piano-roll, object highlighting and envelope highlighting. Scope is built in by allowing the composer to "lasso" or circle a set of notes and treat them as an object.
The SSSP score editing tools are an interesting set of programs in that they are all related to one another and run on the same hardware but are substantially different in their user interfaces and sophistication. The conclusion drawn by the authors are firstly, a graphical approach offers a very congenial method of editing musical scores compared to conventional alphanumeric music input languages. Secondly, the concept of scope is an important one and foreshadows the heirarchical concept of music with instances of the same phrase in a musical composition sharing the same location in computer memory. (W. Buxton et al, "TheUseofHierarchyandInstanceinaDataStructureforComputerMusic", Computer Music Journal, 2(4), 10-20 (1978) revised and updated[15])
The biggest disadvantage of a graphical user interface in a musical input language is the lack of programmability. It is difficult to specify that a certain set of notes is to be repeated at various positions in a musical composition except by physically copying the set of notes. Once copied, however, modification done on the original set will no longer affect the copies.
INV (C. Abbott, A Software Approach to Interactive Processing of Musical Sound, Computer Music Journal, 2(1): 19-23 (1978), reprinted pp. 512 - 522[15]) or "Invoke", is a simple interpretive programming language that performs mixing and editing operations to produce pieces of music. It is only casually related to music generation programs as it has no concept of music at all, it simply 'records' (samples a waveform) musical notes onto a hard disk and allows mixing and editing of the waveform samples before producing them on the output ('playback'). The language is a fairly low level language similar in spirit but not in surface appearance to BASIC.
On the lowest level, event structures and integer variables can be written to and accessed. Control flow and functions are called activities. There can only be one activity per line, corresponding to statements in a programming language. In addition, the following musically oriented primitives are defined:
User defined activities (called macros or functions) are stored in a library. During program execution, the interpreter fetches the macros from disk and executes the macros until the end of the macro. The calling-level stack is organized such that macro symbols are local to the macro.
INV is unique among all the other languages presented in this review in that it has no concept of music at all. The 'knowledge' of INV is limited to that of recording and playing waveform samples, although a limited waveform generation capability does exist. However, INV can produce music, simply because it is a very fast random-access tape recorder. If all the notes of a grand piano are recorded into INV and then a selected set of notes are played in sequence, music can be produced.
Another unique feature of INV is its capability of dealing with 'events' regardless of their type using an 'event structure'. This, plus programmability, makes INV a flexible tool for generating music but the prospective INV composer would be hampered by INV's lack of musical knowledge.
PLAY (J. Chadabe, R. Meyers, An Introduction to the PLAY Program, Computer Music Journal, 2(1): 12-18 (1978), revised and updated pp. 523 - 538[15]) is a program for music generation that is not rooted in traditional music notation but instead tries to to maximise flexibility in the conceptualization of temporal processes. It functions in two distinct stages:
PLAY consists of three subprograms which can be called in any order, even during playback:
PLAY is a flexible interactive music generating program that is clearly descended from the MUSIC N model. However, the interactive aspects of PLAY make it easier to use than MUSIC N. It is clearly intended as a music performance language rather than a music input language. Hence, the specification of musical scores within PLAY is rather tedious.
The Ford-Columbia Input Language (S. Bauer-Mengelberg, The Ford-Columbia Input Language, pp. 48 - 52[20]) is the result of research to allow printing and typesetting of music under computer control using photo-composition equipment. Hence the emphasis was to develop an alphanumeric means of encoding complete musical works (including indications on how the music "looks" in conventional notation) in such a manner that CMN can be completely recovered from the alphanumeric code language. Rather than encoding just incipits, or tones, or harmonies, the Ford-Columbia language (hereinafter referred to as FCL) is a representation of CMN in alphanumeric form.
Although the initial application of FCL was in phototypesetting specification, FCL has been used successfully for musical analysis and comparative musicology and transcription of tablature to standard notation.
Notes in FCL are specified by their positions on the musical staff. Each position (irrespective of clef, key signature etc.) of a note on a staff is encoded numerically. Horizontal displacement between notes are regarded as superfluous and two notes are separated by a single space in FCL.
Initially, information such as clefs, time signatures and key signatures are specified. For example. "25!C" is the notation for the alto clef, where 25 is the staff code of the clef and 'C' is the letter of the clef. Similarly, time signatures are indicated by "!M3:4" for waltz timing, and key signatures by "!K3#" for A-major.
Notes are indicated with one letter mnemonic duration values, and stem indication (normally redundant information, but required for CMN). In addition, dynamic marks, staccato, tenuto, staccatissimo, marcato, fermata and other musicological symbols may all be specified. Repeating passages may be specified with a repeat code, or a macro-definition and expansion.
FCL has proved to be a surprisingly useful and flexible tool for purposes other than which it was designed for. It should not be too difficult to design a music performance system that reads in scores in FCL provided only a subset of the total information presented in FCL is actually used. However, the main disadvantage of FCL is its insistence on representing all aspects of CMN, which makes the input language rather cumbersome for non-typesetting applications.
A similar input language to FCL, but much more compact and easier to key in, is the so called Plain & Easie code (B. S. Brook, The Plaine and Easie Code, pp. 53 - 56[20]) which has been extended into ALMA (M. J. Gould, G. W. Logemann, ALMA : Alphanumeric Language for Music Analysis, pp. 57 - 90 [20])
Plaine and Easie (abbreviated to P&E) is a simple means of notating music alphanumerically but has a close relationship to CMN. Pitch and rhythm can be completely specified and extensions are available for dynamics, phrasing, harmony, stemming etc. After identifying the piece and composer, P&E allows specification of clef, instrument, tempo, key signature and meter in a manner similar to FCL. Notes however are given in alphanumeric symbols (A, B, C, ..., G for pitch values with accidentals) with duration expressed as numbers. Rests are indicated by hyphens and ties using the underscore or plus character. Triplets and unusual rhythmic groupings are enclosed in parentheses preceded by total duration e.g. "4(8 C D E)". Additional symbols such as accidentals, bar lines, trills and grace notes can also be specified.
ALMA is a much more sophisticated notation system based on P&E. The amount of information to be coded, as well as the ordering of the information and choice of symbols, is ultimately up to the researcher but certain mnemonically convenient symbols are suggested. This allows most music in CMN to be adequately transcribed into ALMA, with special notations and symbols easily added to the extensible language.
In ALMA, the two dimendional CMN is represented as a one dimensional string of alphanumeric characters by multiple scan lines, each scan line usually representing one musical voice. The path of the scan is the order of examining music notation. Notes of pitch and duration may be grouped into groupettes which may be manipulated as a single entity. Attributes may give specific musical information about the notes, change the meaning of future symbols (e.g. key signatures) or control the scan. Chords may be specified either by enclosing chord members within the dollar sign ('$') symbol or using the '<' symbol to indicate 'move scan back to start of previous note'. In other respects, ALMA notation is very similar to P&E.
Attributes may be specified for groupettes, and may possess scope. A large variety of suggested attributes ensure that conventional CMN performance symbols are adequately represented. A powerful set of scan control functions are also provided.
P&E and ALMA are very compact music input languages designed primarily for musical analysis and hence are essentially representations of CMN in a more congenial form suitable for data entry. A variety of music input languages for musical analysis have been developed along the lines of P&E and ALMA. (J. Wenker, A Computer Oriented Music Notation including Ethnomusicological Symbols, pp. 91 - 129 [20]) (R. Jackson, P. Berzott, A Musical Input Language and a Sample Program for Musical Analysis, pp. 130 - 150 [20]) Performance systems for P&E is feasible but limited by the lack of additional notation in P&E. A performance system for ALMA would be difficult due to the wide range of attributes available, the possible paths of scan control, and the extensibility of the language.
Adagio (R. Dannenberg, The Adagio Language[3]) is a very simple nonprocedural notation for scores. Each note is specified on a single line, with additional attributes such as duration, time of next note, dynamics, voice and timbre. Multiple notes may be joined together on a single line by terminating each note with a semicolon (';'). Chords can be represented by specifying that the next note starts on the same time as the current note, or by separating notes by a comma ('.').
In addition, global attributes such as tempo and rate (speed relative to current tempo) as well as control change commands such as portamento can also be set. A useful method of entering multiple voices with dissimilar timings or durations is by means of an easy "rewind time" feature that treats the rest of input as starting at the same time as the previous tempo or rate change or rewind command.
Adagio is different from other languages in that it is reasonably close to musical notation (employs similar models and paradigms) and yet Adagio is mainly a performance language. Adagio scores are not used for musical analysis but are instead read in by the Adagio interpreter which converts the notes into MIDI(7) commands for performance on suitably equipped conventional commercially available music synthesizers. The primary advantages of Adagio are its flexibility and ease of use as well as the performance aspects of the Adagio interpreter. However, it is sufficiently primitive compared to CMN as to make it unsuitable for musical analysis.
In the course of this review we have witnessed a surprising large range of music input languages, ranging from primitive low-level sound generating programs like MUSIC N, INV, and PLAY through to musical analysis languages like P&E and ALMA to musical typesetting control languages such as FCL. In between are programmable languages such as PILE and MIR and at the compositional level I have reviewed MUSPEC.
It can be seen that different approaches and models used in developing music languages and music performance systems can lead to radically different designs with different features and limitations. In a sense, the diversity of music input languages parallels the diversity of sound synthesis. What remains to come is a general purpose music language that combines the best features of all the above languages mentioned and allows music performance as well. While this is not difficult, the challenge to preserve simplicity on top of an overall approach may prove to be an interesting problem in computer language design. This is the problem that Rubato attempts to solve.
Next: Chapter 4: A TUTORIAL TO THE RUBATO LANGUAGE
Previous: Chapter 3: LITERATURE REVIEW
Back to: Table of Contents