The Music Notepad

The Music Notepad
Andrew Forsberg, Mark Dieterich, and Robert Zeleznik
Brown University
Department of Computer Science
Providence, RI 02912
(401) 863-7693; asf,mkd,bcz @cs.brown.edu
ABSTRACT
with a number of musicians and composers we believe a fun-
We present a system for entering common music notation
damentally different music notation interface based on a pen-
based on 2D gestural input. The key feature of the system is
based UI will be more desirable and of equal or greater value
the look-and-feel of the interface which approximates sketch-
than a WIMP-based UI.
ing music with paper and pencil. A probability-based inter-
preter integrates sequences of gestural input to perform the
The Music Notepad attempts to be an interactive electronic
most common notation and editing operations. In this paper,
sheet of music paper. Unlike WIMP UIs, the Music Notepad
we present the user’s model of the system, the components
is characterized by a portable display surface (a Wacom PL-
of the high-level recognition system, and a discussion of the
300 Display Tablet) that can be directly drawn upon with a
evolution of the system including user feedback.
stylus. To support what can be done with pencil and paper
interfaces, the Music Notepad interprets gestures speciﬁed
KEYWORDS:
user interface, interaction, music notation,
by the user with a stylus to create notation. Moreover, ges-
gestural input, gesture recognition, handwriting recognition,
tures can also be used to perform more powerful editing op-
direct displays.
erations, to professionally format notation, and to synthesize
instrumental sounds based on the notation.
INTRODUCTION
There are a number of situations that revolve around infor-
The following sections present previous work, the user’s model
mally notating music
of the system, the details of our recognition methodology,
½
, such as when a composer wants to
jot down an idea, when a teacher explains theory to students,
and a discussion of the formative design of the system through
or when a musician wants to visualize (i.e., notate) a musical
user feedback.
idea. Despite the many advantages of applying computers to
PREVIOUS WORK
music notation (e.g., for synthesizing sound, neatly format-
There are two common approaches to music notation: pa-
ting and rendering notation), people in fact resort to using
per and pencil, and computer-based systems. Paper and pen-
just paper and pencil for many tasks even when computer so-
cil has many advantages– notably low cost, simplicity, and
lutions are available. This paradox derives from the nature of
portability. Music can be notated by drawing symbols on
the interfaces for typical music applications.
inexpensive paper. Sheets of paper can be copied and dis-
Most computerized music notation systems employ standard
tributed very easily. However, producing high quality pub-
windows, icons, menus, and point-and-click (WIMP) user in-
lishable documents by hand requires great skill and editing
terfaces (UI’s) as well as a transition to keyboard “hotkeys”
operations that are difﬁcult to perform. Other desirable con-
for frequently used functions. In some cases, these systems
cepts such as automatically performing written notes are not
offer advantages over paper and pencil notation such as rapid
possible. ¾
data entry, editing ﬂexibility, automatic formating, synthe-
There are two main ﬂavors of computer-based systems: se-
sized sound, and high-quality printing. However, the user’s
quencer and notation systems. The goal of sequencers is
model for computerized systems is very different from the
to enter and perform synthesized music, whereas notation
model of paper and pencil notation. Based on discussions
programs are intended only to produce high-quality printed
scores. Both are successful in addressing some aspects of the
½
See [1] and [13] for further information on musical terms and common
music notation
problems of paper and pencil systems such as improved edit-
ing operations (e.g., editing individual or groups of symbols
and transposing), and synthesizing or printing a high-quality
version of the music that has been entered.
¾
Written music can be performed by one or more skilled performers.
However, there are often signiﬁcant barriers to becoming a skilled performer
such as years of practice and expense.

However, the handwriting techniques for creating standard
tation. Although both systems share many comparable fea-
music notation [13] learned by many musicians bears little
tures, the Music Notepad provides unique functionality, such
resemblance to the music software UIs. Instead, they tend
as allowing the user to retain a “non-ﬁnished” look (making
to use the well-established WIMP-based UIs. These UIs are
it appear closer to a pen and paper look), extensive support
sometimes augmented with a MIDI input device such as a pi-
for editing notations, and score playback.
ano keyboard. The primary advantage of a WIMP interface is
its simplicity and learnability. In addition, many applications
SYSTEM DESCRIPTION
such as Finale [6] or Cakewalk [3] provide mechanisms that
This section describes the user model of the system followed
allow users to transition from using the WIMP interface to
by the details of the components used to integrate the various
using keyboard “shortcuts.” Shortcuts are typically learned
types of input.
over time by displaying each shortcut key next to the equiv-
alent WIMP interface command. Gradually, users tend to
interactive playback
transition to using only the shortcuts resulting in very fast,
although indirect, user input. The use of MIDI devices for
input to a notation system at ﬁrst seems appropriate, but is
still not ideal because nearly all input from MIDI devices
requires editing. This problem is rooted in the need to be
skilled performer of a particular MIDI device.
sheet manipulation
There have been several research systems for music notation.
moving symbols
The Mockingbird system [12] was a pioneer in the use of
a graphical UI and MIDI keyboard. However, because this
system relies heavily on MIDI keyboard input nearly all input
drawn
requires skilled performance and editing. The system also
gestures
depends on a WIMP interface-style to edit notation.
Buxton [2] developed a system which included a set of ges-
tures for specifying notes and rests (see Figure 1). While
this is an effective set of gestures for very basic note entry,
the system does not provide gestures for other fundamental
Figure 2: The stylus has four buttons for controlling all
notations such as stem direction, accidentals, and beams. Al-
Notepad operations.
though we have integrated Buxton’s gestures in the Music
Notepad, there are some situations where the gesture scheme
User Model
is cumbersome. For example, multiple short notes often ap-
Music Notepad is intended to appear to the user as an interac-
pear in long sequences, but the gesture for creating short
tive sheet of music paper. Thus the user accesses all function-
notes is unfortunately relatively complicated.
ality gesturally with a stylus. There are four classes of gestu-
ral operations that correspond to the four buttons of the stylus
(see Figure 2). Marking gestures, drawn with just the tip of
the stylus, leave ink trails on the display; sequences of these
gestures are interpreted as either handwritten commands or
as operations for creation and deletion of musical notations,
marking menu [9] invocation and selection, as well as region
selection. The lower button of the pen is used to perform di-
rect manipulation operations for changing note pitches and
graphical placement of symbols. The second lowest button
Figure 1: A set of gestures developed in [2] for creating
notes of various durations. Rests are created by mirroring
allows the user to slide the “music paper” across the display
these same gestures around the horizontal axis.
screen. Finally, the eraser button of the pen is used for play-
back of the entire score or for interactive playback of regions
A similar gestural component is embedded in an otherwise
of musical notation.
conventional WIMP interface in both the NoteWriter and the
NoteAbility systems [14]. These systems provide simple
Creating notation symbols
The most basic operation in the
gestural alphabets that are related to Buxton’s gestures both
Music Notepad is the creation of notes. Users can create
notes using the gestures shown in Figure 3. Gestures convey
in functionality and limitations.
both spatial and symbolic information. Note creation ges-
As we prepared the ﬁnal version of this paper, we learned
tures consisting of only distinct line segments are centered
about a similar system GSCORE [17]. GSCORE provided
on the ﬁrst point of the gesture. Creation gestures involving
both WIMP based and gestural based methods for music no-
drawn noteheads (called “scribbled” noteheads) are placed

based on the computed center of the notehead. Scribbled
to indicate a location followed by a command or text. Com-
notes default in duration to a quarter note, however, the size
mands are speciﬁed through handwriting or speech. For ex-
of the drawn notehead also determines the note’s volume.
ample, to change the key signature the user draws a dot at the
That is, a small notehead plays softer than a fat notehead. If
desired location and writes “D major”.
a scribbled notehead has a tail that extends above or below
Editing
The Music Notepad currently supports the most com-
the notehead, then a sharp or ﬂat is associated with that note.
After a new note is created, the synthesizer sounds that note.
monly used music notation editing operations: deleting, copy-
If the note is created above other notes, a chord is recognized
ing, pasting, and region selection. One or more objects can
and all notes are played back simultaneously.
be deleted by scribbling on top of them. To select a region,
the user draws a lasso around the region. Multiple regions
The duration of an existing note can be modiﬁed by drawing
can be speciﬁed by drawing multiple lassos. To delete a se-
a slash gesture through its stem. A gesture from the left to the
lected region, the user scribbles inside a lasso. There are also
right will half its duration, whereas a gesture from the right
two single-stroke delete gestures derived from text editing
to the left will double its duration.
notation (see Figure 4).
A quick tap on a notehead will mark that notehead as stac-
cato. The staccato marking of a notehead will disappear if
it is already notated to play staccato and is tapped with the
stylus. This quick tap gesture is similar to the meaning of the
staccato marking. Pressing on a notehead for a longer pe-
riod of time will toggle whether that notehead has a “dotted”
value.
Rests can be created by using the gestures from Figure 1 mir-
rored about the horizontal (to distinguish from the “create-
a-note gestures”.) Rest durations can be doubled or halved
in the same way note durations are modiﬁed; by drawing a
short gesture through the rest.
Figure 4: Two single-stroke delete gestures derived from
text editing.
Beams for a set of notes can be created by drawing a straight
line above or below a set of notes. This position and angle of
To copy a region, the user holds the stylus just above a las-
the gesture line determines the distance of the beam from the
soed region, presses the move button on the stylus (which
noteheads as well as the angle of the beam. In some situa-
creates a ghosted copy of the symbols in the lasso), and then
tions, a speciﬁc set of notes can be accurately beamed by ﬁrst
releases the button at the location he or she wishes to place
lassoing the desired notes and then drawing the line above
the copied symbols. To move a region, the user touches
or below the set of notes. Once a group of notes has been
the tablet surface and presses the move button on the stylus,
“beamed,” drawing a gesture line through the stems of notes
moves to the new location (dragging a copy of the lassoed
in the group will half or double the duration of those notes,
symbols), and then releases the move button to place the las-
depending on the direction of the gesture line. Scribbling on
soed symbols. This style of pen-based interaction is similar
a beam erases a beam and ungroups the notes associated with
to Pick-And-Drop [15].
it. Users can also adjust the slant of a beam by directly ma-
nipulating either the left or right side of the beam. If the user
Instrument Assignment
To assign a musical instrument to a
grabs the middle of the beam, then the height of the beam is
staff, the user draws a gesture line from an instrument shown
adjusted.
in the “instrument picker” (see Figure 5) to the staff the in-
strument is to be assigned to. Alternatively, the user can draw
Accidentals for an existing note can be added using a mark-
a dot on a staff and then write or speak the name of the de-
ing menu. If the user holds the stylus on a notehead for
sired instrument (see Figure 3).
some short period of time, a radial menu appears showing
the marking menu choices for accidentals. By dragging and
Visualization
Typically, music is printed on roughly 8.5” by
releasing in the direction of a menu item, a sharp, ﬂat, or nat-
11” paper and two or more sheets are arranged in a row for
ural is associated with the note. The ﬁnal menu option clears
viewing. However, our Wacom display tablet has a total dis-
all accidentals. If the user does not wait for the menu, the op-
play area of only 8” by 6”. To increase the effective display
eration is still performed as with any other marking gesture.
area of our tablet, the user can create a division in the “music
New notes with accidentals can also be created using a sin-
paper” on either side of the display tablet in order to display
gle gesture as previously described in the creating notation
more of the musical score (see Figure 6). This is based on the
section.
perspective wall metaphor [11]. As a result, the user has in-
creased context for the current working area and also makes
Clefs and key signatures are created by drawing a dot or lasso
it easier to navigate through the score.

the token’s features, often in the context of the existing no-
tations. For example, a RECOG that erases lassoed objects
returns a probability based on how well the RECOG consid-
ers that one token describes a lasso around the objects and
that the other describes a squiggle within the lasso. ¿ If a
RECOGs probability is greater than all other RECOG prob-
abilities and it is also greater than some threshold (we used
50% certainty), then the RECOG is executed and the accu-
mulator is cleared. Conceptually, our approach is similar to
the uniﬁcation-based multi-modal integration from [8].
Handwriting
Stylus
Interpreter
Single Stroke
Token
Token
Interpreter
Token
Token
Speech
RECOG
Interpreter
Figure 5: An orchestral instrument picker– the user draws
Microphone
Execute()
a gesture line from an instrument to a staff to assign instru-
Probability (0-100%)
ments.
User Input
Tokenizer
Accumulator
Recognizers
Figure 7: Overview of system components: User input is
transformed to a token data structure that describes the spa-
tial and textual information of input. Command recogniz-
ers (called “RECOG”s) examine the accumulated tokens. If
the top probability of a RECOG exceeds some threshold, its
function is executed.
We employed the concept of an Arbitrator for resolving sit-
uations where multiple RECOGs reported high probability
that they were recognized. For example, the RECOGs that
identify the gesture to insert an end-of-measure symbol and
to insert a quarter note (based on gestures in Figure 1) both
look for nearly the same input tokens. We chose to keep in-
Figure 6: A Perspective Wall of music.
dividual RECOGs simple and in the case where there is a
conﬂict, the RECOGs that reported high recognition proba-
bilities were passed to the Arbitrator which is responsible to
Audio Feedback
In addition to hearing notes or chords when
resolve the conﬂict. The Arbitrator consists of case-speciﬁc
adding new notes, there are two other types of audio feed-
code for expected conﬂicts. For example, in the end-of-measure
back in Music Notepad. First, the user can drag the other end
and quarter note scenario, the Arbitrator has a resolution han-
(the eraser side) of the stylus over notes. As the stylus passes
dler that has been hand-coded to inquire additional context to
over notes, they are played back. Second, using a different
resolve the situation.
gesture with the eraser end of the stylus, the entire piece will
be played back.
There are some important details to the UI system compo-
nents. Stylus input can be directed to one of two tokenizers–
UI System Components
a single stroke interpreter or a handwriting interpreter. By
This section describes the Music Notepad’s UI system com-
default, all stylus input is directed to the single stroke in-
ponents. The ﬂow of input through the system is illustrated
terpreter. If the ﬁrst single stroke gesture drawn is a “dot,”
in Figure 7. The tokenizer converts user input into feature
then the system recognizes this as an out-of-band gesture
vectors, called tokens, data structures that describe an input’s
and channels all remaining stylus input through a handwrit-
spatial and textual content. Each token consists of 25 fea-
ing recognizer. The “dot” gesture generally acts as a spatial
tures including, for example, the time to enter the user in-
marker for the subsequent handwritten command.
put, the length of a stroke, the object under the starting point
of a stroke, and the number of self intersections. Tokens
Our tokenizer interpreters are Rubine’s gesture recognition
are posted to the accumulator where they are examined by
system [16] for single strokes (such as those from Figure 1),
command recognizers, called RECOGs. Each RECOG has
¿
Since probabilities are in essence arbitrary, we had to iteratively reﬁne
a customized procedure for estimating the probability that
different techniques for computing and combining probabilities, with the
the posted tokens are “what it is looking for” by analyzing
result being effective but “ad hoc.”

the Calligrapher [4] system for handwriting recognition, and
they often misplace the notehead. Since users want staffs to
the In-Cube speech recognition system [7]. Additional sup-
be drawn at a standard printed size, the spacing between staff
port code translates the interpreted input to our token data
lines is relatively small. Consequently, as predicted by Fitt’s
structures which are then posted to the accumulator.
law, users have increasing difﬁculty in placing notes accu-
rately as they work faster. Each time a note is misplaced, the
DISCUSSION and USER FEEDBACK
user must perform at least one additional editing operation to
The design of the Music Notepad has undergone a number of
correct the mistake.
iterations guided by formative evaluations by small groups of
musicians and composers. The following discussion presents
In response to our early user experiences with gestures like
some of the results of our user experiences and a description
those of Buxton (see Figure 1), we developed an alternate
of how our designs changed in response to that feedback.
method for entering notes. With this method, users position
a notehead by “scribbling in” a gesture that looks like a note-
A variety of user gestures
head. There are several differences in this method for enter-
Although all users were instructed with both a demonstra-
ing notes. First, this gesture is more accurate than the point-
tion and a description of the gestures used in the system, we
and-click approach because the position of the note is speci-
found wide variation between individual performances of the
ﬁed by the average position of the multiple samples deﬁning
same gesture. Figure 8 illustrates some of the varieties of
the scribbled notehead. Second, this gesture maps directly to
four common, simple gestures.
how a notehead is drawn on paper. Third, since this gesture
can act as the image for a notehead, we can avoid the distrac-
tion of replacing the drawn gesture with a different image.
Last, this technique can be slower than the point and click
technique and it does not convey the note duration.
Visual Representation
In addition to supporting What-you-see-is-what-you-get (WYSI-
WYG), the Music Notepad also supports What-you-see-is-
what-you-entered (WYSIWYE). The goal of WYSIWYE is
to minimize the time a user spends understanding the ef-
Figure 8: Different user styles for drawing the same ges-
fects of an action. There are two instances of WYSIWYE
ture. Clockwise from top left: Erasing notes by squiggling
in the Music Notepad: sketchy noteheads and delayed auto-
on them, selecting notes with a lasso, scribbling to create
formatting.
noteheads, erasing notes with a single gesture lasso.
Sketchy noteheads
When notes are entered by sketching
a notehead, the notehead is represented by exactly the line
In response to the range of gesture styles, we were able to re-
the user sketched instead of replacing the line with a perfect
design our gestures and develop more robust gesture recog-
notehead.
nition algorithms. By analyzing the actual drawn gestures
from a number of users, we identiﬁed what types of features
Delayed auto-formatting
In our evaluation of existing mu-
were important in different situations and used this informa-
sic software systems, we found that each one reformats some
tion to ﬁne tune the recognition probabilities in speciﬁc ges-
subset of the notation every time a new symbol is created.
ture RECOGs.
While automatic reformatting is a useful feature, it can also
be distracting– especially since the look of the document is
Based on informal interviews of users who had tried the Mu-
often irrelevant during informal music entry. In response to
sic Notepad, we found no negative reactions to learning and
the reaction of some users, we delay this automatic format-
using the gestural style of interaction. In addition, some
ting until the user speciﬁcally requests it.
users, including musicians, indicated they would prefer us-
ing a completed version of the Music Notepad over existing
A major issue with WYSIWYE is ensuring that gestures are
alternatives. The PalmPilot reﬂects similar learnability issues
initially interpreted correctly. If not, this may lead to refor-
and has gained widespread acceptance despite its use of the
matting errors that must be painstakingly located by the user.
idiosyncratic graﬁtti alphabet and the time required to mas-
Filled and hollow notes, for example, can be difﬁcult to dis-
ter it. Based on these reactions and the similar idiosyncratic
tinguish even by humans from a sketched notehead. We think
style of gesturing between the two systems, we believe the
it may be effective to incorporate feedback for marking am-
Music Notepad would have similar acceptance.
biguous notes in a similar way to Microsoft Word’s “squiggly
underline” technique for highlighting misspelled words.
Accurate placement of symbols
Some music notation symbols such as noteheads must be ac-
Marking Menus and Direct Input
curately positioned. If the user speciﬁes a position with a sin-
There are many advantages to a direct draw environment,
gle mouse sample (e.g., by pointing and clicking), we found
however, one disadvantage is that the user’s hand can block

part of the display. This is a problem when using mark-
REFERENCES
ing menus in the traditional manner because candidate menu
1. Ammer, C., “The Harper Collins Music Dictionary,”
items are obstructed by the user’s hand.
New York: Harper Peremial, 1991.
We propose two solutions to this problem: ﬁrst, allow the
2. Buxton, W., Sniderman, R., Reeves, W., Patel, S., and
user to lift their hand and the stylus tip from the tablet to view
Baecker R., “The Evolution of the SSSP Score Editing
the choices of the radial menu. After viewing the choices,
Tools,” Computer Music Journal, Issue No. 12, 3:(4),
the user can select an item by touching the item with the sty-
pp. 14-25, 1979.
lus tip or cancel the operation with a different gesture. The
second solution is not to display items underneath the user’s
3. Cakewalk
Pro
Audio,
Cakewalk,
Inc.,
hand. This requires sensing where the user’s hand is and
http://www.cakewalk.com/.
might be accomplished with a Wacom tablet by using the
data that reports the orientation of the stylus.
4. Calligrapher,
ParaGraph
International,
Inc.,
http://www.paragraph.com/.
Mouse versus Stylus Input
In order to validate the need for a stylus-based gestural in-
5. “Common Music Notation,” A free western music no-
terface, we also prototyped our system using a three-button
tation package written in Common Lisp, The Stanford
mouse for input. We found that although some aspects of
University Center for Computer Research in Music and
the mouse-based interface proved beneﬁcial (e.g., the user’s
Acoustics (CCRMA), http://ccrma-www.stanford.edu/.
hand does not occlude the display), users considered most
gestural interactions to be more difﬁcult. Users had particu-
6. Finale,
Coda
Music
Technology,
lar difﬁculty drawing gestures with the mouse that involved
http://www.codamusic.com/.
curved lines, especially handwriting and lassoing. Despite
these difﬁculties, users found that the character of pencil and
7. The In-Cube User Guide, available from Command
paper sketching was still preserved.
Corporation, Inc., Atlanta, GA, 1997.
FUTURE WORK
8. Johnston, M., Cohen, P. R., McGee D., Oviatt, S. L.,
There are many areas of future work for the Music Notepad:
Pittman, J. A., Smith I., “Uniﬁcation-based Multimodal
Integration”, 35th Annual Meeting of the Assoc. for
approaches to learning the gestural interface
¯
Computational Linguistics and 8th Conference of the
accounting for many different styles of drawn gestures
¯
European Chapter of the Assoc. for Computational Lin-
more extensive use of natural speech
¯
guistics, Madrid, Spain, July 7-12, 1997.
apply framework to other 2D applications
¯
incorporate MIDI keyboard interface, voice input
¯
9. Kurtenbach, G. and Buxton, W. “User learning and
greater music functionality in order to perform user studies
¯
performance with marking menus,” In Proceedings of
supporting multiple voices per staff
ACM CHI ’94 Conference on Human Factors in Com-
visual management of score
puting Systems, pp. 258-264, 1994.
better playback that incorporates notation / dynamics
user studies
10. MacKenzie, S., Sellen, A., and Buxton, W., “A com-
¯
comparison with traditional GUI (e.g., paper and pen-
parison of input devices in elemental pointing and drag-
cil and Finale)
ging tasks,” ACM SGICHI - Human Factors in Comput-
apply to speciﬁc area of music: e.g., jazz
ing Systems, pp. 161-166, 1991.
¯
CONCLUSIONS
11. Mackinlay, J. D., Robertson, G. G., Card, S. K., “The
The Music Notepad demonstrates a paper and pencil look-
Perspective Wall: Detail and Context Smoothly Inte-
and-feel to a powerful computer music notation engine. Thus
grated,” In Proceedings of ACM CHI’91 Conference
users can informally jot down music as well as edit, profes-
on Human Factors in Computing Systems, pp. 173-179,
sionally format, and synthesize the music. Although the sys-
1991.
tem is still incomplete, it has beneﬁted from multiple stages
of formative evaluation and development. Musicians and
12. Maxwell, J. T. and Ornstein, S. M., “Mockingbird: A
composers provided feedback that was used to redesign our
Composer’s Amanuensis,” Available as technical report
gesture recognition algorithms and to improve access to the
from Xerox Corporation, January 1983.
available functionality.
13. McGrain, M., “Music Notation,” Berklee Press Publi-
ACKNOWLEDGMENTS
cations, Milwaukee, 1986.
This work is supported in part by the NSF Graphics and Visu-
alization Center, Advanced Networks and Services, Alias/Wavefront,14. Personal conversation with Keith Hamel, Opus1 Music
Autodesk, Microsoft, Sun Microsystems, and TACO.
Software, http://debussy.music.ubc.ca/ opus1/.

15. Rekimoto, J., “Pick-And-Drop: A Direct Manipula-
tion Technique for Multiple Computer Environment,”
In Proceedings of UIST ’97, pp. 31-39, October, 1997.
16. Rubine, D., “Specifying Gestures by Example” In Pro-
ceedings of ACM SIGGRAPH ’91, pp. 329-337, July,
1991.
17. Rubine, D., The Automatic Recognition of Gestures,
PhD Thesis, School of Computer Science, Carnegie
Mellon University, December, 1991.
18. M.T. Vo and C. Wood, “Building an Application
Framework for Speech and Pen Input Integration
in Multimodal Learning Interfaces,” In Proceedings
ICASSP’96, Atlanta, GA, May 1996.

Legend:
Example Gesture:
drawn by computer
dot signifies
drawn by user
start of gesture
=>
=>
delete note with squiggle
create new note
=>
create new note with sharp
=>
delete note with lasso
delete note with lasso &
=>
create new note with flat
=>
squiggle
create new note with
=>
delete group of beamed
=>
low volume
notes
create new note with
=>
medium volume
delete single notes from a
=>
group of beamed notes
=>
create new note with loud
volume
delete beam over group
=>
=>
create new whole note
of notes
=>
halve note duration
=>
create new whole note
=>
double note duration
=>
create new half note
=>
beam all notes, change
create new quarter note
=>
note durations, match
beam slope to gesture
=>
create new eighth note
=>
halve beamed notes
duration
=>
create new sixteenth note
double beamed note
=>
create new
=>
thirtysecond note
duration
=>
create new whole rest
piano
name of instrument for
=>
create new half rest
staff
=>
create new quarter rest
=>
create new eighth rest
piano
oboe
writing "oboe" on staff
=>
create new
=>
changes the instrument
sixteenth rest
create new
=>
thirtysecond rest
Figure 3: A taxonomy of marking gestures and their associated operations.