Free-form document annotation is a crucial part of every
knowledgeable worker’s life. Despite the exponential improvement in computer
performance, when it comes to reading and annotating documents people still
turn to pen and paper. This is reasonable, as pen and paper offer many
advantages. One key advantage is the ease with which the reader may sketch
unstructured notes and drawings in response to document content.
There are definite advantages to emulating this annotation
ability on a computer. While real ink annotations often end up in the recycle
bin, digital annotations can persist throughout the lifetime of a document.
They can be filtered and organized, and like digital documents, they can easily
Now that email and the World Wide Web are well established,
the number of digital documents people interact with on a daily basis has
increased dramatically. Unlike their paper counterparts, these documents are
read in many different formats and they are displayed on diverse devices and in
different-sized windows. They may also be edited, included in other documents,
or they may even dynamically adapt their contents. All of this means that any
given document may reflow to many different layouts throughout its lifetime.
The lack of a permanent layout poses a unique challenge in
the adaptation of freeform pen-and-paper annotation to the digital domain. Each
time the content of a digital document reflows to a new layout, any digital ink
annotations must also reflow to keep up with it.
This represents a significant technological challenge. In
order to meet it, we must follow three broad steps.
First, as a user is marking up a document, we must group and
classify their ink strokes according to rough annotation categories (e.g.
underline, circle, connector, margin comment, etc.).
Second, we must anchor each annotation to its surrounding
context in the document. And third, when the layout of the underlying document
changes, we must transform each annotation to agree with the new layout of its
This third and final step is the primary focus of this
paper. We have implemented an initial, straightforward approach to the problem
of reflowing ink annotations and there is much work left to do in refining it
and developing it into a working solution. Before we develop our approach
further, however, there are significant empirical questions we must answer in
order to guide our future research.
For instance, what do people expect to happen to their annotations
when the underlying document reflows? Does our initial approach achieve the
most basic requirement of reflowable annotations, to preserve each annotation’s
contextual meaning? And do users prefer their own original ink when viewing
their own annotations or are more formalized versions (which are
technologically easier to reflow) acceptable? Most people are not familiar with
experience of having their ink reflow and so their reactions are largely
Many groups have addressed handwriting and diagram
recognition issues, some have looked at annotation anchoring and some have even
looked at modifications to existing ink (for instance “prettying” handwriting),
but none to our knowledge has addressed the issue of how users react when their
free-form digital ink annotations are automatically reflowed.