ThemesArtistWritingVideoContactOld Format

The Earthscore Notational System

for Orchestrating Perceptual Consensus about the Natural World

After a study of modern painters, the biologist C. H. Waddington articulated the following argument (Waddington: 1970). As a species we transmit information over generations both genetically and through speech and writing. Speech and writing inevitably result in authority structures, someone telling someone else what to do. The child is told, “No, don't touch, the oven is hot.” His or her perceptual system is stunted and his or her behavior is linked up to the language commands of others. Based on his examination of how modern painters had learned to see nature without language, Waddington suggested institutionalizing this artistic achievement for the human species as a whole. He thought we could generalize the silent success of painters such as Monet, Cezanne, and van Gogh and evolve an information transmission system based on shared perception of environmental realities rather than language.

This paper presents the Earthscore Notational System as a formal framework for evolving a shared perception of the natural world along the lines posited by Waddington. The Earthscore Notational System grew out of my efforts to use the video medium to interpret nature. I began working on the system in 1971 while living in the Hudson River Valley and trying to interpret the natural world as a video artist. It did not take long for me to realize that no matter how good I became at producing landscape video as an individual artist, it would have little effect on how people actually treated the ecology of the Hudson Valley. To make a difference, what was really needed was a cooperative group of videographers who could interpret the natural world and present it to the community at large on an ongoing basis. In order to produce such an orchestration of perception, a notational system was needed. No such notational system existed. I was thus faced with the necessity of inventing the sort of information transmission system based on perception that Waddington had proposed.

The difficulty of inventing a video notational system suitable for the natural world becomes evident when we compare the two activities of recording nature with video and playing music on a piano. Video is a perceptual device with which we look at the natural world. The natural world can be a buzzing, blooming confusion. We have never codified a clear system of “notes” for reading nature. By contrast, there is a clear system of musical notes encoded in the piano. In fact, the piano was constructed according to specifications determined by these notes. We do not know the “notes” according to which nature was constructed. A notational system designed to interpret the natural world must somehow be based on clear “notes” elicited from the natural world. For example, in order for videographers to record salmon spawning in a way that is faithful to the spawning process itself, they must understand what I will call the “figures of regulation” guiding the “performance” of the salmon. Ecological videographers must know how to read these underlying figures of regulation, or notes in nature, just as dance videographers must know the choreography of the dance they are recording. Once the underlying figures of regulation for salmon spawning in a particular river are identified and put together, i.e. composed into a score, then videographers who know the notational system and that particular score can record and monitor the salmon run year after year, generation after generation. If a particular performance of the salmon as recorded does not comply with the score, then the videographers are in a position to scan the ecological system for perturbations and alert us that something might be disturbing the underlying figures of regulation for the spawning run. This should result in a revision of the score and/or a correction of some human activity that is ecologically destructive to the salmon run.

The five components of what I consider a suitable system of notation for evolving shared perception of the natural world over generations, i.e., the five components of the Earthscore Notational System are:

The categories of firstness, secondness, and thirdness
The relational circuit
The firstness of thirdness
A semiotic system for interpreting the firstness of thirdness

As of April 1991, these five components have not yet been put together in an organized way to create an ongoing perceptual information system. In the space allowed, I will provide a description of each component and cite artwork, workshops, and projects which I have been involved in that make use of these components.

back to top

The Categories of Firstness, Secondness, and Thirdness

 Because I wanted a notational system for video that was responsive to the totality of the environment, I was attracted by the comprehensiveness of the categories of firstness, secondness, and thirdness as developed by the American philosopher Charles Peirce (1839-1914). Following Kant, Peirce subscribed to the architectonic theory of philosophy (Apel: 1981). By architectonic, he meant the art of constructing systems, i.e., uniting manifold ways of knowing under one idea. The idea or concept of a formal whole determines a priori both the scope of the manifold content and the positions that the parts occupy relative to each other. This unity makes it possible to determine, from our knowledge of some parts, what other parts are missing, and to prevent arbitrary additions. Knowledge can grow organically, like the body of an animal.

For Peirce, knowledge corresponds to three modes of being: firstness or positive quality, secondness or actual fact, and thirdness or laws that will govern facts in the future. Peirce held that these categories of being are phenomenologically evident to anyone who pays attention to what happens in the mind. Direct observation will produce these categories of knowledge.

Firstness is positive quality. The taste of banana, warmth, redness, feeling gloomy: these are examples of firstness. Firstness is the realm of spontaneity, freshness, possibility, and freedom. Firstness is being “as is” without regard for any other.

Secondness is a two-sided consciousness of effort and resistance engendered by being up against brute facts. The “facticity” or “thisness” of something, as it exists, here and now, without rhyme or reason constitutes secondness. To convey the pure actuality of secondness, Peirce often used the example of pushing against an unlocked door and meeting silent, unseen resistance.

Thirdness mediates between secondness and firstness, between fact and possibility. Thirdness is the realm of habit, of laws that will govern facts in the future. With a knowledge of thirdness we can predict how certain future events will turn out. It is an 'if...then' sort of knowledge. Thirdness consists in the reality that future facts of secondness will conform to general laws.

When we attempt to interpret a natural site with a video camera, we are confronted with “everything.” We need to make selections. If those selections are arbitrary, the final tape can leave out significant aspects of the ecosystem. Significant omissions can make the interpretation of the site faulty. Peirce's categories of firstness, secondness, and thirdness are, in effect, a theory of everything. Using these comprehensive categories, it is possible to make selections that are responsible to “everything” at the site. The way in which Peirce's categories can be used to organize video perception of ecological sites is evident in my videotape titled Nature in New York City (Ryan: 1989a). Consider the following list of the four sites in the tape and how my interpretation of the sites was guided by using Peirce's categories.

1. Horseshoe crabs laying eggs, Jamaica Bay, Gateway National Recreation Area, Brooklyn and Queens. Firstness: eggs, signifying the possibility of new crabs; secondness: predator birds and meddlesome boys; thirdness: pattern of crabs mating and context of urban habitat.
2. Clay Pit Pond, Clay Pit Pond State Preserve, Staten Island. (Five phenomena selected: deciduous trees, evergreen trees, abandoned cars, grass, reeds.) Firstness: quality of five phenomena plus pond surface; secondness: facticity of five selected phenomena; thirdness: patterns of phenomena in the context of the pond.
3. Stand of trees, forest in Inwood Hill Park, Manhattan. Firstness: melting snow, bark surfaces and sprigs of green; secondness: burnt wood and litter; thirdness: children swinging on rope, pattern of tree crowns.
4. Waterfall, Bronx River, New York Botanical Garden, The Bronx. Firstness: quality of surface water and texture of turbulence; secondness; water turbulence; thirdness: explicit water patterns, geological context of the falls.

This twenty-seven-minute tape was edited in six-second passages set up in 4/4 time for musical interpretation. Each passage corresponds to firstness (F), secondness (S), or thirdness (T) and the passages fade into each other. A given sequence might run FSFT, SFST, TSFS, TFSF.

back to top

The Relational Circuit

 The Nature in New York City tape was composed using what I call “the relational circuit.” The relational circuit organizes the categories of firstness, secondness, and thirdness in unambiguous, relative positions. The circuit is to the Earthscore System what the staff and bars are to classical music notation. I originated the relational circuit based on my own video experimentation and a study of Peirce's failed attempt to develop a logic of relationships.

Peirce thought that the organization of knowledge according to his three fundamental categories required a new kind of formal logic. During his lifetime he made four major attempts to construct a philosophic system, each attempt guided by new discoveries he had made in logic, but none succeeded.

The major difficulty Peirce had was with continuity. Peirce believed all things were continuous and that the concept of the continuum was the master key to philosophy. However, he was never able to organize his categories in a logical continuum (Murphey: 1961).

Working with video, I was able to construct a topological continuum that, I believe, supplies the formal logic necessary to realize Peirce's architectural dream. My purpose here is to present the circuit for the reader's inspection and show how it organizes Peirce's three categories.

The relational circuit is a self-penetrating, tubular continuum with six unambiguous positions. The circuit below is depicted in three dimensions. There is a part contained by two parts, the position of firstness. There is a part contained by another part and containing a part, the position of secondness. There is a part that contains two parts, the position of thirdness. The circuit organizes differences in terms of these three positions and the three “in-between” positions that connect them in the continuum. Peirce's three categories map onto this continuum. The positions are named to correspond to the categories.

Relational Circuit – Three Dimensional Illustration

back to top

The relational circuit provides the core of a notational system which can regulate composing with firstness, secondness, and thirdness in a way that is analogous to the way the painter Cezanne composed with what he fondly called his little blues, little browns, and little whites (Lacan 1978: 110). Nature in New York City is an example of such a composition. However, since this tape was produced by me as a solo artist, it falls short of the ideal in which cooperating videographers interpret the ecology. Videographers can establish cooperation using the relational circuit through a process I call “Threeing.”

back to top


Before beginning work on a video notational system for nature, I was part of a video production team in New York City called Raindance. Raindance often created shared video perception of events spontaneously, without script, formula, fixed roles, or hierarchies. A limited number of portable cameras would be passed around based on affinities, shared sensibilities, and whatever was happening in range of the cameras. I was involved in making many videotapes in this manner. Whether it was the first Earth Day in New York City, a media conference at Goddard College, or a day on a California beach, recording and replaying with video involved spontaneous cooperation within a small group (Raindance: 1969—71).

Threeing is a formal, teachable version of the kind of cooperation that happened spontaneously within Raindance. In the summer of 1989, I taught a dozen young videographers in New York City to observe sites in Central Park using the protocols of Threeing. The youngsters went on to incorporate this cooperative way of observing sites into a series of tapes. One of these tapes won a national prize from the American Film Institute (Urban Conservation Corps: 1989). In the spring of 1990, I taught the three categories of the Earthscore Method to adult videographers on Staten Island. Together with sound artist Charles Potter and poet Alan Ginsberg, we then produced a videotape based on Walt Whitman’s poem “Crossing Brooklyn Ferry” (Ryan et al: 1990b).

Unlike the tradition of filmmaking, in which each member of a production team has a fixed roles, such as director, camera person, or sound person, Threeing enables each member of a video team to play each of three critical roles. The roles correspond to the categories. The videographers take turns paying attention to a phenomenon in terms of firstness, secondness, and thirdness. As an example, if a three-member team—say, yellow, red, and blue— had produced the video interpretation of Clay Pit Pond mentioned above, it would have worked in the following way. Yellow might do a six-second shot of the texture of bark on the evergreen (firstness), red might do a shot of the trunk of the same evergreen rising out of its ground site (secondness), blue might do a shot of the whole tree with the pond in the background (thirdness). When the team approached a deciduous tree, red would do firstness, blue secondness, and yellow thirdness. Another phenomenon, such as an abandoned car left in the pond, would find blue doing firstness, yellow secondness, and red thirdness. To augment this turn taking, there are protocols for decision making in teams of three. These protocols are too complex to specify here. Suffice it to say that videographers who practice Threeing would be capable of orchestrating their perception of the natural world without fixed roles and without a dependence on language.

In addition to incorporating Threeing into the Earthscore Notational System, I have been able to detach threeing from that system and develop it as an “art of behavior” in its own right. As an art of behavior, Threeing is a way in which three people can relate to each other simultaneously without words and without cameras. Threeing works for three people analogous to how T'ai Chi or yoga works for the individual. T'ai Chi and yoga balance a person's well being with a system of changing postures. Threeing is a practice that balances the relationships among three people with a system of changing positions. (See Threeing under Earthscore for artists on Earthscore site.) Just as the practice of T'ai Chi or yoga can prevent certain health problems such as lower back pain, so the practice of Threeing can prevent certain problems in human relationships that bedevil consensus building. In the essay above titled “Relationships,” I discuss at length how Threeing works to preclude specific difficulties in human relationship, including the tendency of two parties to split apart and third party exclusions.

To date I have had three opportunities to develop threeing as an art of behavior. In 1984, I produced a videotape titled The Ritual of Triadic Relationships (Ryan: 1984b) for the Primitivism Show at the Museum of Modern Art. With a commission from V’soske, a custom rug making house in New York City, I have designed a rug for Threeing in collaboration with artist Michael Kalil. The rug, which codes the positions in terms of textures and graded rug pilings, is now in production. As a nominee for a Rockefeller Intercultural Arts Fellowship, I conceptualized a Tricultural Tournament for performing couples from three different cultures who would come together to “invent” cooperative intercultural behavior. In the rules of The Tricultural Tournament only an intercultural team of three can win. Viewers would identify with an intercultural team competing to create cooperation rather than a team representing their own culture against another culture (pp. 434ff.). Ideally, the tournament would stimulate intercultural teams of three to build a perceptual consensus about the natural world using the Earthscore Notational System.

back to top

The Firstness of Thirdness

Video recording and playback, with its possibilities of time lapse and slow motion, enables us to understand natural patterns in a non-verbal way. Think of time lapse film studies of budding flowers and slow motion studies of insects. Watching these moving images, it is possible to understand the pattern presented in a single gestalt without rational inference using language. The moving image allows the natural event to occur in the mind like a fist in the hand. There is a spontaneous, intuitive appreciation of a pattern in nature. Peirce would call this “the firstness of thirdness.” This intuitive appreciation of natural patterns through perception is the fourth component of the Earthscore Notational System. It is important to understand how the firstness of thirdness relates to the categories of firstness, secondness and thirdness.

In Peirce's categories, firstness is not separated from secondness, nor is firstness separated from thirdness. There is a firstness of secondness. The 'ouch' sounded by someone struck with a thrown rock is an instance of the firstness of secondness. The brute fact of the rock hitting the person is actually there, secondness. It is not constructed or determined by the person's feelings alone. Yet for the person a feeling attaches to the brute fact, a feeling evident in the involuntary cry.

Peirce provided as well for the firstness of thirdness, that is, the immediate perceptibility of law. Muybridge's famous photos of a running horse, done on a wager about whether the four hooves were ever all off the ground at the same time, is an instance of such firstness of thirdness. The firstness of thirdness in nature can also be understood in a formal way using the catastrophe theory of the topologist, Rene Thom (1975). Catastrophe theory is a qualitative method for modeling discontinuous phenomena. The theory models the states of nature as smooth surfaces of equilibrium. When the equilibrium is broken, catastrophe or discontinuity occurs. Thom has proven that in natural phenomena controlled by no more than four dimensions, there are only seven possible equilibrium surfaces, hence only seven possible discontinuous breaks, i.e., only seven elementary catastrophes. Thom named these seven as follows: fold, cusp, swallowtail, butterfly, hyperbolic umbilic, eliptic umbilic, and parabolic umbilic.

Catastrophe theory is to the medium of video what Euclidian geometry is to the medium of paper. Television and video monitor and record events (Cavell: 1982). Just as Euclidean geometry offers a formal understanding of geometric surfaces and solid objects, catastrophe theory provides a formal understanding of events or changes from states of equilibrium, i.e, discontinuous phenomena. Based on Euclidean Geometry, someone faced with tiling a wall knows with mathematical certitude that of all possible regular polygons ( equal-sided, two dimensional shapes) only three (hexagon, square, triangle) can fill the plane packed edge to edge. Based on catastrophe theory, someone observing nature with a video camera knows with mathematical certitude that there are only seven kinds of discontinuity possible in any natural phenomena controlled by four dimensions or less. Just as the continuous relational circuit constitutes the “staff” of the Earthscore Notational System, so these seven elementary models of discontinuity constitute the basic “notes” of the system.
To suggest how these notes function in the Earthscore Notational System, I ask the reader to imagine a section of a stream in which there is a continuous flow of smooth water. The flow of water has four dimensions: length, width, depth, and rate of flow. Changes in these dimensions occur because of changes in the shape of the streambed and variations in the amount of rainfall. Catastrophe theory can model how changes in these dimensions control changes in the way the water behaves. The models provide both a control surface for the changing dimensions and a behavioral surface for the discontinuous action of the water itself. For example, if the width of the streambed begins to narrow very gradually, suddenly a fold will appear in the water's shape. If both the rate of flow and the depth of the stream increases the water may jump into the air as if jumping over a cusp. If a twig catches the water as it comes down, you may get a droplet forming at the end of the twig before it falls to the next surface. In catastrophe theory such periodic droplet formation in-between surfaces would map on the butterfly model. The butterfly is a like a cusp except it has another surface half way between the upper and lower surfaces, a pocket, on which the droplet could form. The swallowtail and the three umbilical models function in a similar manner. Whatever way the four controlling dimensions change, there are only seven possible surfaces on which the corresponding changes in the behavior of the water can be mapped, only seven basic “figures of regulation” for the water's behavior. I should note in passing another way of modeling water flow which has developed recently called chaos theory (Gleick: 1987). Chaos theory is particularly useful in approaching turbulence, a domain in which catastrophe theory has not yet been very helpful. To my knowledge, the formal interrelationship of these two modeling systems has yet to be worked out, but in principle both could be integrated into the Earthscore Notational System.

In nature, the combinations of the basic seven catastrophes are multiple and not readily apparent. Yet the underlying structural stability of discontinuous phenomena in nature can be understood by careful observation. Each “event pattern” can be understood in terms of its 'chreod'. Chreod is a term taken from the Greek that means “necessary path:” “chre” meaning “necessary,” and “ode” meaning “path.” If any natural process is disturbed it will return to the pathway necessary for its structural stability, like a flooded river returns to its riverbed. These necessary pathways of nature, or chreods, can be rigorously modeled using the seven elementary catastrophes and variations on these seven (Casti 1988: 149ff.).

In my own work as a video artist, I have repeatedly returned to moving water as the richest single source for developing a vocabulary of “chreods” in nature. Water takes so many different shapes such as billows, droplets, back curls, waves, fantails, and cascades. Each of these shapes exhibits a different pathway in which water can flow, a different chreod. In 1975, I spent the year recording over thirty—five chreods on videotape at the waterfall in High Falls, New York.
 In 1983, I did a study of the Great Falls in Paterson which I edited into a tape with five sets of seven different kinds of chreods. In 1984, I did a study of the coast of Cape Ann above Boston. In 1986, I crossed the Atlantic Ocean on a sixty-foot North Sea Trawler and videotaped over thirty hours of ocean waters. Currently, I am working on a video interpretation of nine different water ecologies in the Shawangunk Mountains at the edge of the Hudson Valley.
Building up a vocabulary of chreods can give us an articulate set of notes with which to score natural phenomena. Horseshoe crabs laying their eggs in Jamaica Bay is a natural process regulated by a chreod. The crabs only lay their eggs in the wet sand during the ebb tides created by the full moon in June. This assures maximum protection for the eggs from predator birds and land animals. The birthing activity takes place within a necessary figure of regulation. If you destroy that figure of regulation, that chreod— by stripping the beach of sand, for example— you have destroyed the natural process of birthing in that site.

To sum up this section on the firstness of thirdness, I am saying that the difficulty of discovering clear “notes” in the buzzing, blooming confusion of nature can be resolved with systematic observation of an ecology by video teams trained in Threeing and schooled to identify the chreods of an ecosystem. The systematic observation of “everything” would insure that we did not miss anything significant. By identifying the chreods we can rigorously model the underlying structural stability of the various events in the ecosystem. We can then find out, through more observation and study, how these various chreods relate to each other. The syntax of interrelationships between these chreods would, in effect, constitute the “score” for the ensemble of recurring events that constitute that particular ecosystem. We would be eliciting the score from the ecosystem itself by careful observation. Once we know the score we can observe and monitor how the ecosystem actually performs or fails to perform in compliance with that score. Failure to comply would mean that we need to reinterpret our score and/or to correct any behavior of ours that is making the ecosystem incapable of performing according to its natural score.

back to top

A Semiotic System for Interpreting the Firstness of Thirdness

 The advantage of identifying the observation-based score of an ecosystem as the firstness of its thirdness is that we can then connect that score to Peirce's entire semiotic system. Semiotics approaches knowledge as a process of generating signs. Peirce's semiotics encompasses both perceptual and linguistic signs. Any kind of local knowledge, any art form and any scientific discipline can be incorporated into Peirce's semiotic system. The system is too complex to present here. In brief, it can be pointed out that for Peirce, semiotics, or the understanding of signs, is consistent with his categories. A sign (firstness) represents an object (secondness) for an interpretant (thirdness). This threefold division exfoliates into a sixty-six-fold classification of signs that is inclusive of everything from a smudge of paint to a syllogism. With Peirce's approach, it is possible to systematize both interdisciplinary and multimedia representations of ecosystems. In the spring of 1989, I organized a pilot project in the Black Rock Forest above New York City with fourteen eighth grade students from the Dalton School. The task was to interpret changes in a steam from winter to spring. In six hours I taught the students how to think in firstness, secondness, and thirdness. I then organized them into a video team, a word team, an image team and a number team. Each team had three members and observed the stream at four different sites. Only the video team used video. The others represented the stream in language, images and numbers. The interpretation of stream changes, which the students presented to teachers and fellow students, was very successful and prompted me to conceptualize a K-12 environmental curriculum based on the Earthscore System (pp. 363ff.).

The most ambitious strategy I have for implementing the Earthscore Notational System is a global network of regional television stations, each responsible for monitoring local ecologies. My design for these television stations is based on a cybernetic adaptation of Peirce's entire phenomenological and semiotic system (pp. 255ff.). I have presented my “Ecochannel Design” at the Museum of Modern Art, the first International EcoCity Conference, and the World Congress of Local Governments for a Sustainable Future at the United Nations. As part of an environmental coalition of over 250 groups in New York City, I have proposed how such a channel could be implemented in an urban setting (pp. 350ff.). In effect, the television ecochannel design is an extension of the Earthscore Notational System. Humans and ecosystem are considered part of one interrelated circuit. The programming offered by the station is organized according to this circuit. Briefly, the circuit can be articulated as follows: differences in the ecosystem make differences in how the ecosystem is represented on television, which make differences in the actual interpretations of the ecosystem by specific people, which make differences in how the community as a whole interprets the ecosystem. This interpretation, in turn, makes differences in how the community behaves toward the ecosystem. By following this circuit, a community can identify and eliminate errors in its relationship to the ecologies that support its life. By linking regional television systems operating according to this circuit, humans could establish a global television network grounded in the perception of ecosystems. The easiest way to build such a global television network would be to integrate Earthscore Notation into the Earth Observation System currently being developed as part of NASA's Mission to Planet Earth (Ryan: 1991c).

back to top


Linguist Derek Bickerton reasons that despite the vast powers that language has conferred on our species, some of the consequences of modeling reality with language threaten the continuation of our life on earth. Language can create dysfunctional representations of reality, representations that result in antibiological conduct such as a heretic who refuses to recant and is burned to death. In a sense, humans have become a heretical species. Orthodoxy upon this earth holds that any species, which destroys its environment, destroys itself. Humans are destroying their environment, hence destroying themselves. Humans are a heretical species. Bickerton ascribes this antibiological behavior to our capacity to misrepresent reality with language. He states, rather abruptly, “Perhaps language is, after all, terminally dysfunctional” (Bickerton 1990: 253).

Without necessarily agreeing with Bickerton's worst-case scenario, I do doubt that our species will ever talk itself out of the trouble we're in. On the other hand, building a system of transmitting information grounded in the perception of environmental realities would help correct some of the dysfunctionality of our language-driven species. Hopefully, by referencing a common perception, created according to the Earthscore Notational System, we could negotiate an operating consensus about living life on earth in realistic ways that are not destructive.

back to top

    © 2006
    Paul Ryan
    all rights reserved