Friday, January 23, 2009

Art Genomes?

Structuralist and post-deconstructionist and neo-post-pseudo-intellectualist-structuralites have been advancing the idea that we can classify and thus, to some extent, interpret or evaluate works of art by their semiotics. Briefly, this means we think of these art works as systems of signs that stand for things, and then consider the way the signs are configured within a work of art, and how that might reflect something about the things the signs stand for. Language is a semiotic system ... we use words to stand for things, and in writing, the combinations and sequences of words says something about the subject we're writing about. At least, we hope it does. Much of the 20th century philosophizing about this dealt with movies or, as it's more pompously known, the art of cinema.

That's all well and good, but now there's a very practical problem with is leading to a new way of thinking about this. Basically, the problem is that you've been listening to music through Pandora, or renting movies via Netflix, and you'd like to know what other music or movies would appeal to you.

These online content sources have tons of movies and music, but the problem is how to classify it so that you can identify which ones might appeal to audience members with certain tastes. Netflix, for example, asks you to rate each movie you rent, and over time, it builds up a database of what movies you've liked.

But there's still the similarity problem: If your favorite movies were Ishtar and Gigli, how to you decide which other movies are similar?

One promising answer is the movie genome. Basically, the idea is to identify a slew of properties that a movie can have, like plot, cast, awards, box office success, etc. Comparisons of movies on the basis of their genomes are likely to good matches and non-matches.

Now, you might think "Isn't this just a fancy way of comparing all these properties? Calling them a genome doesn't really change anything."

The answer is "Well, yes." It is just a fancy way of comparing these things. But treating all these properties as genes accomplishes two things:
  1. It makes the classification of movies more systematic, and
  2. It makes it possible to use some elaborate algorithms for doing the comparisons, finding near-matches, etc.
Obviously this genome idea could also be applied to books, plays, paintings, etc. It may lead to some new ideas about how to classify and interpret works in these art forms.

Or not.

Wednesday, January 7, 2009

Visual Language

Jacques Bertin was a pioneer of visual language with his book, Semiology of Graphics. Bertin, a map maker, set forth a system for using and interpreting signs and symbols in various forms of graphic presentation. His concern was primarily with printed graphics.

Jock Mackinlay proposed a practical implementation of some of Bertin's work in his 1986 paper, Automating the Design of Graphical Presentations of Relational Information. Many others have pushed the area of technology called information visualization forward, notably Edward Tufte in his books and lectures.

In an age of virtual reality, though, the types of graphics that can be presented are as rich and varied as reality itself, and even beyond, and the interpretation just as subtle and ambiguous. Experiencing virtual reality (VR) graphics is, in some ways, like watching a movie, and perhaps the best guides to interpretation are those on the language of cinema. In particular, James Monaco's classic How to Read a Film comes to mind, along with Jennifer van Sijll's Cinematic Storytelling.

One factor is that electronic graphics displays can show animation and other time-based effects. This involves a different set of conventions from printed graphics. For example, in a comic strip, adjacent panels

indicate a sequence of events, separated in time.

However, in a movie, a split-screen like this,

usually represents simultaneous action. The actions in the two portions of the screen are going on concurrently.

That's just a simple example, but it illustrates how our interpretation of very similar visual effects depends on the context they appear in. We'll talk about that more later.

Sunday, January 4, 2009

Signals and Samples

The area of technology that seems most closely related to art is called signal processing. Signal processing is really a collection of techniques and concepts that are used for manipulating pictures, sounds, TV, movies, and other kinds of electronic information. We don't need to know all the technical stuff, but there are two ideas that are useful for understanding digital art media. They're also kind of cool ideas that can be understood visually.


The first, and biggest, idea in this area is signals themselves. What are they? Why should we care?

For our purpose, we can say a signal is a sequence of numbers. There are other kinds of signals that don't quite fall into this definition, but the ones we're thinking about with digital art media are sequences of numbers.

As one simple example, think about the time shown on a clock, compared with actual time that has passed. For example, since the start of this year, 2009, clocks in the U.S., that measure time in 12 hour cycles, produce a signal like this:

The green line shows what time is displayed on the clock. One hour after midnight, it shows 1:00, then 2:00, 3:00, and so on. When the clock display reaches 12:00, we switch from AM to PM (or vice versa) and start counting again. So even though time itself just keeps moving forward, hour after hour and day after day, the time shown by the clock repeats the cycle from 12:00 to 12:00.

I cheated a little bit here, because the green line doesn't look like a series of numbers, which is what we said a signal is. If you think about a digital clock, the time displayed by the clock only changes once per minute: 12:00, 12:01, 12:02, etc. So if this signal represents the time shown on a digital clock, then looking really closely at this signal, we'd see something more like this:

We get this kind of stairstep effect as the time on the clock jumps from one minute to the next. If the digital clock showed seconds, there would be 60 tiny stairsteps for each of the big ones shown here.

A lot of signals are measurements of something over time. For example, this shows the average high and low temperatures in New York City over the course of a year.

You can see that in January, both the high and low temperatures are on the cool side, but they both go up around July and August, and then down again by the end of the year. These could be considered two signals.

But not all signals compare measurements with time. I'll talk in more detail later about treating images as signals which are measurements of color compared with location ... the upper left corner is one color, the pixel next to it is another color, etc.


So if we think of a signal as a sequence of numbers, then each number is a sample. Simple, isn't it? Usually, we get the samples by measurement ... we look at the clock face, or we check the thermometer, or we scan the points in an image to measure the color at each point.

There's an old joke that a clock that's completely stopped is better than one that loses one minute every hour ... the stopped clock is right twice a day! Suppose you had a clock that stopped at 8:17, and you only looked at it at 8:17 AM and 8:17 PM every day. You'd have no way of knowing if the clock was working or not. In other words, you wouldn't be able to tell this signal

from this one

The second signal, the stopped clock, is an alias for the original signal. So the fact that, based on the samples you have, the second signal looks like the first one, is called aliasing. The way to get around this is to take more samples ... look at the clock more than twice a day! Then you'll be able to tell which of these two signals the clock is producing. This way of getting around aliasing is anti-aliasing. It's just like the jagged edges you see in low resolution images. Higher resolution images don't have the same jaggedness, because there are more samples.

There are other ways to reduce the effect of the jaggies with low resolution images, and we'll talk about that later.