Smoking Yahoo!'s Pipes

By Annalee Newitz, AlterNet. Posted April 3, 2007.

I've been playing around with Yahoo!'s latest technological experiment on the Web. It's called Pipes, and it's a system designed to help Web-savvy people write simple programs without ever having to read a book about Java.

If you visit pipes.yahoo.com, you can take a peek. Visitors to the site are presented with a sheet of virtual graph paper and a list of modules that you can drag onto the paper and connect with pipes. In this early stage, the modules mostly allow users to build a really customized news feed or online research tool. You can tell a source module to pull information from, say, a Google search for "Windows Vista" or the RSS feed of your favorite newspaper.

Then you pipe that information to an operator module, which allows you to filter it, list it by date, translate it into another language, and more. Other modules let you do more complicated things, such as annotating each piece of data with geographical information or merging the RSS feeds from several sites so that you get one big daily news feed instead of 20 from various progressive blogs.

Just think: you could mix the latest wankery from porno news site Fleshbot with the latest wonkery from Talking Points Memo! That's the beauty of a customized news feed. Pipes isn't for everyone -- it's too complicated for casual Web surfers, who may not be familiar with the inner workings of RSS feeds and search queries.

But a quick Google search reveals some excellent tutorials that will aid even the most RSS-clueless person in creating a pipe. Plus, you can clone other people's pipes -- so if you want a customized news feed, you can just use one that already exists, fill in your own news sources of choice, and save it to your own account.

There are hundreds of cool pipes available on the site, and they're all cloneable. Now I sound like a cheerleader for Pipes, which I'm not. In fact, I recently spent an evening making fun of Pipes with one of the creators of the RSS standard (no, it wasn't Dave Winer). Our mockery was inspired by two things: one, Pipes could be an overhyped proof of concept that nobody will ever use; and two, it could actually limit people's control over data.

How could a tool designed to help you manipulate all kinds of information actually limit your control? To answer this, we need to delve briefly into the origin of the pipes idea. The name comes from a powerful command in UNIX, one of the first operating systems, which converts the output of one function into the input for another.

It's hard to convey how utterly awesome and time-saving this command was when it was invented. It meant that data could be crunched, sorted, alphabetized, merged, and recombined more easily than ever before. Yahoo! Pipes aims to do the same thing, only the data you use is what's publicly available on the Web.

So if you want to use Pipes to organize or sort your personal data, you'll have to publish it online. This is obviously quite different from the UNIX pipe, which is so powerful in part because you can use it on private stuff such as passwords and financial documents. Yahoo! Pipes treats the Web as if it were the hard drive of your UNIX box -- you can pipe data from Google into a sorting program or pipe the New York Times RSS feed into a filter that will remove all stories that refer to Yahoo! Pipes.

It's marvelously cool, but I worry that it will inspire people to put sensitive data online just because it's more convenient to crunch via Pipes. At this point, my fears are probably unjustified. Pipes is in beta, and it may not catch on with the general public.

More likely, a user-friendly version of Pipes will come along and get widely adopted in a couple years. It will become just one more way we're being seduced into dumping all our personal stuff online. I like the idea of turning all the data on the Web into my raw material, to do with what I please. That's the beautiful part of Pipes. Still, the more data we deposit in the hive-mind of the Web, the less power we have over it.

See more stories tagged with: pipes, yahoo!

Annalee Newitz (annalee@techsploitation.com) is a surly media nerd who still hears the voice of her UNIX teacher in her head saying, "Now pipe it to MORE."

Liked this story? Get top stories in your inbox each week from AlterNet! Sign up now »