Home | Blog | Brainjam | Publications | People

Flickr Fountain of Knowledge

What can we learn from Flickr? Well, for one, we have learned that there are a lot of people who like to take photographs and share them publicly. Who would have guessed! However, my question refers to a different type of knowledge: information about the world that is implicitly encoded in the activity on Flickr.

You do not need to go far to see a simple yet brilliant example of such knowledge: check out Flickr’s tag clusters (here are the clusters for love, jaguar, Taj Mahal, hack). Using tag co-occurrence on Flickr photos, Flickr’s clustering can break down a term into multiple semantics or meanings: Jaguar, for example, is the animal as well as the car and the guitar: the first co-occurs with the tags “zoo” and “cat”; the second meaning of “jaguar” appears with “car” and “auto”. Note that these meanings are not mined from any other resource: they represent some “knowledge” that is generated automatically from the implicit contributions of Flickr users uploading and tagging their photos.

In other examples, Patrick Schmitz developed a different co-occurrence model that allowed him to generate subsumption data in Flickr tags (e.g. San Francisco is subsumed by California). The work at Yahoo! Research on TagLines and at our own lab on Tag Maps had shown that Flickr community activity generates descriptive labels for events and locations.

Last week, in Amsterdam, as part of SIGIR 2007, we added yet another method of extracting knowledge from Flickr. The paper, “Towards Automatic Extraction of Event and Place Semantics from Flickr Tags”, by Tye Rattenbury, Nathan Good (two of our star interns) and myself*, begins to answer a simple question: given a tag that appears on Flickr (such as “dog”, “SIGIR 2007″, or “Yahoo! Research Berkeley”), can we automatically determine whether or not that tag refers to a specific place, and whether or not the tag refers to a specific event? As you may guess, SIGIR 2007 refers to an event, Yahoo! Research Berkeley is a place, and “dog” is neither a place not an event.

Knowing if a tag is a place or event leads to better image search, but can also help us to better visualize the Flickr data; generate automatic event and place gazetteers; associate missing time/location metadata based on tags, and more.

I will not get into the details of how we propose to do extract the place/event knowledge from Flickr; you can get these details in our paper (pdf). I will just mention that we are using the dataset of geotagged Flick photos, and looking at the time and location distributions for each individual tag in the dataset. If the location or time distribution for a tag have specific “structure” to them, we classify that tag as a place or event, accordingly.

Below, you can follow the presentation slides I gave at SIGIR, or just jump directly to the paper to get the full story.

While the debate on the “Is the semantic web is dead?” question continues, “emerging semantics” are alive and kicking. What other knowledge can be extracted from the Flickr dataset?

* “Towards” is a code word in research papers meaning “we didn’t take the research all the way quite yet but want to make the paper sound important nevertheless” - we try not use it too much.

CrowdScapes on the Go from ITP

Mr. Bukhin likes to make us think about places and images. From his adventures into memory and implicit capture at Waymarkr (with Mr. DelGaudio), Mike has been raising more question than answers in his ITP projects. That’s a good thing.

Recently, Mike created CrowdScapes, his final ITP thesis project. CrowdScapes proudly utilizes our own TagMaps data API (as well as the Flickr APIs, of course). Mike describes the project this way:

My mobile application, CrowdScapes, lets a user explore a neighborhood through the crowd’s eyes. CrowdScapes value is that it leverages the critical mass of a large community of photo takers and sharers, not just a small subset of power users using a custom application. CrowdScapes can give places a user passes through everyday but doesn’t really consider a new life and a new possibility. By letting users step outside of themselves and consider what a location means to others, CrowdScapes can give new insight into a place.

The view CrowdScapes provides moves around two core pivots. The ‘familiar view’ shows localized photographs by a participant’s most used tags, it shows the current location through the lens of the participant’s interests. The ‘strange view’ shows the most popular tags and their respective photographs as viewed by the global Flickr crowd.

The TagMaps data is what helps Mike generate the `strange view’.

You can try CrowdScapes on your phone (via WAP) at http://m.crowdscapes.com. In a couple of days, you will see something new from us that will touch on very similar concepts. Start getting excited.

Oh, and congratulation, Mike, for passing your thesis defense!

The Emerging-Semantics Web (”The Semantic Web is Dead”)

Last week, I participated in a WWW2007 panel called “Multimedia Metadata Standards in a Semantic Web 3.0“, where I took the opportunity to declare the Semantic Web dead. As you can imagine, such a declaration in front of a crowd of semantic web researchers provoked many responses. While I believe panels should be provocative and entertaining, I also have specific reasons for why I went as far as calling the Semantic Web “dead”. Let me explain what I mean.

There is no way that we can engage the masses in annotating media with “semantic” labels. At best, we can get the people to annotate content (such as Flickr images or YouTube videos) with short text descriptions or tags. This works only because tags are simple; powerful (can be used for many tasks) and, in some systems, carefully engineered to match the user’s natural motivations. Our best hope is to be able to take this bottoms-up annotation, or folksonomy if you will, and try to assign some semantics to it later - Flickr’s Clustering is a great example, as well as Y!RB’s TagMaps and our upcoming SIGIR paper (”Towards Automatic Extraction of Event and Place Semantics from Flickr Tags”, available in pdf).

Now, developers are people too(tm). When Web2.0 developers have any need for a multimedia standard, they will choose something that is simple, powerful, and answers their immediate needs. Semantic Web? Dublin Core? MPEG-7? I don’t think so. The masses of developers have opted to use simple formats such as (partial list) RSS, Microformats, and recently, Flickr machine tags.

Flickr machine tags are, in essence, a bottom-up way to assign “lightweight semantics” to Flickr tags/images. Machine tags are an easy way for developers to create their own semantics and applications around the Flickr platform by using a triple-tag such as “upcoming:event=144945″ to signify that a photo was taken in the event ID 144945 in the Upcoming.org event database. Machine tags are an “emerging format” that will adapt to the needs of developers and will reflect the salient properties of the data - just like tags. Flickr cannot, and probably would not want to regularize or limit the usage of machine tags. Issues will arise - for example, since the namespace for machine tags is not pre-defined (a good thing!) we may have multiple names for the same semantics. Or, developers may use the same names for semantically different subjects.

As with regular tags, it will be up to us, researchers, to make sense of the machine tags via our “tools” - be it schema mapping, pattern extraction, semantic analysis, or any other technique (content anaylsis may even help in the case of multimedia). There is simply no hope in enforcing a complete set of semantics for media (or content) annotation on the web as a whole. Which led me to declare the [grand vision of the] semantic web dead.

Well, as I mentioned above, this claim engaged the audience quite a bit (a good thing!). Some of the panel responses claimed that the machine tags are, indeed, a success, or even an instance, of the semantic web. I mentioned Aaron Cope’s statement: “machine tags are not RDF but they can play RDF on TV”. I also mentioned Dan Catt’s quadruple-hop from machine tags through RDF to JSON using Triplr and Pipes (partially in support of the connection between machine tags and the semantic web). So, it doesn’t matter whether RDF is used or not (Dave Beckett, friend of the program, and the editor of RDF as well as the man behind Triplr was in the crowd and participated in the discussion). Others pointed out that the semantic web is very successful and useful for some domains (I agree).

To summarize and conclude, let me tone down my claim. The Semantic Web is not dead; but the grand vision of “A Semantic Web” will not be achieved, despite Tim’s noble efforts. Instead, we will see the Emerging-Semantics Web, derived from how people/developers use lightweight formats and tags on popular platforms such as Flickr and YouTube. As for the panel, I concluded with quotes from Nat Torkington’s rather amusing post about the future of the web - right on the spot, but in retrospect, maybe I should have spared the injured crowd from this additional insult at that point…

My slides from the panel are embedded below. I also encourage you to check out the slides of the other panelists, all smarter (and perhaps less combative) than me: Sean Bechhofer, Lynda Hardman, and John Smith. And thanks to Susanne Boll and Raphael Trochy for organizing!

Media from the Green Hell

A few weeks ago when the research community was looking at CHI, Formula 1 racing fans were looking at Germany to watch a very different event. Nick Heidfeld was driving a F1 car around the famed Nürburgring-Nordschleife circuit in Germany. “The Green Hell” as it is popularly known is one of the toughest race tracks on the planet and last hosted an F1 race back in 1976 when Niki Lauda’ near fatal crash put an end to F1 at the track. It has since been used only for sports car racing and also as a public access race track. On April 28, BMW organized a special event where several of their racing cars would be driven around the ring including 3 laps in their 2006 BMW F1 car. The event had F1 fans all over the world speculating about possible laptimes and thousands showed up to watch the event live.

At this point you are probably wondering what this has to do with YRB. The answer is media: the very first images and videos of the event came from Flickr and YouTube. This is not new to the world of journalism, public citizens have been scooping the press for years. With the rise of the Internet, bloggers have started breaking stories and are often authorities on specific topics. However this event was a bit different because it was specifically conceived of as a public relations opportunity. The BMW PR machine had been hyping the event for weeks and had a full team there to capture the event in its full glory. Within 24 hours of the event they released a slickly edited video showing the highlights around the lap. It included onboard footage, helicopter shots and even video from chase cars. Despite all the effort and the fairly quick turnaround time, the corporate PR machine was beaten by a handful of people with cheap camcorders and Internet access. Not only was the fan media the first to be available, it also showed many things that the official video did not. You could see things like the size of the crowd, the sound of a distant F1 engine, the wait for the car to go past, there were even a few shots of the camera car.

This kind of situation leaves the consumer with an interesting choice: On one hand you have the quick and dirty fan media which is immediately available and on the other hand there is the better edited professional media which is available with some delay but usually more comprehensive. The consumer now have to decide which source they want their information from and that will depend on the topic and its time sensitivity. This choice is going to become even harder in the future as fans start editing their media more and the professionals speed up their editing process. I believe that in the end trust and authority will become the deciding factor - all else being equal users will choose the source that they trust the most.

At YRB we are very interested in both sources of media and are studying how people collect, consume, share and remix media. We don’t know what media sources will “win” but we do know that this is a very interesting time for research.

Photo by peve.de.