Friday, August 05, 2005

The long tail and the short head of the Zipf Curve

Chris Anderson of Wired is getting a lot of play since the publication of his October 2004 article on "The Long Tail." He makes some interesting observations about how especially when it comes to selling digital content, niche players can make a lot of money selling many items in low volume.

So we're seeing lots of examples of Zipf Curves, which I first came to understand 4 or 5 years ago when doing Web search log analysis. But you've got to define your terms and understand just how long that long tail can be. Consider two graphs:



Now look at this one:


(Click to see full-size image.)

Here's the important part: These two charts represent the same data. The first one covers the top 1000 unique searches. The second one covers the top 31,000 unique searches. When you look at all searches in a given period, you discover a large number of unique -- one-off -- search phrases.

Another way of understanding the curve is to look at how many unique searches are required to reach certain percentiles of coverage. Look at the top 10 unique search phrases at Michigan State:

(Click to see chart full-size)

This dramatically demonstrates how much the "head" of the curve holds. Now, to understand the rest of the curve, we can examine how far down the curve we have to travel in order to achieve further coverage.

For instance, to cover 20% of all unique searches, we need merely include the top 32 items. To cover the rest of the curve:

Percent Rank
30% 85
40 203
50 441
60 925
70 2182
80 5617
90 13770
100 31000

Suppose, for instance, that we put some effort into providing "Best Bets" for our search engine, ensuring that we direct people to the "best" page for the most popular searches. Based on this sample, we could achieve that for 60% of all searches with a database of only 925 entries.

"The Long Tail" argument goes like this: there's lot to mine in the niches. But for search log analysis and Best Bets, you're a fool if you dedicate resources too far into the tail. You're turning yourself into a research librarian with thousands of customers daily. You can't afford to build a Best Bet for a search that may never be performed again.

I think there's a paradox here. Anderson gives the example of Netflix, which ships DVDs by US Mail. They can afford to keep much deeper inventory than, say, your local Blockbuster. So if you want to rent an obscure documentary, Netflix can make money and you get to see the obscure title you crave.

Anderson argues that retailers who distribute digital content, such as MP3 audio, can really win. Disk is cheap and distribution is practically free.

So here's the paradox: I think the equation is reversed for Web search services. Google is the leading search engine because the PageRank algorithm serves the short head of the curve, and the middle of the curve, so very well. Millions of people find what they seek. But travel too far out on the tail end, and Google will fail you. In fact, if there's only one or two links to your obscure page, Google may drop it right out of the index.

But sometimes you really need some obscure piece of information. You might have a question that you're willing to pay $1000 or $100,000 to have answered. Rather than paying less per unit in the long tail, you might be willing to pay many times more.

Even if you're not willing to pay big bucks, if your query is rare enough that you can't find satisfaction at the short head, if you're motivated enough to find your answer, you may spend real time and/or money to find your answer:


  • You ask your reference library to do some research.
  • You pay money to search a commercial database.
  • You hire an outsider to do the research -- perhaps a freestanding research firm, or perhaps you float your query to Google Answers.

So if your research query ends up being at the end of the long tail, you may pay more per item, not less.

21 comments:

Mike said...

make money quick
is easy. make money quick

lokokid said...

Hi i am totally blown away with the blogs people have created its so much fun to read alot of good info and you have also one of the best blogs !! Have some time check my link to !!Ways to make money from home

lokokid said...

Hi i am totally blown away with the blogs people have created its so much fun to read alot of good info and you have also one of the best blogs !! Have some time check my link to !!Make money online free

cash at home said...

Hi, My name is Donald you have a great blog here! I'm definitely going to bookmark you! I have a make money online and work from home site. It pretty much covers make money online and work from home related stuff.

Come and check it out if you get time :-)

credit cards now said...

don't you easy make money with googleads, it's the best way to easy make money

credit cards now said...

howdy.. reading your blog gave me goose pimples all over my body, but i noticed you don't have google ads anywhere?!?! Goggle adsense will make u mad cash a\if you use google ads. what do ya think?

luckpoo said...

Help! I am lost. I was searching for ways to make money online and somehow ended up here. How that happened I don't know, however I do like your Blog a lot. Would you mind if I add your Blog to my favorites page so others can visit?

riskfreemoney said...

You have a very good site on making money. This is something I also have a large interest in and have set up a website about making money. Please visit and let me know what you think.

Hilda Johnson-Slaton
Risk-Free Money Auto Income Generator
www.riskfreemoney.biz

stony said...

Great Blog! I have a make money related site. Maybe you have a look.

infinitemonies said...

Hey, your blog is awesome. I will definately add it to my favorites. I have a site that pretty much coversmake money on ebay related stuff.

startonline said...

Your blog is excellent. I specialize in card card credit fix help newgoodcredit.com self as well.

Swingin80 said...

Cool blog you have. I have a make money online from home related site. Check it out if you get a chance. The URL is make money online from home

cc Infopage said...

Hello,

I am searching for fresh information
for my cc Infopage, 30,000 daily updated Information Pages about all kind of subjects.

It might interest you to know that your blog has been visited and has been read. I hope you enjoy your "Blogging".

I wish you all the luck I can, keep the good work going!

Kind regards,
Jos
Today's News From & About Google

Johnjon said...

I discuss this topic daily myself. I also have a website that talks about make money working online related things. Go check it out if you get a chance.

Johnjon said...

I discuss this topic daily myself. I also have a website that talks about affiliate internet make make marketing money money online pluginprofit.html program wealth.com related things. Go check it out if you get a chance.

Dominicans resources said...

::FREE BLOGS AND FREE WORDPRESS BLOG HOSTING - CREATE UNLIMITED BLOGS NOW ALL FREE ::

Free blogging software hosting speicalist

haidi said...

Hey, you have a great blog here! I'm definitely going to bookmark you!

I have a **Make Big Money**blog. It pretty much covers Make Big Money articles related stuff.

Come and check it out if you get time :-)

Frank said...

Thanks for the great information, I book marked your site

I have a business idea own start site. It pretty much covers business idea own start related stuff.

Come and check it out if you get time :-)

Internet Marketer Azhar said...

Hi :)

You have a great blog! I'll be sure to bookmark it and visit regularly, keep up the great work!

Please visit my site if you get some time: working from home

Phill said...

I just came across your blog about forex research and wanted to drop you a
note telling you how impressed I was with the information you have posted here.
I have a forex research site. It pretty much covers forex research related stuff.
so I know what I'm talking about when I say your site is top-notch! Keep up the great work, you are providing a great resource
on the Internet here!

Thanks,
Phill

Google Page Rank 6 said...

Want more clicks to your Adsense Ads on your Blog?

Then you have to check out my blog. I have found a FREE and Legitimate way that will increase your earnings.

Come Check us out. How to Boost Your AdSense Revenue