Link Feedback - see and show where your visitors come from!
Sean Nolan yaywastaken.com

come visit us at our new home!

So one of the assertions in Stephen Johnson's excellent book Emergence is that the web alone doesn't show any self-organizing behavior because it's missing feedback, i.e., all the links are one-way.

Browser adjunct systems like alexa analyze the links between sites to generate "related" groups, with some very cool results. Unfortunately, alexa and its kin have not been widely adopted by the browsing public, for a lot of reasons --- privacy being a big one. So the data is valid for only relatively large sites that get hit by many, many people.

Anyways. I thought it'd be fun to do an experiment. I put together a very simple set of links that you can insert into your pages to automatically track incoming links and echo them back to people who visit your site. Basically it's automated link-exchanging, but hopefully with more interesting results. There are two steps:

1. Track incoming links to your site.

Anywhere in the BODY section of your site, put the following script tag:

	<script>
	<!-- 
	var szReferer = document.referrer;
	if (szReferer != '')
		{
		document.write('<img height=1 width=1 border=0 hspace=0 vspace=0 ' +
			'src=http://www.yaywastaken.com/referer/referer.asp?site=FOO' +
			'\046homedomain=MYSITE.BAR.COM\046referer='+escape(szReferer)+'>');
		}
	// -->
	</script>

Basically this guy just checks to see if the browser has sent a "referrer" --- the URL that was clicked to end up at your site. If there is one, it creates an invisible 1x1 pixel image that references a page at yaywastaken.com. ywt checks the referer to be sure it's (1) publically accessible and (2) not from your home domain setting. If so, it adds the link to a small database. The script then returns an empty image so that the layout of your page is not affected.

2. Show the list of recent referrers on your site.

The easiest way to show the list of referrers to your site is to add the following script tag where you'd like it to show up:

	<script src="http://www.yaywastaken.com/referer/referer-js.asp?site=FOO&url=http://MYSITE.BAR.COM"></script>

	<script src="http://www.yaywastaken.com/referer/referer-js.asp?site=FOO&url=http://MYSITE.BAR.COM&xsl=http://MYSITE.BAR.COM/referrers.xsl"></script>

Finally, If you're using a system that understands RSS feeds natively, you can just point it at the url:

	http://www.yaywastaken.com/referer/referer-rss.asp?site=FOO&url=http://MYSITE.BAR.COM
	

A Few Extras

You can pass an additional querystring parameter "maxitems=X" to the referer-js.asp and referer-rss.asp scripts; this will set the maximum number of items to X (default is 10, or "maxitems=10").

You can also pass the querystring parameter "qry=popular" to the scripts. Instead of selecting the last X referrers, it will select the most common referrers to your site --- i.e., the ones with the most hits since you started keeping track.

The querystring parameter "notitle=1" will cause the script to return URLs instead of TITLE tags as the friendly name of each link.

The querystring parameter "maxlen=X" will truncate friendly names at X characters; this can help when you want to use the raw table but don't want your table layout to get messed up.

The querystring parameter "maxwordlen=X" will break words longer than X characters with spaces. This is another approach for keeping long title tags from screwing up your table layout without having to resort to truncation.

Note: This one goes on REFERER.ASP. The querystring parameter "exclude=blah,moo" (where blah,moo is any comma-delimited set of words) will exclude referrers containing these words from your lists. For example, if you don't want Google URLs to show up, pass "exclude=google.com" and you'll be set. Remember that this alters new additions to the db, not queries out (that would be too slow for my db) ... so it may take some time for old entries to work their way down the list.

I've had requests to make the javascript table writer spew class names so that people could more easily customize the display with CSS ... here goes: the table itself is "ywtTABLE"; the tr for the header is "ywtTRH"; the td for the header is "ywtTDH"; each tr for a link is "ywtTR"; each td for the bullet is "ywtTDB"; each td for the link is "ywtTD". Hope that's helpful! (Note: if the grey background or font tags are making it difficult to get styles to work, I've put up a completely naked stylesheet you can use. Just add "xsl=!vanilla.xsl" to your referer-js.asp url and you'll be off to the races.)

Thoughts and Caveats

No warranties, no guarantees, no nothing. I think this is an interesting idea and would love to see a lot of people using it. My server should handle moderate loads, but we'll see. :) If you have trouble, please email me so we can try to figure it out. If all else fails, just yank the tags out --- but please let me know.

Privacy ... referrer data is yours, sent to you by the browser. By using the referer.asp page you'll be sending it to me. I promise not to give this data away, but I do plan to look at it to see if I can find any patterns and if there are interesting aggregate, non-identifying results I reserve the right to talk about it on my weblog. And search engines have some pretty weird terms, so get used to it.

A number of people have asked for the source. Don't get too excited; it's pretty crappy VBScript/ADO stuff because that's what my host provides.... but here you go:

• referer-inc.asp, shared stuff
• referer.asp, the guy that tracks referers
• referer-js.asp, javascript table generator
• referer-rss.asp, rss feed generator
• referer-sql.htm, ddl for the underlying database

URL normalization needs to be better ... For raw google searches I try to compress urls that are logically the same into one, but it doesn't always work. Any ideas here?

I know I spelled "referrer" wrong in my urls.

---S