Feeds basics 101: introduction to newsfeeds (part 1)

Deutsch | Español | Français | Italiano | Português | 日本語 | 한국어 | 简体中文

Feeds, RSS feeds, Atom feeds, XML feeds, newsfeeds, web feeds, they're increasingly common on the internet these days - but what are they, how do you subscribe, and how do you publish and publicise your own news feed? This post is a 3-part introductory tutorial guide to web feeds, aimed at intelligent non-geeks who don't want to be treated like idiots just because they don't wish to struggle through a sea of unnecessarily befuddling tech jargon, but do want to learn rather more about feeds than the average dummy, with a few tips and tricks along the way.

The introduction of the graphical Web, the point-and-click web browser, revolutionised the Internet. Only a relatively small proportion of Net users may be using newsfeeds right now, but more and more are going to. I personally think that feeds are just as revolutionary as the point-and-click Web was. And lately many of the posts I've been wanting to write have been to do with feeds, so it's about time I did a more integrated post for beginners on feeds, as I'm aware that actually a lot of people still don't use them - not counting of course those Web-savvy denizens of the Net who virtually live online.

There are lots of intros or guides out there already on feeds e.g. by Feedburner, or by the BBC (more pictorial BBC version), or on Wikipedia. This is just my own personal take on the subject. My emphasis will be on feeds for Blogger or Blogspot blogs, but most of what I say will be relevant to other blogging platforms too, and indeed other kinds of websites which aren't blogs. As always I'd appreciate any feedback on clarity, any errors etc. (and should any publishers be thinking of a book on feeds or other techie subjects for the intelligent lay person, please see my sidebar for how to reach me!)

First I'll explain the basics generally (but still with a practical bias), then in part 2 I'll explain how to make use of the feed that's associated with your blog i.e. how to publish and publicise your own feed, and finally in part 3 I'll show you how to use the free Feedburner service to "syndicate" your blog or website, including the use of some of its other helpful features, summarising some things I've learned from fiddling with my own feed.

Note that this basic post doesn't discuss the underlying coding or how to create a feed from scratch (phew, I hear some of you say), but rather aims to give you enough info so that you can figure out for yourself how best to make use of feeds. It does assume basic knowledge of HTML (for which see e.g. Blogger's help on HTML) and basic understanding of computer files and folders.

1. Feeds - the very basics

So what's a feed anyway?

A news feed (feed for short) is in fact a special kind of text file written in a particular standardised format, which can be automatically created by certain Web server software from sources such as webpages. Newsfeed files may be flexibly used in all sorts of ways by software that knows how to fetch them and read them (feed readers) and possibly do other things with them.

That all sounds a bit broad and general, but the kinds of things you can get feeds for and how feeds can be put to use really have changed radically, as I'll come to later, and no doubt they will keep on evolving.

It may be clearer if we step back a bit and start with the most basic and popular function of feeds.

With websites whose content often changes, such as news sites or blogs, it can be a pain as well as time-consuming to have to revisit sites constantly just to check what's new. Using newsfeeds, you don't have to.

If a site publishes a newsfeed (also known as "syndicating" the site), you can just "subscribe" to its feed via software or Web services known as "feed readers" or "aggregators". Feeds are also sometimes known as "syndicated content". (Confusingly, "syndicate" sometimes seems to be used for the act of a user subscribing to a site, as well as the act of publishing a feed.)

Subscribe to all the site feeds you want, and then the new content from all these different sources will be automatically fed through to you to skim at your leisure at once.

Effcctively, feeds are a kind of "pull" technology where your desired content is automatically pulled to your computer behind the scenes so that you can access it all together in one single place, rather than your having to go look for it in all sorts of different places. Some might say it's a kind of push technology but strictly it isn't, and I say "push" in hushed tones as it became a bit of a dirty word after it was overhyped and fell flat on its face in the earlier days of the Web. The concept is excellent but it was pushed (if you'll forgive the pun) before its time, that's all. Push technology (e.g. instant messaging), or should I say quasi-push in the case of feeds, is certainly coming into its own now. But I digress.

Here is a (deservedly much linked to) lighthearted video by Common Craft called "RSS in Plain English" which explains the "push" aspect of web feeds very clearly for non-geeks:

Click To Play

Feeds as shop windows

Another analogy I dreamt up for feeds (when writing my intro to podcasting) is this: shopping. Yep, you heard the shopping analogy for feeds here first at A Consuming Experience! Trust a girly geek to come up with that...

So what do I mean, shopping? Well, think of it like this: instead of your having to keep going round your favourite stores to check what's new there (their webpages), shops display their latest wares in their shop windows (their feeds). You put your feet up at home and send your faithful personal shopper (feedreader) out regularly to check if there's anything new in the windows of your favorite shops - if there is anything new (and of course they'll know because, like any decent personal assistant, they've kept track of what they saw the last time they went there), then they'll bring it back home to you.

So you can have a replica of your favourite shop windows in your own living room, which you can rummage through at leisure in your own time, and you haven't even had to leave your house. Or, in the vast majority of cases, even pay a penny for what's been brought home to you.

Now the analogy isn't exact - some shops only put out free samples in their windows (partial feeds or excerpts only feeds) rather than the full product, the full feed, and you'll then have to go into the shop personally (visit the Website) if you want to see their new stuff properly - but I think that analogy helps to give a general but fairly accurate idea.

Feeds content - what do feeds cover, what are they for?

Just as there are blogs on everything under the sun, so there are feeds on every subject under the sun. That's because every blogging platform worth its salt automatically produces a newsfeed for each blog - i.e. it automatically dresses up its own shop window to show off any new goodies inside. You just need to open up the curtains and expose your wares, so to speak (and no, we're not talking Amsterdam shop windows, OK? This is a family blog. Harrumph).

What's more, you can have feeds for more things than just webpages. For instance, for many blogs you can get a comment feed, which is a feed showing all the comments posted to that blog, handy to help watch out for new comments on that blog. And you can have "per post" comment feeds, which show just the comments added on a particular blog post or webpage (see the Blogger feeds help). With Blogger you can even have "label feeds", showing all the posts which have been given a particular "label" by the author (i.e. posts which have been classed as being about a topic like "mobiles"), if the subscriber wants to keep track of new posts from a blog just on a single subject.

Feeds can be used not just for text, such as most Webpage content, but also to distribute other kinds of content too - notably, media feeds are possible for multi-media content such as music, video, even documents like PDFs. For instance, people can get MP3s via feeds, a.k.a. podcasts - e.g. internet radio shows (more on podcasting - feeds for music etc - in my podcasting guide). Video feeds are known as vidcasts, webcasts or videocasts, etc; Blogger for instance have tested and now allow podcasting of videos on Blogger blogs. And photographers can provide their uploaded photos as a photo feed or picture feed (e.g. Photobucket or PicasaWeb; and Flickr photostreams can be available as feeds), and even via Google Pack screensaver to people's desktops. You can get all Web 2.0 and social media with feeds too - e.g. Google Reader lets its users share, on a public webpage (or via its feed, of course) or by email, or as a small clip that can be embedded on any website or blog, individual selected items in their subscribed feeds or all items they've assigned selected tags to.

More and more Web services and applications, for example social networking sites, are providing feeds for their users, or re-purposing feeds. You can keep on top of the status of Blogger and its hiccups via the Blogger Status Feed, for instance, or subscribe to Google Finance or Google News, or government info if you really must. Or, less prosaically, transmit your blog feed into space! Twitter, the social networking service which uses SMS text etc, provides feeds. So does Facebook. For mobile phones, Nokia's WidSets uses feeds to enable users to read blogs and access other things on their cellphones, and feeds are also used by Twitter competitor Jaiku, which was recently acquired by Google after just a year and a half in operation (so Jaiku founder Jyri Engeström's 5 principles for Web 2.0 success are very well worth noting!). You can even get calendar feeds of diary entries or events e.g. from Google Calendar, Upcoming.org etc, and potentially transport schedules on Google Transit. Many Web search engines base their indexing of sites on the feeds which sites like news sites output, e.g. Google's Sitemaps, and Google's Blog Search. (Apologies, this paragraph somehow disappeared from the draft when publishing - it should have been in the original published version.)

How do you subscribe to a newsfeed? - tools: feed readers

You can subscribe to a feed in two main ways:

• on your own computer - use a feed reader program (a.k.a. "news aggregator" or "feed aggregator" programs like Juice - I'm going to say "reader" as it's shorter to type), or
• via any computer connected to the Net - sign up for an online Web-based feed reader service (they're mostly free) like Bloglines (the pic at the top of this post is a Bloglines screenshot), Google Reader which dates from 2005 (and which you can now even read offline), or Netvibes, which you can then access via a browser from any computer, subscribing to feeds via the service and then viewing all your subscribed feeds together on the feed reader's site.

A full roundup of feed readers is beyond the scope of this post - see the Wikipedia list of feed aggregators - but you can get lots of free ones such as SharpReader for Windows, or the multi-platform podcast-dedicated Juice. I have an early beta version of the free multimedia friendly Fireant for Mac and Windows which supports text as well as podcasts (Juice doesn't do text, just media - though what's "media" anyway?):


Increasingly too, feed support is being built into browsers, e.g. the fab free browser Firefox has a basic built-in reader (more a feed subscriber really), known as "Live Bookmarks", plus you can get free feed reader extensions for Fox like Sage. The lightning quick free browser Opera has supported feeds for ages. And Internet Explorer 7 also has a feed reader built in, though earlier versions don't.

We're also seeing feeds being displayed not just in specialised feed readers but also as part of other kinds of info on personalised webpages or customizable webpages like My Yahoo!, via Google's iGoogle personalized homepage with Firefox toolbar integration e.g. through gadgets, or even MyBlogLog (e.g. this blog's MyBlogLog page) and the Blog Friends app on Facebook (more on Blog Friends), and even in people's blog sidebars. In fact, it's possible to display the content of feeds on any webpage just using some Javascript (talk about recursive!), and I'll be doing a separate post on feed display via Javascript. You can even get feeds on mobile phones, and not just through iGoogle.

Feed readers will normally display in a list all the feeds you subscribe to. You can then choose a particular feed to see the latest items from that feed (rather like choosing a folder in your email mailbox to see all the emails within that folder). Links within items will usually be clickable too.

Here's a screenshot of the Web-based Google Reader showing the feed from my pal Kirk's blog:


And here's Fireant showing the contents of a text feed:


While here's Fireant showing an MP3 podcast of Creative Commons-licensed music by Uwe Hermann - I can download individual songs and play them from within Fireant (controls at the top right of the window below):


(A side issue, but the beta 1.0 version of Fireant I have doesn't seem to be able to handle video enclosures properly e.g. from the Bloggerindraft blog - nothing at all. I hope later versions display Blogger video enclosures properly.)

Now, as well as what media types they can handle, readers differ also in how you view, filter and generally manage your subscribed feeds and individual items in the feed, how often to refresh the feed (check for new items in your subscribed feeds and download them), etc, and also as regards the level of control the user has. The most basic feed readers may be able to handle only text, but not podcasts (music files) and the like - or vice versa, e.g. as mentioned Juice handles MP3s but not blog posts text.

How do you subscribe to a newsfeed? - mechanics

In terms of mechanics, to subscribe to a particular feed manually you just:

1. note the feed's Web address or URL (copying it to clipboard is of course the easiest way) - these days feed icons are standardised and you can get the feed URL by rightclicking on the standard orange feed icon , known as a "chicklet", shown on the site - and then copying the link or shortcut (which will be the feed's URL)
   
2. launch or go to your feed reader of choice, if it's not already open
3. go to the relevant "Add Feed" or similar function of your news reader (often it's a + icon)
   
4. paste the feed address link and OK.

After subscribing to a feed, the new feed will be there in your list of feeds, and you can of course then view the added feeds in whichever way works for your reader.

But there are easier ways.

Auto-discovery - what is it?

Many feedreaders can even detect a site or blog's feed URL automatically from the site itself, if the blog or website has been set up to allow that, just by your giving the feed reader the site's main homepage URL. Many feed readers automatically detects the feed URL while you're on the site's webpage, you don't have to give it any further info. That's known as "feed autodiscovery". (I'll cover how to enable and provide prospective subscribers with auto-discovery for your own feed if you're a publisher, in part 2 of this series).

For instance, in the Firefox browser, if you're viewing a webpage which has auto-discovery enabled for its associated feeds you'll see the orange feed icon in the browser address bar, and if you click on the orange feed icon (a little obscured by the mouse in the pic below), it'll give you the option to subscribe. In the case of my blog, I offer readers a choice of 3 possible types of feeds as like others I don't believe in only excerpts (here's how to offer a choice); hence, clicking on the feed icon will list all 3 options to choose from:


There are other ways to autodiscover feed URLs too, if you're a potential subscriber - for instance the online feed aggregation service Bloglines offers users a special bookmarklet or favelet which you can add to you browser toolbar. Click that bookmark while you're on a website and it'll try to auto-discover and list the feeds available for that site or blog, so that you can then choose which ones you'd like to subscribe to via Bloglines.

Oddly, that favelet doesn't work on my blog main page, only offering the comments feed URL:


Whereas the bookmarklet does work properly when viewing individual post pages:


So, as you can see, autodiscovery isn't always foolproof. But in many cases, it does help speed things up for would-be subscribers.

How do newsfeeds work behind the scenes, in simple terms?

RSS (also sometimes known as Really Simple Syndication) is one main standard used by websites to syndicate their content, i.e. a feed file is a text file written in RSS format. The other well known standard format for feeds which you're likely to come across is newer and, many say, better: Atom. (There are in fact several flavours of RSS itself, confusingly. I won't go into the standards wars here. For the history see e.g. Wikipedia). "RSS feed" is sometimes used synonymously for "feed" or "newsfeed", even when referring to Atom feeds - a bit like "hoover" is used for "vacuum".

Obviously, you can view a feed published using RSS only if your reader supports RSS. And similarly with Atom feeds. However, most readers these days can handle both formats, so generally you don't need to worry about that.

To publish a feed, a web site or blogging platform basically takes the HTML file for the webpage or blog post and constructs a new file in RSS or Atom format out of it (the feed file usually ends in ".xml", but it could sometimes be something else like ".rdf" or ".aspx"). It is that specially-produced feed file which feed readers will download and display.

XML is, the way I think of it anyway, the big brother of HTML - which, as you know, is the language in which webpages are written behind the scenes so that they can be interpreted and displayed on your computer (relatively!) consistently by browsers such as Internet Explorer, Firefox, Safari, and Opera. RSS and Atom are basically dialects of XML, and you could say that a feed file is written in a machine-readable way using RSS or Atom.

You don't need to know the coding ins & outs, the syntax for RSS or Atom, to be able to make use of feeds as a publisher or a subscriber. But if you want to, you can actually view the text of a .xml feed file, its underlying RSS or Atom code, via a text editor like Notepad - just as you can view the source of a webpage, which is really just a text file with the ending .html. Indeed, you can view a feed file direct in many browsers - just enter the feed's URL in the browser address bar. But just like the raw HTML of a webpage doesn't look very nice in view source or a text editor, similarly a feed often doesn't look very pretty in a browser, especially older browsers, as it was designed for software not humans - although more up to date browsers can in fact cope fine with their display. And although browsers are incorporating feedreading functionality, viewing a feed via a modern browser is still not as userfriendly, powerful or flexible as using a dedicated feed reader (or, if you prefer, just going in your browser to the original webpages which the feed file was based on). As an example, if you really want to see a feed in your browser click on this for my blog's RSS feed - opens in new window.

But what is in a feed file exactly? What info does the user get sent?

Now having said earlier that a webpage is converted into a feed file, that's not 100% accurate. It's not necessarily the whole of a website or even a single webpage which is converted to produce the feed.

With blogs, commonly it's only the main page or home page of the blog (and not post or item pages, or even archive pages), which is used as the source for the feed file. Or it's the latest X posts only whose contents are copied into the feed. (Those who have lots of control over their site and server where their blog files are stored may be able to create feeds from other pages or other parts of the site, but with most blogging platforms you won't have that level of control, they do what they do and that's it.)

With Blogger blogs, it used to be that the feed displayed exactly the number of posts set to be shown on your main page (so if you'd set your main page to show 10 posts, the feed would contain your 10 latest posts, if 50 then the latest 50 etc). But since since the rollout of the now feature complete fancy New Blogger, formerly known as Blogger Beta, despite its widgety goodness and enhanced developer-friendliness the default number shown in the base Blogger feed is now 25 posts - although there's a way to set Blogger to show more than 25 posts (or indeed fewer), which I'll cover in a later post.

So, taking the Blogger example, because the feed by default just reflects your last 25 posts, as you publish new posts the feed's content will inevitably change, with new posts added to the top of the feed "pushing off" the older posts from the bottom.

Effectively, whenever you publish a new post, Blogger knows to reconstruct your main blog page (what Blogger call the "index" page) - and, at the same time, it will also reconstruct your feed so that the feed will show the 25 newest posts as and when requested by a subscriber.

But remember that a blogging platform only automatically produces feed files if that feature is activated or enabled. It normally is, by default, but if you have problems you might want to check that. I deal with how to do that on Blogger later in part 2.

Also, in terms of the content of your feed, it's not necessarily the full text of each post that is shown in a feed - it could be excerpts only e.g. the first X characters of each post. I'll discuss that in more detail in part 2, but basically with a text feed the blog owner can tweak things to control how much (or how little) of each post is sent with the feed.

Feeds can contain text, but they can also contain links, e.g. Web hyperlinks. As I alluded to earlier, feeds have developed so that they can "transmit" not only text but also media like songs, videos etc. But the feed doesn't "include" or transmit full MP3 files, but rather links to those files. As long as the blogger / feed publisher has their site set up to produce the media feed properly by including special links to media files in a form recognised by feed readers, called "enclosures" (as to which see my podcasting intro), and the reader can cope with media files, then a subscriber can receive audio and video too, even PDF or Word documents etc, via a feed. See the screenshots of Fireant above, for instance.

How does the reader know when a feed has been updated?

As mentioned above, when you publish a new post your feed should automatically be updated by your blogging software constructing a new XML file to include the new post.

When I first heard about feeds I puzzled over the basic question of how a reader knows a feed has been updated before I found out that, often, it doesn't have to. When a user subscribes to a particular feed, what their reader does (assuming it's open and active, of course, where the feed reader is on their own computer) is to check every so often whether a subscribed feed has changed, and then fetch the changes back. How often it does this can usually be set in the options or preferences for the feed reader.

Using my personal shopper analogy, your personal shopper makes a note of the shops you want it to check and what it's fetched back for you so far from those shop windows, and you can tell it how often to check your fave shop windows for new wares. And then it does that. So your reader keeps track of the changes for you and monitors your feeds of choice for changes.

Confession: I rarely use feed readers, myself - somehow, the way my mind works, I prefer email alerts like Google Alerts. And in fact you can get feed updates via email, through services such as Feedblitz and Feedburner's email service. But I still need to know about feeds and I still have to provide them on my blog, because I know many people prefer them. And feeds are now important for many more reasons than just subscribers' convenience, as I've already mentioned.

In part 2, I'll go into more detail about ways to publish and publicise your own feed.

Tags: feed, newsfeed, newsfeeds, RSS, Atom, full feeds, partial feeds, chicklets, feed chicklets, feed icons, Feedburner, auto-discovery, auto discovery, autodiscovery, feed autodiscovery, Blogger.com, Blogspot, blog, blogs, browsers, blogging, blogging tools, tips, tricks, guide, basics, Improbulus, A Consuming Experience