A Consuming Experience: HTML forms: getting "get"

I finally get "method=" and "action=" on Webforms. I've seen loads of stuff explaining how to create HTML forms, how to include various form elements like radio buttons etc, but until recently I'd not found anything I could understand which explained what actually happens when you click the Submit button or hit Enter after you fill in a Webform.

Now I get it, at least for forms where method="get". And this means I also get how to construct working Webforms to search many sites too.

What happens with forms using method=get

Take a Web form like the following (where "formelement1" etc is just a standard form element for a textbox, checkbox, radio button, dropdown list menu etc e.g. input type="textbox", or select, or radio):

<form action="http://someurl" method="get">
<formelement1 name="name1" value="valueA">
<formelement2 name="name2" value="valueX">
<formelement3 name="name3" value="valueZ">
<input type="submit" value="Submit">
</form>

When someone fills in the form and hits Submit or the Enter key, what happens is exactly the same as if they went to the following URL (e.g. by manually typing it into their browser address bar):
http://someurl?name1=valueA&name2=valueX&name3=valueZ

In other words, their browser takes the URL in the "action=" bit of the form, adds a questionmark after it, then inserts the names and associated values of each form element, in the order in whixh they appear in the form, with "&" separating each "name=value" pair. It's just a way to take user input entered into a form and put it into a standardised format which can be used to do something like query a database on a search engine's servers (living at the URL in the "action="), and get back the results.

At least that's my interpretation of these official W3 explanations ("user agent" just means "browser"), which may be shorter than mine but perhaps are a little less comprehensible to the average softcore techie (if I may coin a phrase - I regard myself as a softcore techie too!):

(On the form submission method) "get: With the HTTP "get" method, the form data set is appended to the URI specified by the action attribute (with a question-mark ("?") as separator) and this new URI is sent to the processing agent."
or
(On the form data set) "If the method is "get" and the action is an HTTP URI, the user agent takes the value of action, appends a `?' to it, then appends the form data set, encoded using the "application/x-www-form-urlencoded" content type. The user agent then traverses the link to this URI."

Example form

I'll illustrate with a form to search Blogger.com profiles. I first thought of producing a form for this kind of search when I noticed the pattern in the structure of the URL in the address bar after you click on a keyword in a Blogger profile in order to find other profiles with the same keyword (e.g. interests=leather; or "pot noodles", if you must) - but the best I could manage, in the state of greater ignorance I was in at the time, was to find a Javascript script which I then somehow managed to adapt to do that search (and posted about previously.)

Now I've realised that you don't even need Javascript, a simple HTML form will do (and I've replaced the Javascript version in my Blogger.com template with this now and also edited my previous post to include the HTML version). Here's the code:

<!--  Please do not delete this note
Form to search Blogger.com profiles - results open in new window, to change this delete target="_self"
Copyright Improbulus 2005 http://consumingexperience.blogspot.com/ licensed under Creative Commons License http://creativecommons.org/licenses/by-nc-sa/2.0/
-->
<form action="http://www.blogger.com/profile-find.g" method="get" target="_blank">
<label for="q">Find this word or phrase:</label><br />
<input type="text" size="25" name="q" id="q" /><br />
<label for="t">within this profile section:</label><br />
<select name="t" id="t">
<option value="i">Interests</option>
<option value="m">Movies</option>
<option value="s">Music</option>
<option value="b">Books</option>
</select>
<input type="submit" value="Search" /> <input type="reset" value="Clear" />
</form>

Ignore the "id=" (which is included to go with the "label=for") and the odd "/>"if you like, I'm just trying to acquire good habits and start learning to be XHTML-compliant like my pal Tab...

Now if someone fills in "leather" in the textbox (so that the value of the textbox named "q" becomes "leather"), chooses "Interests" from the dropdown (so that the value of the dropdown menu named "t" is set to "i") and clicks Submit, what happens is the same as if they'd entered, into their browser address bar, the URL after "action=" (here "http://www.blogger.com/profile-find.g") followed by a "?" followed by q (the name of the textbox) = leather & t (the name of the dropdown list) = i (the value selected for Interests), i.e. http://www.blogger.com/profile-find.g?q=leather&t=i (click that link and you'll see the results are the same as if you'd filled in "leather" in the textbox and selected "Interests" using the profile search form in my sidebar - though on reflection, don't click that link unless you're over 18!). Here's the form whose code is given above, which you can test out:

(I used a dropdown list to save screen real estate but, if you prefer them, radio buttons would do as well - e.g. <input type="radio" name="t" value="i">Interests<br /> <input type="radio" name="t" value="m"> Movies<br />, and so on for music and books - remember it's the name and value that are important, not the description e.g."Interests" or "Movies", though obviously the descriptions are needed for human comprehension.)

Note that I deliberately didn't include "name=" for either the submit or reset input button - you don't want to have name/value pairs sent out for those, for obvious reasons they're meaningless and useless if not downright confusing to the poor lil server that's just obediently processing the name/value pairs which do contain info of substance.

Building your own search forms

So if you use a search engine where the results page has a URL in the same format as above, with a URL then a questionmark and then x=y&a=b etc pairs, or indeed if you come across any Webpage whose URL is in that format, you can devise a form to produce the same kind of result or perform the same kind of search.

You can figure out what the URL for "action=" should be set to in your form from the first part of the URL in the address bar, and you know because of the URL format that the method used is "get" (though you could view source on the original search form to check). You will also know what names to give the form elements from what's visible in the URL before the = signs, as well as the associated values (if-pre-set) from what's shown after the = signs, at least with a bit of experimenting and trying different searches and search options (or viewing source!) to figure out all the preset options needed to produce particular results, e.g. that name must be "m" to search movies, or "i" to search interests.

From that analysis you can build your own search form (replicating their name/value pair order just to be cautious), even for a site which doesn't provide a search form themselves (like I did with Blogger.com profile searches), and even include the form in your own blog or webpage - assuming that the search engine site concerned doesn't mind you putting a form onto your own page (I hope most won't mind, especially if they put their ads etc on the results rather than search page).

And vice versa - if you see the URL in a search result, you can sometimes, in simple cases, work out what the name/value pairs are from that and so create a search form even if you don't view source for the form. Which is exactly how I figured out the search form I included in my post on Who's linking to your blog.

A couple of twists. As well as automatically producing the questionmark and the "name=value" pairs in the resulting URL, if method="get" the browser will automatically do clever things like change spaces in the form values to "+" symbols (e.g. if you entered "pot noodles" in the textbox, that would be turned into "pot+noodles"), and "escape" or encode the name/value pairs into characters that are required for use in URLs (the funny % etc codes that you often see in URLs). That's called encoding using the "application/x-www-form-urlencoded" content type, that is.

Also, if a particular site always seems to require a particular name/value pair "this=that" for every search, you can include them on your form using <input name="this" value="that" type="hidden">and then the user won't see them but they'll be transmitted as part of the final URL.

Using a link to replicate a Webform

Finally, as I'd previously learned and exploited (e.g. my posts Technorati: cosmos, and watchlists, Auto-translating Webpages/blogs: code, and Help others bookmark your posts on Del.icio.us or Furl), you can include a link on any Webpage (e.g. blog post or template) that when clicked produces exactly the same result as if you'd filled in a search form with particular terms.

However you need to use a "+" symbol instead of a space in the link URL, and also should encode certain non-alphanumeric characters with "%something" (see the W3 note on form content types.)

The twist here is that I've just discovered that, strictly, the "&" symbols in the link (between the name=value pairs) has to be encoded as "&" in order for the link to work properly (in fact with most browsers and servers you can get away with just using "&" in the link, but strictly you shouldn't be, in order to avoid validation and other problems - see this W3 note).

I have to admit I haven't got round to doing those substitutions in my template yet, but it does explain finally why I kept getting errors in the validation of my template… must get to it soon...