<button> elements are much easier to style than <input> elements. You can add inner HTML content (think <em>, <strong> or even <img>), and make use of :after and :before pseudo-element to achieve complex rendering while <input> only accepts a text value attribute.
IE7 has a bug where when submitting a form with <button type="submit" name="myButton" value="foo">Click me</button>, the POST data sent will result in myButton=Click me instead of myButton=foo.
IE6 has an even worse bug where submitting a form through a button will submit ALL buttons of the form, with the same bug as IE7.
This bug has been fixed in IE8.