Google recently announced on their webmaster blog that the speed of a website would now be a factor in determining its search engine position.
With the emergence of widespread broadband use and greater bandwidth, web design trends have moved, in recent years, towards media rich layouts and interactive script libraries. While these can enhance the user experience, they also add to the overall download size of web pages. Conversely with the rapidly growing use of web browsing on mobile devices such as the iPhone and iPad websites have had to adapt - with many now providing lightweight mobile versions of their main site.
Google’s new speed rule is only currently being applied to results on Google.com searches and effects fewer than 1% of search queries so unless your website is a behemoth then there is no real need to worry.
A useful tool that I am currently using is the YSlow plugin for Firebug which calculates the speed of your site download and suggests possible areas for improvement. Google Webmaster Tools also now provides details of speeds which may be helpful in optimising your site.