Californian-based Marvell has announced a plug-sized computer, called the Sheevaplug. It's got everything you need - cpu, memory, network connectivity and USB ports.
More specifically, it's packing a 1.2GHz CPU, 512MB of flash memory and 512MB of RAM, along with a gigbit ethernet port, USB and mini-USB ports, and an SD card slot.
It lacks any kind of video output, so you'll need some sort of USB video adaptor, but its specs should be sufficient to run a cut-down Linux distro and work as a web server. It only consumes 5W of power, so don't worry about it pumping up your electricity bill.
It's not too pricy either - coming in at $99 for a developer kit, which includes the plug, as well as all of the software tools needed to develop applications for the platform. If you want one that comes with all the stuff you need pre-installed though, then you'll need to wait a bit longer.