Small Disclaimer: WordPress itself doesn’t really use that much memory, the increase is normally noticed on established WordPress installs with plugins and themes installed.
Use A Caching System
One of the biggest memory hogs while using WordPress is actually PHP. All of the memory required to build your pages is used by PHP, which depending on the amount of SQL queries needed & plugins used can be upward of 20-30 MB. To stop the need to use that amount of memory every time a page is loaded the best thing you can do is install a caching plugin. The most well know ones are WP-Super-Cache & Hyper Cache. I have used them both & can recommend both as excellent caching systems.
If you are struggling to understand how a caching plugin will help, here is a little description. A caching plugin basically waits for a page to be loaded via PHP then takes a flat HTML copy of the page and loads that the next time someone wants to view it. It’s obviously more complicated than that, but that is the general idea behind it.
Limit Plugin Usage
One of the greatest things about WordPress is its extendability via plugins, but therein lies another of its problems. Using too many plugins can cause WordPress’ memory usage to go through the roof, so the best thing to do is use only the plugins you need. If you have any you don’t use any more, disable them. Go through ones you are using & consider if you really need them or not. This is not to say that every plugin does increase memory usage, but to the general WordPress user the best advice is to disable as many as you can.
For those that are WordPress savvy try disabling plugins that use a lot of MySQL queries, do heavy calculations, or place a lot of information in memory. If you find that some plugins are essential for the functionality of your site, keep them enabled & try to save memory in other places.
Something not many WordPress users consider is memory usage while you are in the admin. The amount of memory used when saving a post or even just loading a post for editing can be quite high sometimes. To keep memory usage low until you need to save a post/page etc you can try using Google Gears, or Turbo as WordPress calls it. You can enable it by installing Google Gears & then going to the Tools -> Tools in WordPress.
Unfortunately Gears is not compatible with FireFox 3.6 and it is currently unknown if Google plans to update Gears or not with the recent announcement that they are going to concentrate on developing offline file access for HTML5.
Well those are my tips for saving memory, but it is by no means an exhaustive list. If you have any other tips please share them. I’d love to hear other ideas, and if you are ok with it, I’ll add them to this list. Also if anyone has any other tips on reducing the memory used in the admin I’m all ears.