Nando's blog
About Nando Nando's blog Posts about video, movies etc. Posts about computing Posts about music Posts about literature Philosophical posts Posts about programming

Do you really need 21 MB of RAM for a Python website?

I made a little experiment and found that using CherryPy 3.1, plus the packages I need, instead of the whole TurboGears stack, can save a lot of RAM.

How can a TurboGears website use 1/3 less RAM?

(If you host at WebFaction, your account is mainly constrained on memory usage.)

My TurboGears 1.0.4.2 application is using 23,888 KB on startup, then 27,512 KB after many requests for different pages. Pylons is more or less the same, since a Hello World takes up 21 MB.

(Python 2.5, Ubuntu 7.10. I understand this varies a lot based on OS.)

I decided to see what would happen if I rolled my own “framework” on top of CherryPy 3.1, which is vastly superior to CherryPy 2.3 — pity TG hasn’t upgraded!

So I tried a memory consumption test for a CherryPy 3.1 based website.

I started with the CherryPy Hello World, then added import by import, measuring the impact in memory usage. The total memory used is annotated besides the corresponding import.

You can see memory usage with the following Linux shell commands:

ps aux
ps aux | grep python
ps aux | grep hello.py

Don’t confuse virtual memory with resident memory. I am worried about the latter only.

The answer: A CherryPy 3.1 Hello World amounts to mere 7 MB. After I import everything I remember that I need (including Genshi, SQLAlchemy and Routes), I’m using 14 MB.

Here is the (very simple) test for you to reproduce.

Why this enormous difference?

Robert Brewer, CherryPy author, explains to me at #cherrypy:

  • well, I know Sylvain went on a tear recently to make sure we weren’t importing things too early
  • so CP 2.x -> 3.1 might be a good chunk of the difference by itself
  • I also ripped out some of the stdlib modules from CP and rewrote them myself in order to be faster (e.g. no sense importing the whole MIME module when you’re just doing http headers–a lot of the mime stuff is for email)

Of course, I owe my whole Python web programming experience to TurboGears. It provides sane defaults, especially for beginners. I learned Kid, then Genshi. I learned SQLObject, then SQLAlchemy. I also learned how to use Routes as the TG URL dispatcher. And there was the help of a large community.

But now that I am not a beginner, just using plain CherryPy could be a very similar experience.

Based on the result of this simple test, I am thinking of assembling a small TurboGears-like web framework. I would really just steal the things I really use. Make them work under CherryPy 3.1 (TG really should have upgraded.)

TurboGears 2.0 is being implemented on top of Pylons for good reasons... but CherryPy still seems to have several advantages over Paste or even WebOb, so it is not such an easy decision. At least for me.