Deploying a Django app on the desktop
Posted on May 31st, 2007 in Catalyst, Product design, Tool, django, python by siddharta || 33 Comments
One of the cool things about Silver Catalyst (which is a Django app) is that you can start using it right out of the box. I didn’t want the team working their way around Apache and MSSQL configurations, Python version incompatibilities, database access issues and deployment hassles. The final solution was a simple executable, which when run would start everything required to get going.
In this post, I’ll explain how that was achieved.
A bit of background first. The vision for Silver Catalyst was a tool that could be installed by a team without having to get centralised servers and system administrators involved. I wanted the team to be able to set aside one computer on which they could install Silver Catalyst by themselves for use on their project. That made it pretty important that the setup be absolutely painless.
Silver Catalyst was written in Python using the Django framework. My first thought was to package the application as a zip of compiled python files and then have the team deploy that onto a server. This meant that someone would have to setup a web server, get Django working with it, setup a database, install python, deploy the application and finally configure it to point to the right files. Phew!
Luckily, to the rescue came cx_freeze. cx_freeze is a really nifty utility that will take your python code, check the included libraries and package everything along with the current python version into an executable. Very cool! A bonus was that you would not even need python installed to run the tool since the interpreter is packaged in the executable. This also eliminated issues like python version incompatibilities and so on.
Step two was the decision to use sqlite3 as the database. Since sqlite3 is an embedded database, you don’t need to have a database installed at all. Even better, there are standard modules to access sqlite3. In fact, it has been made a part of the standard distribution from Python 2.5 on.
The only dependency remaining was the web server. If I could use a pure python web server, I could simply hook up Django to it via WSGI and include the whole lot in the executable. That is what I did. I used ToofPy with some modifications so that it is pre-configured to run Django.
When cx_freeze is run on this setup, it bundles together the web server, Django, sqlite3, the application code, and the python interpreter into an executable. The application is pre-configured to use the packaged sqlite3 database code and the web server is pre-configured to load Django.
The result?
Just run the executable and the server starts. Access the server via a browser and you go straight into the Silver Catalyst application. In fact, because there are no external dependencies, it is easy to think that it is a native application. But behind the scenes, it is all python and some cx_freeze magic. If you want to see for yourself what the final result was like, you can download a free version of the tool here.
June 2nd, 2007 at 1:51 pm
[...] Deploying a Django app on the desktop (Silver Stripe Blog) With the craze about Google Gears, this is another approach to do offline web applications. Silver Catalyst is a ’scrum tool’ that runs as a server. The server is packaged as an executable. Works great, just tried it: clean interface and interaction. (tags: django scrum webdev offline apollo googlegears) [...]
June 2nd, 2007 at 4:56 pm
One quick question:
Any reason why you used cz_freeze instead of py2exe (http://www.py2exe.org)? Py2exe seems to be more up-to-date (latest update on 2006-12-30) and it works great with the scripts I used it with so far.
June 2nd, 2007 at 9:29 pm
Very interesting!
Can we get some more details on how to do this ? How to organize a project, possible caveats etc ?
Thanks!
June 3rd, 2007 at 9:32 pm
Yeah py2exe was one choice. cx_freeze uses the techniques in py2exe among others, so I decided to try it first and it worked pretty well for my application.
The cool part is that cx_freeze automatically checks for imported modules and includes them in the package. The one caveat, and its a big one is that it obviously cannot find modules that are imported at runtime by using the __import__(..) function.
There are two ways around this.
One is to put all dynamic modules in the same directory (or a fixed relative directory) as the executable so that python can find them when they are imported. The other option is to statically include these modules somewhere.
I actually use both techniques. I have some plugin modules that are loaded dynamically. I leave these at a relative path from the executable because I want to add and remove these at runtime.
Django uses dynamic imports in some places, eg: determining which database code to use. I’ve replaced this with a static import because I know beforehand that I am going to use sqlite3.
I also have a file called imports.py that has statically imports various modules that cx_freeze was unable to detect. Then in the main script I say “import imports” and when cx_freeze is parsing it finds all these modules and includes them.
June 14th, 2007 at 8:40 am
I’ve been making my own executable, also, albeit using py2exe and the Django admin server. I don’t suppose you could post your ToofPy integration code at Django Snippets?
June 15th, 2007 at 2:48 pm
Hmm yeah I suppose I could to that..
June 28th, 2007 at 12:23 pm
Excellent and very inspiring writeup!
July 5th, 2007 at 2:06 am
siddharta,
Thanks for the write-up – very interesting. Did you ever get around to posting the ToofPy integration code to Django Snippets ?
Martin
July 5th, 2007 at 9:25 am
Just did it – http://www.djangosnippets.org/snippets/301/
It’s 2 lines of code
July 9th, 2007 at 11:08 pm
[...] goodies that come with working in Python. Topping the list are the really cool utilities like cx_freeze that allow me to integrate a web server, database (sqlite3), django and my application into one [...]
July 19th, 2007 at 4:50 am
[...] Silver Stripe Blog » Blog Archive » Deploying a Django app on the desktop The vision for Silver Catalyst was a tool that could be installed by a team without having to get centralised servers and system administrators involved. I wanted the team to be able to set aside one computer on which they could install Silver Catalyst by (tags: django tools packs development python) [...]
September 27th, 2007 at 5:08 am
Hi, thanks for the idea. Since Django is distributed with its own lightweight web server, what is the reason you went with ToofPy?. Cheers.
September 27th, 2007 at 4:56 pm
Hi Jack, Django’s server is just for development use. They don’t recommend it for production.
October 10th, 2007 at 8:09 am
I am trying to do something similar. What was the exact command line params that you passed to cx_Freeze to make sure that it picked up everything including the django apps under the main project directory?
October 10th, 2007 at 8:56 am
This is what I did:
FreezePython –install-dir=../dist manage.py
And this is what I get when running the app:
Traceback (most recent call last):
File “/home/nsayeed/Downloads/cx_Freeze-3.0.3/initscripts/Console.py”, line 27, in ?
exec code in m.__dict__
File “manage.py”, line 12, in ?
File “/var/lib/python-support/python2.4/django/core/management.py”, line 1299, in execute_manager
project_module = __import__(project_name, ”, ”, [''])
ImportError: No module named manage
October 11th, 2007 at 12:30 pm
Hi nomadjourney,
When doing FreezePython, it does not package the modules that are imported dynamically at runtime using __import__ because obviously the software does not know beforehand what is going to be imported. What you need to do is to create a file, say imports.py and import all the dynamically loaded modules there, like so.
cx_imports.py
————-
import manage
import xyz
…
mainfile.py
————-
import cx_imports
This way cx_freeze will be able to find the required modules and package them in the exe.
The other option is to put the files in the same directory as the exe so that they can be found dynamically at runtime. Use this option if you don’t know beforehand which modules might be loaded.
October 11th, 2007 at 12:36 pm
So just to give an example, a part of my cx_imports.py file looks like this -
import os, sys
os.environ['DJANGO_SETTINGS_MODULE'] = “your_project.settings”
import your_project.your_app.models
import your_project.your_app.views
import your_project.urls
import your_project.settings
import your_project.manage
import django.template.loaders.filesystem
import django.template.loaders.app_directories
import django.middleware.common
import django.contrib.sessions.middleware
import django.contrib.auth.middleware
import django.middleware.doc
import django.contrib.auth
import django.contrib.contenttypes
import django.contrib.sessions
import django.contrib.sites
import django.core.cache.backends.simple
import django.db.backends.sqlite3.base
import django.db.backends.sqlite3.introspection
import django.db.backends.sqlite3.creation
import django.db.backends.sqlite3.client
import django.template.defaulttags
import django.template.defaultfilters
import django.template.loader_tags
As you can see, I am statically importing the sqlite3 modules because thats what I am using, so although Django supports other databases, I’m not importing them here. The exact combination of files you need to put here depends on the modules your app uses.
April 10th, 2008 at 10:17 pm
Siddharta: I’m trying to package a django application using cx_freeze and I am running into error. I hope you can help me.
WindowsError: [Error 3] The system cannot find the path specified: ‘C:\Documents and Settings\Joseph\Desktop\SOL\sol.exe\django\db\backends
I know this is not a support forum for cx_freeze; but as you’ve already packaged django using it, I am just hoping you can help.
Thank you very much for your time.
Joseph
April 11th, 2008 at 7:12 am
There is a part in the django code where it tries to open django/db/backends to search for the list of installed backends. This happens because it’s not able to import the database code for the backend you specified. Check out __init__.py in django/db directory for this bit of code.
Django uses dynamic imports to import the backend. Since cx_freeze cannot resolve dynamic imports, you’ll can put a static import somewhere in your code to tell cx_freeze to include this module. See Comment 17 above. The example above uses sqlite3 backend. Replace those imports to whichever backend you are using.
May 31st, 2008 at 11:12 pm
I got FreezePython (cx_freeze) working fine on a hello_world.py program, but when I follow your instructions on a skeleton django project, I get the following error from running FreezePython on mainfile.py:
Module cx_imports from file cx_imports.py has invalid syntax (cx_imports.py, line 3).
That is, the os.environ[] command appears to have invalid syntax. I changed ‘your_project’ to ‘mysite’ everywhere applicable. I’m not sure what else to do!
May 31st, 2008 at 11:14 pm
btw, I am just doing everything with the Django development server for now, and there are no databases involved
May 31st, 2008 at 11:55 pm
Ok, I found the problem … Beware that if you copy-paste from the instructions in post number 17 that there may be a non-ASCII character hiding in there somewhere
June 3rd, 2008 at 11:04 am
Hi Ben, you’re right. Wordpress changes the formatting of some characters like the quotes and dashes to special characters (like curly opening and closing quotes), so when copy-pasting, we need to be a bit careful that the right quotes and dashes are used.
June 26th, 2008 at 7:17 pm
Thanks Siddharta.
This was a good starting point, although I ended up using CherryPy (instead of ToofPy) and Py2Exe (instead of cx_freeze).
I ended up doing something more based on Joseph Jude’s work http://www.jjude.com/index.php/archives/70 – itself inspired by your post here.
Anyway I’ve posted my notes here – http://misunderstandings.wordpress.com/2008/06/26/django-desktop-app/
July 8th, 2008 at 6:16 pm
[...] me to Siddharta’s site he was doing what I wanted but without the detailed instructions a simpleton (me) [...]
October 31st, 2008 at 5:21 am
Has anyone seen OpenSway (http://opensway.googlecode.com) It looks promising, but still in early stages. I downloaded it from svn and with some tweeking got it to work. It seems like a cool project though, and it’s based off XULRunner
November 1st, 2008 at 11:42 pm
Just had a quick glance at OpenSway. Looks interesting. Thanks for pointing it out.
March 9th, 2009 at 6:03 am
do you have the source code for the wiki in 15 min.
I am just starting with py and dj.
another question, if you were to design a site that become very popular, do you think dj database query is optimized to handle huge traffic?
thx,
webguy
March 13th, 2009 at 9:10 am
I dont have the source with me anymore, but you can follow along in the screencast and get it – http://showmedo.com/videos/series?name=v7kABKL6R
Note that some parts need to be updated for the latest version of Django
About the django queries, a lot of high traffic websites use it, so I don’t think it should be a major issue.
September 14th, 2009 at 3:07 pm
Hi siddharta,
again a question using cx_freeze for compiling a django app, maybe you could give me a hint …
I got pretty far following your suggestion with providing a file with static imports for all dynamically loaded modules.
However it seems that commands like ‘runserver’ (and all the others) are not available in the compiled django app even though I listed those django modules in the import file. And from the log I can see that they are found by cxfreeze. To me this smells a bit like an initialization problem, what do you think?
./manage runserver
Unknown command: ‘runserver’
Type ‘manage help’ for usage.
Thank you very much for your help,
Ralph
October 5th, 2009 at 3:23 pm
Ralph,
What happens with the management commands is that Django dynamically loads the commands only at runtime by looking at the command files installed with each app. So you also need to include all the command files in the static imports, otherwise Django doesn’t load the command.
December 8th, 2009 at 10:16 am
Various of guys blog about this matter but you wrote down some true words!!
December 12th, 2009 at 7:30 pm
Could you share the size of the executable created?
thx
RS