Tuesday, March 31, 2009

py2exe can't import packages that should be available (simplejson)

In my py2exe build script, I was specifying a few modules to be imported that for some reason would fail to import. I could import these from the python prompt, but the build script would give the following:

running py2exe
*** searching for required modules ***
Traceback (most recent call last):
File "setup.py", line 235, in
console=[ 'runcm.py' ],
File "C:\PYTHON25\LIB\distutils\core.py", line 151, in setup
dist.run_commands()
File "C:\PYTHON25\LIB\distutils\dist.py", line 974, in run_commands
self.run_command(cmd)
File "C:\PYTHON25\LIB\distutils\dist.py", line 994, in run_command
cmd_obj.run()
File "C:\Python25\lib\site-packages\py2exe\build_exe.py", line 243, in run
self._run()
File "C:\Python25\lib\site-packages\py2exe\build_exe.py", line 296, in _run
self.find_needed_modules(mf, required_files, required_modules)
File "C:\Python25\lib\site-packages\py2exe\build_exe.py", line 1297, in find_n
eeded_modules
mf.import_hook(mod)
File "C:\Python25\lib\site-packages\py2exe\mf.py", line 719, in import_hook
return Base.import_hook(self,name,caller,fromlist,level)
File "C:\Python25\lib\site-packages\py2exe\mf.py", line 136, in import_hook
q, tail = self.find_head_package(parent, name)
File "C:\Python25\lib\site-packages\py2exe\mf.py", line 204, in find_head_pack
age
raise ImportError, "No module named " + qname
ImportError: No module named simplejson


Apparently the problem has to do with that the modules in site-packages stored as zipped eggs would not import. Simply unzipping the simplejson egg contents so that the simplejson directory was in the site-packages directory solved the problem.

Thursday, March 12, 2009

'module' object has no attribute 'tostring' - elementtree with py2exe compiled django app

In that Django py2exe compiled app, I am making use of ElementTree. Code that worked properly on my development server was complaining about the ElementTree module I was using not having a tostring method.

Really? Since when does xml.etree.cElementTree (using Python 2.5 here) not have a tostring method?

The answer? When xml.etree.ElementTree isn't present. A number of the cElementTree methods are merely mapped to the ElementTree module, so if the xml.etree.ElementTree isn't included along with xml.etree.cElementTree in the py2exe includes, cElementTree will be missing components.

Friday, March 6, 2009

Merge/Combine two pdf pages into a single pdf page (Perl)

I needed to be able to add markup to an existing pdf file. In particular, I have a template and need to add a barcode programmatically. However, I am working mostly in Python and had already written the code to generate my barcode with placement, and wanted to just overlay that onto my existing pdf template.

With the template pdf:



And the barcode pdf:


I was aiming for:



First, I attempted to do this with the pyPdf library.


from pyPdf import PdfFileWriter, PdfFileReader
barcodepdf = PdfFileReader(file("barcode.pdf", "rb"))
templatepdf = PdfFileReader(file("freecar.pdf", "rb"))
output = PdfFileWriter()
barcodePage = barcodepdf.getPage(0)
templatePage = templatepdf.getPage(0)

templatePage.mergePage(barcodePage)
output.addPage(templatePage)

outputStream = file("output.pdf", "wb")
output.write(outputStream)
outputStream.close()

This works in my samples, but I encountered problems with my application template: my i and E characters were turned invisible. Maybe a font problem?

Here's what I was looking at:



I decided to go with the Perl PDF::API2 library. It properly handled my files and gave me the output I was looking for--with no missing characters.

#!/usr/bin/perl
#------------------#
# PROGRAM:
# mergepdfpage.pl
#
# USAGE:
# mergepdfpage.pl in1.pdf in2.pdf out.pdf
#
# DESCRIPTION:
# Combine the first page from
# two input pdf files
# into a single page in a
# new pdf file.
#------------------#

use PDF::API2;
$input1pdf = PDF::API2->open($ARGV[0]);
$input2pdf = PDF::API2->open($ARGV[1]);
$outputpdf = PDF::API2->new;

# Bring in the template page
$page = $outputpdf->importpage($input1pdf,1);

# Overlay the second input page over the first
$page = $outputpdf->importpage
($input2pdf,1, $outputpdf->openpage(1));

#Save the new file
$outputpdf->saveas($ARGV[2]);

Wednesday, March 4, 2009

Django Admin not showing tables under code compiled with py2exe

I have a Django-based project that I am building using py2exe. Unfortunately, the admin piece does not seem to like something here--my development non-py2exed version works fine, but no tables show up in the py2exe version's admin.



So what's the problem? Well, first off, it's rather hard to debug in py2exed Django:



So, after muddling through with thrown exceptions (logging would have been an option as well), I found the culprit.

In django/contrib/admin/__init__.py lives the admin autodiscover() function. The following part of this function doesn't behave as intended:


# Step 2: use imp.find_module to find the app's admin.py. For some
# reason imp.find_module raises ImportError if the app can't be found
# but doesn't actually try to import the module. So skip this app if
# its admin.py doesn't exist
try:
imp.find_module('admin', app_path)
except ImportError:
continue


I'm not familiar with the imp module, but it would seem it as though maybe it requires the python source? Or at least something more pure than the mangled code provided by py2exe.

Since imp.find_module is failing, we can just look to import the admin module for each app:


try:
imp.find_module('admin', app_path)
except ImportError:
try: #In py2exe compiled code, imp will have failed to find the admin module, so we will guess and try to import the 'app'.admin
__import__(app+'.admin', globals(), locals(), [])
except ImportError:
pass

continue


Now, it's not necessarily going to be convenient to patch the django source, so we can just create our own autodiscover function and call that.

The following goes in my util.py:


from django.contrib.admin import LOADING

LOADING = False

def autodiscover():
"""
Auto-discover INSTALLED_APPS admin.py modules and fail silently when
not present. This forces an import on them to register any admin bits they
may want.
"""
# Bail out if autodiscover didn't finish loading from a previous call so
# that we avoid running autodiscover again when the URLConf is loaded by
# the exception handler to resolve the handler500 view. This prevents an
# admin.py module with errors from re-registering models and raising a
# spurious AlreadyRegistered exception (see #8245).
global LOADING
if LOADING:
return
LOADING = True

import imp
from django.conf import settings

for app in settings.INSTALLED_APPS:
# For each app, we need to look for an admin.py inside that app's
# package. We can't use os.path here -- recall that modules may be
# imported different ways (think zip files) -- so we need to get
# the app's __path__ and look for admin.py on that path.

# Step 1: find out the app's __path__ Import errors here will (and
# should) bubble up, but a missing __path__ (which is legal, but weird)
# fails silently -- apps that do weird things with __path__ might
# need to roll their own admin registration.
try:
app_path = __import__(app, {}, {}, [app.split('.')[-1]]).__path__
except AttributeError:
continue

# Step 2: use imp.find_module to find the app's admin.py. For some
# reason imp.find_module raises ImportError if the app can't be found
# but doesn't actually try to import the module. So skip this app if
# its admin.py doesn't exist
try:
imp.find_module('admin', app_path)
except ImportError:
try: #In py2exe compiled code, imp will have failed to find the admin module, so we will guess and try to import the 'app'.admin
__import__(app+'.admin', globals(), locals(), [])
except ImportError:
pass

continue

# Step 3: import the app's admin file. If this has errors we want them
# to bubble up.
__import__("%s.admin" % app)
# autodiscover was successful, reset loading flag.
LOADING = False


And then in urls.py, where autodiscover() was already being called:


from django.contrib import admin #Still importing the admin module to map our urls
#admin.autodiscover() #We are going to use our custom autodiscover instead
from util import autodiscover
autodiscover()


Rebuild and we have admin.