Python
- Oct 7 2010 - lowering impedance of TDD with python mock ...
So after my post about gaeunit a few weeks ago I’ve since completely thrown out what I was doing there and moved to vanilla python unit tests. I ended up making this move for a few reasons.
1. I was never running my tests.- GAEUnit was nice, but slow. Even when running tests in parallel I still had to go through the process of opening the browser, navigating to the right place and letting the tests run. Compare this to ctrl-f9 and seeing the following:
2. AND I love seeing code coverage numbers and while it might actually be possible with GAEUnit, it wasn’t this easy…
In the end it was relatively simple once I started employing mocking to deal with sessions and realized that I could actually issue real request objects to my handlers with relative ease…webObReq = Request.blank(url)webApReq = webapp.Request(webObReq.environ)handler.initialize(webApReq, webapp.Response())For the mocking I eventually decided to go with this framework:It has so far been fine for my needs and at least my handlers have almost achieved 100% coverage. The syntax is a little foreign compared to Moq, but hasn’t been a blocker to me actually getting work done so I’d recommend it. - Aug 20 2010 - Unicode woes and Python unit testing in GAE ...
One of the really cool aspects of deploying to Google’s cloud offering (GAE) versus the more machine oriented Microsoft Azure and Amazon EC2 approaches are that you really are only dealing with computing resources. You deploy your app not to any particular server, but to the cloud itself. Despite the very real challenges in distributing work across data centers I am still filled with visions of automagical propagation and distribution and unlimited elastic computing. Beautiful. Anyway I was inspecting the logs and was SHOCKED to discover that there have been close to a thousand quotes added to system by 37 users. THIRTY SEVEN USERS!! Now, these are small numbers I know, but given the only publicity I’ve ever given this has been the two posts on this blog I was absolutely amazed to find that people had not only managed to find the app but were able to use it! (It’s quite ugly)
There is nothing quite as motivating as having users.
So I started looking at log files and discovered that I am actually throwing errors for at least some of those users who are not English speaking and are using Unicode characters. Oops. You’d think I’d know better by now. See : The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)
In most cases for these bugs it just meant calls to unicode(var) rather than str(var) . So for example when parsing url variables starting with the example provided here by Google and adding my own processing of an array of arguments to a method like so…
def update_field(self, fieldName, *args):
strings = [str(arg) for arg in args]
….
return
and then calling that code with a unicode character such as ‘’–" would produce an error that looked like this…UnicodeEncodeError: ‘ascii’ codec can’t encode character u’\u2013’ in position 0: ordinal not in range(128)
the same code as above corrected is below. There were a few variants on this but this is a good example of a function where user defined input dictated a change to accommodate.def update_field(self, fieldName, *args):strings = [unicode(arg) for arg in args]….return
So in the course of digging in and finding/fixing these bugs I also recognized I was far overdue for some unit testing. So starting with the premise that I needed a way to reproduce the issue in the lab and write a failing test for this bug I went in search of the best method to do so. Python actually has [pretty good unit test support] built-in but I was also looking for something that would work in the context of GAE. It doesn’t take much looking before stumbling on GAEUnit, which is a useful extension that provides some scaffolding for tests that will run within GAE. The major negative to this is that your tests are part of your application, but I don’t have a problem with that as it’s easy to secure them to administrator access only but for some people it’s a block.
The next issue I ran into with this was the fact that the python SDK for GAE doesn’t include a method for testing content behind authentication. So whereas in Java you can say this :private final LocalServiceTestHelper helper =
new LocalServiceTestHelper(new LocalUserServiceTestConfig())
.setEnvIsAdmin(true).setEnvIsLoggedIn(true);public void testIsAdmin() {
UserService userService = UserServiceFactory.getUserService();
assertTrue(userService.isUserAdmin());
}
But today there is no equivalent for that in python. I wasted a few hours trying to work around this issue before deciding to just login before I run tests. My login lasts for hours so this isn’t much of a constraint. In the long run I plan to drop Google’s authentication any and move to janrain/rpx to allow users to login using whatever they like. That will mean abandoning or at least shimming the infrastructure provided by GAE, but has the additional benefit of reducing my applications direct dependency on Google. Once I have my own definition of Session and session provider I can of course mock it out and control it a lot easier.
Here’s the test, no asserts but if there is a problem we’ll see an exception
def test_unicode_to_json_error(self):
badQuote = q.Quote(quote=‘hi there ’ + u"\u2013")
result = badQuote.jsonSafe()
simplejson.dumps(result)
So one error-type in the log file later I’ve spent a couple days, done a bunch of refactoring and have produced no new functionality… but I do have this!
Of course then, after all that I STILL ran into an error with the JavaScript, for which I have yet to add unit testing for. Sigh. I’ll have to choose between qUnit which I’ve used before and YUI-test which is probably better aligned to the rest of my application.
The JavaScript portion of this bug was basically because I was calling unescape() on a url-encoded variable. URL encoding though only supports 8-bit characters and doesn’t truly allow unicode, so my unicode characters became three ascii characters which could include control characters or other undesired results. There seems to be a lot of people standardizing on UTF-8 within url-encoded variables but it’s purely a convention at this point. Thankfully I found this pre-written code on webtoolkit.info that knows how to do the look ahead at the next three characters in order to handle UTF-8 input properly.
Now I hope it’s safe to say that blogquotes can handle unicode! (and has some unit tests to boot!) - Jul 19 2008 - blogquotes prototype is working ...
I’m supposed to be studying for a challenge exam I’m writing this week in “Advanced” Operating Systems. Instead I spend a good chunk of the day today working on blogquotes in between watching/playing with my daughter.
I can justify this time spent because I did all of this work exclusively from a bash shell using vi to refresh myself on some of the content for the course. In this session I was able to use wget,grep,awk,vi,a shell script and some file permissions. I’m not so sure that will get me through the exam but it was fun and I was finally able to put some time towards my random quote include for the blog.
You can see the quotes being pulled in now in the top right hand corner of this page. So far only my wife and I are using it, as this is very much a proof of concept. It works basically from end to end, but without a lot of features that are going to be necessary as this grows. Paging, searching, caching, tags, some UI polish, and some testing. It’s very gratifying though to reach this first phase and actually get something working. Adding features will probably happen a lot quicker now that I actually have a user. ;-)
I also took the opportunity to try out some of Yahoo’s client side api’s in the YUI. Currently I’m using XHR, Layout and DataTable. I was amazed at how quick it was to basically “assemble” my application. Google’s app engine makes the CRUD a total cake-walk, and yahoo’s user interface library has no dependencies on server side code but works seamlessly with a JSON backed RPC scheme in Python. It’s a whole new world! Now as long as my application doesn’t get popular and I have to start paying for resources. ;-)
Some interesting snags while working on this latest revision:- Randomly selecting an entity in GQL
- Django Utils simplejson can’t serialize google’s db.Model classes, so I had to actually proxy my model class to a simpler structure that I serialize to the client via JSON and XHR
- No unique id’s for google user accounts, all you have is email which isn’t exactly something people are going to appreciate me passing on url’s for the random inclusion widget. The solution was simple for what I needed, a user preference entity keyed on google’s User db type and storing a UUID as the publishingKey, that id now becomes my unique id which won’t change even if you change your google account name
- Google has a very cool AJAX Library SDK for sharing hosting/serving up of all the most popular frameworks like dojo and jquery
