Appengine
- Oct 7 2010 - lowering impedance of TDD with python mock ...
So after my post about gaeunit a few weeks ago I’ve since completely thrown out what I was doing there and moved to vanilla python unit tests. I ended up making this move for a few reasons.
1. I was never running my tests.- GAEUnit was nice, but slow. Even when running tests in parallel I still had to go through the process of opening the browser, navigating to the right place and letting the tests run. Compare this to ctrl-f9 and seeing the following:
2. AND I love seeing code coverage numbers and while it might actually be possible with GAEUnit, it wasn’t this easy…
In the end it was relatively simple once I started employing mocking to deal with sessions and realized that I could actually issue real request objects to my handlers with relative ease…webObReq = Request.blank(url)webApReq = webapp.Request(webObReq.environ)handler.initialize(webApReq, webapp.Response())For the mocking I eventually decided to go with this framework:It has so far been fine for my needs and at least my handlers have almost achieved 100% coverage. The syntax is a little foreign compared to Moq, but hasn’t been a blocker to me actually getting work done so I’d recommend it. - Aug 20 2010 - Unicode woes and Python unit testing in GAE ...
One of the really cool aspects of deploying to Google’s cloud offering (GAE) versus the more machine oriented Microsoft Azure and Amazon EC2 approaches are that you really are only dealing with computing resources. You deploy your app not to any particular server, but to the cloud itself. Despite the very real challenges in distributing work across data centers I am still filled with visions of automagical propagation and distribution and unlimited elastic computing. Beautiful. Anyway I was inspecting the logs and was SHOCKED to discover that there have been close to a thousand quotes added to system by 37 users. THIRTY SEVEN USERS!! Now, these are small numbers I know, but given the only publicity I’ve ever given this has been the two posts on this blog I was absolutely amazed to find that people had not only managed to find the app but were able to use it! (It’s quite ugly)
There is nothing quite as motivating as having users.
So I started looking at log files and discovered that I am actually throwing errors for at least some of those users who are not English speaking and are using Unicode characters. Oops. You’d think I’d know better by now. See : The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)
In most cases for these bugs it just meant calls to unicode(var) rather than str(var) . So for example when parsing url variables starting with the example provided here by Google and adding my own processing of an array of arguments to a method like so…
def update_field(self, fieldName, *args):
strings = [str(arg) for arg in args]
….
return
and then calling that code with a unicode character such as ‘’–" would produce an error that looked like this…UnicodeEncodeError: ‘ascii’ codec can’t encode character u’\u2013’ in position 0: ordinal not in range(128)
the same code as above corrected is below. There were a few variants on this but this is a good example of a function where user defined input dictated a change to accommodate.def update_field(self, fieldName, *args):strings = [unicode(arg) for arg in args]….return
So in the course of digging in and finding/fixing these bugs I also recognized I was far overdue for some unit testing. So starting with the premise that I needed a way to reproduce the issue in the lab and write a failing test for this bug I went in search of the best method to do so. Python actually has [pretty good unit test support] built-in but I was also looking for something that would work in the context of GAE. It doesn’t take much looking before stumbling on GAEUnit, which is a useful extension that provides some scaffolding for tests that will run within GAE. The major negative to this is that your tests are part of your application, but I don’t have a problem with that as it’s easy to secure them to administrator access only but for some people it’s a block.
The next issue I ran into with this was the fact that the python SDK for GAE doesn’t include a method for testing content behind authentication. So whereas in Java you can say this :private final LocalServiceTestHelper helper =
new LocalServiceTestHelper(new LocalUserServiceTestConfig())
.setEnvIsAdmin(true).setEnvIsLoggedIn(true);public void testIsAdmin() {
UserService userService = UserServiceFactory.getUserService();
assertTrue(userService.isUserAdmin());
}
But today there is no equivalent for that in python. I wasted a few hours trying to work around this issue before deciding to just login before I run tests. My login lasts for hours so this isn’t much of a constraint. In the long run I plan to drop Google’s authentication any and move to janrain/rpx to allow users to login using whatever they like. That will mean abandoning or at least shimming the infrastructure provided by GAE, but has the additional benefit of reducing my applications direct dependency on Google. Once I have my own definition of Session and session provider I can of course mock it out and control it a lot easier.
Here’s the test, no asserts but if there is a problem we’ll see an exception
def test_unicode_to_json_error(self):
badQuote = q.Quote(quote=‘hi there ’ + u"\u2013")
result = badQuote.jsonSafe()
simplejson.dumps(result)
So one error-type in the log file later I’ve spent a couple days, done a bunch of refactoring and have produced no new functionality… but I do have this!
Of course then, after all that I STILL ran into an error with the JavaScript, for which I have yet to add unit testing for. Sigh. I’ll have to choose between qUnit which I’ve used before and YUI-test which is probably better aligned to the rest of my application.
The JavaScript portion of this bug was basically because I was calling unescape() on a url-encoded variable. URL encoding though only supports 8-bit characters and doesn’t truly allow unicode, so my unicode characters became three ascii characters which could include control characters or other undesired results. There seems to be a lot of people standardizing on UTF-8 within url-encoded variables but it’s purely a convention at this point. Thankfully I found this pre-written code on webtoolkit.info that knows how to do the look ahead at the next three characters in order to handle UTF-8 input properly.
Now I hope it’s safe to say that blogquotes can handle unicode! (and has some unit tests to boot!) - Jul 19 2008 - blogquotes prototype is working ...
I’m supposed to be studying for a challenge exam I’m writing this week in “Advanced” Operating Systems. Instead I spend a good chunk of the day today working on blogquotes in between watching/playing with my daughter.
I can justify this time spent because I did all of this work exclusively from a bash shell using vi to refresh myself on some of the content for the course. In this session I was able to use wget,grep,awk,vi,a shell script and some file permissions. I’m not so sure that will get me through the exam but it was fun and I was finally able to put some time towards my random quote include for the blog.
You can see the quotes being pulled in now in the top right hand corner of this page. So far only my wife and I are using it, as this is very much a proof of concept. It works basically from end to end, but without a lot of features that are going to be necessary as this grows. Paging, searching, caching, tags, some UI polish, and some testing. It’s very gratifying though to reach this first phase and actually get something working. Adding features will probably happen a lot quicker now that I actually have a user. ;-)
I also took the opportunity to try out some of Yahoo’s client side api’s in the YUI. Currently I’m using XHR, Layout and DataTable. I was amazed at how quick it was to basically “assemble” my application. Google’s app engine makes the CRUD a total cake-walk, and yahoo’s user interface library has no dependencies on server side code but works seamlessly with a JSON backed RPC scheme in Python. It’s a whole new world! Now as long as my application doesn’t get popular and I have to start paying for resources. ;-)
Some interesting snags while working on this latest revision:- Randomly selecting an entity in GQL
- Django Utils simplejson can’t serialize google’s db.Model classes, so I had to actually proxy my model class to a simpler structure that I serialize to the client via JSON and XHR
- No unique id’s for google user accounts, all you have is email which isn’t exactly something people are going to appreciate me passing on url’s for the random inclusion widget. The solution was simple for what I needed, a user preference entity keyed on google’s User db type and storing a UUID as the publishingKey, that id now becomes my unique id which won’t change even if you change your google account name
- Google has a very cool AJAX Library SDK for sharing hosting/serving up of all the most popular frameworks like dojo and jquery
- Jun 10 2008 - Google's changing the rules again (appspot) ...
I am super excited about Google’s recent foray into cloud computing, the app engine. They are not first, I realize Amazon has been doing some pretty cool things with the S3 services and their computing services but the ability to simply and easily have an application that can leverage BigTable and GQL, google account authentication, image manipulation, memcache and having that all live within the seemingly infinite scalability of the google platform is extremely exciting for me. In fact it’s the whole reason for my recent resurgent interest in python. App engine out of the gate is a python runtime environment which is robust enough even to run frameworks like Django for your web apps.
ArsTechnica had some good comments on the service and one of their cheif concerns was being locked into the platform. I think this is not an invalid concern, but that the tools will be written to make migration from one platform to another a relatively trivial matter. From what I understand of GQL it is a subset of normal SQL with limitations like no joins which should make data/code migration easy, but scalability and hosting maybe not so much. There are open source implementations such as HyperTable which is modeled after Google’s data platform, but ultimately the article is right. Personally I don’t see myself writing large scale applications on this platform though, I see this much more as a utility model for small useful services.
I registered for the preview of app engine on the same day it was announced but not early enough to be one of the lucky first 10,000. When I did get my invite a couple weeks ago I immediately started to create applications. It’s clear this is exactly what everyone else did and I’m glad Google has capped usage at three applications per developer. Already it’s almost as bad as trying to register a domain. For each of my three applications I had to try a series of names before getting a clear one. In other words none of my ideas are at all unique. :-) On a positive note when I checked the names I was looking for there were also no uploaded applications for any of them. I predict Google may have to have an expiry date on these - I can see myself taking weeks or months to finally get anything uploaded.
http://silitrader.appspot.com (silverlight port of the old game OilBaron)
http://wherediditallgo.appspot.com (personal spending tracker)
http://blogquotes.appspot.com
I actually created that last application as I was trying to find a simple means of having a rotating list of hand picked quotes show up in the header of my blog. Back when somatose.com was hosted with LunarPages this would have been trivial, throw the quotes into a textfile, write a script to randomly choose and insert one as I was building the page. I moved from a hosted environment to using blogger so that I would stop tinkering and focus on actual content. I spent most of tonight tinkering though, building up to actually building blogquotes. (By the way I am sure there are other blog quote like tools out there, but this is a rare case where I don’t mind reinventing the wheel somewhat)
Now the nature of the tinkering has just changed. Rather than write a small shell script that spits out random lines I will actually build something of a small application/service. The quotes will be stored using the datastore api (GQL), and will be publicly visible. Other users will be able to authenticate and store their quotes in a similar manner. A widget will be built and hosted from that application that enables anyone to paste in a snippet of html to add random quotes from their personal db. I as a user can subscribe to my friend’s quotes. It’s remarkable really, the differences and yet familiarity of this new ecosystem. And while I do miss the control of simply writing a little shell script I have to say I love the idea of the internet being populated by these little widgets of functionality. The emergence of ajax and increasing power on the client means that millions of non-technical users who blog on platforms like blogger and wordpress can very easily use my quote engine. How cool is that?
Google’s API’s (and Amazon’s) are proving to be the next Win32. Massively widespread and base functionality that anyone can build on. The plumbing is there for us to be able to chain together these small tools/widgets in much the same way a clever unix user could within their shell. And once the honeymoon wears off and I actually give some though to hosting data in the US and dealing with popularity should one of my apps get hammered and I end up having to pay Google usage fees… well until all those little issues I can just be excited again about the changing landscape.
