The picture below is the documentation for Mimas, the conference submission and review system I wrote for Agile on the Beach. The documentation is three A4 sheets (2 state diagrams and one short paragraph of an assumption) plus an A3 object model – actually the database object model. All stuck to the wall in my office.
That is all the paper documentation, there are over 100 unit tests too which form pretty good documentation and have allowed me to continue changing the live system. There is also a physical stack of cards on my desk which form the backlog, although the most important changes I need to make are firmly logged in my brain.
A couple of my blogs have mentioned Mimas before but in this entry I just wanted to share some lessons about the system and my first serious programming effort in years.
For the record the system runs on the Google AppEngine, it uses standard Python AppEngine libraries, e.g. webapp2 plus a light coating of BootStrap on the UI – thanks to Sander Hoogendoom for that suggestion – and SendGrid for mail but apart from that is pretty much free of libraries. It comprises 8500 lines of Python including 3300 lines of test code. There are another 2400 lines of HTML which contains about a hundred lines of JavaScript. The JavaScript gave me an interesting insight…
I like to think I know Python but I don’t think I know JavaScript, the JavaScript I used here was 50% scrapheap challenge and 50% programming. On the few occasions I did anything more than a simple function I found I spent an inordinate amount of time getting it to work. I kept putting this down to not really knowing JavaScript – plus perhaps being a bit snobby about JavaScript not being “real” language.
But, fairly late in the process I realised why JavaScript took so much time and why Python was so fast: The Tests.
The Python was properly Unit Tested. When I needed new functionality in the server end I built it test first. And it just worked. Time and time again I surprised myself how quickly I got things working.
Not so in JavaScript, I kept avoiding it and when I did something it just took too long which made me dislike JavaScript all the more.
But… when I came to UI I’d let myself off about the unit tests. I never set up a test framework. And so I never unit tested my JavaScript either. I slowly, lately, realised a lot of my JavaScript mistakes would never have happened if I had unit tested it. And I’d also wager my UI development generally would have been faster if I had properly unit tested it. UI tests take longer, manual tests take longer, it takes longer to find out something doesn’t work and to try something else. Fail fast, fail cheap.
Lesson 1: Be even more honest with unit testing.
Lesson 2: Yes unit testing the UI is more difficult but that shouldn’t be an excuse for not trying.
To some degree I would defend myself by saying I was really new to JS and a lot of the webapp2 UI ideas and was just learning. But that begs the question: would I have learned faster with tests? At some point, before today, I think I would have done.
Although I didn’t unit test the front end I did keep a strict front-back division and pushed as much logic as I could not the back.
Lesson 3: I was already (subconsciously?) compensating for where I knew I was weak.
(By the way, Python support for mocking and stubbing is good but can someone in the Python community please document it properly? Its still a bit of magic too me.)
One of the reasons I avoided third party libraries was because of the time it took to get them working. Take dragular – for example, it is a JS drag and drop library which claims to be “so easy it hurts the brain.” I couldn’t get it to work.
I spent a couple of days with it. (I could really do with drag and drop in one part of the system).
Then I gave up and in a day knocked together a non-drag and drop alternative – not as good but it works.
Lesson 4: Even ready to use code libraries have learning curves, sometimes that learning curve is longer than doing it a different way. The cost of reuse.
Something else I relearned: those really trivial bits, the bits you easily convince yourself you don’t need to test? Like getters and setters.
Well, if you do the tests you may well find they were so trivial you didn’t pay attention and got them wrong.
Lesson 5: Unit testing the most trivial stuff can actually be the most valuable.
I say I relearned this because I learned it with a previous development a year or two back and comparing notes with James Lewis found he’d had a similar experience!
In general I’m impressed by the Google App Engine. There are some places I got stuck and a couple of things which just don’t seem to work or are not really good enough (__init__ constructors and NDB come to mind.)
Mail handling caused me a lot of problems. In some ways I can’t blame Google for this ‘cos their documents discourage you from using their own mail handling system. When I eventually did cut across to SendGrid not only was the change far easier than I expected but it also solved some other issues I had.
Lesson 6: Sometimes we resist changes which are actually good for us.
Live running the system caused surprisingly few problems. The only real problem we had in live was hitting a Google quota limit. It turned out this was a quota limit around mail sending and was eventually force the move to SendGrid.
But… The Google docs on quota limited a far from clear. Or rather, the Google diagnostics are far from clear. The logs told me I’d blown a quota but not which one. It took a lot of hunting, and some guesswork, to find it was mail – not the number of mail messages but the frequency. Even then I got it wrong because the Google quota allowances have changed over time, not only is a lot of old information still on the web but some of the limits aren’t clear or seem to contradict one another.
Lesson 7: Select may not be broken but sometimes it hard to know exactly what isn’t working.
Overall, I managed again to prove Hofstadter’s law but in part that was because I enjoyed myself so much! Really, I’d forgotten how much fun coding, and seeing the code work, is. Quite possibly, I spent too much time coding last year and not enough doing paid work.
And perhaps, this is what non-coders don’t like about coders. They see programmers enjoying their work and they fear the programmers are doing more because they are enjoying themselves. And maybe, they are right. But that isn’t a reason to make programming miserable!
In the first few weeks of coding I kept a very sharp focus on “Only what do we need for Agile on the Beach” but at some point I let myself go and enabled it for other conferences. But at that point I actually found it easy to change the system because it had been built well and built with tests.
Lesson 8: It is really hard to keep a focus on just what you need for now
Lesson 9: If you focus on good engineering broadening it out will come naturally; if you focus on broadening it out you’ll forget the good engineering.
This made me wonder if I could work as a programmer again. I can still program but I’m under no illusions: I’d never get past the CV filtering let alone an interview. Firstly nobody would hire me because I haven’t commercially programmed for over ten years. But, more importantly I don’t have the libraries or other related technologies.
To pick a Python job at random, here is the one that came up first just now on JobServe:
+ Exceptional Python knowledge/experience
No, I’m not an exceptional Python programmer
+ Java, C/C++
My Java is very very rusty, my C++ just rusty but worse, I’m a C++98/2003 guy, I’ve no idea whats in 11 let alone 14.
+ Flask and/or Django
None
+ Agile, TDD, CI/CD
I’m rumoured to know a little about Agile, and a tiny bit about TDD but CI or CD… never really done it.
Fantasy over.
Lesson 10: Middle age ex-progammers may try to recapture their youth in code.
In my next entry I’ll say a thing or two about what Mimas does, but try it yourself. I’ve also set up a little site to publicise it – ConfReview.com.
| Reference: | 10 Lessons from coding the Mimas conference submission & review system from our JCG partner Allan Kelly at the Agile, Lean, Patterns blog. |
Nice post, Alan!
Regards Ted