Paying my technical debt

I’ve been involved in Toastmasters since late 2010. I volunteered to take on the Vice President Education role in my home club, the Silver Tongued Cats, soon after I joined, and looked forward to taking on a District Leadership role so that I could, eventually, become a Distinguished Toastmaster.

In 2012-13, I took on the role of Area B2 Governor. I had to look at several different reports (all of which were manually created in Excel) to figure out what was going on with my clubs. There was one report for progress in the Distinguished Club Program and another one for membership growth; it took some effort to pull all the information together, and many trees died in the process.

The following year, I was a Division Governor, and the effort to pull the info together for the 25 clubs in my Division was significantly greater – and so was the pile of paper I dealt with.

I thought there had to be a better way and started working on Python code to take the information available from the Toastmasters Dashboard and create a consolidated report. I decided to put the code onto GitHub to make it easy to develop on my laptop at home but run the real code on the web server.

Getting the information wasn’t particularly difficult (Toastmasters offers the data in CSV), nor was creating a consolidated report with the information that we needed. When I showed my report to the District Governor, she asked me if I could make it available to the entire leadership team. Of course, I said “yes”, and joined the District 4 Web Team as Statistician and Assistant Webmaster, a role that, unlike Division Governor, had no fixed term.

I didn’t want to fetch a whole year’s worth of data from Toastmasters every time I needed to create my reports, so I needed to save the data after getting it. I’d already had to do some transformations on the data (consolidating information from three separate CSV files into one object, for example), so I decided to save each day’s work as a Python pickle file, and had my reporting program unpickle all of the collected files. This was ugly code, to say the least, but it worked.

The leadership team liked the reports I generated, but they wanted more – they wanted support for District incentive programs, and, soon, support for reforming the District into two Districts. I was happy to oblige, and cranked out lots of code.

During the reformation process, I met George Marshall, the then-Webmaster for District 57 and the person behind Tools for Toastmasters. I told him what I was doing, and he made a very pointed suggestion: use a database instead of my pile of pickle files.

He was right. By using the pickle files, I had, unintentionally, built up a lot of technical debt, and the sooner I moved to a database, the less it would cost me to pay off the debt.

With that problem solved, I was able to support the District leadership fairly easily as we moved forward on the reformation, and took on the role of District 101 Webmaster and Statistician when the reformation happened.

Since I was posting the code to GitHub, I didn’t want to put things like the database credentials in the code, so I moved them to an external file, and then to a separate directory, just to make sure I didn’t accidentally post them.

And as long as I had a separate directory, I thought it would be a good place to store other files, such as the statistics I downloaded from Toastmasters, the maps and reports I created, Dropbox cursors, and so forth. Without planning it, I created a directory that had a hodgepodge of stuff in it – some was critical to the operation of the suite of code, and some was recreated every day. And there was no easy way to tell which was which.

Years passed. I wrote more code. I put more and more files in the “stuff” directory. But everything worked well enough, so I didn’t do anything about it.

Until last year, when I got serious about not being Webmaster and Statistician for Life. I put out a call for volunteers to help me with the code – when meant that I had to be able to explain the code. Most of it was easy to explain and document, but the “stuff” directory turned out to be a real problem.

So I’m fixing it.

When I moved from the pickle files to a database, I had, perhaps, five programs to fix. Today, I have more than 80 programs in my source tree; not all of them are actually in use (getting rid of the dead ones is another technical debt to pay off), but I have to look at each and every one of them.

I’ve been at it for nearly a week, and I’m about half-way through the first pass. Figuring out how to arrange things took a while, and I’ve made some false starts on the way, but now I think I have the setup under control, and adapting a program is fairly quick.

Even after I finish this project, I won’t be out of technical debt – I’m finding other problems that should be fixed, but they’re less urgent (little things, like making “verbose” and “quiet” options consistent through the code). Perhaps my successors will want to tackle them!