The Cost of Hidden Wires

In working on the Calm codebase one of the first things I did was wire in Wallaby.js because, well... it's amazing, and I absolutely love it. Having been in a mix of C# and JavaScript, Wallaby gives me the immediate feedback I find most useful in my TDD cycle. I do have a small love affair with the test runner, but... I will be using it as an example of how I used in a manner that gave me issues once I moved Calm into CI. I don't hold Wallaby responsible for it, I just hit a snag that needed to be addressed.

I've long held the belief that starting a project comes with it a special set of skills that I admittedly continue personal growth within. Wiring a project together to fit the needs of the overall longevity of it caries a set of unknowns sometimes that may or may not be obvious to you in the moment. And often that wiring is hidden away where you're unlikely to bump into it or have a need to go seek it out. Here is my simple example.

There is a small cost in time and a little bit of re-work that I had to pay in detangling some hidden wires in my specs. Wallaby provides a setup function that I had been leaning on until now. It was taking care of wiring some dependancies that each of my spec's needed... like hooking up chai and sinon, and it looked something like this:

'use strict';

module.exports = function () {
    return {
        name: 'Calm',

 ...
        setup: function() {
            chai.should();
            var sinonChai = require('sinon-chai');
            chai.use(sinonChai);
            global.sinon = require('sinon');
            global.next = function() {};
        },
...
    };
};

Having this configuration and wiring in the Wallaby configuration file meant that I did not have to import  those into every test file I created, which was great! That's the point of hidden wires... they are out of your way and do some special sauce to make you more productive. Once I spun up a test file, I could get on with the work that I truly cared about... my tests, without having to pull in that setup every time. I had even setup up my package.json to tell you that you could not run the specs from the command line if you ran "npm run test"... you would get the message:

"scripts": {
	"test": "echo \"Error: use Wallaby to run tests from within your favorite Wallaby configured IDE\"",
	"start": "node app.js"
}

I naturally hit a wall the second I began wiring up Continuous Integration in Circle Ci. Up to this moment, I hadn't had a need to run the tests from the command line. Remember, I'm still very early in the lifetime of this project, so I don't have a ton of code... it's still just basically a holding page with a few experiments in it that I've done along the way:

This issue could have been addressed in several ways, some involving more complexity than others, and I'm still deciding what I want to do long term. My immediate desire was to get specs running in CI... and CI deploying the application, that was it. And I was looking for the smallest commitment I could make in getting that there from a codebase perspective because I knew I would have my hands full enough understanding the CircleCI ecosphere.

In the short term, I've chosen to write a function containing all of the same configuration, moving it out of the Wallaby configuration to somewhere that can be exercised when specs are run from the command line, like this... testHarness.js is then imported into each spec file:

This may not be where things live long term, or maybe they are... the project still has a long way to go. This was a good first step though towards getting all the specs to run in CircleCi and deployments going out when things merge into the primary branch. As for the npm configuration to to run on the command line... it's now a punt to mocha :).

I haven't lost anything in my usage of Wallaby and CI works... a win win. My point in all of this is that the hidden wiring of pieces of our system often give us some efficiency or clarity in some other part of the system. The system is more scannable when reading the codebase, reducing some level of cognitive load along the way which is for sure valuable. Anytime I can reduce the amount of stuff I have to think about is typically a great thing. It can however come out of hiding as the system evolves though, and we need to be prepared for those types of moments. In my case it was a straightforward enough fix to make progress.

Leaning on The Pomodoro Lately

It's funny how accidentally we discover things sometimes... a lot of the time. I think it would be interesting to have a catalog of things I've discovered on purpose, and things that I've discovered by total accident. I suppose that's simultaneously one of the best things about the very nature of the web (accidental discovery) and one of the the worst things (my quest for "accidental discovery" can be a complete timesink). 

I remember sitting in my cubical at my first programming job at TRX in Atlanta, GA, scouring the web for an answer to something (probably how to formulate some regex to scrape data off of a passenger name record), when I stumbled on an article about something called the Pomodoro Technique of all things. I remember reading the article and thinking to myself, "What a novel idea... sounds interesting... I think I'll give it a try sometime." In it's most basic form the Pomodoro Technique is a method for managing your time developed by Francesco Cirillo (when he was a college student) where you:

1. Pick something that must be done.
2. Set a timer for 25 minutes.
3. Work on the task.
4. End work when the timer rings.
5. Take a 5 minute break.
6. Repeat 2-5 until your task is complete.

I think at the time, I toyed around with working on a few things using Pomodoros... but the technique didn't immediatly stick for me. I don't think I was serious enough about it, even if I had seen some benefit out of using it for a few things. 

Flash forward several years... I'm at a new job with a great group of developers that I learned a lot from at VersionOne. They had a deeply-seated tradition of pair programming, which was new to me at the time. One of the things that we all loved to experience when pairing was a sense of getting through the things we needed to get done with a sense of flow in getting through them. To that end, a couple devs liked working via a Pomodoro Timer. 

Kept with us during the 25 minutes was a sheet of paper to record anything that was important to come back to and take care of, but not germane to the task we had at hand. These two practices I have kept with me and have used on and off ever since I learned them. Sometimes I have been more disciplined about the timer, and other times not as much. I have always... always... kept the habit of a small piece of paper to record things on to come back to. As humans, we're not built to keep track of more than about 4 things at any given time. The little piece of paper is the external brain that frees me to be more creative. More on that in another blog post.

Flash forward a few more years and here we all find ourselves wrapped up in the hands of a COVID-19 based pandemic, and for those of us fortunate enough... working from home full time. Many of us also found ourselves as poor substitutes for the amazing teachers that our kids had in school. Here we were, overnight seemingly, tossed into a world where we were expected to be as productive as we had ever been at work... and also be immediately available for our kids when they needed help for something related to school... or any other creative thing that came to their minds. 

Time management and retaining a sense of flow in my work life became an immediate challenge. My kids needed help understanding some of their assignments, and I was their substitute teacher. My employer required me to be continually making progress on all things work related. How was I going to get through this without making my kids feel guilty for asking for help? How was I going to claim the time required to make meaningful progress at work? Hellooooo to tha Pomodoro!!

I've rediscovered my disciplined relationship with the Pomodoro Technique. There were several days where at the end of the day I was asking myself why had I not completed anything. It was in part the nature of the work that I was working on, and in part needing to be the awesome dad/substitute teacher my kids needed me to be. I've been using the Pomodoro Technique as I always have, really no different other than I don't really pair at my current employer. I'm just using it much more regularly lately than I had been over the past year or so. It's helped me with letting my kids know when they can snag me to ask questions they have needed help with. And it's helped me with staying focused and on task to get through the day to day coding work that I do. The family get's me for little windows of time, and I get a state of flow through my day. It's honestly been a godsend.

All of that having been said, my hope is to start utilizing some of the aspects of the technique that I haven't been using. If I'm successful in pulling them in, I'll check back here and share how it has affected my work. If you've not ever tried the technique, I encourage you to give it a shot. I think you'll be surprised at how helpful and useful it can be.

Sometimes... you take a detour

To say it's been a while since I've touched my side project (Calm) would be a complete misrepresentation of how long it's actually been. As you can see from my last post back in... um.... 2018... it's been a little bit longer than "a while". But, that's cool, life happens and I'm back at it, and that feels good 😊.

Since I've started picking back up the pieces and hints that I left myself, I've been somewhat confused by the highly technical notes that I left behind. Current me is trying hard to remember the context of some of these notes that past me left!! Nonetheless, I'm making some progress. One string of the yarnball that I started to pull on was beginning to use an environment file for some of the application level configuration (a tenet of The Twelve-Factor App). If you're unaware of the principals behind what is described at The Twelve-Factor App, I encourage you to give their site a read. As for configuration for your app, they require a strict separation of the config from the code. Things like port numbers, database connection strings, and 3rd party api tokens.

That's all well and fine! I did hit a little bit of a snag with my unit tests gaining the appropriate load of my environment variables when running my tests through Wallaby.js though. I took a dependency on the 'dotenv-flow' npm package into my app.js file, and wrote a test that ensured it was being used and that the port I was expecting was also being used where I anticipated it should be.

Perhaps not the most useful of tests I know, but sometimes I push things to the dogmatic edge just to see if I can, and to experience what challenges there are in doing so. Lo and behold, an unexpected challenge presented itself. My port was continually getting reported as undefined, regardless of what I tried. When I ran the app on it's own via 'npm run start', the port was console.log'ed as expected. I was not as lucky from within my test suite though. I was having a hard time finding an answer, and my google foo... for whatever reason... was failing me.

That was until I stumbled on this issue in the Wallaby.js repo. The Wallaby configuration file needed to load the .env file in order to make it's contents available to my testing context apparently... like this:

'use strict';
require('dotenv-flow').config();

module.exports = function () {
    return {
        name: 'Calm',

        files: [
            {pattern: 'src/**/*.js'},
            {pattern: 'app.js'},
            //list helpers for tests last, per wallaby's suggestion
            {pattern: 'testHelpers/*.js', instrument: false}
        ],

        tests: [
            {pattern: 'test/**/*-spec.js'}
        ],

        env: {
            type: 'node',
            runner: 'node'
        },

        testFramework: 'mocha',

        setup: function() {
            chai.should();
            var sinonChai = require('sinon-chai');
            chai.use(sinonChai);
            global.sinon = require('sinon');
            global.next = function() {};
        },

        lowCoverageThreshold: 99
    };
};

Now, does my use of a .env file completely satisfy the advice given the the author(s) of The Twelve-Factor App philosophy? No, I don't believe it does. However, it is a good first step that is better than having these types of things hard coded into the application itself. I suppose the next extension of their advice would be to have CI set the environment variables on the server itself that the code is deployed to. I'll tackle that one on some other dogmatic day, lol. Did my dogmatic requirement for this project having at least 99% code coverage force me to learn something new today? It did, and that's worth the effort to me.

Using 'host' while Bootstrapping Calm

I've been in the process of bootstrapping Calm (a new app I'm building out as part of my side hustle). Every time I set up a new project from scratch, I'm reminded of those things that I do only very rarely. Setting up the VM, nginx, ssh, firewalls... all of these are things I do rarely in comparison to the day to day development work that I do.

I'm always pleased to find a new small utility or tool along the way. Today I found the host command for verifying that the dns servers that I'm hitting from my house have the updated name servers I just adjusted with my domain registrar. It's an alternative to nslookup, and gives some really useful information.

For my needs, the following did the trick

host -t NS getcalm.io

The -t NS piece tells host to query for name servers, which was just the piece I needed. And yes, I was pleased to find that the new name servers had already propagated.

getcalm.io name server ns3.digitalocean.com.
getcalm.io name server ns1.digitalocean.com.
getcalm.io name server ns2.digitalocean.com.

There Is No Such Thing As A Short Term Hack

One day during a review of some changes, a new user interaction was introduced that was basically making use of a toggle in the place of a checkbox. This gave rise to the desire to make other parts of the system have one user control for this behavior. It was said that the change in one spot to use a toggle, with other spots still using the old antiquated checkbox, would confuse users. We should instead take time to understand how far the change would need to extend.

Fast forward a couple of months, one early morning I suggested to a colleague that we create an email group for our team. He said "yeah, but then you have to keep up with it when people come and go, and it becomes a big pain and get's out of date and kind of a hassle" to which I replied something to the effect of "welcome to my world and what it's like to maintain software".

A parallel to both of these is when we collectively as a team say, "Yeah, let's go with the hack for now so we can release...and then later we'll come back and clean it up." What's the commonality? Where is the parallel? All three are a point in time where a decision has to be made about maintenance in one form or another.

What does a hack look like? It comes in many forms. Oddly placed "if(...)" checks sprinkled around. Ever growing indentation levels in a method, indicating harder and harder to understand logic to get you down into those inner most levels. Duplicated identical switch statements hiding classes that are waiting to be born. Code smells sprinkled here and there, and... and... and everywhere.

This business decision of a "short term hack" leaves a bunch of garbage laying around in the code. As a team, we may have great intentions of "coming back to clean it up". But the reality is, that rarely happens... and when it does, it can be the exceptional case. Look, I get it. There is often not any perceived value of coming back to take care of it. But I as a user of that code, have to deal with that. And so do the poor schlubs that I work with that bump into that hack and are left scratching their heads.

It's confusing on several orders of magnitude larger than someone having to handle grocking a checkbox vs. a toggle, or to maintain an email list. Over time, more and more of these "compromises" cause the code to become more and more brittle. It becomes harder and harder to change. Harder and harder to understand. To reason about. The compromise is on the side of the code. The health of the artifact that should be an enabler of speed... of our ability to pivot late in the release cycle... rather than a drag on our speed, becomes more and more un-healthy.

When things begin to take longer and longer to deliver, do we ever look at one another as a team and say "well, this is because we have made some "compromising" decisions along the way that have affected the code adversely and now we're feeling it". I think that's often not the case. Sometimes it is, don't get me wrong. Sometimes we do recall those compromises. And sometimes we take the time to inject some health back into our code artifact.

But, more often is the case that we have lost all context from those far off historical decisions, and now we just feel the pain. Whether it be from too much time passing, or because people that came to the initial "compromise for a short term hack" are no longer around working on the code / on the project / with the company. Or maybe they are still around, but they are not the one staring that compromise in the face at the moment, you are...you're the poor schlub I mentioned above. And there is no way for you to recapture that intimate moment of the past where yes we were able to deliver back then... but now... now you have a story on your Kanban board that's taking forever to make it out of the "Developing" column. So the "compromise" has come back and is staring the larger company straight in the face, specifically you... your product owner... your team. And there you stand as a developer with a cracked egg all over your face. Given enough complaining room and frustration and you might just start to sound like a crotchety wind bag full of hot air about how difficult the code base is. Product owners don't care to hear about the difficulties of the code... they would much rather hear that your story is complete and your ready to take on more work.

Being honest about what it takes to keep a code base healthy is not an easy thing. It's soo necessary though. Be a professional. Do the right thing by the code, and keep lines of communication open between you and your team about what your doing and how it's coming. No, you can't fix the whole system in the course of one story. But yes, you can chew on a mouthful of it. Always making small improvements, and biting off some larger ones along the way can breath new life back into the code asset as well as your team.

Fight like hell to keep them out. Don't offer them up as an option. But (and I do mean but), if you do have add in a "short term hack" in order to deliver (let's all be honest here... it happens... delivery has to happen, it's what keeps the lights on), insist at the very least that a story is created to come back and address the hack. Leave some artifact as a conversation piece to come back to. And make sure that story has plenty of detail around it, capturing as much of the moment as possible to get you back into the context of when the "compromise for the hack" was made. And then fight like hell to play that story as soon as possible.

Short term hacks become long term hacks that harm code agility. Even more, they become longer term problems as you design more and more of your system around them... making additional compromises... predicated on the far off and often forgotten past compromises that the company as a whole made. Keep the code agile, and your company will be too.

Eventually Crappy People Show Up, and I'm Flattered

ShopToTrot has been out for a little more than a year now. We've gotten enough users for me to say that there is enough interest to keep going, and that feels really really killer. Like anything that attracts people online, some people that come through can be, well Crappy. There, I said it. By Crappy I mean that they have no intention of using the system as it was intended. They are there just to cause trouble.

Well ShopToTrot has received enough attention that we've gotten our first Crappy user that hung out just to put junk ads into the system. This one did have a sense of humor though. He or She at least mocked us a bit by putting in a photo of their "horse" (seen above) that mirrored our stock photo that we use if you don't put any pictures of your horse in (seen below).

For a moment, I was highly irritated. But that shortly gave way to flattery. Someone cared enough to come into our system and stink it up a bit. My point to all of this is simply, enjoy the Crappy people. They can't help themselves, and truly probably need a hug.

More to the point, they are a signal that you are becoming successful. And... if it weren't for them, you might not harden the system.

My Manager Hat vs. My Worker Hat

I mentioned a while back that one of the podcasts that I've been listening to is Simple Programmer, by John Sonmez. He gave some advice to a listener, and I've had it rolling around in my head since I heard it from Episode 120: Do I Have To Be An Entrepreneur To Be Successful. Just recently I've started to walk in that piece of advice.

I may get some of the particulars incorrect, but the overall gist will be in the spirit of the exchange.

John told the guy that to make progress in his extra goals, he needed to have the ability to wear two hats.  As I remember them, they were:

1. A Manager Hat 
2. A Worker Hat (I lovingly call this one the Doer or Executor Hat so if you see me use those alternate forms you’ll know to what personality I’m referring to). 

The Manager Hat:

This is the hat that you wear when you're deciding what should be accomplished by your Worker. So effectively, what he was saying is that you need to take some time to plan out your week. Be deliberate about setting goals for what would get accomplished during that week so you can keep sight of them. I've chosen Sunday night at 9:30 for this hat to adorn my head.

The Worker Hat:

The other hat he mentioned you needed to pick up is a Worker Hat (or a Doer Hat). This is the hat you put on once the Manager Hat comes off. You wear this hat for the rest of the week. Once this hat comes on you simply execute the plan unquestioningly. You don’t deviate from it, you just simply do what your Manager has planned for your Doer.

Once you get through the week, wash, rinse, and repeat for the following week.

In Practice (as I have experienced it thus far):

As I have tried to operate in this, I’ve discovered that my Manager always has unrealistic expectations of what my Worker can accomplish. Big surprise right? That’s often the case in the real world, outside of this little dual personality experiment that I’m doing. So what does that really mean you may ask. Well, for me it’s meant that what my Manager has scheduled for a particular day might not be what my Worker has been able to accomplish for that day. And there isn’t anything scheduled for this thing that my Worker hasn’t finished on the next day coming up.

Sooooo, my Worker had to negotiate with my Manager. The Worker had to tell the Manager, “Hey dude, I’m not done. I haven’t been slacking, it’s just that it’s taking me longer… or life has happened and inserted itself in the middle of your pre-scheduled routine, thus disrupting the schedule."  If you’re anything like me, you’re trying to accomplish these things in the midst of working at your day job, raising a family, and potentially working on these things with other people. Any one of those three things tend to try and get in the middle of your Worker accomplishing what your Manager has laid out. It happens, we’re human, and plans may need to change.

So how have my Manager and Worker adjusted expectations of what will get accomplished. Well, the Worker starts out the week working diligently on those things scheduled by the Manager. And if the Worker isn’t able to complete a day's tasks, then the unfinished task/s get pushed to the next day by the Worker, and the things that were scheduled for the next day get pushed accordingly through the week. The Manager has placed a bucket at the end of the week to catch anything that gets pushed completely out of the week, and things are pulled from that bucket for scheduling the next week.

This way, the Worker has some sense of continuity and doesn’t get disgruntled with not having completed anything at the end of the week, but has spent it thrashing about trying to accomplish everything (when that might not be possible). It’s important for progress and the Worker’s feelings and sense of accomplishment to complete things and get them behind him. If the schedule needs to be adjusted then so be it, the Manager needs to compromise.

This has made the things that I've been trying to accomplish more concrete. And by "more concrete" I mean that there are very real time slots for work to slip into, and expectations on myself that those things scheduled (and only those things scheduled) should be completed.

Life is going to happen. Your plans will get disrupted. You need to be able to adjust, and in the adjustment not loose focus on what your goals are.

In the closing, figuring out the balance between your Manager Hat and your Worker Hat can be Detective work for a while. But in the end you can find some happy medium and hopefully some cadence as to how much you’ll be able to accomplish in your upcoming week.