Waterfall-like Agile-ish Projects

In software just about all project management methodologies get labeled one of two things: Agile or Waterfall. There are formal definitions of both labels, but in practice few companies stick to those definitions particularly in the world of consulting. For people who really care about such things, there are actually many more methodologies out there but largely for marketing reasons we call any process that’s linear in nature Waterfall, and any that is iterative we call Agile.

Classic cartoon of a tree swing being poorly because every team saw it differently.
Failure within project teams leading to disasters is so common and basic that not only is there a cartoon about it but there is a web site dedicated to generating your own versions of that cartoon (http://projectcartoon.com/).

Among consultants I have rarely seen a company that is truly 100% agile or 100% waterfall. In fact I’ve rarely seen a shop that’s close enough to the formal structures of those methodologies to really accurately claim to be one or the other. Nearly all consultancies are some kind of blent of a linear process with stages (sometimes called “a waterfall phase” or “a planning phase”) followed by an iterative process with lots of non-developer input into partially completed features (often called an “agile phase” or “build phase”). Depending on the agency they might cut up the planning into the start of each sprint or they might move it all to the beginning as a separate project phase. Done well it can allow you to merge the highly complex needs of an organization with the predefined structures of an existing platform. Done poorly it can it look like you tried to force a square peg into a round hole. You can see evidence of this around the internet in the articles trying to help you pick a methodology and in the variations on Agile that have been attempted to try to adapt the process to the reality many consultants face.

In 2001 the Agile Manifesto changed how we talk about project management. It challenged standing doctrine about how software development should be done and moved away from trying to mirror manufacturing processes. As the methodology around agile evolved, and proved itself impressively effective for certain projects, it drew adherents and advocates who preach Agile and Scrum structures as rigid rules to be followed. Meanwhile older project methodologies were largely relabeled “Waterfall” and dragged through the mud as out of date and likely to lead to project failure.

But after all this time Agile hasn’t actually won as the only truly useful process because it doesn’t actually work for all projects and all project teams. Particularly among consulting agencies that work on complex platforms like Drupal and Salesforce, you find that regardless of the label the company uses they probably have a mix linear planning with iterative development – or they fail a lot.

Agile works best when you start from scratch and you have a talented team trying to solve a unique problem. Anytime you are building on a mature software platform you are at least a few hundred thousand hours into development before you have your first meeting. These platforms have large feature sets that deliver lots of the functionality needed for most projects just through careful planning and basic configuration – that’s the whole point of using them. So on any enterprise scale data system you have to do a great deal of planning before you start creating the finished product.

If you don’t plan ahead enough to have a generalized, but complete, picture of what you’re building you will discover very large gaps after far too many pieces have been built to elegantly close them, or your solution will have been built far more generically than needed – introducing significant complexity for very little gain. I’ve seen people re-implement features of Drupal within other features of Drupal just to deal with changing requirements or because a major feature was skipped in planning. So those early planning stages are important, but they also need to leave space for new insights into how best to meet the client’s need and discovery of true errors after the planning stage is complete.

Once you have a good plan the team can start to build. But you cannot simply hand a developer the design and say “do this” because your “this” is only as perfect as you are and your plan does not cover all the details. The developer will see things missed during planning, or have questions that everyone else knows but you didn’t think to write down (and if you wrote down every answer to every possible question, you wrote a document no one bothered to actually read). The team needs to implement part of the solution, check with the client to make sure it’s right, adjust to mistakes, and repeat – a very agile-like process that makes waterfall purists uncomfortable because it means the plan they are working from will change.

In all this you also have a client to keep happy and help make successful – that’s why they hired someone in the first place. Giving them a plan that shows you know what they want they are reassured early in the project that you share their vision for a final solution. Being able to see that plan come together while giving chances to refine the details allows you to deliver the best product you are able.

Agile was supposed to fix all our problems, but didn’t. The methodologies used before were supposed to prevent all the problems that agile was trying to fix, but didn’t. But using waterfall-like planning at the start of your project with agile-ish implementation you can combine the best of both approaches giving you the best chances for success.  We all do it, it is about time we all admit it is what we do.

Cartoon of a developer reviewing all the things he's done: check technical specs, unit tests, configuration, permissions, API updates and then says "Just one small detail I need to code it."
Cartoon from CommitStrip

Thoughts on Hacktoberfest 2018

This year I took part in Hacktoberfest. Partially to see what all the fuss is about, partially to get myself involved in projects I didn’t know about, and partially for the free t-shirt (which do come in men’s and women’s cuts).  If you haven’t run into this project before it’s an effort by Digital Ocean to get people to participate in open source projects. Once you sign up they count all public pull requests you make on Github toward a goal of 5. I participated both as a developer, and by tagging a few issues on my own projects so people would find them.

As a developer:

It was a great excuse to go find new projects and look at ways I can contribute.  While I’d have plenty of experience on open source projects, often they have been outside Github or are repos I have commit access to – so I don’t open a lot of pull requests on Github. That meant that Hacktoberfest was a chance to find new projects and practice a basic process for contributing code to teams.

In that regard it was a pretty good success. I opened six PRs on four different projects. Mostly they were small stuff like linting code, updating packages, or tweaking a README file.  

In terms of drawing me into projects we’ll see. I did keep up with one after I finished the 5 required (hence having six PRs), but I didn’t dive into anything truly hard on that project.  

In terms of getting me to provide truly useful code think that was limited. The largest piece of code I wrote was initially rejected so I re-wrote in a different style, and then re-written by the project maintainer the day after he accepted the PR. He was really nice about it, and it helped him get something done that had been on the to-do list for a long time, but even that was example code to be used in classrooms (which was why he was so concerned about style – he didn’t want it to be idiomatically correct for Python he wanted to clear to beginners).

It did give me a chance to play around in other people’s code bases and I did resolve some issues for people that would have otherwise lingered longer than they already had.  It also forced me to meet other people’s standards, lint to their specifications, and pass their automated tests – all good things for everyone to do now and again to see if there are solutions you like better than the ones you use every day.

As a project owner:

Once I got through the contributions I needed to get a shirt, I figured I’d look over my own projects to see if there were issues I could label for beginners to help them find ways to get started. I listed several issues are both Hacktoberfest and good first issues. Almost all the ones I flagged as good first issues got PRs opened – sometimes more than one.

I got two problems solved that I wouldn’t have known how to solve without a bit of research, and those were great. But most of the PRs were simple things that took me longer to solve collaboratively than it would have taken me to solve myself. That’s okay, in part because some of my PRs caused the same problem for their project maintainers, and because it forced me to final learn how to setup CircleCI so the code gets checked and tested automatically when PRs are opened in the future.

What I don’t expect it caused was anyone to be truly interested in the project and helping it move forward over time. So while I solved a couple small problems, I did not get new help that going to keep engaging. That made it useful as a sprint, but not useful to helping build great projects.

But even if there is room for improvement my shirt is ordered and on the way.

Arts in the Heart 2018

We got out last night to the opening night of Arts in the Heart of Augusta 2018. With hurricane Florence creeping over the region that’s probably all we’ll get to this year, but it was great to get a meal and enjoy a bit of the festival.

Update: we did manage to get back on Sunday for another meal and to see a few more performances.

Aiken’s Makin’ 2018

This weekend was the annual Aiken’s Makin’ Arts and Craft show. I took a few hours earlier today to enjoy show, eat some terribly unhealthy food, and take some pictures

Throw away information

When I was first starting out in IT I had a senior colleague who commented to me about all the throw away knowledge she’d collected and tossed over the years. She talked about learning early versions of VB, an obscure OS for an old HP minicomputer, the aging phone system we had, and all kinds of other things she’d learned in her career that was, or was about to become, useless knowledge.

Liz taught me a lot of things, but that basic observation about spending lots of time acquiring information that promptly becomes useless stands out in my mind as a useful caution. It has also become something I’ve try to plan around and compensate for as I advance.  For example I almost never bother to memorize an API: I can generally keep the documentation handy which is often updated when new features are added where my memory would lag behind.

It’s also part of why I love puzzles: they keep my brain in practice at gaining and tossing aside detailed information.  There is just about nothing as useless as the things you teach yourself while doing a jigsaw puzzle. The small details that allow you to associate pieces with parts of the puzzle. The ways you sort pieces for a particular puzzle. All kinds of things that really only apply to the puzzle you’re doing and not any other.

Puzzle underway.
The start of the cat puzzle at the top of this post as it got started. Stacks of red, white, brown, and green pieces are gathered on lower right while I finished the boarder. The wood grain and basket weave would be sorted during later passes.

There are plenty of studies that show we need to keep our brains active doing different things to help keep them healthy – some even look at the value of puzzles. Programming, and other tech work, certainly does that by default, but as any of us works with a specific platform we get used to the patterns it uses and therefore slow our rate of learning.  Puzzles also are good for finding practical applications for basic computer science concepts.

Sorting in particular is something that puzzles encourage different approaches not all of which are routinely needed in development work but are good to remember are out there. The concepts of the core of radix sorts  – or bucket sort – are extremely useful, and can even be used as a form of compression for a puzzle. My puzzle desk is a bit too small to allow me to spread out all the pieces of most incomplete puzzles, but digging through all the pieces over and over is just a waste of time.  So I often find a basic pattern I can do quick sorts by, and stack up pieces by color, or those with book covers, or other other easy to quantify detail. Then as I move through assembling the puzzle I can grab a stack of those pieces and do another round of sorting and assembly. Like a radix sort, you don’t need to first pass to be useful to make progress on the overall solution.

Puzzle pieces spread on a desk showing sample sorting collections.
The next puzzle getting started with edges sorted on the left and rough collections forming on the right. That big collection on the right is text and poster edges (puzzle is a collection of classic Star Wars posters).

Liz was right. I too learned and dropped VB; that stupidly designed phone system was mine until we replaced it with one slightly less stupidly designed; Drupal 4, 5, and 6 all had particularities I recall only to be nostalgic about the past; and the things I’m teaching myself these days about Salesforce will one day be equally useless. That’s okay, it’s what I signed up for, and if you are finding your own way in this field it’s a reality you should plan on being a part of the rest of your career.

Developers need to write more than code

Developers need to be able to in write their primary spoken languages as well as they can writing in their primary programming languages. It takes effort and practice, just like programming does. And you get rusty if you stop for awhile, just like in programming. I started blogging in part to make sure I was writing on a regular basis. When I worked at AFSC I was writing and editing everyday for my work, but the next two jobs I had didn’t expect much writing from me. I’ve written before about the value of my liberal arts education, and the last few weeks have been a testament to effort Hamilton put into making sure I learned the fundamentals of writing.

In my new job we work hard to provide a great deal of information to our clients and to make sure they can understand that information. While each team member brings a different set of strengths and weaknesses to the projects, we are all expected to communicate clearly with each other and our clients. All of us are expected to explain our parts of the project so that our clients can understand our advice and make can informed choices about how the projects should proceed.

In the last month I’ve been responsible for presenting basic overviews of platforms and highly technical reviews of parts of projects. I have contributed to large architectural design documents and focused detailed designs for small subsystems. All these materials were done collaboratively by teams of highly skilled technical people, but our client’s technical skills are much wider ranging. Often they are experts in other areas who need to use the tools we’re building. Others are experts in the technology we work with, and we are serving as added capacity to their in-house teams. We are expected to explain ourselves equally well to all these people, which means we need to both be clear and thorough.

One of the things I try to bring to the project teams is a willingness and ability to edit and be edited.  Because I spent time responsible for all the digital communications of a large organization, I have experience quickly and aggressively editing documents of all sizes. I also know how important it is to have an editor who catches your mistakes, and while I carefully review someone else’s work I am always supportive of anyone who is editing a section I wrote. Not everyone is equally comfortable with these roles, but they are things I believe all developers should try to master.

We have socially come to expect that technical people do not explain their work clearly. We are too often allowed to use a great deal of jargon, skip lightly over hard to understand details, and belittle those who cannot keep up. Complex systems are indeed complex so it does take effort to clearly explain and fully understand them. But the vast majority of developers are perfectly capable of explaining what we do when we take the time. And the vast majority of people are capable of understanding clear descriptions of our work.

Developers who are unwilling to take the time to learn to explain their work well do a disservice to their colleagues, clients, and themselves. Their work will suffer from a lack of good feedback, and so will not be as good as it could have been with more support. Those weaknesses will become bad user experiences, bugs, and other flaws in their final products.

A well done code review (one that’s meant to be supportive and doesn’t include yelling) and a good edit aren’t that different. In both one person is looking over the work of another to try to understand it and check for errors. Done well both give a chance for the reviewer to learn from the material and to help the person whose work is being reviewed do their best work.

When we explain our work well we open ourselves of up for feedback. That feedback gives us chances to validate our designs, improve the experiences of our users, and generally create more awesome products.

Families Belong Together: Augusta, GA

This afternoon my wife and I went to the Families Belong Together protest at the Unitarian Universalist Church of Augusta, and organized by a few of their members, Progressives for Democratic Reform, along with a few other like minded groups that think tearing families apart is repugnant (at best).

What else should I have asked but haven’t yet?

Every time I go through a job search either as a candidate or reviewing applications I try to learn a few things to make sure I am better prepared in the future and to help friends looking for work and talent. I recently completed a job search, so I want to share what I learned this time around.

I found a great question for a candidate to ask:  “What else should I have asked you?”

This is actually a question my wife and I started asking several years ago when we were buying large ticket items we didn’t know much about – like cars, houses, and HVAC systems. One of us, I don’t remember who, asked one of the salesman if there was anything he thought we should be asking him and his competitors. He gave us a couple small tips – likely things he thought he could answer better than his competition – and it gave us an insight into details he thought were important. Once we asked that question of all the sales people we had a list of questions to ask that covered more perspectives than we would have been able to figure out on our own.

It’s now a staple question we ask when starting anything new. Instead of trying to pretend to be experts we ask people for guidance. Often the answer is “No, I think you covered it.” but sometimes we learn things or are told about discounts, features, or services we would have otherwise missed. Outside of purchasing we’ve found the question can help spur conversations and get people to tell us things we need to know – it’s a question we use a lot as Guardian ad Litems.

Most good interviews include a time for the candidate to ask questions. This should not be a pro forma detail crammed in at the end. If the interviewer is taking your needs seriously they will give you several minutes for your questions that give you a chance to round out who you are as a candidate, this is particularly true when talking with the hiring manager (if they don’t take this seriously you should think about whether or not you want to work for that person). This portion of the interview is a critical chance to gather information about the organization, your potential role, their existing team, and vision of the future. It is also a chance to ask questions that highlight your experience and knowledge. Most advice you will find online will tell you to make sure you have a few questions you want to ask to try to draw out the information you need while showing off that you’re smart and talented. Doing this well can be hard. I discovered that having a simple, and reliably unusual, question that I can ask at the end gives a good last impression and this one has gained me unexpected insight more than once.

The exact wording isn’t important here. I’ve asked several versions:

Are there other things I should have asked but haven’t?

Are there questions you aren’t hearing from candidates that you expected?

What else do you think someone should be asking about before joining your team?

Are thing questions you would ask if you were in my shoes?

The idea is to ask an open ended question that shows you know there is always more information to be gained and gets them to think about things they haven’t shared with you, or with other candidates. The question alone often stands out, and if you get them to discuss something with you they didn’t discuss with others that helps you stand out in their minds even more. It also gives them a chance to talk about things they know and you don’t, which can help give a positive impression of you (this is same idea as dating advice that encourages getting your date to talk about themselves in part because it will make them think you’re smarter).

We all tend to want to know the same things when considering a job. This portion of the interview allows you to fill in gaps in their job ads and the conversation you’ve had so far. But since all job seekers want similar information they are asking similar questions. As for showing off, the hiring manager has likely already pre-selected a group that has shared backgrounds they are looking for so you aren’t going to easily stand out from that crowd of people with similar professional backgrounds. But by asking an unexpected question that puts the creativity on the interviewer you might be able to trigger a conversation that gives you that extra attention.

For me the question worked best in group interviews, because finding good questions is hard for me in that setting and it sometimes triggered discussion and debate within the team about things they wanted to hear candidates asking. It gave me a chance to hear a set of perspectives I wouldn’t have heard otherwise, and to see the team disagree about their vision for what’s needed. The most successful was when they fell into a mode of answer each other’s questions. For 15 minutes I moderated a discussion of what the team needed from their newest members and watched the internal team dynamics and politics played out in front of me. Usually the responses more mundane, but still helpful. Never did I feel like it was a foolish thing to have asked since the worst answer I got was a long pause and “Well that’s interesting, but I think we’ve covered everything I think you need to know.” followed by a quick check list of details the person thought it was important for candidates to know (the details of that list helped confirm why I didn’t want to work for that manager).

The question is also practical in a pinch. One of my interviews was rushed, I had just two hours to prepare after a first round interview so I didn’t have time to think of new things to ask. To add to the challenge the interviewer answered most of what I’d come up with before we got to my turn. I think I managed one or two detail questions that mostly clarified something she told me before I switched gears and asked something to the effect of “What else should I ask about before taking the job?” The question allowed me to stand out to her as asking questions that suggested I wanted to make my career move carefully (which was true and a good thing from their perspective too), and got the conversation into a productive place about the team’s role within a large organization. I start the job from that interview this week.

Escape to Savannah

We took a couple days this weekend to relax and visit Savannah.

Making the invisible visible.

This is a guest post from my wife, and co-Guardian ad Litem, Elizabeth Georgian. You can read more about our Guardian ad Litem work in this previous post.

In a world that at times seems to grow increasingly uncaring, chaotic, and impossible to change, two sets of teenagers, a century apart, living remarkably different lives, may offer us a path forward.

On March 25, 1911, 145 textile workers perished in the Triangle Shirtwaist Factory fire. They died of smoke inhalation, flames, or from injuries sustained as they leapt down the elevator shaft or out of the ninth story windows.  The factory owners had locked these young, largely immigrant women in their building so they could inspect their bags as they left and on that day no one remembered to free them.

Two years earlier, the Triangle Shirtwaist factory workers had gone on strike, as part of a larger effort on the part of textile workers across New York city.  As these young women picketed, marched, and struck, police beat them, with the approval of many bystanders. The media paid little attention, except when a small number of wealthy women joined them in their protests.

Few non-working class New Yorkers cared that the children spent their days in factories not schools, immigrants lived in dire poverty, and working conditions were hazardous.  It was only in death that they became human.

From the American Experience documentary about the fire:

Newspapers, public officials, the wider world had begun to attach names to these Triangle workers by then: Rosie Bassino and her sister Irene; Max Lehrer and his brother Sam; Mary Goldstein; the Saracino sisters; Michela Marciano, who had survived an eruption of Mt. Vesuvius before emigrating to America; Rose Manofsky, whose little sister had lost her sole source of support; and Salvatore Maltese, who had buried every female in his household: his wife Catherine, his 20-year old daughter Lucia and his daughter Rosaria, who was – at 14 – the fire’s youngest victim.

Apathy and even hostility towards the labor movement finally changed to outrage, support, and eventually significant reform as these invisible women suddenly became visible.

One hundred years later, the official charge from the family court to Guardian ad Litems charges us with conducting an independent investigation so we can make recommendations to the court.  To conduct those investigations, guardians have a right to sit in on all meetings about the teenagers we advocate for, visit their schools, talk to their teachers and doctors, and see where they live and meet with their caregivers. In doing so, I discovered to my amazement, that I have the power to make an invisible child–often poor, neglected or abused, afraid to open-up, justifiably suspicious of the system, and sometimes openly hostile–appear human. All of a sudden an angry school principle stops seeing a defiant, scary teenager in need of expulsion and instead sees a child afraid of the world and in need of help rebuilding trust.

Recently, while working with the staff in for-profit group homes, I have stumbled on the power of the language of motherhood. While I don’t actually consider the teenagers we work with children, I use that language, I describe them as my children and myself as their mother, at least figuratively.  The effect is polarizing. For a few adults, the reminder that the person they are intent on punishing is a human being and someone’s child makes them angry. But more often than not, that language de-escalates tense situations, helps me refocus conversations around the children’s strengths rather than their perceived failings, and leads us out of confrontation into negotiation or even creative problem solving.

Increasingly I see part of my role as showing the teenagers that they don’t have to be invisible.  That they have rights that deserve to be respected, needs that deserve to be met, feelings that deserve to be honored.  And seeing me stick up for them helps them see themselves as more valuable and also more powerful. Sometimes I am rewarded by watching them learn to successfully advocate for themselves and make a difference in their lives, to see themselves as powerful.

Today, the anonymous victims of textile factory fires are still poor women, still invisible, but this time we ignore them because they live in foreign places that most of us have never seen: India, Pakistan, Bangladesh, even while, like New Yorkers, we wear the clothes they make.

So who else am I still leaving invisible? Who are you making visible?