Category: Mistakes

Still a B Student

Now and then something happens to us all that reminds us we’re not perfect. I’ve written about the strengths of B Students before, and well last week I proved that sometimes I still have room to improve.

When I started this blog I set the goal of posting once a month. I have stuck with, or exceeded, that target except for two months since 2016. I missed September 2019 and last month. Last time around I set the goal of four posts in the following month, so that is my established case law.

I did have a piece written that I had planned to release. But it needed another editing pass. Before I got those edits done a project at work took over my life. I will have those edits done for next weekend, and I already know the general topic of the post what will follow it.

Last time the increased pressure to write actually resulted in some descent pieces. I wrote that B Student piece, which I quite like, in part as a response to that failure. Two others from that month are how-to articles that continue to get good readership years later.

Hopefully this will be an equally good month (although this post probably isn’t the best example of that).

Being Nice vs Being Kind

A few years ago during a job interview at a nonprofit organization, the Executive Director asked me about the culture of the job I was leaving. He had noticed that I was stressing the importance of good feedback during my interviews and wanted to know why it was so important to me. Indeed, part of why I was job hunting was the job I had rejected the notion of mistakes – everyone’s work was always good. I told him that, and that I do my best work when I get honest feedback. He responded by drawing a distinction between being nice and being kind. He was sure everyone I worked with was quite nice, but they didn’t sound very kind to him.

Being nice when giving feedback just means saying positive things.
Being kind when giving feedback requires helping another person understand how to improve.

It was one of those moments in life where someone offers you words to explain a struggle you’re having and suddenly things make more sense. I knew I was unhappy in the job, but his insight brought into focus the reasons I wasn’t fitting in. Being told my work was great did not mesh with my understanding that it’s important to admit we all make mistakes and find ways to avoid repeating them. My desire to examine my own work caused conflict, let alone my efforts to introduce peer feedback.

The CEO at that job once told me that if people heard their work wasn’t perfect, they wouldn’t want to come in the next day. He didn’t seem to realize that getting dishonest feedback causes the same thing. And seeing no chance for improvement was even worse. There was no room to revisit problems that lead to near failure and ongoing technical debt for the client. Instead they convinced clients the technical debt was a feature, and used it to sell support contracts.

All the feedback my colleagues and I got was nice to hear (“This is great.”, “The client will be thrilled”). But unrelenting positive feedback didn’t help us improve as individuals or as a team. Our leadership gave team members a false sense of success and held the company back from improving.

The thing about being kind, is that sometimes you have to tell people things they don’t want to hear. That can be hard, and to do it well requires you to care about the other person. If want people to like everything you say, it will be hard to be kind.

Giving nice feedback involves offering a string of hollow platitudes that leaves people with a false sense of achievement. That makes people vulnerable to failure when they move forward without understanding their weak foundation. Giving kind feedback requires you to provide an honest assessment of a person’s work and helping them recognize errors. Kindness requires listening to people when they push back, and finding ways to help them find ways to excel as they move forward.

When all was said and done the opportunity wasn’t the right fit for me. Turning down their job offer was probably the hardest career decision I ever made. But they and I left good enough impressions with one another that I recently started serving as an volunteer advisor to the program I would have been running. That service has allowed me to reaffirm my sense that it was the right decision not to take the role both for me and for the organization.  And it has given me a chance to help give them some kind feedback to help – I hope – improve their operations. I try to be as kind to them as an organization as they were to me as an applicant.

In Praise of B Students

There is nothing wrong with just being good at something.

When I was in school I was a solid B student. I thought of myself as a B student and I worked hard to make sure I was always working at least a B average. I worked hard, but not too hard. Likely I could have been an A student if I’d worked harder, but I would not have been as happy. I didn’t feel the drive to be the best all the time, just good enough to keep up with what I was learning. Most people brag about being A students, but that wasn’t me and it has worked out pretty well in the long run.

For me a B student is someone who not only gets actually Bs most of the time in school but also knows how to work hard to get the grade they want, and how to relax so they recover and enjoy their lives along the way. If they fall a little short at some point they are perfectly capable of working a little harder to make up the ground. Life is not a sprint, it’s a marathon, we all have to save a bit for later. If you are used to running full out all the time, you do not always have the ability to step it up a little extra when needed nor how to relax and recover without stopping. In a world where we seem to be moving from helicopter parents to snowplow parents, there are still people that have fallen short from time to time, so they know how to pick themselves up and push forward.

There has been plenty written and said about the flaws of perfectionism so you don’t need to trust me, and I’m not going to waste time making the case that there are problems with pushing kids and college students to be a perfect A student all the time. And there are problems when companies only want to hire people who have those backgrounds.

In our current test-driven education culture there is a lot of push for kids to be the best. And the highly competitive nature of admissions for some top colleges can make it seem like a kid has to be perfect to succeed in life. But life isn’t Top Gun there actually are points for second place, and really more or less all the others (well paychecks and some level of happiness anyway). More importantly people who are used to a bit of failure can excel when faced with tough jobs that require a lot of on the job learning and trial and error work. That describes pretty much every modern office job, and lots of others – watch a plumber work on an old house and you’ll see plenty of trial and error being required to solve even basic problems.

Failure is a part of life. We all need to know how to acknowledge we made mistakes, how to recover, and how to ensure we don’t make the same mistake again in the future. People who earned a lot of B’s in school are familiar with all those things. We sometimes got lazy and ended up with a C instead of B – to stay a B student we then worked harder and got ourselves enough A’s to offset our missteps and bring our averages back up. Sometimes we really didn’t understand something and had to struggle through to make sure we stayed on pace.

B students also often know the difference between learning and getting the right answer on an assessment (a skill I would have told you I had but did not need anymore until I starting having to pass professional certification exams on a regular basis). We learned how to pass the test without knowing all the information, but knowing enough to get the grade we wanted. Getting a perfect score on an assessment is only useful if the assessment is perfect – most college professors and anyone who has taken a certification exam can tell you there are no perfect assessments. I had to pass tests in college, and I need to complete and maintain certifications now, but I don’t need to have every piece of information crammed into my brain all the time – that’s what Google is for. Once I pass, I’ve passed – it feels better to get more right answers, but as long as I get enough to pass I’m certified.

When I was in the 6th grade we had a whole day dedicated to studies skills taught by a guy named Mr. Gallagher. He wasn’t a regular teacher at my school, and I have no idea where they found him or what happened to him later, but some of what he taught us that day has stuck with me both in terms of skills and philosophy. He emphasized the need to understand how a test was written, both in terms of following directions so you didn’t give up silly points and so you know where to focus your time (hint: time goes first into the places you are likely to earn the most points), and when to guess and when to work through an answer carefully (again be greedy about points not the number of right answers).

Mr. Gallagher was the first person to put the idea into my head that the point of going to school is to get good grades instead of learning. I can still remember our teachers pushing back on that argument the next day, and I eventually arrived at the conclusion that they were both right: the purpose of school is to learn, but the purpose of tests is to get good enough grades so you can move forward in school. The thought exercise itself helped me see that learning material and passing a test weren’t actually always related. I had a teacher later who helped me further abstract this concept when he pointed out that he could easily write tests we all passed or all failed, so the purpose of the test writing was to find a measure of what he thought we should be able to do (that teacher and his wife were recently on Marketplace – thanks Bill that simple lesson was super useful in college and beyond). If pass and fail are at the whim of the test author, and they aren’t actually playing against me personally but really tracking a group of people, then I can play it as a game where I need to get myself into the top 80%. If I learn to play at that level reliably I’m going to do pretty well and I don’t have to stress being perfect all the time.

Yes, there are things in life that you cannot do unless you are one of the best. Not to mention the fact that the playing field isn’t level, which means some people have to work a whole lot harder than others to earn the same B and a report card full of Bs doesn’t get read the same for everyone. But we cannot all be the best, and pretending that’s not true doesn’t level the social playing fields.

And of course there are plenty of moments we need to strive to be as close to perfect as we are capable. But we also have to hit deadlines and get things done even if we know the work will be less-than perfect. Anyone who works in software development has to deal with this on a regular basis. As much as we may want to be perfect, shifting project requirements, project timelines, platform weakness, and our own mistakes all get in the way. We do the best we can with the skills, time, teams, and the tools we have. We need to know how to accept that we cannot fix every flaw and still get things done well enough to ensure the project is successful.

So when you are hiring people consider looking for people who already know how to be second best. People who know how to do really good work, but also know how to take risks, try something new, and in the process fail. People who know how to get things done well enough to be successful, even if it’s not perfect. We aren’t perfect, our work isn’t perfect, our world isn’t perfect, so don’t pretend your team has to be perfect.

Authors Note: Last month I failed to post to this blog for the first time. If you look back over the record you will see at least one post each month from the time the blog starts through August 2019 – then I missed September. Oops. I could make all kinds of excuses (I really did get sick right at the end of the month so my planned post was delayed a week), but the truth is I had time and material, I just didn’t get anything done. So now to make up for it I am trying to post every weekend this month.  I missed a goal by a little, so now I compensate to make up for it – then I’ll hopefully go back to my nice reliable routine. What can I say? I’m okay with a B.

Preparing for your next crisis

Can your plan handle the bizarre?

Last winter Dries, the Drupal Association, and the whole Drupal community, stumbled when concerns about a leading contributor’s potentially exploitive relationship got caught up in discussions of Gorean subculture and related sexual behaviors (warning researching this topic will quickly lead you to NSFW information). The intriguing details drew in an ever-expanding audience but were actually irrelevant the main concerns and the secondary ones that followed. The DA lost control of the message and the story and the entire community suffered as a consequence. Last spring and summer I was asked to step in to help them regain control of the message and start to resolve the crisis. It wasn’t the first time I was part of a crisis response with unusual details, and likely won’t be the last.

The first time I saw an organization respond to a threat to their reputation was my senior year at Hamilton College when I was intern in the Communications and Development Department. The morning of February 5th, 2001 the administrative assistant who spent every Monday morning scanning major publications for references to the college and its professors (this is before you could use Google Alerts and other tools for a similar purpose) suddenly jumped up from her desk and ran into her boss’s office. A few moments later they both sprinted down the hall to the Vice President’s office. She had found an article in the New York Times Magazine titled The Cloning Mission; A Desire to Duplicate featuring then-chemistry professor Brigitte Boisselier who – unbeknownst to the college – was moonlighting as the research director for Clonaid trying to develop human cloning technology as part of her leadership of Raëlianism.

Yup, the first time I had a front row seat to an organization’s crisis was a college learning from the New York Times that they had a professor doing secret human cloning research for a group that believes aliens created humanity.

Within a hour they had a preliminary message prepared for any alumni who called – and they were calling – and got it to all alumni class presidents to share with any concerned classmates. By noon they had held meetings with all the needed decision makers including the college president, the chemistry department chair, the deans, and the VP of Communications to form a plan of action. By early afternoon that turned into a more formal statement that was recirculated to the class presidents and anyone else expressing concern to the school. Over the course of the semester, and the two years that followed (she continued to make world news after she left the college), they responded to repeated media inquires – I was one of several interns in a mock audience for b-roll footage of a CBC piece on the topic – and even got her to engage in a public debate with an ethicist from the Philosophy department (which went poorly for her). In the end the concerned alumni were pleased with the college’s handling of the matter and the school’s reputation remained intact.

Eventually every organization will face a crisis that requires a public response. Depending on the nature of your organization you may already have a plan that handles the obvious situations: like schools preparing for threats to their students or political campaigns preparing for sexual harassment claims (at least they all should be prepared regardless of what their candidate tells them). But in my experience most of the time the crisis that actually emerges isn’t what you expected and includes strange details that easily distract everyone from the main issue.

Hamilton did not have a plan titled: “What happens if one of our professors is caught leading human cloning research for an alien cult.”  What they had was a general plan for “What happens when it appears someone is going to make us look bad” that was quickly escalated to “what happens when it really is bad.” They knew who they needed to get into a meeting, and that allowed timely decision making. The communications team then had a basis to work quickly so they could get back in control of the story. The plan they had allowed them to brush aside the unimportant details – the involvement of Raëlianism was fun to talk about but didn’t change anything about the response – so they could focus on important details.

Often when faced with these kinds of strange details surrounding a crisis the people who should be leading the response get distracted and start talking about those details. When it gets really weird they really want to say “is this not my fault and not my problem” and ignore it. It doesn’t matter why just that it is your problem. Everyone will want to focus on the salacious details, but you have to focus on the important details and lead your audience to supporting your response.

These are my tips for how to plan for a crisis and building a response that allows you to stay focused:

  1. Know when to initiate your response. Develop a list of people and metrics to use to initiate a crisis response. It should include both some clear markers – e.g. a threat of violence against staff or constituents – and some fuzzier signs – e.g. anytime the ED/CEO or board chair says there is a crisis.
  2. Know who needs to be involved and how to find them. Part of what allowed Hamilton to move quickly was they knew exactly who needed to be involved in the process. Likewise you should determine which staff, board members, volunteers, consultants, etc, need to be in the loop and how you reach them quickly. And know who gets cut out when – if your board chair is on a cruise and can’t be reached until next Tuesday who do you call instead? If your director of communications is the problem you’re probably better off not having them planning the response.
  3. Prepare different types of response. You may want to monitor the situation while saying nothing; you may want to target different messages to specific audiences (your board may get a different message than your donors); you may want to try to use press contacts or avoid them. You should think about how and when to use all your communications channels as part of your response.
  4. Have metrics for testing your plan before following it. In a time of stress it can be easy to overlook important details in your response. While planning write up a checklist to run through to make sure you remembered everything that’s important. It should at least include a reminder to think about everyone’s physical safety and your legal risk (in that order). It should also probably include contacts with important constituent groups (like Hamilton’s alumni class presidents) and any internal audiences so staff and volunteers aren’t surprised by public statements.
  5. Be prepared to share aggressively. Often organizations appear to be hiding information when they share it slowly, or when they say “this is all we can release” and then are pressured into releasing more. With any given statement release everything you can – sometimes with supporting materials if you need to – and then stop.
  6. Be prepared to shut up. This correlates with the previous tip. Sometimes part of a good response will be to be quiet. Usually this will be right before and after a major statement. It gives people a chance to process your response and avoids the sense that you are leaking information in dribs and drabs.
  7. Don’t over plan. Your next crisis will not look like what you planned for, so be prepared to change course from your very first move. If you lock in to many details you will likely make your situation worse by sounding tone-def.

When you have created your draft plan you should practice using it. Some of that practice should be very practical: what do you do when a man in leadership is accused of sexual harassment or abuse? Some of it should be like the CDC and FEMA zombie drills; even if you don’t use zombies use a crisis that you think couldn’t possibly happen – if you hear someone in the office say “good thing that can’t happen here” use it as your scenario. The first makes sure you are prepared for the kinds of things that are most likely to actually happen. The second makes sure you are prepared to think outside the box and handle a messy situation to which you thought you were immune. Your organization might not do any medical research at all, but what would happen if you discovered a board member was marrying the next director of Clonaid (including aliens is almost as fun as including zombies)? What would you tell your major donors?

A good crisis response comes from being prepared for the unexpected. If your plan is flexible enough to handle the utterly expected and adapt to a staff member cloning alien zombies, you can probably handle whatever actually comes your way.

Writing Good Directions

Last fall I wrote about the importance of writing good documentation, and part of writing good documentation includes writing good directions. I have a pet peeve when it comes to poorly written instructions of any kind, but unfortunately I’m still learning to do it well myself.

Writing directions can be thankless: you know you provided good directions when people use them and never complain about them. If you write bad directions everyone who gets stuck complains about your work – and usually not nicely because you left them frustrated.

A greyhound wearing a vest labeled donations
No one ever asks where to put money when Leo wears his donations vest.
Good directions, like good recipes, cover all the steps you need to follow, are easy to read at a glance, and provide extra details to help you stay on track. If I’d written up my Drupal Cake recipe as a large block of text without formatting no one would even recognize it as a recipe let alone be able to follow the steps.

Sign with arrow pointing left label Cake and one pointing right labeled No Cake
You don’t have to ask to know where to get the cake.

Over the last few months at work we’ve been updating our development workflow. It started with a large project to migrate our code repositories to Bitbucket and move all support clients onto new testing infrastructure. With a large number of support clients, we had lots of updates to do, so we shared among all the developers. I did the first few conversions and then wrote up a set of directions for other developers to use. The first few times other people walked through them I got corrections, complaints, and updates, and then after a few edits there was silence.  Every couple of days I noticed another batch of clients got migrated without anyone asking me questions. The directions got to be good because no one struggled with them after the first few corrections. But I didn’t get, or expect, compliments on them, but they achieved their purpose.

Switch for a microwave labeled No fish, no curry.
This is from a hotel, but every office should probably have one on their microwave. I doubt the person who created those labels hears about them much unless someone broke the rule and microwaved fish in their room.

It’s easy to complain about directions, but it’s hard to do them right. There is another set of directions at work that I know are bad: because everyone complained about them and then gave up on the process they explain. I need to try again, but frankly it’s hard to get up the motivation to replace the current silence with either new silence (if I succeed) or complaints (if I fail).

Usually when I’m writing up directions the outcome doesn’t matter much. If your Drupal Cake isn’t the shade of blue you were hoping for, or my colleagues have to ask a couple extra questions while migrating site configuration, the world will not end. But there are people who have to write important directions that can save or cost lives.

Even if your directions aren’t signs that hopefully save lives, it is worth trying to do them right. I’ve already admitted I’m still working on getting this right but here are a few things that help me.

  1. Write down the steps as you do the task. Include pictures or screenshots when they are helpful.
  2. Do the task again following your directions to the letter.
  3. As you edit them (because you will find mistakes) add tips about what happened when you made mistakes during your previous attempt to help people know they are off course and how to recover.
  4. Repeat 2 and 3 until you are sure you have removed the largest errors.
  5. Watch someone else follow your directions and see where they get confused – if the task is complex they will get confused and that’s okay for now. Ideally this person should have a different experience level than you do.
  6. Edit again based on the problems that person ran into.
  7. Repeat 5 and 6 until you run out of colleagues willing to help you or you stop finding major errors.
  8. Release generally, and wait for the complaints.

This of course is an ideal. It’s what I did for the migration instructions, but not what I do most of the time. Rarely do we have the time to really work through a process like that and edit more than once or twice. You can shortcut this process some by limiting the number of edits, but if you don’t edit at all you should expect people to complain to the point of giving up.

One last thing. I’ve often been told the first part of writing good instructions is mastering the process. I disagree with that advice pretty strongly. Most of the time I find that beginners write better instructions since they are paying attention to more details. Once I master a topic I skip steps because they are obvious to me, but not to people who need the instructions. That’s why for step five you want someone of a different experience level, someone more junior to help make sure you didn’t forgot to include the obvious and someone more senior to point out that you made mistakes.

Welcome Piscataway Students. I noticed the start of the annual bump in traffic from the Piscataway LMS and figured I’d say hello. A few years ago one of the teachers started to assign this article, so I see a bump in traffic here when they post the assignment.  Tell them I said thank you, and that I’d be interested in hearing from them about how this article gets used in your classes. I hope this year goes well for all of you.

Fixing the Expert Beginner

I’ve been reading Erik Deitrich’s blog a bunch recently, particularly two pieces he wrote last fall on how developers learn. They are excellent. I recommend them to anyone who thinks they are an expert particularly if you are just starting out.

Failed Sticky bun
I am a good baker but this attempt at a giant sticky bun failed because I am not an expert.

In short he argues for a category of developer he calls the Expert Beginner.  These are people who rose to prominence in their company or community due more to a lack of local competition than raw skill.  Developers who think they are great because they are good but have no real benchmarks to compare themselves to and no one calling them out for doing things poorly.  These developers not only fail to do good work, but will hold back teams because they will discourage people from trying new paths they don’t understand.

I have one big problem with his argument: he treats Expert Beginning as a finished state. He doesn’t provide a path out of that condition for anyone who has realized they are an expert beginner or that they are working with someone who needs help getting back on track to being an actual expert.

That bothers me because I realized that at times I have been, or certainly been close to being, such expert beginner.  When I was first at AFSC, I was the only one doing web development and I lacked feedback from my colleagues about the technical quality of my work.  If I said something could be good, it was good because no one could measure it. If I said the code was secure, it was secure because no one knew how to attack it. Fortunately for me, even before we’d developed our slogan about making new mistakes, I had come to realize that I had no idea what I was doing. Attackers certainly could find the security weaknesses even if I couldn’t.

I was talking to a friend about this recently, and I realized part of how we avoided mediocrity at the time was that we were externally focused for our benchmarks.  The Iraq Peace Pledge gathered tens of thousands of names and email addresses: a huge number for us. But as we were sweating to build that list, MoveOn ran out and got a million. We weren’t playing on the right order of magnitude to keep pace, and it helped humble us.

I think expert beginnerism is a curable condition. It requires three things: mentoring, training, and pressure to get better.

Being an expert beginner is a mindset, and mindsets can be changed. Deitrich is right that it is a toxic mindset that can cause problems for the whole company but change is possible. If it’s you, a colleague, or a supervisee you have noticed being afflicted with expert beginnerism the good news is it is fixable.

Step 1: Make sure the expert beginner has a mentor.

Everyone needs a mentor, expert beginners need one more than everyone else. A good mentor challenges us to step out of the comfort zone of what we know and see how much there is we don’t understand. Good mentors have our backs when we make mistakes and help us learn to advocate for ourselves.

When I was first in the working world I had an excellent boss who provided me mentoring and guidance because he considered it a fundamental part of his job. AFSC’s former IT Director, Bob Goodman, was an excellent mentor who taught me a great deal (even if I didn’t always admit it at the time). Currently no one person in my life can serve as the central point of reference that he did, but I still need mentoring. So I maintain relationships with several people who have experiences they are willing to share with me. Some of these people own companies, some are developers at other shops, some are at Cyberwoven, and probably none think I look at them as mentors.

I also try to make myself available to my junior colleagues as a mentor whenever I can. I offer advice about programming, careers, and on any other topic they raise. Mentoring is a skill I’m still learning – likely always will be. At times I find it easy, at times it’s hard. But I consider it critical that my workplace has mentors for our junior developers so they continue progress toward excellence.

Step 2: Making sure everyone gets training.

Programming is too big a field for any of us to master all of it.  There are things for all of us to learn that someone else already knows. I try to set a standard of constant learning and training. If you are working with, or are, an expert beginner push hard to make sure everyone is getting ongoing training even if they don’t want it. It is critical to the success of any company that everyone be learning all the time. When we stop learning we start moving backward.

Also make sure there are structures for sharing that knowledge. That can take many different shapes: lunch and learns, internal trainings, informal interactions, and many others. Everyone should learn, and everyone should teach.

Step 3: Apply Pressure.

You get to be an expert beginner because you, and the people around you, allow it. To break that mindset you have to push them forward again. Dietrich is right that the toxicity of an expert beginner is the fact that they discourage other people from learning. The other side of that coin is that you can push people into learning by being the model student.

Sometimes this makes other people uncomfortable. It can look arrogant and pushy, but done well (something I’m still mastering) it shows people the advantages of breaking habits and moving forward. This goes best for me is when I find places that other colleagues, particularly expert beginners, can teach me. Expert beginners know things so show them you are willing to learn from what they have to offer as well. They may go out and learn something new just to be able to show off to you again.

Finally, remember this takes time.  You have to be patient with people and give them a chance to change mental gears.  Expert beginners are used to moving slow but it feels fast to them.  By forcing them into the a higher gear you are making them uncomfortable and it will take them time to adjust. Do not let them hold you back while they get up to speed, but don’t give up on them either.

Why I won’t wear your free t-shirt

With DrupalCon coming up I want to talk about a question I will be asking vendors giving out t-shirts: do you have women’s shirts? I’ll then request a men’s large (since it’s the size and cut that actually fits my body). Given the reminder this week about the problems in the Drupal community around misogynism this seems appropriate topic.

Close your eyes and picture a Drupal developer (go ahead I’ll wait). I can’t say who you saw, but too many sales managers for technology companies just pictured a man or group of men. They want those developers to think well of their company and to wear shirts with their logos. They will buy, and in a few weeks give away, t-shirts for the developers they just pictured. They will forget that our community is strong because we’re diverse, and gender is a critically important form of diversity for us.

Thinking about this through t-shirts isn’t a new or original idea. The first time I ran into a form of this discussion was this 2006 blog post from Kathy Sierra (which I probably read in 2007, so I’ve been thinking about this for 10 years). I was a big fan of her work at the time (and was until she was driven from tech circles shortly after), and the support she got from women on the topic was almost as influential on me as the attacks she received from men for daring to talk about her own body. And it’s not just tech: if you listened to the entire Planet Money t-shirt saga you hear the NPR staff complain about the same problem.

For me usually the conversation starts something like:

“Do you have t-shirts for women?”

“Yes, do you want one for your wife or girlfriend?”

“No, I’ll take a men’s large.”

[confused stare from salesperson followed by them giving me a t-shirt.]

Like many tech conferences, there are usually lots of chances to get free t-shirts from various vendors. For anyone who hasn’t thought about it, that t-shirt is meant to make me and you into walking billboards. It puts their brand in the minds of people in your office, neighborhood, and anyplace else you go. And it works. After I got home from New Orleans last year I wore shirts from several different hosting companies everyday for a week. Since we were talking about new hosting relationships at the time, I got questions about the company on my shirt each day – we do business with two of those companies now. So giving me a free t-shirt is a request for me to advertise on their behalf which could easily pay for itself.

A few years ago I was at DrupalCon with a female colleague who was attending for the first time. The all male dev team members had attended several times before that and she had always liked the interesting variety of t-shirts they came home with, and so was looking forward to finally getting something for herself. Only she didn’t.

Very few vendors had women’s shirts at all and none in her size. Even worse, DrupalCon itself didn’t have a t-shirt for her because the men who had checked in before she arrived had thought their female partners would like the design and had taken all the woman’s shirts.

I should have known this was a risk for her since it’s not like the topic was new, but I didn’t and while I felt bad about it, that doesn’t undo the insult.

Instead of making sure she got to fully participate in our ritual, we allowed her to be left out. Several years had passed since the Kathy Sierra triggered discussion and I wrongly assumed we’d made progress as a community. Oops.

As part of trying to apologize to my friend for my own cluelessness, and in part to try to do something to make our community better, I took up asking vendors with t-shirts if they have women’s sizes. Not because I want one for someone else in particular but because I want other people in general to have the same chances I have. And I am not willing to advertise for companies if they are making our community worse. Anything we do that makes women feel less welcome makes our community worse.

Over time I’ve had interesting conversations with people who have said both yes and no.

The people who say yes are sometimes interesting. Usually they are just confused because they never thought the issue through. Sometimes they cite Kathy Sierra, sometimes they cite a similar experience to my own. Usually we’ll exchange thank yous and I’ll move on.

If they say no, the conversation goes a bit differently. I am always nice, but I don’t let them off the hook. I’ll ask “why not?”  With one exception the answers are unfulfilling.

The most common answer is “I don’t know.” While the salesperson is sure no offense was meant (as if they would tell me if one was), I generally offer to wait while they call their boss to figure out why they don’t want women to advertise for them (no one has taken me up on it yet).

The next most popular answer (usually offered right after saying “I don’t know” didn’t get me to go away)  “It’s too expensive to add those sizes.” Typically this is from a person who gives out hundreds or thousands of t-shirts a year. This just isn’t true at that scale. Most vendors are still going to order enough shirts to get over the same price breaks they would have with their current order, and every t-shirt they give a woman is one they didn’t give a man. The only way for this to really be true is if they end up giving away more total shirts because more people want them, which is the whole point of the exercise. The time a RackSpace employee gave me this answer I just stared at the woman, in a woman’s cut RackSpace t-shirt, and said “Really”?  The next year RackSpace had women’s t-shirts (this probably had nothing to do with me, but it’s nice to think it might have).

I’ll also sometimes get “We didn’t have room to pack them.” I usually get this answer from smaller vendors who may not have as many to give out as a company like RackSpace or Pantheon, and are therefore concerned about bringing shirts and not finding a person to take them. But I’ve gotten it from big companies too who will admit shipping five or six boxes of shirts. Either way, sorry, but no, avoid insulting potential customers by splitting your packing space.

The final answer I hear is “Women don’t want them.” This is the go to excuse from men who don’t want to think about why women don’t participate in anything related to technology. I even got that response from a man planning a Drupal Camp when I pointed out that none of the women on the planning committee had shown any interest in having shirts at all – his daughter being one of them. When I commented that maybe they didn’t like shirts that didn’t fit a wave of nodding followed: we had women’s shirts made.

If you are attending DrupalCon, or any other large event for that matter, please ask everyone giving out t-shirts about women’s sizes. If they say yes, take whatever you want, but if they say no I encourage you to think hard about if you want to advertise for them. And draw them into a discussion on the topic.

If you are bringing t-shirts to DrupalCon, or any other large event I attend, please bring t-shirts for as many different types and sizes of attendees as you can fit. The right ratio is a hard problem to solve, and I know you have limited space. Ask the conference for an estimate of attendee demographics and make your best guess. If you guess wrong that’s okay, apologize, and try to do better next time.

Do I think refusing to take free t-shirts from tech companies will suddenly solve all of our the gender issues? Of course not. But it does force people into conversations they aren’t used to having, and it makes at least some stop and think about ways we create unfriendly atmospheres for women in technology.

Good Enough Passwords

I deal with passwords a lot. In any given day I log into five or six servers, another dozen web sites, plus my personal systems and tools. Some are stored in password managers, some are memorized, some are mine, and some are shared. To avoid losing things I need I have several patterns, schemes, and password generation tools I use to try to keep up with it all and make sure I’m using good passwords most of the time.

I don’t have a deep background in cryptography and a definitely don’t consider myself an expert in the related math. But I have spent a lot time time with users and observing behaviors, so I do consider myself reasonably knowledgeable about how people actually behave with their passwords.

A couple years ago this XKCD comic came out, and an admin’s view of it has become one of my measures of their understanding of users and passwords:

XKCD Password Strength Comic

If you’re interested in understanding what technically right and wrong with the math and assumptions in that comic you might find some of the references on explain XKCD’s discussion interesting.

I don’t actually think that’s the interesting part. To me the interesting part is the number of system administrators I have talked to who are convinced the comic is wrong, because they are also convinced that anyone who doesn’t use truly random password and avoid password reuse is stupid and therefore their behavior can be ignored. These people have access to your system, and it’s your job to keep those system secure. The problem isn’t your users, the problem is we have made passwords unreasonably hard to do right.

At this point people generally know they are supposed to use good passwords, like we all know we’re supposed to brush after every meal. Sure there are people that do that, but not most of us. We all know that passwords should be long (even if we fight over the exact length to require) and use lots of different kinds of characters (although many password systems require them to be from the latin character set). But it’s too hard to follow all the rules, and security experts are so concerned about being right they don’t provide useful guidance about when the cheat.

Let me grant those who like long-random passwords the following point: if you use different passwords on every system that needs one, and they are all truly random strings, and you memorize them all so you don’t have them recorded someplace they could be stolen, you have the hardest for an attacker to crack. Great for you. But I work with people who are not perfect, have limited memories, and need to be able to have shared access on a regular basis.

Knowing the perfect random password generation pattern is useful in some cases (or so I’ve heard), but rarely are you in a case where you can use the perfect setup. I don’t care about perfect: I’m not perfect, I don’t work in a perfect office, have perfect colleagues, or perfect clients. So here are my good-enough rules for admins and developers.

0) Make it easier to do the right thing than the hard thing. This gets to be rule zero, because everything else is meant to support this idea. You want the path of least resistance to be the one that gets you the results that are secure enough to protect your systems from the attackers they face. Make sure your users have good tools for storing passwords, settings on password fields to encourage good (not perfect) behavior, and a minimum of stupid rules you don’t really understand but someone told you are “best practice”.

1) If you make it hard for people, they will find a way around you and likely weaken security. It might be post-it notes on monitors, cycles of passwords that are 5 long (because you force them to not use the last 4). If you make it hard to pick a password (because you required punctuation but not  ‘, \, “, &, or !), you will end up with lots of passwords that are curse words – and your attacker will thank you for shrinking the search space. If they are using touch devices to type them, they will do things like repeat as many characters as you allow to make it easier to type (if you ban any repeats: again the attackers thank you for shrinking the search space). All things you want them to stop doing.

2) Do not have a maximum password length. Any time I hit a system with an upper bound of 12 I want to scream (although jokes about chimps might be a better tactic). Even if you are using a secure hashing system that ignores all characters after some point: who cares? Why limit the attacker’s search space to only strings between 8-12 characters?!? Sure that’s a massive search space, but not nearly as big as it could be.

3) Do have minimum lengths. Minimum lengths forces your users to do two things. First, not use passwords that could be broken in less time than it took you to read this article. Second, it gives you leverage to push them to either good phrases or generators. If you’re smart and don’t have legacy systems to support go with something like 15 or 20 characters.

4) Expect people to share passwords. Many times this is actually a basic job function. If I won the lottery tomorrow (unlikely since I don’t play) and don’t come to work (also unlikely since I would wait until I had the money reinvested before making plans) the person taking my place needs to be able to access all the tools, servers, and accounts I’ve setup. If she can’t do those basic things I haven’t done my job responsibly.

5) Provide secure means to share passwords. I have more than once been sent a password in chat (running through Google, Slack, or once upon a time AOL’s servers), email, word documents, text file, and a variety of other terrible solutions. This happens not because my colleagues didn’t know it was a bad thing to do but because they didn’t have a good option. We spend so much time locking down passwords, that we don’t create secure channels to hand them around responsibly which defeats the purpose of secured storage.

6) Pay attention to how users will be using individual passwords. Not all passwords are created equal, which is why I encourage you to support throw away passwords: something short, easy to remember, and only used places it doesn’t matter if it were stolen. But even when a password is important there are issues like the ease of entering them: if I have to enter a password 4 times a day it better be easy to type or I better be able to copy and paste it. If I need it once a month it should be impossible to remember and its okay if it takes me 5 minutes to get it right. Most of us can’t type complicated passwords quickly, and if we have to enter it a bunch we want to be fast.  This is even more true for people using touch interfaces where shift is an extra keystroke as is changing to a different part of the standard keyboard.

7) Stop telling people they have to use a different password every time. This is an extension of number six. People have too many passwords, and that’s not changing soon. Sure we can encourage them to use LastPass, or a tool like it, but most people aren’t going to (and if they did that could be its own problem since it creates single points of failure). Tell them to use a different password when it’s important, and to use a throw away password or scheme when it’s not.

Not everything needs to be Fort Knox so stop pretending it is.  Important things like your bank account(s), your email, Facebook need their own passwords because they can be used to do real damage in the real world. Online communities, games, and other trivial places asking you to sign in do not.

8) Don’t lecture people about bad personal password habits. Honestly, this is probably the hardest one (here I am lecturing you about not lecturing them). Usually the first people to admit they are sloppy about passwords are developers and sysadmins. Sure, they will tell you about the awesome password wallet they use first, and the two factor authentication they created for their blog, but then toss off that all their production servers have the same root password and it’s 8-10 nonrandom characters. Even if you are perfect (if you are still reading this by definition you probably have room for improvement) don’t lecture people who aren’t. It just makes them feel they can’t admit when something has gone wrong, or if they don’t understand something. When you find people doing it wrong, show them how easy it is to do it right, and if it isn’t apologize and fix it.

Responding to Drupal Break-ins

If you support any web site long enough you will suffer a break in. If you support lots of web sites you will suffer them more often than you’ll want to admit in public. A few weeks ago my number came up again in the attack lottery when we discovered a client’s web site was being used as a proxy and redirect to a fake shoe site.

It wasn’t the first time I’d suffered a break in, and unfortunately I don’t expect it to be the last. My last experience with a major break in was shortly after Drupalgeddon (I patched all the clients I was supporting before they were breached but had to clean up sites that weren’t patched by other vendors), and the attackers had learned a few new tricks in the meantime.

If you are responding to a break in on a Drupal site there are directions on drupal.org to help guide you through an attack response, but I thought it might be helpful to talk through a version of what response can look like in practice. I think it’s also useful for us all to admit our weaknesses from time to time to help us all make sure we’re making new mistakes.

Overview

At the outset I’m going to admit we never found the initial source of the attack, what we did find were the tools they placed after the break in. The most likely cause was poor server patching practices by the client’s host, but there were also some Drupal security patches that had been slow to be get installed as well. During the attack I worked with members of the Drupal Security team (particularly Greg Knaddison who generously provided feedback on this article as well – of course any remaining mistakes are mine), who were helpful in giving me suggestions and who were clearly interested in helping us make sure we resolved the problem.

The site was being used as part of a scam advertising network. The attacker was leveraging the reputation of the site to create records in search engine indexes that were redirects to a fake shoe sales site. There were also a number of tools placed on the server that gave them full access to the Drupal database and the ability to run arbitrary PHP scripts. And it was clear by the end they had placed additional backdoors we never found – they may have had full control over the OS as well.

How we found out

Google told us.

We got an alert from Google reporting SPAM content on the site. At first we couldn’t find the content they were talking about, which unfortunately slowed our escalating our response, because it was only directly available to search engines. The junior developer who was initially assigned to review the message from Google eventually figured out how to find the listing on Google (a Google site search for Nikes and some hash codes the attacker was using), but couldn’t figure out how out it got there, and escalated the task to me.

Once I saw what she’d figured out my stomach sank. At first I was still hoping there might be some other explanation, or some simple matter of a single user account getting exploited, but that seemed unlikely (since we couldn’t find the content on the server) and I quickly knew it was going to be a mess.

Initial Response

The first thing I did was make sure we had a copy of the exploited site: code, files, and database. I would have rolled the site back to a recent backup, but our five-day rolling database snapshots were not enough to get back to before the attack began. We spun up new virtual machines for myself and another senior developer to start reviewing copies in environments isolated from other work.

Since the URLs we had for testing were a fairly unique pattern we started to Google those – and we got lots of hits. As soon as we knew the problem was larger than our site, we opened an issue with the Drupal security team and started to feed them all the information we had gathered. While their practice is not to get involved in resolving attacks directly (their role is to ensure the security of Drupal core and contributed modules), they were supportive and helpful in suggesting places to look for problems and resolution strategies.

Attacks we found

By the time I was alerted to the problem there were already several malicious tools installed, some of which I’d seen versions of before, and some were new to me – all were designed to be hidden from sight through some simple but effective obfuscation. Over the course of the next couple of days I found several backdoors manually, wrote tools to help me find more, and played entirely too much whack-a-mole (more on that in a bit).

There were two main categories of attack I was chasing: PHP scripts scattered around the public files directory, and records added to Drupal’s database tables.

Database table exploits

If you dealt with sites in the aftermath of Drupalgeddon, or other hacked Drupal sites, you have probably seen what happens when an attacker inserts PHP into carefully targeted parts of a Drupal database. In the ones I’d seen before attackers replaced the callback functions in Drupal’s menu_router table with PHP of their own. In this case the attacker used the Block module’s ability to use PHP to place a block to provide themselves a way to execute arbitrary PHP by sending a post request to the server. They leveraged the fact that the main system block is always available and therefore is a reliable place to insert a backdoor. By posting a form with a specific form element they were able to execute arbitrary PHP and therefore use that to place additional malicious code.

The attacker also leveraged Drupal’s system table to get more complex attack code loaded. They created a record for a file to be loaded as a module and then uploaded that file to the site’s files directory where they were guaranteed Drupal had write access.

filename: sites/default/files/styles/medium/public/57h3d21.jpg
name:overly
type: module
owner:
status:1
bootstrap: 0
schema_version:0
weight: 0
info: a:11:{s:4:"name";s:6:"overly";s:11:"description";s:58:"Displays the Drupal administration interface in an overly.";s:7:"package";s:4:"Core";s:7:"version";s:4:"7.32";s:4:"core";s:3:"7.x";s:7:"project";s:6:"drupal";s:9:"datestamp";s:10:"1413387510";s:12:"dependencies";a:0:{}s:3:"php";s:5:"5.2.4";s:5:"files";a:0:{}s:9:"bootstrap";i:0;}

This was the script doing the redirects and filtering traffic so that pages only appeared to search engines. Usually these records have filenames that are .module, .php, or .inc files, but in this case it was a .jpg file named to be similarly to actual files on the site to make it hard to spot.

The content of that file was a PHP script not an image. The script did several things, and was the main tool the attacker was actively using during the time we were trying to stop them. It served as a simple proxy of content that they would present to the search engines, and redirect those same pages to the scam site for anyone else. It also provided code to make sure the content of user login forms was sent to the attacker, and a backup backdoor incase some of their others were lost.

We actually had to remove this particular attack more than once (always using the misspelled “overly” module) and each time it came back with a new file, and each time using a different but similar disguise to try to make their code blend in with legitimate files.

.htaccess files in public files

The other trick that was new to me (and a more aggressive stance by Drupal core on this approach is being discussed) was to take advantage of the .htaccess patterns in Apache to re-enable PHP execution within the public files directory. Drupal’s default .htaccess file disables PHP at the root of the public files directory and in theory all subdirectories, but that can simply be undone by a malicious .htaccess file (unless you block it in Apache’s main configuration – which in my opinion defeats the purpose of using .htaccess).

The attacker had placed a number of basic PHP-based exploits on the server using this technique to allow them to run the scripts. The tools themselves were not Drupal-specific, and likely the .htaccess file would work just as well on a number of other PHP-based CMS platforms.
Since the files directory gets deep and complicated there is no reasonable way to scan the whole thing by hand: particularly since several of the files were using inaccurate file extensions (like .jpg or no extension at all) and file names meant to blend into the background. So in addition to checking for any .htaccess files below the files directory root, I wrote a simple Python script to scan a directory for anything that includes the string <?php:

How we fixed it

We immediately made sure all code on the site was up-to-date, and I removed every exploit I could find. And for a couple days I played whack-a-mole with the attacker. Every day I would remove a series of exploits, disable their ability to redirect users to their scam, and every night they would break back in through a backdoor I’d failed to find.

The final solution was to replace the server, deploy a version of the code known to be good, and deploying copies of the database and files that have been scanned for any PHP in places it shouldn’t be – which involved a combination of the scanner above and hand checking every place the database stores PHP, not a fast process.

What we will do better next time

Part of any security event of this nature needs to be a full review of your internal processes and controls to make sure you reduce your risk and improve your response the next time something occurs (because unfortunately there will be a next time for all of us).

One of our first areas of improvement is a shared understanding that it’s more important to resolve the attack than determine the cause. This goes against the question that developers are constantly asked during and after an attack: “How did this happen?” While you need to know something about what happened, in the end it’s more important to make it stop. I still don’t know what happened that started the attack, I know I stopped it by blocking every attack vector I could think of and replacing every part of the stack with a version known to be fully up-to-date. It would have been faster and cheaper if I’d just started there: yes there is a risk I would have missed some of the code in the database if I hadn’t taken the time to review what I was finding, but frankly I doubt that risk is has high as the risk that new exploits will appear while I’m working to understand the previous one.

Beyond that basic shift in approach we developed a three part list of improvements:

  • Things we needed to improve right away.
  • Things we needed to improve soon.
  • Things that should be part of ongoing improvement.

The highest priority items were coming up with better internal process for initial response, and making sure we are deploying all security updates in a timely but still careful manner, including monitoring our hosting partners to ensure servers stay up-to-date as well. These are basics that are easy to let slip over time – particularly monitoring that your partners are doing their job correctly.

The second category of fixes is filled with workflow and procedure improvements. We were already were in the process of improving our code handling (migrating from SVN to Git, better production monitoring, more internal code review, etc), and we accelerated our plans to complete that work. This category also includes a complete review of our existing backup procedures to make sure they provide the level of coverage our clients need.

The final category of longer term adjustments includes tasks that include ensuring all developers are given (and expected to take) professional development opportunities around security best practices, doing more internal sharing about emerging ideas and trends, and encouraging more community engagement so we are better able to leverage the community resources in a crisis.

Yes I wear pants and other advice for working from home.

I’ve worked from home part-time or full-time for the past four years. The first question I’m always asked when people learn I work from home is “Do you wear pants?” The answer is yes, I wear pants to work even when no one can see me. And frankly I don’t quite understand the desire of large numbers of people to work without pants, but it led me to realize that it might be helpful to share a few tips for people starting to work remotely.

Wear Pants

Okay, it doesn’t have to be pants, but get up and get dressed in clothes you don’t mind being seen in. The clothes we pick communicate, not only to other people but also to ourselves. Pick clothes to help you take your work seriously.

Set a routine

Have a routine about when you start, and what you do to prepare yourself before you start for the day. Your commute might be a walk down the hall instead of a drive across town or a ride on public transit, but most people I know still benefit from a pre-work routine. Walk the dog, go for a run, eat breakfast, or some other basic activity. It doesn’t matter a whole lot what it is, but have a routine that gives you a few minutes to get settled into a frame of mind to be focused on what you are going to do at work.

Make sure the technology works

You never want to have to say “I can’t do that from here.” or “I’ll do that next time I’m in the office.” This is particularly true when you have a part-time remote arrangement (i.e. if you work from home 1 or 2 days a week), and the systems may not be setup to fully support remote workers. Push hard to get the technology fixed so you can do everything from home at least as well as you can in the office.

Have back-channels

Make sure you are set up with ways to informally communicate with colleagues. Whether that’s Slack, HipChat, Google hangouts, or some other tool, make sure there is a way to discuss totally unimportant stuff to maintain your personal relationships with your colleagues. When times gets stressful it’s critical to have good will built up and important to have a way to work out issues in private.

Have an office

It doesn’t have to be a nice office, but have a space you go to for work. Make sure it’s setup well for your work, and make sure you don’t spend lots of non-work time in that space. Don’t use the room with your TV or other simple distractions. Ideally you want a space with a door you can close so you can block out any other people who are around and focus.

Have boundaries

Like a morning to routine you need to have boundaries about when you are not working. We all have times we have to work late or step into an emergency, but that shouldn’t be happening most days. Finish your day at a predictable time most days. Make sure your colleagues know when you are available outside normal hours and limit how much you let them cheat. Most importantly sometimes be truly unavailable.

Get out

Get out of your house/apartment at least once a day. When I first worked from home I realized that I’d gone a week without leaving my apartment complex and I was starting to go stir crazy which was bad for me and made less effective at work. Even if it’s just a trip to the store or a walk in the woods, get out and remember there is more to the world than your work.

Be Productive

Lots of work places treat work from home as a privilege they may want to take away again. Even if that doesn’t seem possible in your company, make sure you are proving you are able to be productive with the freedom working from home gives you. Ideally you have a work space free from the normal distractions of a conventional office, so use that environment to get more done than you can in the office.

Get together with colleagues

If you work remotely full-time you need to make sure you still spend face-to-face time with your colleagues. Humans are geared to appreciate time spent together, and for all the technology allows us new freedoms about where and when we work, there is still a different quality when everyone is in one place together. Some companies do this at conferences, some hold retreats, some just call the remote staff to the main office a few times a year. Figure out what makes sense for you and your company and push to make sure it happens.

Show personality

puzzleDo things that help give your colleagues insight into who you are outside work. I keep a puzzle table in my office to help me clear my head and avoid boredom during long conference calls. At a previous job we did a daily stand up video conference, and some days I would point the camera at the puzzle and worked on it as we talked. It served as a friendly way to help people see me as a real person not just a source of code. I also share pictures of my dogs and other things that round out people’s understanding of my life.

Not all of these things are totally within your control, and you will need support from your company to make sure the environment is right for you to be successful. Work with your manager, other remote employees, and other colleagues to make sure the environment is going to allow you to be successful over time.