Category: Programming

SC DUG September 2018

Chis Zietlow – Using Machine Learning to Improve UX

This fall the South Carolina Drupal User’s Group started using Zoom are part of all our meetings. Sometimes the technology has worked better than others, but when it works in our favor we are recording the presentations and sharing them when we can.

Chris Zietlow presented back in September about using Machine Learning to Improve UX.

We frequently use these presentations to practice new presentations and test out new ideas. If you want to see a polished version hunt group members out at camps and cons. So if some of the content of these videos seems a bit rough please understand we are all learning all the time and we are open to constructive feedback.

If you would like to join us please check out our up coming events on Meetup for meeting times, locations, and connection information.

Drupal 8 Batch Services

For this month’s South Carolina Drupal User Group I gave a talk about creating Batch Services in Drupal 8. As a quick side note we are trying to include video conference access to all our meetings so please feel free to join us even if you cannot come in person.

Since Drupal 8 was first released I have been frustrated by the fact that Drupal 8 batch jobs were basically untouched from previous versions. There is nothing strictly wrong with that approach, but it has never felt right to me particularly when doing things in a batch job that I might also want to do in another context – that really should be a service and I should write those core jobs first. After several frustrating experiences trying to find a solution I like, I finally created a module that provides an abstract class that can be used to create a service that handles this problem just more elegantly. The project also includes an example module to provide a sample service.

Some of the text in the slides got cut off by the Zoom video window, so I uploaded them to SlideShare as well:


Quick Batch Overview

If you are new to Drupal batches there are lots of articles around that go into details of traditional implementations, so this will be a super quick overview.

To define a batch you generate an array in a particular format – typically as part of a form submit process – and pass that array to batch_set(). The array defines some basic messages, a list of operations, a function to call when the batch is finished, and optionally a few other details. The minimal array would be something like:

  <?php  // Setup final batch array.
    $batch = [
      'title'    => 'Page title',
      'init_message' => 'Openning message',
      'operations'  => [],
      'finished' => '\some\class\namespace\and\name::finishedBatch',
    ];

The interesting part should be in that operations array, which is a list of tasks to be run, but getting all your functions setup and the batch array generated can often be its own project.

Each operation is a function that implements callback_batch_operation(), and the data to feed that function. The callbacks are just functions that have a final parameter that is an array reference typically called $context. The function can either perform all the needed work on the provided parameters, or perform part of that work and update the $context['sandbox']['finished'] value to be a number between 0 and 1. Once finished reaches 1 (or isn’t set at the end of the function) batch declares that task complete and moves on to the next one in the queue. Once all tasks are complete it calls the function provided as the finished value of the array that defined the batch.

The finish function implements callback_batch_finish() which means it accepts three parameters: $success, $results, and $operations: $success is true when all tasks completed without error; $results is an array of data you can feed into the $context array during processing; $operations is your operations list again.

Those functions are all expected to be static methods on classes or, more commonly, a function defined in a procedural code block imported from a separate file (which can be provided in the batch array).

My replacement batch service

It’s those blocks of procedural code and classes of nothing but static methods that bug me so much. Admittedly the batch system is convenient and works well enough to handle major tasks for lots of modules. But in Drupal 8 we have a whole suite of services and plugins that are designed to be run in specific contexts that batch does not provide by default. While we can access the Drupal service container and get the objects we need the batch code always feels clunky and out of place within a well structured module or project. What’s more I have often created batches that benefit from having the key tasks be functions of a service not just specific to the batch process.

So after several attempts to force batches and services to play nice together I finally created this module to force a marriage. Even though there are places which required a bit of compromise, but I think I have most of that contained in the abstract class so I don’t have to worry about it on a regular basis. That makes my final code with complex logic and processing far cleaner and easier to maintain.

The Batch Service Interface module provides an interface an an abstract class that implements parts of it: abstract class AbstractBatchService implements BatchServiceInterface. The developer extending that class only needs to define a service that handles generating a list of operations that call local methods of the service and the finish batch function (also as a local method). Nearly everything else is handled by the parent class.

The implementation I provided in the example submodule ends up four simple methods. Even in more complex jobs all the real work could be contained in a method that is isolated from the oddities of batch processing.

<?php

namespace Drupal\batch_example;
use Drupal\node\Entity\Node;
use Drupal\batch_service_interface\AbstractBatchService;

/**
 * Class ExampleBatchService logs the name of nodes with id provided on form.
 */
class ExampleBatchService extends AbstractBatchService {

  /**
   * Must be set in child classes to be the service name so the service can
   * bootstrap itself.
   *
   * @var string
   */
  protected static $serviceName = 'batch_example.example_batch';

  /**
   * Data from the form as needed.
   */
  public function generateBatchJob($data) {
    $ops = [];
    for ($i = 0; $i < $data['message_count']; $i++ ) {
      $ops[] = [
        'logMessage' => ['MessageIndex' => $i + 1],
      ];
    }

    return $this->prepBatchArray($this->t('Logging Messages'), $this->t('Starting Batch Processing'), $ops);
  }

  public function logMessage($data, &$context) {

    $this->logger->info($this->getRandomMessage());

    if (!isset($context['results']['message_count'])) {
      $context['results']['message_count'] = 0;
    }
    $context['results']['message_count']++;

  }

  public function doFinishBatch($success, $results, $operations) {
    drupal_set_message($this->t('Logged %count quotes', ['%count' => $results['message_count']]));
  }

  public function getRandomMessage() {
    $messages = [
      // list of messages to select from
    ];

    return $messages[array_rand($messages)];

  }

}

There is the oddity that you have to tell the service its own name so it can bootstrap itself. If there is a way around that I’d love to know it. But really one have one line of code that’s a bit strange, everything else is now fairly clear call and response.

One of the nice upsides to this solution is you could write tests for the service that look and feel just like any other services tests. The methods could all be called once, and you are not trying to run tests against a procedural code block or a class that is nothing but static methods.

I would love to hear ideas about ways I could make this solution stronger. So please drop me a comment or send me a patch.

Related core efforts

There is an effort to try to do similar things in core, but they look like they have some distance left to travel. Obviously once that work is complete it is likely to be better than what I have created, but in the meantime my service allows for a new level of abstraction without waiting for core’s updates to be complete.

Waterfall-like Agile-ish Projects

In software just about all project management methodologies get labeled one of two things: Agile or Waterfall. There are formal definitions of both labels, but in practice few companies stick to those definitions particularly in the world of consulting. For people who really care about such things, there are actually many more methodologies out there but largely for marketing reasons we call any process that’s linear in nature Waterfall, and any that is iterative we call Agile.

Classic cartoon of a tree swing being poorly because every team saw it differently.
Failure within project teams leading to disasters is so common and basic that not only is there a cartoon about it but there is a web site dedicated to generating your own versions of that cartoon (http://projectcartoon.com/).

Among consultants I have rarely seen a company that is truly 100% agile or 100% waterfall. In fact I’ve rarely seen a shop that’s close enough to the formal structures of those methodologies to really accurately claim to be one or the other. Nearly all consultancies are some kind of blent of a linear process with stages (sometimes called “a waterfall phase” or “a planning phase”) followed by an iterative process with lots of non-developer input into partially completed features (often called an “agile phase” or “build phase”). Depending on the agency they might cut up the planning into the start of each sprint or they might move it all to the beginning as a separate project phase. Done well it can allow you to merge the highly complex needs of an organization with the predefined structures of an existing platform. Done poorly it can it look like you tried to force a square peg into a round hole. You can see evidence of this around the internet in the articles trying to help you pick a methodology and in the variations on Agile that have been attempted to try to adapt the process to the reality many consultants face.

In 2001 the Agile Manifesto changed how we talk about project management. It challenged standing doctrine about how software development should be done and moved away from trying to mirror manufacturing processes. As the methodology around agile evolved, and proved itself impressively effective for certain projects, it drew adherents and advocates who preach Agile and Scrum structures as rigid rules to be followed. Meanwhile older project methodologies were largely relabeled “Waterfall” and dragged through the mud as out of date and likely to lead to project failure.

But after all this time Agile hasn’t actually won as the only truly useful process because it doesn’t actually work for all projects and all project teams. Particularly among consulting agencies that work on complex platforms like Drupal and Salesforce, you find that regardless of the label the company uses they probably have a mix linear planning with iterative development – or they fail a lot.

Agile works best when you start from scratch and you have a talented team trying to solve a unique problem. Anytime you are building on a mature software platform you are at least a few hundred thousand hours into development before you have your first meeting. These platforms have large feature sets that deliver lots of the functionality needed for most projects just through careful planning and basic configuration – that’s the whole point of using them. So on any enterprise scale data system you have to do a great deal of planning before you start creating the finished product.

If you don’t plan ahead enough to have a generalized, but complete, picture of what you’re building you will discover very large gaps after far too many pieces have been built to elegantly close them, or your solution will have been built far more generically than needed – introducing significant complexity for very little gain. I’ve seen people re-implement features of Drupal within other features of Drupal just to deal with changing requirements or because a major feature was skipped in planning. So those early planning stages are important, but they also need to leave space for new insights into how best to meet the client’s need and discovery of true errors after the planning stage is complete.

Once you have a good plan the team can start to build. But you cannot simply hand a developer the design and say “do this” because your “this” is only as perfect as you are and your plan does not cover all the details. The developer will see things missed during planning, or have questions that everyone else knows but you didn’t think to write down (and if you wrote down every answer to every possible question, you wrote a document no one bothered to actually read). The team needs to implement part of the solution, check with the client to make sure it’s right, adjust to mistakes, and repeat – a very agile-like process that makes waterfall purists uncomfortable because it means the plan they are working from will change.

In all this you also have a client to keep happy and help make successful – that’s why they hired someone in the first place. Giving them a plan that shows you know what they want they are reassured early in the project that you share their vision for a final solution. Being able to see that plan come together while giving chances to refine the details allows you to deliver the best product you are able.

Agile was supposed to fix all our problems, but didn’t. The methodologies used before were supposed to prevent all the problems that agile was trying to fix, but didn’t. But using waterfall-like planning at the start of your project with agile-ish implementation you can combine the best of both approaches giving you the best chances for success.  We all do it, it is about time we all admit it is what we do.

Cartoon of a developer reviewing all the things he's done: check technical specs, unit tests, configuration, permissions, API updates and then says "Just one small detail I need to code it."
Cartoon from CommitStrip

Thoughts on Hacktoberfest 2018

This year I took part in Hacktoberfest. Partially to see what all the fuss is about, partially to get myself involved in projects I didn’t know about, and partially for the free t-shirt (which do come in men’s and women’s cuts).  If you haven’t run into this project before it’s an effort by Digital Ocean to get people to participate in open source projects. Once you sign up they count all public pull requests you make on Github toward a goal of 5. I participated both as a developer, and by tagging a few issues on my own projects so people would find them.

As a developer:

It was a great excuse to go find new projects and look at ways I can contribute.  While I’d have plenty of experience on open source projects, often they have been outside Github or are repos I have commit access to – so I don’t open a lot of pull requests on Github. That meant that Hacktoberfest was a chance to find new projects and practice a basic process for contributing code to teams.

In that regard it was a pretty good success. I opened six PRs on four different projects. Mostly they were small stuff like linting code, updating packages, or tweaking a README file.  

In terms of drawing me into projects we’ll see. I did keep up with one after I finished the 5 required (hence having six PRs), but I didn’t dive into anything truly hard on that project.  

In terms of getting me to provide truly useful code think that was limited. The largest piece of code I wrote was initially rejected so I re-wrote in a different style, and then re-written by the project maintainer the day after he accepted the PR. He was really nice about it, and it helped him get something done that had been on the to-do list for a long time, but even that was example code to be used in classrooms (which was why he was so concerned about style – he didn’t want it to be idiomatically correct for Python he wanted to clear to beginners).

It did give me a chance to play around in other people’s code bases and I did resolve some issues for people that would have otherwise lingered longer than they already had.  It also forced me to meet other people’s standards, lint to their specifications, and pass their automated tests – all good things for everyone to do now and again to see if there are solutions you like better than the ones you use every day.

As a project owner:

Once I got through the contributions I needed to get a shirt, I figured I’d look over my own projects to see if there were issues I could label for beginners to help them find ways to get started. I listed several issues are both Hacktoberfest and good first issues. Almost all the ones I flagged as good first issues got PRs opened – sometimes more than one.

I got two problems solved that I wouldn’t have known how to solve without a bit of research, and those were great. But most of the PRs were simple things that took me longer to solve collaboratively than it would have taken me to solve myself. That’s okay, in part because some of my PRs caused the same problem for their project maintainers, and because it forced me to final learn how to setup CircleCI so the code gets checked and tested automatically when PRs are opened in the future.

What I don’t expect it caused was anyone to be truly interested in the project and helping it move forward over time. So while I solved a couple small problems, I did not get new help that going to keep engaging. That made it useful as a sprint, but not useful to helping build great projects.

But even if there is room for improvement my shirt is ordered and on the way.

We can do better

This week, for the second time in a year, the unacceptable behavior of a high profile man in the Drupal community has been the topic of public discussion and debate. This time the organizations involved acted more clearly and rapidly, if imperfectly. The issue came to the forefront during #metoo campaign, and again showed that the Drupal community reflects the world around us. I listened to friends and colleagues respond in various ways to the events and to the recognition of several men that they had been allowing this behavior to go on right in front of them for years without intervention. It is clear that in Drupal, like in all parts of our society we can – and must – do better.

As I reflected on these discussions this weekend I read back through some posts from Danah Boyd I’d read a while ago and stashed with my list of ideas of topics for blog posts. In particular I read her comments from last March on how failures to understand people’s hate fuels it and then a piece I had initially missed from July on change in the tech community. Her experiences and perspective are worthy of a few minutes read in their own right, but this week her views seem particularly timely.

One of the things that struck me about Boyd’s piece from July was her clear simple ask:

…what I want from men in tech boils down to four Rs: Recognition. Repentance. Respect. Reparation.

To me the first two are painfully obvious and most men who care about these issues have been working through those for a while now (too many men still need to learn to care at all). I count myself among the group of men who care, and the this post is mostly directed at that group of peers.

More and more you will hear men acknowledge they believe women are telling the truth, recognizing there are more stories we don’t hear than we hear, and apologizing for their own actions or inactions in the past. But of course just believing people and saying sorry doesn’t get us very far. The next two on Boyd’s list are the places where real forward looking change comes from.

Respect should be easy, but too often it is the first place we get into trouble. It is the part her call to action that will always be true no matter what future progress we make on these issues. Respect is an ongoing act requiring constant care, attention, and effort. Meaning to be respectful is not the same as actually being respectful. It requires actively listening to the ideas of women and people of color and considering them as fully as you do anyone else’s. It means tracking in yourself when you fail to do listen and making the personal change required to do better going forward. It includes monitoring our own behavior in meetings, hallway interactions, and one-on-one discussions to make sure you understand how you are being perceived differently by different people – your friendly or silly gesture to one colleague could be insulting or threatening to another. Respect is not something special that women and people of color are suddenly asking for, it’s something that we all already knew we should be extending to all our colleagues but too often fail to show. And when we fail to show true respect for coworkers – regardless of why we failed or which demographic categories they fall into – it’s our responsibility to recognize it and repent.

Finally Boyd also calls for Reparations. Reparations is a word that lots of us fear for no particularly defendable reason since it’s just about attempt to undo some of the harm we’ve benefited from. And in this case her ask is so direct, plain, and frankly easy that I’m giving her the last words:

Every guy out there who wants to see tech thrive owes it to the field to actively seek out and mentor, support, fund, open doors for, and otherwise empower women and people of color. No excuses, no self-justifications, no sexualized bullshit. Just behavior change. Plain and simple. If our sector is about placing bets, let’s bet on a better world. And let’s solve for social equity.

Code Without Community is Dead

With the turmoil in the Drupal community this spring and summer I have seen a wave of calls for open source projects to judge their community members, and other contributors, by the code contributions alone. It’s an idea that sounds great and has been popular among developers for at least forty years. Eric Raymond, while writing about Drupal, described it this spring as a right developers deserve. In some circles it gets treated almost as religious doctrine: “Thou shalt judge by code alone.”

The problem is that it’s not actually true nor just.

There may have been a time it was an ethic that held communities together but now it’s a line we pull out when we want to justify someone’s otherwise unacceptable behavior. Too often it’s a thinly veiled attempt to protect someone from consequences of their own choices and actions. And too often its used at the expense of other community members and other would be contributors.

For obvious reasons open source communities are dominated by developers. And as developers we have faith in our code. We like to believe our code doesn’t care who we are or what we believe. It will do exactly as asked every time. It forgives us all our sins as long as we faithfully fix any bugs and update to new versions of our platforms.

We like to claim that by focusing only on the code, and not the person behind it, we are creating a meritocracy and therefore better results. I don’t need to convince you I’m smart if I can show you my wonderful and plentiful code. We will be faithful to the best code and nothing more.

But none of this is proven to be true.

What does it profit, my brethren, if someone says he has faith but does not have works? Can faith save him? If a brother or sister is naked and destitute of daily food,  and one of you says to them, “Depart in peace, be warmed and filled,” but you do not give them the things which are needed for the body, what does it profit? Thus also faith by itself, if it does not have works, is dead.  James 2:14-17 NKJV

Any project worth doing could be done many different ways. One solution might solve the problem more quickly, another might be easier to maintain in the future, and yet another might be easier to extend to solve even more complex problems later. Which of the solutions is right will depend on the problem, the users, the developers, the timeline, and everyone’s guesses about future needs. There isn’t some objective measure of best. We figure out which solution we’re going to use by working together to sort through the competing ideas. Sometimes the best solution wins, sometimes the loudest voice wins, sometimes the most famous person wins.

When you say that code is the only valid way to judge a developer it tells people how you expect developers to work together. If I see that out of a project lead it tells me expect they me to put up with jerks and bullies as long as those people write good code. It says that women contributing code should expect to be belittled and degraded by colleagues, friends, and strangers as long as those people can solve a problem as well or better than she can (and often even if they can’t do that).

It says these things because for decades that’s been the behavior of large numbers of developers who are still welcomed on projects and praised for their amazing code.

It’s a simplistic way to view people that encourages the bad behavior developers, particularly many famous developers, are known for. Instead of creating a true meritocracy it drives away smart and talented people who don’t want to deal with being called names, threatened, and abused in their work place or while volunteering their time and talents.

~~~~~~~~~~~~~~~~~~~~~~~~~~~

People who damage a community because they mistreat others in it hold everyone back. If you join a project and write a module that brilliantly solves a new problem, but drive three people off the project because you are unbearable to work with, your work now has to offset all of their potential contributions. You also have to offset the loss of productively when people get distracted dealing with you. And you have to offset the reputation loss caused by your bad behavior. That better be an amazing block of code you wrote to do all those things, and you better be ready to do it again, and again, and again just to keep from slowing the project down.

Terrible developers can move a project forward. If someone comes into a project and writes a terrible module that stimulates someone else to solve that same problem better, they are making the project better even if none of their code is ever accepted. Or if someone takes time to make new participants feel welcome and excited to contribute, they may cause more code to be written than anyone else, even if they never write a line of code themselves.

And just because I can solve a problem faster and better than someone else doesn’t prove anything. If I start a project, or contribute for several years, I’m going to have more code credited to me than a new person. I’m going to know the project’s strengths and weaknesses better, and I’ll be able to solve problems more easily. Which means I should produce better code than another equally talented developer, or even someone a smarter than me. I just have a head start on them – anyone can beat Usain Bolt in the 100 meter dash if you give them a big enough head start. If we judge just by code contributions, I can use my head start to make sure no one catches me. That’s not merit, that’s just gaming a system.

Developers are people. People are flawed, complicated, biased, and wonderful creatures. Our value to a project is the sum of all those parts we choose to pour into a community. It matters how we handle discussions in issue queues; debates with community members on Twitter, Slack, or IRC; the content of our blogs; how we act at conferences; and anything else that is going to affect how others experience our contributions.

If you don’t care for how your community functions and if you don’t make sure people are treated well, even when their code is terrible, your project will suffer, and eventually may die. Sure there are examples (big examples like the Linux kernel and OpenBSD) of projects that are doing just fine with communities that allow developers treat each other terribly. The men leading those projects (if anyone has an example of a female lead projects where vile behavior is tolerated let me know and I’ll edit this sentence) are happy with the communities they have built. But can you imagine what those project could have achieved if they hadn’t driven highly talented developers away? Can you show any compelling evidence that those projects succeed because of the bad behavior not despite it?

Communities will be flawed, complicated, biased, and wonderful, just like the people that make them up. But if you only focus on part of someone’s contributions to the community you may miss their value and the damage one person can cause.

Faith in your code, if it isn’t matched with works in your community, is dead.

Helping the Cable Company Help me

Since the beginning of June I’ve been having intermittent issues with my Atlantic Broadband internet service.  When it works, it’s great (except they still won’t let my wife and I have different last names), but when it’s not it’s really annoying.

After several hours on hold, 5 or 6 visits (I’ve lost count), and them testing everything in sight, I finally started writing software to see if I can help find the problem.

At strange intervals there is a run of noise on the line that kills my service.  The noise eventually goes away and my service resumes. But figuring out those intervals has been a challenge since it turns out I don’t actually sit around 24 hours a day watching for service interruptions.

So after the last tech was unable to find a problem, and implied that it was my cable modem, I wrote a simple python script that pings Google every 5 seconds and records the response time (or failure). I setup an old Linux laptop to sit around and do nothing but ping Google 24 hours a day and record the results of each request.

I learned that my service interruptions tend to start and stop on the hour or half hour.  For example, the first night it started to be bad at noon and improved at 2:30 the next morning.  That pattern of human friendly times and data was enough to convince them to try again (although they aren’t willing to consider it’s something scheduled since they aren’t aware of any scheduled processes on their network).

So at the moment I have a loaner modem – which has ruled out my device as the problem – and they are closing on having replaced every piece of hardware between me and the nearest node.  To their credit, it has gotten much less bad.  Most outages now last five minutes (more or less exactly), and I have only have a few a day (instead of a few an hour).

Anyone have guesses about what processes run on cable networks that last five minutes and reattempt for a few hours at a stretch before giving up for roughly 13 or 14 hours?

My takeaway so far has been that if your internet provider is unable to figure out what’s wrong, offer them more data. Even if they don’t know how to use it, it them to take the problem seriously.

Fixing the Expert Beginner

I’ve been reading Erik Deitrich’s blog a bunch recently, particularly two pieces he wrote last fall on how developers learn. They are excellent. I recommend them to anyone who thinks they are an expert particularly if you are just starting out.

Failed Sticky bun
I am a good baker but this attempt at a giant sticky bun failed because I am not an expert.

In short he argues for a category of developer he calls the Expert Beginner.  These are people who rose to prominence in their company or community due more to a lack of local competition than raw skill.  Developers who think they are great because they are good but have no real benchmarks to compare themselves to and no one calling them out for doing things poorly.  These developers not only fail to do good work, but will hold back teams because they will discourage people from trying new paths they don’t understand.

I have one big problem with his argument: he treats Expert Beginning as a finished state. He doesn’t provide a path out of that condition for anyone who has realized they are an expert beginner or that they are working with someone who needs help getting back on track to being an actual expert.

That bothers me because I realized that at times I have been, or certainly been close to being, such expert beginner.  When I was first at AFSC, I was the only one doing web development and I lacked feedback from my colleagues about the technical quality of my work.  If I said something could be good, it was good because no one could measure it. If I said the code was secure, it was secure because no one knew how to attack it. Fortunately for me, even before we’d developed our slogan about making new mistakes, I had come to realize that I had no idea what I was doing. Attackers certainly could find the security weaknesses even if I couldn’t.

I was talking to a friend about this recently, and I realized part of how we avoided mediocrity at the time was that we were externally focused for our benchmarks.  The Iraq Peace Pledge gathered tens of thousands of names and email addresses: a huge number for us. But as we were sweating to build that list, MoveOn ran out and got a million. We weren’t playing on the right order of magnitude to keep pace, and it helped humble us.

I think expert beginnerism is a curable condition. It requires three things: mentoring, training, and pressure to get better.

Being an expert beginner is a mindset, and mindsets can be changed. Deitrich is right that it is a toxic mindset that can cause problems for the whole company but change is possible. If it’s you, a colleague, or a supervisee you have noticed being afflicted with expert beginnerism the good news is it is fixable.

Step 1: Make sure the expert beginner has a mentor.

Everyone needs a mentor, expert beginners need one more than everyone else. A good mentor challenges us to step out of the comfort zone of what we know and see how much there is we don’t understand. Good mentors have our backs when we make mistakes and help us learn to advocate for ourselves.

When I was first in the working world I had an excellent boss who provided me mentoring and guidance because he considered it a fundamental part of his job. AFSC’s former IT Director, Bob Goodman, was an excellent mentor who taught me a great deal (even if I didn’t always admit it at the time). Currently no one person in my life can serve as the central point of reference that he did, but I still need mentoring. So I maintain relationships with several people who have experiences they are willing to share with me. Some of these people own companies, some are developers at other shops, some are at Cyberwoven, and probably none think I look at them as mentors.

I also try to make myself available to my junior colleagues as a mentor whenever I can. I offer advice about programming, careers, and on any other topic they raise. Mentoring is a skill I’m still learning – likely always will be. At times I find it easy, at times it’s hard. But I consider it critical that my workplace has mentors for our junior developers so they continue progress toward excellence.

Step 2: Making sure everyone gets training.

Programming is too big a field for any of us to master all of it.  There are things for all of us to learn that someone else already knows. I try to set a standard of constant learning and training. If you are working with, or are, an expert beginner push hard to make sure everyone is getting ongoing training even if they don’t want it. It is critical to the success of any company that everyone be learning all the time. When we stop learning we start moving backward.

Also make sure there are structures for sharing that knowledge. That can take many different shapes: lunch and learns, internal trainings, informal interactions, and many others. Everyone should learn, and everyone should teach.

Step 3: Apply Pressure.

You get to be an expert beginner because you, and the people around you, allow it. To break that mindset you have to push them forward again. Dietrich is right that the toxicity of an expert beginner is the fact that they discourage other people from learning. The other side of that coin is that you can push people into learning by being the model student.

Sometimes this makes other people uncomfortable. It can look arrogant and pushy, but done well (something I’m still mastering) it shows people the advantages of breaking habits and moving forward. This goes best for me is when I find places that other colleagues, particularly expert beginners, can teach me. Expert beginners know things so show them you are willing to learn from what they have to offer as well. They may go out and learn something new just to be able to show off to you again.

Finally, remember this takes time.  You have to be patient with people and give them a chance to change mental gears.  Expert beginners are used to moving slow but it feels fast to them.  By forcing them into the a higher gear you are making them uncomfortable and it will take them time to adjust. Do not let them hold you back while they get up to speed, but don’t give up on them either.

Good Enough Passwords

I deal with passwords a lot. In any given day I log into five or six servers, another dozen web sites, plus my personal systems and tools. Some are stored in password managers, some are memorized, some are mine, and some are shared. To avoid losing things I need I have several patterns, schemes, and password generation tools I use to try to keep up with it all and make sure I’m using good passwords most of the time.

I don’t have a deep background in cryptography and a definitely don’t consider myself an expert in the related math. But I have spent a lot time time with users and observing behaviors, so I do consider myself reasonably knowledgeable about how people actually behave with their passwords.

A couple years ago this XKCD comic came out, and an admin’s view of it has become one of my measures of their understanding of users and passwords:

XKCD Password Strength Comic

If you’re interested in understanding what technically right and wrong with the math and assumptions in that comic you might find some of the references on explain XKCD’s discussion interesting.

I don’t actually think that’s the interesting part. To me the interesting part is the number of system administrators I have talked to who are convinced the comic is wrong, because they are also convinced that anyone who doesn’t use truly random password and avoid password reuse is stupid and therefore their behavior can be ignored. These people have access to your system, and it’s your job to keep those system secure. The problem isn’t your users, the problem is we have made passwords unreasonably hard to do right.

At this point people generally know they are supposed to use good passwords, like we all know we’re supposed to brush after every meal. Sure there are people that do that, but not most of us. We all know that passwords should be long (even if we fight over the exact length to require) and use lots of different kinds of characters (although many password systems require them to be from the latin character set). But it’s too hard to follow all the rules, and security experts are so concerned about being right they don’t provide useful guidance about when the cheat.

Let me grant those who like long-random passwords the following point: if you use different passwords on every system that needs one, and they are all truly random strings, and you memorize them all so you don’t have them recorded someplace they could be stolen, you have the hardest for an attacker to crack. Great for you. But I work with people who are not perfect, have limited memories, and need to be able to have shared access on a regular basis.

Knowing the perfect random password generation pattern is useful in some cases (or so I’ve heard), but rarely are you in a case where you can use the perfect setup. I don’t care about perfect: I’m not perfect, I don’t work in a perfect office, have perfect colleagues, or perfect clients. So here are my good-enough rules for admins and developers.

0) Make it easier to do the right thing than the hard thing. This gets to be rule zero, because everything else is meant to support this idea. You want the path of least resistance to be the one that gets you the results that are secure enough to protect your systems from the attackers they face. Make sure your users have good tools for storing passwords, settings on password fields to encourage good (not perfect) behavior, and a minimum of stupid rules you don’t really understand but someone told you are “best practice”.

1) If you make it hard for people, they will find a way around you and likely weaken security. It might be post-it notes on monitors, cycles of passwords that are 5 long (because you force them to not use the last 4). If you make it hard to pick a password (because you required punctuation but not  ‘, \, “, &, or !), you will end up with lots of passwords that are curse words – and your attacker will thank you for shrinking the search space. If they are using touch devices to type them, they will do things like repeat as many characters as you allow to make it easier to type (if you ban any repeats: again the attackers thank you for shrinking the search space). All things you want them to stop doing.

2) Do not have a maximum password length. Any time I hit a system with an upper bound of 12 I want to scream (although jokes about chimps might be a better tactic). Even if you are using a secure hashing system that ignores all characters after some point: who cares? Why limit the attacker’s search space to only strings between 8-12 characters?!? Sure that’s a massive search space, but not nearly as big as it could be.

3) Do have minimum lengths. Minimum lengths forces your users to do two things. First, not use passwords that could be broken in less time than it took you to read this article. Second, it gives you leverage to push them to either good phrases or generators. If you’re smart and don’t have legacy systems to support go with something like 15 or 20 characters.

4) Expect people to share passwords. Many times this is actually a basic job function. If I won the lottery tomorrow (unlikely since I don’t play) and don’t come to work (also unlikely since I would wait until I had the money reinvested before making plans) the person taking my place needs to be able to access all the tools, servers, and accounts I’ve setup. If she can’t do those basic things I haven’t done my job responsibly.

5) Provide secure means to share passwords. I have more than once been sent a password in chat (running through Google, Slack, or once upon a time AOL’s servers), email, word documents, text file, and a variety of other terrible solutions. This happens not because my colleagues didn’t know it was a bad thing to do but because they didn’t have a good option. We spend so much time locking down passwords, that we don’t create secure channels to hand them around responsibly which defeats the purpose of secured storage.

6) Pay attention to how users will be using individual passwords. Not all passwords are created equal, which is why I encourage you to support throw away passwords: something short, easy to remember, and only used places it doesn’t matter if it were stolen. But even when a password is important there are issues like the ease of entering them: if I have to enter a password 4 times a day it better be easy to type or I better be able to copy and paste it. If I need it once a month it should be impossible to remember and its okay if it takes me 5 minutes to get it right. Most of us can’t type complicated passwords quickly, and if we have to enter it a bunch we want to be fast.  This is even more true for people using touch interfaces where shift is an extra keystroke as is changing to a different part of the standard keyboard.

7) Stop telling people they have to use a different password every time. This is an extension of number six. People have too many passwords, and that’s not changing soon. Sure we can encourage them to use LastPass, or a tool like it, but most people aren’t going to (and if they did that could be its own problem since it creates single points of failure). Tell them to use a different password when it’s important, and to use a throw away password or scheme when it’s not.

Not everything needs to be Fort Knox so stop pretending it is.  Important things like your bank account(s), your email, Facebook need their own passwords because they can be used to do real damage in the real world. Online communities, games, and other trivial places asking you to sign in do not.

8) Don’t lecture people about bad personal password habits. Honestly, this is probably the hardest one (here I am lecturing you about not lecturing them). Usually the first people to admit they are sloppy about passwords are developers and sysadmins. Sure, they will tell you about the awesome password wallet they use first, and the two factor authentication they created for their blog, but then toss off that all their production servers have the same root password and it’s 8-10 nonrandom characters. Even if you are perfect (if you are still reading this by definition you probably have room for improvement) don’t lecture people who aren’t. It just makes them feel they can’t admit when something has gone wrong, or if they don’t understand something. When you find people doing it wrong, show them how easy it is to do it right, and if it isn’t apologize and fix it.

Are you moving forward or backward?

Every week I try to ask myself: What did I do this week to make myself more valuable? Am I moving forward toward a goal, or further from it?

Handmade sign reading Are you going bkwds? or are you going fwds?
I spotted this the other week in Philly. I didn’t try to sort out the politics of the creator.

In technology, communications, or any other job that involves one of those two things you are either moving forward or moving backward: standing still is not an option. You are either learning new skills, trends, tools, and concepts, or you’re falling behind as other people build new tools that drive new ideas and trends.

I read lots of advice that says to plan your career two or three moves in advance. That is good advice, but I don’t think it’s wise to trust your gut that far out. The technology landscape changes too fast and too dynamically to believe you know where everything will be in three or five years. On the one hand I think it’s important to deepen my skills for the path I want to be on, but at the same time I try to broaden my skills into areas other areas that have things to teach me. In the back of my head there is always plan B and C, just in case plan A doesn’t come together they way I hope. In part, because I’ve never been able to stay on plan A very long: life always intervenes.

When I was 23 (and sure I wanted to be teacher) I was advised to read an hour a day in my field, and that it should not just be reading about the kind work I was already doing. At the time I was the new kid in IT of a mid-sized international nonprofit organization doing whatever no one else wanted to do: which is a great way to learn a variety of things. I didn’t really know anything yet about how to have a career – I had a job, and I liked my job, but couldn’t envision a career path.

But I took that advice to heart and tried to find ways to constantly learn about things I don’t know. I read books, listened to podcasts, and taught myself new skills. I learned about communications planning, economics, corporate strategy, algorithms, and a variety of other topics. The ideas I pick up from those sources help me think about technology more creatively, and helped me understand the importance of making sure I build tools that are useful not just cool to me.

For several years I also taught myself a new programming language every year. I taught myself ASP, C#, PHP, Python, Ruby (on rails and off), R, Haskell, and JavaScript because I heard other people talk about them as important or interesting. I have used five of those professionally to create actual software people used. And the others all forced me to see programming differently and helped me be a better developer. I don’t force myself into learning whole new languages annually, it was too broad and prevented me from deepening my knowledge of individual languages and the ecosystems I work in frequently (although I’m probably overdue to teach myself something like Go, Swift, or Rust.

The biggest thing I’ve learned from all these different inputs is that I need to live in constant fear of getting behind, outmoded, and sidelined. That fear keeps me motivated to learn more and push myself harder. By the time I retire I cannot imagine I will still be paid to be a full-time Drupal developer, I doubt that’s what I’ll be doing five years from now. Certainly by in 30 years Drupal, and the web as we know it, won’t look anything like they do today and I will be doing my job very differently. This is the blerch that keeps me motivated.

So every week ask yourself: what did I learn this week? Did I move forward or fall behind?