Lessons learned from my first Drupal 8 projects

During last month’s SCDUG I gave two presentations. I’ve already posted the Sins Against Drupal talk I gave. That didn’t take up much time, and since I had been prepping some thoughts on the first few Drupal 8 projects I gave a second short talk on what I’ve learned working with Drupal 8 during its first year.


What’s not new?

Drupal 8 brings forward lots of concepts from Drupal 7 (and prior versions). It also brings forward a few standard community realities.

The information architecture tools are basically the same at least at the conceptual level. Nodes, Fields, Taxonomy, and Menus all still exist more or less as we’ve known them for years, and while there are differences on the surface those differences are incremental in nature and scope.

We also still have the constant hunt for modules that do what you’re doing so you don’t reinvent the wheel. Many modules that people are used to using are still missing, but new things arrive daily, and many of the new versions are significant improvements over previous generations of tools.

And there is still a lack of a clear line between front-end and back-end. When does business logic end and interface begin? When does a themer need to understand Drupal’s HTML generation vs when does a backend developer need to figure out how to force Drupal to generate carefully crafted markup?  There are opportunities to form better and clearer lines, but they aren’t automatic by any means: every team will have to solve this problem their own way.

What’s so great?

Drupal 8 opens up a collection of new tools and opportunities for the community.

  • As a backend developer you get to write modern code. The name spacing can feel a little Java-esc at times, but the ability to properly name space code, ditch globals, move from hooks to event listeners, and other basic OOP tools is incredibly nice.
  • With CKEditor in core we get better integration between that interface and the rest of Drupal. And better modules are coming out all the time to solve long standing UX annoyances. For example with the D8 Editor File upload module files and images can both be handled as Drupal file objects, but the editor can know the difference between a file (which should just be a link) and an image (which you should display).
  • The Symfony community provides a large number of packages that provide 3rd party integrations, or the tools to make them easy to build.
  • The two core base themes do not require keelhauling to make viable. If you commonly built your themes from scratch, the ability to have clean default markup that’s easy to override makes both Stable and Classy a major improvement to life.

Things you need to survive

As you dive into your first projects you need to understand that much of what you know at the detail level has changed.  And so you’ll need to learn a few new tricks and be willing to toss aside a few old ideas:

Your old build standards are wrong. Probably not entirely, but you’ll need new modules in your default builds, new best practices about when to use a node vs block vs custom entities, and other basic details you probably have really well develop standards (or at least habits) from years of working with Drupal.

If you’ve been avoiding it to-date, it’s time to develop an understanding about composer, drush, and Drupal console. Not only do you need to be using these tools (probably all three) but you need to understand what each provides and which tool is best for which job.

Nginx is not for the faint of heart: as best I can find, no one has published a complete setup guide yet. In Drupal 6 and 7 there were pretty good guides to setting up Nginx properly, but with Drupal 8 there are enough differences that those guides don’t really work. And all the guides that I have read so far include errors (some of them significant security mistakes). To be clear, it can be done, but if you want to do it yourself be prepared to do a great deal of extra research along the way.

If you have windows you have pain. There are a number of windows specific challenges (speed and NTFS weakness being the biggest we face), and there is little community support to help you overcome the challenges.

Core patches tend to be required. Unlike Drupal 7, core and module patching is the norm not the exception. There are several issues that are frustratingly slow to get fixed that hit some fairly common use cases (like this menu block bug). While this is improving all the time, we haven’t launched a D8 site without at least one patch in place.

Cache tags are great, but require learning. The new caching system is powerful, flexible, and totally different from what we had before.  It’s better, but to use it well you’ll need to spend some time getting to know when to set and clear the tags you want to create.

Twig is great, but requires learning and discipline. I really like twig and the much cleaner syntax to brings to the theme layer.  However, as more and more people use it they are finding ways to move increasingly complex logic out of modules and theme PHP into template files. Please fight this urge! Keep your business logic separate from your display logic. If some object wasn’t loaded into a variable in your twig file do not attempt to load it in twig. If you need some 4 or 5 layer selector to get to the value you want to print: fix that in a preprocess function.

The API improvements are coming all the time and make things interesting. So far the community has stayed on schedule of rolling out new minor versions of 8 and that’s meant great new features in each version. It has also meant that sometimes a solution you built is not using the best techniques given the improvements.  That’s not really a problem, but can add headaches in maintenance cycles. Also you will find places where brand new D8 tools are already deprecated but the replacements don’t have good example implementations yet.

XML to JSON in Five lines

In closing I want to share the feature that caused me to realize that Symfony was worth the cost of admission: a 5 line XML to JSON AJAX callback.

My first site had a location finder that links to a data provider partner locations nationwide. That provider has an XML-base API that cannot be accessed directly from browsers, and therefore needed to be proxied through the main site. The full details involve more than just the following lines of code (providing input checking, settings, routing, etc), but the heart of the process is just five lines.

    $client = new Client(); // 1: Create Guzzle Client
    try{
      $res = $client->get($config->get('api_endpoint'), // 2: Make request
            [
              'http_errors'=>false,
              'query' =>; [
                 'key'=> $config->get('coop_api_key'),
                     // ...
               ],
            ]);
      $decoderRing = new XmlEncoder(); // 3: Create XML Encoder
      $xml = $decoderRing->decode($res->getBody()); // 4: Decode the XML Response into an array.
      return new JsonResponse($xml); // 5: Return the array as JSON
    } catch (RequestException $e) {
      throw new HttpException($this->t('Unable to process request'));
    }

Sins Against Drupal 3

This is part of my ongoing series about ways Drupal can be badly misused. These are generally times someone tried to solve an otherwise interesting problem in just about the worst possible way. All of these will start with a description of the problem, how not to solve it, and then ideas about how to solve it well.

I present these at SC Drupal Users Group meetings from time to time as an entertaining way to discuss ways we can all improve our skills.

This one was presenting during our October event here in Aiken, SC.


The Problem

Provide a custom authentication solution that allows staff to have one backend and members another.

The Sinful Solution

In order to force staff to use the staff login page, during login form validation check to see if the user is a staff member, by authenticating the user, checking their groups, and logging out staff.

The Code

/**
* Prevents staff members from logging in outside of staff login page. <<-- Why?
*/
function my_auth_staff_boot($form, &$form_state) { // NOT actually a hook_boot (thankfully) called as login form validator...
  user_authenticate($form_state['values']);
  global $user;
  if (in_array('An Employee', $user->roles)) {
    form_set_error($form['#id'], l(t('Staff must log in via staff-login', 'staff-login')), TRUE);
    drupal_set_message('Staff must log in via ' . l(t('staff-login', 'staff-login')), 'error', TRUE);
    // Load the user pages in case they have not been loaded.
    module_load_include('inc', 'user', 'user.pages');
    user_logout();
  }
}



Why is this so bad?


This code actually completes the login process before kicking the user out. Why would you ever want to do that to your users? What did they do to you? It also loads an extra file for no apparent reason just before kicking the user back out.


Better Solutions


The goal here is to control what backend the user logs into, and shouldn’t control the page they login from. So the place to look for solutions are modules that already do this and so I propose mimicking the LDAP or GAuth modules’ approaches. LDAP attaches a validator to the form and takes over authentication, but LDAP supports lots of options so the code there is too extensive to use for a clear example. So for discussion I pulled out elements of the GAuth module (although there is still lots of trimming to make this understandable).


The GAuth module adds a submit button to the form and handles all processing for that form directly.


/**
* Implements hook_form_alter().
*/
function gauth_login_form_alter(&$form, &$form_state, $form_id) {
  if ($form_id == 'user_login' || $form_id == 'user_login_block') {
    $form['submit_google'] = array(
      '#type' => 'submit',
      '#value' => t(''),
      '#submit' => array('gauth_login_user_login_submit'),
      '#limit_validation_errors' => array(),
      '#weight' => 1000,
    );
    drupal_add_css(drupal_get_path('module', 'gauth_login') . '/gauth_login.css');
  }
}

/**
* Login using google, submit handler
*/
function gauth_login_user_login_submit() {
  if (variable_get('gauth_login_client_id', FALSE)) {
// .. skipping resource validation ...

  $client = new Google_Client();
// .. skipping client setup ...
  $url = $client->createAuthUrl();
  // Send the user off to Google for processing
  drupal_goto($url);
  }
  // ... skip errors
}



From there we pass through a menu router from the main module, and an API hook to get:


function gauth_login_gauth_google_response() {
  if (isset($_GET['state'])) {
// Skipping some error traps...
    $redirect_url = isset($state['destination']) ? $state['destination'] : '';
    if (isset($_GET['code'])) {
// Skipping a bunch of Client setup...
      $oauth = new Google_Service_Oauth2($client);
      $info = $oauth->userinfo->get();
      if ($uid = gauth_login_load_google_id($info['id'])) {
        $form_state['uid'] = $uid;
        user_login_submit(array(), $form_state); // << That right there with the $form_state['uid'] set does the magic.
      }
      else {
// Skipping other options....
      }
    }
    drupal_goto($redirect_url); // << be nice and handle the destination parameter
  }
}



Share your sins


I’m always looking for new material to include in this series. If you would like to submit a problem with a terrible solution, please remove any personally identifying information about the developer or where the code is running (the goal is not to embarrass individuals), post them as a gist (or a similar public code sharing tool), and leave me a comment here about the problem with a link to the code. I’ll do my best to come up with a reasonable solution and share it with SC DUG and then here. I’m presenting next month so if you have something we want me to look at you should share it soon.


If there are security issues in the code you want to share, please report those to the site owner before you tell anyone else so they can fix it. And please make sure no one could get from the code back to the site in case they ignore your advice.

Documenting your work

Programming books
Your documentation does not need to look like this.

Early in my career I spent a lot of time as the only technical person on project, and therefore believed that I didn’t need to document my work carefully since I was the only person who had to understand it later. It turned out that if a project was back burnered for a few months the details were pushed out of my mind by the details of eight other projects.

Any project that takes more than a couple hours to complete involves too many details for most people to remember for more than a few days. We often think about project documentation as something for other people – and it is – but that other person may be you in six months.

I learned to start keeping notes that I could go back to, those notes would turn into documentation that I could share with other people as the need developed. My solutions were typically ad-hoc: freeform word documents or wiki pages. For a while I had a boss who wanted every piece of documentation created by IT to fit a very predictable format and to be in a very specific system. It took two years for him to settle on the system, process, and format to use. By then I had a mountain of information in wiki pages that documented the organization’s online tools in detail, and no one else is IT had anything substantial. It was two more years before the documentation of other team members got to be as good as my ad-hoc wiki.

That’s not to say a rogue solution is best, but the solution that I used was better than his proposed setup for at least three years. That experience got me to think about what makes documentation useful.

Rules of thumb for good project documentation:

  • Write up the notes you’d want from others when coming into a project: think of this as the Golden Rule of documentation. Think about what you’d want to have if you were coming into the project six months from now. You’d want an outline of the purpose of the project and the solution used, and places they deviated from any standards your team normally uses. You’ve probably read documents that are explaining something technical to an expert that are hard for anyone else to understand – if I’m reading the documentation I want to become an expert, but I’m probably not one already.
  • Keep it easy to create and edit while working: if you have to stop what you’re doing and write your notes in a totally different environment that your day-to-day work you will not do it. Wikis, markdown files, and other similar informal solutions are more likely to actually get written and updated than any formal setup that you can’t update while doing your main work.
  • Document as you go: we all plan to go back and write documentation later and almost none of us do. When we do get back to it, we’ve forgotten half the details we need to make the notes useful to others. So admit you’re not going to get back to it and don’t plan to: write as you go and edit as you need.
  • Make sure you can come in in the middle: People skim project documentation, technical specifications, and any other large block of text. Make sure if someone has skipped the previous three sections they can either pick up where they left off, or give them directions to the parts the need to understand before continuing.
  • Track all contributions: Use a system that automatically tracks changes so you you can see contributions from others and fix mistakes. Tools like MediaWiki, WordPress, and Drupal do this internally. Markdown or text files in a code repository also have this trait. Avoid solutions like MS Word’s track changes that are meant for editing a final document not tracking revisions over time.
  • Be boldDon’t fear editing: follow the Wikipedia community’s encouragement to Be Bold. You should not fear making changes to the team’s documentation. You will be wrong in some of what you write, and you should fix any mistake you find – yours or someone else’s. Don’t get mad if someone makes a change that’s not quite right, revert the change or make a new edit and more forward.
  • There is always an audience: even if you are the only person on the project you have an audience of at least your future-self. Even if it feels like a waste in the moment having documentation will help down the road.

Remember even if you are working alone you’re on a team that includes at least yourself today and yourself in the future. That future version of you probably won’t remember everything you know right now, and will get very annoyed at you if you don’t record what they need to know. And if the rest of your team members aren’t just versions of yourself they may expressed their frustration more directly.

Why I think pushing only STEM is a problem

Over the course of this week I needed to be conversant in PHP, MySQL, Linux, Drupal, search tools, herbal remedies, online education tools and strategy, women’s health, men’s health, canine health, dental health, health insurance, public media outlet fundraising, online advertising, email, open source technology, open source movements, cyber security, physical security, national security, state and national politics, disaster response planning, disaster response fundraising, coffee, vegetarian ethics, economics, retirement planning, videography and photography, child rearing, fiber arts, baking, and a few other topics.

Last week I talked a bit about the things I got from my liberal arts education that made me a better developer. There I focused mostly on the positive side of the argument: why non-stem course work is important. Now I want to flip the coin and talk about why our constant push to have more people graduating with STEM degrees is bad for the country and the economy.

No, I didn’t learn about all the topics in the opening in college – in fact much of the information I gathered in college on many of those topics (particularly things related to technology) is now at least partially out of date. But I spent four years learning how to learn and developed a base of skills in several disciplines that makes it easier for me to adapt to change and new areas of interest.

In 2016 the path to most high paying jobs runs through college. And the majority of people who go to college earn more than they would have if they hadn’t gone to college. But if we talk about college as being all about our work and pay we miss the main point of a good education and we ignore the fact that our education system should help mold engaged citizens not just good workers.

The push to graduate students to be “workplace ready” in engineering and technology (people calling for STEM education rarely care about science that can’t be applied to engineering new technologies and pretty much never mention math) is short-term thinking. That kind of educational model frees companies from having to train new employees while weakening their long-term workforce and our democracy. Our education system, high school and college in particular, needs to be helping kids become productive members of a society none of us can fully predict. To thrive in current and – more importantly – future America we all have to adapt to constant technical and social change. You cannot teach kids in school how to actually handle the situations that don’t exist yet, so we need to be teaching them how to learn so they develop skills as they need them.

Companies go looking for the employees of tomorrow, but not of next year, or of five years from now. They complain that younger workers don’t want to put in time at low-level jobs while refusing to provide training to help those new workers advance. What we get all too often are people who are good-enough to do the work of today and tomorrow, but never get better.

When I first wrote code for work, I was terrible. I got better because I was allowed to fail and learn from my mistakes. If you’d judged my college program on that first project, you’d call Hamilton a fraud. If you judge them by my ability to adapt, change, and improve, you’d rank Hamilton a top school as most rankings do.

College should be part of an education meant to prepare people for a life that will be unpredictable and varied. It should part of an education built around showing kids as much diversity of thought as we can:

  • Kids need art courses because art helps us all see the world fully and because developers need artists on our teams to make our projects successful.
  • Kids need to learn history because history matters – when our society repeats mistakes because we don’t understand our own past we all lose.
  • Kids need to learn math and science because we need them to understand how the universe works and that facts are not something that change when it is politically convenient.
  • Kids need to learn to challenge ideas they are taught because even the best teachers will be wrong sometimes.
  • And kids need to play because it’s good for their bodies, minds, and souls.

I work for a company that develops software every day of the week. There are ten of us whose primary job is to write code, so we need great developers. But there are 30 people at the company which leaves twenty of us who do things other than write code – we need great designers, writers, account managers, and more. Yes, they need to understand the internet and the tools we use at some level, but more importantly they need to understand how do their part in helping our clients be successful so that our company can succeed.

And at the end of the day all thirty of us live in a democracy, and in November all of us should be casting informed votes up and down the ballot for people we think will do the best job. The artists will need to understand the tax policies, and the developers will need to understand the impact of social program proposals. We don’t have to agree, but we all need to be prepared.

If you want a deeper argument about the importance to the world of people with a broader understanding of many topics check out Former Bennington College President Liz Coleman’s Ted Talk.

What I learned by getting a degree from a liberals arts college

I have a degree in Computer Science, with a minor in Economics. But I earned that degree at Hamilton College, a traditional liberal arts college. That meant I was forced (now students there are just encouraged) to take classes in a variety of disciplines. I went to Hamilton in part because they offered degrees in Computer Science and History, and I was interested in both fields. In addition to the courses required by my major I took classes in history, religious studies, philosophy, art, and more. And I learned critical skills for work and life because I took those classes.

Two hand made mugs
Making these mugs made me a better developer.

Ceramics and pottery taught me to be a craftsman, accept critical feedback, and admit failure. For all the courses I took in college, the time I spent in Ceramics and Pottery studio taught me more about how to be good at what I do than any other course work (including CS). Basic pottery is a craft that doesn’t allow you to save a piece once you’ve made mistakes. If you are making cylinders the walls aren’t straight throw it away and start again. My professor in those courses set very high standards and pushed us to meet them through brutally honest, but helpful, critiques. He taught me to appreciate honest feedback, and to be skeptical of my work before showing it others. In those courses we had to do good work, recognize our weaknesses, and hit deadlines.

History taught me to care about communicating. Between the courses I took, and dating (and eventually marrying) a history major (now professor) I came to understand how important it is to communicate well in speech and in writing. I started this blog in part because I wanted to make sure I was spending more time writing to help maintain those skills. I frequently find myself in meetings having to explain highly technical issues to non-technical clients and colleagues. Knowing how to adjust to my audience, without insulting their intelligence, is a critical part of my day to day work.

Religious Studies and Philosophy taught me to make a well reasoned argument and express the ethics of my position. Religious Studies and Philosophy both require you to make arguments based primarily on the strengths of your ideas – you can’t research your way out of a bad concept. If you cannot assemble a well reasoned argument you leave yourself open to easy counter attacks. When you work at a nonprofit you often are building your ideas and your explanations from a mix of the facts of the issue, your worldview, and the assumptions you are forced to make because no one has all the information they really need.

Economics taught me how to view the world through the lens of money and trade. While I don’t think it’s the only, or even best, way to view the world it is a very important worldview that dominates the news and political spaces. Understanding the strengths and weaknesses of economics as a field helps me think about plans put forward by various politicians. It also helps me think about how the companies I work for function, and helps make sure internal systems and plans I propose have a strong business case to support them (and because I learned to throw things away in ceramics I also know that when the case is bad to stop and try something different).

The most important skill I have is the ability to learn new skills. Since I graduated from college the technology I work with every day has changed several times. I’ve had to learn tools, techniques, and strategies that didn’t exist when I was in college. And my career has evolved through multiple employers, disciplines, and areas of expertise.

Bad data systems do not justify sexist your behavior

This week we get a letter from Atlantic Broadband, our ISP, addressed to “Aaron & Eliza Crosman Geor”. My wife has never gone by Eliza and her last name is not “Geor”.

Picture of an envelop from Atlantic Broadband to: Aaron & Eliza Crosman GeorIt’s been this way since we signed up with them, when we ask them to fix it they acknowledge that they cannot because their database cannot correctly handle couples with different last names who both want to appear on the account. Apparently it is the position of Atlantic Broadband that in 2016 it is reasonable to tell a woman she cannot be addressed by her legal name because it would be expensive for them to fix their database, and therefore she must be misaddressed or left out entirely.

I consider this unacceptable from old companies, but Atlantic BB was founded in 2004 – there are probably articles about not making assumptions about people’s names that are older than their company.

Folks, it is 2016, when companies insult people and then blame their databases it is because they do not consider all their customers worthy of equal respect.

So let’s get a few basics out of the way:

  • Software reflects the biases of the people who write it and buy it.
  • If your database tells someone their name is invalid your database is not neutral. Just because you don’t get the push-back that Facebook sees when they mess this up does not mean what you’re doing is okay.
  • If your database assumes my household follows 1950s social norms, the company that uses it considers 1950s social norms acceptable in 2016 – and there are probably a few of those they don’t want to defend (I hope).
  • When an email, phone rep, or letter calls me by my wife’s last name or her by mine, in both cases they are assuming she has my last name not that I have hers. This is a sexist assumption that the company has chosen to allow.

Of course Atlantic isn’t the only company that does this: Verizon calls me Elizabeth in email a couple times a week because she must be primary on that account (one person must lead the family plan), and Nationwide Insurance had to hack their data fields for years so my wife could appear on our car insurance card (as required by law) every time we moved because their web interface no longer allowed the needed changes. The same bad design assumptions can be insulting for other reasons such as ethnic discrimination. My grandmother was mis-addressed by just about everyone until she died because in the 1960s the Social Security Administration could not handle having an ‘ in her name, and no one was willing to fix it in the 50 years that followed SSA’s uninvited edit to her (and many other people’s) name.

In all these cases representatives all say something to the effect of “our computers cannot handle it.” And that of course is simply not true. Your systems may not be setup to handle real people, but that’s because you don’t believe they should be.

Let’s check Atlantic Broadband’s beliefs about their customers based on how they address us (I’m sure there are some additional assumptions not reflected here but these are the ones they managed to encode in one line in this letter):

  • They assume they are addressing one primary account holder: I happen to know from my interactions with them that they list my first name as: “Aaron & Eliza”, and my last name as “Crosman Geor”. Plenty of households have more than one, or even two, adults who expect equal treatment in their home. Our bank and mortgage company know we are both responsible adults why is this so hard for an ISP (or insurance company, or cell provider, credit card, etc)?
  • They assume my first name isn’t very long: They allowed 13 characters, but 4 more is too many. I went to high school with a kid who broke their database by exceeding the 26 character limit it had (they didn’t ask the kid to change his name, the school database admin fixed the database), but Atlantic can barely handle half that.
  • They assume my last name isn’t very long: Only 12 characters were used and they stopped in a strange place. I know many people with last name longer than that: frequently people who have hyphenated last names blow past 12. Also the kid with a 26 character first name – his surname was longer.
  • They assume my middle name isn’t an important part of my name: If they had a middle name field, they could squeeze a few more letters in and make this read more sensibly.  But they only consider first and last names important. Plenty of people have three names – or more – they like to have included on letters.
  • They assume it is okay to mis-address me and my wife: The name listed is just plain wrong, but they believe it’s okay to keep using this greeting. They assume this even after they have been told it’s not, and even after we’ve reduced service with them (if another ISP provided service to my house I’d probably cut it entirely although mostly for other reasons). They believe misaddressed advertisements will convince me I need a landline or cable package again.

Now I’ll be fair for just a minute and note something they got right: they allow & and spaces in a name so Little Bobby Tables might be able to be a customer without causing a crisis (partially because his name is too long for them to fit a valid SQL command into the field).

Frequently you’ll hear customers blame themselves because their names are too long or they have done something outside the “norm”. Let’s be clear: this is the fault of the people who write and buy the software. Software development is entirely too dominated by men, as is the leadership of large companies. When a company lacks diversity in key roles you see that reflected in the systems built to support the work. Atlantic’s leadership’s priorities and views are reflected in how their customers are addressed because they did not demand the developers correct their sexist assumptions.

These problems are too common for us to be able to refuse to do business when it comes up. I will say that when we switched our insurance to State Farm they did not have any trouble understanding that we had different last names and their systems accommodated that by default.

If you do business with a company that makes these (or other similar mistakes) I think it’s totally reasonable to remind them every time you reasonable can that it’s offensive. Explain that they company is denying you, your loved ones, and/or your friends a major marker of their identity. Remind them they are not neutral.

If you write data systems for a living: check the assumptions you’re building into your code. Don’t blame the technology because you used the wrong character set or trimmed the field too short: disk is cheap, UTF-8 has been standard for 15+ years, and processors are fast. If the database or report layout doesn’t work because someone’s name is too long the flaw is not the name.

We all make mistakes and bad assumptions sometimes, but that does not make it okay to deny people basic respect. When we make a bad assumption, that’s a bug, and good developers are obligated to fix it. Good companies are obligated to prevent it from happening in the first place.

Try doing it backwards

As part of my effort not to repeat mistakes I have tried to build a habit in my professional – and personal – life to look for ways to be better at what I do. I recently rediscovered how much you can learn when you try doing something you know well backwards: I drove on the left side of the road.

This is the Holden Barina we rented while in New Zealand.
This is the Holden Barina we rented while in New Zealand, a brand of car I’d never heard of before this trip. It was a good car for the mountain driving even if the wipers and lights controls were reversed from cars at home.

By driving on the left I discovered how many basic driving habits I have that are built around driving on the right. The clearest being that the whole time I was in New Zealand I never knew if anyone was behind me, and the whole time I couldn’t figure out why. The mirrors on the car worked just fine, but it turned out I wasn’t looking at them. Driving home from the airport after we returned to the US I realized that every few seconds my eyes jump to the upper right of my vision to check the mirror. In New Zealand I spent the whole time glancing at the post between the windshield and the driver’s side window (which had seemed massive to me while I was there) instead of the mirror. It made me conscious of my driving habits in a way I haven’t been in years, and as a consequence, I think it’s made me a better driver. I’m thinking about little details again; I’ve been more aware of where I am on the road and what I’m doing to keep track of the other cars around me.

My wife drove this section so I got to take some pictures. Amazing scenery but she had to adjust quickly.
My wife drove this section while I got to take some pictures. She got to learn to drive on the left on winding mountain roads – we don’t recommend that approach.

A few years ago I was watching videos from the MIT Algorithms course to refresh some of my basics, and because I wanted to know what had been added in the decade since I’d taken that class at Hamilton. During the review of QuickSort the professor mentioned that it wasn’t originally a divide-and-conqueror process, but a loop based approach meant to work on a fixed length array (so you could use a fixed block of memory). And as I recall he suggests that the students should work out the loop based version. So riding on the train home from work I pieced it together, and found that it’s an elegant process. It’s not something I ever expect to have cause to implement, but it did help me improve my thinking about when to use recursive functions vs when to use a loop, and helped me think about when to use recursion, loops, and other tools for processing everything in a list. There was a session by John Kary at DrupalCon this year on rethinking loops that pushed me again to revise some of how I made those decisions. Again his talk took the reverse view of much of my previous thinking and was therefore very much worth my time.

If you’re feeling like you are in a good groove on something, try doing it backwards and see what you discover.

My Grandmother’s Hats

Near the end of her life my grandmother made hats – lots of hats. Most of them made from cheap acrylic yarns and most sized for children. She lived alone and spent much of her time sitting in her apartment watching C-SPAN and knitting. At the time her apartment was a few blocks from my office so I went to visit her about once a week and we’d chat about current events, politics, and whatever else was on her mind. If you went to visit her during the winter and you were not wearing a hat when you arrived (and since she was on the 19th floor I usually took my hat off in the elevator) you were strongly encouraged to take a few to keep warm when you left.

Blue and white handmade hat.
This is one of the hats I still have, made from yarn my mother had used several years earlier for her own project.

My mother and aunt would bring her yarn from various sales and the ends of projects, and I would occasionally take a bag or two or hats with me after a visit. I tried giving them away on the street to homeless people who often slept near her apartment or my office but they weren’t usually interested. I sent a couple bags to Afghans for Afghans (before I realized they needed items made from better fiber). We sent some to a friend who taught in Buffalo (Nanny loved the picture of children wearing her hats sent in the thank you note).

Even having given away a couple bags full, when it came time for her to stop living alone we discovered there was a closet full of grocery bags stuffed with hats. We laughed at the large number of hats she had stashed away but as her last charitable act – even though it was one she never knew about – we gave them away. My wife and I gave them to school teachers who had children in need. My sister took a few hundred to a women’s shelter. And we donated them to other useful causes when we could find people about to use them. Slowly my grandmother’s work was spread across several communities.

Fibre on the shelf
Fibre in our house that isn’t used promptly may become a decoration.

In addition to working in technology, I also spin – as in make yarn from wool and other fibers. I get more or less done depending on the ebbs and flows of life, but I generally have some fiber in the process of becoming yarn on a spinning wheel. My fibre stash is small compared to some spinners’ but since lots of my fiber is either a gift or a random purchase from events like the Maryland Sheep and Wool Festival instead of a specific project, I usually have extra yarn around too.

spinning wheel
My Ashford traditional, a gift from my mother-in-law. This is my primary spinning wheel, although I have a few others.
Wool yarn I made recently.
Wool yarn I made recently from fibre my mother-in-law gave me a couple years ago .

I also knit. And yes, I knit hats. Visiting in her apartment talking about the hats she was making Nanny shared – more than once – the pattern she used to make all those hats. She’d memorized it from some magazine or another years earlier (or at least a pattern like it) and so I typically make them about them same as she did:

  1. Pick a size needle that works with the yarn you have, and will result in the size you want (this takes trial and error if you aren’t an experienced knitter who knows their gage).
  2. Cast on 96 stitches (she sometimes said 76, but usually it was 96 and that appeals to my techie nature).
  3. Work 2 inches in Knit 2, Purl 2.
  4. Work 3 inches in Stockinette.
  5. Knit 2 together, Knit 6, repeat to the end of the row. Purl back.
  6. Knit 2 together, Knit 5 repeat to the end of the row. Purl back.
  7. Continue this way until, decreasing the number of stitches between each gather until your gathers start to collide. Then knit 2 together every stitch (still purl back).
  8. When you have 4 or 5 stitches on the needles, cut the yarn with about a foot of extra.
  9. Pull the end back through the remaining loops, and stitch down the open side.
This is one of mine made from some yarn that was started during a workshop I did on spinning with a drop spindle for high school kids.
This is one of mine made from some wool left over from a workshop I did on spinning with a drop spindle and some singles I used to teach a friend’s kids a little about spinning. Once plied together the variation in the singles formed a surprisingly good yarn with interesting color mixes.

I’ve made hats for friends and family, although certainly not in the volume she produced them. Last winter I realized I could use some of my stash of fiber to start making them to give away like we did with hers. And so now I have a slowly growing collection of hats (some with matching scarves) in the hopes that I am able to do something half as useful with them as we managed with my grandmother’s.

Here's one of the matching scarves. These are sized for child, but I'll probably make more sizes over time.
Here’s one of the matching scarves. These are sized for child, but I’ll probably make more sizes over time.

My grandmother was a challenging person in many ways. But she always tried to be nice to strangers and people in need. So this has become my tribute to the parts of her that I loved most.

Looking at a project from different angles

For our 15th anniversary my wife and went to the south island of New Zealand, with a long layover in Sydney. We only had a few hours in Sydney so we went to see the Opera House and then walk through the botanical gardens next door.

As we walked around the harbor I took pictures of the opera house from several different angles. And that got me thinking about the advice I’ve been given both about photography and about my work: make sure you try things from different angles.

A classic angle of the Sydney opera house from across the harbor.

Too often all kinds of experts get into a rut and lose track of the perspective non-experts, and other experts with whom they disagree. Cable news channels like to package those ruts as two talking heads yelling at each other by calling it “debate”.

It’s an easy trap to fall into even without watching the people paid to yell at each other. Sometimes when we look at a problem twice it looks different because we changed something small, and we think we’ve seen all the valid angles. But we’ve just reinforced our sense of superiority not actually explored anything interesting yet.

When you look right at the sun a small change can have a large impact, but you may still be fundamentally in the same place with a fundamentally flawed perspective.

And sometimes you look from a new angle and something easily recognizable becomes new and different, but that’s not always an improvement. There are reasons for best practices, and sometimes we just reinvent the wheel when we try to break our own path.

You don't see pictures of the opera house from this angle often – which is probably for the best.
You don’t see pictures of the opera house from this angle often – which is probably for the best.
This angle was even worse. It's a good thing I wasn't using film for this exercise.
This angle was even worse. It’s a good thing I wasn’t using film for this exercise.

And sometimes it is important to think about the extra details that you can capture by changing perspectives and taking the time to figure out the best approach.

Opera House with sailboat
I had to wait a few minutes for the sailboat to get into a spot that made it look right.
Sometimes too much context is too distracting.
Sometimes too much context is too distracting and makes it hard to know what you’re supposed to look at.

But when you take the time to look at things from different angles, perspectives, and positions sometimes you get to discover something you didn’t know to ask about.

This little guy and an older buddy spend lots of time in the sun on these steps behind the opera house – I had no idea they were there until we were walking around.
This little guy and an older buddy spend lots of time in the sun on these steps behind the opera house – they are well known locally, but I had no idea they were there until we were walking around.

For me the best moments are those gems you find when you take the time to explore ideas and view points and discover something totally new. Nothing beats travel to help you remember to change your perspective now and again.

Picking tools you’ll love: don’t make yourself hate it on day one.

Every few years organizations replace a major system or two: the web site, CMS, CRM, financial databases, grant software, HR system, etc. And too often organizations try to make the new tool behave just like the old tool, and as a result hate the new tool until they realize that they misconfigured it and then spend 5-10 years dealing with problems that could have been avoided. If you’re going to spend a lot of money overhauling a mission critical tool you should love it from day one.

No one can promise you success, but I promise if you take a brand new tool and try to force it to be just like the tool you are replacing you are going to be disappointed (at best).  Salesforce is not CiviCRM, Drupal is not WordPress, Salsa is not Blackbaud. Remember you are replacing the tool for a reason, if everything about your current tool was perfect you wouldn’t be replacing it in the first place. So here are my steps for improving your chances of success:

  1. List the main functions the tool needs to accomplish: This is the most obvious thing to do, but make sure your list only covers the things you need to do, not the ways you currently do it. Try to keep yourself at a relatively high level to avoid describing what you have now as the required system.
  2. List the pros and cons of what you have: Every tool I’ve ever used had pluses and minuses. And most major internal systems have stakeholders who love and hate it – sometimes that’s the same person – make sure you capture both the good and bad to help you with your selection later.
    Develop a list of tools that are well known in the field: Not just tools you know at the start of the project. Make sure you hunt for a few that are new to you. You might think you’ve heard of them all cause you walked around the vendor hall at NTC last year, but I promise you there are more companies that picked a different conference to push their wares, and there are open source tools you might have missed too.
  3. Make sure every tool has a salesperson: Open Source tools can be overlooked because no one sells them to you, and that may mean you miss the perfect tool for your organization. So for open source even the playing field by having a salesperson, or champion, for the tool. This can be an internal person who likes learning new things, or an outside expert (usually paid but sometimes volunteer).
  4. Let the sales teams sell, but don’t trust them: Let sales people run through their presentations, because you will learn something along the way. But at some point you also need to ask them questions that force them off your script. Force a demo of a non-contrived example, or of a feature they don’t show you the first time. Make them improvise and see what happens.
  5. Talk to other users, and make sure you find one who is not happy: Sure your organization is unique but lots of other organizations have similar needs for the basic tools – unless you have a software-based mission you probably do not want an email system that’s totally different from everyone else’s. A good salesperson will have no trouble giving you a list of references of organizations who love the tool, but if you want the complete picture find someone who hates it. They might hate it for totally unfair reasons, but they will shed light on the rough edges you may encounter. Also make sure you ask the people who love it what problems they run into, remember nothing is perfect so everyone should have a complaint of some kind.
  6. Develop a change strategy: In addition to a data migration plan you need to have a plan that covers introducing the new tool to your colleagues, training the users, communicating to leadership the risks and rewards of the new setup, and setting expectations about any disruptions the change over may cause.  I’ve seen an organization spend nearly a half million dollars on customization of a complex toolset only to have the launch fail because they didn’t make sure the staff understood that the new tool would change their day-to-day tasks.
  7. Develop a migration plan: Plan out the migration of all data, features, and functions as soon as you have your new tool selected. This is not the same thing as your change strategy, this is nuts and bolts of how things will work. Do not attempt to do this without an expert. You made yourself an expert in the field, but not of every in-and-out of the new system: hire someone who is.  That could be a setup team from the company that makes it, a 3rd party consultant, or a new internal staff person who has experience with different instances of the tool.
  8. Get staff trained on using the new tool: don’t scrimp on staff training. Make sure they have a chance to learn how to do the things they will actually be doing on a day-to-day basis.  If you can afford to have customized training arranged I highly recommend it, if you cannot have an outside person do it, consider custom building a training for your low-level internal users yourself.
  9. Develop a plan for ongoing improvement: you will not be 100% happy 100% of the time, and over time those problems will get worse as your needs shift. So make sure you are planning to consistently improve your setup. That can take many forms and what makes the most sense will vary from tool to tool and org to org, but it probably will mean a budget so ask for money from the start and build it into your ongoing budget for the project. Plan for constant improvement or you will find a growing list of pain points that push you to redo all this work sooner than expected.You’ll notice I never actually told you to make your choice. Once you’ve completed steps 1-6 you probably will see an obvious choice, of not: guess. You have a list, you listened to 20 boring sales presentations, you’ve read blogs posts, white papers, and ad materials. You now are an expert on the market and the tools. If you can’t make a good pick for your organization, no one else can either so push aside your imposter syndrome and go with your gut. Sure you could be wrong, but do the best you can and move forward. It’s usually better to make a choice than waffle indefinitely.