Good Enough Passwords

I deal with passwords a lot. In any given day I log into five or six servers, another dozen web sites, plus my personal systems and tools. Some are stored in password managers, some are memorized, some are mine, and some are shared. To avoid losing things I need I have several patterns, schemes, and password generation tools I use to try to keep up with it all and make sure I’m using good passwords most of the time.

I don’t have a deep background in cryptography and a definitely don’t consider myself an expert in the related math. But I have spent a lot time time with users and observing behaviors, so I do consider myself reasonably knowledgeable about how people actually behave with their passwords.

A couple years ago this XKCD comic came out, and an admin’s view of it has become one of my measures of their understanding of users and passwords:

XKCD Password Strength Comic

If you’re interested in understanding what technically right and wrong with the math and assumptions in that comic you might find some of the references on explain XKCD’s discussion interesting.

I don’t actually think that’s the interesting part. To me the interesting part is the number of system administrators I have talked to who are convinced the comic is wrong, because they are also convinced that anyone who doesn’t use truly random password and avoid password reuse is stupid and therefore their behavior can be ignored. These people have access to your system, and it’s your job to keep those system secure. The problem isn’t your users, the problem is we have made passwords unreasonably hard to do right.

At this point people generally know they are supposed to use good passwords, like we all know we’re supposed to brush after every meal. Sure there are people that do that, but not most of us. We all know that passwords should be long (even if we fight over the exact length to require) and use lots of different kinds of characters (although many password systems require them to be from the latin character set). But it’s too hard to follow all the rules, and security experts are so concerned about being right they don’t provide useful guidance about when the cheat.

Let me grant those who like long-random passwords the following point: if you use different passwords on every system that needs one, and they are all truly random strings, and you memorize them all so you don’t have them recorded someplace they could be stolen, you have the hardest for an attacker to crack. Great for you. But I work with people who are not perfect, have limited memories, and need to be able to have shared access on a regular basis.

Knowing the perfect random password generation pattern is useful in some cases (or so I’ve heard), but rarely are you in a case where you can use the perfect setup. I don’t care about perfect: I’m not perfect, I don’t work in a perfect office, have perfect colleagues, or perfect clients. So here are my good-enough rules for admins and developers.

0) Make it easier to do the right thing than the hard thing. This gets to be rule zero, because everything else is meant to support this idea. You want the path of least resistance to be the one that gets you the results that are secure enough to protect your systems from the attackers they face. Make sure your users have good tools for storing passwords, settings on password fields to encourage good (not perfect) behavior, and a minimum of stupid rules you don’t really understand but someone told you are “best practice”.

1) If you make it hard for people, they will find a way around you and likely weaken security. It might be post-it notes on monitors, cycles of passwords that are 5 long (because you force them to not use the last 4). If you make it hard to pick a password (because you required punctuation but not  ‘, \, “, &, or !), you will end up with lots of passwords that are curse words – and your attacker will thank you for shrinking the search space. If they are using touch devices to type them, they will do things like repeat as many characters as you allow to make it easier to type (if you ban any repeats: again the attackers thank you for shrinking the search space). All things you want them to stop doing.

2) Do not have a maximum password length. Any time I hit a system with an upper bound of 12 I want to scream (although jokes about chimps might be a better tactic). Even if you are using a secure hashing system that ignores all characters after some point: who cares? Why limit the attacker’s search space to only strings between 8-12 characters?!? Sure that’s a massive search space, but not nearly as big as it could be.

3) Do have minimum lengths. Minimum lengths forces your users to do two things. First, not use passwords that could be broken in less time than it took you to read this article. Second, it gives you leverage to push them to either good phrases or generators. If you’re smart and don’t have legacy systems to support go with something like 15 or 20 characters.

4) Expect people to share passwords. Many times this is actually a basic job function. If I won the lottery tomorrow (unlikely since I don’t play) and don’t come to work (also unlikely since I would wait until I had the money reinvested before making plans) the person taking my place needs to be able to access all the tools, servers, and accounts I’ve setup. If she can’t do those basic things I haven’t done my job responsibly.

5) Provide secure means to share passwords. I have more than once been sent a password in chat (running through Google, Slack, or once upon a time AOL’s servers), email, word documents, text file, and a variety of other terrible solutions. This happens not because my colleagues didn’t know it was a bad thing to do but because they didn’t have a good option. We spend so much time locking down passwords, that we don’t create secure channels to hand them around responsibly which defeats the purpose of secured storage.

6) Pay attention to how users will be using individual passwords. Not all passwords are created equal, which is why I encourage you to support throw away passwords: something short, easy to remember, and only used places it doesn’t matter if it were stolen. But even when a password is important there are issues like the ease of entering them: if I have to enter a password 4 times a day it better be easy to type or I better be able to copy and paste it. If I need it once a month it should be impossible to remember and its okay if it takes me 5 minutes to get it right. Most of us can’t type complicated passwords quickly, and if we have to enter it a bunch we want to be fast.  This is even more true for people using touch interfaces where shift is an extra keystroke as is changing to a different part of the standard keyboard.

7) Stop telling people they have to use a different password every time. This is an extension of number six. People have too many passwords, and that’s not changing soon. Sure we can encourage them to use LastPass, or a tool like it, but most people aren’t going to (and if they did that could be its own problem since it creates single points of failure). Tell them to use a different password when it’s important, and to use a throw away password or scheme when it’s not.

Not everything needs to be Fort Knox so stop pretending it is.  Important things like your bank account(s), your email, Facebook need their own passwords because they can be used to do real damage in the real world. Online communities, games, and other trivial places asking you to sign in do not.

8) Don’t lecture people about bad personal password habits. Honestly, this is probably the hardest one (here I am lecturing you about not lecturing them). Usually the first people to admit they are sloppy about passwords are developers and sysadmins. Sure, they will tell you about the awesome password wallet they use first, and the two factor authentication they created for their blog, but then toss off that all their production servers have the same root password and it’s 8-10 nonrandom characters. Even if you are perfect (if you are still reading this by definition you probably have room for improvement) don’t lecture people who aren’t. It just makes them feel they can’t admit when something has gone wrong, or if they don’t understand something. When you find people doing it wrong, show them how easy it is to do it right, and if it isn’t apologize and fix it.

Are you moving forward or backward?

Every week I try to ask myself: What did I do this week to make myself more valuable? Am I moving forward toward a goal, or further from it?

Handmade sign reading Are you going bkwds? or are you going fwds?
I spotted this the other week in Philly. I didn’t try to sort out the politics of the creator.

In technology, communications, or any other job that involves one of those two things you are either moving forward or moving backward: standing still is not an option. You are either learning new skills, trends, tools, and concepts, or you’re falling behind as other people build new tools that drive new ideas and trends.

I read lots of advice that says to plan your career two or three moves in advance. That is good advice, but I don’t think it’s wise to trust your gut that far out. The technology landscape changes too fast and too dynamically to believe you know where everything will be in three or five years. On the one hand I think it’s important to deepen my skills for the path I want to be on, but at the same time I try to broaden my skills into areas other areas that have things to teach me. In the back of my head there is always plan B and C, just in case plan A doesn’t come together they way I hope. In part, because I’ve never been able to stay on plan A very long: life always intervenes.

When I was 23 (and sure I wanted to be teacher) I was advised to read an hour a day in my field, and that it should not just be reading about the kind work I was already doing. At the time I was the new kid in IT of a mid-sized international nonprofit organization doing whatever no one else wanted to do: which is a great way to learn a variety of things. I didn’t really know anything yet about how to have a career – I had a job, and I liked my job, but couldn’t envision a career path.

But I took that advice to heart and tried to find ways to constantly learn about things I don’t know. I read books, listened to podcasts, and taught myself new skills. I learned about communications planning, economics, corporate strategy, algorithms, and a variety of other topics. The ideas I pick up from those sources help me think about technology more creatively, and helped me understand the importance of making sure I build tools that are useful not just cool to me.

For several years I also taught myself a new programming language every year. I taught myself ASP, C#, PHP, Python, Ruby (on rails and off), R, Haskell, and JavaScript because I heard other people talk about them as important or interesting. I have used five of those professionally to create actual software people used. And the others all forced me to see programming differently and helped me be a better developer. I don’t force myself into learning whole new languages annually, it was too broad and prevented me from deepening my knowledge of individual languages and the ecosystems I work in frequently (although I’m probably overdue to teach myself something like Go, Swift, or Rust.

The biggest thing I’ve learned from all these different inputs is that I need to live in constant fear of getting behind, outmoded, and sidelined. That fear keeps me motivated to learn more and push myself harder. By the time I retire I cannot imagine I will still be paid to be a full-time Drupal developer, I doubt that’s what I’ll be doing five years from now. Certainly by in 30 years Drupal, and the web as we know it, won’t look anything like they do today and I will be doing my job very differently. This is the blerch that keeps me motivated.

So every week ask yourself: what did I learn this week? Did I move forward or fall behind?

How to create a good trouble ticket

This week I was working with a new colleague on our account team. As with all people knew to working with technical teams and bug tracking, she’s having to learn how to create good trouble tickets for when clients report issues. This is a challenge I’ve seen played out in every place I’ve ever worked: developers want detailed tickets so we can dive in without asking 16 follow up questions, and people creating tickets don’t actually know what we want and assume we know how to find and fix the problem.  And so I’d like to try to offer this explanation of what we’re looking for and why.

At the most basic level I need to know at least three things to find and fix a problem on a project (either a web site or some other tool I’m supporting):

  • Where in the program is the problem? This is usually a link to a sample page that has the problem.
  • What happened? I need a clear explanation of what went wrong. Is there a picture missing? Is the text format wrong? Is there a big red error message at the top?
  • What you expected to happen? What is the picture of and where exactly was the picture supposed to appear? What formatting was supposed to appear on the text? Did you do something right before the error message appeared that helps me see that message again?

These things are part of allowing me to reproduce the problem.  If I can’t reproduce the problem, I can’t promise you I fixed it. If you can’t reproduce the problem, you can’t check that I’m right.

Developers will often say that if you can’t give me the step to reproduce a problem I can’t fix it. But in my experience sometimes a problem is actually really hard to reproduce, and you need a developer or a professional tester to actually figure out those steps. So it’s okay if you can’t give me perfect directions, but give me what you have.

If you find yourself writing a ticket that doesn’t say more than “Search is broken” or “Blog post didn’t look right” the problem better be massive (think big red error message level). As an account/support team member that may be all you got from the client but someone has to fill the gaps – and developers are terrible people to have fill those gaps.

As a developer there are several reasons that’s true.

First, rarely do developers get the luxury of working on one project for an extended period, and when they do those tools are large and complex. So we probably don’t have every detail in our heads at any moment. If we could store all that information we wouldn’t need task tracking systems, you could just call us and tell us about a problem as we’d call you back a few hours/days/weeks later and say “fixed”.

Second, we’re terrible at finding mistakes in our own work. Like everyone else, we need editors. If I could see the problem you are reporting, I would probably have fixed it, or at least reported it to you so we could open a task to get it fixed later.

Third, we probably don’t spend as much time in the project documentation as you do. So if someone needs to track down the original design to check for a discrepancy between the design and what’s happening a developer is probably going to be much slower at this task than you are (or you will become soon).

Also remember your developer probably will not look at the problem today unless it’s mission critical to the client. So they need to be able to figure out three weeks from now what you were talking about.  If it just says: “search is broken” and I run a search in two weeks and everything looks fine, you are going to need to tell me what’s broken about it (maybe a result is missing, maybe it’s formatted wrong, maybe it’s working perfectly but the client doesn’t like the results).

Even with all that context, I know it is intimidating for many new support or account team members to crack the code developers use when talking. We over explain this, using technical terms, and get annoyed too quickly when people don’t understand us. And we often forget that teaching by analogy is helpful.

My new colleague is a baker, and so as I was trying to help her understand what I needed to be helpful on tasks I switched to bread:

If I came to you and said “my bread didn’t work out, please tell me how to fix it” how would you start?

That helped her make the connection. Just saying my bread didn’t work out, doesn’t tell her enough to help me do it right next time. She’s going to have to ask several follow up questions before she can be helpful.

  • Did it taste wrong or look wrong?
  • What kind of bread was it?
  • Did you follow the instructions or do something different?
  • Are your ingredients fresh?
  • Did it rise enough?
  • Did you knead it enough?
  • Did you set the oven to the right temperature?

On the other hand if I come to her and say:

I tried to make sourdough oatmeal bread over the weekend. I followed the recipe closely, but my bread turned out really dense instead of having the bready texture I expected.

Now she knows there was a problem getting the bread to rise. So we can focus questions on the yeast and other details of getting air into bread. Yes, there are still several things that could have gone wrong, but now we know where to start.

Frequently new support staff are intimidated by all the technical things they don’t know. And too often developers brush aside new staff who don’t give them the information they need and just say things like “Oh I’ll figure it myself” instead of helping their colleagues learn. Part of the solution is to help people understand that the first set of questions aren’t actually technical. Baking isn’t the right analogy for everyone, but it helped in this case. And hopefully next time I’ll do better at getting to a better explanation quickly.

Also, my bread came out fine and I’m taking her a loaf this weekend. You are welcome to try my Sour dough oatmeal bread recipe.

What to say

I’ve been struggling with what to write about our new president, the protests he’s triggered, the ban on refugees he’s ordered, the families he’s dividing, the hateful things he has said about many groups, and all the related news. While this is mostly a technology blog and much more articulate people are already saying things worth hearing, these are not events I can allow to pass without comment.

For a variety of reasons I haven’t been able to attend the protests, except I accidentally attended part of a rally in Philadelphia (I thought the rally pictured above was over when I met a friend for dinner right in the middle of it), but it’s been exciting to watch the sustained energy the last few weeks.

Last week my wife paraphrased some of the memes that have been going around: Remember all those times in history class you thought “If I had been alive during ____ I would have ____”? What you are doing now is exactly what you would have done then.

She didn’t mean it as a direct challenge to me, but I haven’t been able to let it go. In part because while I’ve had good answers in the past, I’m not sure I have a good answer at the moment. Besides, the challenge takes on another level of importance when coming from a historian.

While I was in Philly I visited American Friends Service Committee’s Waging Peace exhibit marking their 100th year.

For four generations my family has spent time working and volunteering for AFSC, and there were markers of some of our work in the exhibit. Our widely different roles matched the times and events we were living through. As young adults my great grandparents worked in post-World War I Germany and France, and my great-grandfather later led the construction of a housing co-op. My grandfather and father served as conscience objector with AFSC during World War II and Vietnam. Most of my father’s siblings and their children have volunteered in various ways with some AFSC program or another, often packing donated clothing in the basement of Friends Center in Philadelphia. I spent ten years working in the IT and Communications departments in Philadelphia.

Alice Forsythe in Hamburg in 1922.
A Hurford Crosman’s passport picture from the AFSC Archives

My great grandparents there was little formal training or planning. They both took on roles I can barely imagine doing now, let alone doing them at 22 in countries destroyed by war. Later in his life my great grandfather described part of his work after the armistice:

I was sent with an English boy to Bethlainville, near Verdun, none of the now returned owners knew who we were or why we were there. “To plow our land? Oh no! Americans come to take our land away from us?” My two high school years of French were put to a real test. We are here for Love. We are not the Americans you have met with during the war. We want no pay. But one young girl who had lost both her father and brothers, completely alone, said she would take a chance. “Oh but you’ll be raped said the others” O.K. she will have a gamble. So Jacoby plowed from day break until noon and then I plowed from noon until I couldn’t see at dark. Land all plowed, now we will see what these young men really want. Nothing? Nothing but love. Not the kind she expected. From then on everybody wanted their land plowed. Later on another crew of Americans came with seeds, tools, clothing, etc. So the people of Bethlainville were restored to almost normal living.

Relief work has grown a bit more formal in the last century.

Self-Help housing signs from AFSC's 100's anniversary exhibit.
My great grandfather helped this program in Ohio and Philadelphia.

He later returned to AFSC to support the self-help housing program, first in Ohio, and then leading the program in north Philadelphia which renovated a building to create a co-op that still functions today; my cousin, his wife, and daughter are members of the co-op.

My own work at AFSC was much less exciting: I sat in relatively comfortable offices and made sure the technology needed to support the programs worked. I supported mostly communications, and fundraising, and my program involvement tended to be limited to supporting their public facing functions. I was in entirely administrative roles: I was overhead.

Friends for Peace was a fun project, and successful in many ways, but creating a web site for sharing peace images and raising money isn’t the core work of the organization.

Now I easily see what I couldn’t always see then: the value I contributed over 10 years. Too often we denigrate administrative functions at nonprofits as wasteful distractions from their true mission. But the truth is that well done administration is critical to the success of any good organization.  They need money to pay the staff, they need web sites to inform people about their work, they advocate for systemic change, they post pictures of their own pets to grab your attention so you will help their work.

Now I’m struggling to find ways to help as the world seems to spin out of control and the US government does things even worse than we did while I was at AFSC (the US went to war twice, admitted torturing people, and openly spied on citizens, so this is a fairly high bar to cross).

I’m amazed I was able to get much done for peace or anything else in that work space.

While I’ve been pleased, and frankly surprised, at how effective the current protests appear to be, I believe over the next four years the resistance the administration’s fermentation of hatred and distrust will have to be a community by community movement. The lack of trust between communities, and the lack of political power make sustained broad-based movements hard to imagine. There are communities under new attack, and others that have been under assault for decades now starting to find their voices: many are voices of pain and anger. Even as we see mass protests we also see infighting within them and outside resistance from communities with shared interests.

I’ve written in the past about my sister-in-law’s students in Maryland. My wife and I posted similar messages to all our personal and social media networks, and hundreds of postcards and letters have poured in from all corners. The teachers read them to the kids, and sent them home so their families also know there are Americans who welcome them. Some of those letters came from other immigrant children, and her students are now writing back to share their warmth and stickers, with the other children. It doesn’t fix everything, but provides some of what that community needs: love, the kind my great grandparents carried to post-war France and Germany.

There need to be people in the world who are willing to go into the most dangerous places, but we can’t all be those people. We also need people working actively in their community to help spread love, respect, and basic dignity.

I’m still struggling to find ways to have a positive, and effective impact, but here are things I’m trying:

  • Making hats for a newly arrived Iraqi family I learned about from a woman at church this morning.
  • Calling and writing to my elected officials to thank them for the things I think they are doing right and asking them to change what I think they are doing wrong.
  • Writing and speaking openly about my concerns.
  • Attending rallies when I can, and supporting those who do when I cannot.
  • Donating to organizations taking the kinds of action we want to see in the world including Planned Parenthood and likely others (we’re still working on who all that will be).

Please share with me (and others) what you’re doing and what else you think can be done.

I believe that to work for positive change we all need to work in and with marginalized communities and help find positive solutions for their problems – even those expressing their frustration through hate toward others.

A Pattern for Drupal 8 Blocks

For Drupal 7, I have a pattern to help simplify creating and managing custom blocks for a site. Since the standard hook_block_info() and really hook_block_view() implementations tend to get messy and junked up with markup in the code.  My solution was the use the block delta key from the block info array as a helper function name called from the hook_block_view(), and a theme function in hook_theme() to make sure I could easily create a template for each block. I’m not going into detail about it since was hardly original and I’ve seen several variations from other developers.

When I started to work on Drupal 8 I wanted to develop a similar pattern to help create simple conventions for integration with front end work on a project. While the new plugin system helps avoid the ugliness of old hook_block_view() implementations, how to create a twig file for your custom block isn’t obvious and doesn’t impose a naming convention. Worse, I’ve seen lots of example code with blocks that return render arrays of type markup meaning they have HTML in strings in PHP.

Most projects involve a collection of nearly static blocks that provide basic information like the copyright information, a disclaimer text, a link to the firm that built the site, decorative flourishes, and other similar elements that don’t benefit from being managed as content. After a couple experiments I’ve come up with a solution that works pretty well:

A visual of module files also described in main text.
Outline of the custom blocks pattern.
  1. Create a module for simple blocks (although I use the rest of the pattern anytime I’m creating a custom block).
  2. Create the block class for each block within the module at src/Plugin/Block/CopyrightBlock.php.
  3. Create a theme function for each block in the main .module file.
  4. Create a twig file to implement the theme function within the module at templates/custom-blocks-copyright.html.twig.

The for a copyright block the class itself is simple.  We implement the variables we want rendered by the block, and set the cache to expire at midnight (one of the places a timed cache makes more sense that cache tags):

With the block created, we need to define the custom_blocks_copyright theme function we included in the block using hook_theme() from within the dot module file.  Remember any custom variables you used in the block class need to also be defined here (in this case attributes and year):

Finally, create the twig file to provide the actual markup.

Drupal 8 Custom Plugins are addictive

In December I gave a talk at the SCDUG meeting on Drupal 8 plugins.  This is based on that talk with a few improvements and additions from things I’ve learned since. 

Drupal 8 provides many great improvements, but one I’ve been finding to be the most exciting is the easy ability to create my own plugins. They are actually a bit addictive, and it’s easy to start seeing projects as an excuse to create a new plugin manager. At first it can take a little bit of effort to get your head around the process, but I got great details from Unraveling the Drupal 8 Plugin System from Drupalize.me. But the problem with those, and all the examples I could find, was they were either too deep (the drupalize.me article should be considered required reading, but the example is too abstract for most people) or too shallow (the drupal.org Plugin API examples currently lack context about why I would want to do this in practice).

So my goal is to straddle that gap.

What’s a plugin?

Plugins are a design pattern that let you extend a module or service. Drupal 8 has at least four plugin discovery systems:

  • Annotated ← I’m mostly talking about this one, its used for blocks, views, rules, Queue API, and more.
  • YAML ← These are used for menus, routes, and services.
  • Hook ← D7 carry over hook system, still used in many places.
  • Discovery Decorators ← Wrap around other plugin systems to replace/improve on the classic info alter hooks from previous Drupal versions.
  • Static ← These are used for test code, and I’m ignoring here.

Basic Commonly Used Plugins

As a Drupal 8 developer you probably use several plugins when building a site, particularly if there is any custom functionality required. When creating controllers you’ll need a route, and likely a menu item, which means you’ll use YAML files to define those aspects.  For blocks, the queue api, or Rules you’ll create an annotated plugin.  Theme functions are still commonly defined with hook_theme().

Why do I create new plugin?

Between core, and major contrib modules (like Rules and Search API), plugins options abound, but it’s not always obvious why you might want to create one for a more focused project. For large projects that may have a large number of related functions, poorly defined use cases, or use cases that are expected to expand over time Plugins provide three major advantages over other approaches:

  • Plugins make it easy to easily keep classes small and focused.
  • Plugins make it easy to extend a service.
  • Plugins allow one module to easily extend the function of another.

Actual Examples

Cyberwoven has a client who needed an internal training portal. They want to provide some basic gamification, particularly badges, and they wanted a more detailed tracking of user behavior than the core statistics module provides. The badges present a function that has many related, but different, use cases and is likely to grow over time. So I created a pluggable service that allows us to easily define a set of conditions for creating a new badge. Each plugin’s annotation lists the events it cares about, and when one of those events is fired the service runs the plugin’s test function. If the badge has been earned the plugin returns the information needed to generate that entity. All the event listening, entity generation, and error checking happens in the service, so each plugin is only focused on the details of the one badge. The event tracker is similar, with each plugin defining a filter for the major system events to determining what details should be recorded, and the service handling the logging of those details and reporting. As the client needs have evolved we’ve been able to add and modify the plugins to meet their needs.

The second place I’ve created a custom plugin system for was an internal project at Cyberwoven (it was actually the project I used to learn how to create custom plugins). Our testing server has always had a service to perform some basic operations on the server, like pulling repos via webhooks and similar tools to help developers. Setting up our development environments for D8 meant it was time for a new test server, and therefore new tools to manage it. From the start I wanted to create a tool that helped both developers and account managers. So I created a D8 site to manage the sites that provides various tools to perform task operations we all need (not only code handling, but also checking if security releases apply to any sites). All the various operations we want exposed through that site are created as plugins to our custom manager. Each task plugin is focused on just that one task. All the other various needed to track the sites, route requests to run a task, and render their responses are all handled in other places.  It’s also now making it easy to add features that wrap around the plugins since the plugins all have a consistent interface (like adding new displays and running some tasks through cron).

Introducing Site Manager

List of example sites.
The listing page for the site manager tool. This runs on our server and can be run in developer’s sandboxes as well.

The main module defines an entity for each site, a couple displays (like the listing page displayed above), and the annotated plugin base so we can add new features easily. Plugins can be enabled and disabled on a site-by-site basis, and can be run from the plugins dropbutton.

Once we had the basics in place (like getting the git and drush status’s via simple plugins) we started to add more powerful features to turn the tool into a dashboard.  We added the module search block you see on the left as a separate module that searches all sites for a specific Drupal module (nice when security updates are released). The search module also provides a plugin for each site that lists the modules for just that site.

The last site in the screen shot has a reference to scanning the HTTPS certificate for a specific site.  This is another feature we added recently as this tool increasingly starts to serve as a dashboard. The HTTPS Scanner module again provides both a batch job to scan all sites, and a task plugin that scans just one.

As of this writing, all told there are nine working task plugins across four modules (there are a couple more underway as well). Tasks can also have their results displayed in a block when you look at the details for a site.

Building your own manager

Hopefully by now your convinced that it’s worth having your own plugin manager at least sometimes. So the next question is how to build them.

Elements of a custom plugin

Plugins have several parts, most of which are small and simple:

  • Annotation Plugin Definition
  • Plugin Manager Service
  • Plugin Interface
  • Plugin Base
  • Example Implementation
  • Controller (optional)

Annotation Plugin

The first step is to define the annotation for your plugin. This is a simple class that extends Drupal\Component\Annotation\Plugin to define the variables you want listed in the annotation comments of the plugin. This file goes in custom_module/src/Annotation, and as always the file name and class name match:

Plugin Manager Service

The plugin manager itself is a service, but the vast majority of its function comes from the parent classes.  The convention is to place this file in custom_module/src/Plugin (as will the rest of the base definitions we’re about to provide).  In this case we call it: TaskPluginManager.php.

The full version of this provides a few more improvements to this class (I overrode the getDefinitions() method to provide some filtering options to support disabling plugins), but those aren’t actually required for it to work well.

 

In short, this class just provides some general definitions to separate it from all the other annotated plugins.

And since the plugin manager is a service, we add it to our module’s service.yml file:

services:
   plugin.manager.task_plugin.processor:
     class: Drupal\sitemanager\Plugin\TaskPluginManager
     parent: default_plugin_manager

Plugin Interface

Next we define the plugin’s interface.

 

For this plugin I defined 4 critical public functions, three just provide basic information about a plugin, but fourth (run) will be the interesting one in a minute.

Plugin Base

Finally we’re getting to something interesting (or at least something that requires code).  The plugin base is an abstract class which all the actual plugins will extend. This goes in custom_module/src/Plugin/TaskPluginBase.php.

The things worth taking notice of here are the fact that I inject several services into the base class, so I have them for each plugin: translation, main sitemanager module configuration, and a wrapper on Symfony’s process service to make running drush and git commands easier.  The three informational functions are fully defined here, but the run function stays abstract since it’s the function that actually justifies creating a plugin at all.

In theory I could stop right there. I’ve created a plugin manager and defined all the things someone else needs to be able to use it. But there are two technically optional parts that are useful if you like to work and play well with others:  an example implementation and a controller for use or testing purposes.

Example Implementation

We went through all the trouble to create the plugin manager, before you stop you should create at least one plugin to prove all the work we just did actually works.

This is a simple task to use drush to run cron for a site.  The site entities know the location on the file system for Drupal root, and the simple task service handles the extra settings to control where the commands are run (and time outs and error trapping), so we can safely feed that to command and the location to get the response.

The method returns a render array, that can be used as part of a response.

Controller

The controller is truly 100% optional.  There are two reasons you might want to create one: to actually run the plugin if that makes sense (it does for the site manager), and to provide for easy testing of a plugin.

For site manager I created the controller because I actually needed it to run the tasks from the dropbutton (its the link used by all the links on those buttons).  But when I created the others for clients, I realized they can be very hard to test. They are buried inside several layers of complexity, making a test suite is a challenge to create correctly. And since are of the reason to use a plugin for a project with growing definitions, the test suite is often of limited use for rush additions. But by having a carefully built (and secured) controller and route, you can create an endpoint to use to test the plugin.

Because it’s optional, and well covered elsewhere (or heck just use Drupal console to generate it), I’m going to skip over most of the details of actually building the route, and linking that to the controller, and focus on the single function of running the task itself. To keep this short I’ve removed lots of extraneous details like security and error trapping – creating a tool like this isn’t for beginners so I’m assuming you know how to do those things.

The plugin manager service was injected into the controller (code not shown) so I just have it create an instance of the task requested. The controller calls the run function.  If the task is successful the output (a render array) is used for the response, otherwise a simple message is send. This sends an AjaxResponse to work with Drupal’s standard JS handlers, but it could generate a full page just as easily.

Final thoughts

Being able to create your own plugins easily is really nice.  In Drupal 7 custom plugins where generally considered an advanced developer task. The improvements in the abstraction (and the fact that true plugin support is now in core not pulled from ctools or other critical modules), means that while its still not a beginner task it is something all Drupal developers should be learning.

But remember you don’t always need or want the complexity of your own plugins. I’ve found the idea a bit addictive, and I get unreasonably excited when I find a place plugins are used or that it makes sense for me to create a new plugin type. For custom work you can almost always do the same thing using extra functions on a service, controller, or event listener, so this is a judgement call in the end.

Responding to Drupal Break-ins

If you support any web site long enough you will suffer a break in. If you support lots of web sites you will suffer them more often than you’ll want to admit in public. A few weeks ago my number came up again in the attack lottery when we discovered a client’s web site was being used as a proxy and redirect to a fake shoe site.

It wasn’t the first time I’d suffered a break in, and unfortunately I don’t expect it to be the last. My last experience with a major break in was shortly after Drupalgeddon (I patched all the clients I was supporting before they were breached but had to clean up sites that weren’t patched by other vendors), and the attackers had learned a few new tricks in the meantime.

If you are responding to a break in on a Drupal site there are directions on drupal.org to help guide you through an attack response, but I thought it might be helpful to talk through a version of what response can look like in practice. I think it’s also useful for us all to admit our weaknesses from time to time to help us all make sure we’re making new mistakes.

Overview

At the outset I’m going to admit we never found the initial source of the attack, what we did find were the tools they placed after the break in. The most likely cause was poor server patching practices by the client’s host, but there were also some Drupal security patches that had been slow to be get installed as well. During the attack I worked with members of the Drupal Security team (particularly Greg Knaddison who generously provided feedback on this article as well – of course any remaining mistakes are mine), who were helpful in giving me suggestions and who were clearly interested in helping us make sure we resolved the problem.

The site was being used as part of a scam advertising network. The attacker was leveraging the reputation of the site to create records in search engine indexes that were redirects to a fake shoe sales site. There were also a number of tools placed on the server that gave them full access to the Drupal database and the ability to run arbitrary PHP scripts. And it was clear by the end they had placed additional backdoors we never found – they may have had full control over the OS as well.

How we found out

Google told us.

We got an alert from Google reporting SPAM content on the site. At first we couldn’t find the content they were talking about, which unfortunately slowed our escalating our response, because it was only directly available to search engines. The junior developer who was initially assigned to review the message from Google eventually figured out how to find the listing on Google (a Google site search for Nikes and some hash codes the attacker was using), but couldn’t figure out how out it got there, and escalated the task to me.

Once I saw what she’d figured out my stomach sank. At first I was still hoping there might be some other explanation, or some simple matter of a single user account getting exploited, but that seemed unlikely (since we couldn’t find the content on the server) and I quickly knew it was going to be a mess.

Initial Response

The first thing I did was make sure we had a copy of the exploited site: code, files, and database. I would have rolled the site back to a recent backup, but our five-day rolling database snapshots were not enough to get back to before the attack began. We spun up new virtual machines for myself and another senior developer to start reviewing copies in environments isolated from other work.

Since the URLs we had for testing were a fairly unique pattern we started to Google those – and we got lots of hits. As soon as we knew the problem was larger than our site, we opened an issue with the Drupal security team and started to feed them all the information we had gathered. While their practice is not to get involved in resolving attacks directly (their role is to ensure the security of Drupal core and contributed modules), they were supportive and helpful in suggesting places to look for problems and resolution strategies.

Attacks we found

By the time I was alerted to the problem there were already several malicious tools installed, some of which I’d seen versions of before, and some were new to me – all were designed to be hidden from sight through some simple but effective obfuscation. Over the course of the next couple of days I found several backdoors manually, wrote tools to help me find more, and played entirely too much whack-a-mole (more on that in a bit).

There were two main categories of attack I was chasing: PHP scripts scattered around the public files directory, and records added to Drupal’s database tables.

Database table exploits

If you dealt with sites in the aftermath of Drupalgeddon, or other hacked Drupal sites, you have probably seen what happens when an attacker inserts PHP into carefully targeted parts of a Drupal database. In the ones I’d seen before attackers replaced the callback functions in Drupal’s menu_router table with PHP of their own. In this case the attacker used the Block module’s ability to use PHP to place a block to provide themselves a way to execute arbitrary PHP by sending a post request to the server. They leveraged the fact that the main system block is always available and therefore is a reliable place to insert a backdoor. By posting a form with a specific form element they were able to execute arbitrary PHP and therefore use that to place additional malicious code.

The attacker also leveraged Drupal’s system table to get more complex attack code loaded. They created a record for a file to be loaded as a module and then uploaded that file to the site’s files directory where they were guaranteed Drupal had write access.

filename: sites/default/files/styles/medium/public/57h3d21.jpg
name:overly
type: module
owner:
status:1
bootstrap: 0
schema_version:0
weight: 0
info: a:11:{s:4:"name";s:6:"overly";s:11:"description";s:58:"Displays the Drupal administration interface in an overly.";s:7:"package";s:4:"Core";s:7:"version";s:4:"7.32";s:4:"core";s:3:"7.x";s:7:"project";s:6:"drupal";s:9:"datestamp";s:10:"1413387510";s:12:"dependencies";a:0:{}s:3:"php";s:5:"5.2.4";s:5:"files";a:0:{}s:9:"bootstrap";i:0;}

This was the script doing the redirects and filtering traffic so that pages only appeared to search engines. Usually these records have filenames that are .module, .php, or .inc files, but in this case it was a .jpg file named to be similarly to actual files on the site to make it hard to spot.

The content of that file was a PHP script not an image. The script did several things, and was the main tool the attacker was actively using during the time we were trying to stop them. It served as a simple proxy of content that they would present to the search engines, and redirect those same pages to the scam site for anyone else. It also provided code to make sure the content of user login forms was sent to the attacker, and a backup backdoor incase some of their others were lost.

We actually had to remove this particular attack more than once (always using the misspelled “overly” module) and each time it came back with a new file, and each time using a different but similar disguise to try to make their code blend in with legitimate files.

.htaccess files in public files

The other trick that was new to me (and a more aggressive stance by Drupal core on this approach is being discussed) was to take advantage of the .htaccess patterns in Apache to re-enable PHP execution within the public files directory. Drupal’s default .htaccess file disables PHP at the root of the public files directory and in theory all subdirectories, but that can simply be undone by a malicious .htaccess file (unless you block it in Apache’s main configuration – which in my opinion defeats the purpose of using .htaccess).

The attacker had placed a number of basic PHP-based exploits on the server using this technique to allow them to run the scripts. The tools themselves were not Drupal-specific, and likely the .htaccess file would work just as well on a number of other PHP-based CMS platforms.
Since the files directory gets deep and complicated there is no reasonable way to scan the whole thing by hand: particularly since several of the files were using inaccurate file extensions (like .jpg or no extension at all) and file names meant to blend into the background. So in addition to checking for any .htaccess files below the files directory root, I wrote a simple Python script to scan a directory for anything that includes the string <?php:

How we fixed it

We immediately made sure all code on the site was up-to-date, and I removed every exploit I could find. And for a couple days I played whack-a-mole with the attacker. Every day I would remove a series of exploits, disable their ability to redirect users to their scam, and every night they would break back in through a backdoor I’d failed to find.

The final solution was to replace the server, deploy a version of the code known to be good, and deploying copies of the database and files that have been scanned for any PHP in places it shouldn’t be – which involved a combination of the scanner above and hand checking every place the database stores PHP, not a fast process.

What we will do better next time

Part of any security event of this nature needs to be a full review of your internal processes and controls to make sure you reduce your risk and improve your response the next time something occurs (because unfortunately there will be a next time for all of us).

One of our first areas of improvement is a shared understanding that it’s more important to resolve the attack than determine the cause. This goes against the question that developers are constantly asked during and after an attack: “How did this happen?” While you need to know something about what happened, in the end it’s more important to make it stop. I still don’t know what happened that started the attack, I know I stopped it by blocking every attack vector I could think of and replacing every part of the stack with a version known to be fully up-to-date. It would have been faster and cheaper if I’d just started there: yes there is a risk I would have missed some of the code in the database if I hadn’t taken the time to review what I was finding, but frankly I doubt that risk is has high as the risk that new exploits will appear while I’m working to understand the previous one.

Beyond that basic shift in approach we developed a three part list of improvements:

  • Things we needed to improve right away.
  • Things we needed to improve soon.
  • Things that should be part of ongoing improvement.

The highest priority items were coming up with better internal process for initial response, and making sure we are deploying all security updates in a timely but still careful manner, including monitoring our hosting partners to ensure servers stay up-to-date as well. These are basics that are easy to let slip over time – particularly monitoring that your partners are doing their job correctly.

The second category of fixes is filled with workflow and procedure improvements. We were already were in the process of improving our code handling (migrating from SVN to Git, better production monitoring, more internal code review, etc), and we accelerated our plans to complete that work. This category also includes a complete review of our existing backup procedures to make sure they provide the level of coverage our clients need.

The final category of longer term adjustments includes tasks that include ensuring all developers are given (and expected to take) professional development opportunities around security best practices, doing more internal sharing about emerging ideas and trends, and encouraging more community engagement so we are better able to leverage the community resources in a crisis.

Yes I wear pants and other advice for working from home.

I’ve worked from home part-time or full-time for the past four years. The first question I’m always asked when people learn I work from home is “Do you wear pants?” The answer is yes, I wear pants to work even when no one can see me. And frankly I don’t quite understand the desire of large numbers of people to work without pants, but it led me to realize that it might be helpful to share a few tips for people starting to work remotely.

Wear Pants

Okay, it doesn’t have to be pants, but get up and get dressed in clothes you don’t mind being seen in. The clothes we pick communicate, not only to other people but also to ourselves. Pick clothes to help you take your work seriously.

Set a routine

Have a routine about when you start, and what you do to prepare yourself before you start for the day. Your commute might be a walk down the hall instead of a drive across town or a ride on public transit, but most people I know still benefit from a pre-work routine. Walk the dog, go for a run, eat breakfast, or some other basic activity. It doesn’t matter a whole lot what it is, but have a routine that gives you a few minutes to get settled into a frame of mind to be focused on what you are going to do at work.

Make sure the technology works

You never want to have to say “I can’t do that from here.” or “I’ll do that next time I’m in the office.” This is particularly true when you have a part-time remote arrangement (i.e. if you work from home 1 or 2 days a week), and the systems may not be setup to fully support remote workers. Push hard to get the technology fixed so you can do everything from home at least as well as you can in the office.

Have back-channels

Make sure you are set up with ways to informally communicate with colleagues. Whether that’s Slack, HipChat, Google hangouts, or some other tool, make sure there is a way to discuss totally unimportant stuff to maintain your personal relationships with your colleagues. When times gets stressful it’s critical to have good will built up and important to have a way to work out issues in private.

Have an office

It doesn’t have to be a nice office, but have a space you go to for work. Make sure it’s setup well for your work, and make sure you don’t spend lots of non-work time in that space. Don’t use the room with your TV or other simple distractions. Ideally you want a space with a door you can close so you can block out any other people who are around and focus.

Have boundaries

Like a morning to routine you need to have boundaries about when you are not working. We all have times we have to work late or step into an emergency, but that shouldn’t be happening most days. Finish your day at a predictable time most days. Make sure your colleagues know when you are available outside normal hours and limit how much you let them cheat. Most importantly sometimes be truly unavailable.

Get out

Get out of your house/apartment at least once a day. When I first worked from home I realized that I’d gone a week without leaving my apartment complex and I was starting to go stir crazy which was bad for me and made less effective at work. Even if it’s just a trip to the store or a walk in the woods, get out and remember there is more to the world than your work.

Be Productive

Lots of work places treat work from home as a privilege they may want to take away again. Even if that doesn’t seem possible in your company, make sure you are proving you are able to be productive with the freedom working from home gives you. Ideally you have a work space free from the normal distractions of a conventional office, so use that environment to get more done than you can in the office.

Get together with colleagues

If you work remotely full-time you need to make sure you still spend face-to-face time with your colleagues. Humans are geared to appreciate time spent together, and for all the technology allows us new freedoms about where and when we work, there is still a different quality when everyone is in one place together. Some companies do this at conferences, some hold retreats, some just call the remote staff to the main office a few times a year. Figure out what makes sense for you and your company and push to make sure it happens.

Show personality

puzzleDo things that help give your colleagues insight into who you are outside work. I keep a puzzle table in my office to help me clear my head and avoid boredom during long conference calls. At a previous job we did a daily stand up video conference, and some days I would point the camera at the puzzle and worked on it as we talked. It served as a friendly way to help people see me as a real person not just a source of code. I also share pictures of my dogs and other things that round out people’s understanding of my life.

Not all of these things are totally within your control, and you will need support from your company to make sure the environment is right for you to be successful. Work with your manager, other remote employees, and other colleagues to make sure the environment is going to allow you to be successful over time.

Let them eat Drupal Cake

Last week Cyberwoven hosted the local SC DUG. To encourage people to come when we hold these events in Columbia I’ve blatantly started to bribe people with baked goods: I give the presenter a choice of Cookies, Cake, or Pie – everyone picks cookies (unless I swap in Brownies instead of cookies then they pick brownies), no one asks for pie or cake. This month Will gave a talk about Docker and asked for cake! Since I was excited to finally have someone actually ask for something interesting I decided to do something Drupal themed:  I made Drupal Cake.

Turns out if you go Googling for Drupal cake you get ideas for how to make a yellow or white cake and decorate it with a Drupal logo. There are some notable counter examples, but I wanted something with more Drupal baked in.

So I made a Chocolate Blue Velvet Cake with a Drupal logo in blueberries.

This recipe is derived from All Cake’s Considered’s Dark-Chocolate Red Velvet Cake (if you’re an NPR-nerd you should buy a copy but it is also available in its entirety from the Internet Archive – I assume legally).

Of course hers is red, and I wanted blue. But since my actual inspiration came from people providing recipes (mostly bad ones) for baby shower cakes getting suggestions about how much food coloring to use wasn’t hard. So a little marriage of ideas and you have Chocolate Blue Velvet Drupal Cake.

Ingredients

Cake:

  • 2 Sticks of unsalted butter at room temperature
  • 1 ¼ cups sugar
  • 1 ¼ cups brown sugar
  • 6 large eggs at room temperature
  • 2 teaspoons vanilla extract
  • 3 cups all-purpose flour
  • ½ teaspoon baking soda
  • ¼ cup Dutch process cocoa
  • ½ teaspoon baking powder
  • 1 cup sour cream
  • 1 ounce Blue icing coloring (my version used food coloring but didn’t have quite the color I wanted.  Icing coloring is usually stronger so this should get you closer).

Icing:

  • ½ cup (1 stick) unsalted butter at room temperature
  • 2 8-ounce packages cream cheese at room temperature
  • 32-ounces of confectioners’ (powdered) sugar.
  • 1 teaspoon vanilla extract
  • 1 package of fresh Blueberries

Instructions

  1. Preheat the oven to 325°F
  2. Cream the butter and then gradually add the sugars beating well as you go.
  3. Add the eggs one at a time, beating well between each.
  4. Add the vanilla extract and beat for another couple of minutes.
  5. In a separate bowl combine and lightly mix the dry ingredients.
  6. Alternate adding roughly ⅓ of the dry mixture followed by ⅓ of the sour cream, beating well after each addition, until all of both are fully incorporated.
  7. Add the coloring, and continue to beat well.  After a minute stop the mixer and using a spatula to get any unevenly dyed batter off the sides and bottom, and then beat until the color is even.
  8. Pour the batter into a pair of well greased 8 or 9-inch round cake pans, and place them in the oven so they have as similar of conditions as possible.
  9. Bake for 45 minutes or until the cake tests done.
  10. Cool the layers in their pans for 10 minutes and then remove from their pans carefully.  Let them cool to room temperature.

Frosting:

  1. Cream the butter and the cream cheese at medium speed.
  2. Gradually add the confectioners sugar.  Continue beating until the it is light and fluffy.

To Assemble the Cake:

  1. Wait until the cake is fully cooled to room temperature. You need one to be flat (to form the bottom layer) so if domes formed on both cakes, use a long knife to cut the top of one layer to be flat (you can do both if you want the top flat, but drops are round).
  2. Place about a ⅓ of the icing in a separate bowl, and working from this smaller amount (we’ll get back to the rest later – this is just to avoid getting crumbs in your icing) put a smooth later cross the top of the bottom layer.
  3. Place the top layer on the bottom layer, and ice the top of the cake, and then the sides. This first coat should be fairly thin.  Give it a few minutes to dry on the surface before proceeding with the second coat.
  4. Using the rest of the icing, apply a thick final coat of icing, particularly on top.
  5. Using the blueberries attempt to create a Drupal logo pattern on the top of the cake.
Top view of the cake.
I didn’t get a picture of the inside so here’s another view of the logo – yes that’s supposed to be the Drupal 8 logo. Happy 1st birthday to Drupal 8.

Postcards for children

Update (Oct 2017):

In the weeks and months that followed this posting, thanks those who read and shared it and our related message on Facebook, my sister-in-law received most than 500 postcards and letters from across the country welcoming her students, some sent gifts and donations. The messages came from people of all walks of life, including other immigrant children. Many of the messages were read to the entire class and she sent postcards home for their families to enjoy.  These simply gestures were a light in a very dark place for these new and often frightened Americans.

But there are still children, and adults, around the country that live in the same fear these kids do. If you know people in your community who immigrated to the U.S. please try to make an effort to be kind and welcoming to them. Show them the best that America can be, especially when the news shows so much of our worst.

Original Post (Nov 2016):

My sister-in-law is an ESOL teacher in Maryland. Yesterday she found herself trying comfort her class of terrified 6-8 year olds who wanted to know why America hates them. Many of her students are war refugees who have seen bombings, gunfire, and family members killed. Many of them are muslims, and even at young ages are aware of what has been said during the election about them and their families. Coming to America was supposed to mean safety and a government that would protect them, and now they fear that is gone.

Right now we can’t fix the Supreme Court, and we can’t undo all of the damage that has been done to our nation this year, but we can help these children.

We are trying to get as many people as we can from as many places as possible to send them postcards.

Tell them you’re happy they are in America.
Tell them you don’t hate Muslims.
Tell them that you love them.
Tell them whatever is in your heart.

Postcards can be sent to:

Ms. M’s Class
Templeton Elementary School
6001 Carters Lane

RiverdaleMD 20737