Responding to Drupal Break-ins

If you support any web site long enough you will suffer a break in. If you support lots of web sites you will suffer them more often than you’ll want to admit in public. A few weeks ago my number came up again in the attack lottery when we discovered a client’s web site was being used as a proxy and redirect to a fake shoe site.

It wasn’t the first time I’d suffered a break in, and unfortunately I don’t expect it to be the last. My last experience with a major break in was shortly after Drupalgeddon (I patched all the clients I was supporting before they were breached but had to clean up sites that weren’t patched by other vendors), and the attackers had learned a few new tricks in the meantime.

If you are responding to a break in on a Drupal site there are directions on drupal.org to help guide you through an attack response, but I thought it might be helpful to talk through a version of what response can look like in practice. I think it’s also useful for us all to admit our weaknesses from time to time to help us all make sure we’re making new mistakes.

Overview

At the outset I’m going to admit we never found the initial source of the attack, what we did find were the tools they placed after the break in. The most likely cause was poor server patching practices by the client’s host, but there were also some Drupal security patches that had been slow to be get installed as well. During the attack I worked with members of the Drupal Security team (particularly Greg Knaddison who generously provided feedback on this article as well – of course any remaining mistakes are mine), who were helpful in giving me suggestions and who were clearly interested in helping us make sure we resolved the problem.

The site was being used as part of a scam advertising network. The attacker was leveraging the reputation of the site to create records in search engine indexes that were redirects to a fake shoe sales site. There were also a number of tools placed on the server that gave them full access to the Drupal database and the ability to run arbitrary PHP scripts. And it was clear by the end they had placed additional backdoors we never found – they may have had full control over the OS as well.

How we found out

Google told us.

We got an alert from Google reporting SPAM content on the site. At first we couldn’t find the content they were talking about, which unfortunately slowed our escalating our response, because it was only directly available to search engines. The junior developer who was initially assigned to review the message from Google eventually figured out how to find the listing on Google (a Google site search for Nikes and some hash codes the attacker was using), but couldn’t figure out how out it got there, and escalated the task to me.

Once I saw what she’d figured out my stomach sank. At first I was still hoping there might be some other explanation, or some simple matter of a single user account getting exploited, but that seemed unlikely (since we couldn’t find the content on the server) and I quickly knew it was going to be a mess.

Initial Response

The first thing I did was make sure we had a copy of the exploited site: code, files, and database. I would have rolled the site back to a recent backup, but our five-day rolling database snapshots were not enough to get back to before the attack began. We spun up new virtual machines for myself and another senior developer to start reviewing copies in environments isolated from other work.

Since the URLs we had for testing were a fairly unique pattern we started to Google those – and we got lots of hits. As soon as we knew the problem was larger than our site, we opened an issue with the Drupal security team and started to feed them all the information we had gathered. While their practice is not to get involved in resolving attacks directly (their role is to ensure the security of Drupal core and contributed modules), they were supportive and helpful in suggesting places to look for problems and resolution strategies.

Attacks we found

By the time I was alerted to the problem there were already several malicious tools installed, some of which I’d seen versions of before, and some were new to me – all were designed to be hidden from sight through some simple but effective obfuscation. Over the course of the next couple of days I found several backdoors manually, wrote tools to help me find more, and played entirely too much whack-a-mole (more on that in a bit).

There were two main categories of attack I was chasing: PHP scripts scattered around the public files directory, and records added to Drupal’s database tables.

Database table exploits

If you dealt with sites in the aftermath of Drupalgeddon, or other hacked Drupal sites, you have probably seen what happens when an attacker inserts PHP into carefully targeted parts of a Drupal database. In the ones I’d seen before attackers replaced the callback functions in Drupal’s menu_router table with PHP of their own. In this case the attacker used the Block module’s ability to use PHP to place a block to provide themselves a way to execute arbitrary PHP by sending a post request to the server. They leveraged the fact that the main system block is always available and therefore is a reliable place to insert a backdoor. By posting a form with a specific form element they were able to execute arbitrary PHP and therefore use that to place additional malicious code.

The attacker also leveraged Drupal’s system table to get more complex attack code loaded. They created a record for a file to be loaded as a module and then uploaded that file to the site’s files directory where they were guaranteed Drupal had write access.

filename: sites/default/files/styles/medium/public/57h3d21.jpg
name:overly
type: module
owner:
status:1
bootstrap: 0
schema_version:0
weight: 0
info: a:11:{s:4:"name";s:6:"overly";s:11:"description";s:58:"Displays the Drupal administration interface in an overly.";s:7:"package";s:4:"Core";s:7:"version";s:4:"7.32";s:4:"core";s:3:"7.x";s:7:"project";s:6:"drupal";s:9:"datestamp";s:10:"1413387510";s:12:"dependencies";a:0:{}s:3:"php";s:5:"5.2.4";s:5:"files";a:0:{}s:9:"bootstrap";i:0;}

This was the script doing the redirects and filtering traffic so that pages only appeared to search engines. Usually these records have filenames that are .module, .php, or .inc files, but in this case it was a .jpg file named to be similarly to actual files on the site to make it hard to spot.

The content of that file was a PHP script not an image. The script did several things, and was the main tool the attacker was actively using during the time we were trying to stop them. It served as a simple proxy of content that they would present to the search engines, and redirect those same pages to the scam site for anyone else. It also provided code to make sure the content of user login forms was sent to the attacker, and a backup backdoor incase some of their others were lost.

We actually had to remove this particular attack more than once (always using the misspelled “overly” module) and each time it came back with a new file, and each time using a different but similar disguise to try to make their code blend in with legitimate files.

.htaccess files in public files

The other trick that was new to me (and a more aggressive stance by Drupal core on this approach is being discussed) was to take advantage of the .htaccess patterns in Apache to re-enable PHP execution within the public files directory. Drupal’s default .htaccess file disables PHP at the root of the public files directory and in theory all subdirectories, but that can simply be undone by a malicious .htaccess file (unless you block it in Apache’s main configuration – which in my opinion defeats the purpose of using .htaccess).

The attacker had placed a number of basic PHP-based exploits on the server using this technique to allow them to run the scripts. The tools themselves were not Drupal-specific, and likely the .htaccess file would work just as well on a number of other PHP-based CMS platforms.
Since the files directory gets deep and complicated there is no reasonable way to scan the whole thing by hand: particularly since several of the files were using inaccurate file extensions (like .jpg or no extension at all) and file names meant to blend into the background. So in addition to checking for any .htaccess files below the files directory root, I wrote a simple Python script to scan a directory for anything that includes the string <?php:

How we fixed it

We immediately made sure all code on the site was up-to-date, and I removed every exploit I could find. And for a couple days I played whack-a-mole with the attacker. Every day I would remove a series of exploits, disable their ability to redirect users to their scam, and every night they would break back in through a backdoor I’d failed to find.

The final solution was to replace the server, deploy a version of the code known to be good, and deploying copies of the database and files that have been scanned for any PHP in places it shouldn’t be – which involved a combination of the scanner above and hand checking every place the database stores PHP, not a fast process.

What we will do better next time

Part of any security event of this nature needs to be a full review of your internal processes and controls to make sure you reduce your risk and improve your response the next time something occurs (because unfortunately there will be a next time for all of us).

One of our first areas of improvement is a shared understanding that it’s more important to resolve the attack than determine the cause. This goes against the question that developers are constantly asked during and after an attack: “How did this happen?” While you need to know something about what happened, in the end it’s more important to make it stop. I still don’t know what happened that started the attack, I know I stopped it by blocking every attack vector I could think of and replacing every part of the stack with a version known to be fully up-to-date. It would have been faster and cheaper if I’d just started there: yes there is a risk I would have missed some of the code in the database if I hadn’t taken the time to review what I was finding, but frankly I doubt that risk is has high as the risk that new exploits will appear while I’m working to understand the previous one.

Beyond that basic shift in approach we developed a three part list of improvements:

  • Things we needed to improve right away.
  • Things we needed to improve soon.
  • Things that should be part of ongoing improvement.

The highest priority items were coming up with better internal process for initial response, and making sure we are deploying all security updates in a timely but still careful manner, including monitoring our hosting partners to ensure servers stay up-to-date as well. These are basics that are easy to let slip over time – particularly monitoring that your partners are doing their job correctly.

The second category of fixes is filled with workflow and procedure improvements. We were already were in the process of improving our code handling (migrating from SVN to Git, better production monitoring, more internal code review, etc), and we accelerated our plans to complete that work. This category also includes a complete review of our existing backup procedures to make sure they provide the level of coverage our clients need.

The final category of longer term adjustments includes tasks that include ensuring all developers are given (and expected to take) professional development opportunities around security best practices, doing more internal sharing about emerging ideas and trends, and encouraging more community engagement so we are better able to leverage the community resources in a crisis.

Yes I wear pants and other advice for working from home.

I’ve worked from home part-time or full-time for the past four years. The first question I’m always asked when people learn I work from home is “Do you wear pants?” The answer is yes, I wear pants to work even when no one can see me. And frankly I don’t quite understand the desire of large numbers of people to work without pants, but it led me to realize that it might be helpful to share a few tips for people starting to work remotely.

Wear Pants

Okay, it doesn’t have to be pants, but get up and get dressed in clothes you don’t mind being seen in. The clothes we pick communicate, not only to other people but also to ourselves. Pick clothes to help you take your work seriously.

Set a routine

Have a routine about when you start, and what you do to prepare yourself before you start for the day. Your commute might be a walk down the hall instead of a drive across town or a ride on public transit, but most people I know still benefit from a pre-work routine. Walk the dog, go for a run, eat breakfast, or some other basic activity. It doesn’t matter a whole lot what it is, but have a routine that gives you a few minutes to get settled into a frame of mind to be focused on what you are going to do at work.

Make sure the technology works

You never want to have to say “I can’t do that from here.” or “I’ll do that next time I’m in the office.” This is particularly true when you have a part-time remote arrangement (i.e. if you work from home 1 or 2 days a week), and the systems may not be setup to fully support remote workers. Push hard to get the technology fixed so you can do everything from home at least as well as you can in the office.

Have back-channels

Make sure you are set up with ways to informally communicate with colleagues. Whether that’s Slack, HipChat, Google hangouts, or some other tool, make sure there is a way to discuss totally unimportant stuff to maintain your personal relationships with your colleagues. When times gets stressful it’s critical to have good will built up and important to have a way to work out issues in private.

Have an office

It doesn’t have to be a nice office, but have a space you go to for work. Make sure it’s setup well for your work, and make sure you don’t spend lots of non-work time in that space. Don’t use the room with your TV or other simple distractions. Ideally you want a space with a door you can close so you can block out any other people who are around and focus.

Have boundaries

Like a morning to routine you need to have boundaries about when you are not working. We all have times we have to work late or step into an emergency, but that shouldn’t be happening most days. Finish your day at a predictable time most days. Make sure your colleagues know when you are available outside normal hours and limit how much you let them cheat. Most importantly sometimes be truly unavailable.

Get out

Get out of your house/apartment at least once a day. When I first worked from home I realized that I’d gone a week without leaving my apartment complex and I was starting to go stir crazy which was bad for me and made less effective at work. Even if it’s just a trip to the store or a walk in the woods, get out and remember there is more to the world than your work.

Be Productive

Lots of work places treat work from home as a privilege they may want to take away again. Even if that doesn’t seem possible in your company, make sure you are proving you are able to be productive with the freedom working from home gives you. Ideally you have a work space free from the normal distractions of a conventional office, so use that environment to get more done than you can in the office.

Get together with colleagues

If you work remotely full-time you need to make sure you still spend face-to-face time with your colleagues. Humans are geared to appreciate time spent together, and for all the technology allows us new freedoms about where and when we work, there is still a different quality when everyone is in one place together. Some companies do this at conferences, some hold retreats, some just call the remote staff to the main office a few times a year. Figure out what makes sense for you and your company and push to make sure it happens.

Show personality

puzzleDo things that help give your colleagues insight into who you are outside work. I keep a puzzle table in my office to help me clear my head and avoid boredom during long conference calls. At a previous job we did a daily stand up video conference, and some days I would point the camera at the puzzle and worked on it as we talked. It served as a friendly way to help people see me as a real person not just a source of code. I also share pictures of my dogs and other things that round out people’s understanding of my life.

Not all of these things are totally within your control, and you will need support from your company to make sure the environment is right for you to be successful. Work with your manager, other remote employees, and other colleagues to make sure the environment is going to allow you to be successful over time.

Let them eat Drupal Cake

Last week Cyberwoven hosted the local SC DUG. To encourage people to come when we hold these events in Columbia I’ve blatantly started to bribe people with baked goods: I give the presenter a choice of Cookies, Cake, or Pie – everyone picks cookies (unless I swap in Brownies instead of cookies then they pick brownies), no one asks for pie or cake. This month Will gave a talk about Docker and asked for cake! Since I was excited to finally have someone actually ask for something interesting I decided to do something Drupal themed:  I made Drupal Cake.

Turns out if you go Googling for Drupal cake you get ideas for how to make a yellow or white cake and decorate it with a Drupal logo. There are some notable counter examples, but I wanted something with more Drupal baked in.

So I made a Chocolate Blue Velvet Cake with a Drupal logo in blueberries.

This recipe is derived from All Cake’s Considered’s Dark-Chocolate Red Velvet Cake (if you’re an NPR-nerd you should buy a copy but it is also available in its entirety from the Internet Archive – I assume legally).

Of course hers is red, and I wanted blue. But since my actual inspiration came from people providing recipes (mostly bad ones) for baby shower cakes getting suggestions about how much food coloring to use wasn’t hard. So a little marriage of ideas and you have Chocolate Blue Velvet Drupal Cake.

Ingredients

Cake:

  • 2 Sticks of unsalted butter at room temperature
  • 1 ¼ cups sugar
  • 1 ¼ cups brown sugar
  • 6 large eggs at room temperature
  • 2 teaspoons vanilla extract
  • 3 cups all-purpose flour
  • ½ teaspoon baking soda
  • ¼ cup Dutch process cocoa
  • ½ teaspoon baking powder
  • 1 cup sour cream
  • 1 ounce Blue icing coloring (my version used food coloring but didn’t have quite the color I wanted.  Icing coloring is usually stronger so this should get you closer).

Icing:

  • ½ cup (1 stick) unsalted butter at room temperature
  • 2 8-ounce packages cream cheese at room temperature
  • 32-ounces of confectioners’ (powdered) sugar.
  • 1 teaspoon vanilla extract
  • 1 package of fresh Blueberries

Instructions

  1. Preheat the oven to 325°F
  2. Cream the butter and then gradually add the sugars beating well as you go.
  3. Add the eggs one at a time, beating well between each.
  4. Add the vanilla extract and beat for another couple of minutes.
  5. In a separate bowl combine and lightly mix the dry ingredients.
  6. Alternate adding roughly ⅓ of the dry mixture followed by ⅓ of the sour cream, beating well after each addition, until all of both are fully incorporated.
  7. Add the coloring, and continue to beat well.  After a minute stop the mixer and using a spatula to get any unevenly dyed batter off the sides and bottom, and then beat until the color is even.
  8. Pour the batter into a pair of well greased 8 or 9-inch round cake pans, and place them in the oven so they have as similar of conditions as possible.
  9. Bake for 45 minutes or until the cake tests done.
  10. Cool the layers in their pans for 10 minutes and then remove from their pans carefully.  Let them cool to room temperature.

Frosting:

  1. Cream the butter and the cream cheese at medium speed.
  2. Gradually add the confectioners sugar.  Continue beating until the it is light and fluffy.

To Assemble the Cake:

  1. Wait until the cake is fully cooled to room temperature. You need one to be flat (to form the bottom layer) so if domes formed on both cakes, use a long knife to cut the top of one layer to be flat (you can do both if you want the top flat, but drops are round).
  2. Place about a ⅓ of the icing in a separate bowl, and working from this smaller amount (we’ll get back to the rest later – this is just to avoid getting crumbs in your icing) put a smooth later cross the top of the bottom layer.
  3. Place the top layer on the bottom layer, and ice the top of the cake, and then the sides. This first coat should be fairly thin.  Give it a few minutes to dry on the surface before proceeding with the second coat.
  4. Using the rest of the icing, apply a thick final coat of icing, particularly on top.
  5. Using the blueberries attempt to create a Drupal logo pattern on the top of the cake.
Top view of the cake.
I didn’t get a picture of the inside so here’s another view of the logo – yes that’s supposed to be the Drupal 8 logo. Happy 1st birthday to Drupal 8.

Postcards for children

Update (Oct 2017):

In the weeks and months that followed this posting, thanks those who read and shared it and our related message on Facebook, my sister-in-law received most than 500 postcards and letters from across the country welcoming her students, some sent gifts and donations. The messages came from people of all walks of life, including other immigrant children. Many of the messages were read to the entire class and she sent postcards home for their families to enjoy.  These simply gestures were a light in a very dark place for these new and often frightened Americans.

But there are still children, and adults, around the country that live in the same fear these kids do. If you know people in your community who immigrated to the U.S. please try to make an effort to be kind and welcoming to them. Show them the best that America can be, especially when the news shows so much of our worst.

Original Post (Nov 2016):

My sister-in-law is an ESOL teacher in Maryland. Yesterday she found herself trying comfort her class of terrified 6-8 year olds who wanted to know why America hates them. Many of her students are war refugees who have seen bombings, gunfire, and family members killed. Many of them are muslims, and even at young ages are aware of what has been said during the election about them and their families. Coming to America was supposed to mean safety and a government that would protect them, and now they fear that is gone.

Right now we can’t fix the Supreme Court, and we can’t undo all of the damage that has been done to our nation this year, but we can help these children.

We are trying to get as many people as we can from as many places as possible to send them postcards.

Tell them you’re happy they are in America.
Tell them you don’t hate Muslims.
Tell them that you love them.
Tell them whatever is in your heart.

Postcards can be sent to:

Ms. M’s Class
Templeton Elementary School
6001 Carters Lane

Riverdale, MD 20737

Lessons learned from my first Drupal 8 projects

During last month’s SCDUG I gave two presentations. I’ve already posted the Sins Against Drupal talk I gave. That didn’t take up much time, and since I had been prepping some thoughts on the first few Drupal 8 projects I gave a second short talk on what I’ve learned working with Drupal 8 during its first year.


What’s not new?

Drupal 8 brings forward lots of concepts from Drupal 7 (and prior versions). It also brings forward a few standard community realities.

The information architecture tools are basically the same at least at the conceptual level. Nodes, Fields, Taxonomy, and Menus all still exist more or less as we’ve known them for years, and while there are differences on the surface those differences are incremental in nature and scope.

We also still have the constant hunt for modules that do what you’re doing so you don’t reinvent the wheel. Many modules that people are used to using are still missing, but new things arrive daily, and many of the new versions are significant improvements over previous generations of tools.

And there is still a lack of a clear line between front-end and back-end. When does business logic end and interface begin? When does a themer need to understand Drupal’s HTML generation vs when does a backend developer need to figure out how to force Drupal to generate carefully crafted markup?  There are opportunities to form better and clearer lines, but they aren’t automatic by any means: every team will have to solve this problem their own way.

What’s so great?

Drupal 8 opens up a collection of new tools and opportunities for the community.

  • As a backend developer you get to write modern code. The name spacing can feel a little Java-esc at times, but the ability to properly name space code, ditch globals, move from hooks to event listeners, and other basic OOP tools is incredibly nice.
  • With CKEditor in core we get better integration between that interface and the rest of Drupal. And better modules are coming out all the time to solve long standing UX annoyances. For example with the D8 Editor File upload module files and images can both be handled as Drupal file objects, but the editor can know the difference between a file (which should just be a link) and an image (which you should display).
  • The Symfony community provides a large number of packages that provide 3rd party integrations, or the tools to make them easy to build.
  • The two core base themes do not require keelhauling to make viable. If you commonly built your themes from scratch, the ability to have clean default markup that’s easy to override makes both Stable and Classy a major improvement to life.

Things you need to survive

As you dive into your first projects you need to understand that much of what you know at the detail level has changed.  And so you’ll need to learn a few new tricks and be willing to toss aside a few old ideas:

Your old build standards are wrong. Probably not entirely, but you’ll need new modules in your default builds, new best practices about when to use a node vs block vs custom entities, and other basic details you probably have really well develop standards (or at least habits) from years of working with Drupal.

If you’ve been avoiding it to-date, it’s time to develop an understanding about composer, drush, and Drupal console. Not only do you need to be using these tools (probably all three) but you need to understand what each provides and which tool is best for which job.

Nginx is not for the faint of heart: as best I can find, no one has published a complete setup guide yet. In Drupal 6 and 7 there were pretty good guides to setting up Nginx properly, but with Drupal 8 there are enough differences that those guides don’t really work. And all the guides that I have read so far include errors (some of them significant security mistakes). To be clear, it can be done, but if you want to do it yourself be prepared to do a great deal of extra research along the way.

If you have windows you have pain. There are a number of windows specific challenges (speed and NTFS weakness being the biggest we face), and there is little community support to help you overcome the challenges.

Core patches tend to be required. Unlike Drupal 7, core and module patching is the norm not the exception. There are several issues that are frustratingly slow to get fixed that hit some fairly common use cases (like this menu block bug). While this is improving all the time, we haven’t launched a D8 site without at least one patch in place.

Cache tags are great, but require learning. The new caching system is powerful, flexible, and totally different from what we had before.  It’s better, but to use it well you’ll need to spend some time getting to know when to set and clear the tags you want to create.

Twig is great, but requires learning and discipline. I really like twig and the much cleaner syntax to brings to the theme layer.  However, as more and more people use it they are finding ways to move increasingly complex logic out of modules and theme PHP into template files. Please fight this urge! Keep your business logic separate from your display logic. If some object wasn’t loaded into a variable in your twig file do not attempt to load it in twig. If you need some 4 or 5 layer selector to get to the value you want to print: fix that in a preprocess function.

The API improvements are coming all the time and make things interesting. So far the community has stayed on schedule of rolling out new minor versions of 8 and that’s meant great new features in each version. It has also meant that sometimes a solution you built is not using the best techniques given the improvements.  That’s not really a problem, but can add headaches in maintenance cycles. Also you will find places where brand new D8 tools are already deprecated but the replacements don’t have good example implementations yet.

XML to JSON in Five lines

In closing I want to share the feature that caused me to realize that Symfony was worth the cost of admission: a 5 line XML to JSON AJAX callback.

My first site had a location finder that links to a data provider partner locations nationwide. That provider has an XML-base API that cannot be accessed directly from browsers, and therefore needed to be proxied through the main site. The full details involve more than just the following lines of code (providing input checking, settings, routing, etc), but the heart of the process is just five lines.

    $client = new Client(); // 1: Create Guzzle Client
    try{
      $res = $client-&amp;gt;get($config-&amp;gt;get('api_endpoint'), // 2: Make request
            [
              'http_errors'=&amp;gt;false,
              'query' =&amp;gt;; [
                 'key'=&amp;gt; $config-&amp;gt;get('coop_api_key'),
                     // ...
               ],
            ]);
      $decoderRing = new XmlEncoder(); // 3: Create XML Encoder
      $xml = $decoderRing-&amp;gt;decode($res-&amp;gt;getBody()); // 4: Decode the XML Response into an array.
      return new JsonResponse($xml); // 5: Return the array as JSON
    } catch (RequestException $e) {
      throw new HttpException($this-&amp;gt;t('Unable to process request'));
    }

Sins Against Drupal 3

This is part of my ongoing series about ways Drupal can be badly misused. These are generally times someone tried to solve an otherwise interesting problem in just about the worst possible way. All of these will start with a description of the problem, how not to solve it, and then ideas about how to solve it well.

I present these at SC Drupal Users Group meetings from time to time as an entertaining way to discuss ways we can all improve our skills.

This one was presenting during our October event here in Aiken, SC.


The Problem

Provide a custom authentication solution that allows staff to have one backend and members another.

The Sinful Solution

In order to force staff to use the staff login page, during login form validation check to see if the user is a staff member, by authenticating the user, checking their groups, and logging out staff.

The Code

/**
* Prevents staff members from logging in outside of staff login page. &amp;amp;amp;lt;&amp;amp;amp;lt;-- Why?
*/
function my_auth_staff_boot($form, &amp;amp;amp;amp;$form_state) { // NOT actually a hook_boot (thankfully) called as login form validator...
  user_authenticate($form_state['values']);
  global $user;
  if (in_array('An Employee', $user-&amp;amp;amp;gt;roles)) {
    form_set_error($form['#id'], l(t('Staff must log in via staff-login', 'staff-login')), TRUE);
    drupal_set_message('Staff must log in via ' . l(t('staff-login', 'staff-login')), 'error', TRUE);
    // Load the user pages in case they have not been loaded.
    module_load_include('inc', 'user', 'user.pages');
    user_logout();
  }
}



Why is this so bad?


This code actually completes the login process before kicking the user out. Why would you ever want to do that to your users? What did they do to you? It also loads an extra file for no apparent reason just before kicking the user back out.


Better Solutions


The goal here is to control what backend the user logs into, and shouldn’t control the page they login from. So the place to look for solutions are modules that already do this and so I propose mimicking the LDAP or GAuth modules’ approaches. LDAP attaches a validator to the form and takes over authentication, but LDAP supports lots of options so the code there is too extensive to use for a clear example. So for discussion I pulled out elements of the GAuth module (although there is still lots of trimming to make this understandable).


The GAuth module adds a submit button to the form and handles all processing for that form directly.


/**
* Implements hook_form_alter().
*/
function gauth_login_form_alter(&amp;amp;amp;amp;$form, &amp;amp;amp;amp;$form_state, $form_id) {
  if ($form_id == 'user_login' || $form_id == 'user_login_block') {
    $form['submit_google'] = array(
      '#type' =&amp;amp;gt; 'submit',
      '#value' =&amp;amp;gt; t(''),
      '#submit' =&amp;amp;gt; array('gauth_login_user_login_submit'),
      '#limit_validation_errors' =&amp;amp;gt; array(),
      '#weight' =&amp;amp;gt; 1000,
    );
    drupal_add_css(drupal_get_path('module', 'gauth_login') . '/gauth_login.css');
  }
}

/**
* Login using google, submit handler
*/
function gauth_login_user_login_submit() {
  if (variable_get('gauth_login_client_id', FALSE)) {
// .. skipping resource validation ...

  $client = new Google_Client();
// .. skipping client setup ...
  $url = $client-&amp;amp;gt;createAuthUrl();
  // Send the user off to Google for processing
  drupal_goto($url);
  }
  // ... skip errors
}



From there we pass through a menu router from the main module, and an API hook to get:


function gauth_login_gauth_google_response() {
  if (isset($_GET['state'])) {
// Skipping some error traps...
    $redirect_url = isset($state['destination']) ? $state['destination'] : '';
    if (isset($_GET['code'])) {
// Skipping a bunch of Client setup...
      $oauth = new Google_Service_Oauth2($client);
      $info = $oauth-&amp;amp;amp;gt;userinfo-&amp;amp;amp;gt;get();
      if ($uid = gauth_login_load_google_id($info['id'])) {
        $form_state['uid'] = $uid;
        user_login_submit(array(), $form_state); // &amp;amp;lt;&amp;amp;lt; That right there with the $form_state['uid'] set does the magic.
      }
      else {
// Skipping other options....
      }
    }
    drupal_goto($redirect_url); // &amp;amp;amp;lt;&amp;amp;amp;lt; be nice and handle the destination parameter
  }
}



Share your sins


I’m always looking for new material to include in this series. If you would like to submit a problem with a terrible solution, please remove any personally identifying information about the developer or where the code is running (the goal is not to embarrass individuals), post them as a gist (or a similar public code sharing tool), and leave me a comment here about the problem with a link to the code. I’ll do my best to come up with a reasonable solution and share it with SC DUG and then here. I’m presenting next month so if you have something we want me to look at you should share it soon.


If there are security issues in the code you want to share, please report those to the site owner before you tell anyone else so they can fix it. And please make sure no one could get from the code back to the site in case they ignore your advice.

Documenting your work

Programming books
Your documentation does not need to look like this.

Early in my career I spent a lot of time as the only technical person on project, and therefore believed that I didn’t need to document my work carefully since I was the only person who had to understand it later. It turned out that if a project was back burnered for a few months the details were pushed out of my mind by the details of eight other projects.

Any project that takes more than a couple hours to complete involves too many details for most people to remember for more than a few days. We often think about project documentation as something for other people – and it is – but that other person may be you in six months.

I learned to start keeping notes that I could go back to, those notes would turn into documentation that I could share with other people as the need developed. My solutions were typically ad-hoc: freeform word documents or wiki pages. For a while I had a boss who wanted every piece of documentation created by IT to fit a very predictable format and to be in a very specific system. It took two years for him to settle on the system, process, and format to use. By then I had a mountain of information in wiki pages that documented the organization’s online tools in detail, and no one else is IT had anything substantial. It was two more years before the documentation of other team members got to be as good as my ad-hoc wiki.

That’s not to say a rogue solution is best, but the solution that I used was better than his proposed setup for at least three years. That experience got me to think about what makes documentation useful.

Rules of thumb for good project documentation:

  • Write up the notes you’d want from others when coming into a project: think of this as the Golden Rule of documentation. Think about what you’d want to have if you were coming into the project six months from now. You’d want an outline of the purpose of the project and the solution used, and places they deviated from any standards your team normally uses. You’ve probably read documents that are explaining something technical to an expert that are hard for anyone else to understand – if I’m reading the documentation I want to become an expert, but I’m probably not one already.
  • Keep it easy to create and edit while working: if you have to stop what you’re doing and write your notes in a totally different environment that your day-to-day work you will not do it. Wikis, markdown files, and other similar informal solutions are more likely to actually get written and updated than any formal setup that you can’t update while doing your main work.
  • Document as you go: we all plan to go back and write documentation later and almost none of us do. When we do get back to it, we’ve forgotten half the details we need to make the notes useful to others. So admit you’re not going to get back to it and don’t plan to: write as you go and edit as you need.
  • Make sure you can come in in the middle: People skim project documentation, technical specifications, and any other large block of text. Make sure if someone has skipped the previous three sections they can either pick up where they left off, or give them directions to the parts the need to understand before continuing.
  • Track all contributions: Use a system that automatically tracks changes so you you can see contributions from others and fix mistakes. Tools like MediaWiki, WordPress, and Drupal do this internally. Markdown or text files in a code repository also have this trait. Avoid solutions like MS Word’s track changes that are meant for editing a final document not tracking revisions over time.
  • Be boldDon’t fear editing: follow the Wikipedia community’s encouragement to Be Bold. You should not fear making changes to the team’s documentation. You will be wrong in some of what you write, and you should fix any mistake you find – yours or someone else’s. Don’t get mad if someone makes a change that’s not quite right, revert the change or make a new edit and more forward.
  • There is always an audience: even if you are the only person on the project you have an audience of at least your future-self. Even if it feels like a waste in the moment having documentation will help down the road.

Remember even if you are working alone you’re on a team that includes at least yourself today and yourself in the future. That future version of you probably won’t remember everything you know right now, and will get very annoyed at you if you don’t record what they need to know. And if the rest of your team members aren’t just versions of yourself they may expressed their frustration more directly.

Why I think pushing only STEM is a problem

Over the course of this week I needed to be conversant in PHP, MySQL, Linux, Drupal, search tools, herbal remedies, online education tools and strategy, women’s health, men’s health, canine health, dental health, health insurance, public media outlet fundraising, online advertising, email, open source technology, open source movements, cyber security, physical security, national security, state and national politics, disaster response planning, disaster response fundraising, coffee, vegetarian ethics, economics, retirement planning, videography and photography, child rearing, fiber arts, baking, and a few other topics.

Last week I talked a bit about the things I got from my liberal arts education that made me a better developer. There I focused mostly on the positive side of the argument: why non-stem course work is important. Now I want to flip the coin and talk about why our constant push to have more people graduating with STEM degrees is bad for the country and the economy.

No, I didn’t learn about all the topics in the opening in college – in fact much of the information I gathered in college on many of those topics (particularly things related to technology) is now at least partially out of date. But I spent four years learning how to learn and developed a base of skills in several disciplines that makes it easier for me to adapt to change and new areas of interest.

In 2016 the path to most high paying jobs runs through college. And the majority of people who go to college earn more than they would have if they hadn’t gone to college. But if we talk about college as being all about our work and pay we miss the main point of a good education and we ignore the fact that our education system should help mold engaged citizens not just good workers.

The push to graduate students to be “workplace ready” in engineering and technology (people calling for STEM education rarely care about science that can’t be applied to engineering new technologies and pretty much never mention math) is short-term thinking. That kind of educational model frees companies from having to train new employees while weakening their long-term workforce and our democracy. Our education system, high school and college in particular, needs to be helping kids become productive members of a society none of us can fully predict. To thrive in current and – more importantly – future America we all have to adapt to constant technical and social change. You cannot teach kids in school how to actually handle the situations that don’t exist yet, so we need to be teaching them how to learn so they develop skills as they need them.

Companies go looking for the employees of tomorrow, but not of next year, or of five years from now. They complain that younger workers don’t want to put in time at low-level jobs while refusing to provide training to help those new workers advance. What we get all too often are people who are good-enough to do the work of today and tomorrow, but never get better.

When I first wrote code for work, I was terrible. I got better because I was allowed to fail and learn from my mistakes. If you’d judged my college program on that first project, you’d call Hamilton a fraud. If you judge them by my ability to adapt, change, and improve, you’d rank Hamilton a top school as most rankings do.

College should be part of an education meant to prepare people for a life that will be unpredictable and varied. It should part of an education built around showing kids as much diversity of thought as we can:

  • Kids need art courses because art helps us all see the world fully and because developers need artists on our teams to make our projects successful.
  • Kids need to learn history because history matters – when our society repeats mistakes because we don’t understand our own past we all lose.
  • Kids need to learn math and science because we need them to understand how the universe works and that facts are not something that change when it is politically convenient.
  • Kids need to learn to challenge ideas they are taught because even the best teachers will be wrong sometimes.
  • And kids need to play because it’s good for their bodies, minds, and souls.

I work for a company that develops software every day of the week. There are ten of us whose primary job is to write code, so we need great developers. But there are 30 people at the company which leaves twenty of us who do things other than write code – we need great designers, writers, account managers, and more. Yes, they need to understand the internet and the tools we use at some level, but more importantly they need to understand how do their part in helping our clients be successful so that our company can succeed.

And at the end of the day all thirty of us live in a democracy, and in November all of us should be casting informed votes up and down the ballot for people we think will do the best job. The artists will need to understand the tax policies, and the developers will need to understand the impact of social program proposals. We don’t have to agree, but we all need to be prepared.

If you want a deeper argument about the importance to the world of people with a broader understanding of many topics check out Former Bennington College President Liz Coleman’s Ted Talk.

What I learned by getting a degree from a liberals arts college

I have a degree in Computer Science, with a minor in Economics. But I earned that degree at Hamilton College, a traditional liberal arts college. That meant I was forced (now students there are just encouraged) to take classes in a variety of disciplines. I went to Hamilton in part because they offered degrees in Computer Science and History, and I was interested in both fields. In addition to the courses required by my major I took classes in history, religious studies, philosophy, art, and more. And I learned critical skills for work and life because I took those classes.

Two hand made mugs
Making these mugs made me a better developer.

Ceramics and pottery taught me to be a craftsman, accept critical feedback, and admit failure. For all the courses I took in college, the time I spent in Ceramics and Pottery studio taught me more about how to be good at what I do than any other course work (including CS). Basic pottery is a craft that doesn’t allow you to save a piece once you’ve made mistakes. If you are making cylinders the walls aren’t straight throw it away and start again. My professor in those courses set very high standards and pushed us to meet them through brutally honest, but helpful, critiques. He taught me to appreciate honest feedback, and to be skeptical of my work before showing it others. In those courses we had to do good work, recognize our weaknesses, and hit deadlines.

History taught me to care about communicating. Between the courses I took, and dating (and eventually marrying) a history major (now professor) I came to understand how important it is to communicate well in speech and in writing. I started this blog in part because I wanted to make sure I was spending more time writing to help maintain those skills. I frequently find myself in meetings having to explain highly technical issues to non-technical clients and colleagues. Knowing how to adjust to my audience, without insulting their intelligence, is a critical part of my day to day work.

Religious Studies and Philosophy taught me to make a well reasoned argument and express the ethics of my position. Religious Studies and Philosophy both require you to make arguments based primarily on the strengths of your ideas – you can’t research your way out of a bad concept. If you cannot assemble a well reasoned argument you leave yourself open to easy counter attacks. When you work at a nonprofit you often are building your ideas and your explanations from a mix of the facts of the issue, your worldview, and the assumptions you are forced to make because no one has all the information they really need.

Economics taught me how to view the world through the lens of money and trade. While I don’t think it’s the only, or even best, way to view the world it is a very important worldview that dominates the news and political spaces. Understanding the strengths and weaknesses of economics as a field helps me think about plans put forward by various politicians. It also helps me think about how the companies I work for function, and helps make sure internal systems and plans I propose have a strong business case to support them (and because I learned to throw things away in ceramics I also know that when the case is bad to stop and try something different).

The most important skill I have is the ability to learn new skills. Since I graduated from college the technology I work with every day has changed several times. I’ve had to learn tools, techniques, and strategies that didn’t exist when I was in college. And my career has evolved through multiple employers, disciplines, and areas of expertise.

Bad data systems do not justify sexist your behavior

This week we get a letter from Atlantic Broadband, our ISP, addressed to “Aaron & Eliza Crosman Geor”. My wife has never gone by Eliza and her last name is not “Geor”.

Picture of an envelop from Atlantic Broadband to: Aaron & Eliza Crosman GeorIt’s been this way since we signed up with them, when we ask them to fix it they acknowledge that they cannot because their database cannot correctly handle couples with different last names who both want to appear on the account. Apparently it is the position of Atlantic Broadband that in 2016 it is reasonable to tell a woman she cannot be addressed by her legal name because it would be expensive for them to fix their database, and therefore she must be misaddressed or left out entirely.

I consider this unacceptable from old companies, but Atlantic BB was founded in 2004 – there are probably articles about not making assumptions about people’s names that are older than their company.

Folks, it is 2016, when companies insult people and then blame their databases it is because they do not consider all their customers worthy of equal respect.

So let’s get a few basics out of the way:

  • Software reflects the biases of the people who write it and buy it.
  • If your database tells someone their name is invalid your database is not neutral. Just because you don’t get the push-back that Facebook sees when they mess this up does not mean what you’re doing is okay.
  • If your database assumes my household follows 1950s social norms, the company that uses it considers 1950s social norms acceptable in 2016 – and there are probably a few of those they don’t want to defend (I hope).
  • When an email, phone rep, or letter calls me by my wife’s last name or her by mine, in both cases they are assuming she has my last name not that I have hers. This is a sexist assumption that the company has chosen to allow.

Of course Atlantic isn’t the only company that does this: Verizon calls me Elizabeth in email a couple times a week because she must be primary on that account (one person must lead the family plan), and Nationwide Insurance had to hack their data fields for years so my wife could appear on our car insurance card (as required by law) every time we moved because their web interface no longer allowed the needed changes. The same bad design assumptions can be insulting for other reasons such as ethnic discrimination. My grandmother was mis-addressed by just about everyone until she died because in the 1960s the Social Security Administration could not handle having an ‘ in her name, and no one was willing to fix it in the 50 years that followed SSA’s uninvited edit to her (and many other people’s) name.

In all these cases representatives all say something to the effect of “our computers cannot handle it.” And that of course is simply not true. Your systems may not be setup to handle real people, but that’s because you don’t believe they should be.

Let’s check Atlantic Broadband’s beliefs about their customers based on how they address us (I’m sure there are some additional assumptions not reflected here but these are the ones they managed to encode in one line in this letter):

  • They assume they are addressing one primary account holder: I happen to know from my interactions with them that they list my first name as: “Aaron & Eliza”, and my last name as “Crosman Geor”. Plenty of households have more than one, or even two, adults who expect equal treatment in their home. Our bank and mortgage company know we are both responsible adults why is this so hard for an ISP (or insurance company, or cell provider, credit card, etc)?
  • They assume my first name isn’t very long: They allowed 13 characters, but 4 more is too many. I went to high school with a kid who broke their database by exceeding the 26 character limit it had (they didn’t ask the kid to change his name, the school database admin fixed the database), but Atlantic can barely handle half that.
  • They assume my last name isn’t very long: Only 12 characters were used and they stopped in a strange place. I know many people with last name longer than that: frequently people who have hyphenated last names blow past 12. Also the kid with a 26 character first name – his surname was longer.
  • They assume my middle name isn’t an important part of my name: If they had a middle name field, they could squeeze a few more letters in and make this read more sensibly.  But they only consider first and last names important. Plenty of people have three names – or more – they like to have included on letters.
  • They assume it is okay to mis-address me and my wife: The name listed is just plain wrong, but they believe it’s okay to keep using this greeting. They assume this even after they have been told it’s not, and even after we’ve reduced service with them (if another ISP provided service to my house I’d probably cut it entirely although mostly for other reasons). They believe misaddressed advertisements will convince me I need a landline or cable package again.

Now I’ll be fair for just a minute and note something they got right: they allow & and spaces in a name so Little Bobby Tables might be able to be a customer without causing a crisis (partially because his name is too long for them to fit a valid SQL command into the field).

Frequently you’ll hear customers blame themselves because their names are too long or they have done something outside the “norm”. Let’s be clear: this is the fault of the people who write and buy the software. Software development is entirely too dominated by men, as is the leadership of large companies. When a company lacks diversity in key roles you see that reflected in the systems built to support the work. Atlantic’s leadership’s priorities and views are reflected in how their customers are addressed because they did not demand the developers correct their sexist assumptions.

These problems are too common for us to be able to refuse to do business when it comes up. I will say that when we switched our insurance to State Farm they did not have any trouble understanding that we had different last names and their systems accommodated that by default.

If you do business with a company that makes these (or other similar mistakes) I think it’s totally reasonable to remind them every time you reasonable can that it’s offensive. Explain that they company is denying you, your loved ones, and/or your friends a major marker of their identity. Remind them they are not neutral.

If you write data systems for a living: check the assumptions you’re building into your code. Don’t blame the technology because you used the wrong character set or trimmed the field too short: disk is cheap, UTF-8 has been standard for 15+ years, and processors are fast. If the database or report layout doesn’t work because someone’s name is too long the flaw is not the name.

We all make mistakes and bad assumptions sometimes, but that does not make it okay to deny people basic respect. When we make a bad assumption, that’s a bug, and good developers are obligated to fix it. Good companies are obligated to prevent it from happening in the first place.