Responding to Drupal Break-ins

If you support any web site long enough you will suffer a break in. If you support lots of web sites you will suffer them more often than you’ll want to admit in public. A few weeks ago my number came up again in the attack lottery when we discovered a client’s web site was being used as a proxy and redirect to a fake shoe site.

It wasn’t the first time I’d suffered a break in, and unfortunately I don’t expect it to be the last. My last experience with a major break in was shortly after Drupalgeddon (I patched all the clients I was supporting before they were breached but had to clean up sites that weren’t patched by other vendors), and the attackers had learned a few new tricks in the meantime.

If you are responding to a break in on a Drupal site there are directions on drupal.org to help guide you through an attack response, but I thought it might be helpful to talk through a version of what response can look like in practice. I think it’s also useful for us all to admit our weaknesses from time to time to help us all make sure we’re making new mistakes.

Overview

At the outset I’m going to admit we never found the initial source of the attack, what we did find were the tools they placed after the break in. The most likely cause was poor server patching practices by the client’s host, but there were also some Drupal security patches that had been slow to be get installed as well. During the attack I worked with members of the Drupal Security team (particularly Greg Knaddison who generously provided feedback on this article as well – of course any remaining mistakes are mine), who were helpful in giving me suggestions and who were clearly interested in helping us make sure we resolved the problem.

The site was being used as part of a scam advertising network. The attacker was leveraging the reputation of the site to create records in search engine indexes that were redirects to a fake shoe sales site. There were also a number of tools placed on the server that gave them full access to the Drupal database and the ability to run arbitrary PHP scripts. And it was clear by the end they had placed additional backdoors we never found – they may have had full control over the OS as well.

How we found out

Google told us.

We got an alert from Google reporting SPAM content on the site. At first we couldn’t find the content they were talking about, which unfortunately slowed our escalating our response, because it was only directly available to search engines. The junior developer who was initially assigned to review the message from Google eventually figured out how to find the listing on Google (a Google site search for Nikes and some hash codes the attacker was using), but couldn’t figure out how out it got there, and escalated the task to me.

Once I saw what she’d figured out my stomach sank. At first I was still hoping there might be some other explanation, or some simple matter of a single user account getting exploited, but that seemed unlikely (since we couldn’t find the content on the server) and I quickly knew it was going to be a mess.

Initial Response

The first thing I did was make sure we had a copy of the exploited site: code, files, and database. I would have rolled the site back to a recent backup, but our five-day rolling database snapshots were not enough to get back to before the attack began. We spun up new virtual machines for myself and another senior developer to start reviewing copies in environments isolated from other work.

Since the URLs we had for testing were a fairly unique pattern we started to Google those – and we got lots of hits. As soon as we knew the problem was larger than our site, we opened an issue with the Drupal security team and started to feed them all the information we had gathered. While their practice is not to get involved in resolving attacks directly (their role is to ensure the security of Drupal core and contributed modules), they were supportive and helpful in suggesting places to look for problems and resolution strategies.

Attacks we found

By the time I was alerted to the problem there were already several malicious tools installed, some of which I’d seen versions of before, and some were new to me – all were designed to be hidden from sight through some simple but effective obfuscation. Over the course of the next couple of days I found several backdoors manually, wrote tools to help me find more, and played entirely too much whack-a-mole (more on that in a bit).

There were two main categories of attack I was chasing: PHP scripts scattered around the public files directory, and records added to Drupal’s database tables.

Database table exploits

If you dealt with sites in the aftermath of Drupalgeddon, or other hacked Drupal sites, you have probably seen what happens when an attacker inserts PHP into carefully targeted parts of a Drupal database. In the ones I’d seen before attackers replaced the callback functions in Drupal’s menu_router table with PHP of their own. In this case the attacker used the Block module’s ability to use PHP to place a block to provide themselves a way to execute arbitrary PHP by sending a post request to the server. They leveraged the fact that the main system block is always available and therefore is a reliable place to insert a backdoor. By posting a form with a specific form element they were able to execute arbitrary PHP and therefore use that to place additional malicious code.

The attacker also leveraged Drupal’s system table to get more complex attack code loaded. They created a record for a file to be loaded as a module and then uploaded that file to the site’s files directory where they were guaranteed Drupal had write access.

filename: sites/default/files/styles/medium/public/57h3d21.jpg
name:overly
type: module
owner:
status:1
bootstrap: 0
schema_version:0
weight: 0
info: a:11:{s:4:"name";s:6:"overly";s:11:"description";s:58:"Displays the Drupal administration interface in an overly.";s:7:"package";s:4:"Core";s:7:"version";s:4:"7.32";s:4:"core";s:3:"7.x";s:7:"project";s:6:"drupal";s:9:"datestamp";s:10:"1413387510";s:12:"dependencies";a:0:{}s:3:"php";s:5:"5.2.4";s:5:"files";a:0:{}s:9:"bootstrap";i:0;}

This was the script doing the redirects and filtering traffic so that pages only appeared to search engines. Usually these records have filenames that are .module, .php, or .inc files, but in this case it was a .jpg file named to be similarly to actual files on the site to make it hard to spot.

The content of that file was a PHP script not an image. The script did several things, and was the main tool the attacker was actively using during the time we were trying to stop them. It served as a simple proxy of content that they would present to the search engines, and redirect those same pages to the scam site for anyone else. It also provided code to make sure the content of user login forms was sent to the attacker, and a backup backdoor incase some of their others were lost.

We actually had to remove this particular attack more than once (always using the misspelled “overly” module) and each time it came back with a new file, and each time using a different but similar disguise to try to make their code blend in with legitimate files.

.htaccess files in public files

The other trick that was new to me (and a more aggressive stance by Drupal core on this approach is being discussed) was to take advantage of the .htaccess patterns in Apache to re-enable PHP execution within the public files directory. Drupal’s default .htaccess file disables PHP at the root of the public files directory and in theory all subdirectories, but that can simply be undone by a malicious .htaccess file (unless you block it in Apache’s main configuration – which in my opinion defeats the purpose of using .htaccess).

The attacker had placed a number of basic PHP-based exploits on the server using this technique to allow them to run the scripts. The tools themselves were not Drupal-specific, and likely the .htaccess file would work just as well on a number of other PHP-based CMS platforms.
Since the files directory gets deep and complicated there is no reasonable way to scan the whole thing by hand: particularly since several of the files were using inaccurate file extensions (like .jpg or no extension at all) and file names meant to blend into the background. So in addition to checking for any .htaccess files below the files directory root, I wrote a simple Python script to scan a directory for anything that includes the string <?php:

How we fixed it

We immediately made sure all code on the site was up-to-date, and I removed every exploit I could find. And for a couple days I played whack-a-mole with the attacker. Every day I would remove a series of exploits, disable their ability to redirect users to their scam, and every night they would break back in through a backdoor I’d failed to find.

The final solution was to replace the server, deploy a version of the code known to be good, and deploying copies of the database and files that have been scanned for any PHP in places it shouldn’t be – which involved a combination of the scanner above and hand checking every place the database stores PHP, not a fast process.

What we will do better next time

Part of any security event of this nature needs to be a full review of your internal processes and controls to make sure you reduce your risk and improve your response the next time something occurs (because unfortunately there will be a next time for all of us).

One of our first areas of improvement is a shared understanding that it’s more important to resolve the attack than determine the cause. This goes against the question that developers are constantly asked during and after an attack: “How did this happen?” While you need to know something about what happened, in the end it’s more important to make it stop. I still don’t know what happened that started the attack, I know I stopped it by blocking every attack vector I could think of and replacing every part of the stack with a version known to be fully up-to-date. It would have been faster and cheaper if I’d just started there: yes there is a risk I would have missed some of the code in the database if I hadn’t taken the time to review what I was finding, but frankly I doubt that risk is has high as the risk that new exploits will appear while I’m working to understand the previous one.

Beyond that basic shift in approach we developed a three part list of improvements:

  • Things we needed to improve right away.
  • Things we needed to improve soon.
  • Things that should be part of ongoing improvement.

The highest priority items were coming up with better internal process for initial response, and making sure we are deploying all security updates in a timely but still careful manner, including monitoring our hosting partners to ensure servers stay up-to-date as well. These are basics that are easy to let slip over time – particularly monitoring that your partners are doing their job correctly.

The second category of fixes is filled with workflow and procedure improvements. We were already were in the process of improving our code handling (migrating from SVN to Git, better production monitoring, more internal code review, etc), and we accelerated our plans to complete that work. This category also includes a complete review of our existing backup procedures to make sure they provide the level of coverage our clients need.

The final category of longer term adjustments includes tasks that include ensuring all developers are given (and expected to take) professional development opportunities around security best practices, doing more internal sharing about emerging ideas and trends, and encouraging more community engagement so we are better able to leverage the community resources in a crisis.

Let them eat Drupal Cake

Last week Cyberwoven hosted the local SC DUG. To encourage people to come when we hold these events in Columbia I’ve blatantly started to bribe people with baked goods: I give the presenter a choice of Cookies, Cake, or Pie – everyone picks cookies (unless I swap in Brownies instead of cookies then they pick brownies), no one asks for pie or cake. This month Will gave a talk about Docker and asked for cake! Since I was excited to finally have someone actually ask for something interesting I decided to do something Drupal themed:  I made Drupal Cake.

Turns out if you go Googling for Drupal cake you get ideas for how to make a yellow or white cake and decorate it with a Drupal logo. There are some notable counter examples, but I wanted something with more Drupal baked in.

So I made a Chocolate Blue Velvet Cake with a Drupal logo in blueberries.

This recipe is derived from All Cake’s Considered’s Dark-Chocolate Red Velvet Cake (if you’re an NPR-nerd you should buy a copy but it is also available in its entirety from the Internet Archive – I assume legally).

Of course hers is red, and I wanted blue. But since my actual inspiration came from people providing recipes (mostly bad ones) for baby shower cakes getting suggestions about how much food coloring to use wasn’t hard. So a little marriage of ideas and you have Chocolate Blue Velvet Drupal Cake.

Ingredients

Cake:

  • 2 Sticks of unsalted butter at room temperature
  • 1 ¼ cups sugar
  • 1 ¼ cups brown sugar
  • 6 large eggs at room temperature
  • 2 teaspoons vanilla extract
  • 3 cups all-purpose flour
  • ½ teaspoon baking soda
  • ¼ cup Dutch process cocoa
  • ½ teaspoon baking powder
  • 1 cup sour cream
  • 1 ounce Blue icing coloring (my version used food coloring but didn’t have quite the color I wanted.  Icing coloring is usually stronger so this should get you closer).

Icing:

  • ½ cup (1 stick) unsalted butter at room temperature
  • 2 8-ounce packages cream cheese at room temperature
  • 32-ounces of confectioners’ (powdered) sugar.
  • 1 teaspoon vanilla extract
  • 1 package of fresh Blueberries

Instructions

  1. Preheat the oven to 325°F
  2. Cream the butter and then gradually add the sugars beating well as you go.
  3. Add the eggs one at a time, beating well between each.
  4. Add the vanilla extract and beat for another couple of minutes.
  5. In a separate bowl combine and lightly mix the dry ingredients.
  6. Alternate adding roughly ⅓ of the dry mixture followed by ⅓ of the sour cream, beating well after each addition, until all of both are fully incorporated.
  7. Add the coloring, and continue to beat well.  After a minute stop the mixer and using a spatula to get any unevenly dyed batter off the sides and bottom, and then beat until the color is even.
  8. Pour the batter into a pair of well greased 8 or 9-inch round cake pans, and place them in the oven so they have as similar of conditions as possible.
  9. Bake for 45 minutes or until the cake tests done.
  10. Cool the layers in their pans for 10 minutes and then remove from their pans carefully.  Let them cool to room temperature.

Frosting:

  1. Cream the butter and the cream cheese at medium speed.
  2. Gradually add the confectioners sugar.  Continue beating until the it is light and fluffy.

To Assemble the Cake:

  1. Wait until the cake is fully cooled to room temperature. You need one to be flat (to form the bottom layer) so if domes formed on both cakes, use a long knife to cut the top of one layer to be flat (you can do both if you want the top flat, but drops are round).
  2. Place about a ⅓ of the icing in a separate bowl, and working from this smaller amount (we’ll get back to the rest later – this is just to avoid getting crumbs in your icing) put a smooth later cross the top of the bottom layer.
  3. Place the top layer on the bottom layer, and ice the top of the cake, and then the sides. This first coat should be fairly thin.  Give it a few minutes to dry on the surface before proceeding with the second coat.
  4. Using the rest of the icing, apply a thick final coat of icing, particularly on top.
  5. Using the blueberries attempt to create a Drupal logo pattern on the top of the cake.
Top view of the cake.
I didn’t get a picture of the inside so here’s another view of the logo – yes that’s supposed to be the Drupal 8 logo. Happy 1st birthday to Drupal 8.

Lessons learned from my first Drupal 8 projects

During last month’s SCDUG I gave two presentations. I’ve already posted the Sins Against Drupal talk I gave. That didn’t take up much time, and since I had been prepping some thoughts on the first few Drupal 8 projects I gave a second short talk on what I’ve learned working with Drupal 8 during its first year.


What’s not new?

Drupal 8 brings forward lots of concepts from Drupal 7 (and prior versions). It also brings forward a few standard community realities.

The information architecture tools are basically the same at least at the conceptual level. Nodes, Fields, Taxonomy, and Menus all still exist more or less as we’ve known them for years, and while there are differences on the surface those differences are incremental in nature and scope.

We also still have the constant hunt for modules that do what you’re doing so you don’t reinvent the wheel. Many modules that people are used to using are still missing, but new things arrive daily, and many of the new versions are significant improvements over previous generations of tools.

And there is still a lack of a clear line between front-end and back-end. When does business logic end and interface begin? When does a themer need to understand Drupal’s HTML generation vs when does a backend developer need to figure out how to force Drupal to generate carefully crafted markup?  There are opportunities to form better and clearer lines, but they aren’t automatic by any means: every team will have to solve this problem their own way.

What’s so great?

Drupal 8 opens up a collection of new tools and opportunities for the community.

  • As a backend developer you get to write modern code. The name spacing can feel a little Java-esc at times, but the ability to properly name space code, ditch globals, move from hooks to event listeners, and other basic OOP tools is incredibly nice.
  • With CKEditor in core we get better integration between that interface and the rest of Drupal. And better modules are coming out all the time to solve long standing UX annoyances. For example with the D8 Editor File upload module files and images can both be handled as Drupal file objects, but the editor can know the difference between a file (which should just be a link) and an image (which you should display).
  • The Symfony community provides a large number of packages that provide 3rd party integrations, or the tools to make them easy to build.
  • The two core base themes do not require keelhauling to make viable. If you commonly built your themes from scratch, the ability to have clean default markup that’s easy to override makes both Stable and Classy a major improvement to life.

Things you need to survive

As you dive into your first projects you need to understand that much of what you know at the detail level has changed.  And so you’ll need to learn a few new tricks and be willing to toss aside a few old ideas:

Your old build standards are wrong. Probably not entirely, but you’ll need new modules in your default builds, new best practices about when to use a node vs block vs custom entities, and other basic details you probably have really well develop standards (or at least habits) from years of working with Drupal.

If you’ve been avoiding it to-date, it’s time to develop an understanding about composer, drush, and Drupal consoleNot only do you need to be using these tools (probably all three) but you need to understand what each provides and which tool is best for which job.

Nginx is not for the faint of heart: as best I can find, no one has published a complete setup guide yet. In Drupal 6 and 7 there were pretty good guides to setting up Nginx properly, but with Drupal 8 there are enough differences that those guides don’t really work. And all the guides that I have read so far include errors (some of them significant security mistakes). To be clear, it can be done, but if you want to do it yourself be prepared to do a great deal of extra research along the way.

If you have windows you have pain. There are a number of windows specific challenges (speed and NTFS weakness being the biggest we face), and there is little community support to help you overcome the challenges.

Core patches tend to be required. Unlike Drupal 7, core and module patching is the norm not the exception. There are several issues that are frustratingly slow to get fixed that hit some fairly common use cases (like this menu block bug). While this is improving all the time, we haven’t launched a D8 site without at least one patch in place.

Cache tags are great, but require learning. The new caching system is powerful, flexible, and totally different from what we had before.  It’s better, but to use it well you’ll need to spend some time getting to know when to set and clear the tags you want to create.

Twig is great, but requires learning and discipline. I really like twig and the much cleaner syntax to brings to the theme layer.  However, as more and more people use it they are finding ways to move increasingly complex logic out of modules and theme PHP into template files. Please fight this urge! Keep your business logic separate from your display logic. If some object wasn’t loaded into a variable in your twig file do not attempt to load it in twig. If you need some 4 or 5 layer selector to get to the value you want to print: fix that in a preprocess function.

The API improvements are coming all the time and make things interesting. So far the community has stayed on schedule of rolling out new minor versions of 8 and that’s meant great new features in each version. It has also meant that sometimes a solution you built is not using the best techniques given the improvements.  That’s not really a problem, but can add headaches in maintenance cycles. Also you will find places where brand new D8 tools are already deprecated but the replacements don’t have good example implementations yet.

XML to JSON in Five lines

In closing I want to share the feature that caused me to realize that Symfony was worth the cost of admission: a 5 line XML to JSON AJAX callback.

My first site had a location finder that links to a data provider partner locations nationwide. That provider has an XML-base API that cannot be accessed directly from browsers, and therefore needed to be proxied through the main site. The full details involve more than just the following lines of code (providing input checking, settings, routing, etc), but the heart of the process is just five lines.

    $client = new Client(); // 1: Create Guzzle Client
    try{
      $res = $client-&amp;gt;get($config-&amp;gt;get('api_endpoint'), // 2: Make request
            [
              'http_errors'=&amp;gt;false,
              'query' =&amp;gt;; [
                 'key'=&amp;gt; $config-&amp;gt;get('coop_api_key'),
                     // ...
               ],
            ]);
      $decoderRing = new XmlEncoder(); // 3: Create XML Encoder
      $xml = $decoderRing-&amp;gt;decode($res-&amp;gt;getBody()); // 4: Decode the XML Response into an array.
      return new JsonResponse($xml); // 5: Return the array as JSON
    } catch (RequestException $e) {
      throw new HttpException($this-&amp;gt;t('Unable to process request'));
    }

Sins Against Drupal 3

This is part of my ongoing series about ways Drupal can be badly misused. These are generally times someone tried to solve an otherwise interesting problem in just about the worst possible way. All of these will start with a description of the problem, how not to solve it, and then ideas about how to solve it well.

I present these at SC Drupal Users Group meetings from time to time as an entertaining way to discuss ways we can all improve our skills.

This one was presenting during our October event here in Aiken, SC.


The Problem

Provide a custom authentication solution that allows staff to have one backend and members another.

The Sinful Solution

In order to force staff to use the staff login page, during login form validation check to see if the user is a staff member, by authenticating the user, checking their groups, and logging out staff.

The Code

/**
* Prevents staff members from logging in outside of staff login page. &amp;amp;amp;lt;&amp;amp;amp;lt;-- Why?
*/
function my_auth_staff_boot($form, &amp;amp;amp;amp;$form_state) { // NOT actually a hook_boot (thankfully) called as login form validator...
  user_authenticate($form_state['values']);
  global $user;
  if (in_array('An Employee', $user-&amp;amp;amp;gt;roles)) {
    form_set_error($form['#id'], l(t('Staff must log in via staff-login', 'staff-login')), TRUE);
    drupal_set_message('Staff must log in via ' . l(t('staff-login', 'staff-login')), 'error', TRUE);
    // Load the user pages in case they have not been loaded.
    module_load_include('inc', 'user', 'user.pages');
    user_logout();
  }
}

Why is this so bad?

This code actually completes the login process before kicking the user out. Why would you ever want to do that to your users? What did they do to you? It also loads an extra file for no apparent reason just before kicking the user back out.

Better Solutions

The goal here is to control what backend the user logs into, and shouldn’t control the page they login from. So the place to look for solutions are modules that already do this and so I propose mimicking the LDAP or GAuth modules’ approaches. LDAP attaches a validator to the form and takes over authentication, but LDAP supports lots of options so the code there is too extensive to use for a clear example. So for discussion I pulled out elements of the GAuth module (although there is still lots of trimming to make this understandable).

The GAuth module adds a submit button to the form and handles all processing for that form directly.

/**
* Implements hook_form_alter().
*/
function gauth_login_form_alter(&amp;amp;amp;amp;$form, &amp;amp;amp;amp;$form_state, $form_id) {
  if ($form_id == 'user_login' || $form_id == 'user_login_block') {
    $form['submit_google'] = array(
      '#type' =&amp;amp;gt; 'submit',
      '#value' =&amp;amp;gt; t(''),
      '#submit' =&amp;amp;gt; array('gauth_login_user_login_submit'),
      '#limit_validation_errors' =&amp;amp;gt; array(),
      '#weight' =&amp;amp;gt; 1000,
    );
    drupal_add_css(drupal_get_path('module', 'gauth_login') . '/gauth_login.css');
  }
}

/**
* Login using google, submit handler
*/
function gauth_login_user_login_submit() {
  if (variable_get('gauth_login_client_id', FALSE)) {
// .. skipping resource validation ...

  $client = new Google_Client();
// .. skipping client setup ...
  $url = $client-&amp;amp;gt;createAuthUrl();
  // Send the user off to Google for processing
  drupal_goto($url);
  }
  // ... skip errors
}

From there we pass through a menu router from the main module, and an API hook to get:

function gauth_login_gauth_google_response() {
  if (isset($_GET['state'])) {
// Skipping some error traps...
    $redirect_url = isset($state['destination']) ? $state['destination'] : '';
    if (isset($_GET['code'])) {
// Skipping a bunch of Client setup...
      $oauth = new Google_Service_Oauth2($client);
      $info = $oauth-&amp;amp;amp;gt;userinfo-&amp;amp;amp;gt;get();
      if ($uid = gauth_login_load_google_id($info['id'])) {
        $form_state['uid'] = $uid;
        user_login_submit(array(), $form_state); // &amp;amp;lt;&amp;amp;lt; That right there with the $form_state['uid'] set does the magic.
      }
      else {
// Skipping other options....
      }
    }
    drupal_goto($redirect_url); // &amp;amp;amp;lt;&amp;amp;amp;lt; be nice and handle the destination parameter
  }
}

Share your sins

I’m always looking for new material to include in this series. If you would like to submit a problem with a terrible solution, please remove any personally identifying information about the developer or where the code is running (the goal is not to embarrass individuals), post them as a gist (or a similar public code sharing tool), and leave me a comment here about the problem with a link to the code. I’ll do my best to come up with a reasonable solution and share it with SC DUG and then here. I’m presenting next month so if you have something we want me to look at you should share it soon.

If there are security issues in the code you want to share, please report those to the site owner before you tell anyone else so they can fix it. And please make sure no one could get from the code back to the site in case they ignore your advice.

Sins Against Drupal 2

This is part of my ongoing series about ways Drupal can be badly misused. These examples are from times someone tried to solve an otherwise interesting problem in just about the worst possible way.

I present these at SC Drupal Users Group meetings from time to time as an entertaining way to discuss interesting problems and ways we can all improve.

This one was presented about a year ago now (August 2015). Since I wasn’t working with Drupal 8 when I did this presentation the solution here is Drupal 7 (if someone asks I could rewrite for Drupal 8).


The Problem

The developer needed to support existing Flash training games used internally by the client. Drupal was used to provide the user accounts, game state data, and exports for reporting. The games were therefore able to authenticate with Drupal and save data to custom tables in the main Drupal database. The client was looking for some extensions to support new variations of the games and while reviewing the existing setup I noticed major flaws.

 

The Sinful Solution

Create a series of bootstrap scripts to handle all the interactions, turning Drupal into a glorified database layer (also while you’re at it, bypass all SQL injection attack protections to make sure Drupal provides as little value as possible).

The Code

There was a day when bootstrap scripts with a really cool way to do basic task with Drupal. If you’ve never seen or written one: basically you load bootstrap.inc, call drupal_bootstrap() and then write code that takes advantage of basic Drupal functions – in a world without drush this was really useful for a variety of basic tasks. This was outmoded (a long time ago) by drush, migrate, feeds, and a dozen other tools. But in this case I found the developer had created a series of scripts, two for each game, that were really similar, and really really dangerous. The first (an anonymized is version shown below) handled user authentication and initial game state data, and the second allowed the game to save state data back to the database.

As always the script here was modified to protect the guilty, and I should note that this is no longer the production code (but it was):

&amp;lt;?php
require_once './includes/bootstrap.inc';
drupal_bootstrap(DRUPAL_BOOTSTRAP_FULL); // "boot" Drupal
define("KEY", "ed8f5b8efd2a90c37e0b8aac33897cc5"); // set key

// check data
if(!(isset($_POST['hash'])) || !(isset($_POST['username'])) || !(isset($_POST['password']))) {
header('HTTP/1.1 404');
echo "status=100";
exit; // missing a value, force quit
}

// capture data
$hash = $_POST['hash'];
$username = $_POST['username'];
$password = $_POST['password'];

// check hash validity
$generate_hash = md5(KEY.$username.$password);
if($generate_hash != $hash) {
header('HTTP/1.1 404');
echo "status=101";
exit; // hash is wrong, force quit
}

// look for username + password combo
$flashuid = 0;
$query = db_query("SELECT * FROM {users} WHERE name = '$username' AND pass = '$password'");
if ($obj = db_fetch_object($query)){
$flashuid = $obj-&amp;gt;uid;
}

if($flashuid == 0) {
header('HTTP/1.1 404');
echo "status=102";
exit; // no match found
}

// get user game information
$gamequery = db_query("SELECT * FROM {table_with_data_for_flash_objects} WHERE uid = '$flashuid' ORDER BY lastupdate DESC LIMIT 1");

if ($game = db_fetch_object($gamequery)){
$time = $game-&amp;gt;time;
$round = $game-&amp;gt;round;
$winnings = $game-&amp;gt;winnings;
$upgrades = $game-&amp;gt;upgrades;
} else {

// no entry, create one in db
$time = $round = $game_winnings = $long_term_savings = $bonus_list = "0";
$upgrades = "";
$insert = db_query("INSERT INTO {table_with_data_for_flash_objects} (uid, lastupdate) VALUES ('$flashuid',NOW())");
}

$points = userpoints_get_current_points($flashuid);

// echo success and values
header('HTTP/1.1 201');
echo "user_id=$flashuid&amp;amp;points=$points&amp;amp;ime=$time&amp;amp;round=$round&amp;amp;winnings=$winnings&amp;amp;upgrades=$upgrades";

?&amp;gt;

Why is this so bad?

It’s almost hard to know where to be begin on this one, so we’ll start at the beginning.

  • Bootstrap scripts are not longer needed and should never have been used for anything other than a data import or some other ONE TIME task.
  • That key defined in line 3, that’s used to track sessions (see lines 20-21). If you find yourself having to recreate a session handler with a fixed value, you should assume you’re doing something wrong. This is a solved problem, if you are re-solving it you better be sure you know more than everyone else first.
  • Error handling is done inline with a series of random error status codes that are printed on a 404 response (and the flash apps ignored all errors). If you are going to provide an error response you should log it for debugging the system, and you should use existing standards whenever possible. In this case 403 Not Authorized is a far better response when someone fails to authenticate.
  • Lines 15-17, and then line 30: a classic bobby tables SQL Injection vulnerability. Say goodbye to security from here on in. They go on to repeat this mistake several more times.
  • Finally, just to add insult to injury, the developer spends a huge amount of time copying variables around to change their name: $password = $_POST[‘password’]; $round = $game->round; There is nothing wrong just using fields on a standard object, and while there is something wrong with just using a value from $_POST, copying it to a new variable does not make it trustworthy.

Better Solutions

There are several including:

  • Use a custom menu to define paths, and have the application just go there instead.
  • Use Services module: https://www.drupal.org/project/services
  • Hire a call center to ask all your users for their data…

If I were starting something like this from scratch in D7 I would start with services and in D8 I’d start with the built-in support for RESTful web services. Given the actual details of the situation (a pre-existing flash application that you have limited ability to change) I would go with the custom router so you can work around some of the bad design of the application.

In our module’s .module file we start by defining two new menu callbacks:

function hook_menu() {

$items['games/auth'] = array(
'title' =&amp;gt; 'Games Authorization',
'page callback' =&amp;gt; 'game_module_auth_user',
'access arguments' =&amp;gt; array('access content'),
'type' =&amp;gt; MENU_CALLBACK,
);
$items['games/game_name/data'] = array( // yes, you could make that a variable instead of hard code
'title' =&amp;gt; 'Game Data',
'page callback' =&amp;gt; 'game_module_game_name_capture_data', // and if you did you could use one function and pass that variable
'access arguments' =&amp;gt; array('player'),
'type' =&amp;gt; MENU_CALLBACK,
);

return $items;
}

The first allows for remote authentication, and the second is an endpoint to capture data. In this case the name of the game is hard coded, but as noted in the comments in the code you could make that a variable.

In the original example the data was stored in a custom table for each game, but never accessed in Drupal itself. The table was not setup with a hook_install() nor did they need the data normalized since its all just pass-through. In my solution I switch to using hook_install() to add a schema that stores all the data as a blob. There are tradeoffs here, but this is a clean simple solution:

...
'fields' =&amp;gt; array(
'recordID' =&amp;gt; array(
'description' =&amp;gt; 'The primary identifier for a record.',
'type' =&amp;gt; 'serial',

...
'uid' =&amp;gt; array(
'description' =&amp;gt; 'The user ID.',
'type' =&amp;gt; 'int',

...
'game' =&amp;gt; array(
'description' =&amp;gt; 'The game name',
'type' =&amp;gt; 'text',

...
'data' =&amp;gt; array(
'description' =&amp;gt;'Serialized data from Game application',
'type' =&amp;gt; 'blob',

...

You could also take this one step further and make each game an entity and customize the fields, but that’s a great deal more work that the client would not have supported.

The final step is to define the callbacks used by the menu items in hook_menu():

function game_module_auth_user($user_name = '', $pass = '') { // Here I am using GET, but I don’t have to

global $user;
if($user-&amp;gt;uid != 0) { // They are logged in already, so reject them
drupal_access_denied();
}

$account = user_authenticate($user_name, $pass);

//Generate a response based on result....
}

function game_module_[game_name]_capture_data() {
global $user;
if($user-&amp;gt;uid == 0) { // They aren’t logged in, so they can’t save data
drupal_access_denied();
}

$record = drupal_get_query_parameters($query = $_POST); // ← we can work with POST just as well as GET if we ask Drupal to look in the right place.

db_insert('game_data')
-&amp;gt;fields(array(
'uid' =&amp;gt; $user-&amp;gt;uid,
'game' =&amp;gt; '[game_name]',
'data' =&amp;gt; serialize($record),
))
-&amp;gt;execute();
// Provide useful response.
}

For game_module_auth_user() I use a GET request (mostly because I wanted to show I could use either). We get the username and password, have Drupal authenticate them, and move on; I let Drupal handle the complexity.

The capture data callback does pull directly from the $_POST array, but since I don’t care about the content and I’m using a parameterized query I can safely just pass the information through. drupal_get_query_parameters() is a useful function that often gets ignored in favor of more complex solutions.

So What Happened?

The client had limited budget and this was a Drupal 6 site so we did the fastest work we could. I rewrote the existing code to avoid the SQL Injection attacks, moved them to SSL, and did a little other tightening, but the bootstrap scripts remained in place. We then went our separate ways since we did not want to be responsible for supporting such a scary set up, and they didn’t want to fund an upgrade. My understanding is they heard similar feedback from other vendors and eventually began the process of upgrade. You can’t win them all, even when you’re right.

Share your sins

I’m always looking for new material to include in this series. If you would like to submit a problem with a terrible solution, please remove any personally identifying information about the developer or where the code is running (the goal is not to embarrass individuals), post them as a gist (or a similar public code sharing tool), and leave me a comment here about the problem with a link to the code. I’ll do my best to come up with a reasonable solution and share it with SC DUG and then here. I’m presenting next month so if you have something we want me to look at you should share it soon.

If there are security issues in the code you want to share, please report those to the site owner before you tell anyone else so they can fix it. And please make sure no one could get from the code back to the site in case they ignore your advice.

Sins Against Drupal 1

This is the first is an ongoing series about ways Drupal can be badly misused. These are generally times someone tried to solve an otherwise interesting problem in just about the worst possible way. All of these will start with a description of the problem, how not to solve it, and then ideas about how to solve it well.

I present these at SC Drupal Users Group meetings from time to time as an entertaining way to discuss ways we can all improve our skills.

This first one was presented awhile ago now (Feb of 2015).


The Problem

The developer needed to support an existing JavaScript app with access to content in the form of Drupal nodes encoded in JSON. This is a key part of any headless Drupal project (this entire site was not headless, just one application), and in Drupal 7 and earlier there was no way to do this in core.

The Sinful Solution

Create a custom response handler within template.php that executes every time template.php is loaded.

The Code

During a routine code review of this site I found the following code in template.php:

...
function theme_name_preprocess_region(&amp;amp;$vars) {
if ($vars['region'] == 'header') {
$vars['classes_array'][] = 'clearfix';
}
}

if (isset($_POST['mode'])) {
if ($_POST['mode'] == 'get_fields_node') {
$node = node_load($_POST['id']);

$container = array();

$sliderCounter = count($node-&amp;gt;field_event_slider['und']);
for ($i = 0; $i &amp;lt; $sliderCounter; $i++) { $field_collection_id = $node-&amp;gt;field_event_slider['und'][$i]['value'];
$field_collection = entity_load('field_collection_item', array($node-&amp;gt;field_event_slider['und'][$i]['value']));

$currentCollectionItem = $field_collection[$field_collection_id];

if (isset($currentCollectionItem-&amp;gt;field_slider_image['und'][0]['uri'])) {
$container['slider'][$i]['src'] = file_create_url($currentCollectionItem-&amp;gt;field_slider_image['und'][0]['uri']);
}
if (isset($currentCollectionItem-&amp;gt;field_slider_caption['und'][0]['value'])) {
$container['slider'][$i]['caption'] = $currentCollectionItem-&amp;gt;field_slider_caption['und'][0]['value'];
}
}

$focusTid = $node-&amp;gt;field_event_type['und'][0]['tid'];
$eventTerm = taxonomy_term_load($focusTid);
$dateTid = $node-&amp;gt;field_event_date['und'][0]['tid'];
$dateTid = taxonomy_term_load($dateTid);

$container['nid'] = $node-&amp;gt;nid;
$container['focusTid'] = $focusTid;
$container['title'] = $node-&amp;gt;title;
$container['focus'] = $eventTerm-&amp;gt;name;
$container['date'] = $dateTid-&amp;gt;name;
$container['body'] = $node-&amp;gt;body['und'][0]['value'];

print json_encode($container);
die();
}

if ($_POST['mode'] == 'get_fields_focus') {
$focusTerm = taxonomy_term_load(substr($_POST['id'], 4));

$container['nid'] = $_POST['id'];
$container['title'] = $focusTerm-&amp;gt;name;
$container['body'] = $focusTerm-&amp;gt;description;

print json_encode($container);
die();
}
}
...

Notice that between the two functions is the random block of code wrapped in
if (isset($_POST['mode'])) {...}. So on every request that results in load the theme template.php is loaded and in addition to the normal parsing triggers a check to see if the page request was a POST that included a mode. It it was, we then proceed to load up a node, a couple taxonomy terms, and then encode them as JSON for response. The site sends the response and then unceremoniously dies().

Why is this so bad?

First, there is no parameter checking on the ID parameter: node_load($_POST[‘id’]). Anyone on the internet can load any node if they work out the ID. Since nodes are sequentially number, you could just start at 1 and your way up until they noticed that the site was sending the same useless response over and over. It also doesn’t send a 404 if the NID provided is invalid.

Second, no reasonable developer would expect you to hide a custom callback handler in template.php. It should be totally safe to load template.php without generating output under any condition (that should be true all non-tpl.php files).

Third, this code could run during any page request, not just the ones the application designer thought about. The request could have had Drupal do something relatively expensive before reaching this stage, and all that work was just wasted server resources – which creates an additional avenue for an attacker.

Fourth, Drupal has an exit function that actually does useful clean up and allows other modules to do the same. All that gets bypassed when you just die() midstream.

Finally, Drupal has tools to do all this. There was no reason to do this so badly.

Better Solutions

In Drupal 8 this is part of core.  Enable Restful Web Services and optionally the RestUI module, and in a few minutes you can have this more or less out of the box.

In Drupal 7 are two modules that will do 90% of the work for us. If we really just want the raw node as JSON you could use Content as JSON. But often we want more control over field selection, and the option to pull in related content (which the developer in this case did, and used as his argument for the approach taken). Views Datasource gives us the power of Views to select the data and provides us a JSON (and a few other) display option.

Views Datasource based approach:

  1. Install Views, ctools, and Views Datasource
  2. Create your view and set the display format to JSON data document.Drupal Sins 1 Views Definition
  3. Pick your fields, set the path, and define the contextual filter:Drupal Sins 1 Views field

From there save your view and you’re done. That’s all there is to it. No custom code to maintain, you get to rely on popular community tools to handle access checking and other security concerns, and you get multiple layers of caching.

So what happened?

This sin never saw the light of day.

At the time I encountered this “solution” I worked for a company that was asked review this site while it was still under development. Our code review gave the client a chance to go back to the developer and get a fix. The developer chose a more complicated solution than the Views one presented here (they defined a custom menu router with hook_menu() and moved much of this into the callback and added a few security checks) which was good enough for the project. But I still would have done it in views: it is much faster to develop, views plays nicely with Drupal caching to help improve performance, and is a straightforward approach is easy for a future developer to maintain.

Share your sins

I’m always looking for new material to include in this series. If you would like to submit a problem with a terrible solution, please remove any personally identifying information about the developer or where the code is running (the goal is not to embarrass individuals), post them as a gist (or a similar public code sharing tool), and leave me a comment here about the problem with a link to the code. I’ll do my best to come up with a reasonable solution and share it with SC DUG and then here.

If there are security issues in the code you want to share, please report those to the site owner before you tell anyone else so they can fix it. And please make sure no one could get from the code back to the site in case they ignore your advice.

This Week’s Drupal Fire Drill

This week many in the Drupal community lost a lot of sleep Tuesday night because the security team treated us to a warning about major security updates due out on Wednesday. Fortunately for many it wasn’t a crisis in the end, but it gave us all a chance to practice for the worst. Basically, it was like a fire drill in a elementary school: we got to prepare like there was a disaster, but since they wasn’t one we don’t really know how it would have gone if there was actually a fire. We haven’t had a stop-drop-and-roll type of emergency in a while, so it was a good refresher on how to handle a crisis.

For those who don’t know what I’m talking about here’s a quick review. At Cyberwoven, like many Drupal shops, we follow the Drupal Security twitter feed on one of our Slack channels so we saw this mid-afternoon Tuesday:

Slack posting of tweet from the security team.

I read the PSA with images of Drupalgeddon dancing in my head:

There will be multiple releases of Drupal contributed modules on Wednesday July 13th 2016 16:00 UTC that will fix highly critical remote code execution vulnerabilities (risk scores up to 22/25).These contributed modules are used on between 1,000 and 10,000 sites. The Drupal Security Team urges you to reserve time for module updates at that time because exploits are expected to be developed within hours/days. Release announcements will appear at the standard announcement locations.

Drupal core is not affected. Not all sites will be affected. You should review the published advisories on July 13th 2016 to see if any modules you use are affected.

Oh that bold line up there wasn’t part of the original announcement. On Tuesday we didn’t have a sense of scale, were we talking about modules that everyone uses on almost every site (ctools came up more than once). It’s the one thing I wish the security team had done differently: given us that sense of scale.

I read all security postings, and make sure we take prompt steps to address them for clients as needed, but the potential here was that we’d have to update all 70+ sites in a few hours or less which is very different from your run-of-the-mill security update that often aren’t related to use cases and threat profiles for the majority of sites.

Here’s what we did next:

Tuesday

  1. Took a minute to panic, complain, and joke about pending illnesses. This is actually a useful step because it allowed me to burn off some nervous energy and then to focus on the real work.
  2. Pulled out the list of all active clients with Drupal sites, and doubled checked it for accuracy.
  3. Made sure a developer had a working repo for all sites (70+). Since we had a couple people out of the office, and some projects had been reassigned recently to different developers, this was an important step to make sure no sites fell through the cracks during a rush to update them all.
  4. Made sure we knew which were 6 sites, and which were 7 in case we are able to determine that 6 is also affected. Since I knew the announcement would likely skip D6, we needed to accept that we might have to take those sites offline for a time.
  5. Made sure leadership knew that all developers may be busy start at 16:00 UTC on Wednesday. We didn’t actually cancel anything right away, but I didn’t want anyone surprised if we were all too busy posting and testing updates to worry about things like meetings.
  6. Made sure complex projects were thought about ahead of time: sites with unusual setups or ongoing dev work that make sudden updates complex. For example we have one client that has 16 sites that all have an unusual set up, so we agreed who would handle those and made sure she was prepared.

Wednesday

  1. First thing in the morning I saw the update to the notice that gave us a sense of scale and relaxed a little, but still made sure we were fully prepped.
  2. Noon: the announcement of what modules were affected was released and a couple other developers and I immediately reviewed the releases. We relaxed once we determined none of our clients were using any of the modules listed.
  3. I reviewed code from each of the three modules to see what the change was to look for ways to improve my own code to avoid similar errors.
  4. Looked for ways to improve our response for the next time it’s not a drill.

Things would have been more exciting if we’d had to update our sites. Since we were prepared it was a matter of minutes for us to check that all our sites were secure. Each developer checked all the sites in their sandbox, and since I knew all sites were in someone’s sandbox that gave us 100% coverage without having to do lots of double checks.

I think it is too easy to look past doing the code review of modules we weren’t using but I find this kind of follow up really useful. Looking back on Drupalgeddon it’s amazing how much pain was caused by such a small error (16 characters are all that were needed to fix it). And by seeking to understand what went wrong you can look for places that you make similarly invalid assumptions.

If you read my post on making new mistakes, you also know I believe that looking for improvement is the most important detail (particularly when it turned out to be a drill, not a fire). Here’s my initial list of things to improve:

  • Have a system to automatically check every site for specific modules (this was already under development but will take a little while longer to complete).
  • Make sure at least two developers have a working sandbox for all projects at all times in case something comes out during a vacation.
  • Improve internal messaging about what to expect – template message and process.  I tossed together some disjointed thoughts for account managers. But disjointed developer thinking does not make people feel like you’re on top of things.
  • Have better tracking of outliers:  I completely missed that I had a demo site on Pantheon that did have the Coder module running.  Since it was Pantheon they alerted me to this problem and had taken steps until I could do the update myself. But it would have been bad if that site had been someplace else and/or in production.
  • Make sure everyone one knows when the actual release is coming out, and what the outcome was. Several developers were hoping to get lunch before the updates, but hadn’t done so when the announcement came out (which could have been a problem if we’d ended up busy).  And I spent the rest of the day answering one-off questions from the account team who wanted to know if the announcement had been bad news.

Ideally we’d come up with a method to automate security updates (maybe all updates), but that’s not totally straightforward.  We have to worry about required patches, non-standard setups, automated testing, and other details. There has been discussion on the Pantheon power user’s mailing list, but every shop has a slightly different workflow (like the fact that we don’t use Pantheon much at all) so we’ll need to come up with a system that accommodates our system.

When Should I Update my Drupal Site to Drupal 8?

Last year Drupal 8 finally arrived, and brought the question that comes with every new release of Drupal: when should I update? New releases of Drupal mean two things: new features and cool new tools, and the retirement of an old version. We got the power and flexibility of Symfony and Drupal 6 sites are no longer getting community support. Unlike WordPress, which has well defined upgrade paths, each version of Drupal is a new adventure in upgrade pain. The more I watch people suffer with this pain, and the more I watch them try to find a way to do upgrades that preserve their site’s fundamental structure, the more I come to the conclusion that this pain is telling us something: we’re doing it wrong. Not because Drupal’s strategy is wrong, but because keeping all your content in the same structures is usually wrong. Drupal 8 should not make it easy for you to continue to use an old strategy, it should encourage us to update old assumptions.

Here is how I encourage everyone to view their choices:

If you have a Drupal 6 (or older) site you should update right now. Drupal 6 is no longer getting security updates so you are on borrowed time. But more importantly Drupal 8 is a better tool for the current state of the web than your Drupal 6 site. Most sites running on D6 reflect an online communications strategy that’s at least 4 or 5 years old. Those sites probably aren’t responsive, aren’t prepared to support apps, don’t have the right focus on social media and user engagement, and make assumptions about user behaviors that have evolved. Skip to Drupal 8: do not migrate these sites to Drupal 7. If there is a tool that is missing from Drupal 8 that your current site uses make sure you need it before complaining (or paying to have someone port it for you). Maybe that tool hasn’t been ported because it doesn’t make sense anymore. Some things are still missing, but lots of things are being rewritten differently because we have a better platform. The community is smarter than it was 5 or 10 years ago, and the platform is better, take the time to figure out why something hasn’t been ported: is it just no one has bothered, or has something better been built instead?

If you have a Drupal 7 site you should update when your web site no longer supports your work.  This is actually the same advice I just gave, but without a few assumptions like “you need a secure site.” Many Drupal 7 sites have a lot of life left in them. A site you built today will be designed to meet the needs you have now, and the ones you foresee in the near future. Three years from now (when Drupal 7 is scheduled to lose support from the community) you will be operating on assumptions that have probably been wrong for at least two years. Every six months you should ask yourself: does my site reflect my online strategy, and is my strategy still working? If the answer is yes to both of those questions you are fine, if the answer is no to either – particularly the second – you should engage someone to help you update your strategy and rebuild your site.

I’ve been part of projects that failed in part because we tried to port a stale strategy and stale content to a fresh site. We broke the new site before it even launched. Don’t try to make Drupal 8 behave like your old site: embrace the change.