Spinning Code

Some Things Every Developer Should Read

Every developer needs to be learning new things all the time. We need to have a good grounding in the ideas that have come before us and those that are emerging around us within our field, and we need to understand the larger social impact of our work. To be fair this isn’t just true of developers, but we are a field that has a bad habit of arguing our work is totally new and original – which is rarely true in any meaningful way.

Part of that push for constant learning is to read books about software development, about life, about business, about writing, and really anything that pushes how and what you think. There isn’t really a settled cannon of things every developer must have read but there are definitely some works you need to be familiar with the ideas in (even if you haven’t read them) to keep in conversations over lunch at a conference like the Mythical Man Month and the Agile Manifesto (both of which came up over a lunch at my last conference). There is also a much larger set of works that many developers benefit from reading even if they aren’t really about software.

So here is my list of works that I think every developer should read at least once in their life. I’m breaking the list into two pieces: specific books I think every developer should actually read, and a category of types of things that every developer should be on the lookout to read on a regular basis. I’d love to hear suggestions about should be added and what I should have read myself.

Specific Books

These are some specific books that I’ve found helpful for making me a better developer. None of the ones here are directly about programming – that’s not an accident. Many of the books I’ve read about development and the creation of software were helpful and I am glad to have spent time with them but these are works that helped me think more broadly about the craft and the when, why, and how I write code.

Design of everyday things: While many of the specific recommendations for how to fix some devices are out-of-date and flawed (I will never forget reading that all office phones needed to be easy to use was a two-line digital display while working at a desk with a proof the display had not solved the problem), the general philosophy holds up. Also, it will make you justifiably angry at every poorly installed door you struggle to open.
Zen and the Art of Motorcycle Maintenance: This is a long, slow, dense read. And if I hadn’t need assigned it in college I probably never would have gotten through it. But I find myself coming back to its discussion of quality all the time.
Technically Wrong: This is the newest book on this list, and it makes the cut cause the ideas are so important. I had the opportunity to see Sara Wachter-Boettcher speak at DrupalCon before she published the book and it was one of those talks that changed how lots of people in the room think about software and the role of our work in people’s lives.
Elements of Style (aka “Strunk and White”): I was assigned to read this in a history course in college and then again when I was promoted to Web Director at AFSC – cause the Director of Communications wanted to be sure I remembered the importance of writing well in that job (I did, but I re-skimmed the book anyway). If everyone wrote English like Strunk and White suggest writing would be much better on the whole – although fairly boring. Read it not because their rules are perfect, read it because you should think about the rules of writing when you break them.
Blah Blah Blah: This is something you should read after you’ve read a lot about writing clearly since it rather pitches the reverse concept: sometimes a rough picture is clearer and more useful than a carefully written description.

Things that push and inspire

These are examples of works that are helpful to push boundaries. The specific examples I cite may or may not be valuable to you, but they are examples of things within the category I’m describing.

Books that show you code you’d never write:

These are books that talk about code and concepts that either have no application to your work, or are ways of showing off what a language can do – even when you shouldn’t follow the example. I have two examples here from opposite ends of the spectrum:

If Hemingway wrote javascript: This book is brilliant. It’s a series of examples of how various writers might have completed common CS class programming assignments using JavaScript: DO NOT WRITE CODE LIKE THIS. It shows off how wonderfully flexible JavaScript is as a language, and presents some of the worst possible solutions to those problems along the way. It is a great read that will teach almost anyone a few things about JavaScript, and push you to think about different ways to approach a problem. It is a light, fast, and entertaining read.
Beautiful code: This is a fairly dry read. It is made up of series of examples of truly well written code: WRITE CODE LIKE THIS. It looks at code in a variety of languages, not all of which are in common use, solving problems that are sometimes familiar and sometimes rare (in part because of the code you’re reading). It is a slow dense, and at times boring read, but it’ll give you insight into what other people will find graceful in your work.

A college textbook

I mean actually read and actual textbook. Sure if you went to college someone assigned bits of pieces of these things to read, maybe they even assigned all the chapters in order. But how many people actually did all the reading in order? That doesn’t mean the professors weren’t right that those things are useful – they do not get assigned for the professors health (and if you think professors get rich off assigning their own books you need to spend time learning about the economics of academic publishing).

I have a couple around that I kept as reference after college, and one or two I’ve now actually read end to end. I will grant you these are not exciting works. I don’t do this as a regular reading habit, but it’s worth having done. If you don’t have one around, most libraries still have books and often a few reasonable high level textbooks.

Articles by someone whose ideas you don’t like

If you aren’t used to doing this I should be clear that this is about getting outside your comfort zone on topics you think you know a lot about, and reading things that push you to justify – or better yet modify – your understanding of the field. This doesn’t have to mean going and reading material you find deeply troubling or revisit ideas you know are morally reprehensible. Bothering with ideas from ideologies of hate is unlikely beneficial for most of us, think closer to home here. In this category I’m talking about project management practices you think are overblown: love Agile, read people who point out where it routinely fails; hate Agile, read some material about why it always works when done right. For me it is sometimes reading people who still argue technology is morally neutral. But best is often to find a topic you are still trying to resolve your own thinking on and absorb some uncomfortable ideas and wrestle with them.

Salesforce Queries and Proxies in Drupal 8

The Drupal 8 version of the Salesforce Suite provides a powerful combination of features that are ready to use and mechanisms for adding custom add-ons you may need. What it does not yet have is lots of good public documentation to explain all those features.

A recent support issue in the Salesforce issue queue asked for example code for writing queries. While I’ll address some of that here, there is ongoing work to replace the query interface to be more like Drupal core’s. Hopefully once that’s complete I’ll get a chance to revise this article, but be warned some of those details may be a little out of date depending on when you read this post.

To run a simple query for all closed Opportunities related to an Account that closed after a specific date you can do something like the following:

      $query = new SelectQuery('Opportunity');
      $query->fields = [
        'Id',
        'Name',
        'Description',
        'CloseDate',
        'Amount',
        'StageName',
      ];
      $query->addCondition('AccountId', $desiredAccountId, '=');
      $query->conditions[] = [
        "(StageName", '=', "'Closed Won'",
        'OR', 'StageName', '=', "'Closed Lost')",
      ];
      $query->conditions[] = ['CloseDate', '>=', $someSelectedDate];
      $sfResponse = \Drupal::service('salesforce.client')->query($query);

The class would need to include a use statement for to get Drupal\salesforce\SelectQuery; And ideally you would embed this in a service that would allow you to inject the Salesforce Client service more correctly, but hopefully you get the idea.

The main oddity in the code above is the handling of query conditions (which is part of what lead to the forthcoming interface changes). You can use the addCondition() method and provide a field name, value, and comparison as lie 10 does. Or you can add an array of terms directly to the conditions array that will be imploded together. Each element of the conditions array will be ANDed together, so OR conditions need to be inserted the way lines 11-14 handle it.

Running a query in the abstract is pretty straight forward, the main question really is what are you going to do with the data that comes from the query. The suite’s main mapping features provide most of what you need for just pulling down data to store in entities, and you should use the entity mapping features until you have a really good reason not to, so the need for direct querying is somewhat limited.

But there are use cases that make sense to run queries directly. Largely these are around pulling down data that needs to be updated in near-real time (so perhaps that list of opportunities would be ones related to my user that were closed in the last week instead of some random account).

I’ve written about using Drupal 8 to proxy remote APIs before. If you look at the sample code you’ll see the comment that says: // Do some useful stuff to build an array of data. Now is your chance to do something useful:

<?php
 
namespace Drupal\example\Controller;
 
use Symfony\Component\HttpFoundation\Request;
use Drupal\Core\Controller\ControllerBase;
use Drupal\Core\Cache\CacheableJsonResponse;
use Drupal\Core\Cache\CacheableMetadata;
 
class ExampleController extends ControllerBase {
    public function getJson(Request $request) {
        // Securely load the AccountId you want, and set date range.
 
        $data = [];
        $query = new SelectQuery('Opportunity');
        $query->fields = [
            'Id',
            'Name',
            'Description',
            'CloseDate',
            'Amount',
            'StageName',
        ];
        $query->addCondition('AccountId', $desiredAccountId, '=');
        $query->conditions[] = [
            "(StageName", '=', "'Closed Won'",
            'OR', 'StageName', '=', "'Closed Lost')",
        ];
        $query->conditions[] = ['CloseDate', '>=', $someSelectedDate];
        $sfResponse = \Drupal::service('salesforce.client')->query($query);
 
    if (!empty($sfResponse)) {
        $data['opp_count'] = $sfResponse->size();
        $data['opps'] = [];
 
        if ($data['opp_count']) {
            foreach ($sfResponse->records() as $opp) {
                $data['opps'][] = $opp->fields();
            }
        }
    }
    else {
      $data['opp_count'] = 0;
    }
    // Add Cache settings for Max-age and URL context.
    // You can use any of Drupal's contexts, tags, and time.
    $data['#cache'] = [
        'max-age' => 600, 
        'contexts' => [
            'url',
            'user',     
        ],
    ];
    $response = new CacheableJsonResponse($data);
    $response->addCacheableDependency(CacheableMetadata::createFromRenderArray($data));
    return $response;
  }
}

Cautions and Considerations

I left out a couple details above on purpose. Most notable I am not showing ways to get the needed SFID for filtering because you need to apply a little security checking on your route/controller/service. And those checks are probably specific to your project. If you are not careful you could let anonymous users just explore your whole database. It is an easy mistake to make if you do something like use a Salesforce ID as a URL parameter of some kind. You will want to make sure you know who is running queries and that they are allowed to see the data you are about to present. This is on you as the developer, not on Drupal or Salesforce, and I’m not risking giving you a bad example to follow.

Another detail to note is that I used the cache response for a reason. Without caching every request would go through to Salesforce. This is both slower than getting cached results (their REST API is not super fast and you are proxying through Drupal along the way), and leaves you open to a simple DOS where someone makes a bunch of calls and sucks up all your API requests for the day. Think carefully before limiting or removing those cache options (and make sure your cache actually works in production). Setting a context of both URL and User can help ensure the right people see the right data at the right time.

Drupal Salesforce Suite Custom Field Mapping Types

The Drupal 8 Salesforce Suite allows you to map Drupal entities to Salesforce objects using a 1-to-1 mapping. To do this it provides a series of field mapping types that allow you to select how you want to relate the data between the two systems. Each field type provides handling to help ensure the data is handled correctly on each side of the system.

As of this writing the suite provides six usable field mapping types:

Properties — The most common type to handle mapping data fields.
Record Type — A special handler to support Salesforce record type settings when needed.
Related IDs — Handles translating SFIDs to Drupal Entity IDs when two objects are related in both systems.
Related Properties — For handling properties across a relationship (when possible).
Constant — A constant value on the Drupal side that can be pushed to Salesforce.
Token — A value set via Drupal Token.

There is a seventh called Broken to handle mappings that have changed and need a fallback until its fixed. The salesforce_examples module also includes a very simple example called Hardcoded the shows how to create a mapping with a fixed value (similar to, but less powerful than, Constant field).

These six handle the vast majority of use cases but not all. Fortunately the suite was designed using Drupal 8 annotated plugins , so you can add your own as needed. There is an example in the suite’s example module, and you can review the code of the ones that are included, but I think some people would find an overview helpful.

As an example I’m using the plugin I created to add support for related entities to the webform submodule of the suite (I’m referencing the patch in #10 cause that’s current as of this writing, but you should actually use whatever version is most recent or been accepted).

Like all good annotated plugins to tell Drupal about it all we have to do is create the file in the right place. In this case that is: [my_module_root]/src/Plugins/SalesforceMappingField/[ClassName] or more specifically: salesforce_webform/src/Plugin/SalesforceMappingField/WebformEntityElements.php

At the top of the file we need to define the namespace, add some use statements.

<?php
 
namespace Drupal\salesforce_webform\Plugin\SalesforceMappingField;
 
use Drupal\Core\Entity\EntityInterface;
use Drupal\Core\Form\FormStateInterface;
use Drupal\salesforce_mapping\Entity\SalesforceMappingInterface;
use Drupal\salesforce_mapping\SalesforceMappingFieldPluginBase;
use Drupal\salesforce_mapping\MappingConstants;

Next we need to provide the required annotation for the plugin manager to use. In this case it just provides the plugin’s ID, which needs to be unique across all plugins of this type, and a translated label.

/**
 * Adapter for Webform elements.
 *
 * @Plugin(
 *   id = "WebformEntityElements",
 *   label = @Translation("Webform entity elements")
 * )
 */

Now we define the class itself which must extend SalesforceMappingFieldPluginBase.

class WebformEntityElements extends SalesforceMappingFieldPluginBase {

With those things in place we can start the real work. The mapping field plugins are made up of a few parts:

The configuration form elements which display on the mapping settings edit form.
A value function to provide the actual outbound value from the field.
Nice details to limit when the mapping should be used, and support dependency management.

The buildConfigurationForm function returns an array of form elements. The base class provides some basic pieces of that array that you should plan to use and modify. So first we call the function on that parent class, and then make our changes:

 /**
   * {@inheritdoc}
   */
  public function buildConfigurationForm(array $form, FormStateInterface $form_state) {
    $pluginForm = parent::buildConfigurationForm($form, $form_state);
 
    $options = $this->getConfigurationOptions($form['#entity']);
 
    if (empty($options)) {
      $pluginForm['drupal_field_value'] += [
        '#markup' => t('No available webform entity reference elements.'),
      ];
    }
    else {
      $pluginForm['drupal_field_value'] += [
        '#type' => 'select',
        '#options' => $options,
        '#empty_option' => $this->t('- Select -'),
        '#default_value' => $this->config('drupal_field_value'),
        '#description' => $this->t('Select a webform entity reference element.'),
      ];
    }
    // Just allowed to push.
    $pluginForm['direction']['#options'] = [
      MappingConstants::SALESFORCE_MAPPING_DIRECTION_DRUPAL_SF => $pluginForm['direction']['#options'][MappingConstants::SALESFORCE_MAPPING_DIRECTION_DRUPAL_SF],
    ];
    $pluginForm['direction']['#default_value'] =
      MappingConstants::SALESFORCE_MAPPING_DIRECTION_DRUPAL_SF;
    return $pluginForm;
 
  }

In this case we are using a helper function to get us a list of entity reference fields on this plugin (details are in the patch and unimportant to this discussion). We then make those fields the list of Drupal fields for the settings form. The array we got from the parent class already provides a list of Salesforce fields in $pluginForm[‘salesforce_field’] so we don’t have to worry about that part. Since the salesforce_webform module is push-only on its mappings, this plugin was designed to be push only as well, and so limits to direction options to be push only. The default set of options is:

'#options' => [
    MappingConstants::SALESFORCE_MAPPING_DIRECTION_DRUPAL_SF => t('Drupal to SF'),
    MappingConstants::SALESFORCE_MAPPING_DIRECTION_SF_DRUPAL => t('SF to Drupal'),
    MappingConstants::SALESFORCE_MAPPING_DIRECTION_SYNC => t('Sync'),
 ],

And you can limit those anyway that makes sense for your plugin.

With the form array completed, we now move on to the value function. This is generally the most interesting part of the plugin since it does the work of actually setting the value returned by the mapping.

  /**
   * {@inheritdoc}
   */
  public function value(EntityInterface $entity, SalesforceMappingInterface $mapping) {
    $element_parts = explode('__', $this->config('drupal_field_value'));
    $main_element_name = reset($element_parts);
    $webform = $this->entityTypeManager->getStorage('webform')->load($mapping->get('drupal_bundle'));
    $webform_element = $webform->getElement($main_element_name);
    if (!$webform_element) {
      // This reference field does not exist.
      return;
    }
 
    try {
 
      $value = $entity->getElementData($main_element_name);
 
      $referenced_mappings = $this->mappedObjectStorage->loadByDrupal($webform_element['#target_type'], $value);
      if (!empty($referenced_mappings)) {
        $mapping = reset($referenced_mappings);
        return $mapping->sfid();
      }
    }
    catch (\Exception $e) {
      return NULL;
    }
  }

In this case we are finding the entity referred to in the webform submission, loading any mapping objects that may exist for that entity, and returning the Salesforce ID of the mapped object if it exists. Yours will likely need to do something very different.

There are actually two related functions defined by the plugin interface, defined in the base class, and available for override as needed for setting pull and push values independently:

  /**
   * An extension of ::value, ::pushValue does some basic type-checking and
   * validation against Salesforce field types to protect against basic data
   * errors.
   *
   * @param \Drupal\Core\Entity\EntityInterface $entity
   * @param \Drupal\salesforce_mapping\Entity\SalesforceMappingInterface $mapping
   *
   * @return mixed
   */
  public function pushValue(EntityInterface $entity, SalesforceMappingInterface $mapping);
 
  /**
   * An extension of ::value, ::pullValue does some basic type-checking and
   * validation against Drupal field types to protect against basic data
   * errors.
   *
   * @param \Drupal\salesforce\SObject $sf_object
   * @param \Drupal\Core\Entity\EntityInterface $entity
   * @param \Drupal\salesforce_mapping\Entity\SalesforceMappingInterface $mapping
   *
   * @return mixed
   */
  public function pullValue(SObject $sf_object, EntityInterface $entity, SalesforceMappingInterface $mapping);

But be careful overriding them directly. The base class provides some useful handling of various data types that need massaging between Drupal and Salesforce, you may lose that if you aren’t careful. I encourage you to look at the details of both pushValue and pullValue before working on those.

Okay, with the configuration and values handled, we just need to deal with programmatically telling Drupal when it can pull and push these fields. Most of the time you don’t need to do this, but you can simplify some of the processing by overriding pull() and push() to make sure the have the right response hard coded instead of derived from other sources. In this case pulling the field would be bad, so we block that:

  /**
   * {@inheritdoc}
   */
  public function pull() {
    return FALSE;
  }

Also, we only want this mapping to appear as an option if the site has the webform module enabled. Without it there is no point in offering it at all. The plugin interface provides a function called isAllowed() for this purpose:

  /**
   * {@inheritdoc}
   */
  public static function isAllowed(SalesforceMappingInterface $mapping) {
    return \Drupal::service('module_handler')->moduleExists('webform');
  }

You can also use that function to limit a field even more tightly based on the mapping itself.

To further ensure the configuration of this mapping entity defines its dependencies correctly we can define additional dependencies in getDependencies(). Again here we are tied to the Webform module and we should enforce that during and config exports:

  /**
   * {@inheritdoc}
   */
  public function getDependencies(SalesforceMappingInterface $mapping) {
    return ['module' => ['webform']];
  }

And that is about it. Once the class exists and is properly setup, all you need to do is rebuild the caches and you should see your new mapping field as an option on your Salesforce mapping objects (at least when isAllowed() is returning true).

In Praise of B Students

There is nothing wrong with just being good at something.

When I was in school I was a solid B student. I thought of myself as a B student and I worked hard to make sure I was always working at least a B average. I worked hard, but not too hard. Likely I could have been an A student if I’d worked harder, but I would not have been as happy. I didn’t feel the drive to be the best all the time, just good enough to keep up with what I was learning. Most people brag about being A students, but that wasn’t me and it has worked out pretty well in the long run.

For me a B student is someone who not only gets actually Bs most of the time in school but also knows how to work hard to get the grade they want, and how to relax so they recover and enjoy their lives along the way. If they fall a little short at some point they are perfectly capable of working a little harder to make up the ground. Life is not a sprint, it’s a marathon, we all have to save a bit for later. If you are used to running full out all the time, you do not always have the ability to step it up a little extra when needed nor how to relax and recover without stopping. In a world where we seem to be moving from helicopter parents to snowplow parents, there are still people that have fallen short from time to time, so they know how to pick themselves up and push forward.

There has been plenty written and said about the flaws of perfectionism so you don’t need to trust me, and I’m not going to waste time making the case that there are problems with pushing kids and college students to be a perfect A student all the time. And there are problems when companies only want to hire people who have those backgrounds.

In our current test-driven education culture there is a lot of push for kids to be the best. And the highly competitive nature of admissions for some top colleges can make it seem like a kid has to be perfect to succeed in life. But life isn’t Top Gun there actually are points for second place, and really more or less all the others (well paychecks and some level of happiness anyway). More importantly people who are used to a bit of failure can excel when faced with tough jobs that require a lot of on the job learning and trial and error work. That describes pretty much every modern office job, and lots of others – watch a plumber work on an old house and you’ll see plenty of trial and error being required to solve even basic problems.

Failure is a part of life. We all need to know how to acknowledge we made mistakes, how to recover, and how to ensure we don’t make the same mistake again in the future. People who earned a lot of B’s in school are familiar with all those things. We sometimes got lazy and ended up with a C instead of B – to stay a B student we then worked harder and got ourselves enough A’s to offset our missteps and bring our averages back up. Sometimes we really didn’t understand something and had to struggle through to make sure we stayed on pace.

B students also often know the difference between learning and getting the right answer on an assessment (a skill I would have told you I had but did not need anymore until I starting having to pass professional certification exams on a regular basis). We learned how to pass the test without knowing all the information, but knowing enough to get the grade we wanted. Getting a perfect score on an assessment is only useful if the assessment is perfect – most college professors and anyone who has taken a certification exam can tell you there are no perfect assessments. I had to pass tests in college, and I need to complete and maintain certifications now, but I don’t need to have every piece of information crammed into my brain all the time – that’s what Google is for. Once I pass, I’ve passed – it feels better to get more right answers, but as long as I get enough to pass I’m certified.

When I was in the 6th grade we had a whole day dedicated to studies skills taught by a guy named Mr. Gallagher. He wasn’t a regular teacher at my school, and I have no idea where they found him or what happened to him later, but some of what he taught us that day has stuck with me both in terms of skills and philosophy. He emphasized the need to understand how a test was written, both in terms of following directions so you didn’t give up silly points and so you know where to focus your time (hint: time goes first into the places you are likely to earn the most points), and when to guess and when to work through an answer carefully (again be greedy about points not the number of right answers).

Mr. Gallagher was the first person to put the idea into my head that the point of going to school is to get good grades instead of learning. I can still remember our teachers pushing back on that argument the next day, and I eventually arrived at the conclusion that they were both right: the purpose of school is to learn, but the purpose of tests is to get good enough grades so you can move forward in school. The thought exercise itself helped me see that learning material and passing a test weren’t actually always related. I had a teacher later who helped me further abstract this concept when he pointed out that he could easily write tests we all passed or all failed, so the purpose of the test writing was to find a measure of what he thought we should be able to do (that teacher and his wife were recently on Marketplace – thanks Bill that simple lesson was super useful in college and beyond). If pass and fail are at the whim of the test author, and they aren’t actually playing against me personally but really tracking a group of people, then I can play it as a game where I need to get myself into the top 80%. If I learn to play at that level reliably I’m going to do pretty well and I don’t have to stress being perfect all the time.

Yes, there are things in life that you cannot do unless you are one of the best. Not to mention the fact that the playing field isn’t level, which means some people have to work a whole lot harder than others to earn the same B and a report card full of Bs doesn’t get read the same for everyone. But we cannot all be the best, and pretending that’s not true doesn’t level the social playing fields.

And of course there are plenty of moments we need to strive to be as close to perfect as we are capable. But we also have to hit deadlines and get things done even if we know the work will be less-than perfect. Anyone who works in software development has to deal with this on a regular basis. As much as we may want to be perfect, shifting project requirements, project timelines, platform weakness, and our own mistakes all get in the way. We do the best we can with the skills, time, teams, and the tools we have. We need to know how to accept that we cannot fix every flaw and still get things done well enough to ensure the project is successful.

So when you are hiring people consider looking for people who already know how to be second best. People who know how to do really good work, but also know how to take risks, try something new, and in the process fail. People who know how to get things done well enough to be successful, even if it’s not perfect. We aren’t perfect, our work isn’t perfect, our world isn’t perfect, so don’t pretend your team has to be perfect.

Authors Note: Last month I failed to post to this blog for the first time. If you look back over the record you will see at least one post each month from the time the blog starts through August 2019 – then I missed September. Oops. I could make all kinds of excuses (I really did get sick right at the end of the month so my planned post was delayed a week), but the truth is I had time and material, I just didn’t get anything done. So now to make up for it I am trying to post every weekend this month. I missed a goal by a little, so now I compensate to make up for it – then I’ll hopefully go back to my nice reliable routine. What can I say? I’m okay with a B.

Arts in the Heart 2019

This year’s Arts in the Heart of Augusta was not hurricane plagued the way last year’s was, and was a great weekend of good food, performances, and generally great chances to take pictures.

Each year opens with a parade to celebrate the diverse set of communities that have settled in and around Augusta.

Image shows a collection of women of Chinese decent with green t-shirts with drum sticks raised as they perform on a large outdoor stage. — Drummers from the Chinese community welcomed everyone to the annual opening parade.

A collection of people ranging in age from small children to older adults, in ethnic dress from Greece, Jamaica, Latin America, and other parts of the world. The Greek flag is prominent in the foreground. — Drummers from the Chinese community welcomed everyone to the annual opening parade.

The whole event is built around artists selling their work. They manage to pull in people from a fairly large area, but also make space for young local artists to try to get started selling their work.

A young local artist sitting with her paintings and drawings.

A teenage male photographer wearing a Nirvana t-shirt and looking defiant stands by a table with his pictures. A teenage female painter stands by her own table displaying her work.

For us, and for many others, the variety of ethic foods available for purchase is a large part of the draw.

A crowd of people walking passed a series of white tents labeled with the names of countries like Sudan and Lebanon, where they are selling ethnic foods from those countries.

A man wearing a tarboush (Lebanese hat) cutting chicken from a shawarma for sale at the Lebanon food tent.

It’s also a great event for just plain people watching. Between watching the various performances, enjoying the vendors, and the many activities, there are lots of chances to watch people having a good time with their friends and family.

A family walking through the crowd. Including the father pushing a little girl in her stroller – she is wearing a fairy costume. A young teenage boy dressed as Link from Legend of Zelda, and follwed by the mother wearing another fairy costume, hers with wings.

A family waiting in the maian green, the little girls are wearing costumes from a local dance troop, as they wait their turn on the main stage.

Mixed in with the vendors, food, family activities, stages, and other goes on are always a mix of street performers.

A boy standing on the side of the sidewalk playing an accordion, the case is open so people can leave tips. — He said he had picked her up at the rescue that morning. Because where else to you take your new parrot?

A middle-aged magician smiles at a little girl who is petting his live rabbit sitting on a table. — He said he had picked her up at the rescue that morning. Because where else to you take your new parrot?

There are several stages setup throughout the event, spanning several city blocks. The performances range from local signers to dance troops, street performers

An African American woman singing on an indoor stage. She is clearly comfortable with the microphone and is smiling as she sings.

A troop of African American girls dancing on the main stage, their hair is flying as they move with energy.

Until next year…

Two people walking away from the camera as the sun is setting. A few other people milling about. The woman is wearing a traditional Mexican dress and hat.

SC DUG August 2019

After a couple months off, SC DUG met this month with a presentation on super cheap Drupal hosting.

Chris Zietlow from Mindgrub, Will Jackson from Kanopi Studios, and I all gave short talks very cheap ways to host Drupal 8.

Chris opened by talking about using AWS Micro servers. Will shared a solution using a Raspberry Pi for a fully wireless server. I closed the discussion with a review of using Drupal Tome on Netlify.

We all worked from a loose set of rules to help keep us honest and prevent overlapping:

Rules for Cheap D8 Hosting Challenge

The goal is to figure out the cheapest D8 hosting that would actually function for a project, even if it is deeply irresponsible to actually use.

Rules

It has to actually work for D8 (so modern PHP version, working database, etc),
You do not actually have to spend the money, but you do need to know all the steps required to make it work.
It needs to honor the TOS for any networks and services you use (no illegal network taps – legal hidden taps are fair game).
You have to share your idea with the other players so we don’t have two people propose the same solution (first-come-first-serve on ideas).

Reporting

Be prepared to talk for about 5 minutes on how your solution would work. Your talk needs to include:

Estimated Monthly cost for the first year.
Steps required to make it work.
Known weaknesses.

If you have a super cheap hosting solution for Drupal 8 we’d love to hear about it.

Drupal Tome + Docksal + Netlify

Drupal Tome is a static site generator distribution of Drupal 8. It provides mechanisms for taking an entire Drupal site and exporting all the content to HTML for direct service. As part of a recent competition at SCDUG to come up with the cheapest possible Drupal 8 hosting, I decided to do a proof-of-concept level implementation of Drupal 8 with Docksal for local content editing, and Netlify for hosting (total cost was just the domain registration).

The Tome project has directions for setup with Docker, and for setup with Netlify, but they don’t quite line up with each other (I followed the docker instructions, then the Netlify set, but had to chart my own course to get the site from the first project linked to the repo in the second), and since I’m getting used to using Docksal when I had to fall back and do a bit of it myself I realized it was almost painfully easy to setup.

The first step was to go to the Tome documentation for Netlify and setup an account, and site from the template. There is a button in those directions to trigger the Netlify setup, I’ve added one here as well (but if this one fails, check to see if they updated theirs):

Follow Netlify’s directions for setting up DNS so you can have the domain you want, and HTTPS (through Let’s Encrypt). It took it a couple hours to get that detail to run right, but it eventually worked. For this project I chose a subdomain of my main blog domain: tome-netlify.spinningcode.org

Next go to Github (or whatever service you used) and clone the repository to your local machine. There is a generated README on that project, but the directions aren’t 100% correct if you aren’t cloning onto a machine with a working PHP environment. This is when I switched over to docksal, and ran the following series of commands:

fin init
fin composer install
fin drush tome:install
fin drush uli

Then log into your local site using the domain from docksal and the link from drush, and add some content.

Next we export the content from Drupal to send over to Netlify for deployment.

fin drush tome:static
git add .
git commit -m "Adding sample content"
git push

…now we wait while Netlify notices and builds the site…

If you look at the site a few minutes later the new content should be posted.

This is all well and good if I want to use the version of the site generated for the Netlify example, but I wanted to make sure I could do something more interesting. These days Drupal ships with an install profile called Unami that provides a more robust sample site than the more traditional Standard install.

So now let’s try to get Unami onto this site. Go back to the terminal and have Tome reset everything (it’ll warn you that you are about to nuke everything):

fin drush tome:init

…select Unami when it asks for a profile…and wait cause this takes a while…

Now just re-export the content and push it to your repo.

fin drush tome:static
git add .
git commit -m "Converting to Unami"
git push

And wait again, cause this also takes a few minutes…

The Unami home page on my subdomain hosted at Netlify.

That really was all that was involved for a simple site, you can see my repository on Github if you want to see all of what was generated along the way.

The whole process is pretty straight forward, but there are a few things that it helps to understand.

First, Netlify is actually regenerating the markup on their servers with this approach. The Drupal nodes, and other entities, are saved as JSON and then imported during the build. This makes the process reliable, but slow. Unami takes several minutes to deploy since Netlify is installing and configuring Drupal, loading the content, and generating the output. The build command provided in that template is clear enough to follow if you are familiar with composer projects:

command = "composer install && ./vendor/bin/drush tome:install -y && ./vendor/bin/drush tome:static -l $DEPLOY_PRIME_URL"

One upside of this, is that you can use a totally unrelated domain for your local testing and have it adjust correctly to the production domain. When you are using Netlify’s branching workflow for managing dev, test, and production it also protects your work that way.

My directions above load a standard docksal container because that’s quick and easy, which includes MySQL, but Tome falls back to using a Sqlite database since you can be more confident it is there. Again this is reliable but slow. If I were going to do this on a more complete project I’d want a smaller Docksal setup or to switch to using MySQL locally.

A workflow based on this approach might also struggle with concurrent edits or complex configuration of large sites. It would probably make more sense to have the content created on a hidden, but traditional, server and then run through a different workflow. But for someone working on a series small sites that are rarely updated, a totally temporary instance of the site that can be rapidly deployed to a device, have content updated, push out to production, and then deleted locally until needed again.

The final detail to note is that there is no support for forms built into this solution. Netlify has support for that, and Tome has a module that claim to connect to that service but I wasn’t able to quickly determine how to get it connected. I am confident there are solves to this problem, but it is something that would take a little additional work.

Bypass Pantheon Timeouts for Drupal 8

Pantheon is an excellent hosting service for both Drupal and WordPress sites. But to make their platform work and scale well they have a number of limits built into the platform. These include process time limits and memory limits. While they are large enough for the majority of projects, large projects can have trouble.

For data loading their official answer is typically to copy the database to another server, run your job there, and copy the database back onto their server. That’s fine if you can afford to freeze updates to your production site. And have the time to setup a process to mirror changes into your temporary copy. And can afford some additional project overhead. But sometimes those things are not an option. Or the data load takes too long, or happens too often, for that to be practical on a regular basis.

I recently needed to do a very large import of records into Drupal on a Pantheon hosted site. The minimize user impact I started to play around with solutions that would allow me to ignore those time limits. We were looking at dong about 50 million data writes for the project. When I first estimated the process the running time was over a week.

The Outline

Since Drupal’s batch system was created to solve this exact problem it seemed like a good place to start. For this solution you need a file you can load and parse in segments, like a CSV file, which you can read one line at a time. It does not have to represent the final state of your data. While you can actually load the data raw, you can also load each record into a table or a queue to process later.

One quick note about the code samples, I wrote these based on the service-based approach outlined in my post about batch services and the batch service module I discussed there. It could be adapted to a traditional batch job, but I like the clarity the wrapper provides for this discussion.

The general concept here is that we upload the file and then progressively process it from within a batch job. My code samples below provide two classes to achieve this. The first is a form that provides a managed file field which create a file entity that can be reliably passed to the batch processor. From there the batch service uses a bit of basic PHP file handling to copy data into the database. If you need to do more than load the data into the database directly (say create complex entities or other tasks) you can set up a second phase to run through the values to do that heavier lifting.

Load your file

To get us started the form includes this managed file:

   $form['file'] = [
     '#type' => 'managed_file',
     '#name' => 'data_file',
     '#title' => $this->t('Data file'),
     '#description' => $this->t('CSV format for this example.'),
     '#upload_location' => 'private://example_pantheon_loader_data/',
     '#upload_validators' => [
       'file_validate_extensions' => ['csv'],
     ],
   ];

The managed file form element automagically gives you a file entity. The value in the form state is the id of that entity. This file will be temporary and have no references once the process is complete and so depending on your site setup the file will eventually be purged. Which adds up to mean we can pass all the values straight through to our batch processor:

$batch = $this->dataLoaderBatchService->generateBatchJob($form_state->getValues());

If the data file is small, a few thousand rows at most, you can load it right away. But that runs into both time and memory concerns and the whole point of this is to avoid those. With my approach we can ignore those and we’re only limited by Pantheon’s upload file size. If the file size is too large for that you can upload the file via sftp and read from the file system, so you have options.

As we setup the file for processing in the batch job, we really need the file path not the ID. The main reason to use the managed file is they can reliably get the file path on a Pantheon server without us needing to know where they have things stashed. Since we’re about to use generic PHP functions for file processing we need to know that path reliably:

$fid = array_pop($data['file']);
$fileEntity = File::load($fid);
$ops = [];

if (empty($fileEntity)) {
  $this->logger->error('Unable to load file data for processing.');
  return [];
}
$filePath = $this->fileSystem->realpath($fileEntity->getFileUri());
$ops = ['processData' => [$filePath]];

Create your batch

Now we have a file know where it is. Since it’s a csv we can load a few rows at time, process them, and then loop back.

Our batch processing function needs to track two things in addition to the file: the header values and the current file position. So in the first pass we initialize the position to zero and then load the first row as the header. For every pass after that we need to find point we left off. For this we use generic PHP files for loading and seeking the current location:

// Old-school file handling.
$path = array_pop($data);
$file = fopen($path, "r");
...
fseek($file, $filePos);

// Each pass we process 100 lines, if you have to do something complex
// you might want to reduce the run.
for ($i = 0; $i < 100; $i++) {
  $row = fgetcsv($file);
  if (!empty($row)) {
    $data = array_combine($header, $row);
    $member['timestamp'] = time();
    $rowData = [
             'col_one' => $data['field_name'],
             'data' => serialize($data),
             'timestamp' => time(),
    ];
    $row_id = $this->database->insert('example_pantheon_loader_tracker')
             ->fields($rowData)
             ->execute();

    // If you're setting up for a queue you include something like this.
    // $queue = $this->queueFactory->get(‘example_pantheon_loader_remap’);
    // $queue->createItem($row_id);
 }
 else {
   break;
 }
}
$filePos = (float) ftell($file);
$context['finished'] = $filePos / filesize($path);

The example code just dumps this all into a database table. This can be useful as a raw data loader. If you need to add a large data set to an existing site that’s used for reference data or something similar. It can also be used as the base to create more complex objects. The example code includes comments about generating a queue worker to run on cron or as another batch job. the Queue UI module provides a simple interface to run those on a batch job.

Final Considerations

I’ve run this process for several hours at a stretch. Pantheon does have issues with systems errors if left to run a batch job for extreme runs. I ran into problems on some runs after 6-8 hours of run time. So a prep into the database followed by running on queue restart has been more reliable.

Docksal Pantheon Setup from Scratch

I recently had reason to switch over to using Docksal for a project, and on the whole I really like it as a good easy solution for getting a project specific Drupal dev environment up and running quickly. But like many dev tools the docs I found didn’t quite cover what I wanted because they made a bunch of assumptions.

Most assumed either I was starting a generic project or that I was starting a Pantheon specific project – and that I already had Docksal experience. In my case I was looking for a quick emergency replacement environment for a long-running Pantheon project.

Fairly recently Docksal added support for a project init command that helps setup for Acquia, Pantheon, and Pantheon.sh, but pull init isn’t really well documented and requires a few preconditions.

Since I had to run a dozen Google searches, and ask several friends for help, to make it work I figured I’d write it up.

Install Docksal

First follow the basic Docksal installation instructions for your host operating system. Once that completes, if you are using Linux as the host OS log out and log back in (it just added your user to a group and you need that access to start up docker).

Add Pantheon Machine Token

Next you need to have a Pantheon machine token so that terminus can run within the new container you’re about to create. If you don’t have one already follow Pantheon’s instructions to create one and save if someplace safe (like your password manager).

Once you have a machine token you need to tell Docksal about it. There are instructions for that (but they aren’t in the instructions for setting up Docksal with pull init) basically you add the key to your docksal.env file:

SECRET_TERMINUS_TOKEN="HASH_VALUE_PROVIDED_BY_PANTHEON_HERE"

Also if you are using Linux you should note that those instructions linked above say the file goes in $HOME/docksal/docksal.env, but you really want $HOME/.docksal/docksal.env (note the dot in front of docksal to hide the directory).

Setup SSH Key

With the machine token in place you are almost ready to run the setup command, just one more precondition. If you haven’t been using Docker or Docksal they don’t know about your SSH key yet, and pull init assumes it’s around. So you need to tell Docksal to load it but running:
fin ssh-key add

If the whole setup is new, you may also need to create your key and add it to Pantheon. Once you have done that, if you are using a default SSH key name and location it should pick it up automatically (I have not tried this yet on Windows so mileage there may vary – if you know the answer please leave me a comment). It also is a good idea to make sure the key itself is working right but getting the git clone command from your Pantheon dashboard and trying a manual clone on the command line (delete once it’s done, this is just to prove you can get through).

Run Pull Init

Now finally you are ready to run fin pull init:

fin pull init --hostingplatform=pantheon --hostingsite=[site-machine-name] --hosting-env=[environment-name]

Docksal will now setup the site, maybe ask you a couple questions, and clone the repo. It will leave a couple things out you may need: database setup, and .htaccess.

Add .htaccess as needed

Pantheon uses nginx. Docksal’s formula uses Apache. If you don’t keep a .htaccess file in your project (and while there is not reason not to, some Pantheon setups don’t keep anything extra stuff around) you need to put it back. If you don’t have a copy handy, copy and paste the content from the Drupal project repo: https://git.drupalcode.org/project/drupal/blob/8.8.x/.htaccess

Finally, you need to tell Drupal where to find the Docksal copy of the database. For that you need a settings.local.php file. Your project likely has a default version of this, which may contain things you may or may not want so adjust as needed. Docksal creates a default database (named default) and provides a user named…“user”, which has a password of “user”. The host’s name is ‘db’. So into your settings.local.php file you need to include database settings at the very least:

<?php
$databases = array(
  'default' =>
    array(
      'default' =>
      array(
        'database' => 'default',
        'username' => 'user',
        'password' => 'user',
        'host' => 'db',
        'port' => '',
        'driver' => 'mysql',
        'prefix' => '',
      ),
    ),
);

With the database now fully linked up to Drupal, you can now ask Docksal to pull down a copy of the database and a copy of the site files:

fin pull db

fin pull files

In the future you can also pull down code changes:

fin pull code

Bonus points: do this on a server.

On occasion it’s useful to have all this setup on a remote server not just a local machine. There are a few more steps to go to do that safely.

First you may want to enable Basic HTTP Auth just to keep away from the prying eyes of Googlebot and friends. There are directions for that step (you’ll want the Apache instructions). Next you need to make sure that Docksal is actually listing to the host’s requests and that they are forwarded into the containers. Lots of blog posts say DOCKSAL_VHOST_PROXY_IP=0.0.0.0 fin reset proxy. But it turns out that fin reset proxy has been removed, instead you want:

DOCKSAL_VHOST_PROXY_IP=0.0.0.0 fin system reset.

Next you need to add the vhost to the docksal.env file we were working with earlier:

 VIRTUAL_HOST="test.example.org"

Run fin up to get Docksal to pick up the changes (this section is based on these old instructions).

Now you need to add either a DNS entry someplace, or update your machine’s /etc/hosts file to look in the right place (the public IP address of the host machine).

Anything I missed?

If you think I missed anything feel free to let know. Particularly Windows users feel free to let me know changes related to doing things there. I’ll try to work those in if I don’t get to figuring that out on my own in the near future.

SC DUG May 2019

For this month’s SC DUG, Mauricio Orozco from the South Carolina Commission for Minority Affairs shared his notes and lessons learned during his first DrupalCon North America.

We frequently use these presentations to practice new presentations, try out heavily revised versions, and test out new ideas with a friendly audience. If you want to see a polished version checkout our group members’ talks at camps and cons. So if some of the content of these videos seems a bit rough please understand we are all learning all the time and we are open to constructive feedback.

If you would like to join us please check out our up coming events on Meetup for meeting times, locations, and connection information.