Drupal Tome is a static site generator distribution of Drupal 8. It provides mechanisms for taking an entire Drupal site and exporting all the content to HTML for direct service. As part of a recent competition at SCDUG to come up with the cheapest possible Drupal 8 hosting, I decided to do a proof-of-concept level implementation of Drupal 8 with Docksal for local content editing, and Netlify for hosting (total cost was just the domain registration).
The Tome project has directions for setup with Docker, and for setup with Netlify, but they don’t quite line up with each other (I followed the docker instructions, then the Netlify set, but had to chart my own course to get the site from the first project linked to the repo in the second), and since I’m getting used to using Docksal when I had to fall back and do a bit of it myself I realized it was almost painfully easy to setup.
The first step was to go to the Tome documentation for Netlify and setup an account, and site from the template. There is a button in those directions to trigger the Netlify setup, I’ve added one here as well (but if this one fails, check to see if they updated theirs):
Login with Github or similar service, and let it create a repo for your project.
Follow Netlify’s directions for setting up DNS so you can have the domain you want, and HTTPS (through Let’s Encrypt). It took it a couple hours to get that detail to run right, but it eventually worked. For this project I chose a subdomain of my main blog domain: tome-netlify.spinningcode.org
Next go to Github (or whatever service you used) and clone the repository to your local machine. There is a generated README on that project, but the directions aren’t 100% correct if you aren’t cloning onto a machine with a working PHP environment. This is when I switched over to docksal, and ran the following series of commands:
fin init
fin composer install
fin drush tome:install
fin drush uli
Then log into your local site using the domain from docksal and the link from drush, and add some content.
Next we export the content from Drupal to send over to Netlify for deployment.
…now we wait while Netlify notices and builds the site…
If you look at the site a few minutes later the new content should be posted.
This is all well and good if I want to use the version of the site generated for the Netlify example, but I wanted to make sure I could do something more interesting. These days Drupal ships with an install profile called Unami that provides a more robust sample site than the more traditional Standard install.
So now let’s try to get Unami onto this site. Go back to the terminal and have Tome reset everything (it’ll warn you that you are about to nuke everything):
fin drush tome:init
…select Unami when it asks for a profile…and wait cause this takes a while…
Now just re-export the content and push it to your repo.
fin drush tome:static
git add .
git commit -m "Converting to Unami"
git push
And wait again, cause this also takes a few minutes…
The Unami home page on my subdomain hosted at Netlify.
That really was all that was involved for a simple site, you can see my repository on Github if you want to see all of what was generated along the way.
The whole process is pretty straight forward, but there are a few things that it helps to understand.
First, Netlify is actually regenerating the markup on their servers with this approach. The Drupal nodes, and other entities, are saved as JSON and then imported during the build. This makes the process reliable, but slow. Unami takes several minutes to deploy since Netlify is installing and configuring Drupal, loading the content, and generating the output. The build command provided in that template is clear enough to follow if you are familiar with composer projects:
One upside of this, is that you can use a totally unrelated domain for your local testing and have it adjust correctly to the production domain. When you are using Netlify’s branching workflow for managing dev, test, and production it also protects your work that way.
My directions above load a standard docksal container because that’s quick and easy, which includes MySQL, but Tome falls back to using a Sqlite database since you can be more confident it is there. Again this is reliable but slow. If I were going to do this on a more complete project I’d want a smaller Docksal setup or to switch to using MySQL locally.
A workflow based on this approach might also struggle with concurrent edits or complex configuration of large sites. It would probably make more sense to have the content created on a hidden, but traditional, server and then run through a different workflow. But for someone working on a series small sites that are rarely updated, a totally temporary instance of the site that can be rapidly deployed to a device, have content updated, push out to production, and then deleted locally until needed again.
The final detail to note is that there is no support for forms built into this solution. Netlify has support for that, and Tome has a module that claim to connect to that service but I wasn’t able to quickly determine how to get it connected. I am confident there are solves to this problem, but it is something that would take a little additional work.
Pantheon is an excellent hosting service for both Drupal and WordPress sites. But to make their platform work and scale well they have a number of limits built into the platform. These include process time limits and memory limits. While they are large enough for the majority of projects, large projects can have trouble.
For data loading their official answer is typically to copy the database to another server, run your job there, and copy the database back onto their server. That’s fine if you can afford to freeze updates to your production site. And have the time to setup a process to mirror changes into your temporary copy. And can afford some additional project overhead. But sometimes those things are not an option. Or the data load takes too long, or happens too often, for that to be practical on a regular basis.
I recently needed to do a very large import of records into Drupal on a Pantheon hosted site. The minimize user impact I started to play around with solutions that would allow me to ignore those time limits. We were looking at dong about 50 million data writes for the project. When I first estimated the process the running time was over a week.
The Outline
Since Drupal’s batch system was created to solve this exact problem it seemed like a good place to start. For this solution you need a file you can load and parse in segments, like a CSV file, which you can read one line at a time. It does not have to represent the final state of your data. While you can actually load the data raw, you can also load each record into a table or a queue to process later.
One quick note about the code samples, I wrote these based on the service-based approach outlined in my post about batch services and the batch service module I discussed there. It could be adapted to a traditional batch job, but I like the clarity the wrapper provides for this discussion.
The general concept here is that we upload the file and then progressively process it from within a batch job. My code samples below provide two classes to achieve this. The first is a form that provides a managed file field which create a file entity that can be reliably passed to the batch processor. From there the batch service uses a bit of basic PHP file handling to copy data into the database. If you need to do more than load the data into the database directly (say create complex entities or other tasks) you can set up a second phase to run through the values to do that heavier lifting.
Load your file
To get us started the form includes this managed file:
The managed file form element automagically gives you a file entity. The value in the form state is the id of that entity. This file will be temporary and have no references once the process is complete and so depending on your site setup the file will eventually be purged. Which adds up to mean we can pass all the values straight through to our batch processor:
If the data file is small, a few thousand rows at most, you can load it right away. But that runs into both time and memory concerns and the whole point of this is to avoid those. With my approach we can ignore those and we’re only limited by Pantheon’s upload file size. If the file size is too large for that you can upload the file via sftp and read from the file system, so you have options.
As we setup the file for processing in the batch job, we really need the file path not the ID. The main reason to use the managed file is they can reliably get the file path on a Pantheon server without us needing to know where they have things stashed. Since we’re about to use generic PHP functions for file processing we need to know that path reliably:
$fid = array_pop($data['file']);
$fileEntity = File::load($fid);
$ops = [];
if (empty($fileEntity)) {
$this->logger->error('Unable to load file data for processing.');
return [];
}
$filePath = $this->fileSystem->realpath($fileEntity->getFileUri());
$ops = ['processData' => [$filePath]];
Create your batch
Now we have a file know where it is. Since it’s a csv we can load a few rows at time, process them, and then loop back.
Our batch processing function needs to track two things in addition to the file: the header values and the current file position. So in the first pass we initialize the position to zero and then load the first row as the header. For every pass after that we need to find point we left off. For this we use generic PHP files for loading and seeking the current location:
// Old-school file handling.
$path = array_pop($data);
$file = fopen($path, "r");
...
fseek($file, $filePos);
// Each pass we process 100 lines, if you have to do something complex
// you might want to reduce the run.
for ($i = 0; $i < 100; $i++) {
$row = fgetcsv($file);
if (!empty($row)) {
$data = array_combine($header, $row);
$member['timestamp'] = time();
$rowData = [
'col_one' => $data['field_name'],
'data' => serialize($data),
'timestamp' => time(),
];
$row_id = $this->database->insert('example_pantheon_loader_tracker')
->fields($rowData)
->execute();
// If you're setting up for a queue you include something like this.
// $queue = $this->queueFactory->get(‘example_pantheon_loader_remap’);
// $queue->createItem($row_id);
}
else {
break;
}
}
$filePos = (float) ftell($file);
$context['finished'] = $filePos / filesize($path);
The example code just dumps this all into a database table. This can be useful as a raw data loader. If you need to add a large data set to an existing site that’s used for reference data or something similar. It can also be used as the base to create more complex objects. The example code includes comments about generating a queue worker to run on cron or as another batch job. the Queue UI module provides a simple interface to run those on a batch job.
Final Considerations
I’ve run this process for several hours at a stretch. Pantheon does have issues with systems errors if left to run a batch job for extreme runs. I ran into problems on some runs after 6-8 hours of run time. So a prep into the database followed by running on queue restart has been more reliable.
I recently had reason to switch over to using Docksal for a project, and on the whole I really like it as a good easy solution for getting a project specific Drupal dev environment up and running quickly. But like many dev tools the docs I found didn’t quite cover what I wanted because they made a bunch of assumptions.
Most assumed either I was starting a generic project or that I was starting a Pantheon specific project – and that I already had Docksal experience. In my case I was looking for a quick emergency replacement environment for a long-running Pantheon project.
Fairly recently Docksal added support for a project init command that helps setup for Acquia, Pantheon, and Pantheon.sh, but pull init isn’t really well documented and requires a few preconditions.
Since I had to run a dozen Google searches, and ask several friends for help, to make it work I figured I’d write it up.
Install Docksal
First follow the basic Docksal installation instructions for your host operating system. Once that completes, if you are using Linux as the host OS log out and log back in (it just added your user to a group and you need that access to start up docker).
Add Pantheon Machine Token
Next you need to have a Pantheon machine token so that terminus can run within the new container you’re about to create. If you don’t have one already follow Pantheon’s instructions to create one and save if someplace safe (like your password manager).
Once you have a machine token you need to tell Docksal about it. There are instructions for that (but they aren’t in the instructions for setting up Docksal with pull init) basically you add the key to your docksal.env file:
Also if you are using Linux you should note that those instructions linked above say the file goes in $HOME/docksal/docksal.env, but you really want $HOME/.docksal/docksal.env (note the dot in front of docksal to hide the directory).
Setup SSH Key
With the machine token in place you are almost ready to run the setup command, just one more precondition. If you haven’t been using Docker or Docksal they don’t know about your SSH key yet, and pull init assumes it’s around. So you need to tell Docksal to load it but running: fin ssh-key add
If the whole setup is new, you may also need to create your key and add it to Pantheon. Once you have done that, if you are using a default SSH key name and location it should pick it up automatically (I have not tried this yet on Windows so mileage there may vary – if you know the answer please leave me a comment). It also is a good idea to make sure the key itself is working right but getting the git clone command from your Pantheon dashboard and trying a manual clone on the command line (delete once it’s done, this is just to prove you can get through).
Run Pull Init
Now finally you are ready to run fin pull init:
fin pull init --hostingplatform=pantheon --hostingsite=[site-machine-name] --hosting-env=[environment-name]
Docksal will now setup the site, maybe ask you a couple questions, and clone the repo. It will leave a couple things out you may need: database setup, and .htaccess.
Add .htaccess as needed
Pantheon uses nginx. Docksal’s formula uses Apache. If you don’t keep a .htaccess file in your project (and while there is not reason not to, some Pantheon setups don’t keep anything extra stuff around) you need to put it back. If you don’t have a copy handy, copy and paste the content from the Drupal project repo: https://git.drupalcode.org/project/drupal/blob/8.8.x/.htaccess
Finally, you need to tell Drupal where to find the Docksal copy of the database. For that you need a settings.local.php file. Your project likely has a default version of this, which may contain things you may or may not want so adjust as needed. Docksal creates a default database (named default) and provides a user named…“user”, which has a password of “user”. The host’s name is ‘db’. So into your settings.local.php file you need to include database settings at the very least:
With the database now fully linked up to Drupal, you can now ask Docksal to pull down a copy of the database and a copy of the site files:
fin pull db
fin pull files
In the future you can also pull down code changes:
fin pull code
Bonus points: do this on a server.
On occasion it’s useful to have all this setup on a remote server not just a local machine. There are a few more steps to go to do that safely.
First you may want to enable Basic HTTP Auth just to keep away from the prying eyes of Googlebot and friends. There are directions for that step (you’ll want the Apache instructions). Next you need to make sure that Docksal is actually listing to the host’s requests and that they are forwarded into the containers. Lots of blog posts say DOCKSAL_VHOST_PROXY_IP=0.0.0.0 fin reset proxy. But it turns out that fin reset proxy has been removed, instead you want:
DOCKSAL_VHOST_PROXY_IP=0.0.0.0 fin system reset.
Next you need to add the vhost to the docksal.env file we were working with earlier:
Now you need to add either a DNS entry someplace, or update your machine’s /etc/hosts file to look in the right place (the public IP address of the host machine).
Anything I missed?
If you think I missed anything feel free to let know. Particularly Windows users feel free to let me know changes related to doing things there. I’ll try to work those in if I don’t get to figuring that out on my own in the near future.
We frequently use these presentations to practice new presentations, try out heavily revised versions, and test out new ideas with a friendly audience. If you want to see a polished version checkout our group members’ talks at camps and cons. So if some of the content of these videos seems a bit rough please understand we are all learning all the time and we are open to constructive feedback.
If you would like to join us please check out our up coming events on Meetup for meeting times, locations, and connection information.
For the SC DUG meeting this month Will Jackson from Kanopi Studios gave a talk about using Docksal for local Drupal development. Will has the joy of working with some of the Docksal developers and has become an advocate for the simplicity and power Docksal provides.
We frequently use these presentations to practice new presentations, try out heavily revised versions, and test out new ideas with a friendly audience. If you want to see a polished version checkout our group members’ talks at camps and cons. So if some of the content of these videos seems a bit rough please understand we are all learning all the time and we are open to constructive feedback.
If you would like to join us please check out our up coming events on Meetup for meeting times, locations, and connection information.
Kaylan Wagner – Real world lessons from online games.
This fall the South Carolina Drupal User’s Group started using Zoom are part of all our meetings. Sometimes the technology has worked better than others, but when it works in our favor we are recording the presentations and sharing them when we can.
In November Kaylan Wagner gave a draft talk on using experiences in the world of online gaming to be a better remote team member.
We frequently use these presentations to practice new presentations and test out new ideas. If you want to see a polished version hunt group members out at camps and cons. So if some of the content of these videos seems a bit rough please understand we are all learning all the time and we are open to constructive feedback.
If you would like to join us please check out our up coming events on Meetup for meeting times, locations, and connection information.
Chis Zietlow – Using Machine Learning to Improve UX
This fall the South Carolina Drupal User’s Group started using Zoom are part of all our meetings. Sometimes the technology has worked better than others, but when it works in our favor we are recording the presentations and sharing them when we can.
Chris Zietlow presented back in September about using Machine Learning to Improve UX.
We frequently use these presentations to practice new presentations and test out new ideas. If you want to see a polished version hunt group members out at camps and cons. So if some of the content of these videos seems a bit rough please understand we are all learning all the time and we are open to constructive feedback.
If you would like to join us please check out our up coming events on Meetup for meeting times, locations, and connection information.
For this month’s South Carolina Drupal User Group I gave a talk about creating Batch Services in Drupal 8. As a quick side note we are trying to include video conference access to all our meetings so please feel free to join us even if you cannot come in person.
Since Drupal 8 was first released I have been frustrated by the fact that Drupal 8 batch jobs were basically untouched from previous versions. There is nothing strictly wrong with that approach, but it has never felt right to me particularly when doing things in a batch job that I might also want to do in another context – that really should be a service and I should write those core jobs first. After several frustrating experiences trying to find a solution I like, I finally created a module that provides an abstract class that can be used to create a service that handles this problem just more elegantly. The project also includes an example module to provide a sample service.
Some of the text in the slides got cut off by the Zoom video window, so I uploaded them to SlideShare as well:
To define a batch you generate an array in a particular format – typically as part of a form submit process – and pass that array to batch_set(). The array defines some basic messages, a list of operations, a function to call when the batch is finished, and optionally a few other details. The minimal array would be something like:
The interesting part should be in that operations array, which is a list of tasks to be run, but getting all your functions setup and the batch array generated can often be its own project.
Each operation is a function that implements callback_batch_operation(), and the data to feed that function. The callbacks are just functions that have a final parameter that is an array reference typically called $context. The function can either perform all the needed work on the provided parameters, or perform part of that work and update the $context['sandbox']['finished'] value to be a number between 0 and 1. Once finished reaches 1 (or isn’t set at the end of the function) batch declares that task complete and moves on to the next one in the queue. Once all tasks are complete it calls the function provided as the finished value of the array that defined the batch.
The finish function implements callback_batch_finish() which means it accepts three parameters: $success, $results, and $operations: $success is true when all tasks completed without error; $results is an array of data you can feed into the $context array during processing; $operations is your operations list again.
Those functions are all expected to be static methods on classes or, more commonly, a function defined in a procedural code block imported from a separate file (which can be provided in the batch array).
My replacement batch service
It’s those blocks of procedural code and classes of nothing but static methods that bug me so much. Admittedly the batch system is convenient and works well enough to handle major tasks for lots of modules. But in Drupal 8 we have a whole suite of services and plugins that are designed to be run in specific contexts that batch does not provide by default. While we can access the Drupal service container and get the objects we need the batch code always feels clunky and out of place within a well structured module or project. What’s more I have often created batches that benefit from having the key tasks be functions of a service not just specific to the batch process.
So after several attempts to force batches and services to play nice together I finally created this module to force a marriage. Even though there are places which required a bit of compromise, but I think I have most of that contained in the abstract class so I don’t have to worry about it on a regular basis. That makes my final code with complex logic and processing far cleaner and easier to maintain.
The Batch Service Interface module provides an interface an an abstract class that implements parts of it: abstract class AbstractBatchService implements BatchServiceInterface. The developer extending that class only needs to define a service that handles generating a list of operations that call local methods of the service and the finish batch function (also as a local method). Nearly everything else is handled by the parent class.
The implementation I provided in the example submodule ends up four simple methods. Even in more complex jobs all the real work could be contained in a method that is isolated from the oddities of batch processing.
<?php
namespace Drupal\batch_example;
use Drupal\node\Entity\Node;
use Drupal\batch_service_interface\AbstractBatchService;
/**
* Class ExampleBatchService logs the name of nodes with id provided on form.
*/
class ExampleBatchService extends AbstractBatchService {
/**
* Must be set in child classes to be the service name so the service can
* bootstrap itself.
*
* @var string
*/
protected static $serviceName = 'batch_example.example_batch';
/**
* Data from the form as needed.
*/
public function generateBatchJob($data) {
$ops = [];
for ($i = 0; $i < $data['message_count']; $i++ ) {
$ops[] = [
'logMessage' => ['MessageIndex' => $i + 1],
];
}
return $this->prepBatchArray($this->t('Logging Messages'), $this->t('Starting Batch Processing'), $ops);
}
public function logMessage($data, &$context) {
$this->logger->info($this->getRandomMessage());
if (!isset($context['results']['message_count'])) {
$context['results']['message_count'] = 0;
}
$context['results']['message_count']++;
}
public function doFinishBatch($success, $results, $operations) {
drupal_set_message($this->t('Logged %count quotes', ['%count' => $results['message_count']]));
}
public function getRandomMessage() {
$messages = [
// list of messages to select from
];
return $messages[array_rand($messages)];
}
}
There is the oddity that you have to tell the service its own name so it can bootstrap itself. If there is a way around that I’d love to know it. But really one have one line of code that’s a bit strange, everything else is now fairly clear call and response.
One of the nice upsides to this solution is you could write tests for the service that look and feel just like any other services tests. The methods could all be called once, and you are not trying to run tests against a procedural code block or a class that is nothing but static methods.
I would love to hear ideas about ways I could make this solution stronger. So please drop me a comment or send me a patch.
Related core efforts
There is an effort to try to do similar things in core, but they look like they have some distance left to travel. Obviously once that work is complete it is likely to be better than what I have created, but in the meantime my service allows for a new level of abstraction without waiting for core’s updates to be complete.
In software just about all project management methodologies get labeled one of two things: Agile or Waterfall. There are formal definitions of both labels, but in practice few companies stick to those definitions particularly in the world of consulting. For people who really care about such things, there are actually many more methodologies out there but largely for marketing reasons we call any process that’s linear in nature Waterfall, and any that is iterative we call Agile.
Failure within project teams leading to disasters is so common and basic that not only is there a cartoon about it but there is a web site dedicated to generating your own versions of that cartoon (http://projectcartoon.com/).
Among consultants I have rarely seen a company that is truly 100% agile or 100% waterfall. In fact I’ve rarely seen a shop that’s close enough to the formal structures of those methodologies to really accurately claim to be one or the other. Nearly all consultancies are some kind of blent of a linear process with stages (sometimes called “a waterfall phase” or “a planning phase”) followed by an iterative process with lots of non-developer input into partially completed features (often called an “agile phase” or “build phase”). Depending on the agency they might cut up the planning into the start of each sprint or they might move it all to the beginning as a separate project phase. Done well it can allow you to merge the highly complex needs of an organization with the predefined structures of an existing platform. Done poorly it can it look like you tried to force a square peg into a round hole. You can see evidence of this around the internet in the articles trying to help you pick a methodology and in the variations on Agile that have been attempted to try to adapt the process to the reality many consultants face.
In 2001 the Agile Manifesto changed how we talk about project management. It challenged standing doctrine about how software development should be done and moved away from trying to mirror manufacturing processes. As the methodology around agile evolved, and proved itself impressively effective for certain projects, it drew adherents and advocates who preach Agile and Scrum structures as rigid rules to be followed. Meanwhile older project methodologies were largely relabeled “Waterfall” and dragged through the mud as out of date and likely to lead to project failure.
But after all this time Agile hasn’t actually won as the only truly useful process because it doesn’t actually work for all projects and all project teams. Particularly among consulting agencies that work on complex platforms like Drupal and Salesforce, you find that regardless of the label the company uses they probably have a mix linear planning with iterative development – or they fail a lot.
Agile works best when you start from scratch and you have a talented team trying to solve a unique problem. Anytime you are building on a mature software platform you are at least a few hundred thousand hours into development before you have your first meeting. These platforms have large feature sets that deliver lots of the functionality needed for most projects just through careful planning and basic configuration – that’s the whole point of using them. So on any enterprise scale data system you have to do a great deal of planning before you start creating the finished product.
If you don’t plan ahead enough to have a generalized, but complete, picture of what you’re building you will discover very large gaps after far too many pieces have been built to elegantly close them, or your solution will have been built far more generically than needed – introducing significant complexity for very little gain. I’ve seen people re-implement features of Drupal within other features of Drupal just to deal with changing requirements or because a major feature was skipped in planning. So those early planning stages are important, but they also need to leave space for new insights into how best to meet the client’s need and discovery of true errors after the planning stage is complete.
Once you have a good plan the team can start to build. But you cannot simply hand a developer the design and say “do this” because your “this” is only as perfect as you are and your plan does not cover all the details. The developer will see things missed during planning, or have questions that everyone else knows but you didn’t think to write down (and if you wrote down every answer to every possible question, you wrote a document no one bothered to actually read). The team needs to implement part of the solution, check with the client to make sure it’s right, adjust to mistakes, and repeat – a very agile-like process that makes waterfall purists uncomfortable because it means the plan they are working from will change.
In all this you also have a client to keep happy and help make successful – that’s why they hired someone in the first place. Giving them a plan that shows you know what they want they are reassured early in the project that you share their vision for a final solution. Being able to see that plan come together while giving chances to refine the details allows you to deliver the best product you are able.
Agile was supposed to fix all our problems, but didn’t. The methodologies used before were supposed to prevent all the problems that agile was trying to fix, but didn’t. But using waterfall-like planning at the start of your project with agile-ish implementation you can combine the best of both approaches giving you the best chances for success. We all do it, it is about time we all admit it is what we do.
Like last year I’m keeping an extremely rough setup of notes from DrupalCon as a repository of things I’m picking up and tracking of sessions that looked like they would be interested but that I couldn’t attend because I was in another session. I’ll clean then up a bit and add to them over time.
Thank you to everyone who worked so hard to make the event a success.
I’ve assembled a playlist of the various sessions I though were good when I attended, or looked good but couldn’t attend.
Monday I attended the Community Summit, and while I had lots of great discussions, I didn’t take a lot of notes. The biggest two things I noted were that Western New York DUG is doing interesting stuff with online meetings that might be worth checking out and emulating for the SC DUG. And that Mid-Camp keeps a list of all the various channels that have videos of Drupal Camp sessions.
DriesNote:
Roadmap:
The current roadmap looks pretty cool, assuming everything comes together as well as we all hope it will:
JS modernization and a new admin interface design are on their way, media library is part of that, but is likely a year out from being ready for prime-time.
Webchick summed this section of the talk nicely:
So rad to see the Out-of-the-Box and Layout initiatives being shown off together. Feels like #drupal 8 is really coming together for site builders and content authors! #driesnote#drupalcon
My read is that its well intended and has some ground to cover is it gets revised. I haven’t done a deep dive into its details yet, nor the response, but early reviews are mixed.
Frankly that Values statement leaves me a little cold. “Treating others with dignity and respect” !== “Creating a safe and inclusive community” #Driesnote#drupalcon
This was a really good session on accessibility with both a real world set of examples and realistic discussions of what’s hard and what happens when things pass tests but don’t get tested by humans.
Major take aways:
Modern tools support JS and it no longer gets in the way of accessibility. WCAG 1.0 said this was a problem 20 years ago, but that’s not the current best practice.
There are constraints to the work because of accessibility, but it they don’t have.
“There are times that I go to use an interactive calendar on the web and all I hear is 1, 2, 3, 4, 5, 6, 7, 8, 9, and so on to 30 or 31…with no indication that these are dates…just a mass of numbers in the middle of the page.”
We used to test sites by disabling CSS/JS. Now it makes more sense to try to navigate the site with a keyboard and see what happens. Remember that just because something is possible it doesn’t mean it’s obvious or good. This doesn’t get you to a great site, but allows you to pick off errors before someone finds them for you later.
When you tab to things, the visual affordances some designers hate can be put back in as a compromise for people using accessibility tools.
I need to spend more time with the iPhone voice over tool so I can test things better.
This was a really interesting session on the Material Admin theme, and what’s been needed to make it work. It’s not perfect, and may or may not be ready for prime-time, but it looks like a great idea and show what we can do to make the admin much better.
We’re behind, some of fixing that is easy, some of fixing that is hard.
When you’re UX is bad, people perceive things to be slow even if they aren’t. People think that material theme is faster even though it is demonstrably not.
Growth and survival of the project require us to have a better admin.
He’s trying to make sure add-ons for the theme/module are pretty standalone and just work. But theme’s can’t require modules which is silly.
Contenta uses Material by default on front and back because it provides decoupling well.
PDFs in Drupal (I was surprised that this one was overcrowded)
Salesforce BOF:
This BOF was a chance for Cornell to show off some great stuff they have been doing with Message Agency. They have done some cool stuff that shows the power for D8 and a good Salesforce integration.
It should go without saying, but it needs saying too much:
A Salesforce is a CRM. Drupal is a CMS.
Use your tools for what they are best at.
The content in Drupal, actions recorded back into Salesforce.
Lessons:
1) Know strengths of each tool
2) Understand user needs
3) Determine how you will use each tool
4) Get the details right: SSO, Data Mapping, etc.
Drupal is much better at providing accessibility, including Form Assembly which is hard. The SF eco-system is mixed on the whole.
One option for multiple databases is Snap Logic (apparently it is “capital intensive”).
This was a mini session that is worth watching if you’re unsure about the importance and value of having a code of conduct. The hope had been to have a discussion about the importance of Drupal’s CoC, but everyone who attended largely agreed about the broad strokes of the major issues that have been discussed lately in the community. We ended up talking more about how to broaden the discussion than about the CoC itself.
This session was an interesting look at the impact on ACLU’s D6 (yes that’s right) advocacy site running on Pantheon.
Moved to Pantheon in 2013. And that move dealt with limits of their old hosting solution. Unfortunately some of my old-timey knowledge of why that had that solution was so old they couldn’t tell me much about how they had managed to make that move.
“Crazy things happen all the time”
After the their ED made a Rachel Maddow appearance on 11/16/16 they saw an 85x traffic spike. Tag1 was called in to help sort out what happened.
Traffic spiked to just over 500 requests per minute during the interview.
They found it was database bound, which was very common on D6, but still something they see frequently.
Found queries with 3 table join with no indexes on the base table. Able to go from 200,000 rows being scanned, down to 76. They were responding in real-time in crisis response mode.
After the wave passed, they called Pantheon to help build out environments for testing using multi-dev.
During the spikes that followed for the first travel ban, which were even larger they worked to reroute errors to Fastly, which served a PayPal fundraising link: at least the donations kept coming but that wasn’t good enough.
They needed a botnet to replicate the traffic. Tag1 used: Locust to create load tests, SaltStack to organize the bots, and EC2 to be the bots. They were failing at ~600 requests per minute and they were able to get to ~5,000 requests per minute. At that point the payment gateways were also starting to buckle, which isn’t a thing most people see.
The final wave they discussed came after the Net Neutrality lose, which peaked around 1,900 form submissions/min.
ACLU needed more logging, but didn’t want them logging personal information. Turned out the payment gateway’s CDN was detecting a DDOS and blocking them. See curl_log and curl_loadbalance. They also intentionally shift load from MySQL to Redis and PHP(?!?) because they knew Pantheon could scale that are far and as fast as needed to handle the waves, but MySQL was a limiting factor.
Things he argues we pioneered in Open Source:
– Distributed leadership
– Collabortive Development
– Community Engagement
Biggest lessons/takeaways from #Drupal for other open source communities: distributed development, collaborative development, community engagement. #DrupalCon
Interesting to reflect that its about the process and the community, but not about the technology.
Unrelated:
“Who here believes Facebook is unethical?” [hands rise] “Okay, who here has added a tracking pixel to a site at a client’s request?” [hands sheepishly rise] “Okay, now we return to the trolley problem…”
(Note: recording was intentionally stopped after the presentation but the discussion continued for quite a while).
Following Con last year Whitney Hess put forward some ideas, but it wasn’t clear where to go next.
It wasn’t clear that the DA should lead this, so it fell to the CWG cause they were last group standing.
Take aways:
Governance should evolve over time.
Need a values statement
Need to define the community and its membership.
Clearly document that structures and procedures.
CWG needs to improve CoC and enforcement.
Community needs to improve its global outreach.
DA should set higher standards.
Community matters should escalate to groups, not individuals.
We need community onboarding.
We should engage with other communities to discover best practices.
What’s Happened:
Dries stepped down as DA board chair.
DA hired Rachel Lawson.
DA created an updated CoC.
Dries is doing a round table on Thursday.
What’s Next:
Trying to figure that out…
Need to determine if good feedback was gathered so far.
Need to figure out an ongoing and continuous feedback process.
The expected frustrations with Dries and the values statement were expressed. Communication between Dries and other folks continues to be a challenge. The bottleneck of single point of contact is making it hard to stop having a single point of contact.
The main discussion centered around what’s holding back D8 adoption and the ongoing sense that the main forces in the Drupal community no longer concern themselves with the nonprofit sector. This year’s BOF was small because NTC started today in New Orleans. From a rough head count if the people I new were in New Orleans had been at the BOF there would have been a similar number of people.
In 2013 51% of internet traffic was “non-human”. SEO industry calls it NHT. By 2014 it was 61%.
Humans can have superpowers – we have very real opportunities to improve our quality of life, our safety and accessibility or knowledge and resources. #DrupalCon@Amorelandra
Automatic’s Support of Camps and staff to do so: It’s great, but it’s not in the budget (DA budget). He talked about creating it as a DA service that could be self-sustaining, but the WordPress model includes a donation of 8 FTEs.
Yes!! Yes!! Let’s get the drupal al camp organizers together and hash it out!! We can do it! Likes camp organize sprint!! #drupalcon#AskDries@kaleemclarkson
What if your responsible for 1000 D7 sites? When we will know when there is a concrete answer to the question of the EOL for D7: This is an open issue without a good answer that needs a good answer. Move to D8? (but he doesn’t understand why that’s laughable without more detail).
What about the small shops and builders: He doesn’t feel like they were really left behind. Rachel also checked to what the DA could have done better with the new home page, but the language wasn’t a great choice.
"I don't think we're going to beat a Wix or a Squarespace. … Squarespace is really good at page-building, and we can look at them for inspiration. … We can do page-building that plays to our strength, like structured content."@Dries#AskDries#DrupalCon
What can a consumer do to preserve the open web: Not use Facebook. People read the web through Facebook like they do with Google. Don’t install an ad blocker.
Why don’t you hear more about Diversity issues from you? It’s important, and we have to do better. We aren’t were we should be, and I’m happy to show more leadership. I could do more by talking more about it in public and on twitter. Wants to think more about it, and doesn’t feel like an expert. He acknowledged his mistake in the DriesNote in Copanhagen. He also commented about shuttingdown after being called out because of how it was done. Wants understanding of the fact that he’ll make mistakes.
"Why don't you engage more in diversity and inclusion things online?" —@aburke626
"We need to do better, frankly. I'm happy to show more leadership there. … I can certainly do more. I'm going to take you up on that, you have my promise."
When are we moving to Github? A proof of concept is in place to move to GitLab! Our tools are better than GitLab in many ways, but GitLab wants to have our better strengths in their code base. So they are working on doing that for us and for all their users.
Is Drupal 7 Dead? No. Most sites are Drupal 7, and some new sites still launch there. But all the innovation is on 8.
Q: "Is Drupal 7 dead? Releases have slowed down…"@Dries: "Drupal 7 isn't dead…but I think a lot of the innovation has shifted to Drupal 8. People are still launching new websites on Drupal 7, and that's fine."#AskDries#Drupalcon
The new values and principles need work to more fully reflect the community. The process: a group together in December to review the community feedback. And it was clear he needed to do this. He’s been working on it since then, and has found it hard work. He wanted to make it Collabortive, but also wanted to put a stake in the ground. He knows that it needs work, but isn’t entirely sure of the next steps. Doesn’t want to the single owner. He would like to assemble a working group with a charter.
.@dries: "As a next step we're going to put together a working group…a diverse committee of people that can actually take it from here and carry it forward. My next step is to put together a charter for this group."#AskDries#DrupalCon
There were other CMSes in the world, but they were a shit show. I was working on the Linux kernel and liked the modular nature.
Why do you think people are hesitant to update their site? Decided to elect minor updates not major. Mostly that it add complexity.
Q: Why do you think Drupal sites are hesitant to upgrade [minor] releases?
A: "We do add new features in minor releases…that's been a challenge. We've been trying to evolve our releases, trying not to break things." —@Dries#AskDries#DrupalCon
Will the new principles state that destructive beliefs, not just actions, will be banned. He defers to the working group.
Someone just asked if Drupal should police the "toxic beliefs" of community members. #askdries/@Dries has no, repeat no credibility on this topic as he was completely and totally complicit and responsible in the botched @Crell affair.
Rachel acknowledged the tweet, but didn’t know what to do with the fact that it actually called her out. “I wasn’t paying attention.” and then blamed questioners for not asking questions earlier. @drnikki was given a space, and directed people to DD&I meetings.
A really bad response from the audience calling on women lead. Tim Plunket responded appropriately.
"It's the responsibility… of the people in power and the people with privilege and the position and the voice to do this work for everyone else. I don't think it's fair to blame [under-represented people]" —@timplunkett#AskDries#DrupalCon
“Including people in community is more than saying, ‘you’re all included!’ A lot. It’s in our language and our symbols and how we present ourselves and how that communicates ‘what we do here’ to people who are watching.” @blackamazon at #DrupalCon
