Snowfakery Custom Plugins Part 2

Last week I posted part 1 of this series on creating plugins for Snowfakery. This second installment covers creating custom Faker Providers to use as plugins for Snowfakery.

Why create a Faker Provider instead of Snowfakery Plugin

Python’s Faker library provides many useful things, but not everything you might want. Fortunately it is designed to be extended, and Snowfakery is designed to help make it possible for those extensions to be project specific when you need to (or when it makes it easier for your team to share recipes).

Sometimes you need to have all the features of a Snowfakery plugin, like maintaining state. But often you just need to kill a gap left by the Faker community offerings. In that case, a Faker Provider may offer longer term benefits like re-usability in other projects, and useful helper functions, that give it the strong choice.

Things you’ll need

You do not need to have reviewed everything in Part 1, but you will want to look at the project structure section and to make sure you have the following tools:

  1. Python 3 (any recent-ish version should be fine), and the experience to read and write simple Python scripts (they aren’t any harder to follow than YML just different).
  2. Snowfakery 1.12 or later. Note: if you have CCI installed it contains Snowfakery but on Windows you may need extra setup.
  3. The Faker module for Python (probably via: pip3 install Faker). Not strictly required but can help you with testing.
  4. A code editor you like that supports both Python and YML (which is pretty much anything good).
  5. You probably want experience working on at least one or two Snowfakery recipes.

Reminder about project structures

In part 1 we build a project around the built-in search patterns of Snowfakery.

Snowfakery looks for plugins in a select number of places:

  • The Python path.
  • In a plugins directory in the same directory as the recipe.
  • In a plugins directory below the current working directory.
  • A sub-directory of the user’s home directory called .snowfakery/plugins.
Recipe directory layout

For this example the plugins directory will live within the folder with our recipe (called recipes), but you can move the plugins directory up a level and it will work just as well.

To get started, in your project create a recipes directory. Next create a plugins directory within recipes. Then create faker_nonprofit directory in plugins. You can ignore the snowHelper and snow_product.yml examples from part 1 in the screenshot on the right, but that’s generally what we’re doing.

Create your first Snowfakery Faker Provider

Faker itself does not have a community supported generator of Nonprofit organization names, and when you work with nonprofits a lot sometimes you need those. That is exactly what we’ll create in just a minute.

In the plugins directory you created in part 1, create a directory for your new provider with the pattern faker_[my_service_name], in my case faker_nonprofit. Then in that new directory create a file named __init__.py this defines the Faker provider Python module.

You can see the full working example here but I’ll walk through the outline.

The file opens by providing a doc string and importing the faker.providers module.

"""Provider for Faker which adds fake nonprofit names, and program names."""
import faker.providers

The next several lines of the faker_nonprofit module provide arrays of words to use in the generation of fake names. Depending on what your provider does this may or may not be useful to you (but it’s very common).

Then we define the Provider itself as a class that extends the BaseProviders class, with whatever methods you want to call:

class Provider(faker.providers.BaseProvider):
   """Provider for Faker which adds fake nonprofit information."""
 
   def nonprofit_name(self):
       """Fake nonprofit names."""
       prefix = self.random_element(PREFIXES)
       suffix = self.random_element(SUFFIXES)
       topic = self.random_element(TOPICS)
       return " ".join([prefix, topic, suffix]).strip()

Those random element selections are from the arrays of words I mentioned a minute ago. Basically we’re just building a name from some selected words.

A note about tests

Unlike Snowfakery plugins, Faker projects have existing patterns for how to setup tests (they are totally consistent but there are patterns out there). So the complete code on Github includes a testing module as well, and you should consider something similar for yours (particularly if you are considering making it available to the wider community). It’ll allow you to test your provider generically not just when it’s running through Snowfakery.

Linking your Provider to a Snowfakery Recipe

Now that we have a local Faker Provider, we need to connect it to our recipe.

A very simple recipe (in our recipe directory) should demonstrate the output quite nicely:

- plugin: faker_nonprofit.Provider

- object: Account
  fields:
    Name:
      fake: nonprofit_name

Run the recipe through Snowfakery and you can see the name appears believable:

$ snowfakery recipes/sample_recipe.yml
Account(id=1, Name=Southern Unity Community)

The project uses a slightly larger recipe here that uses both plugins and generates more than one object. The full recipe in the repo will create one Salesforce Account object that has a Fake Nonprofit name, a custom field for a Main Service that will be one of our snowy puns, an address, and two related contacts with all their basic information included.

$ snowfakery recipes/sample_recipe.yml 
Account(id=1, Name=Upper Friends Committee, Main_Service__c=Snowmanage, BillingStreet=4773 Giles Plains Suite 878, BillingCity=South Daniel, BillingState=Virginia, BillingPostalCode=34516, BillingCountry=United States, ShippingStreet=7844 Hester Shore Apt. 299, ShippingCity=Maynardview, ShippingState=Indiana, ShippingPostalCode=86323, ShippingCountry=United States, Phone=956.673.3002x471, Fax=+1-786-744-2112x36239, RecordType=Organization)
Contact(id=1, AccountId=Account(1), Salutation=Misc., FirstName=Isaac, LastName=Barr, Email=joypeters@example.com, Phone=+1-808-508-0989x418, MobilePhone=(987)475-7200x8072, Title=Tour manager, Birthdate=1982-03-27)
Contact(id=2, AccountId=Account(1), Salutation=Mx., FirstName=Erica, LastName=Lopez, Email=angel01@example.org, Phone=(011)243-1677x868, MobilePhone=(079)466-5474x52399, Title=Research officer, political party, Birthdate=2000-07-07)

If you have interest in seeing the Nonprofit Provider made into a more complete tool and released as its own project please let me know.

Snowfakery Custom Plugins Part 1

Last November I wrote a bit about creating Salesforce data with Snowfakery.  I’ve continued to use the tool for work, provide feedback to the project maintainer, and help the Salesforce Open Source Commons Data Generation Toolkit Project as we started to build a library of sample recipes. Hopefully I will have more to say on that after the next Community Sprint.

Snowfakery not only gives you a way create carefully shaped relational data sets of nearly any size, it also allows you to create plugins to extend its abilities. those plugins come in two flavors: Snowfakery Plugins, and Faker Providers.

For more technical details you may want to read the project has documentation. My intention here is to provide an end-to-end example of how to make them work.

This article started out as one long piece but to keep it focused I’ve decided to break it into two parts:

  • Part 1 covers Snowfakery Plugins.
  • Part 2 covers creating custom Faker Providers for Snowfakery projects.

The code for both parts is on Github if you want to see the project as a whole.  

Things you’ll need

  1. Python 3 (any recent-ish version should be fine), and the experience to read and write simple Python scripts (they aren’t any harder to follow than YML just different).
  2. Snowfakery 1.12 or later. Note: if you have CCI installed it contains Snowfakery but on Windows you may need extra setup.
  3. A code editor you like that supports both Python and YML (which is pretty much anything good).
  4. You probably want experience working on at least one or two Snowfakery recipes.

Snowfakery Plugin Project Structure

This setup just talks about Snowfakery recipes on their own, not within a larger project, but the concepts are the same even if the details are different.

Snowfakery looks for plugins in a select number of places:

  • The Python path.
  • In a plugins directory in the same directory as the recipe.
  • In a plugins directory below the current working directory.
  • A sub-directory of the user’s home directory called .snowfakery/plugins.
Recipe directory layout

For this example the plugins directory will live within the folder with our recipe (called recipes), but you can move the plugins directory up a level and it will work just as well.

To get started, in your project create a recipes directory. Next create a plugins directory within recipes. Until part 2 you can ignore the faker_nonprofit directory in the screenshot on the right, but that’s generally what we’re doing.

Create Your First Snowfakery Plugin

In the Snowfakery community we use a lot of snow-based puns to name things. So to help create fake sounding products for our projects we might need a simple plugin to generate us new words that match our general naming convention.

In the plugins directory create a new file called snowHelper.py. And copy the follow Python code into your editor:

from snowfakery import SnowfakeryPlugin
 
class SnowPunnary(SnowfakeryPlugin):
   class Functions:
       def snowpunner(self, word):
           return 'Snow' + word

The code here is pretty straight forward if a little nested. We are loading the SnowfakeryPlugin class from Snowfakery itself, and then extending that class to create our plugin. Snowfakery assumes that the plugin has a subclass to hold your plugin functions (called Functions) and that you add your functions to that subclass. Your functions can have a parameter (here the word being punned on) to accept inputs from other parts of the recipe.

Our SnowProduct Recipe

Now we need a recipe that will actually use our plugin. In the recipes directory create a file called snow_product.yml and copy in the following YAML code:

- plugin: snowHelper.SnowPunnary
 
- object: Product
  count: 10
  fields:
    Name:
      SnowPunnary.snowpunner: ${{fake.word}}

In the first line we load our plugin using Python’s module naming convention – because it is getting loaded as a Python module. The pattern here is the file name (without file extension) then the class name. In the last line we then call the plugin’s function by referencing the class name and the function name.

You can run the file directly in Snowfakery and see the outputs:

$ snowfakery snow_product.yml
Product(id=1, Name=Snowplantary)
Product(id=2, Name=Snowbadary)
Product(id=3, Name=Snowmillionary)
Product(id=4, Name=Snoweffortary)
Product(id=5, Name=Snowgreenary)
Product(id=6, Name=Snowbehaviorary)
Product(id=7, Name=Snowcouldary)
Product(id=8, Name=Snowforceary)
Product(id=9, Name=Snowyesary)
Product(id=10, Name=Snowcompanyary)

I’m going to show, but not go into great depth on, one more detail: plugins can save state. Snowfakery provides a mechanism for tracking context variables between calls that allow you to track current state. So we can have ours count the number of times the snowpunner function has been called and return that count in another function:

from snowfakery import SnowfakeryPlugin
 
class SnowPunnary(SnowfakeryPlugin):
   class Functions:
       def snowpunner(self, word):
           context_vars = self.context.context_vars()
           context_vars.setdefault("count", 0)
           context_vars["count"] += 1
           return 'Snow' + word
 
       def currentCounter(self):
           context_vars = self.context.context_vars()
           return context_vars["count"]

Then update the recipe like this:

- object: Product
  count: 10
  fields:
    Name:
      SnowPunnary.snowpunner: ${{fake.word}}
    Index:
      ${{SnowPunnary.currentCounter()}}

Notice that to call the function without a parameter we use the formula syntax.  Run it again and we see the new index that shows the count:

$ snowfakery snow_product.yml
Product(id=1, Name=1: Snowimpact, Index=1)
Product(id=2, Name=2: Snowfund, Index=2)
Product(id=3, Name=3: Snowdark, Index=3)
Product(id=4, Name=4: Snowteach, Index=4)
Product(id=5, Name=5: Snowteam, Index=5)
Product(id=6, Name=6: Snowsummer, Index=6)
Product(id=7, Name=7: Snownew, Index=7)
Product(id=8, Name=8: Snowperform, Index=8)
Product(id=9, Name=9: Snowonto, Index=9)
Product(id=10, Name=10: Snowmodel, Index=10)

It is important to remember that while it would be possible to add a value to the context variable on each iteration, that would cause Snowfakery to consume more memory on each iteration. Snowfakery is designed to generate records by the hundreds of millions if asked, and does so while consuming very little extra memory – you can do things in context variables that would break down on larger runs.

In part 2, I talk about creating a custom Faker Provider and loading it into a Snowfakery recipe.

Why and How to Write Good How-To Articles

Part of contributing to any open source project, or even really being a contributing member of any community, is sharing what you know. That can come in many forms. While many projects over emphasis code, and most of us understand the value of conference talks, good how-to articles are some of the most critical contributions for any software platform. There isn’t much point to a tool if people cannot figure out how to use it.

Why do I write how-to articles

I’ve contributed code to Drupal, some of it even good and useful to others. But usually when I hear someone noticed something I created it’s blog posts about how to solve a problem.

When I struggled to find the answer to a question I expect it is a candidate for a how-to post. I am not so creative that I am often solving a problem no one has, or will want to, solve for another project. And I am good enough at what I do to know that if I struggled to find an answer it was probably harder to find than it could been.

That helps me find topics for articles that are helpful to the community and benefit me.

How-to articles help others in the community use tools better

The goal of a good tutorial is to help accelerate another person’s learning process. The solution does not have to be perfect, and I know most people will have to adapt the answer to their project. I write them when I struggled to find a complete answer in one place, and so I’m hoping to provide one place that gives the reader enough to succeed.

Usually I combine practical experience earned after digging through several references at various levels of technical detail – including things like other people’s blog posts, API documentation, and even slogging through other people’s code. I then write one, hopefully coherent, reference to save others that digging extra reading.

The less time people spend researching how to do something, the more time they have to do interesting work. Better yet, it can mean more time using the tools for their actual purpose.

How-to articles serve as documentation for me, colleagues, and even clients

The best articles serve as high level documentation I can refer back to later to help me repeat a solution instead of recreating it from scratch. When I first wrote how-to articles I was solidifying my own learning, and leaving a trail for later.

They also came to serve as documentation for colleagues. When I don’t have time to sit with them to talk through a solution, or know the person prefers reading, I can provide the link to get them off and running. Colleagues have given me feedback about clarity, typos, and errors to help me improve the writing.

I have even sent posts to clients to help explain how some part of their solution was, or will be, implemented. That additional documentation of their project can help them extend and maintain their own projects.

How-To articles give me practice explaining things

One of the reasons I started blogging in the first place was to keep my writing skills sharpened. How-to articles in-particular tend to be good at helping me refine my process in specific areas. The mere act of writing them gives me practice at explaining technology and that practice pays off in trainings and future articles. If you compare my work on Drupal, Salesforce, and Electron you can see the clarity improve with experience.

How-To articles give me work samples to share

When I’ve been in job applicant mode those articles give me material to share with prospective employers. In addition to Github and Drupal.org, the how-to articles can help a hiring manager understand how I work. They show how explain things to others, how I engage in the community, and serve as samples of my writing.

How-To articles help me control my public reputation

I maintain a blog, in part, to help make sure that I have control over my public reputation. To do that I need inbound links the help maintain page rank and other similar basic SEO games.

From traffic statistics I know the most popular pages on this site are technical how-to articles. From personal anecdotes I know a few of my articles have become canonical descriptions of how to solve the problems.

When I first started my current job we had a client ask if I could implement a specific feature that he’d read about in a post on Planet Drupal. It turned out to be mine. Not only was I happy to agree to his request, it helped him trust our advice. My new colleagues better understood what this Drupal guy brought to the Salesforce team. Besides let’s be honest it’s fun when people cite your own work back at you.

Writing your own

You don’t have to maintain a whole blog to write useful how-to articles. Drupal, like most large open source projects, maintains public wiki-style documentation. Github pages allow anyone to freely publish simple articles and there are many examples of single-page articles out there. And of course there is no shortage of dedicated how-to sites that will also accept content.

The actual writing process isn’t that hard, but often people leave out steps, so I’ll share my process. This is similar to my general advice for writing instructions.

Pick your audience

It’ll be used more widely than whoever you think of, but have an audience in mind. Use that to help target a skill set. I often like to think of myself before I started whatever project inspired the article. The higher your skill set the more you should adjust down, but it’s hard to adjust too far, so be careful is aiming for people with far less experience than you have – make sure you have a reviewer with less experience check your work. Me − 1 is fine, Me − 5 is really hard to do well.

Start from the beginning and go carefully step by step

Start with no code, no setup, nothing. Then walk forward through the project one step at a time writing out each step. If you gloss over a detail because you assume your audience knows about it add reference links. You can have a copy of a reference project open but do not use it directly; it’s there to prevent you from having to re-research everything.

List your assumptions as you go

Anything that you need to have in place but don’t want to describe (like installing Drupal into a local environment, creating a basic module, installing Node, etc) state as an explicit assumption so your reader starts in the same place as you do. Provide links for any assumptions which are likely hard for your expected audience to complete. This is your first check point – if there are no good references to share, start from where that article you cannot find should start (or consider writing that article too). 

Provide detailed examples

Insert code samples, screenshots, or short videos as you progress. Depending on what you are doing in your article the exact details of what works best will vary. Copy and paste as little reference code as possible. This helps you avoid accidentally copying details that may be revealing of a specific project’s details.

If you look at mine you’ll see a lot of places where I include comments in sample code that say things like “Do useful stuff”. That is usually a hint that whoever inspired the article had interesting, and perhaps proprietary, ideas in that section of code (or at least I worried they would think it was interesting). I also try to add quick little asides in the code samples to help people pay attention.

Test as you go

Make sure your directions work without that reference project you’re not sharing. This is both so your directions work properly and further insulation against accidentally sharing information you ought not share.

End with a full example

If you end up with a bunch of code that you’ve introduced piecemeal, provide a complete project repo or gist at the end. You’ll see some of my articles end in all the code being displayed from a gist, and others link to a full repository. Far too many people simply copy and paste code from samples and then either use it blindly or get stuck. Moving it to the end helps get people to at least scan the actual directions along the way.

Give credit where credit is due

If you found partial answers in several places during your initial work, thank those people with links to their articles. Everyone who publishes online likes a little link-love and if the article was helpful to you it may be helpful to others. Give them a slight boost.

Salesforce Lightning Web Components with URL Parameters

A couple weeks ago I needed to create a Salesforce Lightning Web Component (LWC) that pulls values from URL parameters. While the process is very simple it turns out the vast majority of examples on the web are out of date due to a security update Salesforce made sometime last year – and so I spent a frustrating afternoon throwing ideas at the wall until a colleague stumbled into a comment on a blog post that was an incorrect example by a highly trusted expert noting the needed fix.  So, in the hopes of shortening the search for anyone else trying to get this to work, I’m offering an example that works – at least as of this writing.

To be fair the official docs are correct but it is easy to look passed an important detail: if you do not put a namespace on your value the parameter will be deleted.

That change was the security update, before that you could have any value as your parameter name now you have to have __ (two underscores) in the name.  Officially the docs say that in the left side of those underscores you should have the namespace of your package or a “c” for unpackaged code. As far as I can tell at least in sandboxes and trailhead orgs you can have anything you want as long as there are characters before and after the __ (which kinda makes sense since package developers need to be able to write and test their JavaScript before they build their package).

So your final URLs will look something like:
https://orgname.my.salesforce.com/lightning/r/Contact/0034x000009Xy5gAAC/view?c__myUrlParameter=12345

Basic LWC

Now with that main tip out of the way on to a full example.

My assumption going into this is that you know how to create a very basic Hello World quality LWC. If not, start with the Trailhead Hello World example project.

1) Create a new component to work with, mine will be very simple to help keep the details clean, but you can fold this into more interesting code bases.

2) Update the component’s meta.xml file to set isExposed to true, and at least a target of lighning__RecordPage (although any target will do if you know how to use it), and configure the target to connect to Contact (although again any settings you know how to use are fine here).

<?xml version="1.0" encoding="UTF-8" ?>
<LightningComponentBundle xmlns="http://soap.sforce.com/2006/04/metadata">
   <apiVersion>50.0</apiVersion>
   <isExposed>true</isExposed>
   <description>Example Lightning Web Componant to read URL parameters.</description>
   <targets>
       <target>lightning__RecordPage</target>
   </targets>
   <targetConfigs>
       <targetConfig targets="lightning__RecordPage">
           <objects>
               <!-- This is setup to run on contact but you could use any sObject-->
               <object>Contact</object>
           </objects>
       </targetConfig>
   </targetConfigs>
</LightningComponentBundle>

3) In your JS file beyond the main LighningElement you need to add imports for wire, track, and CurrentPageReference from the navigation library:

import { LightningElement, wire, track } from "lwc";
import { CurrentPageReference } from "lightning/navigation";

4) Add a tracked value you want to display inside the main class: 

export default class Parameter_reader extends LightningElement { 
  @track displayValue;

5) Next use the wire decorator to connect CurrentPageReference’s getStateParameters to your own code to get an use the URL parameters:

@wire(CurrentPageReference)
getStateParameters(currentPageReference) {
 if (currentPageReference) {
   const urlValue = currentPageReference.state.c__myUrlParameter;
   if (urlValue) {
     this.displayValue = `URL Value was: ${urlValue}`;
   } else {
     this.displayValue = `URL Value was not set`;
   }
 }

From the code sample above you can see that we’re getting the values from currentPageReferences’s state child object, and then attaching them to our tracked value we created in step four.

6) Update the HTML file to display your value ideally leveraging the SLDS along the way:

<template>
 <div>
   <lightning-card title="Url Sample" icon-name="custom:custom14">
     <div class="slds-m-around_medium">
       <p>{displayValue}</p>
     </div>
   </lightning-card>
 </div>
</template>

7) Deploy all this code to your org.

8) Go to a contact record, and edit the page. Add your new competent to the side bar. Save and activate the page.

9) Return to the record page, the component should appear and say “URL Value was not set”.

10) In the address bar add to the end of the url: ?c__myUrlParameter=Hello, and reload the page, the component should now read “URL Value was Hello”.

A screenshot of the sample component displaying the provided text of "hello".

What about sending the value to APEX?

Now, let’s go one step further and send this parameter over the APEX and post a response.

1) Create an APEX class, and create a public static method using the AuraEnabled decorator.

 @AuraEnabled(cacheable=true)
 public static String reflectValue(String value) {
     // Really you should do something useful here.
     return value;
 }

In this case we’re starting with a method that just passes back the same string it was handed, but obviously you can do whatever you want here.

BE CAREFUL ABOUT SECURITY!

If you take an ID as your parameter make sure you are thinking about what happens when someone sends an ID for an object they should not see, is for an object other than the type you expected, and other similar things. The platform can help you but security is your job here, take it seriously!

2) Create good tests for your class, and deploy the code.

3) Import the new function into your JS file:

import reflectValue from "@salesforce/apex/valueReflection.reflectValue";

Update the getStateParameter handler we wrote before to call this function as a JavaScript promise:

  getStateParameters(currentPageReference) {
    if (currentPageReference) {
      const urlValue = currentPageReference.state.c__myUrlParameter;
      if (urlValue) {
        reflectValue({ value: urlValue })
          .then((result) => {
            this.displayValue = `URL Value was: ${result}`;
          })
          .catch((error) => {
            this.displayValue = `Error during processing: ${error}`;
          });
      } else {
        this.displayValue = `URL Value was not set`;
      }
    }
  }

4) That’s it! Deploy your code and reload the page, and your values should pass through to APEX, come back and get displayed.

The complete SFDX project for this example is up on Github.

Salesforce Electron Starter

Back in August I created an Electron project starter that provides a template to use for electron projects with the goal of outlining how to follow the current best practices for writing secure electron apps. I had extracted that from a couple of personal projects I work on from time to time, one of those projects is ElectronForce – a tool to explore Salesforce orgs.  Because I get ideas of things I want to try out from time to time as Salesforce APIs applications I have now created a derivative project that is setup to create apps that leverage JSForce to interact with Salesforce orgs.

Thus I would like to introduce Electron Salesforce Base

Like my generic project starter it is intended to be a jumping off point that handles some of the basic elements of a project.  It’s a bit more opinionated because comes with a little more plumbing in place – there is a simple interface and it’ll actually log into orgs (assuming you have your security token). 

The interface is built using a Bootstrap dark theme from Bootswatch, and is set up to follow the Airbnb ESLint standards (with a couple small tweaks). The interface generates two windows, one that is meant to be the main interface and includes the controls to log into your org, and a second that is meant to keep a running log of events.

The main thread is fully isolated from the render threads, with all requests and data being passed back and forth using the current inter process communication methodology from Electron leveraging the IPCmain object in the main thread and the contextBridge in the render thread – there is no access to remote in the render thread (actually remote is fully disabled as it should be), and the preload.js file largely serves to filter IPC requests to maintain thread isolation. Currently the main thread of this project isn’t what I would call graceful, and I’m actually working on a refactor for ElectronForce to improve the IPC listener definitions (readers with examples of good design patterns are more than welcome to offer suggestions, whatever I settle on will likely get folded back into this project eventually).

To help understand the general pattern that’s emerging as people get better at sandboxing in Electron (and Electron gets better at demanded it) I find it helpful to think of Electron apps in a client-server model with two largely separate applications and a well defined API for communication between them. You’ll see that reflected here, and in my other recent Electron apps. You can implement whatever you’d like in each layer and just pass messages back and forth. This also means you can totally refactor one part of your application without worrying about the other. So if you hate my proto-interface dump it and build something better.

If you look at the code for this base project, then look at ElectronForce, you will see the render thread provides all the details of the interface – including use of render-friendly libraries like jQuery and a collection of helper functions to make life a little easier – makes an API call (with a filter list provided in preload.js), and then handles responses from the main thread. In main.js you see all the IPC listeners defined, which then pass the needed data to JSForce to make the API calls, before handing back structured data for the interface to render.

As you dig through the code you may notice various @TODO statements that are notes to myself about places with obvious room for improvement. I’m always happy to get suggestions, as comments here, issues there, or as pull requests to help resolve those notes with better solutions.

Generate Sample Data for Salesforce NPSP

Snowfakery is a fairly new tool from the team at Salesforce.org. It is designed to generate any amount of arbitrary relational data you can cook up. The Salesforce Open Source Commons Data Generation Toolkit Project has been starting to work on including it in our work (you can read more about my relationship to these projects on the Attain blog) and as part of that, and as part of my actual day job, I’ve been playing with its recipe language, reviewing the documentation, and generally trying to make sure I can use it well.

Snowfakery itself is excellent, but since it’s new there’s a lot of work left to be done on the documentation. The existing docs focus on the generic ways you could use Snowfakery and outline all the features, but I often need it to create very specific things: data for Salesforce orgs with NPSP installed for our nonprofit clients, and the docs aren’t currently great at getting you started at doing that – this post is intended as partial filler in that gap.

The recipe provided here is based on what I’ve learned from my first real successes getting Snowfakery to work on a real-world project.

It give credit where due my first real break through was when Snowfakery’s creator, Paul Prescod, pointed me to a branch in the repo that had some starter NPSP examples. Paul’s samples are helpful but are incomplete and not super well commented. While my solution leverages his work, it simplifies things and takes some short cuts. My goal is functional and understandable, not perfect.

This recipe creates 30 gifts, 20% from companies, 80% from households. They are spread randomly over a set of preexisting Campaigns, and create the needed Accounts, Contacts, Opportunities, and Payments.  It also assumes you are generally comfortable with the major NPSP objects.

This solution is assembled from three files. The main recipe file snowfakery_npsp_basic_recipe.yml for the objects you want to create, and two layered macro files which are used to generate often repeated objects, npsp_macros.yml, and sf_standard_macros.yml. Both are modified from the versions in the example branch of the repo. The Standard Salesforce objects macro file is derived from this one and the NPSP objects recipe macro file is derived from this one.

The code all of them is in a gist, and embedded below (they are numbered in the gist to control display order, remove numbers before actually using). There are inline comments to explain their details. Once you have local copies of all three files (and have Snowfakery installed) you can generate a JSON file that matches the data described:
$ snowfakery --output-format=JSON --output-file gifts.json snowfakery_npsp_basic_recipe.yml

If you are using CumulusCI for your project, you can have CCI load the data directly into your org:

cci task run generate_and_load_from_yaml -o generator_yaml ./datasets/snowfakery_npsp_basic_recipe.yml -o num_records 30 -o num_records_tablename Account --org my_project_sandbox

Salesforce Queries and Proxies in Drupal 8

The Drupal 8 version of the Salesforce Suite provides a powerful combination of features that are ready to use and mechanisms for adding custom add-ons you may need.  What it does not yet have is lots of good public documentation to explain all those features.

A recent support issue in the Salesforce issue queue asked for example code for writing queries. While I’ll address some of that here, there is ongoing work to replace the query interface to be more like Drupal core’s.  Hopefully once that’s complete I’ll get a chance to revise this article, but be warned some of those details may be a little out of date depending on when you read this post.

To run a simple query for all closed Opportunities related to an Account that closed after a specific date you can do something like the following:

      $query = new SelectQuery('Opportunity');
      $query->fields = [
        'Id',
        'Name',
        'Description',
        'CloseDate',
        'Amount',
        'StageName',
      ];
      $query->addCondition('AccountId', $desiredAccountId, '=');
      $query->conditions[] = [
        "(StageName", '=', "'Closed Won'",
        'OR', 'StageName', '=', "'Closed Lost')",
      ];
      $query->conditions[] = ['CloseDate', '>=', $someSelectedDate];
      $sfResponse = \Drupal::service('salesforce.client')->query($query);

The class would need to include a use statement for to get Drupal\salesforce\SelectQuery; And ideally you would embed this in a service that would allow you to inject the Salesforce Client service more correctly, but hopefully you get the idea.

The main oddity in the code above is the handling of query conditions (which is part of what lead to the forthcoming interface changes). You can use the addCondition() method and provide a field name, value, and comparison as lie 10 does. Or you can add an array of terms directly to the conditions array that will be imploded together. Each element of the conditions array will be ANDed together, so OR conditions need to be inserted the way lines 11-14 handle it.

Running a query in the abstract is pretty straight forward, the main question really is what are you going to do with the data that comes from the query. The suite’s main mapping features provide most of what you need for just pulling down data to store in entities, and you should use the entity mapping features until you have a really good reason not to, so the need for direct querying is somewhat limited.

But there are use cases that make sense to run queries directly. Largely these are around pulling down data that needs to be updated in near-real time (so perhaps that list of opportunities would be ones related to my user that were closed in the last week instead of some random account).

I’ve written about using Drupal 8 to proxy remote APIs before. If you look at the sample code you’ll see the comment that says: // Do some useful stuff to build an array of data.  Now is your chance to do something useful:

<?php
 
namespace Drupal\example\Controller;
 
use Symfony\Component\HttpFoundation\Request;
use Drupal\Core\Controller\ControllerBase;
use Drupal\Core\Cache\CacheableJsonResponse;
use Drupal\Core\Cache\CacheableMetadata;
 
class ExampleController extends ControllerBase {
    public function getJson(Request $request) {
        // Securely load the AccountId you want, and set date range.
 
        $data = [];
        $query = new SelectQuery('Opportunity');
        $query->fields = [
            'Id',
            'Name',
            'Description',
            'CloseDate',
            'Amount',
            'StageName',
        ];
        $query->addCondition('AccountId', $desiredAccountId, '=');
        $query->conditions[] = [
            "(StageName", '=', "'Closed Won'",
            'OR', 'StageName', '=', "'Closed Lost')",
        ];
        $query->conditions[] = ['CloseDate', '>=', $someSelectedDate];
        $sfResponse = \Drupal::service('salesforce.client')->query($query);
 
    if (!empty($sfResponse)) {
        $data['opp_count'] = $sfResponse->size();
        $data['opps'] = [];
 
        if ($data['opp_count']) {
            foreach ($sfResponse->records() as $opp) {
                $data['opps'][] = $opp->fields();
            }
        }
    }
    else {
      $data['opp_count'] = 0;
    }
    // Add Cache settings for Max-age and URL context.
    // You can use any of Drupal's contexts, tags, and time.
    $data['#cache'] = [
        'max-age' => 600, 
        'contexts' => [
            'url',
            'user',     
        ],
    ];
    $response = new CacheableJsonResponse($data);
    $response->addCacheableDependency(CacheableMetadata::createFromRenderArray($data));
    return $response;
  }
}

Cautions and Considerations

I left out a couple details above on purpose. Most notable I am not showing ways to get the needed SFID for filtering because you need to apply a little security checking on your route/controller/service. And those checks are probably specific to your project. If you are not careful you could let anonymous users just explore your whole database. It is an easy mistake to make if you do something like use a Salesforce ID as a URL parameter of some kind. You will want to make sure you know who is running queries and that they are allowed to see the data you are about to present. This is on you as the developer, not on Drupal or Salesforce, and I’m not risking giving you a bad example to follow.

Another detail to note is that I used the cache response for a reason.  Without caching every request would go through to Salesforce. This is both slower than getting cached results (their REST API is not super fast and you are proxying through Drupal along the way), and leaves you open to a simple DOS where someone makes a bunch of calls and sucks up all your API requests for the day. Think carefully before limiting or removing those cache options (and make sure your cache actually works in production).  Setting a context of both URL and User can help ensure the right people see the right data at the right time.

Drupal Salesforce Suite Custom Field Mapping Types

The Drupal 8 Salesforce Suite allows you to map Drupal entities to Salesforce objects using a 1-to-1 mapping. To do this it provides a series of field mapping types that allow you to select how you want to relate the data between the two systems. Each field type provides handling to help ensure the data is handled correctly on each side of the system.

As of this writing the suite provides six usable field mapping types:

  • Properties — The most common type to handle mapping data fields.
  • Record Type — A special handler to support Salesforce record type settings when needed.
  • Related IDs — Handles translating SFIDs to Drupal Entity IDs when two objects are related in both systems.
  • Related Properties — For handling properties across a relationship (when possible).
  • Constant — A constant value on the Drupal side that can be pushed to Salesforce.
  • Token — A value set via Drupal Token.

There is a seventh called Broken to handle mappings that have changed and need a fallback until its fixed. The salesforce_examples module also includes a very simple example called Hardcoded the shows how to create a mapping with a fixed value (similar to, but less powerful than, Constant field).

These six handle the vast majority of use cases but not all.  Fortunately the suite was designed using Drupal 8 annotated plugins , so you can add your own as needed. There is an example in the suite’s example module, and you can review the code of the ones that are included, but I think some people would find an overview helpful.

As an example I’m using the plugin I created to add support for related entities to the webform submodule of the suite (I’m referencing the patch in #10 cause that’s current as of this writing, but you should actually use whatever version is most recent or been accepted).

Like all good annotated plugins to tell Drupal about it all we have to do is create the file in the right place. In this case that is: [my_module_root]/src/Plugins/SalesforceMappingField/[ClassName] or more specifically: salesforce_webform/src/Plugin/SalesforceMappingField/WebformEntityElements.php

At the top of the file we need to define the namespace, add some use statements.

<?php
 
namespace Drupal\salesforce_webform\Plugin\SalesforceMappingField;
 
use Drupal\Core\Entity\EntityInterface;
use Drupal\Core\Form\FormStateInterface;
use Drupal\salesforce_mapping\Entity\SalesforceMappingInterface;
use Drupal\salesforce_mapping\SalesforceMappingFieldPluginBase;
use Drupal\salesforce_mapping\MappingConstants;

Next we need to provide the required annotation for the plugin manager to use. In this case it just provides the plugin’s ID, which needs to be unique across all plugins of this type, and a translated label.

/**
 * Adapter for Webform elements.
 *
 * @Plugin(
 *   id = "WebformEntityElements",
 *   label = @Translation("Webform entity elements")
 * )
 */

Now we define the class itself which must extend SalesforceMappingFieldPluginBase.

class WebformEntityElements extends SalesforceMappingFieldPluginBase {

With those things in place we can start the real work.  The mapping field plugins are made up of a few parts: 

  • The configuration form elements which display on the mapping settings edit form.
  • A value function to provide the actual outbound value from the field.
  • Nice details to limit when the mapping should be used, and support dependency management.

The buildConfigurationForm function returns an array of form elements. The base class provides some basic pieces of that array that you should plan to use and modify. So first we call the function on that parent class, and then make our changes:

 /**
   * {@inheritdoc}
   */
  public function buildConfigurationForm(array $form, FormStateInterface $form_state) {
    $pluginForm = parent::buildConfigurationForm($form, $form_state);
 
    $options = $this->getConfigurationOptions($form['#entity']);
 
    if (empty($options)) {
      $pluginForm['drupal_field_value'] += [
        '#markup' => t('No available webform entity reference elements.'),
      ];
    }
    else {
      $pluginForm['drupal_field_value'] += [
        '#type' => 'select',
        '#options' => $options,
        '#empty_option' => $this->t('- Select -'),
        '#default_value' => $this->config('drupal_field_value'),
        '#description' => $this->t('Select a webform entity reference element.'),
      ];
    }
    // Just allowed to push.
    $pluginForm['direction']['#options'] = [
      MappingConstants::SALESFORCE_MAPPING_DIRECTION_DRUPAL_SF => $pluginForm['direction']['#options'][MappingConstants::SALESFORCE_MAPPING_DIRECTION_DRUPAL_SF],
    ];
    $pluginForm['direction']['#default_value'] =
      MappingConstants::SALESFORCE_MAPPING_DIRECTION_DRUPAL_SF;
    return $pluginForm;
 
  }

In this case we are using a helper function to get us a list of entity reference fields on this plugin (details are in the patch and unimportant to this discussion). We then make those fields the list of Drupal fields for the settings form. The array we got from the parent class already provides a list of Salesforce fields in $pluginForm[‘salesforce_field’] so we don’t have to worry about that part.  Since the salesforce_webform module is push-only on its mappings, this plugin was designed to be push only as well, and so limits to direction options to be push only. The default set of options is:    

'#options' => [
    MappingConstants::SALESFORCE_MAPPING_DIRECTION_DRUPAL_SF => t('Drupal to SF'),
    MappingConstants::SALESFORCE_MAPPING_DIRECTION_SF_DRUPAL => t('SF to Drupal'),
    MappingConstants::SALESFORCE_MAPPING_DIRECTION_SYNC => t('Sync'),
 ],

And you can limit those anyway that makes sense for your plugin.

With the form array completed, we now move on to the value function. This is generally the most interesting part of the plugin since it does the work of actually setting the value returned by the mapping.

  /**
   * {@inheritdoc}
   */
  public function value(EntityInterface $entity, SalesforceMappingInterface $mapping) {
    $element_parts = explode('__', $this->config('drupal_field_value'));
    $main_element_name = reset($element_parts);
    $webform = $this->entityTypeManager->getStorage('webform')->load($mapping->get('drupal_bundle'));
    $webform_element = $webform->getElement($main_element_name);
    if (!$webform_element) {
      // This reference field does not exist.
      return;
    }
 
    try {
 
      $value = $entity->getElementData($main_element_name);
 
      $referenced_mappings = $this->mappedObjectStorage->loadByDrupal($webform_element['#target_type'], $value);
      if (!empty($referenced_mappings)) {
        $mapping = reset($referenced_mappings);
        return $mapping->sfid();
      }
    }
    catch (\Exception $e) {
      return NULL;
    }
  }

In this case we are finding the entity referred to in the webform submission, loading any mapping objects that may exist for that entity, and returning the Salesforce ID of the mapped object if it exists.  Yours will likely need to do something very different.

There are actually two related functions defined by the plugin interface, defined in the base class, and available for override as needed for setting pull and push values independently:

  /**
   * An extension of ::value, ::pushValue does some basic type-checking and
   * validation against Salesforce field types to protect against basic data
   * errors.
   *
   * @param \Drupal\Core\Entity\EntityInterface $entity
   * @param \Drupal\salesforce_mapping\Entity\SalesforceMappingInterface $mapping
   *
   * @return mixed
   */
  public function pushValue(EntityInterface $entity, SalesforceMappingInterface $mapping);
 
  /**
   * An extension of ::value, ::pullValue does some basic type-checking and
   * validation against Drupal field types to protect against basic data
   * errors.
   *
   * @param \Drupal\salesforce\SObject $sf_object
   * @param \Drupal\Core\Entity\EntityInterface $entity
   * @param \Drupal\salesforce_mapping\Entity\SalesforceMappingInterface $mapping
   *
   * @return mixed
   */
  public function pullValue(SObject $sf_object, EntityInterface $entity, SalesforceMappingInterface $mapping);
 

But be careful overriding them directly. The base class provides some useful handling of various data types that need massaging between Drupal and Salesforce, you may lose that if you aren’t careful. I encourage you to look at the details of both pushValue and pullValue before working on those.

Okay, with the configuration and values handled, we just need to deal with programmatically telling Drupal when it can pull and push these fields. Most of the time you don’t need to do this, but you can simplify some of the processing by overriding pull() and push() to make sure the have the right response hard coded instead of derived from other sources. In this case pulling the field would be bad, so we block that:

  /**
   * {@inheritdoc}
   */
  public function pull() {
    return FALSE;
  }

Also, we only want this mapping to appear as an option if the site has the webform module enabled. Without it there is no point in offering it at all. The plugin interface provides a function called isAllowed() for this purpose:

  /**
   * {@inheritdoc}
   */
  public static function isAllowed(SalesforceMappingInterface $mapping) {
    return \Drupal::service('module_handler')->moduleExists('webform');
  }

You can also use that function to limit a field even more tightly based on the mapping itself.

To further ensure the configuration of this mapping entity defines its dependencies correctly we can define additional dependencies in getDependencies(). Again here we are tied to the Webform module and we should enforce that during and config exports:

  /**
   * {@inheritdoc}
   */
  public function getDependencies(SalesforceMappingInterface $mapping) {
    return ['module' => ['webform']];
  }

And that is about it.  Once the class exists and is properly setup, all you need to do is rebuild the caches and you should see your new mapping field as an option on your Salesforce mapping objects (at least when isAllowed() is returning true).