What I Brought from Drupal to Salesforce

If you look over this blog, or know me, you will see that I’ve now have reasonably significant experience as both a Salesforce and Drupal developer. The last couple weeks I have been thinking about what from my Drupal experience supports my work as a Salesforce developer.

I think there are three parallels that are encouraged in Drupal developers that helped me learn to be a good Salesforce developer quickly.

  1. Embrace, don’t fight, Platform Constraints
  2. Extend the Platform’s Strengths
  3. Leverage Events

Embrace Salesforce Platform Constraints

Both Drupal and Salesforce run in constrained environments. Web applications, regardless of their purpose, have to protect themselves against bad actors and bad neighbors. There are execution timeouts and memory limits, for both platforms. Salesforce adds a variety of additional limits and governors, but they are all logical extensions about memory and time. Lots of other platforms allow developers to ignore resource use until they reach a crisis point. Don’t believe me, just check the memory used by Chrome, Electron Apps, or any other Chromium-based application.

Working in resource constrained environments forces you to think through how to use the resources you do have efficiently. While these platforms aren’t like working on hardware with highly limited resources, they still can test your resourcefulness.

Drupal and Salesforce both provide ways to run large jobs across many processing contexts. New developers on both platforms often only resort to using batch and queueable operations as a last resort, but learning to use those solutions is critical to success on most interesting projects. When you try to avoid them you create solutions that appear to work and fail at scale.

Coming to Salesforce from Drupal, I already knew and understood the importance of asynchronous batch processing. So when solutions needed batch processing it was second nature to learn that part of the platform.

Extend the Platform’s Strengths

For all Salesforce’s push and marketing to avoid code, Salesforce developers are often taught that once you write code you just do everything in your code. The interfaces you can use to extend the platform’s existing solutions are treated as advanced topics. But when you work with Drupal, you are encouraged from the start to create modules that build on and extend the platform’s existing strength. Drupal developers are encouraged to leverage the features and utilities all around them.

This has always been true, but even more after Drupal’s move to leverage Symphony plugins and services. As a developer used to extending the platform, I came to Salesforce looking for ways to extend the platform’s declarative tools.

Often Salesforce developers create powerful solutions built purely in code triggered by record changes or simple buttons. They look passed Apex Actions that extend the Flow declarative automatons, platform events, and other tools that extend the system. But when you embrace a platform’s basic structures you often create more flexible solutions than your could with pure code.

The mind set of extending a platform, which I brought with me from my Drupal work helps me create tools and solutions that are designed to adapt over time.

Leverage Events

Event driven architectures are not new, but their popularity continues to grow. Where platforms used to follow informal patterns that equated to event systems, now we see formal event structures being build to replace old habits.

Drupal and Salesforce both have had event frameworks for a long time: Drupal had hooks (events by naming convention), Salesforce had triggers.

Both have seen major upgrades to their event patterns in recent versions. Platform events in Salesforce, still making their full power clear to a lot of developers. Symphony brought proper events to Drupal in version 8 and continue to help push the platform forward.

I have learned to leverage the events systems on both platforms. Understanding them as tightly constrained state machines, and learning to push them to their limits, helps me get the most from both platforms.

My experience with Drupal hooks and events has made it obvious to me when to leverage Salesforce’s Platform events. As Salesforce increases the number of places you can use them in their declarative tools, it increases the value of this approach.

So What?

As a developer, what you learn in one part of your career can make you stronger in the next. As a field we’re not actually that creative. Even if the details are different, the concepts will often carry forward because they are built on the same fundamentals. Whatever platform you are using today, learn how to make it sing – it’ll help you learn the next faster and better.

My Ongoing Liberal Arts Education

I consider listening to good podcasts to be part of my ongoing liberal arts education. We all can benefit from being conversant on many topics, understanding their interplay, conflicts, and mutual inspiration. That broad understand allows us to be more informed voters and more flexible contributors at work. It also just makes me happier.

I make a point of mostly listening to things outside my field of work. I study my work 40+ hours a week already. In my spare time I like to learn other things. Right now my rotation includes shows like Hidden Brain, People I Mostly Admire, Freakonomics, Wait Wait Don’t Tell Me, and more.

I tend to listen to things in fits and starts and then suddenly in waves. If I walk the dogs alone I’ll listen to parts of an episode. While mowing the lawn I’ll squeeze in a couple things. But it’s really the long drives that let me pack in a lot of information. For example, at the end of June I had to do a 12 hour drive on my own, twice. So I got caught up on some episodes that had been building up on my phone.

One of the things I like about listening to these shows is different perspectives on different topics. They provide me a chance to hear overlapping ideas that inspire new ideas related to my work or life. Sometimes those overlaps build on each other, other times conflicts in ideas push me to think deeper about both sides.

This I listened to a People I (Mostly) Admire interview with Philosopher Will MacAskill. They talked about extreme long-term view of morality, effective altruism, and the limits of growth economics.

Currently, economic growth is like 2 percent, 3 percent per year. What happens if that continues for just 10,000 more years? Well, 2 percent compounded over such a long time gets to a very large number indeed where it would mean that for every atom within 10,000 light years — so, every accessible atom within that time would have to produce the economic output of some enormously large number. I think it’s something like 10 to the power of 60. But just think trillions times trillions times trillions of amounts of output as the entire world economy today. And now, I’m not claiming I’m certain that’s impossible. The world today is magical and fantastic and would be judged as such from the perspective of people 1,000 years ago. But it seems really unlikely that every single atom is able to produce many trillion times the economic output of the world today.

Will MacAskill

It was a compelling enough argument to get a Chicago Economist to question his assumption that economic growth is a given.

But it’s funny because, as an economist, I’ve been so indoctrinated into this idea that economic growth is natural, it’s good, it’s part of life, that just something should have been totally obvious to me, but wasn’t because of my indoctrination, all it took was a few lucid arguments from you.

Steve Levitt

During the drives in June, I wandered through several hours of Hidden Brian. The episode that stuck with me was, Do Less featuring Leidy Klotz talking about the power of subtraction from designs. But it’s not just the singular interviews that I find useful. Listening to large amounts of divergent information causes ideas to cross pollinate. The Klotz interview connected in my head with an episode of People I (Mostly) Admire featuring Dan Gilbert, Turning Work into Play, which included the advice to do less, better.

Well, I’m loath to give advice because I’m telling you what worked for me. And that doesn’t necessarily mean it works for anybody else. But I do suspect that many, many people would be much happier if they did less, better.

Dan Gilbert

That whole line of thinking has me trying to re-frame how I talk about building tools as part of my job. How can I make my work better by removing tasks? I even started to write a response piece in my head, which may come together some day.

As a Salesforce Developer (and a Web Developer before that, and a Nonprofit Communications staff person before that), I get encouraged to listen to a very small set of information. People assume I listen to Salesforce Podcasts and not much more. Many of us are encouraged to spend our personal time learning about our work – usually narrowly about the sub-field we’re in.

My point isn’t that they are always right, nor that I’ve picked shows presented by people I always agree with. The guests and hosts in the podcasts often express interesting ideas, but also have view points I disagree with or ideas that conflict. For example, I find that Steve Levitt (who I mostly admire) has an overly simplistic view on gun violence. Shankar Vedantam often has guests on Hidden Brain whose ideas conflict with other guests. My point is that listening to these ideas, letting them challenge my thinking, and building on them my life better.

Life is big and complicated, and involves a lot of not-work. Go find some things worth learning about that aren’t part of your regular activities. See where it takes you.

Pro-Choice in Aiken, SC

Last month the Supreme Court’s Dobbs ruling overturned Roe vs Wade, and ended the right for millions of women to get critical medical care (that care being blocked includes, but is not limited, to abortions) and threatening additional rights all Americans currently enjoy. In the days that followed I joined a small but vocal group of people from Aiken calling for protection of the rights of women’s and others. It’s taken me a few weeks to get these pictures off my camera and posted, but I did want to get them up for anyone who is interested. There are people in South Carolina who dissent.

Getting Started with Salesforce2Sql

Salesforce2Sql is a tool dedicated to doing one thing very well: mirroring Salesforce schema.

A little over a year ago I started work on Salesforce2Sql to help support people doing data work with Salesforce. Salesforce2Sql is a simple Electron app that allows you to clone the schema of a Salesforce org into an SQL database (currently supports MySql/MariaDB and Postgres). These mirrors are useful when you want to stage data during migrations in or out of Salesforce.

When is Salesforce2Sql useful

Salesforce2Sql is a tool dedicated to doing one thing very well: mirroring Salesforce schema. It does not attempt to extract data or convert data in any way.

Salesforce provides excellent APIs for data import and export, but they work best with some prep work first. Salesforce2Sql gives you a staging database that mimics your Salesforce schema. In these schema mirrors you can prepare all your data for high speed processing off-platform. Anyone who is looking to move data into or out of Salesforce at high speeds and large volumes benefits from this setup.

I have seen people write large complex ETL jobs meant to go right from one source system into Salesforce. These jobs can be hard to test and they are slow to run. They generally don’t give you an easy way to review the data before you push to Salesforce. By landing the data in a database clone you can break the jobs into stages, review transformations before they are loaded, and run thousands of iterations for testing instead of dozens.

Setup

Salesforce2Sql is built and released for MacOS, Windows, and Linux (most testing is on Mac and Windows). You can download the installers from the current release and follow the standard patterns for your OS. From there start the application to get to work.

You will also need API access to a Salesforce org and a database to create your schema within.

Basic Salesforce2Sql Use

As of this writing Salesforce2Sql uses the old security token connection method. I would like to add OAuth2 support as well  but haven’t gotten that done; contributions are welcome. So with the application running, and your security token in hand, click the big “Create New Connection” button on the left side of the main interface.

Salesforce2Sql Main Screen

Step 1: Fetch all objects

Once connected click “Fetch Objects”, and the tool will download a list of every object in your org. There will be several hundred. So the next step is to select which you want to mirror. You will notice Salesforce2Sql selected defaults for you (well I did). It will select all custom objects and based on your org’s structure Salesforce2Sql guess which standard objects to select.  There is a search box at the top right to help you find any others you’d like to add but simply checking the box.

Step 2: Fetch all fields

Click the next button to move to the Proposed Schema tab, and then the “Fetch Details” button. Now Salesforce2Sql will query every field on every object you just selected (this may take a moment but honestly I find it much faster than I expected when I first started this project). Once that is complete you can either save that schema to JSON for later re-use, or click Next to move to the “Generated Database” tab.

Step 3: Generate tables

This is the last step. Click “Create Tables” and Salesforce2Sql will ask you for your database credentials. Once you click okay on this final screen the tool will attempt to create all those tables for you. Again this will take a couple minutes if you have a large schema.  Once the process is complete you can also save the SQL statements for editing and/or later re-use.

That’s it, you now have a database with a schema that matches your org’s structure.

Preferences

There are a few preferences you might want to experiment with (although I tried to pick smart defaults) when building mirrors. For me the right choices depend on my use case.

Preference screen from the application with sections for picklist settings, index settings, other defaults, and theme.

Picklists

Salesforce Picklists are a bit of a special beast. The obvious choice is to make a picklist into a SQL Enum to support validation of data. But not all picklists are restricted in Salesforce and aren’t always required. By default the tool will use enum for restricted picklists, varchar for unrestricted picklists, and add blank values to all picklists (since it can’t easily determine if a given picklist is required or not across all page layouts).

If the picklist values in your org are pretty much set, the default settings make a lot of sense. If the picklist values in your org are likely to change you might want to make them all into regular varchar columns.

Auto Indexing

The next important section contains the index settings. While it is possible to over-index a database I think the three sets of default indexes are pretty good guesses: Id columns, external Ids, and picklists. The one you are most likely not to care about are the enums, and therefore the one you might consider disabling if you aren’t processing on those fields at all. Id columns are now case-sensitive even in MySQL by default as of version 0.7.0 since Salesforce Ids are case sensitive.

Additional Settings

The other defaults section let’s you pick a few other system behaviors. The most common to fuss with are first and last in the box.

By default it expects Lookup fields to be 18 character Salesforce Ids – cause that’s what they will be in Salesforce. But during a data migration some people like to put legacy Ids into these fields (I recommend a proper legacy Id field marked as an external Id) so I give the option to use 255 characters instead of 18. 

Salesforce also has two categories of fields that are common to ignore in a migration and you may wish to keep out of your database just for ease of use: the audit fields (createdBy and the like) and read-only fields (like formulas). The final two checkboxes in that other defaults section lets you keep those fields out of your clone schema.

The middle two fields control the behavior of field defaults – generally I like these two settings as is in just about every use case, but you may feel differently.

Salesforce2Sql uses Bootswatch themes for design elements. The preference pane also lets you pick a different look-and-feel from their theme list.

Final Notes

There are a couple other details worth knowing.

First, if the process runs into a problem where the SQL engine complains about row size limits Salesforce2Sql will automatically switch all varchar fields to TEXT fields in an attempt to reduce the row size. That will override all preference settings for these fields.

Second, Knex.js – which Salesforce2Sql uses to handle the actual SQL writing – adds indexes as a table alter even when they could be part of the create statement. This makes the process a bit slow and it means that if there are errors during the creation of indexes you may see some errors in the interface but leave you with a pretty-good schema clone.

Finally, yes there is lots of room for improvement. I work on this project when I can, or when I need a bug fixed for my own work. I am excited to get suggestions, ideas, feedback, documentation edits, and code submissions.

Solve Interesting Problems

During a recent department event my wife introduced me to the sister of one of her students. The event was an award ceremony for some of the history and political science majors – the student’s sister was along to support her brother (and as a smart college student get a free meal outside the dinning hall).

As a CS major, this student is trying to understand her options for what kind of programming she might be interested in. As a good professor my wife introduced us so the CS major wouldn’t have to pretend to be as excited about history as everyone else present. Listening to me talk about what I’ve done in my work she commented that maybe she should be a web developer – my reply was that she should do the work with problems she finds interesting.

Dodge the Gate Keepers

Like many people who are going through a CS education she has been surrounded by people who are confused about the difference between IT and CS. She talked about going to a conference and running into a bunch of guys who belittled her because she wasn’t into computer hardware. Apparently one even criticized her for misstating the directionality of a Lightning adapter (I can’t remember the last time I cared about cable directionality in a digital connector). My suggestion that was old, familiar, and involved one finger. I also pointed out the guy was probably just wrong.

That kind of adolescent gate keeping out of other college students isn’t surprising, but it is annoying. I work in a field that’s short handed, and we need smart people interesting creating great systems. We were short handed before the whole U.S. economy started to run short on workers.

From a short conversation I could tell she was smart, capable, and friendly – exactly the kind of person any employer will be lucky to have some day soon. But she also felt discouraged, as if she was weak in some important part of the field. Assembling her own PC hadn’t been fun for her (I have no shame in admitting that I’ve never built a PC from scratch); fussing with hardware just doesn’t excite her like coding does right now.

I really love having a good IT team to support my work. And having great hardware at my disposal is critical to good work. But I have minimal interest in working on that part of the technology stack myself. Sure, I’ve done my time installing RAM chips onto mother boards, and re-seating PCI cards, but I never really wanted to care about the details of those components.

I always wants to create tools that solved interested problems.

What Problems are Interesting

We are all attracted to ideas and projects that sound exciting. This student became interested in the kind of work I do because I can talk about it with excitement and confidence. I enjoy the problems I get to solve on a day-to-day basis and that shows. I have no idea if she’d enjoy them. Being a Salesforce or web developer might bore her to tears.

A problem is not intrinsically interesting. We find problems interesting for our own reasons. That interest makes us intrinsically motivated to solve them.

I like writing middle-ware and creating related tools. Filling gaps left between other tools is interesting to me. I know people who love to create great UIs because it makes people love the product. Other friends love to work on security problems because it keeps systems secure (and gives them excuses to break into systems they should access). Some of my friends work on creating software to advance science. Still others love to create high performance solutions to handle big data problems. And others who help create games. I could go on, byt you get the picture.

My point is all the problems are interesting – to someone. None of the problems are interesting to all of us.

If you want to have or want a career creating software, look for jobs that solve problems you think are interesting. It doesn’t matter if I think your work is exciting. If you are excited about it, I’ll be excited to hear what you’re doing.

Salesforce Developer Podcast Episode 119

This week’s Salesforce Developer Podcast featured an interview I did with the host, Josh Birk, the end of last year. As much as I still don’t like the sound of my voice on recordings it was a fun interview and I am really excited to see it come out.

We talk about Snowfakery, Salesforce Open Source Commons, the evolution of PHP, Drupal, my career in general, and even a bit about spinning. I’d love to hear what you think.

Supportive Open Source Projects

Data Generation Toolkit Logo features Loinheart Astro with bits falling on him above the project label.

For a little over two years I’ve had the privilege of helping lead the Data Generation Toolkit Project for Salesforce. In general, Open Source projects are not known for their inclusive and supportive communities. I believe it is fair to say our project demonstrates that building a supportive community can yield great results.

Started by a question I wrote on a piece of paper in 2019 and posted to a wall, the project grows more every time I look around. That first meeting inspired the creation of Snowfakery and the recent training for Snowfakery has attracted more than 300 registrants. Contributors created documentation and presentations to help Salesforce admins learn to seed sandboxes. We are building a recipe library to help people starting out on Snowfakery. We launched two faker data providers. This year we reviewing and documenting other tools to generate and move Salesforce data.

More importantly we’ve built a project that is useful to the community, and supports new contributors to open source projects.

We did not get here accidentally. The project leadership wants to support the community members as much as create new tools. From the beginning we chose to encourage people unfamiliar with open source projects, contributions, and technologies.

Leading an Supportive Open Source Project

To be a supportive project starts with the leaders.

The project currently has seven identified leaders: Alisa Edwards, Allison Letts, Jung Mun, Paul Prescod, Samantha Shain, Cassie Supilowski, and myself. For those who like diversity statistics: that can be seen as 5 women, 2 men, 3 countries of residence, 3 counties of origin (not the same 3), 3 developers, 4 Salesforce admins, and at least 3 racial identities. No sub-group perfectly reflects its community, but that’s not a bad start in my opinion.

We have agreed that some of us lead specific sub-projects, while others tend to the health of the community. Everyone has a role.

When we started out with just 4 leaders either Paul (as creator and maintainer of Snowfakery) or I (as the person who started the project) could have dominated the project claiming founder status. Certainly men leading other open source projects have used that status to control the project direction. But Salesforce Open Source Commons projects are designed to discourage that behavior, and Paul and I embraced that design. Cori O’Brien created a wonderful space for our project to grow within.

Any open source project should focus on supporting its users. Our goal as a project is to support the Salesforce community – particularly nonprofit and education users. To do that we need the insights that only come from being open to outside ideas.

Our project’s leadership also established a pattern of self-review and reflection. Each year we will gather to discuss if the leadership group is the right size, and if anyone needs to step down. That creates a space for us to routinely reflect on what we’re doing and if we’re doing it well. The invitation to leave frees people from responsibilities they have to the project without frustration or burn out.

Recipes for Open Source Projects

Beyond our leadership team structure we also are intentional about how we encourage all contributors.

The project as a whole has space for all kinds of contributions. If someone wants to write documentation we will support them. When someone wants to learn to write recipes, we will work in pairs to get through the first one. People who want to write Python code for our faker providers get the chance to do that too.

We try to be kind to all contributors. We thank everyone when they open pull-requests or issues. Even I get thanked when I open a PR. Even if someone needs to make significant changes to a contribution before we commit it, we make sure to praise their effort.

These aren’t shit sandwiches. We genuinely appreciate all contributions and divorce that from any corrections that we request. As a long time open source contributor I am surprised at how much I appreciate the messages.

Maintaining Supportive Discussion Channels

Most open source maintainers know we have to maintain good ways for contributors to ask questions. Over time projects have used a wide variety of tools: IRC, News Groups, Email, issue trackers, and now Slack. In most projects those are the spaces that tend to become ugly. You see RTFM-style answers, personal criticisms, identity-based attacks, and other ugliness. The Slack channels for our project are a key piece of how we engage with each other, and support new contributors. We work hard to make sure people get timely answers, clear directions, and steady encouragement.

Our work falls within a community where our reputations matter and so there is a level of decorum absent in some open source projects. But no group is perfect and we will address issues when they arise. The open source common’s DEI Framework project will hopefully help us continue to deepen our understanding of the community. The kinds of attacks open source community spaces frequently allow in the name of good code, are simply unacceptable.

Come Join Us

The next community sprint for our project is coming up May 4th & 5th.

Still a B Student

Now and then something happens to us all that reminds us we’re not perfect. I’ve written about the strengths of B Students before, and well last week I proved that sometimes I still have room to improve.

When I started this blog I set the goal of posting once a month. I have stuck with, or exceeded, that target except for two months since 2016. I missed September 2019 and last month. Last time around I set the goal of four posts in the following month, so that is my established case law.

I did have a piece written that I had planned to release. But it needed another editing pass. Before I got those edits done a project at work took over my life. I will have those edits done for next weekend, and I already know the general topic of the post what will follow it.

Last time the increased pressure to write actually resulted in some descent pieces. I wrote that B Student piece, which I quite like, in part as a response to that failure. Two others from that month are how-to articles that continue to get good readership years later.

Hopefully this will be an equally good month (although this post probably isn’t the best example of that).

Disable a Salesforce Trigger in Production with VS Code

I recently had to disable a Salesforce Trigger from a client’s production environment. Having discovered the need last minute for a project, I needed to react quickly and make the change quickly. Since the official documentation is a bit lacking I decided it was time for blog post of better directions.

The Salesforce CLI directions in the article above make a few annoying assumptions:

  1. That you’re happy to use MDAPI not SFDX.
  2. That all your tests pass in production (which should be true, but let’s be real it;s always).
  3. You enjoy working out CLI commands in a rush.

What you need

Disabling a Trigger Using SFDX in VS Code.

Connect VS Code to your target org. Easiest solution here is to just go to the command palate and authorize an org.

VS Code command pallet wit with SFDX commands shown.

Pull down the trigger from your org. I find it easiest to go to the metadata browser, find the trigger under Apex Triggers and click the download icon on the right.

Screenshot of VS Code org browser showing an Apex Trigger to download.

Update the trigger’s XML file to change the status. All project code files in SFDX are accompanied by an XML file of class metadata. Open the file and change the status to “Inactive”.

Screenshot of the VS Code file browser with the trigger's metadata file highlighted.
Navigate to the file in the file browser on the left in VS Code
Screenshot of code snippet emphasizing the Status XML tag and Inactive value.
In the editor change the status from Active to Inactive, and save the file.

Generate a Manifest file for your trigger. Right click on the trigger code file in VS Code’s file explorer and select Generate Manifest, provide the file a useful name (in this case used DisableTrigger)

A segment of the contextual menu to show Generate Manifest command's location.
Depending on your setup the context menu might be quite long, SFDX additions are generally near the end of the list.

Deploy the change. If all your org’s tests currently pass you can just right click on the new manifest file and instruct VS Code to deploy the source in the manifest to the org.

Another contextual menu snippet, now with Deploy Source In Manifest circled.

What if tests fail on trigger deploy?

While all our tests should always pass all the time, Salesforce admins frequently find that’s not actually true in practice. Heck this trigger could be part of that problem in your org. But to deploy a code change we have to run some tests. Since we are disabling this trigger, the trigger code doesn’t need to be in tests we just need to run a working test with enough coverage to get the job done. So go find a test class and then we can deploy.

VS Code doesn’t currently have a setting to set the testing mode on a deployment, so you’ll need to do this last step in VS Code’s terminal (available by hitting control-` if you don’t have it open already). In the terminal run the following command (replacing [good_tests] with the name of your test class).

sfdx force:source:deploy -x manifest/DisableTrigger.xml -l RunSpecifiedTests -r [good_tests]

It should deploy pretty quickly, but you can check the status by going to settings in your org and checking the Deployment Status.