Announcing Two New Snowfakery Faker Providers

For the last two years I’ve been fortunate to serve as a leader of the Salesforce Open Source Commons Data Generation Toolkit project. That project has produced and inspired a variety of efforts, including Snowfakery and a collection of starter recipes.

This week, along with my colleague Allison Letts, Salesforce’s Paul Prescod (the creator of Snowfakery), and our fellow project contributor Jung Mun, I helped create two new faker providers for Snowfakery:

What These Faker Providers do

The use of Snowfakery is growing. The more we use it, the more we want the data tailored to specific projects. And the more we find places where the Faker project’s providers do not have quite what we want. In particular the project does not (well did not) have providers for nonprofits and education specific data.

The Nonprofit provider currently just provides organization names:

$ snowfakery snowfakery_nonprofit_example.recipe.yml --target-count 10 nonprofit
nonprofit(id=1, nonprofit_name=Eastern Animal Asscociation)
nonprofit(id=2, nonprofit_name=1st Animal Foundation)
nonprofit(id=3, nonprofit_name=Upper Peace Alliance)
nonprofit(id=4, nonprofit_name=Southern Peace Home)
nonprofit(id=5, nonprofit_name=Unity Home)
nonprofit(id=6, nonprofit_name=Western Peace Home)
nonprofit(id=7, nonprofit_name=Upper History Foundation)
nonprofit(id=8, nonprofit_name=Upper Friends Committee)
nonprofit(id=9, nonprofit_name=Eastern Pets Center)
nonprofit(id=10, nonprofit_name=Northern Animal Foundation)

For the Education provider we have a bit more. You can generate college names, departments, and faculty titles.

$ snowfakery snowfakery_edu_example.recipe.yml
Account(id=1, Name=South Carolina University)
Contact(id=1, FirstName=Roberto, LastName=Stanton, Title=Associate Professor of Microbiology & Immunology)
Account(id=2, Name=French)

The providers can of course run as a standard faker community provider. Once you are setup with Faker just add the new providers with pip:

pip install faker-edu faker-nonprofit

Then you can use the libraries in your code:

from faker import Faker
import faker_edu

fake = Faker()
fake.add_provider(faker_edu.Provider)

for _ in range(10):
    print(fake.institution_name())
from faker import Faker
import faker_nonprofit

fake = Faker()
fake.add_provider(faker_nonprofit.Provider)

for _ in range(10):
    print(fake.nonprofit_name())

How we got here

A few months ago I posted on how to extend Faker to create nonprofit organization names. Allison took that as a starting point to create a similar project to generate names of colleges, departments, and academic titles (and a greatly improved phone number generator but that’s not included since it’s more general). Both of these projects were good proof-of-concept but were rough around the edges. So this week, with Paul’s guidance and Jung’s input, we contributed the more polished versions to the community.

During a virtual working session on Wednesday we restructured the projects, cleaned up code, added sample recipes for Snowfakery, and published them to PyPi. By publishing these as Faker providers on PyPi, and not as Snowfakery plugins, they are available to a wider audience. By having them owned by a larger open source community we are expecting them to enjoy long-term support.

Both are still just getting started. For example the nonprofit one still just generates organization names, but would benefit from job titles, program names, and more. The EDU provider right now just handles colleges, but is expected to generate other education related data in the future. We also have a plan for a third provider to help improve the diversity of the names generated by Faker to make our fake data more representative of real communities.

The Data Generation Toolkit project has been a great example of what happens when you bring people from a wide variety of backgrounds together to solve technical problems. Like the larger Data Generation Toolkit team Paul, Allison, Jung, and I all have different backgrounds, skills, and experiences. By coming together we are able to help each other find better solutions than any one of us would have found on our own.

It’s been an exciting week, and I’m looking forward to more to come.

Faith’s Joy

This is Faith.  Faith has found joy, happiness, and contentment – except when she needs her toenails trimmed, dinner is late, or she gets woken up for a nap.

Life for Faith wasn’t always easy. She spent a little time on the race track before they closed, and was among the last dogs to ever race in Florida. Her first adoptive home didn’t treat her well, and she bounced back to the rescue group underweight, with a cracked tooth, and uncontrolled multi-drug resistant worms. But those problems are gone (at least we hope, the worm treatment takes several months and we just got our first clean test last month – need three in a row). But Faith found a life she loves in our home. 

Two greyhounds cuddling on a dog bed.

She loves to cuddle with her brother. 

Greyhound in a room covered in shredded paper towel with a toy in her mouth.

She loves paper towels. 

Greyhound sleeping on a couch with her feet in the air.

She loves a good inverted nap.

Greyhound curled in a ball, sleeping on a chair.

She even loves hiking with us in the mountains of North Carolina (trust me on this one, she doesn’t hold still again until she’s napping when we are done).

Like many people I aspire to be more like my dog. She has handled life’s ups and down with grace (mostly) and forgiveness (graciously – even after I trim her nails). I hope to learn to be as openly loving as forgiving as her.

Build Cycles of Respect

In consulting we should expected developers to do all the same basic things as everyone else on the team. That includes supporting a culture of mutual respect with other team members.

We should explain our work clearly, be effective in meetings, and take feedback well. In consulting especially, we should engage with clients professionally, effectively, and happily. We should not be isolated off on our own teams working from specs we didn’t write, for clients we haven’t met. Basically, we need to be good team members and we can be expected to be that way. Developers, like most everyone else, do their best when when surrounded by people who treat them with respect.

One of the things I really like about my current job is that we have a respectful culture. It’s not that we don’t have disagreements, at times energetic ones, but we work through those challenges as a team of experts with differing perspectives. We are all expected to be experts who can collaborate with other experts, and to explain our work to clients.

Cycles of Disrespect

But that hasn’t been the consistent norm since I shifted to consulting. I have been on teams, and seen teams, that routinely insult the basic professional behavior of developers and other technologists. In those settings leaders made it clear that the standard of behavior was lower for developers than other team members. Most people I know end up adjusting their behavior to meet the standards set for them. Set a low standard of behavior get bad behavior.

I want to be clear that I have no intention justifying the harassment, bullying, and assault that is too common among developers (and more generally by men in the workplace) – that is not the challenge I’m addressing here (although I’ve addressed those issues before). There are behavioral problems in places that don’t permit and cover up that kind of behavior. Developers and other IT specialists are often still allowed to act as a grumpy trouble maker, or aloof superior jerk. We are told our brains work differently somehow allowing us and our colleagues to ignore healthy social conventions. And by allowing it, stating that we expect it, and taking no actions to correct it, we encourage that behavior.

Signs of Trouble

On more than one occasion, at more than one employer, I’ve been told things like:

  • “You’re talking techie now, I’m not listening anymore.”
  • “Developers hate meetings and never contribute productively.”
  • “You just have a thin skin and act defensive of your work when questioned.”

Often the person will laugh as they say these things like we have some kind of shared inside joke about some basic lack of professionalism. There are lots of reasons why people dismiss developers with those kind of statements, but that doesn’t stop them from being destructive.

When that’s your work environment, it’s tempting for developers to degrade to match expectations. Who takes feedback well when told upfront they have a thin skin? Who likes being in meetings with project managers who explicitly state their contributions aren’t welcome? What expert thinks it’s funny when you stop them and say you are not listening? 

This builds a cycle of disrespect between developers and their colleagues, dividing teams and impeding progress. When people are treated with disrespect while still performing critical tasks, I expect to see them act with disrespect is exchange. That is no okay, but it very human.

Breaking the Cycle

Respect is a two way street which makes it important to break the cycle on both sides. Everyone on a project needs to respect the roles we each have and the contribution we each bring. Those may be, and usually should be, vastly different across the team – otherwise there is no point to having a team at all.

As team members who are often in a position of some power – in part because we are hard to replace in the current job market – developers should take it upon ourselves to be the first to build respectful practices with colleagues.

We can, nicely, point out when we hear statements that feel isolating and rude. We need to try to understand why someone acts intimidated by the technical detail and find ways to help them along. Developers should make a point of soliciting feedback from team members and processing ideas with them. Everyone ought to build personal relationships across our teams to help improve everyone’s ability to work through hard problems (there is a bunch of research on this question).

Creating a Cycle of Respect

To build an ongoing cycle of respect take more than just breaking the old patterns. Developers are often in a position of leading by example and so that is a great first step. Asking friends and colleagues to make an effort with you will help build a re-enforcing cycle in a small group, which makes a great second step.

Understand that cultural change of an organization is hard, but usually it is possible on project teams. So it may help to start with just one team and making a real effort to improve communication and collaboration with the team. From there build out and try to draw others into your new patterns.

When teams work together well, understanding the project as a whole, they do better work. That puts everyone in a position to add ideas, raise concerns, validate suggestions, and adjust to changes.  When teams allow any member to work in isolation they losing the shared vision and project failure.

Being Nice vs Being Kind

A few years ago during a job interview at a nonprofit organization, the Executive Director asked me about the culture of the job I was leaving. He had noticed that I was stressing the importance of good feedback during my interviews and wanted to know why it was so important to me. Indeed, part of why I was job hunting was the job I had rejected the notion of mistakes – everyone’s work was always good. I told him that, and that I do my best work when I get honest feedback. He responded by drawing a distinction between being nice and being kind. He was sure everyone I worked with was quite nice, but they didn’t sound very kind to him.

Being nice when giving feedback just means saying positive things.
Being kind when giving feedback requires helping another person understand how to improve.

It was one of those moments in life where someone offers you words to explain a struggle you’re having and suddenly things make more sense. I knew I was unhappy in the job, but his insight brought into focus the reasons I wasn’t fitting in. Being told my work was great did not mesh with my understanding that it’s important to admit we all make mistakes and find ways to avoid repeating them. My desire to examine my own work caused conflict, let alone my efforts to introduce peer feedback.

The CEO at that job once told me that if people heard their work wasn’t perfect, they wouldn’t want to come in the next day. He didn’t seem to realize that getting dishonest feedback causes the same thing. And seeing no chance for improvement was even worse. There was no room to revisit problems that lead to near failure and ongoing technical debt for the client. Instead they convinced clients the technical debt was a feature, and used it to sell support contracts.

All the feedback my colleagues and I got was nice to hear (“This is great.”, “The client will be thrilled”). But unrelenting positive feedback didn’t help us improve as individuals or as a team. Our leadership gave team members a false sense of success and held the company back from improving.

The thing about being kind, is that sometimes you have to tell people things they don’t want to hear. That can be hard, and to do it well requires you to care about the other person. If want people to like everything you say, it will be hard to be kind.

Giving nice feedback involves offering a string of hollow platitudes that leaves people with a false sense of achievement. That makes people vulnerable to failure when they move forward without understanding their weak foundation. Giving kind feedback requires you to provide an honest assessment of a person’s work and helping them recognize errors. Kindness requires listening to people when they push back, and finding ways to help them find ways to excel as they move forward.

When all was said and done the opportunity wasn’t the right fit for me. Turning down their job offer was probably the hardest career decision I ever made. But they and I left good enough impressions with one another that I recently started serving as an volunteer advisor to the program I would have been running. That service has allowed me to reaffirm my sense that it was the right decision not to take the role both for me and for the organization.  And it has given me a chance to help give them some kind feedback to help – I hope – improve their operations. I try to be as kind to them as an organization as they were to me as an applicant.

College Advice from a Professor’s Spouse

My wife is a college professor. That means, in addition to being married to a very smart person, I also get to hear lots of tips and tricks about how to do well in college – particularly this last year with my wife doing office hours from home on video from her office just outside our kitchen. So as a service to all those headed off to college in the fall (including our oldest nephew), students already working to complete degrees, and to all the professors and instructors out there, I thought I’d offer a few pieces of college advice I’ve picked up as a professor’s spouse. I’m also blending in a few things I learned getting my own degree and from working professionally with colleges on a regular basis (basically, if you don’t like anything in here, blame me not my wife).

The first two are so painfully obvious I hate to have to tell them to people:

  1. Read the syllabus.
    I mean actually read it, not just look at the words (although that is a start). 95% of the time the professor will literally tell you exactly what you need to do to get an A; or if you’re smart and lazy it lays out how to coast to a C. Heck sometimes they even give you extra credit just for showing signs of reading it because they know most of your classmates won’t bother.
  2. Read all the instructions.
    Your professors want you to succeed. Their lives are better when you do good work. They like seeing students learn and grading good work. So for each assignment they give you instructions about what you need to do to get a good grade. Read and follow those instructions.

Okay now for the advice that’s not so obvious and may be harder to follow.

You earn grades, you aren’t given grades.

Your grade should reflect the work you produce in that course, in that semester, for that professor. You aren’t entitled to a good grade unless you do good work by the standards set for everyone in the class. If that’s not happening – well I’ll get to that in a minute. But don’t confuse a hard assignment, or a professor with high standards as a lack of fairness. Grades are fair as long as the professor holds everyone in the course to the same standards.

Also remember professors should not lower their standards just because you are majoring in another subject. Your philosophy grade should reflect your philosophy course performance even if you are a math major (and vice versa for the philosophy majors). Complaining that the assignment is unfair because you don’t write clearly is the wrong way to move forward – ask for help writing better when you need it.

Complain to your friends, not to your professors.

As I just said: yes, sometimes an assignment is hard. Yes, some professors make it hard to earn A’s. Sometimes you stay up all night, work so hard it physically hurts, and your paper earns a D. These are all great things to complain about to your friends, on your own time, in your dorm or apartment. I did in college, so did my wife, and all our college friends – that’s part of college and the learning process.

DO NOT:

  • Complain to your professor that the assignment was too hard.
  • Stand in the hallway outside their office and rail at the injustice of it all.
  • Complain to your other professors.
  • Rant on social media about how it was hard to study cause you were really drunk. (Seriously, professors can see the stuff you post on the school Facebook page – stop it! Aren’t you all supposed to use other platforms these days anyway?!?)

Remember that your professor knows how all your classmates performed. They may know some of your classmates started that paper two weeks ago, asked for advice along the way, and earned an A. They already know if an assignment was actually too hard (it does happen) and almost certainly took that into account when assigning grades.

Hard work and good work are not the same thing.

That paper you stayed up all night writing was probably hard to complete on time, and not your best work. Just because it was hard to do, doesn’t mean you did it well. In your non-college life this is most obvious in things like physical engineering and computer programming. If your phone’s app crashes all the time, it’s bad. If the bridge falls over, no one cares if the engineer was up all night checking their math. It’s also true on standard college work like papers and art projects – if your picture is overexposed or your paper poorly written the outcome was bad and the grade you earn should reflect that reality.

Ask for help – do not demand special treatment.

Most of the people doing classroom instruction in most colleges like helping students succeed (research professors can different – although not all, and you shouldn’t assume they don’t care). They have office hours so you can easily find them to ask questions. This is not so you can easily yell about your grades. Professors do not take well to being yelled at, threatened, and generally treated as if they are supposed to “give” you anything. Remember, you are not an education customer and the professors do not work for you. Never tell a professor you pay their salary; you don’t, and saying it won’t help matters.

Professors all know how to ace their classes. They should, they designed the class after all. Professors understand all the material they are teaching you (even if you don’t feel like it sometimes) that’s how they earned those advanced degrees – probably at a deeper level than you realize exists. If you are confused or struggling they are the best source of help around. They are good at this stuff – give them a chance to help you get there too.

Exceptions can, and sometimes will, be made.

Sometimes a professor they might cut you a break or offer extra help if you ask. But these favors should be asked for rarely and very carefully. Try to understand what you’re really asking them to do for you. Exceptions for you might be deemed unfair to others in the course and have to be explained to a dean. Changes in deadlines, taking incompletes, and other accommodations may require the professor to do extra unpaid work.

When I was in college I wanted to leave early one semester, to attend an international conference. Of my four courses that semester three had open-book exams or projects – so I just finished those before the deadline. The fourth was a Government course with a traditional exam. The professor was excited about the event, so he was willing to let me take the exam early.

That exception required him to trust me not to share details or to write a second exam just for me – neither are trivial requests. If I had asked to take an incomplete, so I could finish late instead of early, I would have been asking him to do even more work. Often that kind of extra work goes unpaid.

I didn’t have a lot of interruptions, and my exception was for an opportunity, not outside interference. I know not everyone is so lucky, and you may have good cause for asking for exceptions to course rules more often. But if you find yourself asking for those exceptions more than once a year, consider talking with your advisor or the school’s academic success center. Asking for too many favors is either a sign you’re expecting special treatment or need support finding a better overall life balance.

Note: Many schools offer some form of hardship accommodations for health challenges and other major life disruptions. Using those when you need them is not asking for a favor. Those are policies meant to help you succeed, not exceptions to the rules of a course. Check your student handbook when things are easy to understand how that works in case something goes really wrong.

Your classmates lie – a lot – so you have to prove you are different.

That syllabus I told you to read (seriously most important thing here – read it), is probably long and boring. Filled with classroom policies, honor codes, assignments, deadlines, and more. When I was in college they were 2-3 pages and nearly all about the assignments. My wife’s, and her colleagues’, have grown from there into a 10 page tome of policy. They hate it. You can also hate it. You should blame every classmate who goes to a dean arguing over a loophole they think they found. Expect that a new rule is written every time you hear another student say some silly thing to justify obviously bad behavior – particularly if sounds unfair to you and then works for them.

Every one of those rules you see probably has a specific former student to blame. Your professors would love to save time and energy by going back to having a 2 pager of assignments and deadlines. Trust me, they sit on my back porch and lament about it – while sharing suggestions about how to describe how often it’s okay to pee during an exam or whatever thing has come up this time.

Exam Tip: pee before the exam. If you cannot regularly go 90 minutes without a bathroom break, and don’t know why, check with a medical professional. After you do that, read the syllabus and student handbook again to check for policies on medical accommodations.

Yes, you may need to prove your loved one died.

If this happens to you, first and foremost, I’m sorry. Your professors are sorry too – even if they don’t show it in front of you. But students lie about this all the time. It goes in and out of fashion as the lie of choice, but it’s always around. It’s called Dead Grandmother Syndrome. There are actual studies on it (even if entertainingly written, it uses real data). That’s why your college has an actual procedure to report the death of a loved one.

My wife’s colleagues all have stories about assignments that were due just before a break that caused a wave of ill-health and death in student families. I think my wife’s record is 5 grandparents with one paper – her record is not very high. If an assignment is due the Wednesday before Thanksgiving or the day before Spring Break, a single paper might wipe out a dozen grandparents (usually grandmothers – see references above). Those people who wanted to leave for vacation a day early are why, when your life actually sucks, your professor may remind you to complete your school’s process.

That said, if you build up trust with you professors by being a reliable and good student, they will probably try to help you first and worry about the paperwork later.

Learn your college’s grade appeal process.

While we’re one the topic of policy let’s talk about times there are actual problems. Sometimes your professor is a jerk and is unfair. This is genuinely pretty rare, but it does happen and it’s bad when it does. Your college has a process for this. I don’t know what that process is, but it’s there and it’s probably in your student manual. Look it up it. Follow it. I can promise you it does not involve your parents calling anyone at the college – if that’s part of your (or their) plan, start again.

If your professor crosses lines beyond being unfair with grades, at least in the U.S., your college is required to have a policy for that as well. But it will be easier to deal with the process if you have support. Go see a professor you like and trust and ask for help. If you don’t have a professor you are comfortable with check to see if the school has a student advocate, or try calling the counseling center to see if they can point you in the right direction (and consider if their direct services might be useful as well). Those are serious situations, and you deserve more help and support than I can offer here.

Professors make mistakes.

Professors, being humans (this fact seems to surprise students when we see them in public), make mistakes (this fact surprises no one). When you think a professor made an actual mistake, don’t be a jerk about it.

During my first programming course in college I got a notice in my mailbox announcing that because I’d failed the first test I should drop the class and change majors. I was pissed. In no small part because I had been disappointed in the B- I’d earned on that test – not what I wanted but a long way from failing. I could have stormed over to the professor’s office and raised my voice to demand justice. Or I could have gone to the department chair and insisted he intervene on my behalf.

Instead, I waited until scheduled office hours, and went to talk calmly with the professor about the situation. I took the test with me – which had “B-” written in his (not my) handwriting on it. I started the conversation by pulling out the exam and handing it to him. He looked at it, cursed at himself, and noticed the off-by-one error in his grade book. He then apologized and offered me a job.

If I’d gone charging in I fully expect I would have gotten my grade fixed, but I would not have gotten a job (I also became his dog sitter so I got free food and laundry service every time he traveled for the next three years because I wasn’t a jerk). And he would have remembered me being a jerk about a simple mistake in every course I took from him until I graduated (there were several). Instead he liked having me in class, which is never a bad thing.

Your professors are nice people.

Seriously, when there aren’t world-wide pandemics they come to my house all the time. I meet them at parties and conferences. We have a good time together.

I’m not saying this just because I am married to one. Sure sometimes they get a little too into their subjects, or lack some social graces, and yes they hold power over your grades. But in the end they are just people like everyone else.

Take them seriously. Listen to what they have to say. Read their syllabi. But don’t lose track of the fact they are person, with a life off campus, a family, hobbies, struggles, and all the rest of it. Ask them open ended questions before or after class and you might learn a few extra things worth knowing.

Good luck and work hard!

There are plenty of other things you need to do to get through college, and lots of other advice to read. But hopefully these tips will help you along the way.  Good luck this year. Hopefully it goes at least a little better than last year did (for all of us).

SC DUG July 2021 – Queries on Queries

For the July 2021 SC DUG, I gave my new talk titled “Queries on Queries” which poses questions to ask yourself when migrating data between systems. Data migrations are often critical to project success, but all too often that are treating as a throw-away process. This talk is intentionally platform agnostic building from my experience with both Drupal and Salesforce.

If you would like to join us please check out our up coming events on MeetUp. You’ll find our meeting times and, once you RSVP, remote connection information.

We frequently use these presentations to practice new presentations, heavily revised versions, and test out new ideas with a friendly audience. So if some of the content of these videos seems a bit rough please understand that is some of the point. If you want to see a polished version checkout our group members’ talks at camps and cons.

If you are interested in giving a practice talk, leave me a comment here, contact me through Drupal.org, or find me on Drupal Slack. We’re excited to hear new voices and ideas. We want to support the community, and that means you.

SC DUG June 2021 – MQTT and Drupal

In June for the SC DUG meeting Will Jackson from Kanopi Studios gave a talk about using MQTT with Drupal to connect to local IoT devices. A fan of home automation, Will has created a Drupal 8/9 version of the MQTT module. He is hoping to encourage more people in the Drupal community to join the fun.

If you would like to join us please check out our up coming events on MeetUp. You’ll find our meeting times and, once you RSVP, remote connection information.

We frequently use these presentations to practice new presentations, heavily revised versions, and test out new ideas with a friendly audience. So if some of the content of these videos seems a bit rough please understand that is some of the point. If you want to see a polished version checkout our group members’ talks at camps and cons.

If you are interested in giving a practice talk, leave me a comment here, contact me through Drupal.org, or find me on Drupal Slack. We’re excited to hear new voices and ideas. We want to support the community, and that means you.

Snowfakery Custom Plugins Part 2

Last week I posted part 1 of this series on creating plugins for Snowfakery. This second installment covers creating custom Faker Providers to use as plugins for Snowfakery.

Why create a Faker Provider instead of Snowfakery Plugin

Python’s Faker library provides many useful things, but not everything you might want. Fortunately it is designed to be extended, and Snowfakery is designed to help make it possible for those extensions to be project specific when you need to (or when it makes it easier for your team to share recipes).

Sometimes you need to have all the features of a Snowfakery plugin, like maintaining state. But often you just need to kill a gap left by the Faker community offerings. In that case, a Faker Provider may offer longer term benefits like re-usability in other projects, and useful helper functions, that give it the strong choice.

Things you’ll need

You do not need to have reviewed everything in Part 1, but you will want to look at the project structure section and to make sure you have the following tools:

  1. Python 3 (any recent-ish version should be fine), and the experience to read and write simple Python scripts (they aren’t any harder to follow than YML just different).
  2. Snowfakery 1.12 or later. Note: if you have CCI installed it contains Snowfakery but on Windows you may need extra setup.
  3. The Faker module for Python (probably via: pip3 install Faker). Not strictly required but can help you with testing.
  4. A code editor you like that supports both Python and YML (which is pretty much anything good).
  5. You probably want experience working on at least one or two Snowfakery recipes.

Reminder about project structures

In part 1 we build a project around the built-in search patterns of Snowfakery.

Snowfakery looks for plugins in a select number of places:

  • The Python path.
  • In a plugins directory in the same directory as the recipe.
  • In a plugins directory below the current working directory.
  • A sub-directory of the user’s home directory called .snowfakery/plugins.
Recipe directory layout

For this example the plugins directory will live within the folder with our recipe (called recipes), but you can move the plugins directory up a level and it will work just as well.

To get started, in your project create a recipes directory. Next create a plugins directory within recipes. Then create faker_nonprofit directory in plugins. You can ignore the snowHelper and snow_product.yml examples from part 1 in the screenshot on the right, but that’s generally what we’re doing.

Create your first Snowfakery Faker Provider

Faker itself does not have a community supported generator of Nonprofit organization names, and when you work with nonprofits a lot sometimes you need those. That is exactly what we’ll create in just a minute.

In the plugins directory you created in part 1, create a directory for your new provider with the pattern faker_[my_service_name], in my case faker_nonprofit. Then in that new directory create a file named __init__.py this defines the Faker provider Python module.

You can see the full working example here but I’ll walk through the outline.

The file opens by providing a doc string and importing the faker.providers module.

"""Provider for Faker which adds fake nonprofit names, and program names."""
import faker.providers

The next several lines of the faker_nonprofit module provide arrays of words to use in the generation of fake names. Depending on what your provider does this may or may not be useful to you (but it’s very common).

Then we define the Provider itself as a class that extends the BaseProviders class, with whatever methods you want to call:

class Provider(faker.providers.BaseProvider):
   """Provider for Faker which adds fake nonprofit information."""
 
   def nonprofit_name(self):
       """Fake nonprofit names."""
       prefix = self.random_element(PREFIXES)
       suffix = self.random_element(SUFFIXES)
       topic = self.random_element(TOPICS)
       return " ".join([prefix, topic, suffix]).strip()

Those random element selections are from the arrays of words I mentioned a minute ago. Basically we’re just building a name from some selected words.

A note about tests

Unlike Snowfakery plugins, Faker projects have existing patterns for how to setup tests (they are not totally consistent but there are patterns out there). So the complete code on Github includes a testing module as well, and you should consider something similar for yours (particularly if you are considering making it available to the wider community). It’ll allow you to test your provider generically not just when it’s running through Snowfakery.

Linking your Provider to a Snowfakery Recipe

Now that we have a local Faker Provider, we need to connect it to our recipe.

A very simple recipe (in our recipe directory) should demonstrate the output quite nicely:

- plugin: faker_nonprofit.Provider

- object: Account
  fields:
    Name:
      fake: nonprofit_name

Run the recipe through Snowfakery and you can see the name appears believable:

$ snowfakery recipes/sample_recipe.yml
Account(id=1, Name=Southern Unity Community)

The project uses a slightly larger recipe here that uses both plugins and generates more than one object. The full recipe in the repo will create one Salesforce Account object that has a Fake Nonprofit name, a custom field for a Main Service that will be one of our snowy puns, an address, and two related contacts with all their basic information included.

$ snowfakery recipes/sample_recipe.yml 
Account(id=1, Name=Upper Friends Committee, Main_Service__c=Snowmanage, BillingStreet=4773 Giles Plains Suite 878, BillingCity=South Daniel, BillingState=Virginia, BillingPostalCode=34516, BillingCountry=United States, ShippingStreet=7844 Hester Shore Apt. 299, ShippingCity=Maynardview, ShippingState=Indiana, ShippingPostalCode=86323, ShippingCountry=United States, Phone=956.673.3002x471, Fax=+1-786-744-2112x36239, RecordType=Organization)
Contact(id=1, AccountId=Account(1), Salutation=Misc., FirstName=Isaac, LastName=Barr, Email=joypeters@example.com, Phone=+1-808-508-0989x418, MobilePhone=(987)475-7200x8072, Title=Tour manager, Birthdate=1982-03-27)
Contact(id=2, AccountId=Account(1), Salutation=Mx., FirstName=Erica, LastName=Lopez, Email=angel01@example.org, Phone=(011)243-1677x868, MobilePhone=(079)466-5474x52399, Title=Research officer, political party, Birthdate=2000-07-07)

If you have interest in seeing the Nonprofit Provider made into a more complete tool and released as its own project please let me know.

Snowfakery Custom Plugins Part 1

Last November I wrote a bit about creating Salesforce data with Snowfakery.  I’ve continued to use the tool for work, provide feedback to the project maintainer, and help the Salesforce Open Source Commons Data Generation Toolkit Project as we started to build a library of sample recipes. Hopefully I will have more to say on that after the next Community Sprint.

Snowfakery not only gives you a way create carefully shaped relational data sets of nearly any size, it also allows you to create plugins to extend its abilities. those plugins come in two flavors: Snowfakery Plugins, and Faker Providers.

For more technical details you may want to read the project has documentation. My intention here is to provide an end-to-end example of how to make them work.

This article started out as one long piece but to keep it focused I’ve decided to break it into two parts:

  • Part 1 covers Snowfakery Plugins.
  • Part 2 covers creating custom Faker Providers for Snowfakery projects.

The code for both parts is on Github if you want to see the project as a whole.  

Things you’ll need

  1. Python 3 (any recent-ish version should be fine), and the experience to read and write simple Python scripts (they aren’t any harder to follow than YML just different).
  2. Snowfakery 1.12 or later. Note: if you have CCI installed it contains Snowfakery but on Windows you may need extra setup.
  3. A code editor you like that supports both Python and YML (which is pretty much anything good).
  4. You probably want experience working on at least one or two Snowfakery recipes.

Snowfakery Plugin Project Structure

This setup just talks about Snowfakery recipes on their own, not within a larger project, but the concepts are the same even if the details are different.

Snowfakery looks for plugins in a select number of places:

  • The Python path.
  • In a plugins directory in the same directory as the recipe.
  • In a plugins directory below the current working directory.
  • A sub-directory of the user’s home directory called .snowfakery/plugins.
Recipe directory layout

For this example the plugins directory will live within the folder with our recipe (called recipes), but you can move the plugins directory up a level and it will work just as well.

To get started, in your project create a recipes directory. Next create a plugins directory within recipes. Until part 2 you can ignore the faker_nonprofit directory in the screenshot on the right, but that’s generally what we’re doing.

Create Your First Snowfakery Plugin

In the Snowfakery community we use a lot of snow-based puns to name things. So to help create fake sounding products for our projects we might need a simple plugin to generate us new words that match our general naming convention.

In the plugins directory create a new file called snowHelper.py. And copy the follow Python code into your editor:

from snowfakery import SnowfakeryPlugin
 
class SnowPunnary(SnowfakeryPlugin):
   class Functions:
       def snowpunner(self, word):
           return 'Snow' + word

The code here is pretty straight forward if a little nested. We are loading the SnowfakeryPlugin class from Snowfakery itself, and then extending that class to create our plugin. Snowfakery assumes that the plugin has a subclass to hold your plugin functions (called Functions) and that you add your functions to that subclass. Your functions can have a parameter (here the word being punned on) to accept inputs from other parts of the recipe.

Our SnowProduct Recipe

Now we need a recipe that will actually use our plugin. In the recipes directory create a file called snow_product.yml and copy in the following YAML code:

- plugin: snowHelper.SnowPunnary
 
- object: Product
  count: 10
  fields:
    Name:
      SnowPunnary.snowpunner: ${{fake.word}}

In the first line we load our plugin using Python’s module naming convention – because it is getting loaded as a Python module. The pattern here is the file name (without file extension) then the class name. In the last line we then call the plugin’s function by referencing the class name and the function name.

You can run the file directly in Snowfakery and see the outputs:

$ snowfakery snow_product.yml
Product(id=1, Name=Snowplantary)
Product(id=2, Name=Snowbadary)
Product(id=3, Name=Snowmillionary)
Product(id=4, Name=Snoweffortary)
Product(id=5, Name=Snowgreenary)
Product(id=6, Name=Snowbehaviorary)
Product(id=7, Name=Snowcouldary)
Product(id=8, Name=Snowforceary)
Product(id=9, Name=Snowyesary)
Product(id=10, Name=Snowcompanyary)

I’m going to show, but not go into great depth on, one more detail: plugins can save state. Snowfakery provides a mechanism for tracking context variables between calls that allow you to track current state. So we can have ours count the number of times the snowpunner function has been called and return that count in another function:

from snowfakery import SnowfakeryPlugin
 
class SnowPunnary(SnowfakeryPlugin):
   class Functions:
       def snowpunner(self, word):
           context_vars = self.context.context_vars()
           context_vars.setdefault("count", 0)
           context_vars["count"] += 1
           return 'Snow' + word
 
       def currentCounter(self):
           context_vars = self.context.context_vars()
           return context_vars["count"]

Then update the recipe like this:

- object: Product
  count: 10
  fields:
    Name:
      SnowPunnary.snowpunner: ${{fake.word}}
    Index:
      ${{SnowPunnary.currentCounter()}}

Notice that to call the function without a parameter we use the formula syntax.  Run it again and we see the new index that shows the count:

$ snowfakery snow_product.yml
Product(id=1, Name=1: Snowimpact, Index=1)
Product(id=2, Name=2: Snowfund, Index=2)
Product(id=3, Name=3: Snowdark, Index=3)
Product(id=4, Name=4: Snowteach, Index=4)
Product(id=5, Name=5: Snowteam, Index=5)
Product(id=6, Name=6: Snowsummer, Index=6)
Product(id=7, Name=7: Snownew, Index=7)
Product(id=8, Name=8: Snowperform, Index=8)
Product(id=9, Name=9: Snowonto, Index=9)
Product(id=10, Name=10: Snowmodel, Index=10)

It is important to remember that while it would be possible to add a value to the context variable on each iteration, that would cause Snowfakery to consume more memory on each iteration. Snowfakery is designed to generate records by the hundreds of millions if asked, and does so while consuming very little extra memory – you can do things in context variables that would break down on larger runs.

In part 2, I talk about creating a custom Faker Provider and loading it into a Snowfakery recipe.

SC DUG April 2021 – Getting Started with Electron

This month I gave a talk at South Carolina Drupal User Group on Getting Started with Electron. Electron allows you to use your web developer skills to create desktop applications. I based this talk on some of my recent side projects and the Electron Project Starter I posted the end of last year.

If you would like to join us please check out our up coming events on MeetUp for meeting times, locations, and remote connection information.

We frequently use these presentations to practice new presentations, try out heavily revised versions, and test out new ideas with a friendly audience. So if some of the content of these videos seems a bit rough please understand we are all learning all the time and we are open to constructive feedback. If you want to see a polished version checkout our group members’ talks at camps and cons.

If you are interested in giving a practice talk, leave me a comment here, contact me through Drupal.org, or find me on Drupal Slack. We’re excited to hear new voices and ideas. We want to support the community, and that means you.