As you are about to see I have concerns about using AI in your education. That doesn’t mean I’m opposed to all things AI. I using it in my work when it’s useful, and in keeping with company policy. I use it on side projects even more. Generative AIs are down right useful – when used in the right context.

Using AI for College Assignments

AI’s were not created as a learning tool. It’s not that they can’t be useful in helping your learn. Tools are coming out that are hoping to use generative AI as study platforms. You can use AI to generate extra problem sets, flash cards, quizzes, and other tools for you to use as practice. But if AI is generating the answers for you, or writing your paper, than you are not doing the work. If you are not working, you are not learning.

Learning should be hard. Learning friction is good. If you are using AI to make learning easy, you’re doing it wrong.

If you want to learn don’t use AI. If you want to be employable, don’t use AI for your course work. If you don’t want to learn, please skip college: don’t waste your money or your professor’s time.

Learning Requires Practice

When I give talks conferences about communications skills, I say practice is critical for improvement. I also tell them never use AI when practicing.

The analogy I use is that learning is like working out at the gym:

If I want to get stronger, I can go to a gym and lift weights. If I go on a regular basis, and put in sufficient effort, I’ll get stronger. But if I decide that my goal is to get heavy things off the ground, I could use a forklift. That forklift will get much heavier stuff off the ground than I ever will. But I won’t get stronger.

AI is like a forklift: it’s powerful when used by a skillful operator, but it is not designed to make you smarter.

College assignments are a form of practice. They help you build new skills. Even if they feel useless, you need to do them yourself.

Their Work Isn’t That Good Anyway

On the whole AIs write bad college essays. I don’t care that their creators claim they do college level work – they don’t.

Some of you feel like AI’s produce good college essays. How sure are you that you know what good work looks like? Are you sure your professors agree?

You may also believe that no one notices when you hand in an AI-written or edited essay. Why do you assume because you don’t get in trouble no one sees what happened?

My wife generally reports that AI generated papers handed in by her students usually fail to meet the assignment in basic ways, and therefore frequently don’t get graded at all. Her colleagues tell me similar things. That generally means they get a zero.

As for if someone notices: just because you don’t get busted doesn’t mean they don’t notice. Right now, in far too many colleges, there is little incentive for professors to punish you. Punishing cheaters often requires doing hours of miserable unpaid work. A zero, or other low grades, requires minimal work.

I know professors who admit they are letting students get away with AI generated work. They assume the professional struggles that come later will be punishment enough. In short, they expect me to punish you by not hiring you or firing you for incompetence.

This is bad for all of us. I blame the administrators who worry more about your current satisfaction than your long-term success.

But My Professors Use It!

Students like to raise the argument that professors use AI, therefore students should be allowed to use AI. This is a hollow argument, but let’s spend some time with it anyway.

The core of this argument is that it’s hypocritical to suggest using AI is bad in one context and acceptable in another. But in life context matters. Why should they be held to the same standard as you are? Their job is to support and guide your learning. You goal should be to learn.

There are lots of important things students benefit from doing that professionals rarely do.

A good Computer Science program will put you through a course on algorithms. In that course you should learn about sorting algorithms and one of your assignments should be to code one. I took that course, wrote that code, and have never written another sort from scratch.

In fact most of the actual algorithms you learn in that course most professional developers will never write. There are excellent libraries already written that do these basic operations really well. That doesn’t mean writing sorting algorithms in college was a waste of my time – it was extremely useful! I learned a lot doing it.

Same with writing in English (or other human languages) – you need to be good at it. To get good you need to do it – a lot.

You need the practice. Your professors don’t.

Letting the AI do the Work Makes You Lazy

One of the reasons that your professors are right to prevent you from using AI, while potentially using it themselves, is that AI makes it easy to be lazy.

When I use an AI to write code I notice very quickly the temptation to be lazy about checking the output of the AI. I know I cannot trust the output, but it is hard to stay engaged in reviewing all the code an AI generates.

The only thing, really, that keeps me focused is that I am extremely price sensitive: I once spent 25¢ on a task with Cline (one of the better AI tools for developers). A whole QUARTER! Worse, the code looked good but wasn’t actually good.

You have to read, and understand, everything you have an AI generate. That can take nearly as much effort as creating it in first place. But because you didn’t create it, your brain isn’t engaged with the details. That means you have to work harder to do the review.

A Tiny Exception

There is a tiny exception to all this: Assignments designed to help you examine how an LLM works and what they do well.

These will be assignments where your professor tells you to use AI. That is the one and only time I think they can make sense in an academic setting – at least currently. One day that could change, but we are a long way from that.

I Need to Know How to Use AI for My Job

Yes. The future of work, heck the present of work, involves people using AI to make us more efficient. But college is not a job training program – it’s about learning how to learn. College should equip you with the skills to grow and adapt over time.

AIs are one type of tool you will need. There are lots of tools you will need to learn to use. To use any tool you also need to understand what good and bad results look like when you use it. That takes understanding of the fundamentals of what the tool is doing. College isn’t the only way to learn fundamental skills, it isn’t even the best way for everyone, but it’s a great way for a lot of people if you do the work.

Leverage the skills you learn while learning in college to teach yourself how to use AI tools.

Remember That Integrity Matters

In the working world good managers don’t have time to check every detail – they need to be able to trust you to do the right thing when no one is watching.

I need to be able to trust the members of my team. I need to know that they are following company policy without my reminding them. They need to be honest with me when there are challenges and they need support. When they make mistakes I need them to admit it so we can work together to rectify the situation.

If you cheat your way through college, why should I believe that you are going to have integrity in any of those conditions?

If I have determine that you used AI to short cut your college learning, why should I believe that you aren’t going to use AI when it’s a security or privacy risk? Why should I believe you can do the work yourself at all?

Please, take the time to do the hard work and show up knowing how to learn the tools that do not exist today.

To be clear I have been experimenting with Github Copilot and Salesforce’s Agentforce for Developers since they were released. Most of my earlier experiments were failures. The tools wrote trivial code terribly and simply couldn’t understand more complex requests. They made me slower, not faster, so I put them aside for a time. But they are evolving fast and were worth another look.

I learn by doing. So I set about creating a few tools that were on my mind to see if I could get good code out of the LLMs. Or at least if they could save me enough time to be worth the learning curve to get good at using them.

Project 1: Simple OpenWRT Status Call Out

I recently updated my home wifi router to run OpenWRT. When I bought the router I selected one compatible with that OS. After realizing T-Link would never fix some security issues in the firmware I made the switch.

Our internet service from Breezeline has always been spotty; sometimes bad enough I need to relocate to get stable service. One of the things I want to know when I’m working elsewhere is: has my internet come back up yet?

I have a programmable router, I’m a programmer, therefore this is a problem I could solve for myself.

But I’ve never written code to work with OpenWRT. Worse, I generally work in high-storage environments. I mostly do side projects in Python or NodeJS, but my router doesn’t have the storage needed for those run-times. OpenWRT recommends C or Lua for scripting. I’ve done my time in C. Jumping through all the hoops required to make C do something this simple did not feel like fun. So Lua it is. Too bad I hadn’t written code in Lua before…enter Copilot.

Throughout this project I used the current defaults – which meant GPT-4o was the selected LLM.

Initial Code

I actually found Lua by asking Copilot. My first prompt was just:

Generate a program for openwrt that sends a post request to an API endpoint at spinngingcode.org

And it generated a reasonable looking Lua script. Since I don’t know Lua, or at that point why Copilot chose it, I dove into a little research into the language. After a few minutes I confirmed that, yes, this is the right use case for Lua. I also confirmed Copilot made reasonable choices in how the code was written.

I went on to ask Copilot to change call outs from curl to a library, add features to pull in the network status information from the router, and other adjustments. It consistently generated code that was close to correct. I did have to tell it to adjust details after validating them against my router’s actual setup and commands. It took some time but on the whole I was able to make steady progress. Eventually I had a good-enough solution.

Tests

Automated test generation has been a promise of these tools since they arrived. It always feels like something that’s easy to do: read code, write test. Which is, of course, the 100% wrong way to write tests. However, the point of the exercise was to see what I could get the tools to do, so I gave it a try.

It took several tries for it generate valid tests. The first time I asked, it wanted to know what framework to use (a fair request), but generated a useless hello_world test file. On subsequent tries it switched frameworks without asking, and took several tries before generating something that looked valid.

Too bad it wasn’t valid.

The test framework I picked, luatest, was a bad choice. But with little experience in the language I didn’t know that. When I finally had it refactor to busted, I got tests I could setup and run…and they all failed cause they were still all wrong.

I spent more time getting a valid test setup than primary code – eventually I gave up. It would have been faster to do the research and figure it out from scratch.

Code Review

All in all, Copilot gave me a reasonable piece of code for the primary request. Using Copilot was far faster than I could have done the research to write something equally good. It still needs a lot of supervision to get it right, but the refactoring requests seemed to go okay.

The code commentary Copilot provided in the response is detailed and accurate, where I checked it. It’s chatty so I didn’t check every piece of commentary just every line of code. The code itself came with passable comments about what each block and function does. And functional decomposition was vastly improved over previous experiments I’ve done with Copilot.

It had to make up a few details, like the actually API endpoint name (it picked https://spinningcode.org/api/endpoint – okay sure reasonably good fake answer that is clear but also obviously wrong), but those were good enough for the limited details I gave it.

Outside of the automated test disaster, the biggest issue was that it did nothing to encourage secure design. Copilot recommended no security for that endpoint, nor does the commentary flag that it skipped security suggestions.

Project 2: Simple PHP Status Server

Since I have PHP on my personal server, I decided that PHP was the right choice for the server side of the router project. Despite being a little rusty I am much more comfortable with PHP than Lua. I have standards and expectations about how to write and organize good PHP code.

The prompt I provided:

I have a Lau script that looks like this:

[a copy of the then current version of the Lua file]

Now I need a PHP application that will listen to those calls, record the data into a CSV file, and includes a page that displays the last 100 lines of the csv file.

It managed to generate valid PHP code that matched my request – sorta.

Code Review

The PHP it generated is not to my standards – not even close.

It lacked security checks, was poorly organized, and had borderline useless comments. If you look at the repo there are both a lot of generated refactors and then a lot of hand edits cause that’s web facing code, security matters, and I could fix it faster than debating with an AI.

I made some effort to get it to highlight the important information I wanted, but it struggled to regain the context from the OpenWRT requests that the Lua script was built around.

Unlike the Lua script, the PHP saved me little or no time. It was terrible in this context even with lots of attempts to refine the code. This piece of code I’ll probably maintain by hand.

Project 3: Data Migration Scripts for Salesforce

One of the hardest things to get people to fund sufficiently for a Salesforce project is the data migration. They are hard, boring (well not for everyone – I have undying respect for people who like these projects), and super time consuming. So we are always looking for ways to do them faster.

A few years ago I proposed a training exercise for data migrations that involved taking the data from my SC Salary Data repo and loading it into Salesforce. It’s a great training project because there is lots of data and it’s real-world messy. But it’s also simple: just two or three objects depending on the data model you choose. So I decided to test if I could use AI code to generate basic scripts to clean and load this data quickly.

For this project I made two changes to my approach. First, I switched the model to Claude 3.5. Second, I decided to treat the LLM like a junior developer and lead it through the project more intentionally. Part of this second change was driven by my initial experiences with the other projects. I assumed from the start that the code would be sub-standard, but that I could encourage Copilot to make changes to meet my expectations – just like I do when working with a human.

Code Review

This project was startling successful. The first stage was to have it create a script that prepped all the data into a simple SQLite3 database (it does some clean up, but not all that’s possible with this data). The second stage was to have it create a script to load the data. That second stage became it’s own repo because I wanted to include a Salesforce SFDX project to setup the objects and fields I wanted correctly.

I generated probably 90+% of the code in that repo with the LLM. The Python to load the data, most of the Salesforce Metadata to create objects and fields needed, even the read me file I generated in large part with the LLM.

Frankly this is a project I’ve seen developers struggle to get right, and with a little guidance the LLM was able to do it. It probably needed as much guidance as a human would have (a very junior human developer), but the iteration cycles were much faster. I am not convinced when all is said and done the project was faster – it took a few hours to complete each stage – than my doing it solo. However, it required far less mental engagement from me than if I had written the code unassisted.

This project went well enough I’m using it to help drive a conversation about how to change our approach to migrations in general. There are major differences between what I did and I do in a migration of a database with thousands of tables targeting hundreds of objects with millions of records. But I was impressed enough to want to move on to fill those next gaps.

Conclusions

LLM generated code is improving fast, but it is still generating low-quality code by default. Unlike experiments I ran a year ago, it now generates functional code that is equal to the code that a brand new developer could write.

Generated code is not to be trusted, particularly in production or public facing settings. If you don’t ask for security, you don’t get security. It organizes code as if it’s following train of thought, not in the order that well structured code should follow. And it does not write tests worth having without significant effort.

Good AI driven tools can make good developers faster. They are not good enough to replace a good developer – yet. Developers will need to learn to work these tools into the process, but we need to be aware of the gaps and how to close them.

I expect the quality also depends deeply on the quality of the code in public repos on the internet. I believe the terrible quality of the PHP I got vs the reasonably good Lua and Python is because there is lots of terrible PHP out there used to train these models. As long the LLMs write code that conforms to the average of the examples online communities will need to sweat the details of what is out there.