Apologies

Before we get started, an apology, there is no code in this post. Not a single line. Partly because there is too much to show, but also because it's tightly coupled to the Change Management Software, and the coding standards of our team. That being said, hopefully it will still provide some value for the reader to see the journey, the issues, and how they've been tackled.

Automating Tests

Some time ago, I decided to have a go at writing unit tests for RPGLE code. Fortunately someone had already done most of the hard work by writing the excellent RPGUNIT framework, which I downloaded, and immediately started playing with.

With a few tests under my belt, I found that I wanted the tests to run automatically (I know, I'm kind of lazy like that). We use Rocket Aldon Lifecycle Manager to manage our changes, and it's fully extensible with exit programs, so I set about adding the following functionality:

After a *SRVPGM or *PGM compiles successfully, any tests are automatically run.
Missing unit tests are added after a *SRVPGM compiles. e.g. When new procedure postToBlog() is added to S_BLOG, then test_postToBlog() is automatically added to T_BLOG.
Objects cannot be promoted to QA if they have tests which have run and failed.

I also configured a daily job to run all available tests on our QA machine and report any errors.

The key to automatically running tests, was a simple naming convention. All our *SRVPGM objects were already prefixed with S_, or C_ (e.g. S_STRING, C_OBJECT), so I named the related test objects using T_ as an alternate prefix. e.g. T_STRING had all the tests for C_STRING. For *PGM objects, I just prefixed the *PGM name with T_ e.g. PS123 is tested by T_PS123.

Test Data

The next problem to tackle was test data. Lots of languages use mocking to remove the need for a data access, but that isn't quite as easy with RPG, plus I still wanted to test file I/O without trashing the data in our development/QA database. With a bit more thought, I came up with the following solution:

As part of every test, a unique test library is created, and it is set as one of the two available product libraries in the library list of the job running the test. This means that the tests can still leverage a standard database in your library list, but you have ability to add copies of files in your test library.
In the setup() procedure, any required files are added to the test library, by copying objects from the jobs *LIBL (Including automatically adding any dependant logicals/indexes, and triggers etc).
Running command STRDBG UPDPROD(*NO) at the start of the test ensures that the tests fail if any code tries to open a file for update that is outside the test library.

Getting There, But Slowly

Tests were now running over the test library, but re-creating the library and a large number of files every time the test ran just took too long, so it was time for some more improvements:

All files in the test library are journalled.
Every time the test suite runs, it saves a copy of the code that sets it up the test library.
If the setup code hasn't changed since the previous run, then it just rolls back the journaled changes since the last run, which is must faster than clearing and rebuilding the library.
I created a command DSPTSTSQL to quickly and easily generate the RPG code I would need to insert data into a file based on the records in an existing database file.

Enter TDD

It was at this point that I started trying TDD. I liked the concept, and could definitely see the benefits, but again, I found that it just took too long. TDD requires frequent changes to both the code being tested, and the tests themselves, but unlike other languages where compilations and tests run on the client machine, RPG has to be compiled on the server, and this could take between 20-30 seconds, so I implemented the following changes to shorten the coding feedback loop:

Created command BUILDMOD to compile a *MODULE, it's parent *SRVPGM/*PGM and then automatically run any unit tests. This had a lot less overhead than the Rocket ACMS compilation process.
Ran BUILDMOD automatically whenever a source member was saved, making the process as efficient as possible.
Created a "Console" web page, which automatically refreshed and showed the result of compilations and test execution.

The difference this made was quite profound - the time between writing code, and getting feedback on whether it had worked was reduced from over 30 seconds to 5-10, which might not seem like much of an improvement, but when you're making 10 changes an hour, that adds up to 3 minutes (5%) saved an hour, and that's before you consider the additional time that was wasted by checking email/surfing the web whilst waiting for a compile to complete.

You can see the process in action here:

What Have I Learned?

Lots, and there is still more to be done, but here's a few tips:

Avoid having too many tests in one *SRVPGM. If the tests take more than 30 seconds to run, then they are too long. One possible solution I'm considering is *MODULE level tests rather than *SRVPGM level tests. e.g. *SRVPGM C_STRING exports code from *MODULE C_STRING1 and C_STRING2. I'm proposing using T_STRING1 to test procedures exported by C_STRING1 and T_STRING2 to test C_STRING2, rather than having T_STRING test all the code exported by C_STRING.
Have different tests procedures for different cases, rather than a single test procedure testing many cases - it makes resolving issues much easier. e.g. Use single assertions in test_addBlogPostEmpty(), test_addBlogPostBrief() and test_addBlogPostSuperLong() rather than 3 assertions in test_addBlogPost()
Always think about your test data, if there is a chance that it will change, then it's worth adding that data to your test library, rather than relying on the data in your standard database not changing.

In Conclusion

Some of the inspiration for this comes from playing around with other unit tests frameworks like JUnit in Eclipse, and marvelling at the near-instant feedback you get to changes made to your code. It's obviously going to be more challenging getting RPG unit tests to run that fast, but hopefully what you can see from this post that you can write automated unit tests in RPG, quickly, efficiently, and perhaps most importantly, have fun whilst you're doing it.