Tester Prototype User Manual

Contents:

Foreword

This is a basic manual for the REBOL case test tool called Tester. The purpose of this tool is to create test cases for your program code which can be run as many times as you want to let you very quickly find bugs in the code, you may not otherwise have found. Tester can be compared to the Flight Data Recorder or black box on airplanes, which record everything. The data recorded by them are studied after a plane crashes to understand why it crashed.

Depressing as it may sound, this analogy fits well with Tester, to help you understand why your REBOL scripts and functions don't work correctly.

For me, this has sped up development of certain libraries by a factor of 5-10, moving weeks of development and testing into a couple of days.

There are a few things to understand about Tester before we move on:

• Tester is not a performance measuring tool.
• Tester does not record other than what you ask it to record.
• Tester doesn't provide access to interactive debugging. Once the test is started, it will continue at full speed until no more tests are available in the test list.
• Tester is not a traditional step-by-step debugger, so if your code consists of multi-layered loops that are run millions of times, Tester may not be suited to debug that.
• Tester is a concept prototype at this point with many things that might seem stupid or backwards, as it was originally designed as a tool for me alone.
• Tester is really not very complex. It could have been written by anybody with a fair bit of REBOL knowledge.

Installing Tester

Tester consists of the tester.r script, toolbar.r and list-view.r, always in their latest version from:

http://www.hmkdesign.dk/rebol/

The simplest way is to run it directly from the site:

http://www.hmkdesign.dk/rebol/tester/tester.r

This makes it possible to create an icon for it in the Rebol/View desktop and run it from there, which I think also is the easiest way to use Tester.

Tester downloads some toolbar images, list-view.r and toolbar.r and then should start up. If a window like this comes up:

Then you can begin!

Tester creates a configuration file for each project in the same directory as it is run from. These files are called tester_27Aug200625644200. Tester automatically saves to these file very frequently, as there is no Save buttons in Tester.

Be aware also that these configuration files can quickly grow in size as your projects grow, because all test results are stored in them.

A Tour

Tester consists of the main window and a few sub-windows that you can open at any time and keep open if you desire.

The parts of the main window are:

	Sets list	Sets let you store multiple tests for the same project, so you can freely browse old test results. They are stored with a time stamp. You can decide whether or not to make a set every time you run the test. It?s also possible to delete a set at a time using the Delete button. Sets lists do NOT store the the test cases themselves, only the results of the test cases.
	Result list	Test results are stored here, shown as they are created in the Tests window. When clicking a test, more indepth results will be displayed in the boxes and lists to the right.
	Testcode text box	This displays the test source code for the current result row.
	Result text box	This displays the returned value of the test.
	Status text field	This returns the status of the result, whether it was OK, or if it generated an error. This is useful in cases where you want a specific result, so Tester can compare this specific result to what actually happened.
	Watched Variables	This list contains a list of variables that are watched over the course of the test run. Each entry in the result list, contain the variable contents as it was recorded at that specific time.
	Function Marks	This contains a list of words that you can watch, for example, to study program flow.

Sub Windows

Projects

This is where you make projects and handle include files for each project.

Test List

This is where you enter the tests you need to perform. It's opened by pressing the Tests button in the main window toolbar, or by double clicking an entry in the result list.

Watched Variables

This is where you enter REBOL variables that you need to watch during the test. It's opened by pressing the Watch... button in the main window right above the list of watched variables.

Functions

This is a list of system/words and acts as a code viewer. It has no practical purpose other than let you view the code of the functions. I have a habit of forgetting the arguments to various functions, so I tend to use this as a function reference.

In the future this window might grow into an actual code editor.

Variable Viewer

This window simply lets you view the contents of a watched variable, so you can study larger objects in a comfortably big window. It can be brought up by double clicking a variable in the watched variables list.

Entry Result Viewer

This window lets you browse the full contents of one of the text areas in the right side of the main window. From within the window, you can navigate back and forth in the results. It can be activated by pressing the Full button above the text area you wish to view.

The Next and Previous buttons are for browsing through results, so you can observe the changes to this specific variable over time.

Basic Workflow

1. First add your script if you have one to test
2. Add/modify the tests that need to be performed
3. Add/modify the variables you need to watch
4. Add/modify the function marks you need
5. Run the test
6. Study the output
7. Repeat the first step

Simple Tests

To test something, you can either add some existing program code and add tests to that or just create stand alone tests. You can write code and test it directly in Tester.

To create tests, you must first create a project. This is done by opening the project window and clicking Add Project.

Give the project a name and ignore the settings in the right side of the window for now. Close the window and notice the Active Project area in the toolbar contains the name of the currently active project.

Now you can create tests.

The Test List, which resides in a separate window must be opened. This is done by pressing the Tests button in the toolbar.

In this window, you'll have a list view and a code window.

To add a simple test, do the following:

1. Press Add Line
2. Type in your test code. Let's type in: var: 3
3. Tab out of the code window. This stores your test.

Now we have one single test. We can now go back to the main window and press Run Test to the upper left in the toolbar.

You should notice the result list now sporting a single entry with your code and the status value should be OK. All that happened was that the code you entered, was run and the result was that it didn't produce an error. This was stored in the result list.

Let's add some more tests:

1. Go back to the Test List window
2. Press Add Line
3. Enter something different, such as: var: 5
4. Then tab out again. Now press Copy to duplicate the entry. Go into the code window and change the code to: var: "Hello World!"

After storing the result, go back to the main window and press Run Test again. Now there should be 3 entries in the result list all with the status OK.

Watching Variables

When running these three tests, all that happens is equivalent to saying this in your console:

>> var: 3
== 3
>> var: 5
== 5
>> var: "Hello World!"
== "Hello World!"

But in Tester, when the test run is over, you can go back and track what value var had at a particular point even when var has been overwritten many times.

To monitor var, you need to enter the word in the Watch List:

1. Press Watch... and the Watch Variables window should open.
2. Press Add to add an entry for editing.
3. Type in var and press Enter.
4. Go back to the main window and press Run Test.

The variable should now appear in the Watched Variables list in the right side of the main window when you click an entry in the result list. Click another entry and the contents of the list changes to fit whatever you set var to at that particular entry in the test.

The value will of course be visible at all entries and show the value that was last assigned to it.

You can't watch variables local to a specific context at this time. Only global variables can be watched.

Testing Existing Program Code

To test existing program code, for example, your own library of functions, you must load that code before the test is run. This is done by going to the Project window, and add include files in the right side of the window for the current project.

When the files are specified, those files are loaded in the order specified in the file list, every time you press Run Test. The code is loaded into the same process as Tester (not launched!) and then run. It's very simple and has limitations, but for simple testing of functions, this works quite well.

After the code is loaded and run, the tests in the test list are performed.

Function Marks

Function marks is simply putting in the function called MARK inside your own program code, similar to when using PROBE to read out variables from the code. The variable is passed through the function in the same manner as PROBE, so it can be dropped in anywhere PROBE can.

Instead of delivering the result on the console, MARK puts the result in the Function Marks list in the lower right corner of the main window.

Studying Program Flow

By letting MARK label parts of the code, you can study the program flow of your code for a single test.

The basic syntax is:

mark <label>

The label can be any identifier, a word, a string, a number, anything that is practical to read in the function marks list.

This means when you create code like this:

either 12:00 > now/time [
  "Good Morning"
][
  "Good Afternoon"
]

You can simply add your marks to see where the program flows:

either 12:00 > now/time [
  mark 1
  "Good Morning"
][
  mark 2
  "Good Afternoon"
]

Since we are not probing anything directly, the Probed column will only display a "-".

You can add marks both to your tests and inside the loaded program code you are testing. Just remember that the program code will not work outside of Tester, while those marks are present.

Studying Output

By using the /prb refinement, you can probe return variables to study temporary results from anywhere in your test code. This basically works like PROBE, except, you still have the label to use.

The syntax then becomes:

mark/prb <label> <value>

This helps you to tell what the time actually is during the test, to see if you got the strings right:

mark/prb 'output either 12:00 > mark/prb 'time now/time [
  "Good Morning"
][
  "Good Afternoon"
]

Notice that I used a word here for a more meaningful label.

By comparing the result of the mark with the returned value, you can easily see if you got the strings correctly.

Intended Results

Tester has the ability to compare an intended result with a returned value from a test. This makes it possible to easily spot if a function that returned an error, was intentional. In the Test List window, below the code text area, you can enter the intended result information now, but no action will be taken by it.

This will be better documented in later versions.

Stages

Stages is basically your test divided in stages. This makes it easy to separate initialization from actual test code. It's purely a visual tool and does not affect the performance or behaviour of your code.

The functionality is limited currently to coloring each row of your test in a specific color, which you can select from the Tests window.

Comments

Like Stages, you can also divide your tests visually with a blue line in the result list. In code, this is read as a comment line:

; ---------- My Comment

To make comment lines like this in an easy way, click the Add Comment button in the Test List window.

Generating Source

All tests made in Tester can be exported as a single source file. When this happens, all marks are removed so you can run the code directly, if you want.

To generate the source, go to the Test List window and click Generate Source.

Known Limitations

• Tester exists under its own context called CTX-TESTER, so you normally can't mess it up. Of course it's possible to do, if you access CTX-TESTER from a test and perform changes to it.
• Tester runs all tests under the same REBOL runtime environment as itself, which means that memory stats and such, may not be accurately measured. Tester affects those results.
• If you create a test with an infinite loop, there's nothing Tester can do about that. You'll have to force quit Tester.
• Tester can't be used to test GUI based applications.

Known Bugs

There are many problems in the program that I'm already aware of, so I'll list the ones I remember off-hand here:

• When creating a new project and closing the project window, the program will crash due to a bug in the REBOL event system. Just start the program again and you'll be able to continue.
• Windows are not resizable and the display requires a minimum 1024x768 resolution for usage.
• Text areas are hardcoded to use the Windows XP Terminal 8x8 font. On OSX it uses a Monaco 8x8 font. This is nice and small for me, but it may not be suitable for you.
• Some list views might be a tad small.
• The UI is not particularly logical, someplaces even backwards. I adopted the "Add then Edit" paradigm, instead of "Enter and then Add" in most places.
• Using marks inside of tests will sometimes corrupt the Probed column Function Marks list view.
• Next Error and Previous Error in the main window don't do anything yet.
• Full and Copy buttons for Watched Variables and Function marks don't do anything yet.
• Series lengths in the watched variables list are not printed.
• Some text areas don't yet have a scroller.

What's Next

• A complete rewrite to bring it up to alpha quality with a better UI, resizable windows and better logic.
• Allow tests to be made on Rebol/Core only. Importing tests.
• Find set variables and functions so they can be unset after a test run.
• Possibility to run code in a separate process for greater control and less run time environment contamination.