Add documentation on how to add new drawing library support

2 files changed, 214 insertions, 0 deletions

diff --git a/DESIGN b/DESIGN
new file mode 100644
index 0000000..4fa14f2
--- /dev/null
+++ b/DESIGN
@@ -0,0 +1,104 @@
+Caskbench's Design
+==================
+
+Caskbench strives to be flexible, with abstracted core routines and lots
+of command line options to customize behavior.  Tests are very simple to
+add.  This abstraction also makes it feasible to extend caskbench with
+more rendering backends or even add new drawing libraries to test.
+
+
+== The Main Loop ==
+
+The main() routine in caskbench.cpp does all the setup, dispatching of
+tests, and results creation.  It runs through all the selected tests (or
+all tests) and alternatively runs the cairo or skia version of that
+test.  For each test, one rendering context is created for the given
+drawing library, the test's setup routine is called, and the test is
+given a free practice run (to warm caches or whatever).  Then the clock
+is started and the test is run for the given number of iterations.  In
+each iteration the screen is cleared, the test run, and the context
+updated.  After the given test has finished all its iterations we do
+some bookkeeping and cleanup everything, then move on to the next test.
+
+Essentially, this is the core logic:
+
+  foreach test c:
+    perf_tests[c].context_setup(&context, config);  // calls your setup routine
+    ...
+    foreach iteration i {
+      perf_tests[c].context_clear(&context);
+      start timer
+      perf_tests[c].test_case(&context);  // run the test
+      perf_tests[c].context_update(&context);  // calls your update routine
+      stop timer
+    }
+    ...
+    perf_tests[c].context_destroy(&context); // calls your destroy routine
+
+The list of available tests, perf_tests[], is generated by CMake when it
+runs the `generate_tests` script to create the tests.cpp file.  Each
+member of the array is a table of function pointers to do the various
+renderer-backend-specific and drawing-library-specific
+creation/destruction/update operations.
+
+
+== Test Design Theory ==
+
+The central purpose of caskbench is to help improve Skia performance on
+the Tizen platform (phones in particular).
+
+As a tester, you generally want to avoid having to hack and recompile
+code. Indeed, there may not even be a compiler on the device itself, and
+having to build new packages for a minor tweak can add a lot of hassle.
+
+So, a design objective for caskbench is to build everything together,
+and provide a rich array of command line options that enable the analyst
+to undertake widely differing testing plans, without needing to
+recompile.
+
+
+I envision at least four distinct use cases for caskbench, that I think
+we should try to support:
+
+A. Quick check. All test cases are run with just randomized
+   settings. Objective is full functional coverage in short period of
+   time to identify bugs in the code. We use pre-defined seed value(s)
+   in order to ensure reproducibility. This is mainly used during
+   development or porting, or for a quick tire kicking.
+
+B. Detailed survey. Several or all test cases are run at a series of
+   different setting values, and multiple iterations, using a specified
+   seed. These are intended to generate graphable data for analysis, and
+   to cover corner-cases that may not have an immediate relation to
+   real-world cases, but would help the analyst gain insight into
+   performance limits.
+
+C. Individual study. One test is run with specific settings and a
+   pre-defined seed to study a specific performance issue. This might be
+   used by a developer while optimizing the code.
+
+D. Benchmark. Test cases and settings are selected that more closely
+   resemble real-world use cases. Corner-cases are avoided. The tests
+   are iterated many times to get good performance averages, and the
+   seed is randomized. Objective is for comparing different rendering
+   technologies, backends, or algorithms for their impact on end user
+   experience, or for testing performance of different types of hardware
+   or drivers.
+
+What all this implies for individual test cases is:
+
+    Each test should have a defined core "mission". When run with all
+    parameters set to specific values, it should effectively let us do
+    stress testing of that core function. For instance, drawing a red
+    circle at a specific location repeatedly as fast as possible.
+    Each test should allow specifying randomization. E.g. random fill
+    colors, random font selections, or random shape dimensions.
+    Each test should allow increasing or narrowing the scope of
+    particular drawing parameters. E.g., increasing or decreasing the
+    variety of shapes to draw, or the number of gradient stops in a
+    radial gradient, or the min and max widths for line stroking.
+    Given a specific seed to the RNG at the start of a test, both the
+    Skia and Cairo implementations of the given test should produce
+    visually identical output (they needn't be pixel-perfect, but should
+    be darn close). 
+
diff --git a/HACKING b/HACKING
new file mode 100644
index 0000000..ac86914
--- /dev/null
+++ b/HACKING
@@ -0,0 +1,110 @@
+Adding New Tests
+================
+Tests are easy to add.  Decide a name for the test, then implement these
+three routines in cairo-tests/${name}.cpp:
+
+ * ca_setup_${name}
+ * ca_teardown_${name}
+ * ca_test_${name}
+
+The setup routine is executed just once, before the main iterations
+begin.  For tests that are directly measuring low level drawing
+operations, you may not have much - if any - setup.  But you could use
+this for one-time setup or precalculations that would be expensive to do
+in the test proper and that shouldn't be charged against performance.
+
+Now reimplement the same test for skia (and any other drawing
+libraries), with the same three routines but prefixed with sk_*.
+
+
+Adding New Rendering Backends
+=============================
+Tasks for adding another backend:
+
+ * cairo-[backend].cpp - surface creation/update/destruction
+ * skia-[backend].cpp - surface creation/update/destruction
+ * [backend].cpp - shared code for both of the above
+ * caskbench-context.cpp - register your backend
+ * caskbench.cpp - add command line options you need
+
+
+Adding New Drawing Library Backends
+===================================
+Adding a new drawing library is a bit more involved:
+
+ * config.h.cmake
+ * CMakeScripts/Modules/FindXxx.cmake
+ * CMakeLists.txt - add library to build system
+
+ * caskbench.cpp - add command line options, etc.
+ * caskbench_context.* - register your drawing library
+ * [drawlib]-[backend].cpp - surface creation/update/deletion
+ * [drawlib]-shapes.* - implement shape drawing
+ * [drawlib]-utils.cpp - misc. routines
+ * [drawlib]-tests/*.cpp - reimplement all tests
+
+Start by adding your library to the build system via the cmake files.
+The FindXxx.cmake module searches through lists of paths where your
+drawing library might be, and figures out the XXX_LIBRARIES path and
+XXX_INCLUDE_DIR.  It should set XXX_FOUND to "YES" if the library was
+detected.  Use the cairo and skia modules as examples. Now add a USE_XXX
+for your library to config.h.cmake, and then register your library in
+CMakeLists.txt - you can basically just copy the logic for cairo.  Run
+cmake and verify things get detected correctly.
+
+With the actual coding, the easiest place to start is probably the
+command line arguments in caskbench.cpp.  Add your library to
+print_drawing_libs_available() and update process_drawing_libs() to
+check for your drawing library's name.  You should now be able to invoke
+caskbench with your drawing library specified, and it'll bail out when
+it can't find the matching tests.
+
+We can go ahead and stub in all the tests at this point.  Create a
+directory src/xxx-tests/ and add the following code to line.cpp:
+
+  #include <config.h>
+  int  xx_setup_line() { return 1; }
+  void xx_teardown_line() { }
+  int  xx_test_line { return 1; }
+
+Copy this file to all the other tests, substituting the test name for
+'line' in the filename and the above three lines.
+
+Then in the generate_tests bash script, add code to write the
+declarations for these tests, and to set up the perf_tests data
+structure.  To test this part is working, run:
+
+  $ src/generate_tests src/ /dev/stdout
+
+Now go back to caskbench.cpp and extend populate_user_tests() with your
+drawing library.
+
+
+Next, enable your new drawing library in caskbench_context.cpp by adding
+definitions for these routines:
+
+  context_setup_xxx()
+  context_destroy_xxx()
+  context_update_xxx()
+
+These three routines are called indirectly from caskbench.cpp's main
+routine; it will probably help to review the DESIGN document to learn
+how all these bits interoperate.  As you implement these three routines,
+you'll add your own xxx-egl.cpp and xxx-image.cpp with your
+backend-specific code.  Use cairo-egl.cpp and cairo-image.cpp as
+examples.
+
+Also add these two routines to xxx-util.cpp:
+[Maybe all the drawing lib context routines should be moved to -utils?]
+
+  context_clear_xxx()
+  write_image_file_xxx()
+
+
+Now you're ready to create a shapes library, xxx-shapes.cpp.  This
+should be a pretty paint-by-numbers reimplementing of the cairo or
+skia shape library.
+
+Finally, go through and fill in all the tests.  This should also be
+pretty straightforward.
+