Regression testing protocol

Below we describe the regression testing procedure we use to verify XGAP importing, exporting and database backend capabilities. This procedure involves a series of automated imports and exports; the results of the exports should match the imports otherwise there is an error somewhere in the XGAP infrastructure. Users extending XGAP can use this method to verify proper functioning of their XGAP instance.

Requirements being tested

The testing protocol aims to test the following:

• A connection with MySQL can be established
• The referred database exists in MySQL
• The default user ('molgenis') exists and has been granted database rights
• The molgenis' generated SQL data model file is correct
• MySQL is set up correctly and is able to insert the datastructure
• Tar.gz extraction functions works correctly
• Generated CSV importer/exporter uses the correct input/output files
• Generated CSV importer/exporter can instantiate the database
• Generated CSV importer/exporter finds the corresponding datatypes and mappers
• Generated CSV importer/exporter performs import/export process correctly according to datatypes, mappers, contraints, table names, etc.
• All imported components are contained within an investigation and cannot conflict with another investigation
• All investigations are unique
• Custom XGAP CSV exporter is able to use queryrules to export one specific investigation to a unique location
• Custom XGAP CSV exporter procedure can successfully query the database with additional argument "all investigations"
• Directory/file compare tool can determine if directory content is equal and file content is equal
• Control data for directory/file compare tool yields correct results (two positive and two negative controls)

Procedure: full data roundtrip with multiple investigations

We assume that we have two data sets with each one investigation. In this example we use 'Bailey' and 'Lyons' from XGAP DataSets. The test protocol involves:

1) Start with an empty database by cleaning it
Passes only if: The connection with MySQL is established, database exists, user exists and has rights, generated SQL file is correct (syntax, constraints, char encoding etc), MySQL is set up correctly and is able to insert the datastructure. (thereby removing all existing XGAP content)

2) Extract investigation "Bailey" from an archive and import from extract directory A
Passes only if: Tar.gz extraction tool works correctly, CSV import procedure detects files and imports correctly according to contraints, table names, etc.

3) Extract investigation "Lyons" from an archive and import from extract directory B
Passes only if: Same as 2), plus there must be no 'id' like fields present that could conflict with "Bailey", all entries are independant of previous imports, investigation must be unique.

4) Perform an 'Export all' on the database to a single tar.gz archive
Passes only if: CSV export procedure can successfully query the database with argument 'all investigations, write out files in the right format, in the appropriate locations.

5) Empty the database
Passes only if: Same as 1).

6) Import the previously exported data from the tar.gz archive
Passes only if: Same as 3), but now the CSV import procedure has one dataset containing two investigations that are imported simultaniously, so there may be no conflicts.

7) Perform an 'Export one investigation' on the database by selecting "Bailey" into directory C
Passes only if: CSV export procedure can successfully query the database for one specific investigation amongst many, write out the appropriate files in the right format, in the right locations.

8) Perform an 'Export one investigation' on the database by selecting "Lyons" into directory D
Passes only if: Same as 7), plus locations of files may not overlap.

9) Check whether file compare function works properly by executing it on positive and negative controls
Passes only if: Both positive and both negative controls for the compare function on existing test files are successful.

10) Compare directory A vs. C and B vs. D. Both comparisons should yield "true"
Passes only if: Directory content is equal (number and names of files) and file content is equal (number and content of each line).

Example result:

Step 9
Difference in files detected:
 * File 1: containsA.txt
 * File 2: containsB.txt
 * Line in file 1: A
 * Line in file 2: B
Difference in files detected:
 * File 1: containsB.txt
 * File 2: containsA.txt
 * Line in file 1: B
 * Line in file 2: A
Positive 'file content compare' control 1 (true): true
Positive 'file content compare' control 2 (true): true
Negative 'file content compare' control 1 (false): false
Negative 'file content compare' control 2 (false): false
Step 10
Comparing 'Bailey_extract' vs 'bailey_export'..
Comparing 'data' vs 'data'..
--> Directories are equal.
--> Directories are equal.
Comparing 'Lyons_extract' vs 'lyons_export'..
Comparing 'data' vs 'data'..
--> Directories are equal.
--> Directories are equal.
Directories of Bailey are equal: true
Directories of Lyons are equal: true

Extending the automated test cases

The following test cases have been implemented:

• Full data roundtrip using test sets Bailey and Lyons
• Full data roundtrip using test sets Keurentjes and Beamer

This code can be found in ​Case1.java and ​Case2.java in the folder ​handwritten/java/regressiontest/csv. Interested users can add more test cases in a similar fashion.