User manual for XGAP

Introduction

The core prodocut of the dbGG project is:

• a data model for genetical gemics that researchers can use to describe relevant information on genetical genomics investigations in a standard way. We refer to the dbGG manuscript (submitted) and ‘description of data model’

From the data model a software infrastructure is generated to directly start using the model:

• a database for genetical genomics (dbGG) that researchers can use to store and retrieve actual investigation data in the data model on a large scale.

• a tab/comma delimited flat file format that researchers can use to exchange investigation data between dbGG instances.

• a graphical user interface that researchers can use to navigate, search and update individual data in the database software infrastructure

• several programmatic interfaces, currently in R-project, Java and web services, that can be used by programming biologists to automate data uploads/downloads on a large scale.

• a commandline import/export program that can be used from the commandline to upload/download complete investigations from/to the delimited flat file format.

This document describes use of the software infrastructure.

Using the grapical user interface

TODO.

Using the R interface

The R-interface of dbGG distinguishes between two classes of data types:

1. Annotations.

Annotations are lists of data that are stored as data.frame, e.g., each row describes a Marker. Each columnname refers to a particular property, e.g. ‘name’ or ‘molgenisid’. Rownames are ignored. For example:

name	chr	cm
PVV4	1	0
AXR-1	1	6.398
HH.335C-Col	1	10.786
DF.162L/164C-Col	1	12.913
EC.480C	1	15.059
EC.66C	1	21.846
GD.86L	1	23.802
g2395	1	27.749
CC.98L-Col/101C	1	31.212
AD.121C	1	41.271

2. Data matrices.

A data matrix contains data in tabular format, e.g. rownames refer to Marker, colnames refer to Probe, values indicate QTL p-value. Rownames refer to annotations and columnnames refering to annotations. Rownames and Columnnames are required. For example:

(note how first row has one element less because of the rownames column):

X1	X3	X4	X5	X6	X7	X8
PVV4	1	1	2	1	2	2	1
AXR-1	1	1	2	1	2	2	1
HH.335C-Col	1	1	1	1	2	2	1
DF.162L/164C-Col	1	1	1	1	2	2	1
EC.480C	1	1	1	1	2	2	1
EC.66C	1	1	1	1	2	2	1
GD.86L	1	1	1	1	2	2	1
g2395	2	1	1	1	2	2	1
CC.98L-Col/101C	1	1	1	NA	2	2	1

Below is described how to use to R-interface and its annotation and data matrix facilities.

Connect to dbGG

Connect to your dbGG server using command (edit to your servername!)

source("http://<yourhost>:8080/dbgg/api/R/")

#e.g. using demonstration server

source("http://gbicserver1.biol.rug.nl:8080/dbgg/api/R/")

#e.g. using local install

source("http://localhost:8080/dbgg/api/R/")

Download and upload annotations

Annotation data is described in this section.

• All annotations are handled inside R in tabular form using data.frames. E.g.

• Each has a name and molgenisid

• See document ‘TAB delimited format’ for details.

• For each annotation type there are ‘find’, ‘add’, and ‘find’ functions. E.g there are

• find.investigation(), add.investigation(), remove.investigation()

• find.marker(), add.marker, remove.marker()

• See all methods by calling ls()

• Find results can be limited by setting search parameters:

# limit to only markers from experiment 1.

find.marker(investigation=1)

• Default find parameters can be set. These parameters are then always used as filter.

# use only data from investigation 1

use.investigation(molgenisid=1)

# also can be done using investigation name

use.investigation(name=”My investigation”)

find.marker()

# identical results to find.marker(investigation=1)

• Add or remove annotations either by setting the properties individually or by passing them all in one data.frame. Note that the result of ‘add’ is a dataframe with the added information, but now including any default or autogenerated values (e.g. molgenisid)

my_investigations = add.investigation(name=c(“Inv1”,”Inv2”)

remove.investigation(my_investigations)

Download and upload data matrices

The dbGG data model has a flexible structure to deal with data matrices.

In the database these are stored using Data and DataElement:

• ‘Data’ to store the properties of the matrix (rowtype, coltype, valuetype).

• ‘DoubleDataElement’ or ‘TextDataElement’ to store the double or text values of the matrix.

• Each record of Double/TextDataElement must refer to DimensionElement annotations (e.g. Probe, Strain, Individual).

An conventient interface to deal with data matrices has been added. Instead of using find/add/remove.Data and find/add/remove.DataElement. one can use find.datamatrix, add.datamatrix and remove.datamatrix:

add.datamatrix

add.datamatrix(.data_matrix, name=, investigation= , rowtype= , coltype= , valuetype=)

Description of parameters:

.data_matrix First parameter is the data matrix to be stured (as.matrix)

name The name of the data set. Should be unique within and investigation.

investigation The molgenisid of the investigation. Doesn’t need to be set if use.investigation() has been called before.

rowtype The type of the rows. Each rowname must refer to an instance of this type. E.g. rowtype=”marker” means that for each rowname there can be a marker$name found.

coltype The type of the rows.

Each rowname must refer to an instance of this type. E.g. rowtype=”marker” means that for each rowname there can be a marker$name found.

valuetype The type of the values in the matrix, either ‘text’ or ‘double’.

If ‘text’ then each matrix cel is added as one row in TextDataElement. If ‘double’ each matrix cel is added as one row in DoubleDataElement.

When executed succesfully, one row is added to Data, and many rows to either DoubleDataElement or TextDataElement.

find.datamatrix / remove.datamatrix

Functions:

find.datamatrix(molgenisid=, name=, investigation=)

#retrieves a data matrix

remove.datamatrix(molgenisid=, name=, investigation=)

#removes a data matrix

Description of parameters:

molgenisid the unique idea of the data set.

Use ‘find.data()’ to get a list of data matrices available.

name the name of the dataset (unique within this investigation).

investigation the molgenisid of the investigation

Note: to search one must either provide a {molgenisid} or the {name and investigation id).

Examples of data matrix functions

Use find.datamatrix, add.datamatrix, remove.datamatrix:

#add text matrix with rows refer to Marker and column to Individual

add.datamatrix(matrix, name=”my genotypes”, rowtype=”Marker”, coltype=”Individual”, valuetype=”Text”)

#add double matrix with rows refer to Probe and column to Individual

add.datamatrix(matrix, name=”my gene expression”, rowtype=”Probe”, coltype=”Individual”, valuetype=”Double”)

#add double matrix with rows refer to Probe and column to Marker

#assume Probe and Marker are not known

add.marker(name=colnames(matrix) #adds marker without annotation

add.probe(name=rownames(matrix) #adds probes without annotation

add.datamatrix(matrix, name=”my QTLs”, rowtype=”Probe”, coltype=”Individual”, valuetype=”Double”)

#find a data matrix

#note: max one result, in contrast to find.annotation

geno <- find.datamatrix(name=”my genotypes)

#remove a data matrix

remove.datamatrix(name=”my gene expression”)

#list existing data matrices

#note: is a normal annotation function

find.data()

Using the web services interface

TODO

Using the commandline client

Import whole investigation data from tab delimited files

Export whole investigation as tab delimited files.

TODO

Appendix: a complete R script using dbGG

Copy paste ready example code, given that you update the host (first line)

(Tested on R 2.4.1 and 2.7.0)

#connect to dbGG

#source("http://gbicserver1.biol.rug.nl:8080/molgenis4dbgg/api/R")

#Uncomment if RCurl is missing

#source("http://bioconductor.org/biocLite.R")

#biocLite("RCurl")

#use existing data from MetaNetwork for example

#install from zipfile from http://gbic.biol.rug.nl/spip.php?rubrique48

library(MetaNetwork)

#

#ADD DATA

#-first annotations

#-second data matrices (referering to annotatations)

#

#add investigation

investigation_return = add.investigation(name="Example investigation MetaNetwork", start="2008-05-31", end="2009-05-31")

use.investigation(name="Example investigation MetaNetwork")

#use sets globabl parameter so we don't need to pass parameter'investigation=<number>' on every call

#add markers

data(markers)

markers = as.data.frame(markers)

markers_return = add.markers(name=rownames(markers), chr=markers$chr, cm=markers$cm)

#add individuals (take name from genotypes)

data(genotypes)

individuals = data.frame(name=colnames(genotypes))

individuals_return = add.individual(individuals)

#add metabolites (take name from traits)

data(traits)

metabolites = data.frame(name=rownames(traits))

metabolites_return = add.metabolites(metabolites)

#add data matrices for genotypes, metabolite expression and qtl profiles

#data(traits)

#data(genotypes)

data(qtlProfiles)

add.datamatrix(genotypes, name="the genotypes", rowtype="marker", coltype="individual", valuetype="text")

add.datamatrix(traits, name="the metabolite expression", rowtype="metabolite", coltype="individual", valuetype="text")

add.datamatrix(qtlProfiles, name="the QTL profiles", rowtype="metabolite", coltype="marker", valuetype="double")

#

# VERIFY DATA uploaded and downloaded data

#

#retrieve the uploaded data

geno2 <- find.datamatrix(name="the genotypes")

traits2 <- find.datamatrix(name="the metabolite expression")

qtls2 <- find.datamatrix(name="the QTL profiles")

#is it identical???

identical(genotypes,geno2)

identical(traits,traits)

identical(qtlProfiles,qtls2)

#ai, there is rounding going on somewhere!

format(qtlProfiles[12,1],digits=20)

format(qtls2[12,1],digits=20)

#as this already happens during write.csv this seems partly due to R itself !!!

#write.table(qtlProfiles, file="c:/test.txt")

#qtlProfiles_copy = read.table(file="c:/test.txt")

#identical(qtlProfiles,qtlProfiles_copy)

#

all.equal(qtlProfiles,qtls2)

#compare annotations

identical(markers_return$name,rownames(markers))

identical(markers_return$name,rownames(genotypes))

identical(markers_return$name,colnames(qtlProfiles))

identical(metabolites_return$name,rownames(traits))

identical(individuals_return$name,colnames(genotypes))

identical(individuals_return$name,colnames(traits))

#

# REMOVE DATA again

# in reverse order

#

#remove matrices

remove.datamatrix(name="the genotypes")

remove.datamatrix(name="the metabolite expression")

remove.datamatrix(name="the QTL profiles")

#remove annotations

remove.metabolite(metabolites_return)

remove.individual(individuals_return)

remove.marker(markers_return)

remove.investigation(investigation_return)