---
title: "The LongTable Class"
author:
- name: Petr Smirnov
  affiliation: &pm Princess Margaret Cancer Centre, University Health Network, Toronto Canada
  email: petr.smirnov@uhnresearch.ca
- name: Christopher Eeles
  affiliation: *pm
  email: christopher.eeles@uhnresearch.ca
package: CoreGx
output:
  BiocStyle::html_document
vignette: |
  %\VignetteIndexEntry{The LongTable Class}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---


```{r setup, include=FALSE}
library(formatR)
knitr::opts_chunk$set(echo = FALSE)
```


```{r dependencies, include=FALSE}
library(CoreGx)
library(data.table)
```

# Why Do We Need A New Class?

The current implementation for the `@sensitivity` slot in a `PharmacoSet` has some
limitations.

Firstly, it does not natively support dose-response experiments with
multiple drugs and/or cancer cell lines. As a result we have not been
able to include this data into a PharmacoSet thus far.

Secondly, drug combination data has the potential to scale to high dimensionality.
As a result we need an object that is highly performant to ensure
computations on such data can be completed in a timely manner.


# Design Philosophy

The current use case is supporting drug and cell-line combinations in
`PharmacoGx`, but we wanted to create something flexible enough to fit
other use cases. As such, the current class makes no mention of drugs or cell-lines,
nor anything specifically related to Bioinformatics or Computation Biology. Rather, we tried to design a general purpose data structure which could support
high dimensional data for any use case.

Our design takes the best aspects
of the `SummarizedExperiment` and `MultiAssayExperiment` class and implements
them using the `data.table` package, which provides an R API to a rich set of
tools for high performance data processing implemented in C.

# Anatomy of a LongTable

## Class Diagram

```{r class_diagram, fig.wide=TRUE, caption='LongTable Class Diagram'}
knitr::include_graphics('LongTableClassDiagram.png')
```

We have borrowed directly from the `SummarizedExperiment` class
for the `rowData`, `colData`, `metadata` and `assays` slot names.
We also implemented the `SummarizedExperiment` accessor generics for the `LongTable`.

## Object Structure and Cardinality

There are, however, some important differences which make this object more
flexible when dealing with high dimensional data.

```{r long_table_structure, fig.wide=TRUE, caption='LongTable Structure'}
knitr::include_graphics('LongTableStructure.png')
```

Unlike a `SummarizedExperiment`, there are three distinct
classes of columns in `rowData` and `colData`.

The first is the `rowKey` or `colKey`, these are implemented internally to keep mappings between each assay and the associated samples or drugs; these will not be returned by the accessors by default. The second is the `rowIDs` and `colIDs`, these hold all of the
information necessary to uniquely identify a row or column and are used to
generate the `rowKey` and `colKey`. Finally, there are the `rowMeta` and `colMeta`
columns, which store any additional data about samples or drugs not required
to uniquely identify a row in either table.

Within the assays the `rowKey` and `colKey` are combined to form a primary key
for each assay row. This is required because each assay is stored in 'long'
format, instead of wide format as in the assay matrices within a
`SummarizedExperiment`. Thanks to the fast implementation of binary search
within the `data.table` package, assay tables can scale up to tens or even
hundreds of millions of rows while still being relatively performant.

Also worth noting is the cardinality between `rowData` and `colData` for a given
assay within the assays list. As indicated by the lower connection between these
tables and an assay, for each row or column key there may be zero or more rows in
the assay table. Conversely for each row in the assay there may be zero or one key
in `colData` or `rowData`. When combined, the `rowKey` and `colKey` for a given
row in an assay become a composite key which maps that to


# Building a LongTable

he current implementation of the `buildLongTable` function is able to assemble
a `LongTable` object from two sources. The first is a single large table with
all assays, row and column data contained within it. This is the structure of the Merck drug combination data that has been used to test the data structure thus far.

```{r head_data, echo=TRUE}

filePath <- '../data/merckLongTable.csv'
merckDT <- fread(filePath, na.strings=c('NULL'))
colnames(merckDT)
```


```{r sample_data, fig.width=80, echo=TRUE}
knitr::kable(head(merckDT)[, 1:5])
```

```{r sample_data2, fig.width=80, echo=TRUE}
knitr::kable(head(merckDT)[, 5:ncol(merckDT)])
```
We can see that all the data related to the treatment response experiment is contained within this table.

## Single Assays Table

To build a `LongTable` object from this file:

```{r build_from_single_table_file, echo=TRUE}
rowDataCols <- list(
    c(cell_line1="cell_line", BatchID="BatchID"))
colDataCols <- list(
    c(drug1='drugA_name', drug2='drugB_name',
     drug1dose='drugA Conc (uM)', drug2dose='drugB Conc (uM)'),
    c(comboName='combination_name'))
assayCols <- list(viability=paste0('viability', seq_len(4)),
                  viability_summary=c('mu/muMax', 'X/X0'))
longTable <- buildLongTable(from=filePath, rowDataCols,
                            colDataCols, assayCols)
```


```{r show_method}
longTable
```
This function will also work if directly passed a `data.table` or `data.frame`
object:

```{r from_single_table, eche=TRUE}
longTable1 <- buildLongTable(from=merckDT, rowDataCols, colDataCols, assayCols)

paste0('All equal? ', all.equal(longTable, longTable1))
```

## Multiple Assay Tables

The second option for building a `LongTable` is to pass it a list of different
assays with a shared set of row and column identifiers. We haven't had a chance to testing this functionality with real data yet, but do have a toy example.


```{r from_list_of_tables, echo=TRUE}
assayList <- assays(longTable, withDimnames=TRUE, metadata=TRUE, key=FALSE)
assayList$new_viability <- assayList$viability  # Add a fake additional assay
assayCols$new_viability <-  assayCols$viability  # Add column names for fake assay
longTable2 <- buildLongTable(from=assayList, lapply(rowDataCols, names), lapply(colDataCols, names), assayCols)
```

```{r show_method2}
longTable2
```
We can see that a new assay has been added to the `LongTable` object when passed
a list of assay tables containing the required row and column IDs. Additionally, any row or
column IDs not already in rowData or colData will be appended to these slots automatically!

# LongTable Object

As mentioned previously, a `LongTable` has both list and table like behaviours.
For table like operations, a given `LongTable` can be thought of as a `rowKey`
by `colKey` rectangular object.

To support `data.frame` like sub-setting for this
object, the constructor makes pseudo row and column names, which are the ID columns
for each row of `rowData` or `colData` pasted together with a ':'.

## Row and Column Names

```{r rownames, echo=TRUE}
head(rownames(longTable))
```

We see that the rownames for the Merck `LongTable` are the cell line name
pasted to the batch id.

```{r colnames, echo=TRUE}
head(colnames(longTable))
```

For the column names, a similar pattern is followed by combining the colID
columns in the form 'drug1:drug2:drug1dose:drug2dose'.

## `data.frame` Subsetting

We can subset a `LongTable` using the same row and column name syntax as
with a `data.frame` or `matrix`.

```{r subset_dataframe_character, echo=TRUE}
row <- rownames(longTable)[1]
columns <- colnames(longTable)[1:2]
longTable[row, columns]
```

### Regex Queries

However, unlike a `data.frame` or `matrix` this subsetting also accepts partial
row and column names as well as regex queries.

```{r rowdata_coldata, echo=TRUE}
head(rowData(longTable), 3)
head(colData(longTable), 3)
```

For example, if we want to get all instance where '5-FU' is the drug:

```{r simple_regex, echo=TRUE}
longTable[, '5-FU']
```

This has matched all colnames where 5-FU was in either drug1 or drug2. If we only want to match drug1, we have several options:

```{r column_specific_regex, echo=TRUE}
all.equal(longTable[, '5-FU:*:*:*'], longTable[, '^5-FU'])
```

## `data.table` Subsetting

In addition to regex queries, a `LongTable` object supports arbitrarily complex
subset queries using the `data.table` API. To access this API, you will need to use the `.` function, which allows you
to pass raw R expressions to be evaluated inside the `i` and `j` arguments
for `dataTable[i, j]`.

For example if I want to subset to rows where the cell line is VCAP and columns
where drug1 is Temozolomide and drug2 is either Lapatinib or Bortezomib:

```{r , echo=TRUE}
longTable[.(cell_line1 == 'CAOV3'),  # row query
          .(drug1 == 'Temozolomide' & drug2 %in% c('Lapatinib', 'Bortezomib'))]  # column query
```

We can also invert matches or subset on other columns in `rowData` or `colData`:

```{r echo=TRUE}
subLongTable <-
  longTable[.(BatchID != 2),
            .(drug1 == 'Temozolomide' & drug2 != 'Lapatinib')]
```

To show that it works as expected:

```{r echo=TRUE}
print(paste0('BatchID: ', paste0(unique(rowData(subLongTable)$BatchID), collapse=', ')))
print(paste0('drug2: ', paste0(unique(colData(subLongTable)$drug2), collapse=', ')))
```

# Accessor Methods

## rowData

```{r echo=TRUE}
head(rowData(longTable), 3)
```

```{r echo=TRUE}
head(rowData(longTable, key=TRUE), 3)
```

## colData

```{r echo=TRUE}
head(colData(longTable), 3)
```

```{r echo=TRUE}
head(colData(longTable, key=TRUE), 3)
```

## assays

```{r echo=TRUE}
assays <- assays(longTable)
assays[[1]]
```


```{r echo=TRUE}
assays[[2]]
```

```{r echo=TRUE}
assays <- assays(longTable, withDimnames=TRUE)
colnames(assays[[1]])
```

```{r echo=TRUE}
assays <- assays(longTable, withDimnames=TRUE, metadata=TRUE)
colnames(assays[[2]])
```


```{r echo=TRUE}
assayNames(longTable)
```

Using these names we can access specific assays within a `LongTable`.

## assay

```{r echo=TRUE}
colnames(assay(longTable, 'viability'))
assay(longTable, 'viability')
```

```{r echo=TRUE}
colnames(assay(longTable, 'viability', withDimnames=TRUE))
assay(longTable, 'viability', withDimnames=TRUE)
```