preprocessing

Preprocess Corpus Data

The rec-emb data model is optimized for fast training of rec-emb models. It includes the following:

• a lexicon: it holds id-string mappings, string-hash mappings and embedding vectors
• a structure: a serialized, typed graph that consists of two numpy arrays. One data array is holding the symbol type for each node. That are string hashes which can be converted into lexicon ids. The other is a sparse adjacency matrix encoding edge data.

Currently, the project provides three data sources:

• SICK corpus(paper). The SICK corpus consists of english sentence pairs extracted from image descriptions that are annotated with a relatedness score and an entailment class (one of neutral, entailment, or contradiction).
• SemEval 2010 Task 8 relation extraction corpus(paper). It contains english sentences where argument pairs are annotated with one out of nine abstract relation types, e.g. cause-effect or message-topic.
• Large Movie Review Dataset (IMDB)(paper). This binary sentiment analysis dataset consists of 50,000 english IMDB movie reviews with clear ratings: negative reviews have a score <= 4 out of 10, and a positive reviews have a score >= 7 out of 10.

NOTE: SICK and SemEval2010T8 are included here (datasets folder). The IMDB dataset has to be downloaded from http://ai.stanford.edu/~amaas/data/sentiment and extracted into the same folder as the other datasets.

The preprocessing happens in two steps:

1. parsing: This results in one jsonline file per input file (or directory in the case of IMDB) containing records in an intermediate format that is a very much simplified version of NIF
2. conversion: Convert into compact, graph based format (rec-emb format) to train the models with.

NOTE: Per default, two structural variants are created in the conversion step: (1) word nodes are linked directly, or (2) word nodes are linked via dependency edge type nodes. However, the former still includes dependency type data by adding that as direct children of the respective word nodes.

The final rec-emb data can be visualized with the visualization tool.

HOW TO preprocess

1. Install docker and docker-compose.
2. Clone the repo and switch into this folder:

git clone https://github.com/ArneBinder/recursive-embedding.git
cd recursive-embedding/docker/preprocessing

3. Download and extract glove.840B.300d.txt, e.g. from here.
4. Rename .env.dev (or copy) to .env and adapt its parameters.
5. Set execution permission for scripts: chmod +x scripts/*.sh
6. Optional: adapt the script files, e.g.
   • use Spacy instead of CoreNLP (parameter --parser),
   • restrict output to create only direct (default) or edge (flag -e) linked structure, or
   • adjust the minimal node type count (parameter -m): node types (e.g. words) that occur less then this value are replaced with the UNKNOWN type.
7. To start the parsing process, execute from current folder:

docker-compose up corpus-parse

9. Afterwards, convert to rec-emb data format:

docker-compose up corpus-convert