Word embedding
Lua error in package.lua at line 80: module 'strict' not found.
Word embedding is the collective name for a set of language modeling and feature learning techniques in natural language processing where words or phrases from the vocabulary are mapped to vectors of real numbers in a low-dimensional space relative to the vocabulary size ("continuous space").
Methods to generate this mapping include neural networks,[1][2] dimensionality reduction on the word co-occurrence matrix,[3][4][5] probabilistic models,[6] and explicit representation in terms of the context in which words appear.[7]
Word and phrase embeddings, when used as the underlying input representation, have been shown to boost the performance in NLP tasks such as syntactic parsing[8] and sentiment analysis.[9]
Contents
Development of Word Embedding Technique
The Word Embedding technique began to develop since 2000. Bengio et al. provided in a series of papers the "Neural probabilistic language models" to reduce the high dimensionality of words representations in contexts by "learning a distributed representation for words". (Benjo et al, 2003).[10] Roweis and Saul published in science how to use "locally linear embedding"(LLE) to discover representations of high dimensional data structure. [11]The area developed gradually and really took off after 2010, partly because important advances had been made since then on the quality of vectors and the training speed of the model.
There are many branches and many research groups working on word embedding. For example, the probably most famous group is lead by Tomas Mikolov (ex-Google employee, now in Facebook). In 2013, they offered a word2vec toolkit that can train words embedding models much faster than the previous approaches. Most of new word embedding techniques rely on a neural network architecture instead of more traditional "n-gram" models and unsupervised learning. [12]
For biological sequences: BioVectors
Word embeddings for n-grams in biological sequences (e.g. DNA, RNA, and Proteins) for bioinformatics applications have been proposed by Asgari and Mofrad.[13] Named bio-vectors (BioVec) to refer to biological sequences in general with protein-vectors (ProtVec) for proteins (amino-acid sequences) and gene-vectors (GeneVec) for gene sequences, this representation can be widely used in applications of deep learning in proteomics and genomics. The results presented by[13] suggest that BioVectors can characterize biological sequences in terms of biochemical and biophysical interpretations of the underlying patterns.
Software
Software for training and using word embeddings includes Tomas Mikolov's Word2vec, Stanford University's GloVe[14] and Deeplearning4j. Principal Component Analysis (PCA) and T-Distributed Stochastic Neighbor Embedding (t-SNE) are both used to reduce the dimensionality of word vector spaces and visualize word embeddings and clusters.[15]
See also
References
- ↑ Lua error in package.lua at line 80: module 'strict' not found.
- ↑ Lua error in package.lua at line 80: module 'strict' not found.
- ↑ Lua error in package.lua at line 80: module 'strict' not found.
- ↑ Lua error in package.lua at line 80: module 'strict' not found.
- ↑ Lua error in package.lua at line 80: module 'strict' not found.
- ↑ Lua error in package.lua at line 80: module 'strict' not found.
- ↑ Lua error in package.lua at line 80: module 'strict' not found.
- ↑ Lua error in package.lua at line 80: module 'strict' not found.
- ↑ Lua error in package.lua at line 80: module 'strict' not found.
- ↑ Lua error in package.lua at line 80: module 'strict' not found.
- ↑ Lua error in package.lua at line 80: module 'strict' not found.
- ↑ Lua error in package.lua at line 80: module 'strict' not found.
- ↑ 13.0 13.1 Lua error in package.lua at line 80: module 'strict' not found.
- ↑ Lua error in package.lua at line 80: module 'strict' not found.
- ↑ Lua error in package.lua at line 80: module 'strict' not found.
