Preprocessing data using different techniques

Getting ready

1
2
3
4

import numpy as np
from sklearn import preprocessing 

data = np.array([[3, -1.5, 2, -5.4], [0, 4, -0.3, 2.1], [1, 3.3, -1.9, -4.3]]) ~~~~

How to do it…

1
2
3
4

# Mean removal
data_standardized = preprocessing.scale(data)
print "\nMean =", data_standardized.mean(axis=0)
print "Std deviation =", data_standardized.std(axis=0)

• python preprocessor.py

Mean = [ 5.55111512e-17 -1.11022302e-16 -7.40148683e-17 -7.40148683e-17]

Std deviation = [ 1. 1. 1. 1.]

1
2
3
4

# Scaling 
data_scaler = preprocessing.MinMaxScaler(feature_range=(0.1))
data_scaled = data_scaler.fit_transform(data)
print "\nMin max scaled data =", data_scaled 

1
2
3

# Normalization
data_normalized = preprocessing.normalize(data, norm='l1')
print "\nL1 normalized data =", data_normalized

1
2
3

# Binarization
data_binarized = preprocessing.Binarizer(threshold=1.4).transform(data)
print "\nBinarized data =", data_binarized

>

1
2
3
4
5

# One Hot Encoding
encoder = preprocessing.OneHotEncoder()
encoder.fit([[0, 2, 1, 12], [1, 3, 5, 3], [2, 3, 2, 12],[1, 2, 4, 3]])
encoded_vector = encoder.transform([[2, 3, 5, 3]]).toarray()
print "\nEncoded vector =", encoded_vector

Label encoding

How to do it…

1
2
3
4
5
6

from sklearn import preprocessing
label_encoder = preprocessing.LabelEncoder()
input_classes = ['audi', 'ford', 'audi', 'toyota', 'ford', 'bmw']
label_encoder.fit(input_classes)
print "\nClass mapping:"
for i, item in enumerate(label_encoder.classes_): print item, '-->', i

>

1

labels =

Linear Algebra

• Scalar

• Vector

  $$\mathbf x=\begin{bmatrix} x_1\\x_2\\ \vdots\\x_n \end{bmatrix}$$

• The $L_p$ norm of a vector

• The $L_1$ norm of a vector

• The $L_2$ norm of a vector

• The $L_\infty$ norm of a vector

• The distance of two vectors $\mathbf x_1,\mathbf x_2$

• A set of vectors $\mathbf x_1,\mathbf x_2,\cdots,\mathbf x_m$ are linearly independent

not exist a set of scalars $\lambda_1,\lambda_2,\cdots,\lambda_m,$ which are not all 0

• Matrix

  $$\mathbf A=\begin{bmatrix} a_{11}&a_{12}&\cdots&a_{1n}\\ a_{21}&a_{22}&\cdots&a_{2n}\\ \vdots&\vdots&\ddots&\vdots\\ a_{m1}&a_{m2}&\cdots&a_{mn} \end{bmatrix}$$

• Matrix product

  $$\mathbf A\in\mathbb R^{m\times n}$$ $$\mathbf B\in\mathbb R^{n\times p}$$ $$\mathbf C_{ij}=\sum_{k=1}^{n}\mathbf A_{ik}\mathbf B_{kj}$$ $$\mathbf C\in\mathbb R^{m\times p}$$

• determinant

  $$det(\mathbf A)=\sum_{k_1k_2\cdots k_n}(-1)^{\tau(k_1k_2\cdots k_n)} a_{1k_1}a_{2k_2}\cdots a_{nk_n}$$ $$k_1k_2\cdots k_n\ is\ a\ permutation\ of\ 1,2,\cdots,n$$ $$\tau(k_1k_2\cdots k_n)\ is\ the\ inversion\ number\ of\ the\ permutationk_1k_2\cdots k_n$$

• The inverse matrix

  $$\mathbf A^{-1}\mathbf A=\mathbf I$$

• The transpose of matrix

  $$\mathbf A_{ij}^{T}=\mathbf A_{ji}$$

• The Hadamard product

  $$\mathbf A\in\mathbb R^{m\times n}$$ $$\mathbf B\in\mathbb R^{m\times n}$$ $$\mathbf C_{ij}=\mathbf A_{ij}\mathbf B_{ij}$$ $$\mathbf C\in\mathbb R^{m\times n}$$

• Tensor: An array with arbitrary dimension

Probability theory

• joint probability

  $$P(X=x_1,Y=y_1)$$

• conditional probability

  $$P(X=x_1|Y=y_1)$$

• The sum rule

  $$P(X=x)=\sum_yP(X=x,Y=y)$$

• The product rule

  $$P(X=x,Y=y)=P(Y=y|X=x)P(X=x)$$

• Bayes formula

  $$P(Y=y|X=x)=\frac{P(X=x,Y=y)}{P(X=x)}=\frac{P(X=x|Y=y)P(Y=y)}{P(X=x)}$$ $$P(X_i=x_i|Y=y)=\frac{P(Y=y|X_i=x_i)P(X_i=x_i)}{\sum_{j=1}^nP(Y=y|X_j=x_j)}$$

• The chain rule

  $$P(X_1=x_1,\cdots,X_n=x_n)\\ =P(X_1=x_1)\prod_{i=2}^{n}P(X_i=x_i|X_1=x_1,\cdots,X_{i-1}=x_{i-1})$$

• The expectation of f(x)

  $$\mathbb E[f(x)]=\sum_xP(x)f(x)$$

• The variance of f(x)

  $$\begin{aligned} Var(f(x)&=\mathbb E[(f(x)-\mathbb E[f(x)])^2]\\ &=\mathbb E[f(x)^2]-\mathbb E[f(x)]^2 \end{aligned}$$

• The standard deviation

  $$\sqrt{Var(f(x)}$$

• Covariance

  $$Cov(f(x),g(y))=\mathbb E[(f(x)-\mathbb E[f(x)])(g(y)-\mathbb E[g(y)])]$$

• Gaussian distribution

  $$N(x|\mu,\sigma^2)=\sqrt{\frac{1}{2\pi\sigma^2}}exp(-\frac{1}{2\sigma^2}(x-\mu)^2)$$ $$\mu\ is\ the\ mean\ of\ variable\ x$$ $$\sigma^2\ is\ the\ variance$$

• Bernoulli distribution

  $$P(X=x)=p^x(1-p)^{(1-x)},x \in \left\{0,1\right\}$$

• Binomial distribution

  $$P(Y=k)=\begin{pmatrix}N\\k\end{pmatrix}p^k(1-p)^{(N-k)}$$

• Laplace distribution

  $$P(x|\mu,b)=\frac{1}{2b}exp(-\frac{\lvert x -\mu\rvert}{b})$$

Graph theory

• Adjacency matrix

  $$A_{ij}=\left\{ \begin{aligned} &1\ if\left\{v_i,\ v_j\right\}\in E\ and\ i\ \neq\ j,\\ &0\ otherwise. \end{aligned} \right.$$

• Degree matrix

  $$D_{ii}=d(v_i)$$

• Laplacian matrix

  $$L=D-A$$ $$L_{ij}=\left\{ \begin{aligned} &d(v_i)\ if\ i=j,\\ &-1\ if\left\{v_i,v_j\right\}\in E\ and\ i\ \neq\ j,\\ &0\ otherwise. \end{aligned} \right.$$

• Symmetric normalized Laplacian

  $$\begin{aligned} L^{sym}&=D^{-\frac{1}{2}}LD^{-\frac{1}{2}}\\ &=I-D^{-\frac{1}{2}}AD^{-\frac{1}{2}} \end{aligned}$$

• Random walk normalized Laplacian $$L^{rw}=D^{-1}L=I-D^{-1}A$$ $$L_{ij}^{rw}= \left\{ \begin{aligned} &1\ if\ i=j\ and\ d(v_i)\ \neq\ 0,\\ &-\frac{1}{d(v_i)}\ if\left\{v_i,v_j\right\}\in E\ and\ i\ \neq\ j,\\ &0\ otherwise. \end{aligned} \right.$$
• Incidence matrix

For a directed graph

For a undirected graph

Abstract

Graph neural networks are neural models that capture the dependence of graphs via message passing between the nodes of graphs.

• propose a general design pipeline for GNN models

• discuss the variants of each component

• systematically categorize the applications

• propose four open problems for future research

Introduction

Graphs can be used across various areas

social networks

Graph Convolutional Networks with Markov Random Field Reasoning for Social Spammer Detection, 2020

natural science

Graph Networks as Learnable Physics Engines for Inference and Control, 2018

Interaction Networks for Learning about Objects, Relations and Physics, 2016

protein-protein interaction networks

Protein Interface Prediction using Graph Convolutional Networks, 2017

knowledge graphs

Knowledge Transfer for Out-of-Knowledge-Base Entities: A Graph Neural Network Approach, 2017

other research areas

Learning Combinatorial Optimization Algorithms over Graphs, 2017

The fundamental motivations of graph neural networks

• Recursive Neural Networks are first utilized on directed acyclic graphs

  Supervised neural networks for the classification of structures,1997
  A general framework for adaptive processing of data structures, 1998

• Recurent Neural Networks and Feedforward Neural Networks

  The Graph Neural Network Model, 2009
  Neural Network for Graphs: A Contextual Constructive Approach, 2009

• CNNs result in the rediscovery of GNNs

  Gradient-based learning applied to document recognition, 1998

• the new era of deep learning

  Deep learning, 2015

• geometric deep learning

  Geometric deep learning: going beyond Euclidean data, 2017
  

• graph representation learning

  A Survey on Network Embedding, 2017
  Representation Learning on Graphs: Methods and Applications, 2017
  Network Representation Learning: A Survey, 2017
  A Comprehensive Survey of Graph Embedding: Problems, Techniques and Applications, 2017
  Graph embedding techniques, applications, and performance: A survey,2018

• word representations

  Efficient Estimation of Word Representations in Vector Space, 2013

• DeepWalk

  DeepWalk: Online Learning of Social Representations, 2014

• SkipGram model

  Efficient Estimation of Word Representations in Vector Space, 2013

• node2vec

  node2vec: Scalable Feature Learning for Networks, 2016

• LINE

  LINE: Large-scale Information Network Embedding, 2015

• TADW

  Network representation learning with rich text information, 2015

• drawbacks

  First, no parameters are shared between nodes in the encoder, which leads to computationally inefficiency, since it means the number of parameters grows linearly with the number of nodes.
  Second, the direct embedding methods lack the ability of generalization, which means they cannot deal with dynamic graphs or generalize to new graphs.

several comprehensive reviews on graph neural networks

Geometric deep learning: going beyond Euclidean data, 2017
Graph convolutional networks: a comprehensive review, 2019

• the most up-to-date survey papers on GNNs

  Deep Learning on Graphs: A Survey, 2018
  A Comprehensive Survey on Graph Neural Networks, 2019
  Machine Learning on Graphs: A Model and Comprehensive Taxonomy, 2020

several surveys focusing on some specific graph learning fields

• adversarial learning methods on graphs

  Adversarial Attack and Defense on Graph Data: A Survey, 2018
  A Survey of Adversarial Learning on Graphs, 2020

• a review over graph attention models

  Attention Models in Graphs: A Survey, 2018

• heterogeneous graph representation learning

  Heterogeneous Network Representation Learning: Survey, Benchmark, Evaluation, and Beyond

• existing GNN models for dynamic graphs

  Modeling Complex Spatial Patterns with Temporal Features via Heterogenous Graph Embedding Networks, 2020

• graph embeddings methods for combinatorial optimization.

  Graph Embedding for Combinatorial Optimization: A Survey

contributions

• We provide a detailed review over existing graph neural network models. We present a general design pipeline and discuss the variants of each module. We also introduce researches on theoretical and empirical analyses of GNN models.

• We systematically categorize the applications and divide the applications into structural scenarios and non-structural scenarios. We present several major applications and their corresponding methods for each scenario.

• We propose four open problems for future research. We provide a thorough analysis of each problem and propose future research directions.

General design pipeline of GNNs

Find graph structure

structural scenarios
non-structural scenarios

Specify graph type and scale

Directed/Undirected Graphs

Homogeneous/Heterogeneous Graphs

Static/Dynamic Graphs

Design loss function

Node-level
Edge-level
Graph-level

Supervised setting
Semi-supervised setting
Unsupervised setting

Build model using computational modules

Propagation Module
Sampling Module
Pooling Module

NDCN(Neural Dynamics on Complex Networks) combines ordinary differential equation systems (ODEs) and GNNs.

Instantiations of computational modules

Propagation modules - convolution operator

The emerging field of signal processing on graphs: Extending high-dimensional data analysis to networks and other irregular domains

• Spectral approaches $$\mathscr F(\mathbf x)=\mathbf U^T\mathbf x$$ $$\mathscr F^{-1}(\mathbf x)=\mathbf U\mathbf x$$

the normalized graph Laplacian

$$\mathbf L=\mathbf I_N-\mathbf D^{-\frac{1}{2}}\mathbf A\mathbf D^{-\frac{1}{2}}$$

A wavelet tour of signal processing

several typical spectral methods which design different filters $\mathbf g_w$

Spectral Network

Spectral Networks and Locally Connected Networks on Graphs

Deep Convolutional Networks on Graph-Structured Data

ChebNet

Wavelets on Graphs via Spectral Graph Theory

Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering

Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering

GCN

• Basic spatial approaches

• Attention-based spatial approaches

• General frameworks for spatial approaches

Propagation modules - recurrent operator

Propagation modules - skip connection

Reference Paper
Graph Neural Networks: A Review of Methods and Applications, 2018

Review

• Graph Neural Networks: A Review of Methods and Applications, 2018

• Deep Learning on Graphs: A Survey, 2018

• Relationanl Inductive Biases, Deep Learning, and Graph Neural Networks, 2018

• Geometric Deep Learning: Going beyond Euclidean data, 2017

• Computational Capabilities of Graph Neural Networks, 2009

• Neural Message Passing for Quantum Chemistry, 2017

• Non-local Neural Networks, 2018

• The Graph Neural Network Model, 2009

Model

• A new model for learning in graph domains, 2005

• Graph Neural Networks for Ranking Web Pages, 2005

• Gated Graph Sequence Neural Networks, 2015

• Geometric deep learning on graphs and manifolds using mixture model CNNs, 2016

• Spectral Networks and locally Connected Networks on Graphs, 2013

• Deep Convolutional Networks on Graph-Structure Data, 2015

• Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering, 2016

• Learning Convolutional Neural Networks for Graphs, 2016

• Semi-Supervised Classification with Graph Convolutional Networks, 2016

• Graph Attention Networks, 2017

• Deep Sets, 2017

• Graph Partition Neural Networks for Semi-Supervised Classification, 2018

• Covariant Compositional Networks For Learning Graphs, 2018

• Modeling Relational Data with Graph Convolutional Networks, 2018

• Stochastic Training of Graph Convolutional Networks with Variance Reduction, 2018

• Learning Steady-States of Iterative Algorithms over Graphs, 2018

• Deriving Neural Architectures from Sequence and Graph Kernels, 2017

• Adaptive Graph Convolutional Neural Networks, 2018

• Graph-to-Sequence Learning using Gated Graph Neural Networks, 2018

• Deeper insights into Graph Convolutional Networks for Semi-Supervised Learning, 2018

• Graphical-Based Learning Environments for Pattern Recognition, 2004

• A Comparison between Recursive Neural Networks and Graph Neural Networks, 2006

• Graph Neural Networks for Object Localization, 2006

• Knowledge-Guided Recurrent Neural Network Learning for Task-Oriented Action Prediction, 2017

• Semantic Object Parsing with Graph LSTM, 2016

• CelebrityNet: A Social Network Constructed from Large-Scale Online Celebrity Images, 2015

• Inductive Representation Learning on Large Graphs, 2017

• Graph Classification using Structural Attention, 2018

• Adversarial Attacks on Neural Networks for Graph Data, 2018

• Large-Scale Learnable Graph Convolutional Networks, 2018

• Contextual Graph Markov Model: A Deep and Generative Approach to Graph Processing, 2018

• Diffusion-Convolutional Neural Networks, 2016

• Neural networks for relational learning: an experimental comparison, 2011

• FastGCN: Fast Learning with Graph Convolutional Networks via Importance Sampling, 2018

• Adaptive Sampling Towards Fast Graph Representation Learning, 2018

Application

• Discovering objects and their relations from entanglrd scene representations, 2017

• A simple neural network module for relational reasoning, 2017

• Attend, Infer, Repeat: Fast Scene Understanding with Generative Models

• Beyond Categories: The Visual Memex Model for Reasoning About Object Relationships

• Understanding Kin Relationships in a Photo

• Graph-Structured Representations for Visual Question Answering

• Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition

• Few-Shot Learning with Graph Neural Networks

• The More You Know: Using Knowledge Graphs for Image Classification

• Zero-shot Recognition via Semantic Embeddings and Knowledge Graphs

• Rethinking Knowledge Graph Propagation for Zero-Shot Learning

• Interaction Networks for Learning about Objects, Relations and Physics

• A Compositional Object-Based Approach to Learning Physical Dynamics

• Visual Interaction Networks: Learning a Physics Simulator from Video

• Relational neural expectation maximization: Unsupervised discovery of objects and their interactions

• Graph networks as learnable physics engines for inference and control

• Learning Multiagent Communication with Backpropagation

• VAIN: Attentional Multi-agent Predictive Modeling

• Neural Relational Inference for Interacting Systems

• Translating Embeddings for Modeling Multi-relational Data

2D points

homogeneous vector $\mathbf{\tilde x}$

2D projective space

$$\textit P^2=\textit R^3-(0,0,0)$$

inhomogeneous vector $\mathbf{\overline x}$

2D lines

homogeneous vector

line equation

$$\mathbf{\textit l}=(\hat n_x,\hat n_y,d)=(\hat{\mathbf n},d)with\Vert{\hat{\mathbf n}}\Vert=1$$

Reference Book
Computer Vision: Algorithms and Applications by Richard Szeliski

Anaconda Prompt Operation

Create new environment

conda create - -name environment pakage

conda create - -name python3 python=3.8

Activate environment

conda activate environment

Exit environment

deactivate environment

Delete environment

conda remove -n environment - -all

Copy environment

conda create -n environment - -clone existing_environment

View environment information

conda info -e
conda env list
conda info - -envs

View python version

python -V

Install package

conda install package

View package information

conda search package

Install package in a specify environment

conda install -n environment package

Update package in a specify environment

conda update -n environment package

Delete package in a specify environment

conda remove -n environment package

View the installed packages in the current environment

conda list

View the installed packages in a specify environment

conda list -n environment