Binary classification

2017 · Machine Learning

Problem description:

Determine whether a person makes over 50K a year

Hello!

Logistic regression is a statistical model that in its basic form uses a logistic function to model a binary dependent variable
Step 1: Function Set \[ f_{w,b}(x) = \sigma\left(\sum_i w_i x_i+b\right) \] Sigmoid function: \[\sigma(z)=\frac{1}{1+e^{-z}}\]
Step 2: Goodness of a Function Training data: $(x^n,\hat(y)^n)$ \[L(f)=\sum_n C(f(x^n),\hat(y)^n)\]
Step 3: Find the best function \[w_i\leftarrow w_i-\eta\sum_n \left(\hat(y)^n-f_{w,b}(x^n)\right)x^n_i\]

In General, A Discriminative model ‌models the decision boundary between the classes. A Generative Model ‌explicitly models the actual distribution of each class.
Step 1: Find $\mu_1, \mu_1, \Sigma_1, \Sigma_2$
Step 2: Compute the corresponding $w,b$ \[w^T=(\mu_1-\mu_2)^T\Sigma^{-1}\] \[b=-\frac{1}{2}(\mu_1)^T(\Sigma_1)^{-1}\mu_1+\frac{1}{2}(\mu_2)^T(\Sigma_2)^{-1}\mu_2+\text{ln}\frac{N_1}{N_2}\]