## Sunday, November 29, 2015

### Perceptron Learning using the Delta rule - Gradient Descent - Java (Training for the boolean AND function)

The delta rule uses gradient descent to approximate the target function. It is guaranteed to converge even if the function is not linearly separable, unlike the perceptron learning rule.

We'll be training the perceptron to learn the boolean AND function. We'll use two weights w1, w2. These weights are approximated by the delta rule to fit the training examples most closely. Next we need to find the threshold value (w0) which we will use to classify the training examples accurately.

I've represented 0 with -1, 1 with 1 itself.

Code:

```class Example{
int x1, x2;
double o;
Example(int x1, int x2, double o){
this.x1 = x1;
this.x2 = x2;
this.o = o;
}
}

public class DeltaRulePerceptron {
double w1, w2;
double dw1, dw2;
double n;

DeltaRulePerceptron(){
//Random values
w1 = 0.1;
w2 = 0.7;
n = 0.05;
}

public double computeOutput(Example example){
return example.x1 * w1 + example.x2*w2;
}

public void trainWithDelta(Example[] examples){
for(int i=0;i<1000;++i){

//            System.out.println("Iteration #"+i);
dw1 = 0;
dw2 = 0;

for(Example ex:examples){
double o = computeOutput(ex);
double t = ex.o;
//                System.out.println("o = "+o+" t = "+t);

dw1 = dw1 + n*(t-o)*ex.x1;
dw2 = dw2 + n*(t-o)*ex.x2;
}

w1 += dw1;
w2 += dw2;
}
}

public static void main(String[] args){
DeltaRulePerceptron d = new DeltaRulePerceptron();

//AND boolean function
Example[] examples = new Example[]{
new Example(-1, -1, -1),
new Example(-1 , 1, -1),
new Example( 1, -1, -1),
new Example( 1,  1, 1)
};
d.trainWithDelta(examples);
System.out.println("(AND) Trained weights : "+d.w1+" "+d.w2);

//bias
double w0 = 0.5;
System.out.println("w0 = "+w0);

//Test
System.out.println(d.sign(w0, d.computeOutput(examples[0])));
System.out.println(d.sign(w0, d.computeOutput(examples[1])));
System.out.println(d.sign(w0, d.computeOutput(examples[2])));
System.out.println(d.sign(w0, d.computeOutput(examples[3])));

//XOR - fails - A single layer perceptron can't represent a XOR function
examples = new Example[]{
new Example(-1, -1, 1),
new Example(-1 , 1, -1),
new Example( 1, -1, -1),
new Example( 1,  1, 1)
};
d.trainWithDelta(examples);
System.out.println("(XOR) Trained weights : "+d.w1+" "+d.w2);
System.out.println("w0 = "+w0);

System.out.println(d.sign(w0, d.computeOutput(examples[0])));
System.out.println(d.sign(w0, d.computeOutput(examples[1])));
System.out.println(d.sign(w0, d.computeOutput(examples[2])));
System.out.println(d.sign(w0, d.computeOutput(examples[3])));
}

public int sign(double w0, double output){
return output-w0>0?+1:-1;
}
}

```
Output:

(AND) Trained weights : 0.49999999999999994 0.5000000000000002
w0 = 0.5
-1
-1
-1
1

(XOR) Trained weights : 0.0 5.551115123125783E-17
w0 = 0.5
-1
-1
-1
-1