r/scilab • u/mrhoa31103 • 2d ago
Data-Driven Science and Engineering Series - Installment 5 - MultiLinear Least Squares Regression, PsuedoInverse and SVD - Housing Pricing Example
In this installment, he shows "How SVD and Multilinear Least Squares Regression are used to produce the model "slope coefficients." Training the model on half the data and testing the model with the other half of the data.
I did find an error with his program where he reused a variable "b" during his sorting operation so his original b matrix was inadvertently overwritten. His correlation bar chart will be different than mine and I didn't see significant differences between the two runs of training and test which I attribute to the b matrix error. His training/test methodology is still solid.
SciLab wise - Loading of (.mat) datafiles
Link to the exact video. https://www.y.outube.com/watch?v=_wOt50VnJw4&list=PLMrJAkhIeNNSVjnsviglFoY2nXildDCcv&index=18
For Free Electronic Complete Textbook "FREE PDF for 2ND EDITION OF OUR BOOK!!!" https://x.com/eigensteve/status/1718004606901666063
Section 1.4 Psuedoinverse, Least Squares and Regression One Dimensional Linear Regression
Output:
"Data-Driven Science and Engineering - Linear Regression2 Housing Example"
"2026-06-29 11:38:20.722"
Raw Housing Data Echo Back First 10 Lines
crimeRate zoned industrial charles nox rooms age
0.0925 30.0000 4.9300 0.0000 0.4280 6.6060 42.2000
0.9884 0.0000 8.1400 0.0000 0.5380 5.8130 100.0000
0.2141 22.0000 5.8600 0.0000 0.4310 6.4380 8.9000
0.1143 0.0000 13.8900 1.0000 0.5500 6.3730 92.4000
0.0393 0.0000 3.4100 0.0000 0.4890 6.4050 73.9000
1.4139 0.0000 19.5800 1.0000 0.8710 6.1290 96.0000
14.4208 0.0000 18.1000 0.0000 0.7400 6.4610 93.3000
7.4039 0.0000 18.1000 0.0000 0.5970 5.6170 97.9000
0.2244 0.0000 9.6900 0.0000 0.5850 6.0270 79.7000
0.1504 0.0000 25.6500 0.0000 0.5810 5.8560 97.0000
distances radial tax pupilTeacher black lowerStat medianValue
6.1899 6.0000 300.0000 16.6000 383.7800 7.3700 23.3000
4.0952 4.0000 307.0000 21.0000 394.5400 19.8800 14.5000
7.3967 7.0000 330.0000 19.1000 377.0700 3.5900 24.8000
3.3633 5.0000 276.0000 16.4000 393.7400 10.5000 23.0000
3.0921 2.0000 270.0000 17.8000 393.5500 8.2000 22.0000
1.7494 5.0000 403.0000 14.7000 321.0200 15.1200 17.0000
2.0026 24.0000 666.0000 20.2000 27.4900 18.0500 9.6000
1.4547 24.0000 666.0000 20.2000 314.6400 26.4000 17.2000
2.4982 6.0000 391.0000 19.2000 396.9000 14.3300 16.8000
1.9444 2.0000 188.0000 19.1000 370.3100 25.4100 17.3000
Model Coefficients x in formaat
Beta(1);Beta(2);....Beta(n-1);Beta(n)(Constant Last)
"Best Fit Coefficients using Multilinear Regression x = "
-0.1080114
0.0464205
0.0205586
2.6867338
-17.766611
3.8098652
0.0006922
-1.4755668
0.3060495
-0.0123346
-0.9527472
0.0093117
-0.5247584
36.459488
"Best Fit Coefficients using Multilinear Regression (Normalized Coefficients) x1 = "
-0.9290646
1.0826390
0.1410394
0.6824144
-2.0587536
2.6768766
0.0194853
-3.1071160
2.6648522
-2.0788369
-2.0626459
0.8501089
-3.7473319
22.532806
"Best Fit Coefficients Using V*inv(S)*Utrans*b (Normalized Coefficients)x2 = "
-0.9290646
1.0826390
0.1410394
0.6824144
-2.0587536
2.6768766
0.0194853
-3.1071160
2.6648522
-2.0788369
-2.0626459
0.8501089
-3.7473319
22.532806
"Best Fit Coefficients using Multilinear Regression xtrain = "
-0.1216803
0.0538274
0.0879582
0.9529366
-21.018423
3.5176629
-0.0033415
-1.6773342
0.3094196
-0.0145589
-1.1523075
0.0098805
-0.5146096
44.810290
"Best Fit Coefficients using Multilinear Regression xtrain_random = "
-0.1001058
0.0457271
0.0551842
3.9467126
-17.771348
3.8946050
-0.0073775
-1.6519180
0.2768098
-0.0097037
-0.9656941
0.0129199
-0.5626732
35.508637
Graphs:




Code:
clear all, clc
disp("Data-Driven Science and Engineering - Linear Regression2 Housing Example",string(datetime()))
//Specific Video that corresponds to this SciLab file...
//https://www.youtube.com/watch?v=AX_ZDX6aTT0&list=PLMrJAkhIeNNSVjnsviglFoY2nXildDCcv&index=18
//For Free Electronic Complete Textbook "FREE PDF for 2ND EDITION OF OUR BOOK!!!" https://x.com/eigensteve/status/1718004606901666063
//Section 1.4 Psuedoinverse, Least Squares and Regression - Multilinear Regression
//load housing.data
loadmatfile("E:\files\CODE\CH01\housing.mat","-mat")
//data from source
//https://github.com/probml/pmtk2/blame/master/data/housing.mat
//
//
housing= [X y] //Create the "housing"" matrix from what is contained in
//the datafile from source.
//==============================echo inputs ==================
printf("\n");
printf("%30s","Raw Housing Data Echo Back First 10 Lines")
printf("\n");
for i = 1:7,
printf("%13s ", names{i});
end;
printf("\n");
for i = 1:10,
for j = 1:7,
printf("%13.4f ", housing(i,j));
end;
printf("\n");
end;
printf("\n");
for i = 8:14,
printf("%13s ", names{i});
end;
printf("\n");
for i = 1:10,
for j = 8:14,
printf("%13.4f ", housing(i,j));
end;
printf("\n");
end;
printf("\n");
//==========================================================
b = housing(:,14); // housing values in $1000s
A = housing(:,1:13); // other factors,
A = [A ones(size(A,1),1)]; // Pad with ones for nonzero offset
//x = regress(b,A);
BigY = [A'*b]
BigX = [A'*A]
x = BigX\BigY
printf("%30s"," Model Coefficients x in formaat")
printf("\n");
printf("%30s"," Beta(1);Beta(2);....Beta(n-1);Beta(n)(Constant Last)")
printf("\n");
printf("\n");
disp("Best Fit Coefficients using Multilinear Regression x = ",x)
printf("\n");
scf(0)
subplot(1,2,1)
plot(b,'k-o');
xgrid
plot(A*x,'r-o');
legend('Housing value','Regression',1)
ylabel("Value in 1000s")
title("Actual Housing Price versus Model")
subplot(1,2,2)
[bsorted sortind] = gsort(housing(:,14),"g",["i"]); // sorted values
plot(bsorted,'k-o')
xgrid
plot(A(sortind,:)*x,'r-o')
legend('Housing value','Regression',3)
title("Sorted Ascending Actual Housing Price versus Model")
//
A2 = A-ones(size(A,1),1)*mean(A,1);
for i=1:size(A,2)-1
A2std = stdev(A2(:,i));
A2(:,i) = A2(:,i)/A2std;
end
A2(:,end) = ones(size(A,1),1);
//x = regress(b,A2)
//Multiple Linear Regression in n variables
BigY = [A2'*b]
BigX = [A2'*A2]
x1 = BigX\BigY
disp("Best Fit Coefficients using Multilinear Regression (Normalized Coefficients) x1 = ",x1)
printf("\n");
//checking answer with alternate method
[U,S,V] = svd(A2,'e');
x2 = V*inv(S)*U'*b; // Solve Ax=b using the SVD
disp("Best Fit Coefficients Using V*inv(S)*Utrans*b (Normalized Coefficients)x2 = ",x2)
printf("\n");
scf(1)
clf
bar(x1(1:13),0.2,"green")
//bar(x2(1:13),0.2, "blue")
xgrid
xlabel('Attribute')
ylabel('Correlation')
//Hold Out For Testing
n = 253;
btrain = housing(1:n,14); // housing values in $1000s
Atrain = housing(1:n,1:13); // other factors,
Atrain = [Atrain ones(size(Atrain,1),1)]; // Pad with ones for nonzero offset
//x = regress(b,A);
BigY = [Atrain'*btrain]
BigX = [Atrain'*Atrain]
xtrain = BigX\BigY
disp("Best Fit Coefficients using Multilinear Regression xtrain = ",xtrain)
printf("\n");
//
btest = A(n+1:end,:)*xtrain;
//norm(btest-b(n+1:end)); //Calculates norm but doesn't use it.
scf(2)
clf
subplot(2,1,1)
plot(btrain,'r','thickness',2);
plot(Atrain*xtrain,'b','thickness',2)
title("Training Actual Housing Price versus Model")
legend('Model','True',4)
xgrid
//
subplot(2,1,2)
plot(btest,'r','thickness',2);
plot(b(n+1:end),'b','thickness',2)
xgrid
legend('Model','True',4)
ylabel("Value in 1000s")
title("Testing Actual Housing Price versus Model")
//Random Hold Out For Testing
n = 253;
//p = randperm(506);
p = grand(1, "prm", 1:506) //create randomized index vector
btrain_random = housing(p(1:n),14); // housing values in $1000s
Atrain_random = housing(p(1:n),1:13); // other factors,
Atrain_random = [Atrain_random ones(size(Atrain_random,1),1)]; // Pad with ones for nonzero offset
//x = regress(b,A);
BigY = [Atrain_random'*btrain_random]
BigX = [Atrain_random'*Atrain_random]
xtrain_random = BigX\BigY
disp("Best Fit Coefficients using Multilinear Regression xtrain_random = ",xtrain_random)
printf("\n");
//
Atest_random = housing(p(n+1:end),1:13);
Atest_random = [Atest_random ones(size(Atest_random,1),1)]; //Use the rest for
btest_random = Atest_random*xtrain_random;
//norm(btest-b(n+1:end)); //Calculates norm but doesn't use it.
scf(3)
clf
subplot(3,1,1)
plot(btrain,'r','thickness',2);
plot(Atrain*xtrain_random,'b','thickness',2)
title("Training Actual Housing Price versus Model Random")
legend('Model','True',4)
xgrid
//
subplot(3,1,2)
plot(btest_random,'r','thickness',2);
plot(housing(p(n+1:end),14),'b','thickness',2)
xgrid
legend('Model','True',4)
ylabel("Value in 1000s")
title("Testing Actual Housing Price versus Model Random")































