por What is the method of least squares?
The method of Least Squares It is one of the most important applications in the approximation of functions. The idea is to find a curve such that, given a set of ordered pairs, that function best approximates the data. The function can be a line, a quadratic curve, a cubic, etc.
The idea of the method is to minimize the sum of squares of the differences in the ordinates (Y component), between the points generated by the chosen function and the points belonging to the data set.
method of least squares
Before giving the method, we must first be clear about what «best approximate» means. Suppose we are looking for a line y=b+mx that best represents a set of n points, namely {(x1,y1),(x2,y2)…,(xn,yn)}.
As shown in the previous figure, if the variables x and y were related by the line y=b+mx, then for x=x1 the corresponding value of y would be b+mx1. However, this value is different from the true value of y, which is y=y1.
Remember that in the plane, the distance between two points is given by the following formula:
With this in mind, to determine the way to choose the line y=b+mx that best approximates the given data, it sounds logical to use as a criterion the selection of the line that minimizes the sum of the squares of the distances between the points. and the straight
Since the distance between the points (x1,y1) and (x1,b+mx1) is y1-(b+mx1), our problem reduces to finding numbers m and b such that the following sum is minimal:
The line that meets this condition is known as the “approximation of the least-squares line to the points (x1,y1),(x2,y2),…,(xn,yn)”.
Once the problem is obtained, it only remains to choose a method to find the approximation by least squares. If the points (x1,y1),(x2,y2),…,(xn,yn) are all on the line y=mx+b, we would have that they are collinear and:
In this expression:
Finally, if the points are not collinear, then y-Au=0 and the problem can be translated into finding a vector u such that the Euclidean norm is minimal.
Finding the minimizing vector u is not as difficult as one might think. Since A is an nx2 matrix and u is a 2×1 matrix, we have that the vector Au is a vector in Rn and belongs to the image of A, which is a subspace of Rn with dimension no greater than two.
We will assume that n=3 to show what is the procedure that must be followed. If n=3, the image of A will be a plane or a line through the origin.
Let v be the minimizing vector. In the figure we observe that y-Au is minimized when it is orthogonal to the image of A. That is, if v is the minimizing vector, then it happens that:
Then, we can express the above in this way:
This can only happen if:
Finally, isolating v, we have that:
It is possible to do this since AtA is invertible as long as the n data points are not collinear.
Now, if instead of looking for a straight line we wanted to find a parabola (whose expression would be of the form y=a+bx+cx2) that was a better approximation to the n data points, the procedure would be as described below.
If the n data points were in said parabola, we would have to:
Then:
Similarly we can write y=Au. If all the points are not on the parabola, we have that y-Au is non-zero for any vector u and our problem becomes again: find a vector u in R3 such that its norm ||y-Au|| be as small as possible.
By repeating the above procedure, we can arrive at the searched vector as:
solved exercises
Exercise 1
Find the line that best fits the points (1,4), (-2,5), (3,-1) and (4,1).
Solution
We have to:
Then:
Therefore, we conclude that the line that best fits the points is given by:
Exercise 2
Suppose an object is dropped from a height of 200 m. While falling, the following measures are taken:
We know that the height of said object, after a time t has elapsed, is given by:
If we wish to obtain the value of g, we can look for a parabola that is a better approximation to the five points given in the table, and thus we would have that the coefficient that accompanies t2 will be a reasonable approximation to (-1/2)g if the measurements are exact.
We have to:
And then:
So the data points are fitted by the following quadratic expression:
So, you have to:
This is a value that is reasonably close to the correct one, which is g=9.81 m/s2. In order to obtain a more exact approximation of g, it would be necessary to start from more precise observations.
What is the method of least squares used for?
In problems that occur in the natural or social sciences, it is convenient to write the relationships that occur between different variables by means of some mathematical expression.
For example, in economics we can relate cost (C), income (I) and profit (U) by means of a simple formula:
In physics, we can relate the acceleration caused by gravity, the time an object has been falling, and the height of the object by law:
In the above expression so is the initial height of said object and vo is its initial velocity.
However, finding formulas like these is not an easy task; Usually it is up to the professional on duty to work with a lot of data and repeatedly carry out various experiments (in order to verify that the results obtained are constant) to find relationships between the different data.
A common way to accomplish this is to represent the data obtained on a plane as points and find a continuous function that best approximates those points.
One of the ways to find the function that «best approximates» the given data is by the method of least squares.
Furthermore, as we also saw in the exercise, thanks to this method we can get quite close approximations to physical constants.