Matrices and systems of equations

2.1 – Matrices as Transformations

If vectors allowed us to imagine abstract concepts as points within an abstract space, matrices invite us to take one step further: it is no longer just a matter of representing where something is, but of understanding how it changes or, more precisely, how we get from one point to another. A matrix can be viewed as a transformation rule, a silent machine that takes one representation and turns it into another. Where a vector describes a position, a matrix describes a possible movement.

This idea is fundamental to understanding much of modern Artificial Intelligence. When a neural network processes an image, a sentence, or a signal, it does not work with those objects directly, but with vector representations that pass through a succession of transformations, each modeled by a matrix. Each layer modifies how the information is organized: it mixes some dimensions with others, amplifies certain features, attenuates others, and gradually builds a new way of looking at the data. In this sense, a matrix is not simply a table of numbers. It is a compact way of describing how information flows between components. Each entry in the matrix indicates how much one dimension influences another, or how an initial structure can be deformed until patterns that were previously hidden are revealed.

In this chapter we will focus on the idea of a matrix as a transformation filter. But a matrix is also a natural way of organizing information in many contexts. A digital image can be represented as a matrix in which each number indicates the light intensity of a pixel. A data table can be viewed as a matrix in which each row describes an example and each column a feature. The mathematical concept is the same in both cases; only the perspective and the way we use it change.

2.2 – Multiplying a Matrix by a Vector

Although somewhat implicitly, we already introduced matrices and transformations in the previous chapter when discussing the concept of a basis. Recall that every vector can be expressed as a linear combination of the vectors in a basis. These basis vectors are independent of one another and therefore generate different directions. For two-dimensional space, we can represent a basis as a \( 2 \times 2 \) matrix, with each direction represented by a different column. The standard basis would be:

From these basis vectors, we can understand any linear combination as a set of instructions—for example, scale the horizontal direction by two and the vertical direction by three. We encode these instructions as a particular vector, representing the linear combination as the multiplication of a matrix by a vector:

This example is simple, but… what happens if we transform the basis vectors? The previous instructions encoded in the vector (two times to the right, three times upward) still apply, but the result is different when applied to a different matrix.

With this different basis, every vector will now point to a different location, so we can generalize this idea to the entire space being deformed. We can understand matrices as transformations insofar as they describe how the vectors that define a basis are moved. For spaces with more than two dimensions, the matrices will no longer be \( 2 \times 2 \), but \( 3 \times 3 \), \( 4 \times 4 \), and so on.

The animation begins with standard basis vectors. The matrix in the upper-left corner allows you to transform that basis and, with it, the entire plane. The formula at the bottom shows the multiplication of the basis transformation matrix by the vector that you can freely position in the plane. The background grid changes because the transformation affects not only the turquoise vector, but every other vector in the plane or, in other words, the space itself.

2.3 – Non-Square Matrices

A matrix, in principle, is nothing more than a table of numbers arranged in an orderly way into columns and rows. However, this simple appearance conceals an extremely useful mathematical tool that we will explore throughout this chapter.

Matrices help us represent bases, but they go much further. We are not restricted to having the same number of rows and columns—that is, to the matrix being square. Here is an example of a matrix of dimension \( 3 \times 2 \), that is, two rows and three columns.

As we anticipated in the previous section, multiplying a matrix by a vector is simply a linear combination of the column vectors.

Clearly, for this operation to be possible, the number of columns in the matrix must match the number of components in the vector. This is why not every matrix can be multiplied by every vector. The columns of a matrix indicate the number of components of the vectors it can transform. The number of rows indicates the number of components of the output vector.

Ultimately, this means that we can generalize the conclusions of the previous section: a matrix can transform one space into another, even one of a different dimension. Following the previous example with a \( 3 \times 2 \) matrix, we can say that:

$$A: \mathbb{R}^2 \to \mathbb{R}^3$$

or, equivalently, vectors in the plane are mapped into 3D space.

2.4 – Linear Transformations

Once we understand a matrix as a way of transforming space, it is worth being precise about the type of transformations we are describing. Matrices represent a very special kind of transformation: linear transformations. They can deform the plane in many different ways: stretching it, compressing it, rotating it, shearing it, or even flattening it onto a line. However, they cannot do so arbitrarily.

A linear transformation preserves the basic structure of space. This means that if a vector is constructed as a combination of other vectors, its transformation will be constructed by combining the transformed vectors in the same way. If a vector was originally the result of moving twice in one direction and three times in another, then after applying the transformation it will still be the result of taking the new first direction twice and the new second direction three times. This is why matrices are so useful: they do not need to specify what happens to every point in the plane one by one. It is enough to know what happens to the basis vectors. From there, the destination of any other vector is determined automatically.

Geometrically, a linear transformation keeps the origin fixed and turns the grid of the plane into another grid that is deformed but still orderly. Straight lines remain straight, and the combination relationships between vectors are preserved. The matrix can change the shape of the space, but it does not break its internal logic.

In this animation you can try applying the basic linear transformations: the identity matrix (which has no transformative effect), rotations in both directions, reflections, and shears (shear).

2.5 – Matrix Multiplication

Multiplying matrices means chaining transformations. If a matrix \( A\) transforms the space in a certain way and another matrix \( B\) then applies a second transformation, their combined effect can be described by a single matrix. The full composition is condensed into the matrix product \( BA\) (the subsequent transformation appears on the left).

$$ B(A\mathbf{v}) = BA\mathbf{v}$$

Rotating a figure and then reflecting it does not produce the same result as reflecting it first and then rotating it. Matrices do not behave like ordinary numbers. In general

$$ BA \neq AB $$

and this difference contains enormous geometric richness. Each multiplication represents a specific sequence of actions on the space.

As we have already seen when multiplying a matrix by a vector, not all matrices can be multiplied by one another. Composing matrices is like connecting pipes: the output of the first transformation must have exactly the form expected by the input of the second. For a product \( AB\), matrix \( B\) must have the same number of rows as \( A\) has columns. The result is a matrix with the same number of rows as the first matrix and the same number of columns as the second.

2.6 – Determinant of a Matrix

As we have already noted, matrix transformations do not merely change the position of a vector; they modify the space itself. The determinant of a matrix is a value that measures how much space expands or contracts under the transformation. Computing the determinant is straightforward in the case of the plane, but becomes more complex in higher dimensions. We will not dwell on the mechanics of calculating the determinant, only on the concept.

A matrix whose determinant is \( 1\) means that the transformation has neither stretched nor compressed the space. This would be the case, for example, with a rotation matrix. Determinants greater than \( 1\) would imply that the matrix has expanded the original area. Negative values would indicate a reflection of the space, something like its mirror image. This means that the determinant is related not only to volume, but also to orientation.

There is one case that deserves special attention: when the determinant is \( 0\). This means that the area has disappeared completely and the matrix has collapsed one of the directions: the transformation has turned one of the vectors into a linear combination of the others. This occurs when there is linear dependence among the columns/rows of the matrix.

It is important to note that non-square matrices do not have a determinant. This is because there is no way to compare volumes when they do not have the same dimensionality.

Here we visualize the parallelogram formed by the basis vectors as the transformed space, although, as we have already discussed in relation to matrices, the transformation would actually apply to any region of the plane (or to a volume in higher-dimensional spaces).

You can verify that when applying a rotation matrix such as \begin{bmatrix} 0 & -1 \\ 1 & 0\end{bmatrix}, the area moves while the determinant remains the same. If, instead of a rotation, we apply a reflection across the \( x \)-axis, \begin{bmatrix} 1 & 0 \\ 0 & -1\end{bmatrix}, we will see that the sign of the determinant changes. If we try a matrix whose columns are not linearly independent, such as \begin{bmatrix} 2 & 4 \\ 3 & 6\end{bmatrix}, we will see that one direction overlaps the other, collapsing one of the dimensions and resulting in a zero determinant.

2.7 – Image Space and Rank

We said a few sections ago that the number of rows in a matrix determines the number of components in the output vector. But this does not mean that the output space has the same dimension as the number of components in the vectors. For example, these three vectors have three components:

$$ \begin{bmatrix} -2 \\ 1 \\ 3 \end{bmatrix} \; \begin{bmatrix} 4 \\ 1 \\ 3 \end{bmatrix} \; \begin{bmatrix} 2 \\ 2 \\ 6 \end{bmatrix}$$

However, all three “live” in the same two-dimensional plane. We know this because the vector on the right is the sum of the two vectors to its left. In other words, they are not linearly independent. If we put them together in a matrix, its determinant would be \( 0\).

The space of possible vectors produced by the matrix is called the image space, and when the determinant is zero, this image space will have fewer dimensions than the space before the transformation.

The rank of a matrix is the number of dimensions of this image space. It is useful to distinguish it from the determinant: when the latter is different from \( 0\), the matrix has full rank, meaning that no dimension has been lost. The dimensions of the image space are the same as those of the original space. By contrast, if the determinant is \( 0\), the transformation flattens some direction. Notice that the determinant does not tell us how many dimensions are lost, only whether any have been lost.

From this perspective, rank measures the geometric richness of a transformation. The greater the rank, the more dimensions it preserves, although its value cannot exceed the number of dimensions of the original space. For any square matrix \( n \times n\), the maximum rank is clearly \( n\), corresponding to the case in which all column vectors are independent. For non-square matrices \( m \times n\), the maximum rank is always the smaller of the two values. Conversely, the lower the rank, the more severe the loss of information.

Even if the matrix has three column vectors, the dimensions of the image space will depend on how many of those vectors are linearly independent. If any of them is not, the matrix will no longer have full rank: its rank will be less than \( 3\), or even \( 0\) if all directions collapse to the zero vector.

2.8 – Null Space

We will now focus on matrices that do not have full rank, that is, matrices in which some directions have collapsed. Once this idea is firmly established, any collapse of this kind means that infinitely many vectors in the original space are transformed into a single point in the image space. And when that point is the zero vector, we can introduce a new concept. The null space (also called the kernel) of a matrix consists of all vectors that, after the transformation is applied, end up exactly at the zero vector, effectively disappearing.

We already said that the determinant does not indicate the number of dimensions lost. For that, we need to look at the dimension of the null space, which can be thought of as a kind of counterpart to the image space. The dimension lost when the matrix does not have full rank is gained by the kernel, so the dimension of the kernel plus the dimension of the image space equals the dimension of the original space before the transformation. This is known as the Rank–Nullity Theorem:

$$ dim Img (A) + dim Kern (A) = n$$

for a matrix \( A\) with \( n\) columns.

As in the previous section, the animation shows a three-dimensional space created by three independent vectors.

2.9 – Inverse Matrix

When a linear transformation modifies space, an immediate question arises: is it possible to go back? In many cases, the answer is yes. If we rotate a figure by \( 90º \), we can rotate it another \( 90º \) in the opposite direction and recover exactly its original position. If we enlarge an area to twice its size, we can shrink it by half to reconstruct it. Some transformations seem to have a kind of “reverse gear”.

The inverse matrix formalizes precisely this idea. An inverse matrix is a second transformation capable of completely undoing the effect of the first. Given a matrix $A$, we denote its inverse by $A^{-1}$, such that

$$A^{-1}A = I$$

or, equivalently, applying the transformation $A$ and then its inverse $A^{-1}$ is equivalent to performing no transformation at all, represented by the identity matrix $I$.

Computing the inverse matrix requires mechanical procedures that we will not detail here. What matters to us is understanding when this matrix can be computed and when it cannot. Not every transformation can be undone. Imagine a matrix that flattens the entire plane onto a single line. As we anticipated in the previous section, after that transformation, points that were originally different have ended up occupying the same position. Looking only at the result, there is no longer any way to reconstruct that space. If an inverse matrix existed, we would be forcing it to return an entire line of vectors for every point in the collapsed space, which is not how matrices behave. The information has been lost. This is why a transformation can have an inverse only when it preserves all the information in the original space. Geometrically, it must avoid collapsing dimensions. And this connects with what we saw earlier: if the determinant of a matrix is \( 0\), then one of the directions has collapsed and the matrix has no inverse. Any other determinant value means that the transformation can be reversed.

In three-dimensional space, the determinant measures volume and how much that volume has expanded or contracted. The animation shows the transformation matrix and its inverse. Computing the inverse involves a procedure that we will not explain here, since our goal is to explore the concept rather than the calculation itself.

You can try entering a transformation matrix in which one of the column vectors is simply the sum of the other two. You will see the volume collapse into a plane. One of the directions will have collapsed and the resulting determinant will be \( 0\)

2.10 – Systems of Linear Equations I

Let us look at how matrices can be used to solve systems of linear equations: simple equations without powers that share one or more unknowns. The goal is to solve for those unknowns so that all the equations are satisfied at the same time. Here is an example of a system of three equations with two unknowns:

\(2x \; – \; y = -7\)

\(-3x + 2y = 12\)

\(-x \; – \; 2y = -4\)

Translating this into matrix notation, we can see that the same system of three linear equations can be represented as the multiplication of a coefficient matrix by a vector of unknowns, in the form \(A\mathbf{v} = \mathbf{u}\):

Suppose we considered only the first equation, \(2x \; – \; y = -7\). We would not have enough information to determine the values of \( x\) and \( y\). There would be infinitely many valid possibilities, including \( \left\{ x = -3, y = 1 \right\}\), \( \left\{ x = 0, y = 7 \right\}\), or \( \left\{ x = 1, y = 9 \right\}\). Even so, an equation still describes a constraint between two unknown values: \(x\) and \(y\) cannot take just any values. The value of one determines the value of the other. This is why all possible solutions live in the same one-dimensional space, a line in the plane. This is called the solution space, and for a single equation its dimension will always be the number of unknowns minus one.

Ideally, each new equation added to the system further restricts the solution space, reducing its dimension by one. Thus, if we add the second equation, that space would go from being one-dimensional to a single point in the plane, and we would consider the system solved. If instead of two unknowns, as in this example, we had three, we would need at least three equations to determine the exact point in three-dimensional space that solves the system.

We said ideally because this reduction does not always occur: for the dimension of the solution space to decrease by one, the equations must be linearly independent. Otherwise, they impose the same directional constraint on the solution space and add no new restriction. Thus, a system could contain a large number of equations and still have infinitely many solutions. We can say that the dimension of the solution space is equal to the number of unknowns minus the rank of the coefficient matrix.

There is also another circumstance in which the previous relationship does not apply: one of the equations may be completely incompatible with the others, meaning that there are no values of the unknowns that satisfy all of them at the same time. In that case, the system has no solution.

Each equation establishes a fixed relationship between the two dimensions \(x\) and \(y\). The equation \(2x \; – \; y = -7\) tells us that the values of \( y\) are restricted to those satisfying the algebraic relationship \( y = 2x +7\). This is why the solution space is restricted to a line in the plane. If we solve the remaining equations for \(y\), each one defines its own line.

Each plane in this space represents the solution space of one equation, the set of three values that could satisfy it. Since our goal is to solve the entire system, we must locate the point that satisfies all three simultaneously. If the three equations are linearly independent, that point will give a single exact solution.

The same does not happen if they are linearly dependent, which can be modeled in the animation by creating columns that are linear combinations of others. A single dependent column would produce a plane intersecting two parallel solution planes: the system would have infinitely many solutions. If all of them are linear combinations, then the planes are all parallel and distinct: the system has no solution.

2.11 – Systems of Linear Equations II

In the previous section, we viewed systems of linear equations as constraints on a solution space. But we can also understand the three possibilities we have seen—a unique solution, infinitely many solutions, or no solution—by returning to the interpretation of matrices as transformations.

Let us return to the coefficient matrix and the form \(A\mathbf{v} = \mathbf{u}\). Instead of looking at the rows of \(A\) as equations that constrain solutions, we now look at the columns of \(A\) as vectors that must be combined to reach \(\mathbf{u}\). Then

\(2x – y = -7\)

\(-3x + 2y = 12\)

\(-x – 2y = -4\)

now becomes:

Solving the equations now means finding the scalars \(x\) and \(y\) that allow those columns to reach the target vector \(\mathbf{u}\). This is simply an alternative perspective to the one presented in the previous section, intended to capture the geometric intuition behind solving linear equations. The result is exactly the same.

Can we construct the target vector as a linear combination of the columns of the matrix? The answer depends on where the target vector lies relative to the space spanned by the columns of the matrix. If \(\mathbf{u}\) does not belong to the image space of the matrix, no combination of its columns can reach it, and the system has no solution. If it does belong to that space and the columns are linearly independent, there will be a unique combination that produces it. By contrast, if the columns are linearly dependent, different combinations can lead to the same target vector, resulting in infinitely many solutions.

In this animation, the system of equations is solved automatically by calculating the scalar (and therefore the extent) of each column vector of the matrix. When the system is solvable, the sum of the vectors points directly to the target vector.