If GPU works with vectors why we(humans) need to work with matrices. It is a lot easier to work with vectors than matrices...

What?!

Matrices are a way to _encode_ transformations (translation, scaling, rotation) performed on vectors. Nothing more, nothing less.

If you want your models in the scene to show up somewhere else, be rotated a bit and scaled down, you _encode_ this information into a matrix, as simple as that.