Ok, I found the problem.
I was sending an Authorization header, because that was the default option in Postman. Now I tried it with the other option which is "Send client credentials in body" and it works.
The documentation about the token endpoint had me a bit confused, that's why I kept sending an (invalid) Authorization header. Now that I post an empty client_secret parameter in the body and NO Authorization header to the token endpoint, things are working fine.
Thanks for pointing me in the right direction, @robotdan and @dan.