Refresh Token flow not enabled on OAuth2 Client can still use Refresh Token flow

Bug description

OAuth2 Clients are able to use a Refresh Token to get a new Access Token despite the Refresh Token flow not being enabled on the Client

How to reproduce the issue

1. Enable OAuth2 Provider (enable Refresh Token)
2. Add OAuth2 client (default is fine, don't add Refresh Token flow)
3. Get Access Token, Refresh Token etc.
4. Use Refresh Token flow to get a new Access Token

Expected behaviour

Error = The authenticated client is not authorized to use this authorization grant type.

Current behaviour

New Access Token issued