When trying to send an assertion using the urn:ietf:params:oauth:grant-type:saml2-bearer grant_type, it appears that the "assertion" parameter is expected in Base64 encoded format.
According to the RFC:
The SAML Assertion XML data MUST be encoded using base64url, where
the encoding adheres to the definition in Section 5 of RFC 4648
[RFC4648] and where the padding bits are set to zero. To avoid the
need for subsequent encoding steps (by "application/x-www-form-
urlencoded" [W3C.REC-html401-19991224], for example), the base64url-
encoded data MUST NOT be line wrapped and pad characters ("=") MUST
NOT be included.
Both Saml2GrantTypeHandler and OAuth2Saml2GrantSPAdapter should be updated to properly encode the assertion.