[OPENAM-3750] REST authentication failed if unicode/utf8 login/password Created: 17/Mar/14  Updated: 28/Jul/17  Resolved: 28/Apr/16

Status: Resolved
Project: OpenAM
Component/s: rest
Affects Version/s: 12.0.0, 13.0.0
Fix Version/s: 12.0.4, 13.5.0

Type: Bug Priority: Major
Reporter: gamaliel.sick@yahoo.fr [X] (Inactive) Assignee: Kamal Sivanandam
Resolution: Fixed Votes: 1
Labels: 13.0.0-Backlog, EDISON
Remaining Estimate: 0h
Time Spent: 8h
Original Estimate: 8h

Issue Links:
relates to OPENAM-4818 Cannot update Protected Attributes in... Resolved
Target Version/s:
Sprint: Sprint 67 - Team Newton, Sprint 69 - Team Newton, AM Sustaining Sprint 21
Support Ticket IDs:
QA Assignee: Filip Kubáň [X] (Inactive)
Verified Version/s:



REST authentication api with "X-OpenAM-Username" and "X-OpenAM-Password" headers doesn't work with login and/or password encoded in utf8.

From RFC, headers values are encoded in ISO-8859-1 but it just bytes...

curl cmd:

curl --request POST --header "Content-Type: application/json" --header "X-OpenAM-Username: 測試測試測試測試" --header "X-OpenAM-Password: 測試測試測試測試" --data "{}" http://...../openam/json/authenticate


{"code":401,"reason":"Unauthorized","message":"Authentication Failed!!"}



In "RestAuthNameCallbackHandler" and "RestAuthPasswordCallbackHandler", header value should be convert in unicode.

Roughly example:

String username = new String(((String)request.getHeader(USERNAME_HEADER_KEY)).getBytes("ISO-8859-1"), "UTF-8");

String password = new String(((String)request.getHeader(PASSWORD_HEADER_KEY)).getBytes("ISO-8859-1"), "UTF-8");

Comment by gamaliel.sick@yahoo.fr [X] (Inactive) [ 25/Mar/14 ]


Any news?

It should be 12.0.0-SNAPSHOT but I cannot edit the ticket.
The issue should exist on 11

Comment by Dirk Hogan [ 20/Oct/14 ]

I am struggling to understand why this is a defect. I agree that the http spec mandates that non-mime-encoded header values should comply with the ISO-8859-1 encoding. However, this encoding is a single-byte encoding. UTF-8, the default encoding, is a multi-byte encoding that can encode a massive super-set of the ISO-8859-1 encoding. Why would we want to constrain the Rest name and password callback handlers in that fashion, when UTF-8 encodes all of the characters which ISO-8859-1 encodes? The particular bit-sequence which encodes a particular character in UTF-8 and ISO-8859-1 is irrelevant, as long as a particular encoding is used consistently. Furthermore, the username and password values you use in your examples certainly don't have an encoding in ISO-8859-1, and thus if the code were written as you suggest, both the username and password would be represented as "????????", as there are no ISO-8859-1 code points for the characters in your example.

I will close the defect as won't-fix/not-a-defect. Please point out fallacies in my thinking if you think I am missing something.

Comment by Dirk Hogan [ 20/Oct/14 ]

See previous comment for explanation.

Comment by Sam Drew [ 23/Oct/14 ]

Dirk, could you clarify this slightly more. At the moment we use HTTP Headers to provide the password on some occasions, including for changing password. The bug here is that if a password uses characters outside of ISO-8859-1 it will not be possible to use to send in the HTTP header and validate.

My understanding is that you are resolving this because according to rfc2616 it is only possible to send ISO-8859-1 characters in the header. While this limitation exists within the spec, we should be supporting UTF-8 passwords, and so probably need to find a workaround for this, even if not as shown in the example above.

Comment by Dirk Hogan [ 23/Oct/14 ]

The RestAuthNameCallbackHandler and the RestAuthPasswordCallbackHandler both pull the header values without considering any string encoding, which means that they default to the platform encoding, which is UTF-8. Which means that if user-agents actually support passing header values not encoded by the ISO-8859-1 character set, we will be able to handle it, because UTF-8 can accommodate these characters. The changes described in this defect submission would actually constrain our existing ability to handle characters of the sort described in the defect filing. I bet that the inability to log in with 測試測試測試測試 is most likely a function of curl's sending this value as '????????', because none of the characters have an encoding in the ISO-8859-1 encoding. To repeat: as far as I can tell, there is nothing in the RestAuthPasswordCallbackHandler and the RestAuthNameCallbackHandler which would prevent the correct representation of these characters.

Comment by Sam Drew [ 23/Oct/14 ]

Assuming the RestAuthName/PasswordCallbackHandlers are pulling the header values, I strongly suspect that they will default to the HTTP specification, rather than the platform so ISO-8859-1 rather than UTF-8. The issue isn't with cURL, or any other client, so much as it is with the HTTP specification.

Effectively HTTP headers are unsuitable for use where we want to support extended character sets. I'm going to re-open the issue, unassign it, and remove the MUST-FIX label. Fortunately this is a less serious issue that originally believed as I can only think of one situation where we are passing a parameter that should be extended character encoding in the HTTP header rather than as a POST or GET parameter.

Comment by gamaliel.sick@yahoo.fr [X] (Inactive) [ 24/Oct/14 ]

More clarification. The example presented above is actually how this issue has been worked around in a SaaS application, in production (multi-data center, etc), and international. Without this workaround it was impossible for OpenAM to be usable internationally.

In reply to @samandrew. This is really not an edge case but a genuine drawback or a flaw in the product. Please fix this or provide out-of-box (supported) plugin as pointed in the example.

Comment by Dirk Hogan [ 25/Oct/14 ]

Thinking about it a bit more:
it is possible that this is a defect iff there is a divergence between the UTF-8 and ISO-8859-1 encodings. If memory serves, there was substantial overlap, but also a divergence. If that is true, getting a header with no specified encoding (will default to UTF-8), and with the ISO-8859-1 encoding, might result in a different characters. I will investigate this a bit more on Monday and report.

Regardless, pulling the headers with the ISO-8859-1 encoding will not resolve the characters in the defect filing, as there is no mapping for these characters in ISO-8859-1.

Comment by Dirk Hogan [ 27/Oct/14 ]

An update: according to the wikipedia article http://en.wikipedia.org/wiki/ISO/IEC_8859-1:
"ISO-8859-1 was incorporated as the first 256 code points of ISO/IEC 10646 and Unicode."
Which makes sense - it would be a migration nightmare if these code points diverged.

So I am back to my original position that this is not a defect. If the defect filer could provide an example of how a ISO-8859-1 encoded string could result in a character different from the character resolved via UTF-8 (for the same byte[] obviously), that would be very helpful.

Comment by Sam Drew [ 27/Oct/14 ]

Because of the deficiencies in HTTP headers, my recommendation for this issue would be to avoid relying upon them for cases where user defined variables and as such UTF-8 content might be required, as the workaround as suggested above will cause compatibility issues where clients have implemented them to specification.

At the moment the only place I know of where XUI fails to handle extended character sets as a result of relying on HTTP-Headers is in updating protected attributes.

Comment by Sam Drew [ 29/Oct/14 ]

For zero-callback authentication that supports UTF-8 use of headers is unsuitable. Instead use POST body, e.g.

curl \
--request POST \
--header "Content-Type: application/json" \
--data '{"callbacks":[{"type":"NameCallback","input":[{"name":"IDToken1","value”:"USER_NAMEユーザ名"}]},{"type":"PasswordCallback","input":[{"name":"IDToken2","value”:"PASSWORDパスワード"}]}]}' \
Comment by David Luna [ 29/Oct/14 ]

We could use the HTTP authorization-header approach of simply base64 encoding passwords before transmission and decoding up on the other end?

Comment by Peter Major [X] (Inactive) [ 18/Nov/14 ]

Reopening this, as we have resolved the very same problem in OPENAM-4818 using MIME encoding. For the sake of consistency using MIME encoded header values for username/password should be possible with the authenticate endpoint just as well.

Comment by Filip Kubáň [X] (Inactive) [ 23/Sep/16 ]

Verified on: OpenAM 12.0.4-RC5 Build 8f3551671e (2016-September-19 17:53)

Comment by Peter Trischberger [X] (Inactive) [ 29/Sep/16 ]

openam now allows for base-64 encoded characters (as per RFC 2047) to be sent in headers to /json/authenticate

Password = changeitÖ
Base64 encoded password = Y2hhbmdlaXTDlg==
Header = X-OpenAM-Password: =?UTF-8?B?Y2hhbmdlaXTDlg==?=

(also possible with X-OpenAM-Username)

Generated at Sat Sep 26 00:08:06 UTC 2020 using Jira 7.13.12#713012-sha1:6e07c38070d5191bbf7353952ed38f111754533a.