SAML2 audit logging sometimes records the IP address of the client (from the http request) while other times it will log the server's own IP address.
A general observation seems to be that successful calls, where a session is created/exists, have the client IP address whereas failed calls where a session does not exist contains the server IP address.
In my examples 192.168.56.1 is my real 'client' making a request. 192.168.56.3 is the server.
Successful sign on to SP:
Failed sign on:
In regular authentication audit logging, things look to be more consistent - even failures are logged with the client IP address: