Uploaded image for project: 'OpenDJ'
  1. OpenDJ
  2. OPENDJ-7821

Stress test: Rest jobs could fail due to IOException: Premature close

    XMLWordPrintable

    Details

    • Bug
    • Status: Dev in Progress
    • Critical
    • Resolution: Unresolved
    • 7.1.0
    • None
    • regression, rest
    • None
    • OpenDJ 7.1.0-SNAPSHOT (196efed513a), Rest2LdapGateway 7.1.0-SNAPSHOT, Java OpenJDK 11.0.4, Gatling 3.0.2, Comte lab machine
    • 0.5

      Description

      The stress jobs `rest2ldap.rest` and `rest2ldap.rest_disable_cache` failed due to fail test `rest_delete_operations_embedded*`. Tests are removing 200000 users with 200000 delete requests, until 30th of January 2021 some requests failed (to ten) with `j.i.IOException: Premature close` or status 404.

      I am not able to reproduce on my machine, however on lab comte machine I was able to reproduce with automation tests and with manually.

      Tests for reproduce:

      • stress duration on jenkins is 1800
      • stress number users must be minimal set up to 200000
        run-pybot.py -v -c stress -s rest.rest OpenDJ

      Steps to reproduce:

      1. Download, unzip and setup OpenDJ latest snapshot with enabled http
      2. Import entries (e.g. created file from makeldif)
        makeldif -c numusers=200000 -o ~/tmp/stress_rest2ldap_startup.ldif ~PyBot/OpenDJ/testcases/data/stress_rest2ldap_startup.template
        
        import-ldif -h comte.internal.forgerock.com -p 4444 -D "uid=admin" -w password -X -n userData -O  --skipFile ~/tmp/skip-DJ1-20210209-145649.txt --rejectFile ~/tmp/reject-DJ1-20210209-145649.txt -l ~/tmp/stress_rest2ldap_startup.ldif
        
      3. Add rest2ldap user and set basic auth for rest2ldap
        ldapmodify -h comte.internal.forgerock.com -p 4444 -D "uid=admin" -w password -X -Z
        n: dc=com
        changetype: modify
        add: aci
        aci: (targetcontrol = "1.2.840.113556.1.4.805")(version 3.0;acl "Admin user can use the Proxy Authorization control";
         allow (read)(userdn = "ldap:///uid=rest2ldap,ou=admins,dc=com");)
        aci: (target = "ldap:///dc=com")(targetattr = "*")(version 3.0;acl "Allow admin user proxied auth";
         allow (all,proxy)(userdn = "ldap:///uid=rest2ldap,ou=admins,dc=com");)
        
        ldapmodify -h comte.internal.forgerock.com -p 4444 -D "uid=admin" -w password -X -Z
        dn: ou=admins,dc=com
        changetype: add
        objectclass: top
        objectclass: organizationalUnit
        ou: admins
        
        ldapmodify -h comte.internal.forgerock.com -p 4444 -D "uid=admin" -w password -X -Z
        dn: uid=rest2ldap,ou=admins,dc=com
        changetype: add
        objectclass: top
        objectclass: person
        objectclass: organizationalPerson
        objectclass: inetOrgPerson
        cn: openam
        sn: openam
        uid: openam
        userPassword: 5ecret12!
        ds-privilege-name: proxied-auth
        ds-privilege-name: password-reset
        
      4. Create mapping from example
        dsconfig -h comte.internal.forgerock.com -p 4444 -D "uid=admin" -w password -X create-identity-mapper --mapper-name id_mapper_cn --set enabled:true --type exact-match --set match-base-dn:ou=Auth,dc=com --set match-attribute:cn -n
        
        dsconfig -h comte.internal.forgerock.com -p 4444 -D "uid=admin" -w password -X create-identity-mapper --mapper-name id_mapper_uid --set enabled:true --type exact-match --set match-base-dn:ou=Auth,dc=com --set match-attribute:uid -n
        
        dsconfig -h comte.internal.forgerock.com -p 4444 -D "uid=admin" -w password -X set-http-authorization-mechanism-prop --mechanism-name "HTTP Basic" --set identity-mapper:id_mapper_cn -n
        
        dsconfig -h comte.internal.forgerock.com -p 4444 -D "uid=admin" -w password -X set-http-authorization-mechanism-prop --mechanism-name "HTTP Basic" --add identity-mapper:id_mapper_uid -n
        
        dsconfig -h comte.internal.forgerock.com -p 4444 -D "uid=admin" -w password -X set-http-endpoint-prop --endpoint-name "/api" --set enabled:false -n
        
        dsconfig -h comte.internal.forgerock.com -p 4444 -D "uid=admin" -w password -X set-http-endpoint-prop --endpoint-name "/api" --set enabled:true -n
        
      5. Run galing sh for delete (I used scala script from autoamtion tests)
        gatling.sh -rf ~tmp/graph -s opendj.rest2ldap.DeleteUserAsAdmin -sf~/gatling/user-files/simulations/forgerock/ -rd opendj.rest2ldap.DeleteUserAsAdmin

      Results:

      ---- Global Information --------------------------------------------------------
      > request count                                     200000 (OK=199995 KO=5     )
      > min response time                                      2 (OK=2      KO=33050 )
      > max response time                                  60889 (OK=29345  KO=60889 )
      > mean response time                                    10 (OK=8      KO=47287 )
      > std deviation                                        287 (OK=151    KO=11641 )
      > response time 50th percentile                          4 (OK=4      KO=44620 )
      > response time 95th percentile                         11 (OK=11     KO=60854 )
      > response time 99th percentile                         22 (OK=22     KO=60882 )
      > response time 99.9th percentile                      945 (OK=920    KO=60888 )
      > mean requests/sec                                1030.928 (OK=1030.902 KO=0.026 )
      ---- Response Time Distribution ------------------------------------------------
      > t < 800 ms                                        199781 (100%)
      > 800 ms < t < 1200 ms                                 125 (  0%)
      > t > 1200 ms                                           89 (  0%)
      > failed                                                 5 (  0%)
      ---- Errors --------------------------------------------------------------------
      > status.find.is(200), but actually found 404                         3 (60.00%)
      > j.i.IOException: Premature close                                    2 (40.00%)
      ================================================================================

      IOException Premature close:

      >>>>>>>>>>>>>>>>>>>>>>>>>>
      Request:
      DeleteUserAsAdmin: KO j.i.IOException: Premature close
      =========================
      Session:
      Session(DeleteUserAsAdmin,9,1612880495016,Map(gatling.http.cache.baseUrl -> http://comte.internal.forgerock.com:8080/api/example, admin_username -> admin_user, gatling.http.cache.dns -> io.gatling.http.cache.DnsCacheSupport$$anon$1@2bf4ddfe, gatling.http.cache.contentCache -> io.gatling.core.util.cache.Cache@5f5fff89, username -> user_20007, 6cfa0354-9970-4a85-af8d-506bde375bdc -> 2008, timestamp.6cfa0354-9970-4a85-af8d-506bde375bdc -> 1612880495018, admin_password -> admin_password, gatling.http.ssl.sslContexts -> SslContexts(io.netty.handler.ssl.OpenSslClientContext@1a1c4e16,None)),187,KO,List(ExitAsapLoopBlock(6cfa0354-9970-4a85-af8d-506bde375bdc,io.gatling.core.action.builder.LoopBuilder$$Lambda$406/1957078537@75193b64,io.gatling.core.action.Exit@788f1314)),io.gatling.core.protocol.ProtocolComponentsRegistry$$Lambda$478/1746257394@588c083b)
      =========================
      HTTP request:
      DELETE http://comte.internal.forgerock.com:8080/api/example/users/user_20007
      headers=
      X-OpenIDM-Username: admin_user
      X-OpenIDM-Password: admin_password
      accept: */*
      origin: http://comte.internal.forgerock.com:8080
      host: comte.internal.forgerock.com:8080
      =========================
      HTTP response:
      <<<<<<<<<<<<<<<<<<<<<<<<<

      404 failed:

      >>>>>>>>>>>>>>>>>>>>>>>>>>
      Request:
      DeleteUserAsAdmin: KO status.find.is(200), but actually found 404
      =========================
      Session:
      Session(DeleteUserAsAdmin,5,1612880495015,Map(gatling.http.cache.baseUrl -> http://comte.internal.forgerock.com:8080/api/example, admin_username -> admin_user, gatling.http.cache.dns -> io.gatling.http.cache.DnsCacheSupport$$anon$1@2bf4ddfe, gatling.http.cache.contentCache -> io.gatling.core.util.cache.Cache@54addf9, username -> user_48831, 6cfa0354-9970-4a85-af8d-506bde375bdc -> 4551, timestamp.6cfa0354-9970-4a85-af8d-506bde375bdc -> 1612880495018, admin_password -> admin_password, gatling.http.ssl.sslContexts -> SslContexts(io.netty.handler.ssl.OpenSslClientContext@c380222,None)),262,KO,List(ExitAsapLoopBlock(6cfa0354-9970-4a85-af8d-506bde375bdc,io.gatling.core.action.builder.LoopBuilder$$Lambda$406/1957078537@75193b64,io.gatling.core.action.Exit@788f1314)),io.gatling.core.protocol.ProtocolComponentsRegistry$$Lambda$478/1746257394@588c083b)
      =========================
      HTTP request:
      DELETE http://comte.internal.forgerock.com:8080/api/example/users/user_48831
      headers=
      X-OpenIDM-Username: admin_user
      X-OpenIDM-Password: admin_password
      accept: */*
      origin: http://comte.internal.forgerock.com:8080
      host: comte.internal.forgerock.com:8080
      =========================
      HTTP response:
      status=
      404 Not Found
      headers= 
      Cache-Control: no-store
      Content-API-Version: protocol=2.1,resource=1.0
      Expires: 0
      Pragma: no-cache
      X-Content-Type-Options: nosniff
      Date: Tue, 09 Feb 2021 14:22:57 GMT
      Server: Apache-HttpCore/5.0.3 (Java/11.0.4)
      Transfer-Encoding: chunked
      Content-Type: application/json; charset=UTF-8
      
      body=
      {"code":404,"reason":"Not Found","message":"No Such Entry: The search base entry 'uid=user_48831,ou=people,dc=example,dc=com' does not exist"}
      <<<<<<<<<<<<<<<<<<<<<<<<<

      Links on failed jobs:

        Attachments

          Issue Links

            Activity

              People

              nicolas.capponi@forgerock.com Nicolas Capponi
              miroslav.meca Miroslav Meca
              Nicolas Capponi Nicolas Capponi
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated: