Uploaded image for project: 'OpenIDM'
  1. OpenIDM
  2. OPENIDM-10081

Lower throughput in LiveSync when external DJ is used as repo

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: OpenIDM 6.0.0
    • Fix Version/s: OpenIDM 6.0.0
    • Labels:
    • Environment:
      IDM 6.0.0 master runs on a CentOS7 using external DJ 6.0.0 M1 as repo

      Description

      When run livesync create with external DJ as repo, the throughput is pretty low, in one instance, LiveSync 10K users from DJ to managed(external DJ as repo)

      ...
      20180127 01:31:09.453 - INFO - current = 9959, prev = 9944, sample = 15 , tps= 2/s
      20180127 01:31:15.488 - INFO - current = 9974, prev = 9959, sample = 15 , tps= 2/s
      20180127 01:31:21.537 - INFO - current = 10000, prev = 9974, sample = 26 , tps= 4/s
      20180127 01:31:21.538 - INFO - The total time spent in livesync create in seconds is 2715.03377295
      20180127 01:31:21.538 - INFO - The throughput in livesync create is 3.6831954356
      

      With MySQL as repo, LiveSync 50K users from DJ to managed, we normally have

      ...
      17:18:21.609	INFO	current = 49243, prev = 48778, sample = 465 , tps= 77/s	
      17:18:27.763	INFO	current = 49708, prev = 49243, sample = 465 , tps= 77/s	
      17:18:33.913	INFO	current = 50000, prev = 49708, sample = 292 , tps= 48/s	
      17:18:33.913	INFO	The total time spent in livesync create in seconds is 728.032843113	
      17:18:33.913	INFO	The throughput in livesync create is 68.6782203207
      

      The throughput in the case of external DJ as repo is much lower then the JDBC case.
      Attached are the metrics, IDM log and the pyforge debug file(can see the livesync progress).

      To reproduce with Pyforge:
      with necessary setup(num_users, duration and IDM repo type), run command

      python -u run-pybot.py -c stress -i gatling -s *LiveSync -t Time_And_Throughput_For_Livesync_Create OpenIDM
      

      Note that In both external DJ as repo or MySQL as repo, see some connection failures with DJ where livesync from.
      also tested with uniqueConstrains on link removed and see the same result.

        Attachments

        1. debug.txt
          182 kB
        2. dj-logs-with-indexes.zip
          1.09 MB
        3. embedded-debug.txt
          48 kB
        4. embedded-metrics
          9 kB
        5. external-debug.txt
          73 kB
        6. external-metrics
          9 kB
        7. livesync.metrics5
          10 kB
        8. openidm0.log.0
          94 kB

          Activity

            People

            Assignee:
            jim.mitchener Jim Mitchener
            Reporter:
            Tinghua.Xu Tinghua Xu
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: