PeopleSoft Search Framework - Elastic Indexing rate is slowing down as it progresses
I am trying to index 114mil rows of data and 41.5 attachments with 150mil rows of psfile_attdet data from a PeopleSoft CRM application to ES 6.1 (due to compatibility with PeopleSoft, can't upgrade to higher versions).
Elastic Configuration:
A single node (Master/Coordinator, ingest and data node) cluster with 200 primary shards with no replica
OS: Oracle Linux 5.4.17-2136.336.5.1.el7uek.x86_64 x86_64
OCPU: 32
Memory: 128GB
Disk: 10TB
elasticsearch.yml options:
http.max_content_length: 300mb
indices.memory.index_buffer_size: 20%
bootstrap.memory_lock: true
jvm.options
-Xms31g
-Xmx31g
-XX:ParallelGCThreads=20
-XX:NewRatio=2
refresh_interval : -1 » changed to 120s
Number of primary shards: 200