PeopleTools and Lifecycle Management - PSFT (MOSC)

MOSC Banner

PeopleSoft Search Framework - Elastic Indexing rate is slowing down as it progresses

in PeopleTools and Lifecycle Management - PSFT (MOSC) 10 commentsAnswered

I am trying to index 114mil rows of data and 41.5 attachments with 150mil rows of psfile_attdet data from a PeopleSoft CRM application to ES 6.1 (due to compatibility with PeopleSoft, can't upgrade to higher versions).

Elastic Configuration:

A single node (Master/Coordinator, ingest and data node) cluster with 200 primary shards with no replica
OS: Oracle Linux 5.4.17-2136.336.5.1.el7uek.x86_64 x86_64
OCPU: 32
Memory: 128GB
Disk: 10TB

elasticsearch.yml options:
http.max_content_length: 300mb
indices.memory.index_buffer_size: 20%
bootstrap.memory_lock: true

jvm.options

-Xms31g

-Xmx31g

-XX:ParallelGCThreads=20

-XX:NewRatio=2

refresh_interval : -1 » changed to 120s

Number of primary shards: 200

Howdy, Stranger!

Log In

To view full details, sign in to My Oracle Support Community.

Register

Don't have a My Oracle Support Community account? Click here to get started.

Category Leaderboard

Top contributors this month

PSFT PeopleTools Tags

Instructions for posting

×

1. Select a discussion category from the picklist.


2. Enter a title that clearly identifies the subject of your question.


3. In the body, insert detailed information, including Oracle product and version.


Please abide by the Oracle Community guidelines and refrain from posting any customer or personally identifiable information (PI/CI).


Cloud / Fusion customers - Our Cloud community has moved! Please go to Cloud Customer Connect .


New to My Oracle Support Community? Visit our Welcome Center

MOSC Help Center