In a distributed system, if two events came back to back, and we do not take any locks in case of update. Then the update due to second update may potentially override the first one.
The API:

public <T> T find(Class<T> entityClass, Object primaryKey)
doesn't provide any kind of locks.

Fix:

Approach 1: Using the overloaded version of the same,
public <T> T find(Class<T> entityClass, Object primaryKey,LockModeType lockMode)
With LockModeType PessimisticLock, we can force them to be serial operations at MySQL level.

Problem: This is an overkill for solving a condition which happens once in the lifecycle.

Approach 2: Using Optimistic locks, in JPA the annotation @Version makes sure that we are locking the table optimistically.

Some problems, challenges and steps taken to overcome:
Problem: Since the data is mocked, so with test cases testing is not possible.
Tentative solutions: In case of multi-threaded test cases, there are some tools like CyclicBarrier, CountDownLatch is a good read.

Solution: Now the question is, how did I test my code if its working properly or not. All I needed was to hit my app on local with 10 threads with update request.

Problem: All MySql commits happens in detached process state. So Exceptions are not coming in program context. [Blocker for Retry logic]

Solution: Commit the changes to underlying database layer, this was achieved by using JPA API's like flush and clear. Now the problem is, the default flush type doesn't commit to the database, so FlushModeType.Commit is needed.

Problem: Even in case of retries, JPA was reading from its cache layer, not the underlying database

Solution: Needed to add a few more JPA API's to the app for refreshing the data from database.
And adding a retry logic programmatically in a for loop doesn't seem too good. So here function interceptor comes to rescue, just create an annotation which intercepts a function and all is set!

Class signature for retry:
RetryInterceptor implements MethodInterceptor { @Override public Object invoke(MethodInvocation invocation) Throwable { //Retry logic goes here } //Define Annotation according to your use case!

And in the method we want to retry, we need to add an annotation:
@Retry(retryCount = 3, exception = OptimisticLockException.class)

And it will take care of all the rest.