Spring Boot in Production: Race Conditions, Outbox Pattern & Connection Pooling in a High-Load Payment System

Code blocks are styled like a dark IDE — select and copy/paste; no code was written into the diagram images.

Everything works flawlessly in the local environment (localhost): unit and integration tests pass green, and testing with a handful of concurrent users shows no problems at all. But once the system goes live in production and starts receiving thousands of concurrent real payment requests, unexpected cascading failures start showing up on the backend.

As you move from monolithic architectures to distributed microservice systems, the hardest part usually isn't the business logic itself — it's network latency, data consistency, and resource management. In this article, I'll walk through three major problems I ran into in a real production environment and how we solved them in a Spring Boot setup.

1. Race Conditions and Locking Strategies: The Double-Spending Problem

What happens when the same user hits the "Pay" button twice in a row because of a spotty internet connection, or when two different microservices touch the same user's balance at the exact same millisecond?

Imagine a user has $100 in their balance and sends two separate $50 requests. Both threads read the same $100 balance from the database at the same moment. Both check the business condition (balance >= 50), both pass, and both threads deduct $50 and write the result back to the database. End result: the user starts with $100, ends up buying $100 worth of goods, but the database still shows $50 left instead of $0 — this is the double-spending problem.

To fix this, we compared two main JPA locking strategies:

• Optimistic Locking (@Version): Adds a version column to the table. It doesn't lock anything on read, only checks the version on write. It's very fast under light load. But when thousands of parallel requests hit the same balance at once, it triggers a flood of OptimisticLockException errors, and most user requests end up getting rejected.

• Pessimistic Locking (SELECT FOR UPDATE): This is the approach we ended up favoring for financial transactions. By applying the @Lock(LockModeType.PESSIMISTIC_WRITE) annotation inside the JPA Repository, we enforce a row-level lock at the database level:

JavaWalletRepository.java

public interface WalletRepository extends JpaRepository<Wallet, Long> {
    @Lock(LockModeType.PESSIMISTIC_WRITE)
    @Query("SELECT w FROM Wallet w WHERE w.userId = :userId")
    Optional<Wallet> findByUserIdWithLock(@Param("userId") Long userId);
}

Copy tip: click inside the code panel → Cmd+A → Cmd+C. The syntax colors are purely for display; what gets pasted is clean Java code.

With this in place, as soon as the first thread finishes its work and commits the transaction, every other thread waits in the database's queue until then. As a result, the balance is updated accurately and consistently.

Figure 1: The moment a race condition occurs, and the Pessimistic Locking (SELECT FOR UPDATE) mechanism

2. Distributed Transactions and the Transactional Outbox Pattern

Once a payment completes successfully, the system needs to notify other services (say, the Notification or Invoice service) by publishing a PaymentCompletedEvent to Kafka. In our very first version, we made a classic mistake:

JavaDual-write (incorrect)

@Transactionalpublic void processPayment(PaymentRequest request) {
    Wallet wallet = walletRepository.findByUserIdWithLock(request.getUserId());
    wallet.deduct(request.getAmount());
    walletRepository.save(wallet); // 1. Update balance in DB
    kafkaTemplate.send("payment-topic", new PaymentEvent(wallet.getId())); // 2. Send message to Kafka
}

This approach carries a serious risk — the classic Dual-Write Problem:

• If the database transaction commits but the message fails to reach Kafka due to a network error, the payment goes through but no notification is ever sent.

• If the Kafka message goes out successfully but the database transaction fails right after and rolls back, no money actually gets deducted, yet other services think the payment went through.

We found the fix by adopting the Transactional Outbox Pattern:

• Instead of publishing directly to Kafka, we write the event into an outbox table as part of the same database transaction.

• A separate asynchronous background worker (or Debezium CDC) reads new entries from the outbox table and reliably delivers them to Kafka (at-least-once delivery).

• Once a message is successfully delivered to Kafka, its outbox record is marked as PROCESSED.

Figure 2: Transactional Outbox Pattern — writing to the outbox within the DB transaction and asynchronously forwarding to Kafka

3. HikariCP Connection Pool and Long-Running Transactions

One of the most critical issues we hit under heavy traffic was the Connection is not available, request timed out after 30000ms error.

Digging into the root cause, it turned out developers were calling the bank's external REST API from inside a @Transactional method.