Why five servers could not handle the load: how we redesigned the database architecture

Category: Database Optimization / Infrastructure / High Load

Technologies: Galera Cluster, MySQL, Proxmox VE, ProxySQL, Database Replication, Cloudflare

Automated bots were generating more load than even a cluster of five physical servers could handle. Restricting traffic through Cloudflare was not an option because it caused access problems for real users. Here is how we redesigned the database architecture, introduced replication and query caching, and eliminated the cause of recurring failures without buying additional hardware.

The problem: heavy load on Galera Cluster

The project ran on five physical servers using Galera Cluster. This configuration provided a distributed database setup, but over time the project developed a serious problem: instability under heavy automated traffic.

Bots regularly visited the website to collect and process published information. Their activity generated a large number of database queries, placing substantial load on the infrastructure.

During particularly intensive scraping, the database could no longer keep up with the volume of requests, causing service failures.

Why not simply block the bots?

One obvious solution was to restrict automated traffic through Cloudflare.

In practice, however, this approach created additional problems. When restrictions were introduced, some real users had trouble accessing the website.

We were effectively being asked to choose between reducing infrastructure load and keeping the service accessible to customers.

That trade-off was unacceptable to us.

The goal was to keep the database stable without restricting user access or blocking automated requests.

The analysis: why adding servers did not solve the problem

The original architecture consisted of five physical servers running as a Galera Cluster.

Our analysis showed that increasing the number of servers was not delivering the expected benefits. Even with the load distributed, intensive queries continued to put considerable pressure on the database subsystem.

The issue was not just the available computing resources, but also the way the data was being accessed.

Automated systems repeatedly requested information that did not require the same read operations to be executed directly against the database every time.

Simply adding more servers would therefore not address the root cause of the overload.

We decided to stop expanding resources and redesign the way requests were processed.

The solution: virtualization, replication and caching

Instead of buying additional hardware, we reallocated the infrastructure's existing resources.

Two of the five physical servers were removed from the original cluster and repurposed to host separate databases.

1. Deploying virtual infrastructure on Proxmox VE

We installed the Proxmox VE hypervisor on the two repurposed physical servers.

We created four virtual machines on each server, giving us eight virtual machines for the databases in total.

The first physical server hosted four independent databases containing the information most frequently requested by automated systems.

The second physical server hosted the corresponding replicas of those databases.

Each of the four databases therefore had its own replica on a different physical server.

The database instances communicated over the local network.

2. Separating writes from reads

The next step was to separate write and read operations.

The application wrote data to the databases on the first physical server. Replication propagated changes to the corresponding replicas on the second server.

Client requests were given read-only access.

This separated data modifications from the intensive stream of queries generated by automated data collection.

As a result, the load was distributed across separate database instances, while access rules kept writes separate from the main stream of client queries.

3. Caching SELECT queries with ProxySQL

Introducing ProxySQL was a key part of the new architecture.

We configured SELECT query caching to reduce repeated requests directly to the databases.

When an eligible query was repeated, ProxySQL could return the cached result without forwarding the request to the database server.

This is particularly useful for projects regularly visited by automated systems: bots may request the same information repeatedly even when the underlying data has not changed.

Caching meant that a substantial share of repeated queries no longer placed additional load on the database servers.

Automated traffic continued to be served, and Cloudflare restrictions were no longer necessary.

The result: stable operation without additional hardware

After implementing the new architecture, we eliminated the database overload that had previously caused service failures.

The main results:

• Lower database load. Repeated SELECT queries were served using the ProxySQL cache.
• Stable service operation. After the architecture change, we observed no further database failures caused by the previous overload.
• Continued access for users. There was no need to restrict automated traffic through Cloudflare in ways that affected real visitors.
• Separate handling of operations. Data writes and client read queries were processed through separate mechanisms.
• Better use of existing resources. The solution used existing hardware without purchasing additional physical servers.

Instead of increasing infrastructure capacity further, we changed the approach to storing and processing data.

The five physical servers in the original configuration were reallocated: three remained in the main cluster, while two formed a separate virtualized infrastructure with four pairs of replicated databases.

This approach removed the previous bottlenecks and enabled stable query processing.

The takeaway: architecture matters more than server count

This project showed that high load does not always require more powerful hardware.

Even distributed infrastructure can suffer performance problems when its architecture does not account for query patterns.

In our case, a combination of virtualization, replication and caching solved the problem by changing how the data was handled.

The main achievement was making better use of the resources already available, rather than increasing the number of servers.

Having server or database performance problems?

High load, slow queries and unstable applications do not always mean your infrastructure lacks capacity.

Sometimes, identifying the bottleneck and redistributing the load is enough.

The inSave Hosting team helps diagnose performance problems, optimize databases and design server infrastructure around a project's actual needs.

Contact us to discuss your requirements and find the right technical solution.