We built Document360 in a bootstrapped model by making use of the right toolset for each purpose. We chose Algolia as our search partner. Here I’m sharing how we bumped into Algolia QPS Violation and how did we fix it.
Algolia as a Search Partner
For Document360, we chose Algolia as our Search Partner. Algolia is one of the most reliable platforms for building search capabilities. The hosted search API caters to billions of queries for different websites every month.
Algolia offers different tiers of subscription and with a higher tier they offer a higher Query per second (QPS). QPS is the highest number of hits received for any given second.
Let’s say, you have 10 customers searching something on your website at the same time and 10 requests arrive to Algolia for information – it will be counted as 10 QPS. Read more on how Algolia calculates the QPS here.
With our current subscription, our allowed limit is 75 QPS. This means that 75 requests are allowed to hit Algolia server per second. This value is quite enough unless you have a high volume e-commerce application.
We received a notification from Algolia about the QPS violation. We were quite surprised to see this and were clueless. Our immediate attention went to the customer-facing documentation websites (e.g. docs.document360.io).
The first question we had in mind was “Do we have enough customers searching for something at the same time simultaneously?” The answer was No. We then started to dig deeper into the problem.
What caused the Algolia QPS violation?
We use a background job for operational backup of all the projects (one at a time). This job also includes Algolia indexes for the backed up project. We use Algolia’s C# .NET SDK to manage the indexes from our codes.
The implementation performed a .BrowseAll() operation for given index. What we did not realize was that the index.BrowseAll() operation was very similar to performing a “Search” operation on the website. This resulted in the Algolia QPS violation.
We did not know of this violation until there were enough documents in any project. With growing number of customers and the documents in their projects, the number of Algolia index objects were also gradually increasing. We looked into the dashboard and found out that the count was 192 QPS for a certain date and between 80 to 100 on some other dates.
After investigation, we found that ~94% of the indexing operations triggered by the C# .NET SDK. The Algolia Dashboard was of real help in finding out the root cause.
This gave us the confirmation that the index.BrowseAll() operation was the culprit. We had to change the way we implemented Algolia Index backup process to fix the issue. We removed the usage of index.BrowseAll() completely and this sorted out the Algolia QPS violation issue.
If you are using any of Algolia’s SDK in the backend, you must cut down the usage of index.BrowseAll() operation to zero. This might not hurt for occasional usage. But for a SaaS application, as the number of users grows, it will become a trouble sooner or later.