Do you have some recommendations that we could utilize?
Hi @safi_khan,
I saw your other new thread but wanted to comment regarding the screenshots you’ve shared here. From the first screenshot of the database’s CPU utilization (which I’m attaching a cropped version of here for clarity) it appears there are frequent spikes up to roughly 50% utilization.
It’s difficult to tell the duration of those spikes from the graph, but relative to the other screenshots of the database CPU utilization, like the 3 hour graph and the query load statistics, I don’t feel confident those actually captured what occurs during those spikes. For example, the 3 hour graph shows the CPU utilization never exceeding 14% which is nowhere near the 50% spikes in the first graph.
Inline with what @richard has shared, including the other thread: narrowing in on what occurs during those spikes, their duration, and any possible connection to page load issues observed would be really helpful.
Lastly, if you intend to implement load balancing or autoscaling, I highly recommend getting Redis setup along the lines of what @richard shared here, for session storage but also for caching. If Kolibri stalls during startup, it probably can’t connect to Redis, so I would suggest searching for ways to test the connection to Redis from those servers, outside of Kolibri.
Regards,
Blaine
Those DB spikes were the effects of me running a cronjob to import users every hour. I disabled the cronjob but we are still seeing the high load time issue
DB after disabling the cronjob
If we move to Redis connection then Kolibri won’t be using the files for storing session information? Also, how can I confirm if kolibri is using the redis server that I attached in the options.ini file?
To avoid Kolibri using the file backed session storage, you would still need to switch out the session backend as I described here: Kolibri Performance Issue - #19 by richard
I did forget to mention there how to use the new settings file, but you would just need to set the environment variable DJANGO_SETTINGS_MODULE=<python path of settings module> in your base docker image.
I am still not sure how to make that change and build the kolibri application to be installed on an ubuntu instance.
For testing purposes: I created a new postgres DB using the same data that we have in production and also created a new ec2 instance that is running kolibri-server.
I tried loading the coach’s tab for running reports, and it still took a long time to load (it didn’t crash though as it would in production). So, can we assume that a big number of concurrent users on kolibri has too little effect on performance?
Hi @richard,
We are still running into issues, and it still seems like the kolibri application is not using all the potential resources available. How can I set kolibri application to use all the resources that I available to use? Like memory, CPU.
We have a high load on the system but all metrics are below 10%.
This is from last 12 hours
This is the targerResponseTIme in seconds
How many processes is uwsgi currently configured to use in the kolibri-server setup? Have you confirmed that kolibri-server is definitely serving the requests now?
It rather looks like something other than CPU and memory are the bottlenecks here, so the only other things I can think of are disk I/O and database queries, could either of these be having an impact?
If there is any sort of cache stampede happening, this might also be the cause, where multiple users are simultaneously trying to refresh the same cache, and then there is write contention to write the cached resource at the same time. Unfortunately, you need to dig further using your system administration knowledge to understand where the bottlenecks are so we can help further.