Problems synchronizing

[billio]

I have installed kolibri on a raspberry to work locally, every so often I share internet to synchronize, however, the first few weeks if it worked but now no longer synchronizes from local to server.

[blaine]

Hi @billio,

Welcome to the Learning Equality forums!

Could you share more information about your usage of Kolibri? Some questions:

1. What version of Kolibri are you running on the Raspberry Pi and the ‘server’?
2. Is the ‘server’ located on the internet?
3. How are you sharing internet with the Raspberry Pi?
4. Do you see errors in the Kolibri UI for the sync task?
5. Could you re-attempt the sync and then share the Kolibri logs from the Raspberry Pi? Information on where to find the logs

Regards,
Blaine

[David_Nicolas]

Hi blaine I work together with billy, thank you very much for the support, answering your questions.

1. What version of Kolibri are you running on the Raspberry Pi and the ‘server’?
   Version 0.15.06 is being used.
   Yes, it is in a DigitalOcean droplet on an Ubuntu 22.10 x64 version.
2. Is the ‘server’ located on the internet?
   Yes, with its own domain and virtual machine.
3. How are you sharing internet with the Raspberry Pi?
   Internet is shared to the raspberry through a cellular hostpot.
4. Do you see errors in the Kolibri UI for the sync task?
   In the kolibri interface only the message “Error in synchronization” appears.
5. Could you re-attempt the sync and then share the Kolibri logs from the Raspberry Pi?
   When performing the synchronization through the console of the raspberry pi is frozen in this part, I attach a picture.
   
   

[blaine]

Hi @David_Nicolas,

Thank you for the information. It is helpful.

I see in the picture that you provided credentials when running the sync from the console. Since it asked for credentials, this would indicate that the Kolibri instances do not recognize each other-- have not synced before. This may be caused by some issue or change to Kolibri on the Raspberry Pi. As a result, this sync will act like an initial complete sync as opposed to an incremental sync, and it will take some time if the database is large.

If the above sounds correct, I have a number of ideas which I recommend you proceed through in order until it is resolved, testing the sync after each:

1. If possible, I highly recommend you upgrade the instances to at least v0.15.8, as there were a number of bug fixes between that version and the version you’re running.
2. I suggest you run the command kolibri manage cleanupsyncs on the DigitalOcean droplet. The command will cleanup the buffers from the failed syncs. The command takes an --expiration parameter, which is the number of hours of inactivity after which a sync should be considered stale and cleaned up. If there are no other Kolibri instances syncing with it, you may use --expiration 0 when running it. Full example:

kolibri manage cleanupsyncs --expiration 0

3. The sync appears to be freezing during the stage where the droplet instance is performing the most processing. During this time, the raspberry pi will have an open HTTP request to the droplet. If it takes a long time, the connection between the two may be closed or interrupted. We have a Kolibri plugin which you may enable on the droplet Kolibri instance to prevent such issues. You’ll need to add the following to Kolibri’s options.ini file on the droplet:

[Sync]
BACKGROUND_INITIALIZATION = True

Regards,
Blaine

[billio]

Hi @blaine! Thank you for your response, we will be trying what you recommend and we will let you know what the results were.

[David_Nicolas]

Thank you Blaine for your help, it is very valuable to us, as we have around 2000 students in rural communities using the platform right now.
-We have already updated to v0.15.12 on both Digital Ocean and the Raspberry Pi.
-We added the plugin to Digital Ocean.
We managed to synchronize after a long period of 2 hours.
Now our problem is that locally, the Raspberry Pi has progress for a student, but when synchronizing in CMD, it shows a successful synchronization, but the progress for this student does not appear on Digital Ocean. It does show that there was a synchronization in the Super Admin profile.

Thank you Blaine for continuing to support us, and greetings from Guatemala!

[blaine]

@David_Nicolas What installation method was used to install Kolibri on the Raspberry Pi?

Kolibri stores its sqlite databases and logs in a directory referred to as the ‘Kolibri Home’ directory. This defaults to a location specific to the user starting or executing Kolibri commands. By default it should be /home/<user>/.kolibri on the Raspberry Pi.

Some Kolibri installers create a new kolibri user for running Kolibri, which if the user you’re running the sync command is different than the user that Kolibri is normally running with, then it wouldn’t access the same Kolibri home directory and thus wouldn’t sync the learner progress. This would also explain why the sync command asked for credentials for syncing to the droplet.

[JuanJo]

@David_Nicolas How are you doing with this topic and Blaine’s suggestions? Can you please update us?

[David_Nicolas]

Hi @blaine sorry for the delay

I installed kolibri according to the documentation of Learning quality

The credentials shown in the image above was because it was a fresh install, as you mentioned, but it was freezing at that part of the sync. After the SYNC plugin change it was able to move forward and the sync was done.

Now another problem arose, the synchronization, I run it automatically from the raspberry with a Crontab -e runs successfully in a unidirectional way, for example I add a lesson from the online part and synchronizes perfect with the local, but when creating an offline lesson is not synchronized online, that is a detail and the following is that the progress of students are not synchronized, how could I know if the database is not duplicated or do not know if that’s the problem?

Thank you very much for your support, sorry for the delay.

[blaine]

Crontab -e runs successfully in a unidirectional way

@David_Nicolas By “unidirectional way”, do you mean that you’ve configured it to run Kolibri’s sync command with the --no-push argument? And that’s the only command you’ve configured to run a sync via the crontab?

[David_Nicolas]

Hi blaine thanks for replying, the command I run with crontab -e is the sync command.
@reboot sleep 120 && kolibri manage sync --baseurl “att.1bot.online:80”
@hourly kolibri manage sync --baseurl “att.1bot.online:80”

With the issue of unidirectional synchronization I mean that for example I create a lesson on the digital ocean server and at the time of synchronizing if it appears on the raspberry server, but for example if I create a lesson on the raspberry server is not synchronized on the digital ocean server. The same situation happens to me with the students’ progress.

[blaine]

@David_Nicolas Have you checked the logs on the two instances?

A sync initiated from the raspberry server will first pull from the digital ocean server, then push to it. It sounds like the push portion of the sync is failing somewhere. So I expect that there’s an error in the logs on either or both sides

[David_Nicolas]

Ok, I understand, I haven’t checked yet, how can I check the records, sorry?

[blaine]

No problem, @David_Nicolas!

There’s a logs/ directory in the Kolibri home directory, which I mentioned previously. Here’s the documentation on how to find them.

Previously, you said:

the Raspberry Pi has progress for a student, but when synchronizing in CMD, it shows a successful synchronization, but the progress for this student does not appear on Digital Ocean. It does show that there was a synchronization in the Super Admin profile.

That synchronization information indicating it did sync you saw in the Super Admin profile on the digital ocean server?

[David_Nicolas]

These are the Digital Ocean logs

And these are the RPI logs

Regarding the other question, the information does appear in the Educational Centers tab as synchronization “just now” in both devices in Digital Ocean and in the RPI.

But the student’s progress is not synchronized

[blaine]

Hi @David_Nicolas,

So the just now in the UI should indicate the sync did complete successfully, which leads me to suspect some deserialization errors are causing this.

A couple of followup tasks:

1. Please use the following command, working from the Kolibri home directory, to search the Kolibri logs on the Digital Ocean server:

egrep -i "error deserializing|validation error" logs/kolibri.txt

2. If the Digital Ocean server doesn’t have sqlite3 installed, please install it. Then please use the following command, working from the Kolibri home directory again, on the Digital Ocean server:

sqlite3 ./db.sqlite3 "select id, model_name, deserialization_error from morango_store where coalesce(deserialization_error, '') != ''"

3. Lastly, this command also requires sqlite3, but may expose private information, so please be aware before sharing its output (you can DM me otherwise):

sqlite3 ./db.sqlite3 "select id, model_name, conflicting_serialized_data from morango_buffer where conflicting_serialized_data != ''"

[David_Nicolas]

Hi @blaine
I will send you by DM the screenshot of what Digital Ocean returned.

[blaine]

Thanks @David_Nicolas

Since there wasn’t any output from any of the commands, that seems to indicate there are no errors.

Could you navigate to Device > Info on each Kolibri instance, and copy and paste the information from the Advanced section?

[blaine]

@David_Nicolas While grabbing that Advanced info, could you also check the Coach > Reports tab and verify that you also do not see the student’s progress there either?

[blaine]

I suspect that this is no longer an issue related to syncing, but a bug within Kolibri. It seems the Digital Ocean server is still on Kolibri 0.15.6, not the latest 0.15.x release.

Knowing whether or not the student’s progress is visible in the Coach Reports would help narrow it down. The screens shown in screenshots are mostly driven by a notifications mechanism, which could have differences with the underlying data if there’s a bug.

EDIT:
Seems I was mistaken, the server is updated to 0.15.12, my apologies.

I believe the primary issue is buffer bloat from failed or interrupted syncs. When a sync gets interrupted, it leaves the transferred data in limbo unless the sync was resumed. This data accumulates with recurrent syncs and causes the database to grow very large (your database is already very large) and queries against the syncing tables to slowdown. I suggest you occasionally configure the Digital Ocean server to run the cleanupsyncs command, but at a time when no syncs should be occurring to be safe.

For example, for your installation:

KOLIBRI_HOME="/var/kolibri/.kolibri" kolibri manage cleanupsyncs

I also suggest you stagger the crontab syncs so that they avoid syncing to the Digital Ocean server around the same time.

I hope this helps,
Blaine