Upgrade from 0.15.12 in Ubuntu 20.04.6 server

Support Kolibri
1

Hello support
Hope all is well with you

I see there is new 0.16 version available, but when using apt update in existing Kolibri server there is no upgrade available

Server info

Version: 0.15.12
OS: Linux-5.4.0-174-generic-x86_64-with-glibc2.29
Python: 3.8.10
Installer: deb kolibri-server - 0.4.0-0ubuntu2
Server: nginx/1.18.0 (Ubuntu)
Database: /home/kolibri/.kolibri/db.sqlite3
Free disk space: 281 GB
Server time: Thu Apr 04 2024 19:56:52 GMT-0400 (Eastern Daylight Time)
Server timezone: Etc/UTC
Device ID: 3d938bfcc2e0d88860c4eb158da2fdd4

CLI attempt at update

kolibri@kol-srv:/$ sudo apt update
Hit:1 Index of /ubuntu focal InRelease
Hit:2 Index of /ubuntu focal-updates InRelease
Hit:3 Index of /ubuntu focal-backports InRelease
Hit:4 Index of /ubuntu focal-security InRelease
Hit:5 Index of /learningequality/kolibri/ubuntu focal InRelease
Reading package lists… Done
Building dependency tree
Reading state information… Done
All packages are up to date.

Do we need to upgrade the version of Ubuntu?

Thank you for your help

2

Hello @mrdavidhaag
Yes, the version is already available at Releases · learningequality/kolibri · GitHub but a new bug fixing version is about to be released with version 0.16.1.
If you need 0.16.0 you can download the package directly from https://github.com/learningequality/kolibri/releases/download/v0.16.0/kolibri_0.16.0-0ubuntu1_all.deb

After 0.16.1 is released It’ll be available using apt, at least, for Ubuntu 20.04 & 22.04
If you need 0.16.0 urgently, let me know it and It’ll be available today (if Ubuntu launchpad recovers , it’s down now)

Regards
José

3

Hello @jredrejo
Thank you for the prompt response and positive solution

We are not in a hurry, so will wait until 0.16.1

The server I am trying to upgrade is at ObAnggen School in Bokondini, Papua - the server has been suffering a bit with difficulties in user access. Users are unable to login and if successful have long delays in server response. This issue is not new and the work around has been to make a new user account for the student.

I believe the issue is caused by the students account being several years old and the student is moved from class to next class in normal progression of school years. Maybe a database issue because the db at ObAnggen is now over 5GB?

image

Sorry for running down a different support trail. I have been a bit lazy in resolving the delay issue because the work around “fixed” the issue.

The barrier to start fresh with new server, updated ubuntu and updated content has been the teachers have many quizzes, lesson plans and other created work that they use year to year. The legacy teachers do not want to have to re-create their work in a new server. Plus, if a new teacher arrives, the created work is done and the only need for training is how to implement existing lesson plans and quizzes in Kolibri.

Thank you again for your time and support.

4

hi @mrdavidhaag
Definitively, the size of your db is not optimal, but if your server memory is big enough, it should not be a big problem unless the db file is not in a good state.

After so long, it’s probably “defragmented”. My first try would be to do a manual “vacuum” of the database. If using a Linux terminal is something where you feel comfortable I can provide you some instructions to do a proper vacuum.

Also, if you don’t mind losing it, some commands can be used to delete years of logging of your users. That won’t delete any of the created classes, exams or quizzes. But, in any case, doing a vacuum is the first step. Only after checking it does not improve things I’d begin to think of deleting some data.

Regards
José

5

Hi @jredrejo
thank you for the advice on vacuum process for db.

Yes, I am comfortable with CLI in Linux terminal. Please forward instructions and I will give the vacuum a try. Also can you please include instructions for making a backup of the db

Right now in the /home/kolibri/.kolibri/backups/ folder there are two .dump files which I have no idea why they are there with today’s date

image

The kolibri server is a VM in a HP Microserver Gen10, so plenty of memory and storage available.

image

Thank you for your help
Kind regards,
David

6

hello again @mrdavidhaag
those dumps you see are automatically done whenever kolibri is upgraded, they seem to have been done yesterday so they should be updated. In any case, we can do another backup before applying vacuum to be completely safe.
These should be the steps (all of them to be executed in a Linux terminal):

  1. First of all, stop kolibri: sudo systemctl stop kolibri
  2. Be sure to have installed the tool we’re going to need: apt install sqlite3
  3. Inside your /home/kolibri/.kolibri folder, do a backup of the current db: cp db.sqlite3 db.sqlite3.bak
  4. Load the database with sqlite: sqlite3 db.sqlite3
  5. Do the vacuum, with the size of your database it’ll take some time, maybe minutes: vacuum;
  6. Exit from the tool: .quit
  7. Restart kolibri and test if it’s going better: sudo systemctl start kolibri

if it’s still having problems, try to apply these steps: Troubleshooting — Kolibri User Guide

As a last option we can think of deleting your users logs, let me know how things go with these instructions.

José

7

hello again @jredrejo
Thank you for the instructions. The process did not go so well.

I followed your instructions and this is terminal from the session

Last login: Fri Apr 5 08:40:18 2024 from 10.10.30.1
kolibri@kol-oa:~$ sudo systemctl stop kolibri
[sudo] password for kolibri:
kolibri@kol-oa:~$ sudo apt install sqlite3
Reading package lists… Done
Building dependency tree
Reading state information… Done
Suggested packages:
sqlite3-doc
The following NEW packages will be installed:
sqlite3
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 860 kB of archives.
After this operation, 2803 kB of additional disk space will be used.
Get:1 Index of /ubuntu focal-updates/main amd64 sqlite3 amd64 3.31.1-4ubuntu0.6 [860 kB]
Fetched 860 kB in 7s (119 kB/s)
Selecting previously unselected package sqlite3.
(Reading database … 121955 files and directories currently installed.)
Preparing to unpack …/sqlite3_3.31.1-4ubuntu0.6_amd64.deb …
Unpacking sqlite3 (3.31.1-4ubuntu0.6) …
Setting up sqlite3 (3.31.1-4ubuntu0.6) …
Processing triggers for man-db (2.9.1-1) …
kolibri@kol-oa:~$ cd /home/kolibri/.kolibri/
kolibri@kol-oa:~/.kolibri$ cp db.sqlite3 db.sqlite3.bak
kolibri@kol-oa:~/.kolibri$ pwd
/home/kolibri/.kolibri
kolibri@kol-oa:~/.kolibri$ ls
backups logs process_cache
content networklocation.sqlite3 sessions
databases nginx.conf static
db.sqlite3 notifications.sqlite3 syncqueue.sqlite3
db.sqlite3.bak options.ini temp
job_storage.sqlite3 options.ini.kolibri-server-backup
log_export plugins.json
kolibri@kol-oa:~/.kolibri$ sqlite3 db.sqlite3
SQLite version 3.31.1 2020-01-27 19:55:54
Enter “.help” for usage hints.
sqlite> vacuum
…>

Afte entering the vacuum command the prompt immediately went to …> and sat with no change as above …>

In my inexperience of the sqlite tool, I did not know how long I had to wait or what the prompt (…>) will look like when finished vacuum. I waited 10 minutes (using stopwatch) and no change.

I wanted to compare how the sqlite3 tool worked on kolibri server with very small database. Then in my hopes of seeing something different in small database, I performed the same steps on local kolibri server with 150kb database.

The response was the same, after giving vacuum command the prompt immediately went to …> and just sat there with no change

The time elapsed on vacuum command in the ObAnggen (OA) server was now closing in on 15 minutes. I perhaps gave up too early.

I then tried to exit the vacuum command tool with the .quit command but that did not work on either server. I looked up exiting sqlite3 and found the CTRL-D option and this worked to exit the vacuum tool.

After exit of sqlite3 tool with CTRL-D, I tried to restart kolibri service in the OA server and was unable to start the service. I was able to restart kolibri service in the local server.

I chose to reboot the OA server and this is status of kolibri service

lines 1-21/21 (END)
● kolibri.service - LSB: kolibri daemon, an offline education platform
Loaded: loaded (/etc/init.d/kolibri; enabled; vendor preset: enabled)
Active: activating (start) since Sat 2024-04-06 20:57:52 WIT; 1min 42s ago
Cntrl PID: 845 (kolibri)
Tasks: 3 (limit: 19088)
Memory: 2.4G
CGroup: /system.slice/kolibri.service
├─845 /bin/sh /etc/init.d/kolibri start
├─873 runuser kolibri -c kolibri start
└─887 /usr/bin/python3 /bin/kolibri start

Apr 06 20:57:52 kol-oa systemd[1]: Starting LSB: kolibri daemon, an offline education platform…
Apr 06 20:57:52 kol-oa runuser[873]: pam_unix(runuser:session): session opened for user kolibri by (uid=0)
Apr 06 20:57:53 kol-oa kolibri[887]: WARNING 2024-04-06 20:57:53,837 Option CHERRYPY_START in section [Server] is deprecate>
Apr 06 20:57:53 kol-oa kolibri[887]: INFO 2024-04-06 20:57:53,839 Option DEBUG in section [Server] being overridden by e>
Apr 06 20:57:53 kol-oa kolibri[887]: INFO 2024-04-06 20:57:53,839 Option DEBUG_LOG_DATABASE in section [Server] being ov>
Apr 06 20:57:54 kol-oa kolibri[887]: /usr/lib/python3/dist-packages/kolibri/dist/rest_framework/utils/serializer_helpers.py:>
Apr 06 20:57:54 kol-oa kolibri[887]: if value is None or value is ‘’:
Apr 06 20:57:54 kol-oa kolibri[887]: INFO 2024-04-06 20:57:54,425 Running Kolibri with the following settings: kolibri.d>
Apr 06 20:57:54 kol-oa kolibri[887]: INFO 2024-04-06 20:57:54,463 Configuring Redis: maxmemory-policy allkeys-lru
Apr 06 20:57:54 kol-oa kolibri[887]: INFO 2024-04-06 20:57:54,463 Configuring Redis: maxmemory 1516069683

I then took a look at the db files and no change

image

As far as I can tell the OA server is functioning OK. Will know for sure when school start on Monday.

This is journalctl for kolibri service for past day

kolibri@kol-oa:~$ sudo journalctl -u kolibri.service --since “1 day ago”
[sudo] password for kolibri:
– Logs begin at Sun 2022-02-20 04:14:03 WIT, end at Sun 2024-04-07 03:34:04 WIT. –
Apr 06 20:55:22 kol-oa systemd[1]: Starting LSB: kolibri daemon, an offline education platform…
Apr 06 20:55:22 kol-oa runuser[4443]: pam_unix(runuser:session): session opened for user kolibri by (uid=0)
Apr 06 20:55:23 kol-oa kolibri[4444]: WARNING 2024-04-06 20:55:23,666 Option CHERRYPY_START in section [Server] is deprecat>
Apr 06 20:55:23 kol-oa kolibri[4444]: INFO 2024-04-06 20:55:23,669 Option DEBUG in section [Server] being overridden by >
Apr 06 20:55:23 kol-oa kolibri[4444]: INFO 2024-04-06 20:55:23,669 Option DEBUG_LOG_DATABASE in section [Server] being o>
Apr 06 20:55:24 kol-oa kolibri[4444]: /usr/lib/python3/dist-packages/kolibri/dist/rest_framework/utils/serializer_helpers.py>
Apr 06 20:55:24 kol-oa kolibri[4444]: if value is None or value is ‘’:
Apr 06 20:55:24 kol-oa kolibri[4444]: INFO 2024-04-06 20:55:24,257 Running Kolibri with the following settings: kolibri.>
– Reboot –
Apr 06 20:57:52 kol-oa systemd[1]: Starting LSB: kolibri daemon, an offline education platform…
Apr 06 20:57:52 kol-oa runuser[873]: pam_unix(runuser:session): session opened for user kolibri by (uid=0)
Apr 06 20:57:53 kol-oa kolibri[887]: WARNING 2024-04-06 20:57:53,837 Option CHERRYPY_START in section [Server] is deprecate>
Apr 06 20:57:53 kol-oa kolibri[887]: INFO 2024-04-06 20:57:53,839 Option DEBUG in section [Server] being overridden by e>
Apr 06 20:57:53 kol-oa kolibri[887]: INFO 2024-04-06 20:57:53,839 Option DEBUG_LOG_DATABASE in section [Server] being ov>
Apr 06 20:57:54 kol-oa kolibri[887]: /usr/lib/python3/dist-packages/kolibri/dist/rest_framework/utils/serializer_helpers.py:>
Apr 06 20:57:54 kol-oa kolibri[887]: if value is None or value is ‘’:
Apr 06 20:57:54 kol-oa kolibri[887]: INFO 2024-04-06 20:57:54,425 Running Kolibri with the following settings: kolibri.d>
Apr 06 20:57:54 kol-oa kolibri[887]: INFO 2024-04-06 20:57:54,463 Configuring Redis: maxmemory-policy allkeys-lru
Apr 06 20:57:54 kol-oa kolibri[887]: INFO 2024-04-06 20:57:54,463 Configuring Redis: maxmemory 1516069683
Apr 06 21:03:20 kol-oa kolibri[887]: INFO 2024-04-06 21:03:20,766 Backed up database to: /home/kolibri/.kolibri/backups/>
Apr 06 21:03:20 kol-oa kolibri[887]: INFO 2024-04-06 21:03:20,767 Version was 0.14.7, new version: 0.15.12
Apr 06 21:03:20 kol-oa kolibri[887]: INFO 2024-04-06 21:03:20,767 Running update routines for new version…
Apr 06 21:03:21 kol-oa kolibri[887]: Operations to perform:
Apr 06 21:03:21 kol-oa kolibri[887]: Apply all migrations: admin, analytics, auth, bookmarks, content, contenttypes, devic>
Apr 06 21:03:21 kol-oa kolibri[887]: Running migrations:
Apr 06 21:03:21 kol-oa kolibri[887]: No migrations to apply.
Apr 06 21:03:21 kol-oa kolibri[887]: Operations to perform:
Apr 06 21:03:21 kol-oa kolibri[887]: Apply all migrations: admin, analytics, auth, bookmarks, content, contenttypes, devic>
Apr 06 21:03:21 kol-oa kolibri[887]: Running migrations:
Apr 06 21:03:21 kol-oa kolibri[887]: No migrations to apply.
Apr 06 21:03:21 kol-oa kolibri[887]: Operations to perform:
Apr 06 21:03:21 kol-oa kolibri[887]: Apply all migrations: admin, analytics, auth, bookmarks, content, contenttypes, devic>
Apr 06 21:03:21 kol-oa kolibri[887]: Running migrations:
Apr 06 21:03:21 kol-oa kolibri[887]: No migrations to apply.
Apr 06 21:04:02 kol-oa kolibri[887]: Operations to perform:
Apr 06 21:04:02 kol-oa kolibri[887]: Apply all migrations: admin, analytics, auth, bookmarks, content, contenttypes, devic>
Apr 06 21:04:02 kol-oa kolibri[887]: Running migrations:
Apr 06 21:04:02 kol-oa kolibri[887]: No migrations to apply.
Apr 06 21:04:02 kol-oa kolibri[887]: Installed 2 object(s) from 1 fixture(s)
Apr 06 21:04:22 kol-oa kolibri[887]: Error: Traceback (most recent call last):
Apr 06 21:04:22 kol-oa kolibri[887]: File “/usr/lib/python3/dist-packages/kolibri/dist/django/db/backends/utils.py”, line >
Apr 06 21:04:22 kol-oa kolibri[887]: return self.cursor.execute(sql, params)

During the process I certainly was confused by my inexperience with sqlite3 tool and the .quit command not working while still in vacuum command.

I made a Snapshot of the OA server VM before starting the vacuum process, so I can rollback if problems arise on Monday.

My questions are:
Did I enter the wrong command for vacuum? Perhaps I needed a statement after the vacuum command?
Do you know how the sqlite3 prompt is supposed to act during a vacuum?
Do you think the 15 minute time period was sufficient? or did I exit too early?
Are there other reports you would like to see from the OA server to help troubleshoot the failed vacuum process?

Thank you for your help and apologies for my inexperience/impatience in the vacuum process.

Kind regards,
David

8

@mrdavidhaag I’m afraid the problem comes from the origin. It’s one of those nitpicking syntax issues: you just forgot a semicolon.
I.e. you have to execute
vacuum;
instead of
vacuum

All the commands inside sqlite need a semicolon at the end. Excepting the .quit one that needs a dot at the beginning.

As you noticed, in your tries you didn’t apply any vacuum command because of the missing semicolon.

Hope this helps
José

9

Hi @jredrejo Thank you for the correction. I tried the process again with proper syntax and the vacuum; process finished after 8 minutes and 7 seconds.

Last login: Sat Apr 6 21:34:58 2024 from 10.10.30.1
kolibri@kol-oa:~$ sudo systemctl stop kolibri
[sudo] password for kolibri:
kolibri@kol-oa:~$ cd /home/kolibri/.kolibri/
kolibri@kol-oa:~/.kolibri$ cp db.sqlite3 db.sqlite3.bak
kolibri@kol-oa:~/.kolibri$ sqlite3 db.sqlite3
SQLite version 3.31.1 2020-01-27 19:55:54
Enter “.help” for usage hints.
sqlite> vacuum;
sqlite> .quit
kolibri@kol-oa:~/.kolibri$ sudo systemctl start kolibri
[sudo] password for kolibri:
Job for kolibri.service failed because the control process exited with error cod e.
See “systemctl status kolibri.service” and “journalctl -xe” for details.
kolibri@kol-oa:~/.kolibri$

I was able to .quit the sqlite3 tool and took a look at the db.sqlite3 file. The size of the db file was reduced from 5.3GB down to 4.8GB.

after-vacuum

However, when attempting to restart kolibri.service the start failed.

This is systemctl status

kolibri@kol-oa:~/.kolibri$ systemctl status kolibri.service
● kolibri.service - LSB: kolibri daemon, an offline education platform
Loaded: loaded (/etc/init.d/kolibri; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Mon 2024-04-08 23:24:53 WIT; 4min 3s ago
Process: 6284 ExecStart=/etc/init.d/kolibri start (code=exited, status=1/FAILURE)

Apr 08 23:24:52 kol-oa kolibri[6309]: raise value.with_traceback(tb)
Apr 08 23:24:52 kol-oa kolibri[6309]: File “/usr/lib/python3/dist-packages/kolibri/dist/django/db/backends/utils.py”, line>
Apr 08 23:24:52 kol-oa kolibri[6309]: return self.cursor.execute(sql, params)
Apr 08 23:24:52 kol-oa kolibri[6309]: File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/backends/sqlite3/base.py>
Apr 08 23:24:52 kol-oa kolibri[6309]: return Database.Cursor.execute(self, query, params)
Apr 08 23:24:52 kol-oa kolibri[6309]: django.db.utils.IntegrityError: NOT NULL constraint failed: logger_attemptlog.sessionl>
Apr 08 23:24:53 kol-oa runuser[6308]: pam_unix(runuser:session): session closed for user kolibri
Apr 08 23:24:53 kol-oa systemd[1]: kolibri.service: Control process exited, code=exited, status=1/FAILURE
Apr 08 23:24:53 kol-oa systemd[1]: kolibri.service: Failed with result ‘exit-code’.
Apr 08 23:24:53 kol-oa systemd[1]: Failed to start LSB: kolibri daemon, an offline education platform.

and this is journal -xe

Apr 08 23:24:53 kol-oa systemd[1]: Failed to start LSB: kolibri daemon, an offline education platform.
kolibri@kol-oa:~/.kolibri$ journalctl -xe
Apr 08 23:24:52 kol-oa kolibri[6309]: self._insert(item, fields=fields, using=self.db)
Apr 08 23:24:52 kol-oa kolibri[6309]: File “/usr/lib/python3/dist-packages/kolibri/dist/django/db/models/query.py”, line 1>
Apr 08 23:24:52 kol-oa kolibri[6309]: return query.get_compiler(using=using).execute_sql(return_id)
Apr 08 23:24:52 kol-oa kolibri[6309]: File “/usr/lib/python3/dist-packages/kolibri/dist/django/db/models/sql/compiler.py”,>
Apr 08 23:24:52 kol-oa kolibri[6309]: cursor.execute(sql, params)
Apr 08 23:24:52 kol-oa kolibri[6309]: File “/usr/lib/python3/dist-packages/kolibri/dist/django/db/backends/utils.py”, line>
Apr 08 23:24:52 kol-oa kolibri[6309]: return self.cursor.execute(sql, params)
Apr 08 23:24:52 kol-oa kolibri[6309]: File “/usr/lib/python3/dist-packages/kolibri/dist/django/db/utils.py”, line 94, in _>
Apr 08 23:24:52 kol-oa kolibri[6309]: six.reraise(dj_exc_type, dj_exc_value, traceback)
Apr 08 23:24:52 kol-oa kolibri[6309]: File “/usr/lib/python3/dist-packages/kolibri/dist/django/utils/six.py”, line 685, in>
Apr 08 23:24:52 kol-oa kolibri[6309]: raise value.with_traceback(tb)
Apr 08 23:24:52 kol-oa kolibri[6309]: File “/usr/lib/python3/dist-packages/kolibri/dist/django/db/backends/utils.py”, line>
Apr 08 23:24:52 kol-oa kolibri[6309]: return self.cursor.execute(sql, params)
Apr 08 23:24:52 kol-oa kolibri[6309]: File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/backends/sqlite3/base.py>
Apr 08 23:24:52 kol-oa kolibri[6309]: return Database.Cursor.execute(self, query, params)
Apr 08 23:24:52 kol-oa kolibri[6309]: django.db.utils.IntegrityError: NOT NULL constraint failed: logger_attemptlog.sessionl>
Apr 08 23:24:53 kol-oa runuser[6308]: pam_unix(runuser:session): session closed for user kolibri
Apr 08 23:24:53 kol-oa systemd[1]: kolibri.service: Control process exited, code=exited, status=1/FAILURE
– Subject: Unit process exited
– Defined-By: systemd
– Support: http://www.ubuntu.com/support

– An ExecStart= process belonging to unit kolibri.service has exited.

– The process’ exit code is ‘exited’ and its exit status is 1.
Apr 08 23:24:53 kol-oa systemd[1]: kolibri.service: Failed with result ‘exit-code’.
– Subject: Unit failed
– Defined-By: systemd
– Support: http://www.ubuntu.com/support

– The unit kolibri.service has entered the ‘failed’ state with result ‘exit-code’.
Apr 08 23:24:53 kol-oa systemd[1]: Failed to start LSB: kolibri daemon, an offline education platform.
– Subject: A start job for unit kolibri.service has failed
– Defined-By: systemd
– Support: http://www.ubuntu.com/support

– A start job for unit kolibri.service has finished with a failure.

– The job identifier is 6758 and the job result is failed.
Apr 08 23:24:53 kol-oa sudo[6281]: pam_unix(sudo:session): session closed for user root
Apr 08 23:30:01 kol-oa CRON[6428]: pam_unix(cron:session): session opened for user root by (uid=0)
Apr 08 23:30:01 kol-oa CRON[6429]: (root) CMD ([ -x /etc/init.d/anacron ] && if [ ! -d /run/systemd/system ]; then /usr/sbin>
Apr 08 23:30:01 kol-oa CRON[6428]: pam_unix(cron:session): session closed for user root

I chose to reboot and this is systemctl status after reboot

kolibri@kol-oa:~$ systemctl status kolibri.service
● kolibri.service - LSB: kolibri daemon, an offline education platform
Loaded: loaded (/etc/init.d/kolibri; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Mon 2024-04-08 23:36:00 WIT; 48s ago
Process: 830 ExecStart=/etc/init.d/kolibri start (code=exited, status=1/FAILURE)

Apr 08 23:35:59 kol-oa kolibri[861]: initialize(**get_initialize_params())
Apr 08 23:35:59 kol-oa kolibri[861]: File “/usr/lib/python3/dist-packages/kolibri/utils/main.py”, line 290, in initialize
Apr 08 23:35:59 kol-oa kolibri[861]: conditional_backup(kolibri.version, version)
Apr 08 23:35:59 kol-oa kolibri[861]: File “/usr/lib/python3/dist-packages/kolibri/utils/main.py”, line 90, in conditional_>
Apr 08 23:35:59 kol-oa kolibri[861]: os.remove(os.path.join(default_path, old_backup))
Apr 08 23:35:59 kol-oa kolibri[861]: FileNotFoundError: [Errno 2] No such file or directory: '/home/kolibri/.kolibri/backups>
Apr 08 23:36:00 kol-oa runuser[858]: pam_unix(runuser:session): session closed for user kolibri
Apr 08 23:36:00 kol-oa systemd[1]: kolibri.service: Control process exited, code=exited, status=1/FAILURE
Apr 08 23:36:00 kol-oa systemd[1]: kolibri.service: Failed with result ‘exit-code’.
Apr 08 23:36:00 kol-oa systemd[1]: Failed to start LSB: kolibri daemon, an offline education platform.
lines 1-15/15 (END)

This is all I have done so far. The school is in time zone +9UTC so they are sleeping now. I did not want to roll back to previous snapshot or take other actions before talking with you.

I am a little confused because I am able to login to Kolibri server as admin and move between screens and topics OK.

Thank you for your patience and support. Please advise next steps.

Kind regards,
David

10

Honestly, I don’t know what’s happening in your system now, there’s no reason for that. The backup command should be creating the folder if it doesn’t exist. (And it did exist in some previous screenshots you sent).

I only can do some guesses that might not be certain:

  • You had more than one instance of kolibri running. Maybe as another user? This would explain why you still can login into kolibri even if it does not start: i.e. you had it started.
  • You started the sqlite3 command without stopping kolibri (it does not look that from your logs)
  • your file system is failing, what would explain missing directories
  • your db was corrupted. I think you could try the other link I provided above

In all the cases, please, ensure kolibri is stopped before taking any action (using sudo systemctl stop kolibri or even being more radical to be sure you don’t have more instances running: sudo killall kolibri, you might need to do apt install psmisc if it complains that killall does not exist). If none of them work, you can always restore the previous db with:
cp db.sqlite3.bak db.sqlite3

Hope it helps
José

11

Hi @jredrejo

Thank you for the prompt response
The /home/kolibri/.kolibri/backups folder does indeed exist and no changes have been made to that folder or the contents

I logged into server again and began with your advice to stop the service and double check with killall - this is terminal

Last login: Mon Apr 8 23:34:42 2024 from 10.10.30.1
kolibri@kol-oa:~$ sudo systemctl stop kolibri
[sudo] password for kolibri:
kolibri@kol-oa:~$ sudo killall kolibri
kolibri: no process found
kolibri@kol-oa:~$ cd ~/.kolibri/
kolibri@kol-oa:~/.kolibri$ mkdir -p malformed
kolibri@kol-oa:~/.kolibri$ cp -b db.sqlite3* malformed/

kolibri@kol-oa:~/.kolibri$
kolibri@kol-oa:~/.kolibri$

From what I see, the killall reported that there were no other instances of kolibri running. But even after the killall command I am able to login and the server is working. I tried using a different browser with same results of being able to login to server. This is confusing to me.

I stopped my work after the cp -b db.sqlite3* malformed/ because I was unsure if I should have copied both db into the malformed directory

image

Should I only be dealing with the db.sqlite3 file and not copy the db.sqlite3.bak file as the advised commands do?

Can you please clarify if I need to run the malformed or the corrupted procedures from the link you provided?

Do you have other suggestions about how I can better understand why I can login to the server even after killall? Or am I confusing myself with being concerned about ability to login after killall?

Thank you,
David

12

This is actually very confusing.

Can you check if you’re using another method to start kolibri? maybe using uwsgi or gunicorn?
If you have some kind of custom installation most of what we’ve tested before is useless as kolibri has not been stopped.

If not I only can think of some kind of confussion between the server networking. You can check using
ifconfig (or ip a , depending on your linux version) that your browser is pointing exactly to this same server.

Regards
José

13

Hi @jredrejo
I am sorry for the confusing problem of being able to login after killall

results from ip a

kolibri@kol-oa:~$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether ae:4e:84:00:33:5e brd ff:ff:ff:ff:ff:ff
inet 10.10.30.6/24 brd 10.10.30.255 scope global ens18
valid_lft forever preferred_lft forever
inet6 fe80::ac4e:84ff:fe00:335e/64 scope link
valid_lft forever preferred_lft forever
kolibri@kol-oa:~$

I am fairly certain that this is normal install of kolibri because I do not know what uwgsi or gunicorn is.

If I shut down the VM, I cannot login to the server at 10.10.30.6:8080 - so I am sure this is correct server

When restarting the server the kolibri service is in loading state for almost 4 minutes. This is systemctl status while loading

kolibri@kol-oa:~$ systemctl status kolibri.service
● kolibri.service - LSB: kolibri daemon, an offline education platform
Loaded: loaded (/etc/init.d/kolibri; enabled; vendor preset: enabled)
Active: activating (start) since Tue 2024-04-09 02:49:04 WIT; 3min 48s ago
Cntrl PID: 835 (kolibri)
Tasks: 3 (limit: 19087)
Memory: 6.9G
CGroup: /system.slice/kolibri.service
├─835 /bin/sh /etc/init.d/kolibri start
├─864 runuser kolibri -c kolibri start
└─868 /usr/bin/python3 /bin/kolibri start

Apr 09 02:51:01 kol-oa kolibri[868]: No migrations to apply.
Apr 09 02:51:01 kol-oa kolibri[868]: Operations to perform:
Apr 09 02:51:01 kol-oa kolibri[868]: Apply all migrations: admin, analytics, >
Apr 09 02:51:01 kol-oa kolibri[868]: Running migrations:
Apr 09 02:51:01 kol-oa kolibri[868]: No migrations to apply.
Apr 09 02:52:41 kol-oa kolibri[868]: Operations to perform:
Apr 09 02:52:41 kol-oa kolibri[868]: Apply all migrations: admin, analytics, >
Apr 09 02:52:41 kol-oa kolibri[868]: Running migrations:
Apr 09 02:52:41 kol-oa kolibri[868]: No migrations to apply.
Apr 09 02:52:41 kol-oa kolibri[868]: Installed 2 object(s) from 1 fixture(s)

Then shortly thereafter fails and this is systemctl status

kolibri@kol-oa:~$ systemctl status kolibri.service
● kolibri.service - LSB: kolibri daemon, an offline education platform
Loaded: loaded (/etc/init.d/kolibri; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Tue 2024-04-09 02:53:01 WIT; 55s ago
Process: 835 ExecStart=/etc/init.d/kolibri start (code=exited, status=1/FAILURE)

Apr 09 02:53:00 kol-oa kolibri[868]: raise value.with_traceback(tb)
Apr 09 02:53:00 kol-oa kolibri[868]: File “/usr/lib/python3/dist-packages/kolibri/dist/django/db/backends/utils.py”, line >
Apr 09 02:53:00 kol-oa kolibri[868]: return self.cursor.execute(sql, params)
Apr 09 02:53:00 kol-oa kolibri[868]: File “/usr/lib/python3/dist-packages/kolibri/dist/django/db/backends/sqlite3/base.py”>
Apr 09 02:53:00 kol-oa kolibri[868]: return Database.Cursor.execute(self, query, params)
Apr 09 02:53:00 kol-oa kolibri[868]: django.db.utils.IntegrityError: NOT NULL constraint failed: logger_attemptlog.sessionlo>
Apr 09 02:53:01 kol-oa runuser[864]: pam_unix(runuser:session): session closed for user kolibri
Apr 09 02:53:01 kol-oa systemd[1]: kolibri.service: Control process exited, code=exited, status=1/FAILURE
Apr 09 02:53:01 kol-oa systemd[1]: kolibri.service: Failed with result ‘exit-code’.
Apr 09 02:53:01 kol-oa systemd[1]: Failed to start LSB: kolibri daemon, an offline education platform.
lines 1-15/15 (END)

What I am wondering now is perhaps this server always has failed at loading service and I never have checked the status of systemctl.

What I know is the IP address is correct and there are no other servers on the network at the 10.10.30.6:8080 IP address because if I shut down VM, I can no longer login

As the server stands now, even with the killall and still logging in confusion and the failed service load – the server is working.

I am leaning towards leaving the server as is and ask the local tech and teachers to put the server through its paces tomorrow (this evening our time).

Any thoughts or further advice?

Kind regards,
David

14

If you could still use the server with the database you had after doing vacuum, I would recommend you to use that one. It should have better perfomance.

Yes, maybe , in any case the IntegrityError I see in your logs it’s a signal of some bad state in the database. I’d investigate it more after you test the server tomorrow.

On the other hand, related to the beginning of this thread, release of 0.16.1 is being done tomorrow. It’ll be likely available in the servers to be upgraded tomorrow if there are no problems in the deployment.

Regards
José

15

Hi @jredrejo
I appreciate your input on trying the server “as-is” with the database after the successful vacuum

I will ask the local tech to perform some heavy testing and to also try some of the “pre-vacuum” trouble accounts and give a report of results.

Thank you for your time and assistance. I appreciate your clear responses and solid advice.

Kind regards,
David