Question on error report at end of 0.15 (server) upgrade - not a trouble issue

Support Kolibri
1

Hello and Merry Christmas - Thank you for your hard work on 0.15
Upgrade was successful but had error? reported on job for nginx.service failed

Kolibri is up and running, so perhaps no issue. Below is Kolibri system info and the error reports

ObAnggen server - Bokondini, Papua, Indonesia
Version: 0.15.0
OS: Linux-5.4.0-91-generic-x86_64-with-glibc2.29
Python: 3.8.10
Installer: deb kolibri-server - 0.4.0-0ubuntu2
Server: nginx/1.18.0 (Ubuntu)
Database: /home/kolibri/.kolibri/db.sqlite3
Free disk space: 312 GB
Server time: Thu Dec 23 2021 21:06:55 GMT+0900 (Japan Standard Time)
Server timezone: Asia/Jayapura
Device ID: 1a7cfa3ad9fc1886fccc88287c50efd4

Putty upgrade info
kolibri@kol-oa:~$ sudo apt upgrade
Reading package lists… Done
Building dependency tree
Reading state information… Done
Calculating upgrade… Done
The following packages will be upgraded:
kolibri kolibri-server
2 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 58.9 MB of archives.
After this operation, 45.4 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 Index of /learningequality/kolibri/ubuntu focal/main amd64 kolibri all 0.15.0-0ubuntu1 [58.9 MB]
Get:2 Index of /learningequality/kolibri/ubuntu focal/main amd64 kolibri-server all 0.4.0-0ubuntu2 [22.5 kB]
Fetched 58.9 MB in 5min 3s (194 kB/s)
Preconfiguring packages …
(Reading database … 122180 files and directories currently installed.)
Preparing to unpack …/kolibri_0.15.0-0ubuntu1_all.deb …
Unpacking kolibri (0.15.0-0ubuntu1) over (0.14.7-0ubuntu1) …
Preparing to unpack …/kolibri-server_0.4.0-0ubuntu2_all.deb …
Unpacking kolibri-server (0.4.0-0ubuntu2) over (0.3.8-0ubuntu1) …
Setting up kolibri (0.15.0-0ubuntu1) …
Installing new version of config file /etc/default/kolibri …
Synchronizing state of kolibri.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable kolibri
Job for kolibri.service failed because the control process exited with error code.
See “systemctl status kolibri.service” and “journalctl -xe” for details.
Setting up kolibri-server (0.4.0-0ubuntu2) …
Installing new version of config file /etc/init.d/kolibri-server …
Installing new version of config file /etc/kolibri/dist/nginx.conf …
WARNING Option CHERRYPY_START in section [Server] is deprecated, please remove it from your options.ini file
WARNING Ignoring unknown section in options file /home/kolibri/.kolibri/options.ini under top level: HTML5.
WARNING Option CHERRYPY_START in section [Server] is deprecated, please remove it from your options.ini file
WARNING Ignoring unknown section in options file /home/kolibri/.kolibri/options.ini under top level: HTML5.
WARNING Options file /home/kolibri/.kolibri/options.ini has been updated; server restart is required before change will take effect.
WARNING Option CHERRYPY_START in section [Server] is deprecated, please remove it from your options.ini file
WARNING Options file /home/kolibri/.kolibri/options.ini has been updated; server restart is required before change will take effect.
Job for nginx.service failed.
See “systemctl status nginx.service” and “journalctl -xe” for details.
Processing triggers for systemd (245.4-4ubuntu3.13) …
kolibri@kol-oa:~$

systemctl status nginx.service info
Active: active (running) since Sat 2021-12-18 20:31:52 WIT; 5 days ago
Docs: man:nginx(8)
Main PID: 760 (nginx)
Tasks: 6 (limit: 19110)
Memory: 140.2M
CGroup: /system.slice/nginx.service
├─ 760 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
├─11327 nginx: worker process
├─11328 nginx: worker process
├─11329 nginx: worker process
├─11330 nginx: worker process
└─11331 nginx: cache manager process
Dec 18 20:31:54 kol-oa systemd[1]: Reloading A high performance web server and a reverse proxy server.
Dec 18 20:31:54 kol-oa systemd[1]: Reloaded A high performance web server and a reverse proxy server.
Dec 23 20:57:59 kol-oa systemd[1]: Reloading A high performance web server and a reverse proxy server.
Dec 23 20:57:59 kol-oa nginx[11232]: nginx: [emerg] “location” directive is not allowed here in /home/kolibri/.kolibri/nginx>
Dec 23 20:57:59 kol-oa systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE
Dec 23 20:57:59 kol-oa systemd[1]: Reload failed for A high performance web server and a reverse proxy server.
Dec 23 20:58:01 kol-oa systemd[1]: Reloading A high performance web server and a reverse proxy server.
Dec 23 20:58:01 kol-oa systemd[1]: Reloaded A high performance web server and a reverse proxy server.
Dec 23 20:58:01 kol-oa systemd[1]: Reloading A high performance web server and a reverse proxy server.
Dec 23 20:58:01 kol-oa systemd[1]: Reloaded A high performance web server and a reverse proxy server.

journalctl -xe info
Dec 23 20:58:51 kol-oa kolibri-server[11309]: initialize(**get_initialize_params())
Dec 23 20:58:51 kol-oa kolibri-server[11309]: File “/usr/lib/python3/dist-packages/kolibri/utils/main.py”, line 289, in in>
Dec 23 20:58:51 kol-oa kolibri-server[11309]: update(version, kolibri.version)
Dec 23 20:58:51 kol-oa kolibri-server[11309]: File “/usr/lib/python3/dist-packages/kolibri/utils/main.py”, line 347, in up>
Dec 23 20:58:51 kol-oa kolibri-server[11309]: run_upgrades(old_version, new_version)
Dec 23 20:58:51 kol-oa kolibri-server[11309]: File “/usr/lib/python3/dist-packages/kolibri/core/upgrade.py”, line 156, in >
Dec 23 20:58:51 kol-oa kolibri-server[11309]: version_upgrade()
Dec 23 20:58:51 kol-oa kolibri-server[11309]: File “/usr/lib/python3/dist-packages/kolibri/core/upgrade.py”, line 51, in _>
Dec 23 20:58:51 kol-oa kolibri-server[11309]: return self.upgrade()
Dec 23 20:58:51 kol-oa kolibri-server[11309]: File “/usr/lib/python3/dist-packages/kolibri/core/logger/upgrade.py”, line 1>
Dec 23 20:58:51 kol-oa kolibri-server[11309]: migrate_from_exam_logs(ExamLog.objects.all())
Dec 23 20:58:51 kol-oa kolibri-server[11309]: File "/usr/lib/python3/dist-packages/kolibri/core/logger/utils/exam_log_migr>
Dec 23 20:58:51 kol-oa kolibri-server[11309]: _bulk_create(ContentSessionLog, compress(content_session_logs, mask))
Dec 23 20:58:51 kol-oa kolibri-server[11309]: File "/usr/lib/python3/dist-packages/kolibri/core/logger/utils/exam_log_migr>
Dec 23 20:58:51 kol-oa kolibri-server[11309]: return LogModel.objects.bulk_create(logs, batch_size=batch_size)
Dec 23 20:58:51 kol-oa kolibri-server[11309]: File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/models/manager.p>
Dec 23 20:58:51 kol-oa kolibri-server[11309]: return getattr(self.get_queryset(), name)(*args, **kwargs)
Dec 23 20:58:51 kol-oa kolibri-server[11309]: File “/usr/lib/python3/dist-packages/kolibri/dist/django/db/models/query.py”>
Dec 23 20:58:51 kol-oa kolibri-server[11309]: self._batched_insert(objs_with_pk, fields, batch_size)
Dec 23 20:58:51 kol-oa kolibri-server[11309]: File “/usr/lib/python3/dist-packages/kolibri/dist/django/db/models/query.py”>
Dec 23 20:58:51 kol-oa kolibri-server[11309]: self._insert(item, fields=fields, using=self.db)
Dec 23 20:58:51 kol-oa kolibri-server[11309]: File “/usr/lib/python3/dist-packages/kolibri/dist/django/db/models/query.py”>
Dec 23 20:58:51 kol-oa kolibri-server[11309]: return query.get_compiler(using=using).execute_sql(return_id)
Dec 23 20:58:51 kol-oa kolibri-server[11309]: File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/models/sql/compi>
Dec 23 20:58:51 kol-oa kolibri-server[11309]: cursor.execute(sql, params)
Dec 23 20:58:51 kol-oa kolibri-server[11309]: File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/backends/utils.p>
Dec 23 20:58:51 kol-oa kolibri-server[11309]: return self.cursor.execute(sql, params)
Dec 23 20:58:51 kol-oa kolibri-server[11309]: File “/usr/lib/python3/dist-packages/kolibri/dist/django/db/utils.py”, line >
Dec 23 20:58:51 kol-oa kolibri-server[11309]: six.reraise(dj_exc_type, dj_exc_value, traceback)
Dec 23 20:58:51 kol-oa kolibri-server[11309]: File “/usr/lib/python3/dist-packages/kolibri/dist/django/utils/six.py”, line>
Dec 23 20:58:51 kol-oa kolibri-server[11309]: raise value.with_traceback(tb)
Dec 23 20:58:51 kol-oa kolibri-server[11309]: File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/backends/utils.p>
Dec 23 20:58:51 kol-oa kolibri-server[11309]: return self.cursor.execute(sql, params)
Dec 23 20:58:51 kol-oa kolibri-server[11309]: File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/backends/sqlite3>
Dec 23 20:58:51 kol-oa kolibri-server[11309]: return Database.Cursor.execute(self, query, params)
Dec 23 20:58:51 kol-oa kolibri-server[11309]: django.db.utils.OperationalError: too many terms in compound SELECT
Dec 23 21:03:44 kol-oa PackageKit[11432]: daemon quit
Dec 23 21:03:44 kol-oa systemd[1]: packagekit.service: Succeeded.
– Subject: Unit succeeded
– Defined-By: systemd
– Support: Enterprise open source support | Ubuntu
–
– The unit packagekit.service has successfully entered the ‘dead’ state.

Please enjoy your holiday time - the Kolibri server is running and online - the school is on break until Jan 14 and I believe this is not a trouble issue.

My question is do I need to be concerned or take any action about this error during upgrade?

Kind regards,
David

2

hello @mrdavidhaag thanks for your report.

Looking at the logs you sent it seems some of the database migration had problems. I can’t be completely sure of which of them was but it looks like it was in the the logs of the exams, what means that some of the reports logs of the previous exams can have some missed data.

For the server if systemctl status kolibri-server returns an ok response, it’s all right.

In any case if you have a chance of sending me in DM the old database that should be in your ./kolibri/backups directory I can try to replicate the problem and find out what might have caused the migration problems (as you mentioned in your message it’s a holiday season so don’t expect to receive a quick answer but you’ll get it)

Regards
José

3

Hello Jose,
Thank you for the quick response.

The systemctl status kolibri-server returned

kolibri@kol-oa:~$ sudo systemctl status kolibri-server
● kolibri-server.service - A high performance web server setup for Kolibri
Loaded: loaded (/lib/systemd/system/kolibri-server.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2021-12-23 20:58:01 WIT; 10h ago
Tasks: 6 (limit: 19110)
Memory: 2.3G
CGroup: /system.slice/kolibri-server.service
├─11316 /usr/bin/uwsgi --ini /etc/kolibri/dist/uwsgi.ini --uid=kolibri --gid=1000 --env=KOLIBRI_HOME=/home/koli>
├─11319 /usr/bin/uwsgi --ini /etc/kolibri/dist/hashi_uwsgi.ini --uid=kolibri --gid=1000 --env=KOLIBRI_HOME=/hom>
├─12403 /usr/bin/uwsgi --ini /etc/kolibri/dist/hashi_uwsgi.ini --uid=kolibri --gid=1000 --env=KOLIBRI_HOME=/hom>
├─12404 /usr/bin/uwsgi --ini /etc/kolibri/dist/hashi_uwsgi.ini --uid=kolibri --gid=1000 --env=KOLIBRI_HOME=/hom>
├─12405 /usr/bin/uwsgi --ini /etc/kolibri/dist/uwsgi.ini --uid=kolibri --gid=1000 --env=KOLIBRI_HOME=/home/koli>
└─12406 /usr/bin/uwsgi --ini /etc/kolibri/dist/uwsgi.ini --uid=kolibri --gid=1000 --env=KOLIBRI_HOME=/home/koli>
Dec 23 20:58:51 kol-oa kolibri-server[11309]: return self.cursor.execute(sql, params)
Dec 23 20:58:51 kol-oa kolibri-server[11309]: File “/usr/lib/python3/dist-packages/kolibri/dist/django/db/utils.py”, line >
Dec 23 20:58:51 kol-oa kolibri-server[11309]: six.reraise(dj_exc_type, dj_exc_value, traceback)
Dec 23 20:58:51 kol-oa kolibri-server[11309]: File “/usr/lib/python3/dist-packages/kolibri/dist/django/utils/six.py”, line>
Dec 23 20:58:51 kol-oa kolibri-server[11309]: raise value.with_traceback(tb)
Dec 23 20:58:51 kol-oa kolibri-server[11309]: File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/backends/utils.p>
Dec 23 20:58:51 kol-oa kolibri-server[11309]: return self.cursor.execute(sql, params)
Dec 23 20:58:51 kol-oa kolibri-server[11309]: File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/backends/sqlite3>
Dec 23 20:58:51 kol-oa kolibri-server[11309]: return Database.Cursor.execute(self, query, params)
Dec 23 20:58:51 kol-oa kolibri-server[11309]: django.db.utils.OperationalError: too many terms in compound SELECT

I checked the backups folder and there are two 0.14.7 databases that were saved a few minutes apart with slightly different file size.

OA-KOL-Backups

Will you need both db files? The server is on a Vsat and will take many hours to upload 2GB file.

Kind regards,
David

4

Hello Jose,
I performed a 0.15 update on a local VM with Kolibri server and had same nginx error. The upgrade log, server info, nginx status and journalctl is below.

Upgrade log

kolibri@kol-srv:~$ sudo apt upgrade
Reading package lists… Done
Building dependency tree
Reading state information… Done
Calculating upgrade… Done
The following packages will be upgraded:
kolibri kolibri-server
2 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 58.9 MB of archives.
After this operation, 45.4 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 Index of /learningequality/kolibri/ubuntu focal/main amd64 kolibri all 0.15.0-0ubuntu1 [58.9 MB]
Get:2 Index of /learningequality/kolibri/ubuntu focal/main amd64 kolibri-server all 0.4.0-0ubuntu2 [22.5 kB]
Fetched 58.9 MB in 6s (9187 kB/s)
Preconfiguring packages …
(Reading database … 122170 files and directories currently installed.)
Preparing to unpack …/kolibri_0.15.0-0ubuntu1_all.deb …
Unpacking kolibri (0.15.0-0ubuntu1) over (0.14.7-0ubuntu1) …
Preparing to unpack …/kolibri-server_0.4.0-0ubuntu2_all.deb …
Unpacking kolibri-server (0.4.0-0ubuntu2) over (0.3.8-0ubuntu1) …
Setting up kolibri (0.15.0-0ubuntu1) …
Installing new version of config file /etc/default/kolibri …
Synchronizing state of kolibri.service with SysV service script with /lib/system d/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable kolibri
Setting up kolibri-server (0.4.0-0ubuntu2) …
Installing new version of config file /etc/init.d/kolibri-server …
Installing new version of config file /etc/kolibri/dist/nginx.conf …
WARNING Option CHERRYPY_START in section [Server] is deprecated, please remove it from your options.ini file
WARNING Ignoring unknown section in options file /home/kolibri/.kolibri/options .ini under top level: HTML5.
WARNING Option CHERRYPY_START in section [Server] is deprecated, please remove it from your options.ini file
WARNING Ignoring unknown section in options file /home/kolibri/.kolibri/options .ini under top level: HTML5.
WARNING Options file /home/kolibri/.kolibri/options.ini has been updated; serve r restart is required before change will take effect.
WARNING Option CHERRYPY_START in section [Server] is deprecated, please remove it from your options.ini file
WARNING Options file /home/kolibri/.kolibri/options.ini has been updated; serve r restart is required before change will take effect.
Job for nginx.service failed.
See “systemctl status nginx.service” and “journalctl -xe” for details.
Processing triggers for systemd (245.4-4ubuntu3.13) …

Server info
Version: 0.15.0
OS: Linux-5.4.0-91-generic-x86_64-with-glibc2.29
Python: 3.8.10
Installer: deb kolibri-server - 0.4.0-0ubuntu2
Server: nginx/1.18.0 (Ubuntu)
Database: /home/kolibri/.kolibri/db.sqlite3
Free disk space: 392 GB
Server time: Mon Dec 27 2021 09:53:50 GMT-0500 (Eastern Standard Time)
Server timezone: Etc/UTC
Device ID: 3d938bfcc2e0d88860c4eb158da2fdd4

nginx status

kolibri@kol-srv:~$ systemctl status nginx.service
● nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2021-12-27 14:36:13 UTC; 22min ago
Docs: man:nginx(8)
Main PID: 696 (nginx)
Tasks: 6 (limit: 9447)
Memory: 20.2M
CGroup: /system.slice/nginx.service
├─ 696 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
├─2088 nginx: worker process
├─2089 nginx: worker process
├─2090 nginx: worker process
├─2091 nginx: worker process
└─2092 nginx: cache manager process
Dec 27 14:36:14 kol-srv systemd[1]: Reloading A high performance web server and a reverse proxy server.
Dec 27 14:36:14 kol-srv systemd[1]: Reloaded A high performance web server and a reverse proxy server.
Dec 27 14:49:04 kol-srv systemd[1]: Reloading A high performance web server and a reverse proxy server.
Dec 27 14:49:04 kol-srv nginx[2001]: nginx: [emerg] “location” directive is not allowed here in /home/kolibri/.kolib>
Dec 27 14:49:04 kol-srv systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE
Dec 27 14:49:04 kol-srv systemd[1]: Reload failed for A high performance web server and a reverse proxy server.
Dec 27 14:49:05 kol-srv systemd[1]: Reloading A high performance web server and a reverse proxy server.
Dec 27 14:49:05 kol-srv systemd[1]: Reloaded A high performance web server and a reverse proxy server.
Dec 27 14:49:05 kol-srv systemd[1]: Reloading A high performance web server and a reverse proxy server.
Dec 27 14:49:05 kol-srv systemd[1]: Reloaded A high performance web server and a reverse proxy server.

journalctl

kolibri@kol-srv:~$ journalctl -xe
Dec 27 14:49:07 kol-srv kolibri-server[2071]: WARNING Option CHERRYPY_START in section [Server] is deprecated, plea>
Dec 27 14:49:07 kol-srv kolibri-server[2071]: INFO Option DEBUG in section [Server] being overridden by environm>
Dec 27 14:49:07 kol-srv kolibri-server[2071]: INFO Option DEBUG_LOG_DATABASE in section [Server] being overridde>
Dec 27 14:49:07 kol-srv kolibri-server[2071]: /usr/lib/python3/dist-packages/kolibri/dist/rest_framework/utils/seria>
Dec 27 14:49:07 kol-srv kolibri-server[2071]: if value is None or value is ‘’:
Dec 27 14:49:07 kol-srv kolibri-server[2071]: INFO Running Kolibri with the following settings: kolibri.deployme>
Dec 27 14:49:07 kol-srv kolibri-server[2071]: INFO Importing ‘tasks’ module from django apps
Dec 27 14:49:07 kol-srv kolibri-server[2071]: INFO Starting Kolibri background services
Dec 27 14:49:07 kol-srv kolibri-server[2071]: INFO Starting Kolibri 0.15.0
Dec 27 14:49:07 kol-srv kolibri-server[2071]: INFO Bus state: ENTER
Dec 27 14:49:07 kol-srv kolibri-server[2071]: INFO Running Kolibri as background process
Dec 27 14:49:07 kol-srv kolibri-server[2071]: INFO Going to background mode, logging to /home/kolibri/.kolibri/l>
Dec 27 14:49:08 kol-srv sudo[1563]: pam_unix(sudo:session): session closed for user root
Dec 27 14:51:14 kol-srv systemd[1]: Starting Cleanup of Temporary Directories…
– Subject: A start job for unit systemd-tmpfiles-clean.service has begun execution
– Defined-By: systemd
– Support: Enterprise open source support | Ubuntu
–
– A start job for unit systemd-tmpfiles-clean.service has begun execution.
–
– The job identifier is 1055.
Dec 27 14:51:14 kol-srv systemd[1]: systemd-tmpfiles-clean.service: Succeeded.
– Subject: Unit succeeded
– Defined-By: systemd
– Support: Enterprise open source support | Ubuntu
–
– The unit systemd-tmpfiles-clean.service has successfully entered the ‘dead’ state.
Dec 27 14:51:14 kol-srv systemd[1]: Finished Cleanup of Temporary Directories.
– Subject: A start job for unit systemd-tmpfiles-clean.service has finished successfully
– Defined-By: systemd
– Support: Enterprise open source support | Ubuntu
–
– A start job for unit systemd-tmpfiles-clean.service has finished successfully.
–
– The job identifier is 1055.
Dec 27 14:54:19 kol-srv PackageKit[1526]: daemon quit
Dec 27 14:54:19 kol-srv systemd[1]: packagekit.service: Succeeded.
– Subject: Unit succeeded
– Defined-By: systemd
– Support: Enterprise open source support | Ubuntu
–
– The unit packagekit.service has successfully entered the ‘dead’ state.
Dec 27 14:55:13 kol-srv sshd[2267]: Accepted password for kolibri from 10.20.30.200 port 34856 ssh2
Dec 27 14:55:13 kol-srv sshd[2267]: pam_unix(sshd:session): session opened for user kolibri by (uid=0)
Dec 27 14:55:13 kol-srv systemd-logind[648]: New session 3 of user kolibri.
– Subject: A new session 3 has been created for user kolibri
– Defined-By: systemd
– Support: Enterprise open source support | Ubuntu
– Documentation: multiseat
–
– A new session with the ID 3 has been created for the user kolibri.
–
– The leading process of the session is 2267.
Dec 27 14:55:13 kol-srv systemd[1]: Started Session 3 of user kolibri.
– Subject: A start job for unit session-3.scope has finished successfully
– Defined-By: systemd
– Support: Enterprise open source support | Ubuntu
–
– A start job for unit session-3.scope has finished successfully.
–
– The job identifier is 1059.

Perhaps the nginx error is not limited to the ObAnggen database. This local Kolibri server has a very small database.

image

Please let me know if you would like more information or a copy of this database.

Kind regards,
David

5

yes @mrdavidhaag a copy of that db backup would be helpful, but please, send it to me via a DM and not in public in case it contains real students information.
Also, I’m on vacation these days, so I won’t be able to take a look at the db in the next days, but I’ll keep you reported when I look at it.

Regards
José

1 Like