Upgrade to 017.5 failed, content folder/database disappeared

If you have a specific technical inquiry, construct your post with the following:

Summary

I did an update via apt of one of our RPi400 systems. It went from 0.15… to 0.17.5. Kolibri was working fine before this.

Upon completion, Kolibri would not start. Here are the errors that came from checking status:

**pi@kolibri-madina**:**~ $** kolibri status

Stopped (1)

**pi@kolibri-madina**:**~ $** sudo systemctl status kolibri

**●** kolibri.service - LSB: kolibri daemon, an offline education platform

Loaded: loaded (/etc/init.d/kolibri; enabled; vendor preset: enabled)

Active: **failed** (Result: exit-code) since Mon 2025-01-13 21:13:11 WAT; 22s ago

Process: 749 ExecStart=/etc/init.d/kolibri start **(code=exited, status=1/FAILURE)**

CPU: 31.708s

Jan 13 21:12:57 kolibri-madina kolibri[789]: File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/backends/sqlite3/schema.py", line 35, in __exit>

Jan 13 21:12:57 kolibri-madina kolibri[789]: self.connection.check_constraints()

Jan 13 21:12:57 kolibri-madina kolibri[789]: File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/backends/sqlite3/base.py", line 353, in check_c>

Jan 13 21:12:57 kolibri-madina kolibri[789]: raise IntegrityError(

Jan 13 21:12:57 kolibri-madina kolibri[789]: django.db.utils.IntegrityError: The row in table 'content_channelmetadata_included_languages' with primary ke>

Jan 13 21:13:11 kolibri-madina runuser[787]: pam_unix(runuser:session): session closed for user kolibri

Jan 13 21:13:11 kolibri-madina systemd[1]: **kolibri.service: Control process exited, code=exited, status=1/FAILURE**

Jan 13 21:13:11 kolibri-madina systemd[1]: **kolibri.service: Failed with result 'exit-code'.**

Jan 13 21:13:11 kolibri-madina systemd[1]: **Failed to start LSB: kolibri daemon, an offline education platform.**

Jan 13 21:13:11 kolibri-madina systemd[1]: **kolibri.service: Consumed 31.708s CPU time.**

So, thinking to check out the database, I installed sqlite3 and went looking looking for the database. It no longer exists! I looked everywhere I could think of on the drive but no database, options.ini, nor content could be found.

So I tried to start Kolibri service again, and this is the outcome:

pi@kolibri-madina:~/.kolibri $ sudo systemctl start kolibri
Job for kolibri.service failed because the control process exited with error code.
See "systemctl status kolibri.service" and "journalctl -xe" for details.
pi@kolibri-madina:~/.kolibri $ journalctl -xe
Jan 14 19:51:03 kolibri-madina kolibri[3865]:   File "/usr/lib/python3/dist-packages/kolibri/dis>
Jan 14 19:51:03 kolibri-madina kolibri[3865]:     res = handle_func(*args, **kwargs)
Jan 14 19:51:03 kolibri-madina kolibri[3865]:   File "/usr/lib/python3/dist-packages/kolibri/dis>
Jan 14 19:51:03 kolibri-madina kolibri[3865]:     post_migrate_state = executor.migrate(
Jan 14 19:51:03 kolibri-madina kolibri[3865]:   File "/usr/lib/python3/dist-packages/kolibri/dis>
Jan 14 19:51:03 kolibri-madina kolibri[3865]:     state = self._migrate_all_forwards(state, plan>
Jan 14 19:51:03 kolibri-madina kolibri[3865]:   File "/usr/lib/python3/dist-packages/kolibri/dis>
Jan 14 19:51:03 kolibri-madina kolibri[3865]:     state = self.apply_migration(state, migration,>
Jan 14 19:51:03 kolibri-madina kolibri[3865]:   File "/usr/lib/python3/dist-packages/kolibri/dis>
Jan 14 19:51:03 kolibri-madina kolibri[3865]:     migration_recorded = True
Jan 14 19:51:03 kolibri-madina kolibri[3865]:   File "/usr/lib/python3/dist-packages/kolibri/dis>
Jan 14 19:51:03 kolibri-madina kolibri[3865]:     self.connection.check_constraints()
Jan 14 19:51:03 kolibri-madina kolibri[3865]:   File "/usr/lib/python3/dist-packages/kolibri/dis>
Jan 14 19:51:03 kolibri-madina kolibri[3865]:     raise IntegrityError(
Jan 14 19:51:03 kolibri-madina kolibri[3865]: django.db.utils.IntegrityError: The row in table '>
Jan 14 19:51:04 kolibri-madina runuser[3864]: pam_unix(runuser:session): session closed for user>
Jan 14 19:51:04 kolibri-madina systemd[1]: kolibri.service: Control process exited, code=exited,
lines 1-16/16 (END)

So, I thought to try to start Kolibri not as a service and here are the results:

pi@kolibri-madina:~/.kolibri $ kolibri start
INFO     2025-01-14 20:03:20,236 Option DEBUG in section [Server] being overridden by environment variable KOLIBRI_DEBUG
INFO     2025-01-14 20:03:20,237 Option DEBUG_LOG_DATABASE in section [Server] being overridden by environment variable KOLIBRI_DEBUG_LOG_DATABASE
INFO     2025-01-14 20:03:20,252 Option DEBUG in section [Server] being overridden by environment variable KOLIBRI_DEBUG
INFO     2025-01-14 20:03:20,253 Option DEBUG_LOG_DATABASE in section [Server] being overridden by environment variable KOLIBRI_DEBUG_LOG_DATABASE
INFO     2025-01-14 20:03:20,260 Attempting to setup using pre-migrated databases
INFO     2025-01-14 20:03:20,276 Copied preseeded database from /usr/lib/python3/dist-packages/kolibri/dist/home/db.sqlite3 to /home/pi/.kolibri/db.sqlite3
INFO     2025-01-14 20:03:20,278 Copied preseeded database from /usr/lib/python3/dist-packages/kolibri/dist/home/syncqueue.sqlite3 to /home/pi/.kolibri/syncqueue.sqlite3
INFO     2025-01-14 20:03:20,280 Copied preseeded database from /usr/lib/python3/dist-packages/kolibri/dist/home/networklocation.sqlite3 to /home/pi/.kolibri/networklocation.sqlite3
INFO     2025-01-14 20:03:20,282 Copied preseeded database from /usr/lib/python3/dist-packages/kolibri/dist/home/notifications.sqlite3 to /home/pi/.kolibri/notifications.sqlite3
INFO     2025-01-14 20:03:22,148 Running Kolibri with the following settings: kolibri.deployment.default.settings.base
INFO     2025-01-14 20:03:23,369 New install, version: 0.17.5
INFO     2025-01-14 20:03:23,369 Running update routines for new version...
Operations to perform:
  Apply all migrations: analytics, auth, bookmarks, content, contenttypes, device, discovery, exams, kolibriauth, lessons, logger, morango, notifications, sessions
Running migrations:
  No migrations to apply.
Operations to perform:
  Apply all migrations: analytics, auth, bookmarks, content, contenttypes, device, discovery, exams, kolibriauth, lessons, logger, morango, notifications, sessions
Running migrations:
  No migrations to apply.
Operations to perform:
  Apply all migrations: analytics, auth, bookmarks, content, contenttypes, device, discovery, exams, kolibriauth, lessons, logger, morango, notifications, sessions
Running migrations:
  No migrations to apply.
Operations to perform:
  Apply all migrations: analytics, auth, bookmarks, content, contenttypes, device, discovery, exams, kolibriauth, lessons, logger, morango, notifications, sessions
Running migrations:
  No migrations to apply.
Installed 2 object(s) from 1 fixture(s)
INFO     2025-01-14 20:03:27,356 Detected updates to plugins: kolibri.plugins.pdf_viewer, kolibri.plugins.media_player, kolibri.plugins.user_profile, kolibri.plugins.epub_viewer, kolibri.plugins.facility, kolibri.plugins.default_theme, kolibri.plugins.user_auth, kolibri.plugins.html5_viewer, kolibri.plugins.device, kolibri.plugins.learn, kolibri.plugins.slideshow_viewer, kolibri.plugins.setup_wizard, kolibri.plugins.perseus_viewer, kolibri.plugins.coach
INFO     2025-01-14 20:03:27,392 Running installation routines for kolibri.plugins.pdf_viewer, installing 0.17.5
INFO     2025-01-14 20:03:27,392 kolibri.plugins.pdf_viewer successfully updated
INFO     2025-01-14 20:03:27,394 Running installation routines for kolibri.plugins.media_player, installing 0.17.5
INFO     2025-01-14 20:03:27,394 kolibri.plugins.media_player successfully updated
INFO     2025-01-14 20:03:27,396 Running installation routines for kolibri.plugins.user_profile, installing 0.17.5
INFO     2025-01-14 20:03:27,396 kolibri.plugins.user_profile successfully updated
INFO     2025-01-14 20:03:27,397 Running installation routines for kolibri.plugins.epub_viewer, installing 0.17.5
INFO     2025-01-14 20:03:27,398 kolibri.plugins.epub_viewer successfully updated
INFO     2025-01-14 20:03:27,399 Running installation routines for kolibri.plugins.facility, installing 0.17.5
INFO     2025-01-14 20:03:27,400 kolibri.plugins.facility successfully updated
INFO     2025-01-14 20:03:27,401 Running installation routines for kolibri.plugins.default_theme, installing 0.17.5
INFO     2025-01-14 20:03:27,402 kolibri.plugins.default_theme successfully updated
INFO     2025-01-14 20:03:27,403 Running installation routines for kolibri.plugins.user_auth, installing 0.17.5
INFO     2025-01-14 20:03:27,404 kolibri.plugins.user_auth successfully updated
INFO     2025-01-14 20:03:27,405 Running installation routines for kolibri.plugins.html5_viewer, installing 0.17.5
INFO     2025-01-14 20:03:27,405 kolibri.plugins.html5_viewer successfully updated
INFO     2025-01-14 20:03:27,407 Running installation routines for kolibri.plugins.device, installing 0.17.5
INFO     2025-01-14 20:03:27,407 kolibri.plugins.device successfully updated
INFO     2025-01-14 20:03:27,409 Running installation routines for kolibri.plugins.learn, installing 0.17.5
INFO     2025-01-14 20:03:27,409 kolibri.plugins.learn successfully updated
INFO     2025-01-14 20:03:27,410 Running installation routines for kolibri.plugins.slideshow_viewer, installing 0.17.5
INFO     2025-01-14 20:03:27,411 kolibri.plugins.slideshow_viewer successfully updated
INFO     2025-01-14 20:03:27,412 Running installation routines for kolibri.plugins.setup_wizard, installing 0.17.5
INFO     2025-01-14 20:03:27,413 kolibri.plugins.setup_wizard successfully updated
INFO     2025-01-14 20:03:27,414 Running installation routines for kolibri.plugins.perseus_viewer, installing 0.17.5
INFO     2025-01-14 20:03:27,414 kolibri.plugins.perseus_viewer successfully updated
INFO     2025-01-14 20:03:27,416 Running installation routines for kolibri.plugins.coach, installing 0.17.5
INFO     2025-01-14 20:03:27,416 kolibri.plugins.coach successfully updated
INFO     2025-01-14 20:03:27,507 Starting Kolibri 0.17.5
INFO     2025-01-14 20:03:27,812 Bus state: ENTER
INFO     2025-01-14 20:03:27,813 Running Kolibri as background process
INFO     2025-01-14 20:03:27,816 Kolibri running on: http://10.0.2.209:8080/
INFO     2025-01-14 20:03:27,816 Kolibri running on: http://127.0.0.1:8080/
INFO     2025-01-14 20:03:27,817 Kolibri running on: http://10.10.10.1:8080/
INFO     2025-01-14 20:03:27,817 Going to background mode, logging to /home/pi/.kolibri/logs/kolibri.txt
pi@kolibri-madina:~/.kolibri $ kolibri status
INFO     2025-01-14 20:03:50,172 Option DEBUG in section [Server] being overridden by environment variable KOLIBRI_DEBUG
INFO     2025-01-14 20:03:50,174 Option DEBUG_LOG_DATABASE in section [Server] being overridden by environment variable KOLIBRI_DEBUG_LOG_DATABASE
OK, running (0)
Kolibri running on:

	http://10.0.2.209:8080/
	http://127.0.0.1:8080/
	http://10.10.10.1:8080/

Following this, I successfully accessed Kolibri, but it showed up as a new install. Looking in the ~/.kolibri folder I found that it had been populated with all of what was missing, but as a new installation. So, the contents that previously existed disappeared. Based on the info on the drive, there is still many GB of data there, but not visible.

Technical details

  • 017.5
  • Debian Raspian OS 11.11
  • Browser

Hi @cjackson,

By running kolibri start instead of systemctl start kolibri, it will run Kolibri as the currently logged in user, which may not be the system user the Kolibri service installation uses. Since Kolibri stores its databases and configuration in a directory that is dependent on the system user’s home directory location, this would cause the effect you described-- a completely new installation of Kolibri running.

My recommendation is to first check that /etc/kolibri/username exists, which I believe it should. If it does, please follow these instructions:

  1. Stop the Kolibri instance running as the pi user: kolibri stop
  2. Ensure the system installation is stopped: sudo systemctl stop kolibri
  3. Change to the user the system installation uses: sudo su $(cat /etc/kolibri/username)
  4. Manually run the migrations: kolibri manage migrate
  5. Log out of the system user: exit
  6. Start the system installation: sudo systemctl start kolibri

Regards,
Blaine

Thank you for your response and explanation. In fact there was that very directory you mentioned. I did as you suggested with errors as follows below.

After completing steps 1-3, step 4 resulted in:

pi@kolibri-madina:~ $ sudo su $(cat /etc/kolibri/username)
kolibri@kolibri-madina:/home/pi$ kolibri manage migrate
INFO     2025-01-15 19:59:56,924 Option DEBUG in section [Server] being overridden by environment variable KOLIBRI_DEBUG
INFO     2025-01-15 19:59:56,925 Option DEBUG_LOG_DATABASE in section [Server] being overridden by environment variable KOLIBRI_DEBUG_LOG_DATABASE
WARNING  2025-01-15 20:00:00,715 Foreign key constraint failed. Trying to fix integrity errors...
ERROR    2025-01-15 20:00:00,716 Foreign key constraint failed in content_channelmetadata_included_languages table, rowid 17. Please fix it manually.
INFO     2025-01-15 20:00:02,461 Running Kolibri with the following settings: kolibri.deployment.default.settings.base
INFO     2025-01-15 20:00:14,858 Backed up database to: /var/kolibri/.kolibri/backups/db-v0.15.12_2025-01-15_20-00-03.dump
INFO     2025-01-15 20:00:14,859 Version was 0.15.12, new version: 0.17.5
INFO     2025-01-15 20:00:14,860 Running update routines for new version...
Operations to perform:
  Apply all migrations: analytics, auth, bookmarks, content, contenttypes, device, discovery, exams, kolibriauth, lessons, logger, morango, notifications, sessions
Running migrations:
  Applying auth.0009_alter_user_last_name_max_length...Error: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/kolibri/utils/cli.py", line 189, in invoke
    initialize(**get_initialize_params())
  File "/usr/lib/python3/dist-packages/kolibri/utils/main.py", line 346, in initialize
    update(version, kolibri.__version__)
  File "/usr/lib/python3/dist-packages/kolibri/utils/main.py", line 400, in update
    _migrate_databases()
  File "/usr/lib/python3/dist-packages/kolibri/utils/main.py", line 109, in _migrate_databases
    call_command("migrate", interactive=False, database=database)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/core/management/__init__.py", line 181, in call_command
    return command.execute(*args, **defaults)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/core/management/base.py", line 89, in wrapped
    res = handle_func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/core/management/commands/migrate.py", line 244, in handle
    post_migrate_state = executor.migrate(
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/migrations/executor.py", line 117, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/migrations/executor.py", line 230, in apply_migration
    migration_recorded = True
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/backends/sqlite3/schema.py", line 35, in __exit__
    self.connection.check_constraints()
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/backends/sqlite3/base.py", line 353, in check_constraints
    raise IntegrityError(
django.db.utils.IntegrityError: The row in table 'content_channelmetadata_included_languages' with primary key '17' has an invalid foreign key: content_channelmetadata_included_languages.channelmetadata_id contains a value 'eabc76dbfc5f4d77a6ec9812d4ed1993' that does not have a corresponding value in content_channelmetadata.id.

kolibri@kolibri-madina:/home/pi$ exit
exit

I went ahead and started the Kolibri service with the outcome as:

pi@kolibri-madina:~ $ sudo systemctl start kolibri
Job for kolibri.service failed because the control process exited with error code.
See "systemctl status kolibri.service" and "journalctl -xe" for details.

Looking into the two sources recommended for additional information:

pi@kolibri-madina:~ $ systemctl status kolibri.service
● kolibri.service - LSB: kolibri daemon, an offline education platform
     Loaded: loaded (/etc/init.d/kolibri; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Wed 2025-01-15 20:01:33 WAT; 20s ago
    Process: 2745 ExecStart=/etc/init.d/kolibri start (code=exited, status=1/FAILURE)
        CPU: 27.429s

Jan 15 20:01:33 kolibri-madina kolibri[2758]:   File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/backends/sqlite3/schema.py", line 35, in __exit__
Jan 15 20:01:33 kolibri-madina kolibri[2758]:     self.connection.check_constraints()
Jan 15 20:01:33 kolibri-madina kolibri[2758]:   File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/backends/sqlite3/base.py", line 353, in check_con>
Jan 15 20:01:33 kolibri-madina kolibri[2758]:     raise IntegrityError(
Jan 15 20:01:33 kolibri-madina kolibri[2758]: django.db.utils.IntegrityError: The row in table 'content_channelmetadata_included_languages' with primary key >
Jan 15 20:01:33 kolibri-madina runuser[2757]: pam_unix(runuser:session): session closed for user kolibri
Jan 15 20:01:33 kolibri-madina systemd[1]: kolibri.service: Control process exited, code=exited, status=1/FAILURE
Jan 15 20:01:33 kolibri-madina systemd[1]: kolibri.service: Failed with result 'exit-code'.
Jan 15 20:01:33 kolibri-madina systemd[1]: Failed to start LSB: kolibri daemon, an offline education platform.
Jan 15 20:01:33 kolibri-madina systemd[1]: kolibri.service: Consumed 27.429s CPU time.

pi@kolibri-madina:~ $ journalctl -xe
░░ Support: https://www.debian.org/support
░░ 
░░ The unit kolibri.service has entered the 'failed' state with result 'exit-code'.
Jan 15 20:01:33 kolibri-madina systemd[1]: Failed to start LSB: kolibri daemon, an offline education platform.
░░ Subject: A start job for unit kolibri.service has failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A start job for unit kolibri.service has finished with a failure.
░░ 
░░ The job identifier is 1823 and the job result is failed.
Jan 15 20:01:33 kolibri-madina systemd[1]: kolibri.service: Consumed 27.429s CPU time.
░░ Subject: Resources consumed by unit runtime
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ The unit kolibri.service completed and consumed the indicated resources.
Jan 15 20:01:33 kolibri-madina sudo[2742]: pam_unix(sudo:session): session closed for user root

Hi @cjackson,

Okay looks like Kolibri was updated okay, but there’s a database integrity error preventing startup.

Here are some next steps:

  1. Again, ensure the system installation is stopped: sudo systemctl stop kolibri
  2. Change to the kolibri user: sudo su $(cat /etc/kolibri/username)
  3. Make a backup of the sqlite database just in case: cp ~/.kolibri/db.sqlite3 ~/.kolibri/db.sqlite3.backup
  4. Open the database with sqlite, which will provide you with a shell (install sqlite3 if command does not exist): sqlite3 ~/.kolibri/db.sqlite3
  5. Run the following SQL in the sqlite shell:
DELETE FROM content_channelmetadata_included_languages AS ccil WHERE NOT EXISTS (SELECT 1 FROM content_channelmetadata AS cc WHERE cc.id = ccil.channelmetadata_id);
  1. Exit the sqlite shell: .exit
  2. Log out of the kolibri user: exit
  3. Try to start the system installation again: sudo systemctl start kolibri

If Kolibri now starts up, you can delete the database backup.

Regards,
Blaine

@blaine ,

Thank you again for your help on this. That fixed the problem! I didn’t look in the /var directory to see if the original content database was there. In fact, I didn’t know that is where all of the Kolibri files and data are kept when running it as a service.

I assume, then, that the files created in the /home/pi/.kolibri directory which were created when I mistakenly chose to run Kolibri as a non-service are not needed. Is that correct? If so, can I remove them from the drive?

Yes, that Kolibri home directory was likely created when you ran kolibri start on its own as the pi user, and shouldn’t be needed.

Glad your issue is resolved!