AWS hosting recommendation

Brian_LePore · September 30, 2022, 8:03pm

We were looking to run Kolibri in AWS for an organization. We were looking to ensure that this software could scale and were looking for recommendation on how best to set this up such as setting this up with an Auto Scaling Group.

Should the entire Kolibri directory (or just the content/database folder) reside on a EFS (a network file system) so that each server would have access to the same users and content.

I do see that Kolibri can handle Redis which could be used for scaling.

richard · October 2, 2022, 9:08pm

Hi @Brian_LePore,

Kolibri does also support PostgreSQL as a database backend, so that may be worth considering. As you note, you will still need to share a network file system for any instances of the application that will be serving the resource files.

Kind Regards,
Richard

giakonda · October 3, 2022, 2:50pm

I would think it’s easier and a lot cheaper to use a Synology server!

Brian_LePore · October 3, 2022, 7:40pm

So because I destroyed something when I was disconnected while running “kolibri manage content movedirectory” and got disconnected partway through and think I lost data when I went to run it again so I decided to start over from scratch with a fresh ubuntu 22.04.1 LTS. Installed Kolibri, configured a new PostgreSQL RDS instance and used pgloader to migrate the data and was able to see data in the database. Verified from the command line the server could access the database so I thought it was good.

I then edited the options.ini file and restarted kolibri and I’m now getting:

ERROR 2022-10-03 19:09:58,952 root Tried to check that the database was accessible and an error occurred: Not able to access the database while checking it.

Exception: database connection isn't set to UTC

I’m not seeing anything in Kolibri config for setting the DB connection. Am I missing something?

richard · October 3, 2022, 7:45pm

Best to downgrade to psycopg2 == 2.7.5: kolibri/postgres.txt at release-v0.15.x · learningequality/kolibri · GitHub

It may work up to 2.8.6, but 2.7.5 is what we run tests with.

Brian_LePore · October 3, 2022, 8:23pm

I’m having trouble downgrading that far.

Is there a version of Ubuntu that this recommended with? I saw previously that it said 16+ but this is seeming to be problematic and I’d like to get things more in line with what is tested.

richard · October 3, 2022, 9:14pm

I think psycopg2-binary==2.8.6 should be the safest bet - seems to be after this that the UTC issues appear and probably need a newer Django version to resolve.

Brian_LePore · October 3, 2022, 9:41pm

That seemed to just cause another error …

INFO     Bus state: ENTER
INFO     Running Kolibri as background process
INFO     Going to background mode, logging to /var/kolibri/.kolibri/logs/kolibri.txt
ERROR    Error in 'ENTER' listener <bound method DaemonizePlugin.ENTER of <kolibri.utils.server.DaemonizePlugin object at 0x7fc555dfca00>>
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/kolibri/dist/magicbus/base.py", line 273, in publish
    result = listener(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/utils/server.py", line 362, in ENTER
    become_daemon(**kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/utils/system.py", line 184, in become_daemon
    _become_daemon_function(**kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/utils/system.py", line 76, in _posix_become_daemon
    os.chdir(our_home_dir)
PermissionError: [Errno 13] Permission denied: '.'

INFO     Listening for SIGTERM.
INFO     Listening for SIGHUP.
INFO     Listening for SIGUSR1.
INFO     Listening for SIGINT.
INFO     Bus state: STOP_ERROR
ERROR    Exiting due to error in stop listener:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/kolibri/dist/magicbus/base.py", line 215, in _transition
    return self.publish(newstate, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/dist/magicbus/base.py", line 291, in publish
    raise exc
magicbus.base.ChannelFailures: PermissionError(13, 'Permission denied')

INFO     Bus state: EXIT
INFO     Waiting for child threads to terminate...
INFO     Bus state: EXITED

richard · October 3, 2022, 9:44pm

Seems like it’s a folder permissions error in os.chdir? Does kolibri have appropriate permissions?

Brian_LePore · October 3, 2022, 10:00pm

Where would os.chdir reside in this context?

Brian_LePore · October 3, 2022, 10:04pm

INFO 2022-10-03 22:02:06,464 kolibri.utils.server Bus state: STOP
INFO 2022-10-03 22:02:06,478 kolibri.utils.server HTTP Server kolibri.utils.server.Server(('0.0.0.0', 36741)) shut down
INFO 2022-10-03 22:02:06,490 kolibri.utils.server HTTP Server kolibri.utils.server.Server(('0.0.0.0', 8080)) shut down
INFO 2022-10-03 22:02:06,490 kolibri.utils.server Joining 'ProcessControlPlugin'
INFO 2022-10-03 22:02:06,856 kolibri.utils.server Stopped thread 'ProcessControlPlugin'.
INFO 2022-10-03 22:02:06,857 kolibri.utils.server Joining 'ZeroConfPlugin'
INFO 2022-10-03 22:02:07,855 kolibri.utils.server Stopped thread 'ZeroConfPlugin'.
ERROR 2022-10-03 22:02:07,857 kolibri.utils.server Error in 'STOP' listener <bound method ZeroConfPlugin.STOP of <kolibri.utils.server.ZeroConfPlugin object at 0x7fbf30b875b0>>
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/kolibri/dist/magicbus/base.py", line 215, in _transition
    return self.publish(newstate, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/dist/magicbus/base.py", line 291, in publish
    raise exc
magicbus.base.ChannelFailures: ProgrammingError('relation "discovery_networklocation" does not exist\nLINE 1: ... "discovery_networklocation"."last_accessed" FROM "discovery...\n                                                             ^\n')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/kolibri/dist/magicbus/base.py", line 273, in publish
    result = listener(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/utils/server.py", line 272, in STOP
    self.broadcast.stop_broadcast()
  File "/usr/lib/python3/dist-packages/kolibri/core/discovery/utils/network/broadcast.py", line 393, in stop_broadcast
    self.unregister()
  File "/usr/lib/python3/dist-packages/kolibri/core/discovery/utils/network/broadcast.py", line 461, in unregister
    self.events.publish(EVENT_UNREGISTER_INSTANCE, self.instance)
  File "/usr/lib/python3/dist-packages/kolibri/dist/magicbus/base.py", line 291, in publish
    raise exc
magicbus.base.ChannelFailures: ProgrammingError('relation "discovery_networklocation" does not exist\nLINE 1: ... "discovery_networklocation"."last_accessed" FROM "discovery...\n                                                             ^\n')

INFO 2022-10-03 22:02:07,863 kolibri.core.tasks.worker Asking job schedulers to shut down.
INFO 2022-10-03 22:02:07,981 kolibri.utils.server Bus state: STOP_ERROR
ERROR 2022-10-03 22:02:07,981 kolibri.utils.server Exiting due to error in stop listener:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/kolibri/dist/magicbus/base.py", line 215, in _transition
    return self.publish(newstate, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/dist/magicbus/base.py", line 291, in publish
    raise exc
magicbus.base.ChannelFailures: ProgrammingError('relation "discovery_networklocation" does not exist\nLINE 1: ... "discovery_networklocation"."last_accessed" FROM "discovery...\n                                                             ^\n')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/kolibri/dist/magicbus/base.py", line 215, in _transition
    return self.publish(newstate, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/dist/magicbus/base.py", line 291, in publish
    raise exc
magicbus.base.ChannelFailures: ProgrammingError('relation "discovery_networklocation" does not exist\nLINE 1: ... "discovery_networklocation"."last_accessed" FROM "discovery...\n                                                             ^\n')

INFO 2022-10-03 22:02:07,987 kolibri.utils.server Bus state: EXIT
INFO 2022-10-03 22:02:07,988 kolibri.utils.server Waiting for child threads to terminate...
INFO 2022-10-03 22:02:07,988 kolibri.utils.server Bus state: EXITED

Brian_LePore · October 3, 2022, 10:08pm

$ kolibri status
INFO: No C extensions are available for this platform
/usr/lib/python3/dist-packages/pkg_resources/__init__.py:116: PkgResourcesDeprecationWarning: 0.1.43ubuntu1 is an invalid version and will not be supported in a future release
  warnings.warn(
/usr/lib/python3/dist-packages/pkg_resources/__init__.py:116: PkgResourcesDeprecationWarning: 1.1build1 is an invalid version and will not be supported in a future release
  warnings.warn(
Stopped (1)

richard · October 3, 2022, 10:44pm

I think for now, maybe running kolibri start --foreground might be best? That will run without attempting to handle daemonization. I assume that you will be starting it in a way where the daemonization of the process is less important anyway.

Brian_LePore · October 3, 2022, 10:54pm

Well, we wanted it to run at boot and had 1-2 servers running in an AutoScaling Group to handle load so we might have needed that.

But it isn’t looking like foreground is helping any …

$ kolibri start --foreground
INFO: No C extensions are available for this platform
/usr/lib/python3/dist-packages/pkg_resources/__init__.py:116: PkgResourcesDeprecationWarning: 0.1.43ubuntu1 is an invalid version and will not be supported in a future release
  warnings.warn(
/usr/lib/python3/dist-packages/pkg_resources/__init__.py:116: PkgResourcesDeprecationWarning: 1.1build1 is an invalid version and will not be supported in a future release
  warnings.warn(
INFO     Option DEBUG in section [Server] being overridden by environment variable KOLIBRI_DEBUG
INFO     Option DEBUG_LOG_DATABASE in section [Server] being overridden by environment variable KOLIBRI_DEBUG_LOG_DATABASE
INFO     Running Kolibri with the following settings: kolibri.deployment.default.settings.base
INFO     Importing 'tasks' module from django apps
INFO     Starting Kolibri 0.15.7
INFO     Bus state: ENTER
INFO     Listening for SIGTERM.
INFO     Listening for SIGHUP.
INFO     Listening for SIGUSR1.
INFO     Listening for SIGINT.
INFO     Bus state: IDLE
INFO     Bus state: START
INFO     Starting async task workers.
INFO     Started monitor thread 'ProcessControlPlugin'.
INFO     Started monitor thread 'ZeroConfPlugin'.
INFO     Starting on 0.0.0.0:0
INFO     Waiting for 0.0.0.0:37391
INFO     Serving on 0.0.0.0:37391
INFO     Starting on 0.0.0.0:8080
INFO     Waiting for 0.0.0.0:8080
INFO     Serving on 0.0.0.0:8080
INFO     Registering ourselves to zeroconf network with id '09b7345875e25fc3f76ac5707914c771' and port '8080'
ERROR    Error in 'SERVING' listener <bound method ZeroConfPlugin.SERVING of <kolibri.utils.server.ZeroConfPlugin object at 0x7efd779f5930>>
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/kolibri/dist/magicbus/base.py", line 273, in publish
    result = listener(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/utils/server.py", line 261, in SERVING
    self.broadcast.start_broadcast()
  File "/usr/lib/python3/dist-packages/kolibri/core/discovery/utils/network/broadcast.py", line 357, in start_broadcast
    self.register()
  File "/usr/lib/python3/dist-packages/kolibri/core/discovery/utils/network/broadcast.py", line 429, in register
    self.events.publish(EVENT_REGISTER_INSTANCE, self.instance)
  File "/usr/lib/python3/dist-packages/kolibri/dist/magicbus/base.py", line 291, in publish
    raise exc
magicbus.base.ChannelFailures: ProgrammingError('relation "discovery_networklocation" does not exist\nLINE 1: ... "discovery_networklocation"."last_accessed" FROM "discovery...\n                                                             ^\n')

ERROR    Error in 'START' listener <bound method KolibriServerPlugin.START of <kolibri.utils.server.KolibriServerPlugin object at 0x7efd779f49a0>>
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/kolibri/dist/magicbus/base.py", line 273, in publish
    result = listener(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/utils/server.py", line 177, in START
    self.bus.publish("SERVING", bind_port)
  File "/usr/lib/python3/dist-packages/kolibri/dist/magicbus/base.py", line 291, in publish
    raise exc
magicbus.base.ChannelFailures: ProgrammingError('relation "discovery_networklocation" does not exist\nLINE 1: ... "discovery_networklocation"."last_accessed" FROM "discovery...\n                                                             ^\n')

INFO     Bus state: START_ERROR
ERROR    Exiting due to error in start listener:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/kolibri/dist/magicbus/base.py", line 215, in _transition
    return self.publish(newstate, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/dist/magicbus/base.py", line 291, in publish
    raise exc
magicbus.base.ChannelFailures: ProgrammingError('relation "discovery_networklocation" does not exist\nLINE 1: ... "discovery_networklocation"."last_accessed" FROM "discovery...\n                                                             ^\n')

INFO     Bus state: STOP
INFO     HTTP Server kolibri.utils.server.Server(('0.0.0.0', 8080)) shut down
INFO     HTTP Server kolibri.utils.server.Server(('0.0.0.0', 37391)) shut down
INFO     Asking job schedulers to shut down.
INFO     Joining 'ZeroConfPlugin'
INFO     Stopped thread 'ZeroConfPlugin'.
ERROR    Error in 'STOP' listener <bound method ZeroConfPlugin.STOP of <kolibri.utils.server.ZeroConfPlugin object at 0x7efd779f5930>>
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/kolibri/dist/magicbus/base.py", line 215, in _transition
    return self.publish(newstate, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/dist/magicbus/base.py", line 291, in publish
    raise exc
magicbus.base.ChannelFailures: ProgrammingError('relation "discovery_networklocation" does not exist\nLINE 1: ... "discovery_networklocation"."last_accessed" FROM "discovery...\n                                                             ^\n')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/kolibri/dist/magicbus/base.py", line 273, in publish
    result = listener(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/utils/server.py", line 272, in STOP
    self.broadcast.stop_broadcast()
  File "/usr/lib/python3/dist-packages/kolibri/core/discovery/utils/network/broadcast.py", line 393, in stop_broadcast
    self.unregister()
  File "/usr/lib/python3/dist-packages/kolibri/core/discovery/utils/network/broadcast.py", line 461, in unregister
    self.events.publish(EVENT_UNREGISTER_INSTANCE, self.instance)
  File "/usr/lib/python3/dist-packages/kolibri/dist/magicbus/base.py", line 291, in publish
    raise exc
magicbus.base.ChannelFailures: ProgrammingError('relation "discovery_networklocation" does not exist\nLINE 1: ... "discovery_networklocation"."last_accessed" FROM "discovery...\n                                                             ^\n')

INFO     Joining 'ProcessControlPlugin'
INFO     Stopped thread 'ProcessControlPlugin'.
INFO     Bus state: STOP_ERROR
ERROR    Exiting due to error in stop listener:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/kolibri/dist/magicbus/base.py", line 215, in _transition
    return self.publish(newstate, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/dist/magicbus/base.py", line 291, in publish
    raise exc
magicbus.base.ChannelFailures: ProgrammingError('relation "discovery_networklocation" does not exist\nLINE 1: ... "discovery_networklocation"."last_accessed" FROM "discovery...\n                                                             ^\n')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/kolibri/dist/magicbus/base.py", line 215, in _transition
    return self.publish(newstate, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/dist/magicbus/base.py", line 291, in publish
    raise exc
magicbus.base.ChannelFailures: ProgrammingError('relation "discovery_networklocation" does not exist\nLINE 1: ... "discovery_networklocation"."last_accessed" FROM "discovery...\n                                                             ^\n')

INFO     Bus state: EXIT
INFO     Waiting for child threads to terminate...
INFO     Bus state: EXITED

richard · October 3, 2022, 11:01pm

It seems like the database has not been migrated? kolibri manage migrate needs to be run after switching database backends.

Brian_LePore · October 3, 2022, 11:20pm

While I definitely did not run kolibri manage migrate as I never saw anything in the documentation about it, doing that and then trying to start and start --foreground did not seem to cause any change.

Was I supposed to add any parameters to that or just run kolibri manage migrate?

I was definitely in the wrong directory when I got the permission error. That I felt dumb about.

I’m not sure if this matters, but the DB is powered by RDS and is not local. That said, I can connect to the database from the command line using pgsl.

richard · October 3, 2022, 11:27pm

Have you configured Kolibri to connect to the database? This can be configured either using environment variables, or using the options.ini file (Customize Kolibri settings with the options.ini file — Kolibri User Guide). The specific settings for database backends are only documented in code (kolibri/options.py at release-v0.15.x · learningequality/kolibri · GitHub) similar to the Redis configuration.

To see a complete documentation of all available environment variables kolibri configure list-env will give a full list plus descriptions. Environment variables will override anything set in the options.ini when both are set.

Brian_LePore · October 3, 2022, 11:38pm

Yes, it’s set in the options.ini in the .kolibri directory.

[Database]
# Which database backend to use, choices are 'sqlite' or 'postgres'
DATABASE_ENGINE = postgres

# For SQLite - the name of a database file to use for the main Kolibri database.
# For Postgresql, the name of the database to use for all Kolibri data.
DATABASE_NAME = EDITED_OUT

# The password to authenticate with when connecting to the database, Postgresql only.
DATABASE_PASSWORD = EDITED_OUT

# The user to authenticate with when connecting to the database, Postgresql only.
DATABASE_USER = EDITED_OUT

# The host on which to connect to the database, Postgresql only.
DATABASE_HOST = EDITED_OUT

# The port on which to connect to the database, Postgresql only.
DATABASE_PORT = 5432

It feels to me like it’s not following the options.ini file / not connecting to the DB.

Brian_LePore · October 11, 2022, 8:40pm

So just an FYI on things. It seems I only ported over the db.sqlite3 file and didn’t move over the other 4 sqlite files. Once I moved them over I was able to start Kolibri at least.

Topic		Replies	Views
Self host kolibri - full docker compose Support	1	193	June 27, 2023
Installing Kolibri on a Synology DiskStation Hardware	4	1292	October 3, 2022
Install Kolibri in a docker container and configure as a webserver Kolibri	2	1042	November 14, 2018
How to manage Kolibri content in kolibri Docker container Support	5	1927	August 12, 2019
Kolibri content serving from S3 or cloud storage Kolibri content	2	438	June 1, 2021

AWS hosting recommendation

Related Topics