Blended learning rollout plan for a school of 2000 students, Kolkata, India

Planning Your Kolibri Program
1

Scenario:

  1. School catering to ~2000 underprivileged children in central Kolkata, eastern India of Grades 1 to 12. It is a West Bengal state board school. It is run by a Trust but in the current circumstances, they are clueless about how to systematically restart the education process. Teachers are trying to communicate with students over whatsapp
  2. Schools in India may not open till December, 2020 due to the Covid-19 pandemic.
  3. Most students have at best a shared smart phone at home and a LCD / LED TV with a HDMI/VGA port. Smart phones configurations may range from Android 6, 1 GB, 16 GB to more recent Android 9, 6 GB, 64 GB (rare).
  4. Most teachers have a smart phone with similar configurations as that of the students. Few have a laptop or desktop at home, most with un-licensed or not updated Windows 7 or older. Awareness and adoption of Linux is non-existent.
  5. Most have internet access via their mobiles with a cap of 1.5 to 2 GB per day. The same is mostly shared by all family members.

Kolibri deployment plan:

  1. A Kolibri server would be housed on the school campus or on cloud. The later seems more feasible for 24x7 access.
  2. This one server would have 1 “Facility” in Kolibri and Grades 1 to 12 x sections A, B, high school has C, D also i.e. 36+ “Classes”.
  3. School teachers would have Coach accounts on the main server and assigned to their respective Classes. Students too shall be enrolled to their respective classes.
  4. We have built and tried the apk on Android phones with 2 GB RAM, Android 6. It works reliably thus far in our tests. So for those who have 2 GB, Android 6 phones, we shall start with a rollout to those users.
  5. For those with desktop / laptop, we shall remotely guide them with Kolibri installation. If the same fails due to OS issues, then we shall try to convince them to install Ubuntu Linux with dual boot.
  6. For those who don’t have a smart phone or a desktop or laptop, we plan to equip them with a RPi4 based full computer experience when connected with their TV and a KB + mouse. We are in the final stages of a POC on RPi4 devices. Via bulk procurement and hopefully some sponsorship campaigns, we aim to get one in the hands of each student and teacher.
  7. As per the Kolibri workflow, coaches will design lessons and we shall seed the phones via the internet or manually, depending on what Covid-19 restrictions permit at the time of deployment. The RPi4 devices shall be pre-seeded with grade-wise content, lessons and users.
  8. The RPi4 which has a WiFi hotspot will do well for a household having multiple students from this same school. They will need only one RPi device and can use the same as a computer as well as access Kolibri over the WiFi on even a “low-end” smart phone.
  9. Learners shall be instructed to ensure that they connect to the internet via mobile USB tether at least once every 24 hours to get the coaches up to speed with their progress. With Kolibri 0.14.x on the RPi4 we can configure a cron (with admin credentials) to check internet availability and execute a sync with the “cloud” server. The same sync mechanism can also be triggered whenever network device status is up.
  10. The issue lies with the Android app users. There is no way a user can sync his progress without having admin access as yet.
  11. With progress data analyzed by the coaches, they can have targeted whatsapp / jitsi / zoom / meet sessions with the group of students / individuals who need help.
  12. Post Covid, or even as the schools start with staggered classes with partial attendance, the blended learning model shall remain relevant and thus the school shall have smoothly transitioned into the “new normal”.

Planned Timeline:

We shall start with just one grade i.e. Grade 9 and 2 subjects.
Week 1: Conduct teacher training of 2 teachers and assist them in lesson planning. (Do note that teachers are coming to schools but students are not.)
Week 2: Plan to roll out to students and teachers with eligible smart phones, laptops and desktops. Week 3: 4 RPi4 based device should be ready for trials.
Week 4 and 5: Monitor usage of all coaches and students.
From weeks 3 to 5, lesson plans for more subjects shall be prepared.
If all goes fine, plans to scale the same shall be discussed with the school management.

Queries:

While our friends at LE are working hard to get the game changing sync features perfected, we need help with some immediate “hacks” that we are attempting -

  1. Is it possible to somehow get the Android app to also attempt a periodic sync or have it triggered upon internet connectivity detection on an Android smart phone?
  2. What is the planned feature roadmap regarding learner / teacher initiated sync?

Rationale behind a RPi4B when a RPi3B or a RPi0 could also be enough for Kolibri -
We want each child to get an opportunity to be a prosumer = producer + consumer, for which a full computing experience is needed at a minimum. Hence to ensure that an investment of close to $200,000 has the maximum impact, we have embarked on this plan.

There are many other schools in our vicinity and am sure across the globe which are in a similar situation and are waiting for a way to get on with the learning process. Our belief is that if only we can showcase effective learning outcomes with a blended learning model supported by the right digital means, can this global pandemic of the digital divide induced learning deficit be addressed. Covid-19 has presented all of us an opportunity to effect the change. Let us make it happen.

Any tweaks / suggestions in addition to the requested assistance for this plan are most welcome.

2 Likes
2

To address the immediate needs of this project we have implemented this periodic auto sync method for the Android app and for Linux installations -
https://github.com/cyberorg/kolibri-autosync/blob/master/src/usr/sbin/kolibri-autosync.py

Must confess that it is a hack that is working in our test runs with 5-7 simultaneous users. Does this look like an acceptable solution?

We are currently hard coding sync user credentials as well as configuring sync server URL in an .ini file. We can do away with the former by studying and using the same process as the P2P sync does.
If this functionality to configure and save the P2P sync (URL, sync user credentials and periodicity) can be built into the UI then we won’t have to hard code any sensitive information.

The app deployment process, which too needs to be done remotely, becomes slightly lengthy and convoluted as import facility can’t be done using kolibri manage. Hence we create and share temporary admin credentials for a set of users who need to be on-boarded using their Android device, trusting them with not to make themselves super users but yet enable to manage content of the device. Once they are on-boarded, we delete/change the admin creds.

The content import process is as yet explicit from the lesson plans and from what we can gather from the discussions in Github, related content shall auto-accompany lesson plans in the future. This too shall reduce a few complex steps for a learner while on-boarding.

We eagerly await queries, suggestions and criticism. Please guide.

3

Thanks for sharing more details from your use case!

We don’t yet have a timeline on the next phases of functionality, but we’re working on it.

One quick note on the script: The credentials only need to be included during the first sync. It uses them to obtain a certificate that can then be used to sync to any other device with the same facility. So it would be best not to hardcode them in there, for security reasons.

4

Thank you @jamalex, Could you please guide me to the section of code or the related dev docs that be used to obtain the certificate? I am guessing that the process to get the certificate is in these tests.

5

That happens automatically, when the first sync happens. If it’s the first sync, and credentials aren’t provided, it asks for credentials. In subsequent syncs, you can just omit the credentials, because it already has the certificate.

6

Great. Shall try removing the syncuser then and see how it functions because a user is required to import a facility for the first time to get started, which I guess should be same as a first sync.

Is there a way as yet to import a facility using kolibri manage?

7

Sorry, I’m not so clear on the question – kolibri manage sync ... is the way to import a facility.

8

In my experience with kolibri manage sync ..., I could only do a sync once a facility had already been imported using the Devices -> Facilities -> Import Facility in the UI.
Are you suggesting that on a pristine Kolibri installation, I should try importing a facility using kolibri manage sync ...?

9

Aim is to import a facility non-interactively into a pristine installation of Kolibri with sqlite and Postgres.

I recall having experienced the below errors even earlier when trying to import a facility via CLI. I had tried when I first noticed the --no-provision option during 0.14.x beta testing. But I gave up assuming that I was attempting an unsupported feature.

I get the following error when using postgres -

$ kolibri manage sync --baseurl <url> --facility <facilityid> --no-push --username <superuser> --password <superuser_password>
WARNING:root:No C Extensions available for this platform.

/usr/lib/python3/dist-packages/kolibri/dist/rest_framework/utils/serializer_helpers.py:107: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if value is None or value is '':
INFO     Running Kolibri with the following settings: kolibri.deployment.default.settings.base
WARNING  Redis is configured without a maximum memory policy. Using Redis with Kolibri, the following is suggested: maxmemory-policy allkeys-lru
WARNING  Redis is configured without a maximum memory size.
WARNING  Problematic Redis settings detected, please see Redis configuration documentation for details: https://redis.io/topics/config
INFO     Invoking command <truncated>
INFO     Attempting connections to variations of the URL: http://content.myscoolserver.in
INFO     Attempting connection to: http://content.myscoolserver.in/
INFO     Success! We connected to: http://content.myscoolserver.in/api/public/info/
INFO     Syncing has been initiated (this may take a while)...
INFO     Creating pull transfer session
Remotely preparing data  [##############################################################################################################################################################]  100%INFO     There are no records to transfer
INFO     Completed pull transfer session
Traceback (most recent call last):
  File "/usr/bin/kolibri", line 11, in <module>
    load_entry_point('kolibri==0.14.3', 'console_scripts', 'kolibri')()
  File "/usr/lib/python3/dist-packages/kolibri/dist/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/dist/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3/dist-packages/kolibri/dist/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3/dist-packages/kolibri/utils/cli.py", line 277, in invoke
    return super(KolibriDjangoCommand, self).invoke(ctx)
  File "/usr/lib/python3/dist-packages/kolibri/dist/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3/dist-packages/kolibri/dist/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/dist/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/utils/cli.py", line 727, in manage
    execute_from_command_line(["kolibri manage"] + ctx.args)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/usr/lib/python3/dist-packages/kolibri/core/tasks/management/commands/base.py", line 113, in handle
    return self.handle_async(*args, **options)
  File "/usr/lib/python3/dist-packages/kolibri/core/auth/management/commands/sync.py", line 195, in handle_async
    create_superuser_and_provision_device(
  File "/usr/lib/python3/dist-packages/kolibri/core/auth/management/utils.py", line 209, in create_superuser_and_provision_device
    facility = Facility.objects.get(dataset_id=dataset_id)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/models/query.py", line 378, in get
    raise self.model.DoesNotExist(
kolibri.core.auth.models.DoesNotExist: Facility matching query does not exist.

Cleaned all sqlite files from .kolibri and then one attempt with sqlite did seem to succeed and it was asking to create a local superuser but I terminated it mistakenly.

Subsequent attempt with sqlite yielded this error after a complete system and kolibri restart to avoid issues with redis locks -

$ sudo systemctl stop kolibri-server
$ rm .kolibri/db.sqlite3* .kolibri/notifications.sqlite3 .kolibri/job_storage.sqlite3
$ sudo systemctl stop kolibri-server

(checked that kolibri has started and landed at startup wizard)

$ kolibri manage sync --baseurl <url> --facility <facilityid> --no-push --username <superuser> --password <superuser_password>
WARNING:root:No C Extensions available for this platform.

/usr/lib/python3/dist-packages/kolibri/dist/rest_framework/utils/serializer_helpers.py:107: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if value is None or value is '':
INFO     Running Kolibri with the following settings: kolibri.deployment.default.settings.base
WARNING  Redis is configured without a maximum memory policy. Using Redis with Kolibri, the following is suggested: maxmemory-policy allkeys-lru
WARNING  Redis is configured without a maximum memory size.
WARNING  Problematic Redis settings detected, please see Redis configuration documentation for details: https://redis.io/topics/config
INFO     Invoking command <truncated>
INFO     Attempting connections to variations of the URL: http://content.myscoolserver.in
INFO     Attempting connection to: http://content.myscoolserver.in/
INFO     Success! We connected to: http://content.myscoolserver.in/api/public/info/
INFO     Syncing has been initiated (this may take a while)...
INFO     Creating pull transfer session
Receiving data (1000/1353, 2.38MB)  [#####################################################################################################------------------------------------]   74%  00:00:42INFO     Receiving data (1000/1353, 2.38MB)
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/backends/sqlite3/base.py", line 328, in execute
    return Database.Cursor.execute(self, query, params)
sqlite3.IntegrityError: UNIQUE constraint failed: morango_buffer.transfer_session_id, morango_buffer.model_uuid

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/bin/kolibri", line 11, in <module>
    load_entry_point('kolibri==0.14.3', 'console_scripts', 'kolibri')()
  File "/usr/lib/python3/dist-packages/kolibri/dist/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/dist/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3/dist-packages/kolibri/dist/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3/dist-packages/kolibri/utils/cli.py", line 277, in invoke
    return super(KolibriDjangoCommand, self).invoke(ctx)
  File "/usr/lib/python3/dist-packages/kolibri/dist/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3/dist-packages/kolibri/dist/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/dist/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/utils/cli.py", line 727, in manage
    execute_from_command_line(["kolibri manage"] + ctx.args)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/usr/lib/python3/dist-packages/kolibri/core/tasks/management/commands/base.py", line 113, in handle
    return self.handle_async(*args, **options)
  File "/usr/lib/python3/dist-packages/kolibri/core/auth/management/commands/sync.py", line 184, in handle_async
    self._handle_pull(
  File "/usr/lib/python3/dist-packages/kolibri/core/auth/management/commands/sync.py", line 269, in _handle_pull
    sync_client.run()
  File "/usr/lib/python3/dist-packages/kolibri/dist/morango/sync/syncsession.py", line 692, in run
    self._pull_records(callback=status.in_progress.fire)
  File "/usr/lib/python3/dist-packages/kolibri/dist/morango/sync/syncsession.py", line 757, in _pull_records
    validate_and_create_buffer_data(
  File "/usr/lib/python3/dist-packages/kolibri/dist/morango/sync/utils.py", line 146, in validate_and_create_buffer_data
    Buffer.objects.bulk_create(buffer_list)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/models/query.py", line 443, in bulk_create
    ids = self._batched_insert(objs_without_pk, fields, batch_size)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/models/query.py", line 1102, in _batched_insert
    self._insert(item, fields=fields, using=self.db)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/models/query.py", line 1079, in _insert
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/models/sql/compiler.py", line 1112, in execute_sql
    cursor.execute(sql, params)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/utils.py", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/utils/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/backends/sqlite3/base.py", line 328, in execute
    return Database.Cursor.execute(self, query, params)
django.db.utils.IntegrityError: UNIQUE constraint failed: morango_buffer.transfer_session_id, morango_buffer.model_uuid

All subsequent attempts after a system reboot with sqlite eventually results in the above error -

django.db.utils.IntegrityError: UNIQUE constraint failed: morango_buffer.transfer_session_id, morango_buffer.model_uuid

or even at times -

ERROR 2020-09-02 16:45:57,144 kolibri.core.tasks.worker Job 9201d5e21fdc445295232684c37605df raised an exception: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/kolibri/core/tasks/worker.py", line 72, in handle_finished_future
    result = future.result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
  File "/usr/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/core/tasks/worker.py", line 217, in wrap
    return f(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/core/tasks/job.py", line 194, in y
    result = func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/core/analytics/utils.py", line 453, in _ping
    ping_once(started, server=server)
  File "/usr/lib/python3/dist-packages/kolibri/core/analytics/utils.py", line 445, in ping_once
    create_and_update_notifications(data, nutrition_endpoints.PINGBACK)
  File "/usr/lib/python3.8/contextlib.py", line 75, in inner
    return func(*args, **kwds)
  File "/usr/lib/python3/dist-packages/kolibri/core/analytics/utils.py", line 374, in create_and_update_notifications
    PingbackNotification.objects.filter(source=source).exclude(
  File "/usr/lib/python3/dist-packages/kolibri/core/utils/cache.py", line 70, in __exit__
    self.release()
  File "/usr/lib/python3/dist-packages/kolibri/core/utils/cache.py", line 64, in release
    self._lock.release()
  File "/usr/lib/python3/dist-packages/kolibri/dist/redis/lock.py", line 224, in release
    raise LockError("Cannot release an unlocked lock")
redis.exceptions.LockError: Cannot release an unlocked lock

With Postgres pristine db and a complete server restart -

$ kolibri manage sync --baseurl <url> --facility <facilityid> --no-push --username <superuser> --password <superuser_password>
WARNING:root:No C Extensions available for this platform.

/usr/lib/python3/dist-packages/kolibri/dist/rest_framework/utils/serializer_helpers.py:107: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if value is None or value is '':
INFO     Running Kolibri with the following settings: kolibri.deployment.default.settings.base
WARNING  Redis is configured without a maximum memory policy. Using Redis with Kolibri, the following is suggested: maxmemory-policy allkeys-lru
WARNING  Redis is configured without a maximum memory size.
WARNING  Problematic Redis settings detected, please see Redis configuration documentation for details: https://redis.io/topics/config
INFO     Invoking command <truncated>
INFO     Attempting connections to variations of the URL: http://content.myscoolserver.in
INFO     Attempting connection to: http://content.myscoolserver.in/
INFO     Success! We connected to: http://content.myscoolserver.in/api/public/info/
INFO     Syncing has been initiated (this may take a while)...
INFO     Creating pull transfer session
Receiving data (1353/1353, 2.92MB)  [###################################################################################################################################################]  100%INFO     Receiving data (1353/1353, 2.92MB)
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "morango_buffer_transfer_session_id_model_uuid_2a7288db_uniq"
DETAIL:  Key (transfer_session_id, model_uuid)=(7aa3fb3e-6a2c-4522-aafc-242fc9d8ff95, 0e2cf944-d328-1e1a-1575-7f845211ade3) already exists.


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/bin/kolibri", line 11, in <module>
    load_entry_point('kolibri==0.14.3', 'console_scripts', 'kolibri')()
  File "/usr/lib/python3/dist-packages/kolibri/dist/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/dist/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3/dist-packages/kolibri/dist/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3/dist-packages/kolibri/utils/cli.py", line 277, in invoke
    return super(KolibriDjangoCommand, self).invoke(ctx)
  File "/usr/lib/python3/dist-packages/kolibri/dist/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3/dist-packages/kolibri/dist/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/dist/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/utils/cli.py", line 727, in manage
    execute_from_command_line(["kolibri manage"] + ctx.args)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/usr/lib/python3/dist-packages/kolibri/core/tasks/management/commands/base.py", line 113, in handle
    return self.handle_async(*args, **options)
  File "/usr/lib/python3/dist-packages/kolibri/core/auth/management/commands/sync.py", line 184, in handle_async
    self._handle_pull(
  File "/usr/lib/python3/dist-packages/kolibri/core/auth/management/commands/sync.py", line 269, in _handle_pull
    sync_client.run()
  File "/usr/lib/python3/dist-packages/kolibri/dist/morango/sync/syncsession.py", line 692, in run
    self._pull_records(callback=status.in_progress.fire)
  File "/usr/lib/python3/dist-packages/kolibri/dist/morango/sync/syncsession.py", line 757, in _pull_records
    validate_and_create_buffer_data(
  File "/usr/lib/python3/dist-packages/kolibri/dist/morango/sync/utils.py", line 146, in validate_and_create_buffer_data
    Buffer.objects.bulk_create(buffer_list)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/models/query.py", line 443, in bulk_create
    ids = self._batched_insert(objs_without_pk, fields, batch_size)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/models/query.py", line 1096, in _batched_insert
    inserted_id = self._insert(item, fields=fields, using=self.db, return_id=True)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/models/query.py", line 1079, in _insert
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/models/sql/compiler.py", line 1112, in execute_sql
    cursor.execute(sql, params)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/utils.py", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/utils/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
django.db.utils.IntegrityError: duplicate key value violates unique constraint "morango_buffer_transfer_session_id_model_uuid_2a7288db_uniq"
DETAIL:  Key (transfer_session_id, model_uuid)=(7aa3fb3e-6a2c-4522-aafc-242fc9d8ff95, 0e2cf944-d328-1e1a-1575-7f845211ade3) already exists.

Every attempt to import a facility from UI, including a complete reinstall of Kolibri and clearing of KOLIBRI_HOME results in a failure with this in the logs -

INFO 2020-09-02 19:24:45,374 kolibri.core.auth.management.commands.sync Completed pull transfer session
ERROR 2020-09-02 19:24:45,390 kolibri.core.tasks.worker Job b35765430c37409fa745d9b25e6bb778 raised an exception: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/kolibri/core/tasks/worker.py", line 72, in handle_finished_future
    result = future.result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/usr/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
  File "/usr/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/core/tasks/worker.py", line 217, in wrap
    return f(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/core/tasks/job.py", line 194, in y
    result = func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/core/management/__init__.py", line 131, in call_command
    return command.execute(*args, **defaults)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/usr/lib/python3/dist-packages/kolibri/core/tasks/management/commands/base.py", line 113, in handle
    return self.handle_async(*args, **options)
  File "/usr/lib/python3/dist-packages/kolibri/core/auth/management/commands/sync.py", line 184, in handle_async
    self._handle_pull(
  File "/usr/lib/python3/dist-packages/kolibri/core/auth/management/commands/sync.py", line 271, in _handle_pull
    sync_client.finalize()
  File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__
    next(self.gen)
  File "/usr/lib/python3/dist-packages/kolibri/core/auth/management/commands/sync.py", line 222, in _lock
    yield
  File "/usr/lib/python3/dist-packages/kolibri/core/utils/cache.py", line 70, in __exit__
    self.release()
  File "/usr/lib/python3/dist-packages/kolibri/core/utils/cache.py", line 64, in release
    self._lock.release()
  File "/usr/lib/python3/dist-packages/kolibri/dist/redis/lock.py", line 222, in release
    expected_token = self.local.token
AttributeError: '_thread._local' object has no attribute 'token'

Environment:
Kolibri 0.14.3 deb + Kolibri server 0.3.8~beta2-0ubuntu1 + redis 5.0.7 + Ubuntu 20.04

I don’t even see a pattern apart from that all attempts to import are now failing.
Is there a pattern that I am missing? Not sure if I am missing anything obvious and doing it all wrong.

10

Aim achieved. The problem is somewhere on my trial device. Switched to another and all working as expected. :ok_hand:

Thereafter I tried to import two resources from the KA CBSE India channel and ran into this error -

$ kolibri manage importcontent --node_ids 1dd8f2bf51325a1dacf95e0dadd9f9ea,0d2ebd88cc9653dda5703413a8c5396c network 2fd54ca47a8f59c99fcefaaa3894c19e
WARNING:root:No C Extensions available for this platform.

/usr/lib/python3/dist-packages/kolibri/dist/rest_framework/utils/serializer_helpers.py:107: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if value is None or value is '':
INFO     Running Kolibri with the following settings: kolibri.deployment.default.settings.base
WARNING  Redis is configured without a maximum memory policy. Using Redis with Kolibri, the following is suggested: maxmemory-policy allkeys-lru
WARNING  Redis is configured without a maximum memory size.
WARNING  Problematic Redis settings detected, please see Redis configuration documentation for details: https://redis.io/topics/config
INFO     Invoking command importcontent --node_ids 1dd8f2bf51325a1dacf95e0dadd9f9ea,0d2ebd88cc9653dda5703413a8c5396c network 2fd54ca47a8f59c99fcefaaa3894c19e
Traceback (most recent call last):
  File "/usr/bin/kolibri", line 11, in <module>
    load_entry_point('kolibri==0.14.3', 'console_scripts', 'kolibri')()
  File "/usr/lib/python3/dist-packages/kolibri/dist/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/dist/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3/dist-packages/kolibri/dist/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3/dist-packages/kolibri/utils/cli.py", line 277, in invoke
    return super(KolibriDjangoCommand, self).invoke(ctx)
  File "/usr/lib/python3/dist-packages/kolibri/dist/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3/dist-packages/kolibri/dist/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/dist/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/lib/python3/dist-packages/kolibri/utils/cli.py", line 727, in manage
    execute_from_command_line(["kolibri manage"] + ctx.args)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/lib/python3/dist-packages/kolibri/dist/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/usr/lib/python3/dist-packages/kolibri/core/tasks/management/commands/base.py", line 113, in handle
    return self.handle_async(*args, **options)
  File "/usr/lib/python3/dist-packages/kolibri/core/content/management/commands/importcontent.py", line 441, in handle_async
    self.download_content(
  File "/usr/lib/python3/dist-packages/kolibri/core/content/management/commands/importcontent.py", line 159, in download_content
    self._transfer(
  File "/usr/lib/python3/dist-packages/kolibri/core/content/management/commands/importcontent.py", line 366, in _transfer
    annotation.set_content_visibility(
  File "/usr/lib/python3/dist-packages/kolibri/core/content/utils/annotation.py", line 681, in set_content_visibility
    update_content_metadata(
  File "/usr/lib/python3/dist-packages/kolibri/core/content/utils/annotation.py", line 669, in update_content_metadata
    set_leaf_node_availability_from_local_file_availability(
  File "/usr/lib/python3/dist-packages/kolibri/core/content/utils/annotation.py", line 350, in set_leaf_node_availability_from_local_file_availability
    max_rght, dynamic_chunksize = _calculate_batch_params(
  File "/usr/lib/python3/dist-packages/kolibri/core/content/utils/annotation.py", line 237, in _calculate_batch_params
    min(CHUNKSIZE, ceil(250 * max_rght / (constraint_count or 1)))
TypeError: unsupported operand type(s) for *: 'int' and 'NoneType'

Checked that max_rght is None.

Version: 0.14.3
OS: Linux-5.4.0-1015-raspi-aarch64-with-glibc2.29
Python: 3.8.2
Installer: kolibri(apt) with kolibri-server Version: 0.3.8~beta2-0ubuntu1
Server: nginx/1.18.0 (Ubuntu)
Database: postgresql

The importcontent is working fine on a few other systems we tested. What edge case may I have run into? Is this worth creating an issue?

11 
>>> main(["manage", "importcontent", "--node_ids", "33672001c7735123b5ba424283587a14", "network", "3e464ee12f6a50a781cddf59147b48b1"])
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/home/kivy/build/android/kolibri/dist/click/core.py", line 764, in _call_
  File "/home/kivy/build/android/kolibri/dist/click/core.py", line 717, in main
  File "/home/kivy/build/android/kolibri/dist/click/core.py", line 1137, in invoke
  File "/home/kivy/build/android/kolibri/utils/cli.py", line 277, in invoke
  File "/home/kivy/build/android/kolibri/dist/click/core.py", line 956, in invoke
  File "/home/kivy/build/android/kolibri/dist/click/core.py", line 555, in invoke
  File "/home/kivy/build/android/kolibri/dist/click/decorators.py", line 17, in new_func
  File "/home/kivy/build/android/kolibri/utils/cli.py", line 727, in manage
  File "/home/kivy/build/android/kolibri/dist/django/core/management/_init_.py", line 364, in execute_from_command_line
  File "/home/kivy/build/android/kolibri/dist/django/core/management/_init_.py", line 356, in execute
  File "/home/kivy/build/android/kolibri/dist/django/core/management/base.py", line 283, in run_from_argv
  File "/home/kivy/build/android/kolibri/dist/django/core/management/base.py", line 330, in execute
  File "/home/kivy/build/android/kolibri/core/tasks/management/commands/base.py", line 113, in handle
  File "/home/kivy/build/android/kolibri/core/content/management/commands/importcontent.py", line 441, in handle_async
  File "/home/kivy/build/android/kolibri/core/content/management/commands/importcontent.py", line 159, in download_content
  File "/home/kivy/build/android/kolibri/core/content/management/commands/importcontent.py", line 366, in _transfer
  File "/home/kivy/build/android/kolibri/core/content/utils/annotation.py", line 681, in set_content_visibility
  File "/home/kivy/build/android/kolibri/core/content/utils/annotation.py", line 669, in update_content_metadata
  File "/home/kivy/build/android/kolibri/core/content/utils/annotation.py", line 350, in set_leaf_node_availability_from_local_file_availability
  File "/home/kivy/build/android/kolibri/core/content/utils/annotation.py", line 237, in _calculate_batch_params
TypeError: unsupported operand type(s) for *: 'int' and 'NoneType'
>>>```

Same in Android, thing common between the failing system is ARM.
12

Lots of stuff going on in parallel in this thread! :slight_smile:

Starting from the bottom: The unsupported operand type(s) for *: 'int' and 'NoneType' error happens when you try to run importcontent before you’ve run importchannel. max_rght is computed from the channel metadata in the database; if you haven’t run importchannel, there’s no metadata for that channel. Opening an issue for that would be great, specifically around “Displaying a more meaningful error message”, as this can be confusing.

The redis-locking stuff should have been resolved in 0.14.3, but it’s possible there is still something left over in the redis database that’s interfering. Completely clearing everything in redis and trying again for that one could be good.

For the morango-syncing pieces:

  • The “uniqueness” error we saw on an older version but we expected should be resolved. If you can replicate that on a clean install (and clarify where you’re syncing from), we can try to look into it.
  • For the “Facility matching query does not exist.” error, we’d similarly need to know where you’re syncing from, and what’s there. Something funny is happening, because it says “There are no records to transfer” – so maybe the facility you requested was deleted from the remote server, or something similar?
13

@jamalex Thanks for all the clarifications.

  • importcontent - https://github.com/learningequality/kolibri/issues/7506
  • redis-locking - I shall try with redis-cli FLUSHALL and if can’t reproduce then shall let it rest
  • morango-syncing - Was syncing from our hosted instance at content.myscoolserver.in from which syncing the same facility to other devices is working fine. Will let these rest till I can reproduce consistently.
14

Tried import after redis-cli FLUSHALL and now all is fine on the “broken” system. Tried a few permutations and combinations of the steps I had done earlier but could not replicate how to reach the broken state again.

1 Like
15

@jamalex @kollivier Till 3 days ago, we were able to successfully build an apk with the 0.14.3 whl following the documented process and test our customization. However, since yesterday, even though the apk is built smoothly, upon first run on the Android env we encounter this error related to morango migrations.

Thereafter have extracted the pre-seeded KOLIBRI_HOME and tried to run the same on a PC with 0.14.3 installed. It lands up at the setup wizard without any issues reported in the logs. Hence, it looks like an issue specific to the kolibri core being built into the apk.

Do note that nothing has changed in the build environment over the last 3 days apart from some custom code related to kolibri-installer-android.

Herein is the dump of the build process - https://pastebin.ubuntu.com/p/Sg5wFwBY6c/

Request you please guide us with the next steps to help debug the same.

16

Fixed.

17

In relation to the above project, need to determine the size of user progress data for 1000 users over 6 months. I understand that this could depend on factors like number of resources, number of interactions and more. Is there any method that can be used to logically arrive at an estimate?

1 Like
18

Hi @shrenik in Brazil we have the same reality. I am from a non-profit organization and we managed to allocate funds to implement a mesh network, we have some partners who have experience to assist in the implementation of these networks. We will have a server that will send the exclusive Kolibri signal (later we can implement other services) to students in the reach of the network’s Wifi, I dont know of the legal conditions for such an implementation in India, in Brazil there are some restrictions for this type of use, but we do fit (like free conn and others conditions) .

For 1000 students it would be a different reality from the one we designed (our case up to 100 simultaneous, as a test), we will leave an instance of Kolibri in the clouds to synchronize with our mesh network and with notebooks that will take the content produced

As soon as we had the first results, I will forward a report with details of the implementation, forecasts and realities faced.

Our plan will also consist of weekly support for students with virtual meetings, those who do not have internet will meet with borrowed 4G modems or in person at the Association’s premises.

Researched some equipment on Indian sites with mesh network equipment

If you have an Internet instance of Kolibri as a base and one on a server in a good location for the mesh network

Waiting for more updates and sucess!!!

1 Like
19

Hi @tmalmeida Thanks for the idea but this won’t be possible in our use case due to population density and scattered students over a large geography. We shall have to wait for the user based sync related development that seems planned for 0.15. Till then, to limit the sync load we may resort to a per grade facility. We are facing teacher capacity building challenges currently. So have yet some ground to cover before we run into the sync issues.

1 Like