- Cleanship documentation
- Klarschiff migration
- Indices and tables
Cleanship is a citizen participation (Bürgerbeteiligung) / complaints & suggestion management (Anliegenmanagement) / … online platform that allows you to submit issues about the public infrastructure. The local administration will then review your note and the solution process is public visible.
It is the successor of Klarschiff, a platform where citizens report problems / ideas concerning public infrastructure. The local administration will then review your note and the solution process is public visible.
The code is Python3 using the Django 2 framework and bootstrap4 webfrontend toolkit.
Currently alpha , so expect that we will break your installation / data / modules / … !
Migrations will cause data lost!
Currently we port only existing features, while avoiding old bottlenecks. New features will be introduced starting with version 0.3. New features will be introduced starting with version 0.3 (*IX*)
- This version is a preview with a lot of limitations:
- frontend UI not polished
- read-only API
- some performance issues
- hardcoded settings
- issues with georeference
- groups to maintain / delegate issues
- staff frontend (office)
- admin backend
- API compatible with CitySDK
- import Klarschiff legacy data
python3 manage.py runserver --settings cleanship.settings.local
localhost:8000/adminto maintain issues
localhost:8000/officefor staff backoffice
localhost:8000/citysdkfor REST API
localhost:8000/feedfor GeoRSS feed
On Linux you need to follow this steps to get a working instance
Setup postgres DBMS with geoextension¶
sudo apt install pgadmin3 postgresql postgresql-10-postgis-2.4 postgresql-10-postgis-scripts sudo -u postgres psql
CREATE USER cleanship WITH PASSWORD 'mysecretpass'; CREATE DATABASE cleanship OWNER cleanship; ALTER ROLE cleanship CREATEDB SUPERUSER; /*setting up test-dbs with GIS extension requires high privileges)*/
You quit with ‘q’. Now work on specific ‘cleanship’ DB:
CREATE EXTENSION postgis;
Setup python virtualenv¶
sudo apt install python3-dev libpq-dev binutils libproj-dev gdal-bin mkvirtualenv -p /usr/bin/python3 cleanship workon cleanship
git clone cleanship pip install -R requirements/base.txt (dev.txt for contributing)
cp /cleanship/settings/example.py /cleanship/settings/local.py
- Adapt your settings in
- Apply DB tables with
python3 manage.py migrate --settings cleanship.settings.local
- Gather static assets to ./static
python3 manage.py collectstatic --settings cleanship.settings.local
- Test startup with
python3 manage.py runserver --settings cleanship.settings.local
- Create first admin user with
python3 manage.py createsuperuser --settings cleanship.settings.local
/municipality_area.jsonwhich contains the outer border as polygon in CRS:4326 (e.g. of Rostock
/eigentumsangaben.geojsonwhich contains disjunct polygones CRS:25833 with char field
- Perform single tests with e.g.
python3 manage.py test legacy/tests -v 2 --settings cleanship.settings.local
Assign users to groups¶
Could be done via admin frontend or programmatically via python shell
from common.models import User, Group myself = User.objects.get(username='test') group = Group.objects.get(name='a group')') group = Group.objects.get(name='a group')') group.user_set.add(myself) group.save()
You can transfer your existing issues from Klarschiff (tested v1.9) to cleanship including issues, categories, groups. We highly recommend a fresh cleanship setup to avoid troubles!
This migration is a more complex process and will take some time for tuning till you get the desired results. Please start with a cloned Klarschiff copy to proceed step by step:
- import categories - create the ideas / problems / tipps hierachy
- import issues - create issue with all details and creates user groups
- export old user details and manually merge them
- import users - create unified user that are referenced
- import comments
- import feedback
The order of the steps is essential, as e.g. comments refer users, issues refer categories, … .
When you have a working mapping and reproducible import steps, you can run all steps in the right order, using
python3 manage.py import --settings cleanship.settings.local.
The following instructions are tested to transfer your Klarschiff content:
- export old data as CSV via this shell-script at your current Klarschiff DB server
export PGPASSWORD="mypass" psql -h localhost -d klarschiff -U admin -Atc "select tablename from pg_tables" |\ while read TBL; do if [[ $TBL == *"klarschiff_"* ]]; then psql -h localhost -d klarschiff -U admin -c "COPY $TBL TO STDOUT WITH (FORMAT CSV, HEADER);" > $TBL.csv fi done
- copy all full size photos to /media directory:
cp /srv/www/klarschiff/static/*_gross_*.jpg ./media
create list of legacy user details:
python3 manage.py exportLegacyUsers --settings cleanship.settings.local
Use the resulting users.txt, which contains of 3 separated blocks (full names, emails, usernames) of all of your Klarschiff users, to manually create a new mapping file called users.csv. It will be used to create the listed users in the next step and needs to contain the following fields:
You need to fill the lines using the information from the txt listing and maybe your Klarschiff administration interface. To merge a user from multiple old names, just add dublicated lines. This mapping is nessesary, to deal with Klarschiff different legacy user references when importing users / comments / feedback / edit history. The goal is to normalize old user datasets (encoding, old migration artifacts, ..) and match the new user-id (lowercase username) against your LDAP user-ids. You can also re-arrange users to the groups.
Create users using this mapping file using
python3 manage.py import --users --settings cleanship.settings.local
Import all categories:
python3 manage.py import --categories --settings cleanship.settings.local
- The import skips some checks to improve performance:
- is in boundary polygon
- updating location description
- updating landowner
- Start full import via
python3 manage.py import --issues --settings cleanship.settings.local
- Import will take only a few minutes
Import all comments
python3 manage.py import --comments --settings cleanship.settings.local.
With the first runs, you will noticed some warnings about non-existing users, which we can’t find in the old_fullname mapping. You will need to find / add them the user mapping file, and run the user import again!
Import all comments
python3 manage.py import --feedback --settings cleanship.settings.local.
With the first runs, you will noticed some warnings about non-existing users, which we can’t find in the email mapping. You will need to find / add them the user mapping file, and run the user import again! There are also Feedback with multiple, or no recipent, which don’t need further attention.
Cleanship focus to be a enterprise-grade selfhosted solution for the public administrations. so it integrates in your existing IT and scales for huge amount of users and notes. You can receive notes with geolocation easily and maintain this issues to find a solution to this note step-by-step.
- LDAP support
- scale with huge amount of issues (>40k tested so far)
- history and logging
- open API following CitySDK protocoll
To get a basic understanding of the internals, you might have a look at the base object definitions:
- issue - problem / idea / tipp for a location, submitted by external or interal authors. e.g. a pothole within a specific street. Focus is the reported damage, not the solution itself. It’s allways assigned to a group of a organisation
- category - a 3 level categorisation by type (problem / idea / tipp), main-category and sub-category e.g. problem - waste - bulky refuse
- role - a overall qualification for a user e.g. admin, editors, field service
- group - a organisation unit of multiple users e.g. civil engineering office
- user - a member of a organsiation
The django project is splitted in different apps focussing on single aspects:
- common - general aspects esp. shared models, admin frontend
- legacy - compatibility features to migrate from old predecessor project Klarschiff
- office - internal frontend for staff
- citysdk - CitySDK API for public frontend / 3rd party apps & platforms
A issue has a status which indicates it’s current progress and which transition can trigger various actions in detail.
- submitted - User submitted issue, but didn’t verified his email yet. Issue will be removed, if user don’t open confirmation link.
- review - User verified his email, but no internal group assigned and no person did a review of the issue content yet. Issue displayed on the map, but details and photo stay hidden.
- work in progress (wip) - A internal group is assigned and working on the issue. Details and photo become public.
- solved - Final state, the core issue could be solved. An explaination is in status text.
- impossible - Final state, the core issue couldn’t be solved. An explaination is in status text.
- A REST like API which CitySDK participation component was implemented at Klarschiff. It is a updated version of the Open311 protocol, by exteding some core concepts. Klarschiff also added even more features, are completely covered here.
- Is the previous (legacy) software suite for civic participation management. It was splitted in a public and internal frontend, both for mobile and desktop users. The database was managed by a backend component, which offered a CitySDK API to the frontends. See Wikipedia (Klarschiff) Cleanship replaces everything below the public frontends.