iritmaximus/oodikone
1
0
Fork 0
mirror of https://github.com/UniversityOfHelsinkiCS/oodikone.git synced 2026-10-12 19:03:39 +03:00
Code Issues Packages Projects Releases Wiki Activity

[Docs] Update instructions on creating or updating a test database

Browse Source
This commit is contained in:
Valtteri Kantanen
2024-09-03 12:35:12 +03:00
parent 56a38e342e
commit 121ab7e9de
4 changed files with 67 additions and 223 deletions
Download Patch File Download Diff File Expand all files Collapse all files
anonyymioodi/README.md
+67 -13
View File
@@ -1,18 +1,72 @@
# Anonyymioodi - test database setup
This folder contains a guide and scripts to modify or create anonymous database images for oodikone, mainly used for testing.
This folder contains a guide and scripts to modify or create anonymous database images for Oodikone, mainly used for testing.
This quick guide should be enough for most purposes: Updating schema, data or postgres version. If not, a more elaborate guide is stored in [old readme](old_readme.md) (though it may have some outdated info).
The quick guide should be enough for most purposes: updating schema, data, or the PostgreSQL version. If you need to generate completely new `sis-importer-db` and `sis-db` images (which shouldn't happen too often), follow the instructions in the section [Generate new sis-importer-db and sis-db completely from scratch](#generate-new-sis-importer-db-and-sis-db-completely-from-scratch).
### Quick guide to update a test database
## Quick guide to updating a test database
- Make sure you have the latest test-db. Just pulling images doesn't seem to replace your already existing volume, so you may need to remove it by hand. This may work: `docker volume rm oodikone_sis-db-data`. The new image including data should be pulled next time you launch oodikone normally.
- Do your dev stuff normally with the test database, for example with `npm run both`. Edit data however you like, run updater via frontend for example, then run tests to see if they pass.
- When you're happy with the database, create a dump:
- `docker exec -i <dbname> pg_dump -Fc -U postgres <dbname> > anonyymioodi/<dbname>.sqz`
- Now you can shut it down `npm run docker:down`
- Navigate to this folder `cd anonyymioodi`
- If you need to **update postgres version**, do it now by changing it in this directory's `docker-compose.yml`.
- Create new container from dump `./create_container_from_dump.sh <dbname>`
- Commit and push it to toska registry `./commit_and_push_to_toska_hub.sh <dbname>`
- `docker compose down`
1. Ensure you have the latest test-db. Just pulling images might not replace your existing volume, so you may need to remove it manually. This command might work: `docker volume rm oodikone_sis-db-data` (replace `oodikone_sis-db-data` with the correct volume name). The new image, including data, should be pulled the next time you launch Oodikone normally.
2. Do your dev work as usual with the test database, for example, using `npm run both`. Edit data as needed, (for example, run the updater via the frontend), and then run tests to see if they pass (follow the instructions in the [README](../README.md) to run the tests more quickly).
3. If you're updating `sis-db`:
- Modify the scheduler limit in [config.js](../updater/sis-updater-scheduler/src/config.js) by setting `module.exports.DEV_SCHEDULE_COUNT = null`.
- Follow the `sis-updater-worker` logs to ensure everything is running smoothly (`docker logs --follow oodikone_sis-updater-worker_1`).
4. When you're satisfied with the database, create a dump:
- Use the command: `docker exec -i <dbname> pg_dump -Fc -U postgres <dbname> > anonyymioodi/<dbname>.sqz`.
5. Shut down the environment with `npm run docker:down`.
6. Navigate to this folder: `cd anonyymioodi`.
7. If you need to update the PostgreSQL version, do it now by changing the version in this directory's `docker-compose.yml`. The version should match the one used on the server to avoid compatibility issues.
8. Create a new container from the dump: `./create_container_from_dump.sh <dbname>`.
9. Commit and push it to the Toska registry: `./commit_and_push_to_toska_hub.sh <dbname>`.
10. Bring down the environment with `docker compose down`.
## Generate new sis-importer-db and sis-db completely from scratch
> [!CAUTION]
> The more the data in `sis-importer-db` and `sis-db` changes, the more tests might break. Because of this, it's highly recommended to make as few changes as possible (for example, avoid changing the degree programs whose students are used in the test data).
>
> If you only need to make small changes, follow the quick guide above.
### Sis-importer-db
1. To generate a sample dump from test Sisu data, follow the instructions at [importer-db-staging-sampletaker](https://github.com/UniversityOfHelsinkiCS/sis-importer/blob/56f85b3196438426806072b923f41b16a1f7bfe1/importer-db-staging-sampletaker/README.md) inside the `sis-importer` repo.
2. Copy `sis-importer-db.sqz` into this folder.
3. Run the following commands (in this folder) to create a new image from the dump and push it to the Toska registry:
```bash
./create_container_from_dump.sh sis-importer-db
./commit_and_push_to_toska_hub.sh sis-importer-db
docker compose down
```
### Sis-db
1. To ensure the newest `sis-importer-db` image is used:
- Delete all `sis-importer-db` images you have locally (list images with `docker images`, delete with `docker rmi <image ID>`).
- Also, delete the old `sis-importer-db` volume (`docker volume rm sis-importer-db-data`).
2. Start Oodikone and Updater with `npm run both` (this will pull the newest `sis-importer-db` image you just pushed in the previous step).
3. Truncate all tables in `sis-db` **_except for `migrations` and `semesters`_** (the easiest way is to use Adminer).
4. Modify the scheduler limit in [config.js](../updater/sis-updater-scheduler/src/config.js) by setting `module.exports.DEV_SCHEDULE_COUNT = null`.
5. Navigate to <http://localhost:3000/updater> and click the three buttons (Update meta, Update students, Update curriculums) one by one. This will populate `sis-db` with data from the new `sis-importer-db`. You might want to do each update a couple of times, as the database tables affect each other, and not all necessary data might be generated on the first try.
6. Follow the `sis-updater-worker` logs to ensure everything is running smoothly (`docker logs --follow oodikone_sis-updater-worker_1`). If you see any errors, try clicking the buttons again.
7. Run all the tests to ensure everything is working as expected (follow the instructions in the [README](../README.md) to run the tests more quickly). If any tests fail, update the test files accordingly.
anonyymioodi/docker-compose.yml
-6
View File
@@ -10,8 +10,6 @@ services:
POSTGRES_DB: kone-db
POSTGRES_HOST_AUTH_METHOD: trust
image: postgres:15.8
profiles:
- oodikone
sis-db:
<<: *defaults
@@ -31,8 +29,6 @@ services:
POSTGRES_DB: sis-importer-db
POSTGRES_HOST_AUTH_METHOD: trust
image: postgres:15.8
profiles:
- updater
user-db:
<<: *defaults
@@ -42,5 +38,3 @@ services:
POSTGRES_DB: user-db
POSTGRES_HOST_AUTH_METHOD: trust
image: postgres:15.8
profiles:
- oodikone
anonyymioodi/enrich_sampledata.sql
-133
View File
@@ -1,133 +0,0 @@
-- add these to sis-importer db by running:
-- docker exec -i sis-importer-db psql -U postgres sis-importer-db < enrich_sampledata.sql
-- if you need to run updater for these students only, see studentnumberlist in the end
-- this file
-- Add admission types (valintaperusteet) to studyrights for some of the computer
-- science 2018-2019 population students. See person_ids / student_number below
-- (in case you need to update dumps).
-- TODO: this should be moved into the sampledata creator script in
-- importer repo, but since this file contains studentnumbers and person ids, it's
-- better to keep this here
-- KP (koepisteet)
-- hy-hlo-1412620 | 013683126
-- hy-hlo-1485246 | 014340523
-- hy-hlo-32230770 | 010589388
-- hy-hlo-35626233 | 010775888
-- hy-hlo-40150144 | 011065995
UPDATE studyrights
SET admission_type_urn = 'urn:code:admission-type:kp'
WHERE person_id IN (
'hy-hlo-1412620',
'hy-hlo-1485246',
'hy-hlo-32230770',
'hy-hlo-35626233',
'hy-hlo-40150144'
);
-- TV (todistusvalinta)
-- hy-hlo-45783433 | 011321150
-- hy-hlo-48537946 | 010764653
-- hy-hlo-49801905 | 011194400
-- hy-hlo-53321302 | 011338871
-- hy-hlo-51223300 | 013951984
-- hy-hlo-53310137 | 012520688
-- hy-hlo-53621583 | 011196796
-- hy-hlo-53863064 | 011736884
-- hy-hlo-57711592 | 010262566
-- hy-hlo-57732872 | 013670647
UPDATE studyrights
SET admission_type_urn = 'urn:code:admission-type:tv'
WHERE person_id IN (
'hy-hlo-45783433',
'hy-hlo-48537946',
'hy-hlo-49801905',
'hy-hlo-53321302',
'hy-hlo-51223300',
'hy-hlo-53310137',
'hy-hlo-53621583',
'hy-hlo-53863064',
'hy-hlo-57711592',
'hy-hlo-57732872'
);
-- YP:
-- hy-hlo-67768022 | 011952325
-- hy-hlo-73740025 | 011736606
-- hy-hlo-74780917 | 012775433
-- hy-hlo-79615371 | 014113941
-- hy-hlo-78118455 | 014300615
UPDATE studyrights
SET admission_type_urn = 'urn:code:admission-type:yp'
WHERE person_id IN (
'hy-hlo-67768022',
'hy-hlo-73740025',
'hy-hlo-74780917',
'hy-hlo-79615371',
'hy-hlo-78118455'
);
-- AV:
-- hy-hlo-78777265 | 010779392
-- hy-hlo-81909550 | 011894153
-- hy-hlo-84571890 | 013897158
-- hy-hlo-80330834 | 015085926
-- hy-hlo-84775722 | 011791272
UPDATE studyrights
SET admission_type_urn = 'urn:code:admission-type:av'
WHERE person_id IN (
'hy-hlo-78777265',
'hy-hlo-81909550',
'hy-hlo-84571890',
'hy-hlo-80330834',
'hy-hlo-84775722'
);
-- M:
-- hy-hlo-87439827 | 011928456
-- hy-hlo-96612371 | 014418361
-- hy-hlo-84840261 | 010975895
-- hy-hlo-90330681 | 010522688
-- hy-hlo-96594603 | 013951573
UPDATE studyrights
SET admission_type_urn = 'urn:code:admission-type:m'
WHERE person_id IN (
'hy-hlo-87439827',
'hy-hlo-96612371',
'hy-hlo-84840261',
'hy-hlo-90330681',
'hy-hlo-96594603'
);
-- all studentnumbers for running updater:
013683126
014340523
010589388
010775888
011065995
011321150
010764653
011194400
011338871
013951984
012520688
011196796
011736884
010262566
013670647
011952325
011736606
012775433
014113941
014300615
010779392
011894153
013897158
015085926
011791272
011928456
014418361
010975895
010522688
013951573
anonyymioodi/old_readme.md
-71
View File
@@ -1,71 +0,0 @@
# Old readme of anonyymioodi
This is mostly obsolete, but may help with something sometimes.
## Anonyymioodi - anonymous database setup for oodikone
(name anonyymioodi for :culture: reasons)
This folder contains a guide and scripts to create anonymous database images for oodikone, mainly used for testing.
## How to create / modify database images
Creating / modifying images varies a bit depending on the db. Generally, to modify images, you can either use `psql` directly in docker by running `docker exec -it <dbname> psql -U postgres <dbname>` or start adminer with oodikone's docker-compose.
By default, previous db images from Toska Hub are used as a base. If needed, comment out Toska hub image and uncomment corresponding clean postgres image in docker-compose before running other steps.
See more details on how to create / modify databases below.
### Sis-importer-db
The following steps are for building a clean sis-importer-db image from scratch. If you want to just modify the existing image a bit, follow the steps defined for user-db / kone-db.
- See [sis-importer/importer-db-staging-sampletaker](https://github.com/UniversityOfHelsinkiCS/sis-importer/tree/master/importer-db-staging-sampletaker) and create `sis-importer-db.sqz` dump
- Move `sis-importer-db.sqz` to this folder
- Run `./create_container_from_dump.sh sis-importer-db`
- Enrich db with command in file `enrich_data.sql` either by piping command to container or by running command inside container.
### Sis-db:
The following steps are for building a clean sis-db image from scratch using oodikone's updater. If you want to just modify the existing image a bit, follow the steps defined for user-db / kone-db.
- Temporarily replace sis-db -service in oodikone's docker-compose.yml with sis-db -service from this repo's docker-compose.yml
- Ensure the newest sis-importer-db image is pulled from toska hub in oodikone repo
- Modify sis scheduler limit in config.js by setting: `module.exports.DEV_SCHEDULE_COUNT = null`
- Start updater, see oodikone readme for more info.
- Populate sis-db from sis-importer-db:
```
curl --request GET --url 'http://localhost:8082/v1/meta?token=dev' && \
curl --request GET --url 'http://localhost:8082/v1/programmes?token=dev' && \
curl --request GET --url 'http://localhost:8082/v1/students?token=dev'
```
(or run these from frontend. You might also want to follow sis-updater-workers logs to ensure things are running smoothly.)
- Clean up any extra stuff you don't want to keep in sis-db.
- Bonus: check that cypress tests run okay
### User-db / kone-db
- Temporarily replace the db service in oodikone's docker-compose.yml with the same db service from this folder's docker-compose.yml
- Ensure the newest user-db/kone-db image is pulled from the registry with `docker-compose pull`
- Modify db to wanted state. Tips:
- Run oodikone if you want to upgrade something from frontend (e.g. tags in kone-db, courses in sis-db etc.).
- Easy way to create new users to user-db is to modify dev user details in the frontend API config. A new user is created during login if the given uid is not present in user-db.
## Publishing the image
After completing the steps above and making sure your local version has the correct data, run `./commit_and_push_to_toska_hub.sh <dbname>` to publish the db image to Toska Hub.
## How to update Postgres versions
When Postgres is updated on the server, anon db images need to be updated to match server versions. Do following:
- Ensure you're using toska hub image in docker-compose and that you've pulled the newest image from toska hub: `docker-compose pull <dbname>`.
- Start database with docker-compose
- Create dump from database by running `docker exec -i <dbname> pg_dump -Fc -U postgres <dbname> > <dbname>.sqz`
- Close container with docker-compose down.
- Update the Postgres version in docker-compose.yml in this folder.
- Uncomment row with Postgres image and comment row with Toskahub image.
- Run `./create_container_from_dump.sh <dbname>`, followed by `./commit_and_push_to_toska_hub.sh <dbname>`