iritmaximus/oodikone
1
0
Fork 0
mirror of https://github.com/UniversityOfHelsinkiCS/oodikone.git synced 2026-10-12 19:03:39 +03:00
Code Issues Packages Projects Releases Wiki Activity

Update README.md

Browse Source
This commit is contained in:
Riku Rauhala
2024-06-05 09:37:32 +03:00
parent 7af4cd4668
commit 9aafd50a66
1 changed files with 46 additions and 43 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+46 -43
View File
@@ -6,21 +6,23 @@ An application for analyzing university data, running at [https://oodikone.helsi
To run Oodikone locally, you will need the following:
1. Applications:
- [Docker](https://docs.docker.com/install/) (version 20.10+),
- [Docker Compose](https://docs.docker.com/compose/install/) (version 1.29+) and
- [npm](https://docs.npmjs.com/cli/v7) (version 7+)
2. Access to Toska Docker Hub,
3. For real data setup, access to Toskas production servers.
4. For some features to work, access to Toskas service tokens
1. Applications
- [Docker](https://docs.docker.com/install/) (version `20.10+`),
- [Docker Compose](https://docs.docker.com/compose/install/) (version `1.29+`)
- [npm](https://docs.npmjs.com/cli/v7) (version `7+`)
2. Access to Toska Docker Hub
3. For real data setup, access to Toska's production servers
4. For some features to work, access to Toska's service tokens
## 🚀 Installation
Pre-installation steps:
- Clone this repo
- Login to toskas docker hub (see Toskas Gitlab for more information)
- Populate `.env.template` file with correct tokens and rename it to `.env`
- `git@github.com:UniversityOfHelsinkiCS/oodikone.git`
- Login to Toska's Docker Hub (see Toska's GitLab for more information)
- Copy the `.env.template` file to `.env` and populate with correct tokens
- `cp .env.template .env`
Then launch the CLI and follow the instructions:
@@ -32,23 +34,26 @@ _Please use a terminal at least 80 characters wide, the CLI is a bit rudimentary
What different CLI options do:
1. Set up oodikone from scratch:
1. Set up Oodikone from scratch
- Cleans up any previous installations
- Installs all needed npm packages locally
- Sets up pre-commit linting hooks.
- Pulls and builds all needed Docker images and sets up dockerized development environment.
- Setup happens with anonymized data, which developers should use by default.
- **Note:** Running this option cleans up all real data too, so please don't run option 2 before this.
2. Reset all real data:
- Sets up pre-commit linting hooks
- Pulls and builds all needed Docker images and sets up dockerized development environment
- Setup happens with anonymized data, which developers should use by default
- **Note:** Running this option cleans up all real data too, so please don' run option 2 before this.
2. Reset all real data
- Cleans up any previous real data databases
- Downloads needed database dumps from production servers.
- Creates real data databases and populates them with downloaded dumps.
3. Reset single database
- Like option 2, but only for single database. Leaves other databases untouched.
3. Reset a single database
- Like option 2, but only for single database
- Leaves other databases untouched
4. Restore data from dumps
- Like option 2, but skips downloading, and simply restores databases from the dumps downloaded earlier.
- Like option 2, but skips downloading, and simply restores databases from the dumps downloaded earlier
5. Docker system prune
- Runs `docker system prune -a && docker volume prune -a`. This will remove all your docker data, also related to any other applications than oodikone! It is sometimes necessary after strange errors due to caching.
- Runs `docker system prune -a && docker volume prune -a`
- This will remove all your Docker data, also related to any other applications than Oodikone!
- Sometimes necessary after strange errors due to caching
## ⌨️ Development
@@ -94,7 +99,7 @@ graph TB
end
```
In development, backend and updater use the same redis, but in production they are separate.
In development, backend and updater use the same Redis, but in production they are separate.
See also the [older graph](documentation/img/oodikone.png) for more information.
@@ -105,55 +110,55 @@ To read a bit more details about what updater-scheduler and updater-worker do, s
### Documentation
The `documentation` directory contains some notes on sis-db schema and is meant as a place for storing useful information on Oodikone to make life easier for future developers. Feel free to add anything relevant!
The `documentation` directory contains some notes on the sis-db schema and is meant as a place for storing useful information on Oodikone to make life easier for future developers. Feel free to add anything relevant!
For internal meeting notes and sensitive information, see Toska GitLab.
### Basics
The development environment is entirely configured in the `docker-compose.yml` file located in this repository. The file defines the services for oodikone's two main components: **oodikone** and **updater**.
The development environment is entirely configured in the `docker-compose.yml` file located in this repository. The file defines the services for Oodikone's two main components: **oodikone** and **updater**.
Running oodikone with real data requires separated databases and redis, which are defined in `docker-compose.real.yml` file. Otherwise real data development environment uses the configuration as anonymized data development environment.
Running Oodikone with real data requires separated databases and redis, which are defined in `docker-compose.real.yml` file. Otherwise real data development environment uses the configuration as anonymized data development environment.
Some useful commands are defined in `package.json` and can be run with `npm run <command>` as follows:
- `npm run oodikone`: starts oodikone with anonymized data
- `npm run oodikone:real`: starts oodikone with real data
- `npm run oodikone`: starts Oodikone with anonymized data
- `npm run oodikone:real`: starts Oodikone with real data
- `npm run updater`: starts updater with anonymized data
- `npm run updater:real`: starts oodikone with real data
- `npm run both`: starts oodikone and updater with anonymized data
- `npm run both:real`: starts oodikone and updater with real data
- `npm run updater:real`: starts updater with real data
- `npm run both`: starts Oodikone and updater with anonymized data
- `npm run both:real`: starts Oodikone and updater with real data
- `npm run docker:down`: stops the whole environment
- `npm run testupdater`: runs tests for updater
- `npm run flushredis`: clears redis, which forces big calculations to be redone in e.g. studyprogramme overview and faculty views.
- `npm run flushredis`: clears Redis, which forces big calculations to be redone in e.g. studyprogramme overview and faculty views
- `npm run rapodiff`: runs [Rapodiff](./services/backend/src/rapodiff/)
- `npm run install:local`: installs `node_modules` for each package (backend, frontend, sis-updater-scheduler and sis-updater-worker) locally
- `npm run install:docker`: installs `node_modules` for each package inside containers
- `npm run install:both`: runs both `npm run install:local` and `npm run install:docker`
Once you have ran setup for oodikone , you can just execute the first one (`npm run oodikone`). After starting and waiting for a while for containers to compile, oodikone can be accessed at [http://localhost:3000/](http://localhost:3000/) and Adminer (database investigation tool) at [http://localhost:5050/](http://localhost:5050/). Adminer requires you to login with username `postgres` and with any password you choose (for example `p`).
Once you have ran setup for Oodikone , you can just execute the first one (`npm run oodikone`). After starting and waiting for a while for containers to compile, Oodikone can be accessed at [http://localhost:3000](http://localhost:3000/) and Adminer (database investigation tool) at [http://localhost:5050](http://localhost:5050/). Adminer requires you to login with username `postgres` and with any password you choose (for example `p`).
### Run.sh script
As said, the development environment runs entirely inside docker containers. To keep `package.json` clean and not filled with predefined scripts, we have created a simple helper script called `./run.sh`. The script allows you to use docker-compose commands without the need to write long list of parameters. Try to run `./run.sh` in the root of the project and see what happens!
As said, the development environment runs entirely inside Docker containers. To keep `package.json` clean and not filled with predefined scripts, we have created a simple helper script called `./run.sh`. The script allows you to use docker-compose commands without the need to write long list of parameters. Try to run `./run.sh` in the root of the project and see what happens!
`./run.sh` is simply a wrapper script to run oodikone, updater or both services in either anon or real mode. If you take a look at `package.json`, you can see that most of the predefined scripts above use `./run.sh` under the hood.
`./run.sh` is simply a wrapper script to run Oodikone, updater or both services in either anon or real mode. If you take a look at `package.json`, you can see that most of the predefined scripts above use `./run.sh` under the hood.
It is recommended to spend some time to become familiar with `docker` and `docker compose` cli commands. You can then use them directly or with `./run.sh` wrapper. Here is some examples for day-to-day development situations:
- `./run.sh oodikone anon pull`: Pull all images related to oodikone development
- `./run.sh updater real up --build --force-recreate --detach`: Start updater detached (=in the background) in real data mode, but build new images before starting
- `./run.sh oodikone anon pull`: pull all images related to Oodikone development
- `./run.sh updater real up --build --force-recreate --detach`: start updater detached (in the background) in real data mode, but build new images before starting
- `docker compose ps`: view the containers in the running environment
- `docker compose logs frontend`: print logs for just frontend
- `docker compose logs --follow --tail 100 backend`: print last hundred rows of backend logs and begin to follow them in your terminal window
- `docker exec -it backend sh`: open bash terminal inside backend container
- `docker exec -it sis-db psql -U postgres sis-db-real`: open psql client to investigate sis real data database.
- `docker exec -it sis-db psql -U postgres sis-db-real`: open psql client to investigate sis real data database
### User types for development
By default, you're login as `mluukkai` dev user when running oodikone in development mode. If you want to debug with certain type of user, you can use mocking: go to "Users" -page, click on "edit" on user you want to use and then click icon on the right corner of user card. Another way is to mock user headers - see frontend's api configuration for how this is done.
By default, you're login as `mluukkai` dev user when running Oodikone in development mode. If you want to debug with certain type of user, you can use mocking: go to the "Users" page, click "edit" on the user you want to use and then click the icon on the right corner of user card. Another way is to mock user headers - see frontend's API configuration for how this is done.
Anon data contains some preset user types, corresponding to most usual user types in real data. These are used in testing too (see "Testing" below for more info).
Anonymous data contains some preset user types, corresponding to most usual user types in real data. These are used in testing too (see "Testing" below for more info).
### Linting
@@ -179,8 +184,8 @@ Continuous integration (CI) works with Github actions and is defined in workflow
- Oodikone setup for cypress and other tests in CI is defined in `docker-compose.ci.yml`. Take a look at this too if debugging github action workflows.
- Tests are run on every push
- After a successful test run, oodikone is deployed to staging
- After creating a release, oodikone is deployed to production
- After a successful test run, Oodikone is deployed to staging
- After creating a release, Oodikone is deployed to production
- Updater is deployed to production when pushed if tests pass
- Test databases can be updated, instructions in [anonyymioodi README](anonyymioodi/README.md)
@@ -200,13 +205,11 @@ Make sure you have your VPN on.
First: Try `npm run cli` option 1: _Set up oodikone from scratch_ and then option 2 to download and restore databases (or option 4 to skip downloading, if you've done it recently).
If that does not help, try option 5: _Docker system prune_. Notice that this clears up **all docker data**, including data related to other projects. If this is not ok, you'll need to identify the oodikone-specific docker volumes yourself and caches and clear those invididually.
If that does not help, try option 5: _Docker system prune_. Notice that this clears up **all Docker data**, including data related to other projects. If this is not ok, you'll need to identify the Oodikone-specific Docker volumes yourself and caches and clear those invididually.
## ✌🏼 Maintainers and contribution
[Toska](https://toska.dev) of course.
University of Helsinki
[Toska](https://toska.dev) - University of Helsinki
## Tietosuoja / data protection