mirror of
https://github.com/nowcommunity/nowdatabase.git
synced 2026-10-12 19:13:52 +03:00
reorganize docs
This commit is contained in:
Sampo Riitala
@@ -27,18 +27,18 @@ The new version is not deployed yet.
|
||||
|
||||
**Further documentation**
|
||||
|
||||
:rocket: [Setup app & restore db](documentation/setup.md) How to get the app up & running, and initialize database from sql-dumps
|
||||
:rocket: [Setup app & restore db](documentation/devops/setup.md) How to get the app up & running, and initialize database from sql-dumps
|
||||
|
||||
:keyboard: [List of all commands](documentation/commands.md) A comprehensive list of all of the most important commands
|
||||
:keyboard: [List of all commands](documentation/devops/commands.md) A comprehensive list of all of the most important commands
|
||||
|
||||
:wrench: [Technical explanations](documentation/guides/technical_explanations.md) Explanations of how things work, and guides for some more rare tasks like updating database-schema
|
||||
:wrench: [Technical explanations](documentation/devops/technical_explanations.md) Explanations of how things work, and guides for some more rare tasks like updating database-schema
|
||||
|
||||
:raised_hands: [Contributing & integration flow](documentation/guides/contributing.md) Where to commit? Where are tests run? How is the code deployed?
|
||||
:raised_hands: [Contributing & integration flow](documentation/contributing.md) Where to commit? Where are tests run? How is the code deployed?
|
||||
|
||||
:memo: [Frontend](documentation/components/frontend.md) Explanation of the frontend logic and how to add views
|
||||
:memo: [Frontend](documentation/frontend/components/frontend.md) Explanation of the frontend logic and how to add views
|
||||
|
||||
:mag_right: [Validators](frontend/src/shared/validators/) The [readme](frontend/src/shared/validators/README.md) explains how validators are written
|
||||
|
||||
:mag_right: [Class diagram](documentation/class_diagram.md) See the relations of relevant tables. Notice that it omits some unimportant tables and does not show most columns.
|
||||
|
||||
:earth_africa: [Map features](/documentation/components/map_features.md) Maps' functionality and development notes
|
||||
:earth_africa: [Map features](documentation/frontend/components/map_features.md) Maps' functionality and development notes
|
||||
|
||||
@@ -1,91 +0,0 @@
|
||||
# npm commands
|
||||
|
||||
## Initialization
|
||||
* `setup`
|
||||
* Installs node_modules to project root, backend and frontend
|
||||
* Sets up Prisma in backend
|
||||
* Creates all config files
|
||||
* `config:all`
|
||||
* Creates all config files: `.dev.env`, `.anon.env` and `.test.env`
|
||||
|
||||
## Running the application
|
||||
* `start`
|
||||
* Runs the application in the production environment
|
||||
* Opens backend, frontend and a MariaDB database (initialized from `data` directory)
|
||||
* Utilizes `.env`
|
||||
* `start:anon`
|
||||
* Runs the application in dev environment
|
||||
* Opens backend, frontend, phpMyAdmin and a MariaDB database (initialized from `test_data` directory)
|
||||
* Utilizes `.anon.env`
|
||||
* `dev`
|
||||
* Runs the application in development environment
|
||||
* Enables hotswap (docker compose doesn't need to be restarted on changes inside `backend/src` or `frontend/src`)
|
||||
* Note: When editing for example `package.json` the docker needs to be manually restarted.
|
||||
* Opens backend, frontend, phpMyAdmin, and a MariaDB database (initialized from `data` directory)
|
||||
* Utilizes `.dev.env`
|
||||
* To create test users visit `http://localhost:4000/test/create-test-users`
|
||||
|
||||
## Testing
|
||||
* `test:api`
|
||||
* Runs tests for backend API
|
||||
* Tests are located in `backend/src/api-tests`
|
||||
* Tests are executed inside Docker
|
||||
* Requires a database running with (preferrably, see last point) `test_data`
|
||||
* Utilizes `.test.env`
|
||||
* Anon version of the database is recommended be used by these tests
|
||||
* Resets the database with `test_data` multiple times!
|
||||
* `test:api:local`
|
||||
* Same as `test:api` but tests are executed locally
|
||||
* `test:e2e`
|
||||
* Runs e2e tests with cypress
|
||||
* Tests are located in `cypress/e2e`
|
||||
* Tests are executed inside Docker
|
||||
* Requires backend, frontend and database (with preferably `test_data`, see last point) to be running
|
||||
* Expects to find frontend at `<baseUrl>` defined in `cypress/config.js`
|
||||
* Expects to find database reset API-point at `<databaseResetUrl>` defined in `cypress.config.js`
|
||||
* Both can be also defined as a environment variable
|
||||
* Resets the database with `test_data` multiple times!
|
||||
* `test:e2e:local`
|
||||
* Same as `test:e2e` but tests are executed locally
|
||||
* Change the browser by running `npm run test:e2e:local -- --browser <YOUR_BROWSER_NAME>`
|
||||
* `test:unit`
|
||||
* Runs all unit tests defined in `backend/src/unit-tests` and `frontend/src/tests`
|
||||
* Doesn't require containers running
|
||||
* Executed locally
|
||||
* `test:{api,e2e}:windows`
|
||||
* Executes api or e2e command with (supposed) Windows support
|
||||
* Not actually tested on Windows: "should work"
|
||||
* For `api-tests` to work with Windows: inside `.test.env`, you need to change all instances of `localhost` to `host.docker.internal`.
|
||||
* These include `MARIADB_HOST`, `DATABASE_URL` and `LOG_DATABASE_URL`.
|
||||
* A short script that should do the same: `sed -i -e s|localhost|host.docker.internal|g .test.env`
|
||||
|
||||
|
||||
## Utilities
|
||||
* `check`
|
||||
* Runs linting and typescript checking for backend, frontend and cypress
|
||||
* `anon:down`
|
||||
* Closes all docker containers running anon versions
|
||||
* REMOVES `nowdb-db-anon` VOLUME PERMANENTLY
|
||||
* This is usually not a problem as the normal development environment uses a different volume
|
||||
* `dev:down`
|
||||
* Closes all docker containers running dev versions
|
||||
* REMOVES `nowdb-db-dev` VOLUME PERMANENTLY
|
||||
* `clean`
|
||||
* Removes all node_modules, build directories, prisma's autogenerated files, .nyc_output and coverage directories
|
||||
* `coverage`
|
||||
* Creates coverage report based on `.nyc_output`
|
||||
* Replaces the incorrect paths of the files from the path inside Docker container to the current working directory (`$PWD`)
|
||||
* Automatically run by `e2e` each time
|
||||
* `coverage:report`
|
||||
* Alias to `coverage` for cypress to create coverage while running tests
|
||||
* https://github.com/cypress-io/code-coverage?tab=readme-ov-file#custom-nyc-command
|
||||
|
||||
|
||||
### Be aware
|
||||
* Weird problems with docker?
|
||||
* Add `-- --build` after for example `npm run start:anon` to force rebuilding all containers
|
||||
* Sometimes docker doesn't rebuild containers when needed and it leads to hard-to-find bugs
|
||||
|
||||
* Difference between `cmd1 && cmd2` and `cmd1; cmd2`
|
||||
* `&&`: `cmd2` is executed only when `cmd1` returns 0 (=is succesful)
|
||||
* `;`: `cmd2` is always executed (after `cmd1`)
|
||||
@@ -1,3 +0,0 @@
|
||||
# EditableTable
|
||||
|
||||
[EditableTable](../../frontend/src/components/DetailView/common/EditableTable.tsx) is a table component that allows the user to add and remove rows during editing. Some EditableTables might be tables of other entities, like the Species-table in the Species-tab of a Locality. In these cases, there is a link to the detail view of those entities on every row. To add links to an EditableTable, give it the `idFieldName` and `url` parameters.
|
||||
@@ -1,3 +0,0 @@
|
||||
# SimpleTable
|
||||
|
||||
[SimpleTable](../../frontend/src/components/DetailView/common/SimpleTable.tsx) is a simple component as its name suggests. If it is a table of entities, it can have links to the detail views of those entities. To add links to a SimpleTable, give it the `idFieldName` and `url` parameters.
|
||||
@@ -1,5 +0,0 @@
|
||||
# TableView
|
||||
|
||||
[TableView](../../frontend/src/components/TableView/TableView.tsx) is a generic component that takes in column definitions, data, and renders a [material-react-table (click for docs)](https://www.material-react-table.com/). It also handles the tables state management: for example saving column filters into url, and restoring them on first render.
|
||||
|
||||
The table itself is constructed with the useMaterialReactTable-hook. The toolbar at the top is defined in the [helpers](../../frontend/src/components/TableView/helpers.tsx) file and rendered using the `renderToolbarInternalActions` option, and it contains buttons for showing/hiding columns and showing the table in fullscreen mode, as well as exporting the content of the table. If the toolbar needs to be changed in the future (e.g. adding more rows to it) checkout the documentation for rendering a custom toolbar [here](https://www.material-react-table.com/docs/guides/toolbar-customization#override-with-custom-toolbar-components) since the current method doesn't really allow any major changes to its layout.
|
||||
@@ -0,0 +1,84 @@
|
||||
### Contributing and CI / CD
|
||||
|
||||
**Contributing workflow**
|
||||
|
||||
- See an issue on Github Issues or Projects tab that you want to fix/implement.
|
||||
- Assign the issue to yourself and start work on the issue on a new local branch. _REMEMBER TO PULL MAIN BEFORE CREATING THE BRANCH!_ Just for your own sake.
|
||||
- Move the task on Github Projects to "In progress".
|
||||
- Work on the task on hand while making small commits along the way.
|
||||
- Finish work? Remember to check if all tests still pass!
|
||||
- Test passing, it's now time to push the changes to a new branch on Github and create a pull request to main.
|
||||
- Now wait for all of the checks to pass (and fixing them if they do not pass) and then it's a waiting game for someone else to approve the pull request.
|
||||
- When merged you can safely delete the local branch, pull main and admire your work.
|
||||
- Don't forget to close your issue and to tag the pull request in the issue by just adding #PR_NUMBER. Also move the task on GH Projects to "Done".
|
||||
- Repeat.
|
||||
|
||||
**Backlog and taskboard**
|
||||
This is for contributors that have access to the nowdatabase Github Project.
|
||||
|
||||
Creating issues.
|
||||
|
||||
- Come up with a feature / find a bug
|
||||
- In case of a feature, write an user story for it. / Bugs => Write an issue that clearly describes what happens, when it happens and under what circumstances it happens.
|
||||
- Create an issue to nowdatabase repo that has the user story / short description of the bug as the title.
|
||||
- Next think of smaller tasks that need to be done in order to implement the wanted feature / fix the bug. Don't forget to add tests as a task!
|
||||
- Now create issues for those tasks too. If there is relevant information regarding the task, it can be written to the description or added as a comment under the issue. _Hint, take note of the issue numbers that Github automatically assigns._
|
||||
- Create a checkbox by typing `- [ ]` and link the tasks to the user story / bug issue by referencing the issue by it's number: `#1234`. With the example issue number we would get: `- [ ] #1234`.
|
||||
|
||||
Now the issue is ready for the backlog and the taskboard.
|
||||
|
||||
- Open the original user story / bug issue and look to the menu on the right and find a section `Projects` and under that `nowdatabase`.
|
||||
- Click on the arrow that expands the view to include all of the fields available on the Github Project. _Don't go to the Github Projects website yet._
|
||||
- Set the priority, size and category according to the issue at hand.
|
||||
- Do this for all tasks listed in the issue.
|
||||
- After that all of the issues should end up to the Github Project `nowdatabase`. All of the issues should be seen on the `Product backlog`, User storys on `Backlog` and Tasks on `Taskboard`.
|
||||
|
||||
During sprint planning.
|
||||
|
||||
- In `Backlog` move all of the selected user stories to `Sprint Backlog` and set the `Iteration` field to the current sprint.
|
||||
- Do this also for all of the tasks and bug reports.
|
||||
|
||||
Handling the Project while developing.
|
||||
|
||||
- When you start work on an user story / bug, change the matching entries on `Backlog` and `Taskboard` to `In Progress`. You can also assign the user story / bug / task to yourself by clicking on the entry and by selecting yourself on the `Assignees` field on the right.
|
||||
- When you finish a task, move it straght to `Done`.
|
||||
- User story can be moved to `Done` when all of the tasks and tests are done for that story.
|
||||
- All user stories that have been demoed approved by the client are moved to the `Approved`
|
||||
|
||||
**Tests & linting**
|
||||
|
||||
Dynamic tests
|
||||
|
||||
- End-to-end tests with cypress
|
||||
- `cypress` folder in the repository root
|
||||
- Main purpose is to check that mostly correct things are being shown and that views don't crash. Feel free to implement more of these.
|
||||
- To run, have the anon-version running, then in repo root type `npm run test:e2e` to run in headless mode, or to see the tests running type `npm run cypress open`.
|
||||
- API-tests using node test runner
|
||||
- `backend/src/api-tests` folder. Runs tests whose filename match `*.test.ts`.
|
||||
- More critical features should have api-tests, such as write & logging operations as well as user access.
|
||||
- To run, type `npm run test:api` in repo root. These tests edit the database and assume that on each run the database is in initial state, so after running the tests or editing it yourself, you must reset it before running the tests again: `npm run reset-anon-db`
|
||||
|
||||
Static tests
|
||||
|
||||
- TypeScript compiler
|
||||
- Checks the basic TypeScript correctness with settings configured in tsconfig-files
|
||||
- ESLint checks
|
||||
- More code quality checks, also related to code style, configured by `.eslintrc.cjs` files
|
||||
- To run both static checks in both `frontend` and `backend`, do `npm run check`. Notice that this runs the commands sequentially and will stop at errors; so after fixing one error you must always run again and it may find other errors.
|
||||
|
||||
**GitHub Actions workflows**
|
||||
|
||||
- Linting and tsc is ran in GitHub Actions on push and pull request to any branch.
|
||||
- End-to-end tests are ran in GitHub as well. They use anonymous data.
|
||||
|
||||
**Deployment**
|
||||
|
||||
- Any code pushes to `main`-branch will trigger the `build` workflow. This will build the docker images (staging-version) and push them to GitHub image registry. Staging-OpenShift will then notice this once every 15 minutes, pull the images, and deploy them to staging. Though now there is a bug in OpenShift which prevents this so you must trigger the image update there yourself.
|
||||
- Production versions are built & deployed whenever you create a Release.
|
||||
|
||||
**Naming conventions**
|
||||
|
||||
- Types, React-components, classes: `PascalCase`
|
||||
- Field names that come directly from database columns or tables are the same name as in the db: `snake_case`
|
||||
- Environment variables: `SCREAMING_SNAKE_CASE`
|
||||
- Everything else - variables, functions etc: `camelCase`
|
||||
@@ -0,0 +1,95 @@
|
||||
# npm commands
|
||||
|
||||
## Initialization
|
||||
|
||||
- `setup`
|
||||
- Installs node_modules to project root, backend and frontend
|
||||
- Sets up Prisma in backend
|
||||
- Creates all config files
|
||||
- `config:all`
|
||||
- Creates all config files: `.dev.env`, `.anon.env` and `.test.env`
|
||||
|
||||
## Running the application
|
||||
|
||||
- `start`
|
||||
- Runs the application in the production environment
|
||||
- Opens backend, frontend and a MariaDB database (initialized from `data` directory)
|
||||
- Utilizes `.env`
|
||||
- `start:anon`
|
||||
- Runs the application in dev environment
|
||||
- Opens backend, frontend, phpMyAdmin and a MariaDB database (initialized from `test_data` directory)
|
||||
- Utilizes `.anon.env`
|
||||
- `dev`
|
||||
- Runs the application in development environment
|
||||
- Enables hotswap (docker compose doesn't need to be restarted on changes inside `backend/src` or `frontend/src`)
|
||||
- Note: When editing for example `package.json` the docker needs to be manually restarted.
|
||||
- Opens backend, frontend, phpMyAdmin, and a MariaDB database (initialized from `data` directory)
|
||||
- Utilizes `.dev.env`
|
||||
- To create test users visit `http://localhost:4000/test/create-test-users`
|
||||
|
||||
## Testing
|
||||
|
||||
- `test:api`
|
||||
- Runs tests for backend API
|
||||
- Tests are located in `backend/src/api-tests`
|
||||
- Tests are executed inside Docker
|
||||
- Requires a database running with (preferrably, see last point) `test_data`
|
||||
- Utilizes `.test.env`
|
||||
- Anon version of the database is recommended be used by these tests
|
||||
- Resets the database with `test_data` multiple times!
|
||||
- `test:api:local`
|
||||
- Same as `test:api` but tests are executed locally
|
||||
- `test:e2e`
|
||||
- Runs e2e tests with cypress
|
||||
- Tests are located in `cypress/e2e`
|
||||
- Tests are executed inside Docker
|
||||
- Requires backend, frontend and database (with preferably `test_data`, see last point) to be running
|
||||
- Expects to find frontend at `<baseUrl>` defined in `cypress/config.js`
|
||||
- Expects to find database reset API-point at `<databaseResetUrl>` defined in `cypress.config.js`
|
||||
- Both can be also defined as a environment variable
|
||||
- Resets the database with `test_data` multiple times!
|
||||
- `test:e2e:local`
|
||||
- Same as `test:e2e` but tests are executed locally
|
||||
- Change the browser by running `npm run test:e2e:local -- --browser <YOUR_BROWSER_NAME>`
|
||||
- `test:unit`
|
||||
- Runs all unit tests defined in `backend/src/unit-tests` and `frontend/src/tests`
|
||||
- Doesn't require containers running
|
||||
- Executed locally
|
||||
- `test:{api,e2e}:windows`
|
||||
- Executes api or e2e command with (supposed) Windows support
|
||||
- Not actually tested on Windows: "should work"
|
||||
- For `api-tests` to work with Windows: inside `.test.env`, you need to change all instances of `localhost` to `host.docker.internal`.
|
||||
- These include `MARIADB_HOST`, `DATABASE_URL` and `LOG_DATABASE_URL`.
|
||||
- A short script that should do the same: `sed -i -e s|localhost|host.docker.internal|g .test.env`
|
||||
|
||||
## Utilities
|
||||
|
||||
- `check`
|
||||
- Runs linting and typescript checking for backend, frontend and cypress
|
||||
- `anon:down`
|
||||
- Closes all docker containers running anon versions
|
||||
- REMOVES `nowdb-db-anon` VOLUME PERMANENTLY
|
||||
- This is usually not a problem as the normal development environment uses a different volume
|
||||
- `dev:down`
|
||||
- Closes all docker containers running dev versions
|
||||
- REMOVES `nowdb-db-dev` VOLUME PERMANENTLY
|
||||
- `clean`
|
||||
- Removes all node_modules, build directories, prisma's autogenerated files, .nyc_output and coverage directories
|
||||
- `coverage`
|
||||
- Creates coverage report based on `.nyc_output`
|
||||
- Replaces the incorrect paths of the files from the path inside Docker container to the current working directory (`$PWD`)
|
||||
- Automatically run by `e2e` each time
|
||||
- `coverage:report`
|
||||
- Alias to `coverage` for cypress to create coverage while running tests
|
||||
- https://github.com/cypress-io/code-coverage?tab=readme-ov-file#custom-nyc-command
|
||||
|
||||
### Be aware
|
||||
|
||||
- Weird problems with docker?
|
||||
|
||||
- Add `-- --build` after for example `npm run start:anon` to force rebuilding all containers
|
||||
- Sometimes docker doesn't rebuild containers when needed and it leads to hard-to-find bugs
|
||||
|
||||
- Difference between `cmd1 && cmd2` and `cmd1; cmd2`
|
||||
- `&&`: `cmd2` is executed only when `cmd1` returns 0 (=is succesful)
|
||||
- `;`: `cmd2` is always executed (after `cmd1`)
|
||||
@@ -0,0 +1,42 @@
|
||||
### Database
|
||||
|
||||
**DB-dump restoring process**
|
||||
|
||||
The mariadb-container by default executes all .sql files it finds in the containers `/docker-entrypoint-initdb.d/` directory when the container is run the first time. The `data` directory is mounted there. It contains an initialization file (`restore_and_create.sql`) which will then restore the database from the sql-dumpfiles in `sqlfiles/`, and also create a user `'now_test'@'localhost'` with a password defined as MARIADB_PASSWORD in the .db.dev.env file for the database.
|
||||
|
||||
**Accessing database inside docker**
|
||||
|
||||
`docker exec -it nowdb-db mariadb -u now_test -p --database now_test`
|
||||
|
||||
Enter the password defined in `.dev.env`.
|
||||
|
||||
**Updating prisma-schema**
|
||||
|
||||
If the database schema is changed, prisma-schema must be also updated. The smart prisma-way would be to run migrations via prisma, but I did this:
|
||||
|
||||
- The database was modified directly in SQL
|
||||
- Anon-db dump was updated to reflect the changes by hand (for both the table structure and the data inputted to it)
|
||||
- Also the test-db dump used locally was changed by hand, but this may be more complicated if theres a lot of data - in which case a new dump must be taken from the db after changes
|
||||
- Notice that for the db-container to initialize data, you must remove the container and then the volume and rebuild
|
||||
- I ran the dev version of the app `npm run dev`
|
||||
- I ran `docker exec -it nowdb-backend-dev npx prisma db pull`
|
||||
- The schema was updated and the changes can be committed to github.
|
||||
- After any schema update, to run the app, the **Prisma client must be generated again** (the client is located in "prisma/generated"). I did it by running `npm run prisma` in `backend/` directory (outside of docker). Running it inside the container would not re-generate it correctly, not sure why.
|
||||
|
||||
Prisma documentation: https://www.prisma.io/docs/orm/reference/prisma-cli-reference#db-pull
|
||||
|
||||
**Updating test data**
|
||||
|
||||
Sometimes you may want to add some test user or entries into the anonymous data in order to test something better. This can be done by `mariadb-dump` program which is included in the `mariadb` official docker image:
|
||||
|
||||
- Reset the anonymous data to the initial state: `npm run reset-anon-db`
|
||||
- Do the necessary edits, either via code, UI, or by executing commands in the mariadb cli.
|
||||
- Take a dump of the relevant tables: `docker exec nowdb-db-anon mariadb-dump <db_name> -uroot -padmin > ./<db_name>.sql`
|
||||
- The db names are probably "now_test" or "now_log_test".
|
||||
|
||||
* Replace the .sql-files in `test_data/sqlfiles/` with the new dump you created.
|
||||
* Commit and push the new dump - unless you did this for the staging-data dump, in which case you shouldn't commit and push.
|
||||
|
||||
**Mariadb-connector and transactions**
|
||||
|
||||
Prisma does not have proper support for connecting to multiple databases, and transactionality between them is therefore impossible. We need transactional write to both now and now_log databases. Therefore we use mariadb-connector for the write operations that need to be logged into the log-database.
|
||||
@@ -0,0 +1,37 @@
|
||||
### Docker
|
||||
|
||||
#### Dockerfile
|
||||
|
||||
There are two variants of Dockerfile used for both backend and frontend:
|
||||
|
||||
1. Production (`Dockerfile`)
|
||||
- Installs only packages required in production (no devDependencies)
|
||||
- Copies only required files to the image
|
||||
- Tries to be minimal so the image would be fast and small
|
||||
2. Development (`dev.Dockerfile`)
|
||||
- Installs all packages from `package.json`
|
||||
- Copies all files from `backend` to the image (not counting `.dockerignore`d files)
|
||||
- Has multiple stages (dev and api-tests)
|
||||
- Learn more about multi-stage Dockerfiles: https://docs.docker.com/build/building/multi-stage/
|
||||
- Includes everything that is required / helps with development and testing
|
||||
|
||||
#### Docker compose
|
||||
|
||||
There are three variants of docker compose used in this project.
|
||||
|
||||
1. Production (`docker-compose.yml`)
|
||||
- Uses `{backend,frontend}/Dockerfile` for both backend and frontend
|
||||
- Uses `.env` for environment variables
|
||||
- Uses `data` for database initialization
|
||||
- Doesn't start phpMyAdmin
|
||||
- Can be started with `npm run start`
|
||||
2. Dev (`docker-compose.dev.yml`)
|
||||
- Uses `{backend,frontend}/dev.Dockerfile`
|
||||
- Uses `.dev.env`
|
||||
- Uses `data` for database initialization
|
||||
- Can be started with `npm run dev`
|
||||
3. Anon (`docker-compose.anon.yml`)
|
||||
- Uses `{backend,frontend}/dev.Dockerfile`
|
||||
- Uses `.anon.env`
|
||||
- Uses `test_data` for database initialization
|
||||
- Can be started with `npm run start:anon`
|
||||
@@ -0,0 +1,23 @@
|
||||
### Github actions
|
||||
|
||||
This project utilizes Github actions for linting, testing and building the project. At the moment
|
||||
pull requests cannot be merged if the all of the pull-request-actions do not pass.
|
||||
|
||||
The following actions are run on every pull request:
|
||||
|
||||
- `lint.yaml`
|
||||
- Lints the whole project and fails if there is linting issues
|
||||
- `api-tests.yaml`
|
||||
- Runs api tests inside Docker, passes if all tests pass
|
||||
- `e2e.yaml`
|
||||
- Runs e2e tests inside Docker, passes if all tests pass
|
||||
- `unit-tests.yaml`
|
||||
- Runs all unit tests, passes if all tests pass
|
||||
|
||||
And the following are run on only merges to main:
|
||||
|
||||
- `build.yaml`
|
||||
- Builds staging version of both backend and frontend and creates Github package out of
|
||||
both of them. If the triggering event happens to be a release, a production package is made.
|
||||
- `phpmyadmin.yaml`
|
||||
- Builds the patched phpMyAdmin image and creates a Github package
|
||||
@@ -0,0 +1,25 @@
|
||||
### Openshift
|
||||
|
||||
Staging and production is deployed in Openshift. You should read more about Openshift's own documentation
|
||||
and Kubernetes documentation to understand more.
|
||||
|
||||
#### Updating deployments
|
||||
|
||||
Auto-deploying pods when new images are pulled from Github doesn't work at the moment because of
|
||||
a bug in Openshift. Until it is fixed the deployments have to be updated manually.
|
||||
|
||||
1. Select role as developer if there is an option. Otherwise skip this.
|
||||
2. Go to `Topology`
|
||||
3. Select backend or frontend
|
||||
4. From `Actions` dropdown menu select "`Edit Deployment`"
|
||||
5. Select `Form view` if not selected.
|
||||
6. Tick `Auto deploy when new Image is available`
|
||||
7. Save
|
||||
|
||||
If there was a new version of an image, a new pod should be created and the old one destroyed after
|
||||
the new pod is up and running.
|
||||
|
||||
Same goes for production.
|
||||
|
||||
Trying to update the staging right after merging a pull request might not always work because
|
||||
Openshift pulls the new image from Github packages roughly every 15mins.
|
||||
@@ -6,7 +6,7 @@
|
||||
2. Place the database dump files `now_test.sql`, `now_log_test.sql` and `now_view.sql` inside it.
|
||||
3. Notice, that the container will have to be able to read these files and execute `data/restore_and_create_user.sql`, so you may have to adjust the necessary rights depending on your operating system.
|
||||
|
||||
See [technical explanations](./guides/technical_explanations.md) for how this works.
|
||||
See [database docs](./database.md) for how this works.
|
||||
|
||||
**Init the project** :rocket:
|
||||
|
||||
@@ -20,6 +20,7 @@ Requirements: npm and some modern installation of Docker engine with Docker comp
|
||||
**Errors while setting up**
|
||||
|
||||
Before any other debugging steps:
|
||||
|
||||
1. Make sure main branch is up to date
|
||||
2. Make sure again, you didn't actually check did you...
|
||||
3. Re-setup your environment with `npm run setup`
|
||||
@@ -29,19 +30,21 @@ Before any other debugging steps:
|
||||
7. Continue with other debugging steps
|
||||
8. Repeat
|
||||
|
||||
***Database***
|
||||
**_Database_**
|
||||
If the database fails to start with an error message `Cannot open directory /docker-entrypoint-initdb.d/. Permission denied`, the problem is most likely permissions. `dev` uses `data` directory and `anon` + `test` uses `test_data`.
|
||||
Both directories and the files inside need the permissions `read` and `execute`. In Linux/GNU this can be achieved by running `chmod 755 DIRECTORY -R` where DIRECTORY is the wanted directory, `test_data` or `data`.
|
||||
(NOTE: This gives `write` for all files and directories for current user. You can disable this by changing the first number to match the other two numbers.)
|
||||
|
||||
***Backend or frontend***
|
||||
* `Cannot find module /usr/src/app/backend/src/index.ts`
|
||||
**_Backend or frontend_**
|
||||
|
||||
- `Cannot find module /usr/src/app/backend/src/index.ts`
|
||||
|
||||
- Permission error with `backend/src`
|
||||
- Fixed by running `find backend/src -type d -exec chmod 755 {} \; && find backend/src -type f -exec chmod 644 {} \;` (on Linux).
|
||||
- The command gives permissions `read` and `execute` for directories and `read` for files inside `backend/src`.
|
||||
- NOTE: This gives `write` for all files and directories for current user. You can disable this by changing the first number to match the others.
|
||||
|
||||
* `Cannot find module ../../../frontend/src/shared/types`
|
||||
- `Cannot find module ../../../frontend/src/shared/types`
|
||||
- Permission error with `frontend/src`
|
||||
- Fixed by running `find frontend/src -type d -exec chmod 755 {} \; && find frontend/src -type f -exec chmod 644 {} \;` (on Linux).
|
||||
- The command gives permissions `read` and `execute` for directories and `read` for files inside `frontend/src`.
|
||||
@@ -0,0 +1,3 @@
|
||||
# EditableTable
|
||||
|
||||
[EditableTable](../../../frontend/src/components/DetailView/common/EditableTable.tsx) is a table component that allows the user to add and remove rows during editing. Some EditableTables might be tables of other entities, like the Species-table in the Species-tab of a Locality. In these cases, there is a link to the detail view of those entities on every row. To add links to an EditableTable, give it the `idFieldName` and `url` parameters.
|
||||
@@ -0,0 +1,3 @@
|
||||
# SimpleTable
|
||||
|
||||
[SimpleTable](../../../frontend/src/components/DetailView/common/SimpleTable.tsx) is a simple component as its name suggests. If it is a table of entities, it can have links to the detail views of those entities. To add links to a SimpleTable, give it the `idFieldName` and `url` parameters.
|
||||
@@ -0,0 +1,29 @@
|
||||
# TableView
|
||||
|
||||
[TableView](../../../frontend/src/components/TableView/TableView.tsx) is a generic component that takes in column definitions, data, and renders a [material-react-table (click for docs)](https://www.material-react-table.com/). It also handles the tables state management: for example saving column filters into url, and restoring them on first render.
|
||||
|
||||
The table itself is constructed with the useMaterialReactTable-hook. The toolbar at the top is defined in the [helpers](../../../frontend/src/components/TableView/helpers.tsx) file and rendered using the `renderToolbarInternalActions` option, and it contains buttons for showing/hiding columns and showing the table in fullscreen mode, as well as exporting the content of the table. If the toolbar needs to be changed in the future (e.g. adding more rows to it) checkout the documentation for rendering a custom toolbar [here](https://www.material-react-table.com/docs/guides/toolbar-customization#override-with-custom-toolbar-components) since the current method doesn't really allow any major changes to its layout.
|
||||
|
||||
### Setting up a new table view
|
||||
|
||||
Following things are needed to create a new page with a table (examples are all from Cross-Search):
|
||||
|
||||
- Backend:
|
||||
|
||||
- The [App](../../../backend/src/app.ts) should use the route.
|
||||
- A [route](../../../backend/src/routes/crossSearch.ts) function calling the service.
|
||||
- A [service](../../../backend/src/services/crossSearch.ts) to get the data from the database. There might be a necessities to clean the result, filter the result by project ids (pid) and user's permissions, and create a temp type for typescript.
|
||||
- Earlier we used prisma.findMany-function to fetch the data, but with cross search's 90k rows, the GET-call took approx 40s in staging environment. We found out that prisma.rawSQL with LEFT JOIN would reduce time of the call from 10s to 3s in dev environment. This reduced the call time for staging to approx 10s. Could possible be implemented for other tables also for better performance.
|
||||
|
||||
- Frontend:
|
||||
- Create necessary types in [shared types](../../../frontend/src/shared/types)
|
||||
- Create a button for the component in [navigation bar](../../../frontend/src/components/NavBar.tsx)
|
||||
- Create a relevant page-object for [Pages](../../../frontend/src/components/pages.tsx)
|
||||
- Add a route for the component to [App](../../../frontend/src/App.tsx)
|
||||
- Create a [component](../../../frontend/src/components/CrossSearch/CrossSearchTable.tsx) (note that cross search borrows the detail parts from localities).
|
||||
- Table view:
|
||||
The [TableView](../../../frontend/src/components/TableView/TableView.tsx) is defined via variables columns and visibleColumns. Columns in the most simple form only require an accessorKey and a header. If an accessorFn is used, a separate id is required as well. The accessorkey functions as a key to fetch data as well as an id to reference the column as well. The variable visibleColumns takes in accessorKeys or ids for columns, set to false, to be hidden by default.
|
||||
- Non-default filtering (eg filterVariant 'range' for numbers or filterFn 'contains' for text substrings) require no null values in the data for given column. It is needed to use accessorFn to map nulls to an empty string for example.
|
||||
- non-hideable columns can be set by setting parameter `enableHiding: false`
|
||||
|
||||
More info can be found from official [MRT docs](https://www.material-react-table.com/)
|
||||
+1
-1
@@ -18,6 +18,6 @@ If `disabled` is set to true, the EditableTextField cannot be clicked.
|
||||
|
||||
If `readonly` is set to true, the user can't change the value of the input field directly.
|
||||
|
||||
The `handleSetEditData` function is used to set the editData state using setEditData, instead of the default handling. This is useful if you want changing the text field to have an effect on some other input field, e.g. another component. An example use is `handleDmsLatChange` (and three similar functions) in [LocalityTab](../../frontend/src/components/Locality/Tabs/LocalityTab.tsx)
|
||||
The `handleSetEditData` function is used to set the editData state using setEditData, instead of the default handling. This is useful if you want changing the text field to have an effect on some other input field, e.g. another component. An example use is `handleDmsLatChange` (and three similar functions) in [LocalityTab](../../../../frontend/src/components/Locality/Tabs/LocalityTab.tsx)
|
||||
|
||||
If the selected value is invalid, the EditableTextField will show a red text underneath it explaining the issue. This is done by getting the validator from the current context (if one exists), and then passing it the current `editData` (also from context) and the `field` value. The validator will check if the value in editData corresponding to `field` is valid.
|
||||
+1
-3
@@ -15,6 +15,4 @@ The values that the RadioSelector supports are strings, numbers, and booleans as
|
||||
|
||||
If a `defaultValue` is not given, the RadioSelector will choose the first option's value as the default value. This means that a RadioSelector always has a value. The selector will also immediately add this default value to the editData.
|
||||
|
||||
The `handleSetEditData` function is used to set the editData state using setEditData, instead of the default handling. This is useful if you want changing the radio selector to have an effect on some other input field, e.g. another component. An example use is `handleDateMethChange` in [AgeTab](../../frontend/src/components/Locality/Tabs/AgeTab.tsx)
|
||||
|
||||
|
||||
The `handleSetEditData` function is used to set the editData state using setEditData, instead of the default handling. This is useful if you want changing the radio selector to have an effect on some other input field, e.g. another component. An example use is `handleDateMethChange` in [AgeTab](../../../../frontend/src/components/Locality/Tabs/AgeTab.tsx)
|
||||
@@ -2,9 +2,9 @@
|
||||
|
||||
**Data tables**
|
||||
|
||||
- [TableView](../../frontend/src/components/TableView/TableView.tsx) is a generic component that takes in column definitions, data, and renders a [material-react-table (click for docs)](https://www.material-react-table.com/). It also handles the tables state management: for example saving column filters into url, and restoring them on first render. See [creating a new tableview](../guides/creating_new_tableview.md) for more details, and check out the [TableView](./tableview.md) documentation too.
|
||||
- [TableView](../../frontend/src/components/TableView/TableView.tsx) is a generic component that takes in column definitions, data, and renders a [material-react-table (click for docs)](https://www.material-react-table.com/). It also handles the tables state management: for example saving column filters into url, and restoring them on first render. See [TableView docs](./components/TableView.md) for more details.
|
||||
|
||||
- [DetailView](../../frontend/src/components/DetailView/)-folder has components that are used by the detailed view: The view which allows looking at and editing a single data item, for example a locality. The DetailView-component is generic, allowing different data types to use it.
|
||||
- [DetailView](../../frontend/src/components/DetailView)-folder has components that are used by the detailed view: The view which allows looking at and editing a single data item, for example a locality. The DetailView-component is generic, allowing different data types to use it.
|
||||
|
||||
**Example of a detail view rendering tree**
|
||||
|
||||
@@ -59,11 +59,11 @@ This ensures that when multiple components try to update the state at the same t
|
||||
|
||||
**Detailed documentation for editingComponents**
|
||||
|
||||
- [RadioSelector](../editingComponents/RadioSelector.md)
|
||||
- [DropDownSelector](../editingComponents/DropDownSelector.md)
|
||||
- [FieldWithTableSelection](../editingComponents/FieldWithTableSelection.md)
|
||||
- [TimeBoundSelection](../editingComponents/TimeBoundSelection.md)
|
||||
- [EditableTextField](../editingComponents/EditableTextField.md)
|
||||
- [RadioSelector](./components/editingComponents/RadioSelector.md)
|
||||
- [DropDownSelector](./components/editingComponents/DropdownSelector.md)
|
||||
- [FieldWithTableSelection](./components/editingComponents/FieldWithTableSelection.md)
|
||||
- [TimeBoundSelection](./components/editingComponents/TimeBoundSelection.md)
|
||||
- [EditableTextField](./components/editingComponents/EditableTextField.md)
|
||||
|
||||
**Documentation for validators**
|
||||
|
||||
@@ -8,6 +8,6 @@ Both editing and creating new entries go via same route in the app. Following th
|
||||
- A validator and a write functions in [services](../../backend/src/services/species.ts) for the route to call.
|
||||
|
||||
- Frontend:
|
||||
- [Spesific validators](../../frontend/src/shared/validators/species.ts) are defined here.
|
||||
- [Specific validators](../../frontend/src/shared/validators/species.ts) are defined here.
|
||||
- An api-endpoint for edit needs to be defined in [reducers](../../frontend/src/redux/speciesReducer.ts) (this serves for creating a new entry as well)
|
||||
- An onWrite-function needs to be defined with the help of the api-endpoint and call navigate-function to end up in the [details-view](../../frontend/src/components/Species/SpeciesDetails.tsx) of the edited/created entry. (At the moment there's a 15ms time-out before the navigation for species as sometimes it tried to navigate before knowing where to (no id yet). This could be investigated if this is a more general bug or something weird happening only in species)
|
||||
|
||||
@@ -1,79 +0,0 @@
|
||||
### Contributing and CI / CD
|
||||
|
||||
**Contributing workflow**
|
||||
+ See an issue on Github Issues or Projects tab that you want to fix/implement.
|
||||
+ Assign the issue to yourself and start work on the issue on a new local branch. *REMEMBER TO PULL MAIN BEFORE CREATING THE BRANCH!* Just for your own sake.
|
||||
+ Move the task on Github Projects to "In progress".
|
||||
+ Work on the task on hand while making small commits along the way.
|
||||
+ Finish work? Remember to check if all tests still pass!
|
||||
+ Test passing, it's now time to push the changes to a new branch on Github and create a pull request to main.
|
||||
+ Now wait for all of the checks to pass (and fixing them if they do not pass) and then it's a waiting game for someone else to approve the pull request.
|
||||
+ When merged you can safely delete the local branch, pull main and admire your work.
|
||||
+ Don't forget to close your issue and to tag the pull request in the issue by just adding #PR_NUMBER. Also move the task on GH Projects to "Done".
|
||||
+ Repeat.
|
||||
|
||||
**Backlog and taskboard**
|
||||
This is for contributors that have access to the nowdatabase Github Project.
|
||||
|
||||
Creating issues.
|
||||
|
||||
+ Come up with a feature / find a bug
|
||||
+ In case of a feature, write an user story for it. / Bugs => Write an issue that clearly describes what happens, when it happens and under what circumstances it happens.
|
||||
+ Create an issue to nowdatabase repo that has the user story / short description of the bug as the title.
|
||||
+ Next think of smaller tasks that need to be done in order to implement the wanted feature / fix the bug. Don't forget to add tests as a task!
|
||||
+ Now create issues for those tasks too. If there is relevant information regarding the task, it can be written to the description or added as a comment under the issue. *Hint, take note of the issue numbers that Github automatically assigns.*
|
||||
+ Create a checkbox by typing `- [ ]` and link the tasks to the user story / bug issue by referencing the issue by it's number: `#1234`. With the example issue number we would get: `- [ ] #1234`.
|
||||
|
||||
Now the issue is ready for the backlog and the taskboard.
|
||||
|
||||
+ Open the original user story / bug issue and look to the menu on the right and find a section `Projects` and under that `nowdatabase`.
|
||||
+ Click on the arrow that expands the view to include all of the fields available on the Github Project. _Don't go to the Github Projects website yet._
|
||||
+ Set the priority, size and category according to the issue at hand.
|
||||
+ Do this for all tasks listed in the issue.
|
||||
+ After that all of the issues should end up to the Github Project `nowdatabase`. All of the issues should be seen on the `Product backlog`, User storys on `Backlog` and Tasks on `Taskboard`.
|
||||
|
||||
During sprint planning.
|
||||
|
||||
+ In `Backlog` move all of the selected user stories to `Sprint Backlog` and set the `Iteration` field to the current sprint.
|
||||
+ Do this also for all of the tasks and bug reports.
|
||||
|
||||
Handling the Project while developing.
|
||||
|
||||
+ When you start work on an user story / bug, change the matching entries on `Backlog` and `Taskboard` to `In Progress`. You can also assign the user story / bug / task to yourself by clicking on the entry and by selecting yourself on the `Assignees` field on the right.
|
||||
+ When you finish a task, move it straght to `Done`.
|
||||
+ User story can be moved to `Done` when all of the tasks and tests are done for that story.
|
||||
+ All user stories that have been demoed approved by the client are moved to the `Approved`
|
||||
|
||||
|
||||
**Tests & linting**
|
||||
|
||||
Dynamic tests
|
||||
+ End-to-end tests with cypress
|
||||
+ `cypress` folder in the repository root
|
||||
+ Main purpose is to check that mostly correct things are being shown and that views don't crash. Feel free to implement more of these.
|
||||
+ To run, have the anon-version running, then in repo root type `npm run test:e2e` to run in headless mode, or to see the tests running type `npm run cypress open`.
|
||||
+ API-tests using node test runner
|
||||
+ `backend/src/api-tests` folder. Runs tests whose filename match `*.test.ts`.
|
||||
+ More critical features should have api-tests, such as write & logging operations as well as user access.
|
||||
+ To run, type `npm run test:api` in repo root. These tests edit the database and assume that on each run the database is in initial state, so after running the tests or editing it yourself, you must reset it before running the tests again: `npm run reset-anon-db`
|
||||
|
||||
Static tests
|
||||
+ TypeScript compiler
|
||||
+ Checks the basic TypeScript correctness with settings configured in tsconfig-files
|
||||
+ ESLint checks
|
||||
+ More code quality checks, also related to code style, configured by `.eslintrc.cjs` files
|
||||
+ To run both static checks in both `frontend` and `backend`, do `npm run check`. Notice that this runs the commands sequentially and will stop at errors; so after fixing one error you must always run again and it may find other errors.
|
||||
|
||||
**GitHub Actions workflows**
|
||||
+ Linting and tsc is ran in GitHub Actions on push and pull request to any branch.
|
||||
+ End-to-end tests are ran in GitHub as well. They use anonymous data.
|
||||
|
||||
**Deployment**
|
||||
+ Any code pushes to `main`-branch will trigger the `build` workflow. This will build the docker images (staging-version) and push them to GitHub image registry. Staging-OpenShift will then notice this once every 15 minutes, pull the images, and deploy them to staging. Though now there is a bug in OpenShift which prevents this so you must trigger the image update there yourself.
|
||||
+ Production versions are built & deployed whenever you create a Release.
|
||||
|
||||
**Naming conventions**
|
||||
+ Types, React-components, classes: `PascalCase`
|
||||
+ Field names that come directly from database columns or tables are the same name as in the db: `snake_case`
|
||||
+ Environment variables: `SCREAMING_SNAKE_CASE`
|
||||
+ Everything else - variables, functions etc: `camelCase`
|
||||
@@ -1,23 +0,0 @@
|
||||
### Setting up a new table view
|
||||
|
||||
Following things are needed to create a new page with a table (examples are all from Cross-Search):
|
||||
|
||||
- Backend:
|
||||
|
||||
- The [App](../../backend/src/app.ts) should use the route.
|
||||
- A [route](../../backend/src/routes/crossSearch.ts) function calling the service.
|
||||
- A [service](../../backend/src/services/crossSearch.ts) to get the data from the database. There might be a necessities to clean the result, filter the result by project ids (pid) and user's permissions, and create a temp type for typescript.
|
||||
- Earlier we used prisma.findMany-function to fetch the data, but with cross search's 90k rows, the GET-call took approx 40s in staging environment. We found out that prisma.rawSQL with LEFT JOIN would reduce time of the call from 10s to 3s in dev environment. This reduced the call time for staging to approx 10s. Could possible be implemented for other tables also for better performance.
|
||||
|
||||
- Frontend:
|
||||
- Create necessary types in [backendtypes](../../frontend/src/backendTypes.d.ts)
|
||||
- Create a button for the component in [navigation bar](../../frontend/src/components/NavBar.tsx)
|
||||
- Create a relevant page-object for [Pages](../../frontend/src/components/pages.tsx)
|
||||
- Add a route for the component to [App](../../frontend/src/App.tsx)
|
||||
- Create a [component](../../frontend/src/components/CrossSearch/CrossSearchTable.tsx) (note that cross search borrows the detail parts from localities).
|
||||
- Table view:
|
||||
The [TableView](../../frontend/src/components/TableView/TableView.tsx) is defined via variables columns and visibleColumns. Columns in the most simple form only require an accessorKey and a header. If an accessorFn is used, a separate id is required as well. The accessorkey functions as a key to fetch data as well as an id to reference the column as well. The variable visibleColumns takes in accessorKeys or ids for columns, set to false, to be hidden by default.
|
||||
- Non-default filtering (eg filterVariant 'range' for numbers or filterFn 'contains' for text substrings) require no null values in the data for given column. It is needed to use accessorFn to map nulls to an empty string for example.
|
||||
- non-hideable columns can be set by setting parameter `enableHiding: false`
|
||||
|
||||
More info can be found from official [MRT docs](https://www.material-react-table.com/)
|
||||
@@ -1,133 +0,0 @@
|
||||
### Database
|
||||
|
||||
**DB-dump restoring process**
|
||||
|
||||
The mariadb-container by default executes all .sql files it finds in the containers `/docker-entrypoint-initdb.d/` directory when the container is run the first time. The `data` directory is mounted there. It contains an initialization file (`restore_and_create.sql`) which will then restore the database from the sql-dumpfiles in `sqlfiles/`, and also create a user `'now_test'@'localhost'` with a password defined as MARIADB_PASSWORD in the .db.dev.env file for the database.
|
||||
|
||||
**Accessing database inside docker**
|
||||
|
||||
`docker exec -it nowdb-db mariadb -u now_test -p --database now_test`
|
||||
|
||||
Enter the password defined in `.dev.env`.
|
||||
|
||||
**Updating prisma-schema**
|
||||
|
||||
If the database schema is changed, prisma-schema must be also updated. The smart prisma-way would be to run migrations via prisma, but I did this:
|
||||
|
||||
+ The database was modified directly in SQL
|
||||
+ Anon-db dump was updated to reflect the changes by hand (for both the table structure and the data inputted to it)
|
||||
+ Also the test-db dump used locally was changed by hand, but this may be more complicated if theres a lot of data - in which case a new dump must be taken from the db after changes
|
||||
+ Notice that for the db-container to initialize data, you must remove the container and then the volume and rebuild
|
||||
+ I ran the dev version of the app `npm run dev`
|
||||
+ I ran `docker exec -it nowdb-backend-dev npx prisma db pull`
|
||||
+ The schema was updated and the changes can be committed to github.
|
||||
+ After any schema update, to run the app, the **Prisma client must be generated again** (the client is located in "prisma/generated"). I did it by running `npm run prisma` in `backend/` directory (outside of docker). Running it inside the container would not re-generate it correctly, not sure why.
|
||||
|
||||
|
||||
Prisma documentation: https://www.prisma.io/docs/orm/reference/prisma-cli-reference#db-pull
|
||||
|
||||
**Updating test data**
|
||||
|
||||
Sometimes you may want to add some test user or entries into the anonymous data in order to test something better. This can be done by `mariadb-dump` program which is included in the `mariadb` official docker image:
|
||||
|
||||
* Reset the anonymous data to the initial state: `npm run reset-anon-db`
|
||||
* Do the necessary edits, either via code, UI, or by executing commands in the mariadb cli.
|
||||
* Take a dump of the relevant tables: `docker exec nowdb-db-anon mariadb-dump <db_name> -uroot -padmin > ./<db_name>.sql`
|
||||
+ The db names are probably "now_test" or "now_log_test".
|
||||
+ Replace the .sql-files in `test_data/sqlfiles/` with the new dump you created.
|
||||
+ Commit and push the new dump - unless you did this for the staging-data dump, in which case you shouldn't commit and push.
|
||||
|
||||
**Mariadb-connector and transactions**
|
||||
|
||||
Prisma does not have proper support for connecting to multiple databases, and transactionality between them is therefore impossible. We need transactional write to both now and now_log databases. Therefore we use mariadb-connector for the write operations that need to be logged into the log-database.
|
||||
|
||||
|
||||
### Docker
|
||||
|
||||
#### Dockerfile
|
||||
|
||||
There are two variants of Dockerfile used for both backend and frontend:
|
||||
|
||||
1. Production (`Dockerfile`)
|
||||
* Installs only packages required in production (no devDependencies)
|
||||
* Copies only required files to the image
|
||||
* Tries to be minimal so the image would be fast and small
|
||||
2. Development (`dev.Dockerfile`)
|
||||
* Installs all packages from `package.json`
|
||||
* Copies all files from `backend` to the image (not counting `.dockerignore`d files)
|
||||
* Has multiple stages (dev and api-tests)
|
||||
* Learn more about multi-stage Dockerfiles: https://docs.docker.com/build/building/multi-stage/
|
||||
* Includes everything that is required / helps with development and testing
|
||||
|
||||
|
||||
#### Docker compose
|
||||
|
||||
There are three variants of docker compose used in this project.
|
||||
|
||||
1. Production (`docker-compose.yml`)
|
||||
* Uses `{backend,frontend}/Dockerfile` for both backend and frontend
|
||||
* Uses `.env` for environment variables
|
||||
* Uses `data` for database initialization
|
||||
* Doesn't start phpMyAdmin
|
||||
* Can be started with `npm run start`
|
||||
2. Dev (`docker-compose.dev.yml`)
|
||||
* Uses `{backend,frontend}/dev.Dockerfile`
|
||||
* Uses `.dev.env`
|
||||
* Uses `data` for database initialization
|
||||
* Can be started with `npm run dev`
|
||||
3. Anon (`docker-compose.anon.yml`)
|
||||
* Uses `{backend,frontend}/dev.Dockerfile`
|
||||
* Uses `.anon.env`
|
||||
* Uses `test_data` for database initialization
|
||||
* Can be started with `npm run start:anon`
|
||||
|
||||
|
||||
### Openshift
|
||||
|
||||
Staging and production is deployed in Openshift. You should read more about Openshift's own documentation
|
||||
and Kubernetes documentation to understand more.
|
||||
|
||||
#### Updating deployments
|
||||
|
||||
Auto-deploying pods when new images are pulled from Github doesn't work at the moment because of
|
||||
a bug in Openshift. Until it is fixed the deployments have to be updated manually.
|
||||
|
||||
1. Select role as developer if there is an option. Otherwise skip this.
|
||||
2. Go to `Topology`
|
||||
3. Select backend or frontend
|
||||
4. From `Actions` dropdown menu select "`Edit Deployment`"
|
||||
5. Select `Form view` if not selected.
|
||||
6. Tick `Auto deploy when new Image is available`
|
||||
7. Save
|
||||
|
||||
If there was a new version of an image, a new pod should be created and the old one destroyed after
|
||||
the new pod is up and running.
|
||||
|
||||
Same goes for production.
|
||||
|
||||
Trying to update the staging right after merging a pull request might not always work because
|
||||
Openshift pulls the new image from Github packages roughly every 15mins.
|
||||
|
||||
|
||||
|
||||
### Github actions
|
||||
|
||||
This project utilizes Github actions for linting, testing and building the project. At the moment
|
||||
pull requests cannot be merged if the all of the pull-request-actions do not pass.
|
||||
|
||||
The following actions are run on every pull request:
|
||||
* `lint.yaml`
|
||||
* Lints the whole project and fails if there is linting issues
|
||||
* `api-tests.yaml`
|
||||
* Runs api tests inside Docker, passes if all tests pass
|
||||
* `e2e.yaml`
|
||||
* Runs e2e tests inside Docker, passes if all tests pass
|
||||
* `unit-tests.yaml`
|
||||
* Runs all unit tests, passes if all tests pass
|
||||
|
||||
And the following are run on only merges to main:
|
||||
* `build.yaml`
|
||||
* Builds staging version of both backend and frontend and creates Github package out of
|
||||
both of them. If the triggering event happens to be a release, a production package is made.
|
||||
* `phpmyadmin.yaml`
|
||||
* Builds the patched phpMyAdmin image and creates a Github package
|
||||
Reference in New Issue
Block a user