axel/setup db n migrations (#29)

* setup db & migrations

* bugfixes

* README updates

* updates to readme

* not tracking .env.dev

* README update

* cleanup

* updated db logic to allow for test db

* remove unnecessary await

---------

Co-authored-by: Jukka Kuusto <38991418+jkuusto@users.noreply.github.com>
This commit is contained in:
Axel Adlercreutz
2025-01-24 15:09:46 +02:00
committed by GitHub
co-authored by Jukka Kuusto
parent c0c4d96e0d
commit 6ded19a4ec
16 changed files with 449 additions and 25 deletions
+34 -1
View File
@@ -2,8 +2,41 @@
GRADESA is a freely accessible website where language learners at language centres as well as university students of German studies can autonomously work on their German grammar skills and knowledge.
## Install
This should be all that's needed:
- `node`
- `npm`
- https://github.com/peterldowns/pgmigrate
- MAC: Install the newest version with `brew install peterldowns/tap/pgmigrate`
- Linux: # Install pgmigrate
- `wget https://github.com/peterldowns/pgmigrate/releases/download/v0.0.6%2Bcommit.4f90829/pgmigrate-linux-amd64 -O /usr/local/bin/pgmigrate`
- `chmod +x /usr/local/bin/pgmigrate`
- Docker, either of these installation options work (if you're new to docker, I recommend Docker Desktop):
- `brew install docker` + `brew install colima` (https://github.com/abiosoft/colima)
- [Docker Desktop](https://www.docker.com/products/docker-desktop/)
- psql: `brew install postgresql@15`
## Running locally
- Run PostgreSQL:
- Init .env files `./init-env.sh` — This will create the required .env files for environment variables. The local db credentials are
- DB_HOST=localhost
- DB_PORT=6543
- DB_USER=ohtu
- DB_PASSWORD=password
- DB_NAME=gradesa
- `docker-compose up -d` — This command starts the services defined in the docker-compose.yml file in detached mode, meaning the services run in the background and the terminal is freed up for other tasks. For more information, checkout [Docker Compose docs](https://docs.docker.com/compose/)
- `npm run dev` — Starts the dev server.
- You can now find the NextJS app on localhost:3000
## Documentation
- [Product and Sprint Backlogs](https://github.com/orgs/OHTU-German-learning-website/projects/3)
- Migrations: check [the migration README](data/README.md)
## Definition of Done
"*The requirement has been analyzed, designed, programmed, tested, automated testing implemented, documented, integrated into other software, and deployed to the production environment.*"
"_The requirement has been analyzed, designed, programmed, tested, automated testing implemented, documented, integrated into other software, and deployed to the production environment._"
+25
View File
@@ -0,0 +1,25 @@
# Database Migration with pgmigrate
This project uses `pgmigrate` to manage database migrations. `pgmigrate` is a tool that helps apply, and manage database schema changes in a consistent and reliable manner. Ran migrations are tracked in the `pg_migrate_migrations` table.
Having a migration system that's based on sql files is good because it keeps track of schema changes in git and guarantees that everyone is using the same schema.
## Prerequisites
Ensure you have the following installed on your system:
- PostgreSQL
- `pgmigrate` (You can find more information and installation instructions on the [pgmigrate GitHub page](https://github.com/peterldowns/pgmigrate))
## Migration Script
The migration script is located at `data/migrate.sh`. This script uses `pgmigrate` to apply migrations to the database.
### Running Migrations
To run the migrations, execute the following command (in root): `npm run db:migrate`
### Resetting the database
To reset the database, you can use the following command: `npm run db:reset`
This will clear all tables, and rerun migrations.
+8
View File
@@ -0,0 +1,8 @@
#!/bin/bash
set -euo pipefail
cd "$(dirname "$0")"
cd ..
pgmigrate migrate -d postgres://ohtu:password@localhost:6543/gradesa -m data/migrations
+10
View File
@@ -0,0 +1,10 @@
#!/usr/bin/env bash
set -euo pipefail
cd "$(dirname "$0")"
psql "postgres://ohtu:password@localhost:6543/gradesa" -c "DROP SCHEMA public CASCADE;CREATE SCHEMA public;"
cd ../
pnpm run db:migrate
echo "Migration DONE"
+1 -3
View File
@@ -5,8 +5,6 @@ services:
container_name: gradesa-db
environment:
POSTGRES_PASSWORD: password
POSTGRES_USER: tsoha
POSTGRES_DB: gradesa
volumes:
- pg-data:/var/lib/postgresql/data/
- ./provision_db.sql:/docker-entrypoint-initdb.d/1_provision_db.sql
@@ -14,7 +12,7 @@ services:
ports:
- "6543:5432"
healthcheck:
test: ["CMD", "pg_isready", "-U", "tsoha"]
test: ["CMD", "pg_isready", "-U", "postgres"]
interval: 10s
retries: 20
volumes:
+5
View File
@@ -0,0 +1,5 @@
DB_HOST=
DB_PORT=
DB_USER=
DB_PASSWORD=
DB_NAME=
+145
View File
@@ -8,7 +8,9 @@
"name": "gradesa",
"version": "0.1.0",
"dependencies": {
"dotenv": "^16.4.7",
"next": "15.1.5",
"pg": "^8.13.1",
"react": "^19.0.0",
"react-dom": "^19.0.0"
},
@@ -1580,6 +1582,17 @@
"node": ">=0.10.0"
}
},
"node_modules/dotenv": {
"version": "16.4.7",
"resolved": "https://registry.npmjs.org/dotenv/-/dotenv-16.4.7.tgz",
"integrity": "sha512-47qPchRCykZC03FhkYAhrvwU4xDBFIj1QPqaarj6mdM/hgUzfPHcpkHJOn3mJAufFeeAxAzeGsr5X0M4k6fLZQ==",
"engines": {
"node": ">=12"
},
"funding": {
"url": "https://dotenvx.com"
}
},
"node_modules/dunder-proto": {
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/dunder-proto/-/dunder-proto-1.0.1.tgz",
@@ -3527,6 +3540,87 @@
"integrity": "sha512-LDJzPVEEEPR+y48z93A0Ed0yXb8pAByGWo/k5YYdYgpY2/2EsOsksJrq7lOHxryrVOn1ejG6oAp8ahvOIQD8sw==",
"dev": true
},
"node_modules/pg": {
"version": "8.13.1",
"resolved": "https://registry.npmjs.org/pg/-/pg-8.13.1.tgz",
"integrity": "sha512-OUir1A0rPNZlX//c7ksiu7crsGZTKSOXJPgtNiHGIlC9H0lO+NC6ZDYksSgBYY/thSWhnSRBv8w1lieNNGATNQ==",
"dependencies": {
"pg-connection-string": "^2.7.0",
"pg-pool": "^3.7.0",
"pg-protocol": "^1.7.0",
"pg-types": "^2.1.0",
"pgpass": "1.x"
},
"engines": {
"node": ">= 8.0.0"
},
"optionalDependencies": {
"pg-cloudflare": "^1.1.1"
},
"peerDependencies": {
"pg-native": ">=3.0.1"
},
"peerDependenciesMeta": {
"pg-native": {
"optional": true
}
}
},
"node_modules/pg-cloudflare": {
"version": "1.1.1",
"resolved": "https://registry.npmjs.org/pg-cloudflare/-/pg-cloudflare-1.1.1.tgz",
"integrity": "sha512-xWPagP/4B6BgFO+EKz3JONXv3YDgvkbVrGw2mTo3D6tVDQRh1e7cqVGvyR3BE+eQgAvx1XhW/iEASj4/jCWl3Q==",
"optional": true
},
"node_modules/pg-connection-string": {
"version": "2.7.0",
"resolved": "https://registry.npmjs.org/pg-connection-string/-/pg-connection-string-2.7.0.tgz",
"integrity": "sha512-PI2W9mv53rXJQEOb8xNR8lH7Hr+EKa6oJa38zsK0S/ky2er16ios1wLKhZyxzD7jUReiWokc9WK5nxSnC7W1TA=="
},
"node_modules/pg-int8": {
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/pg-int8/-/pg-int8-1.0.1.tgz",
"integrity": "sha512-WCtabS6t3c8SkpDBUlb1kjOs7l66xsGdKpIPZsg4wR+B3+u9UAum2odSsF9tnvxg80h4ZxLWMy4pRjOsFIqQpw==",
"engines": {
"node": ">=4.0.0"
}
},
"node_modules/pg-pool": {
"version": "3.7.0",
"resolved": "https://registry.npmjs.org/pg-pool/-/pg-pool-3.7.0.tgz",
"integrity": "sha512-ZOBQForurqh4zZWjrgSwwAtzJ7QiRX0ovFkZr2klsen3Nm0aoh33Ls0fzfv3imeH/nw/O27cjdz5kzYJfeGp/g==",
"peerDependencies": {
"pg": ">=8.0"
}
},
"node_modules/pg-protocol": {
"version": "1.7.0",
"resolved": "https://registry.npmjs.org/pg-protocol/-/pg-protocol-1.7.0.tgz",
"integrity": "sha512-hTK/mE36i8fDDhgDFjy6xNOG+LCorxLG3WO17tku+ij6sVHXh1jQUJ8hYAnRhNla4QVD2H8er/FOjc/+EgC6yQ=="
},
"node_modules/pg-types": {
"version": "2.2.0",
"resolved": "https://registry.npmjs.org/pg-types/-/pg-types-2.2.0.tgz",
"integrity": "sha512-qTAAlrEsl8s4OiEQY69wDvcMIdQN6wdz5ojQiOy6YRMuynxenON0O5oCpJI6lshc6scgAY8qvJ2On/p+CXY0GA==",
"dependencies": {
"pg-int8": "1.0.1",
"postgres-array": "~2.0.0",
"postgres-bytea": "~1.0.0",
"postgres-date": "~1.0.4",
"postgres-interval": "^1.1.0"
},
"engines": {
"node": ">=4"
}
},
"node_modules/pgpass": {
"version": "1.0.5",
"resolved": "https://registry.npmjs.org/pgpass/-/pgpass-1.0.5.tgz",
"integrity": "sha512-FdW9r/jQZhSeohs1Z3sI1yxFQNFvMcnmfuj4WBMUTxOrAyLMaTcE1aAMBiTlbMNaXvBCQuVi0R7hd8udDSP7ug==",
"dependencies": {
"split2": "^4.1.0"
}
},
"node_modules/picocolors": {
"version": "1.1.1",
"resolved": "https://registry.npmjs.org/picocolors/-/picocolors-1.1.1.tgz",
@@ -3580,6 +3674,41 @@
"node": "^10 || ^12 || >=14"
}
},
"node_modules/postgres-array": {
"version": "2.0.0",
"resolved": "https://registry.npmjs.org/postgres-array/-/postgres-array-2.0.0.tgz",
"integrity": "sha512-VpZrUqU5A69eQyW2c5CA1jtLecCsN2U/bD6VilrFDWq5+5UIEVO7nazS3TEcHf1zuPYO/sqGvUvW62g86RXZuA==",
"engines": {
"node": ">=4"
}
},
"node_modules/postgres-bytea": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/postgres-bytea/-/postgres-bytea-1.0.0.tgz",
"integrity": "sha512-xy3pmLuQqRBZBXDULy7KbaitYqLcmxigw14Q5sj8QBVLqEwXfeybIKVWiqAXTlcvdvb0+xkOtDbfQMOf4lST1w==",
"engines": {
"node": ">=0.10.0"
}
},
"node_modules/postgres-date": {
"version": "1.0.7",
"resolved": "https://registry.npmjs.org/postgres-date/-/postgres-date-1.0.7.tgz",
"integrity": "sha512-suDmjLVQg78nMK2UZ454hAG+OAW+HQPZ6n++TNDUX+L0+uUlLywnoxJKDou51Zm+zTCjrCl0Nq6J9C5hP9vK/Q==",
"engines": {
"node": ">=0.10.0"
}
},
"node_modules/postgres-interval": {
"version": "1.2.0",
"resolved": "https://registry.npmjs.org/postgres-interval/-/postgres-interval-1.2.0.tgz",
"integrity": "sha512-9ZhXKM/rw350N1ovuWHbGxnGh/SNJ4cnxHiM0rxE4VN41wsg8P8zWn9hv/buK00RP4WvlOyr/RBDiptyxVbkZQ==",
"dependencies": {
"xtend": "^4.0.0"
},
"engines": {
"node": ">=0.10.0"
}
},
"node_modules/prelude-ls": {
"version": "1.2.1",
"resolved": "https://registry.npmjs.org/prelude-ls/-/prelude-ls-1.2.1.tgz",
@@ -4031,6 +4160,14 @@
"node": ">=0.10.0"
}
},
"node_modules/split2": {
"version": "4.2.0",
"resolved": "https://registry.npmjs.org/split2/-/split2-4.2.0.tgz",
"integrity": "sha512-UcjcJOWknrNkF6PLX83qcHM6KHgVKNkV62Y8a5uYDVv9ydGQVwAHMKqHdJje1VTWpljG0WYpCDhrCdAOYH4TWg==",
"engines": {
"node": ">= 10.x"
}
},
"node_modules/stable-hash": {
"version": "0.0.4",
"resolved": "https://registry.npmjs.org/stable-hash/-/stable-hash-0.0.4.tgz",
@@ -4504,6 +4641,14 @@
"node": ">=0.10.0"
}
},
"node_modules/xtend": {
"version": "4.0.2",
"resolved": "https://registry.npmjs.org/xtend/-/xtend-4.0.2.tgz",
"integrity": "sha512-LKYU1iAXJXUgAXn9URjiu+MWhyUXHsvfp7mcuYm9dSUKK0/CjtrUwFAxD82/mCWbtLsGjFIad0wIsod4zrTAEQ==",
"engines": {
"node": ">=0.4"
}
},
"node_modules/yocto-queue": {
"version": "0.1.0",
"resolved": "https://registry.npmjs.org/yocto-queue/-/yocto-queue-0.1.0.tgz",
+7 -5
View File
@@ -3,19 +3,21 @@
"version": "0.1.0",
"private": true,
"scripts": {
"dev": "next dev --turbopack",
"dev": "next dev",
"build": "next build",
"start": "next start",
"lint": "next lint"
},
"dependencies": {
"dotenv": "^16.4.7",
"next": "15.1.5",
"pg": "^8.13.1",
"react": "^19.0.0",
"react-dom": "^19.0.0",
"next": "15.1.5"
"react-dom": "^19.0.0"
},
"devDependencies": {
"@eslint/eslintrc": "^3",
"eslint": "^9",
"eslint-config-next": "15.1.5",
"@eslint/eslintrc": "^3"
"eslint-config-next": "15.1.5"
}
}
+7
View File
@@ -0,0 +1,7 @@
import { query } from '../../../backend/db';
export async function GET(request) {
const result = await query('SELECT now()');
const now = result.rows[0].now;
return Response.json({ message: 'Hello, world!', now });
}
+10 -2
View File
@@ -1,7 +1,14 @@
'use client';
import { useEffect, useState } from "react";
import Image from "next/image";
import styles from "./page.module.css";
export default function Home() {
const [data, setData] = useState(null);
useEffect(() => {
fetch('/api/hello')
.then(res => res.json())
.then(data => setData(data));
}, []);
return (
<div className={styles.page}>
<main className={styles.main}>
@@ -19,7 +26,8 @@ export default function Home() {
</li>
<li>Save and see your changes instantly.</li>
</ol>
{data && <p>{data.message}</p>}
DB TIME: {data && <p>{data.now}</p>}
<div className={styles.ctas}>
<a
className={styles.primary}
+90
View File
@@ -0,0 +1,90 @@
import { config } from 'dotenv';
// Loads .env and .env.NODE_ENV (e.g. .env.development)
// Shared environment variables are in .env
// Environment specific variables are in .env.NODE_ENV
// NextJS defaults to 'development' when running npm run dev
export const initConfig = async () => {
const env = process.env.NODE_ENV;
switch (env) {
case 'test':
case 'development':
case 'production':
config({ path: `${__dirname}/dotenv/.env`, override: true });
config({ path: `${__dirname}/dotenv/.env.${env}`, override: true });
break;
default:
throw new Error('Unsupported environment: ' + env);
}
};
initConfig();
// The following is an environment variable helper.
// It's used to read environment variables from .env and .env.NODE_ENV
// So instead of using process.env.DB_HOST, you can use getConfig().db.host
// This is useful for keeping the code clean and gives you clear errors if you miss a variable.
let cachedConfig;
export const getConfig = () => {
if (cachedConfig) {
return cachedConfig;
}
cachedConfig = create();
return cachedConfig;
};
function getFromEnv(key) {
const val = process.env[key];
if (val === undefined) {
return undefined;
}
return val;
}
function create() {
let errors = [];
function readString(key) {
const val = getFromEnv(key);
if (val === undefined || val.trim() === '') {
errors.push(key);
}
return val ?? '';
}
function readStringOptional(key, defaultValue) {
return (getFromEnv(key)) ?? defaultValue;
}
function readInt(key) {
const val = getFromEnv(key);
if (val === undefined) {
errors.push(key);
return -1;
}
return parseInt(val);
}
function readIntOptional(key, defaultValue) {
const value = getFromEnv(key);
return value === undefined ? defaultValue : parseInt(value);
}
const config = {
db: {
host: readString('DB_HOST'),
port: readInt('DB_PORT'),
user: readString('DB_USER'),
password: readString('DB_PASSWORD'),
database: readString('DB_NAME'),
},
};
if (errors.length > 0) {
throw new Error(`Missing or invalid environment variables: ${errors.join(', ')}`);
}
return config;
}
+71
View File
@@ -0,0 +1,71 @@
import { Pool } from 'pg';
import { getConfig } from './config';
class DB {
#pool;
constructor() { }
async get() {
if (!this.#pool) {
const config = getConfig();
this.#createPool(config.db);
await this.#connect(this.#pool);
}
return this.#pool;
}
#createPool(config) {
this.#pool = new Pool(config);
}
async #connect(pool) {
try {
if (!pool) {
throw new Error('Pool not initialized');
}
await pool.connect();
const row = await pool.query('SELECT current_database() as database_name, now()');
console.log('\x1b[34m%s\x1b[0m', `Connected to database ${row.rows[0].database_name} at ${row.rows[0].now.toISOString()}`);
} catch (err) {
console.error('Failed to connect to database', err);
}
}
async #set(config) {
const newPool = new Pool(config);
await this.#connect(newPool);
this.#pool = newPool;
}
}
export const db = new DB();
/**
* Executes a database query using the connection pool.
*
* @param {string} text - The SQL query text to be executed.
* @param {Array} [params] - An optional array of parameters to be used in the query.
* @returns {Promise<Object>} - A promise that resolves to the result of the query.
*/
export const query = async (text, params) => (await db.get()).query(text, params);
/**
* Executes a database transaction using the connection pool.
*
* @param {Function} fn - A function that takes a client as an argument and returns a promise.
* @returns {Promise<Object>} - A promise that resolves to the result of the transaction.
*/
export const transaction = async (fn) => {
const client = await (await db.get()).connect();
try {
await client.query('BEGIN');
const result = await fn(client);
await client.query('COMMIT');
return result;
} catch (err) {
await client.query('ROLLBACK');
throw err;
}
};
Executable
+9
View File
@@ -0,0 +1,9 @@
#!/usr/bin/env bash
touch gradesa/.env
echo "TZ=UTC" >> gradesa/.env
cp gradesa/.env.template gradesa/.env.development
cp gradesa/.env.template gradesa/.env.test
cp gradesa/.env.template gradesa/.env.production
echo "Created .env files"
+13
View File
@@ -0,0 +1,13 @@
{
"name": "ohtu-german-learning-website",
"version": "1.0.0",
"description": "GRADESA is a freely accessible website where language learners at language centres as well as university students of German studies can autonomously work on their German grammar skills and knowledge.",
"scripts": {
"db:migrate": "./data/migrate.sh",
"db:reset": "./data/reset-db.sh",
"dev": "npm run --prefix gradesa dev",
"test": "echo \"Error: no test specified\" && exit 1"
},
"author": "",
"license": "ISC"
}
-12
View File
@@ -1,12 +0,0 @@
DO $do$
BEGIN
IF NOT EXISTS (SELECT FROM pg_catalog.pg_roles WHERE rolname = 'tsoha') THEN
CREATE ROLE tsoha;
END IF;
ALTER ROLE tsoha WITH LOGIN PASSWORD 'password' NOSUPERUSER CREATEDB NOCREATEROLE;
END
$do$;
DROP DATABASE IF EXISTS gradesa;
CREATE DATABASE gradesa OWNER tsoha;
GRANT ALL ON DATABASE gradesa TO tsoha;
+12
View File
@@ -0,0 +1,12 @@
DO $do$
BEGIN
IF NOT EXISTS (SELECT FROM pg_catalog.pg_roles WHERE rolname = 'ohtu') THEN
CREATE ROLE ohtu;
END IF;
ALTER ROLE ohtu WITH LOGIN PASSWORD 'password' NOSUPERUSER CREATEDB NOCREATEROLE;
END
$do$;
DROP DATABASE IF EXISTS gradesa;
CREATE DATABASE gradesa OWNER ohtu;
GRANT ALL ON DATABASE gradesa TO ohtu;