mirror of
https://github.com/UniversityOfHelsinkiCS/oodikone.git
synced 2026-10-12 19:03:39 +03:00
[Scripts] Greatly improve db scripts
This commit is contained in:
Aaro Koivukangas
@@ -4,6 +4,29 @@ This folder contains a guide and scripts to modify or create anonymous database
|
||||
|
||||
The quick guide should be enough for most purposes: updating schema, data, or the PostgreSQL version. If you need to generate completely new `sis-importer-db` and `sis-db` images (which shouldn't happen too often), follow the instructions in the section [Generate new sis-importer-db and sis-db completely from scratch](#generate-new-sis-importer-db-and-sis-db-completely-from-scratch).
|
||||
|
||||
## Scripts
|
||||
|
||||
`manage_test_dbs.sh` is the front end for the test database tooling. Run it without arguments for an interactive menu, or give it an action and one or more databases (or `all`) to run it directly:
|
||||
|
||||
```bash
|
||||
./manage_test_dbs.sh dump sis-db # dump sis-db from its running container
|
||||
./manage_test_dbs.sh upload sis-db # upload the current dump to s3
|
||||
./manage_test_dbs.sh release all # dump, upload and push images for every database
|
||||
```
|
||||
|
||||
Actions are `download`, `dump`, `upload`, `push`, `release` (dump + upload + push) and `rebuild` (download + push). Each one is a thin call to a script in this folder, which can also be used on its own:
|
||||
|
||||
| Script | What it does |
|
||||
| -------------------------- | ----------------------------------------------------------------------------------------------------- |
|
||||
| `dump_local_db.sh` | `pg_dump`s the databases from the running containers into `.databasedumps/` |
|
||||
| `download_dump_from_s3.sh` | Downloads the newest dump of each database from s3 (`-b <branch>` to pick dumps from a single branch) |
|
||||
| `upload_dump_to_s3.sh` | Uploads the dumps in `.databasedumps/test` to s3 as `<database>-<git_branch>-<datetime>.sql` |
|
||||
| `build_and_push_image.sh` | Builds database images from the current dumps and pushes them to the Toska registry |
|
||||
| `create_s3_bucket.sh` | Creates the s3 bucket if it doesn't exist yet, run automatically before an upload |
|
||||
| `lib/common.sh` | Shared configuration (dump directories, s3 bucket, registry, database names) |
|
||||
|
||||
Dumps end up in `.databasedumps/test/<database>.sql`, which is where `db.Dockerfile` reads them from when an image is built. The s3 commands read their credentials from the `ACCESS_KEY` and `SECRET_KEY` environment variables (e.g. in CI). If neither is set, they fall back to `~/.s3cfg`, which can be found on the toska/dokumentaatio repo.
|
||||
|
||||
## Quick guide to updating a test database
|
||||
|
||||
1. Ensure you have the latest test-db. Just pulling images might not replace your existing volume, so you may need to remove it manually. This command might work: `docker volume rm oodikone-sis-db-data` (replace `oodikone-sis-db-data` with the correct volume name). The new image, including data, should be pulled the next time you launch Oodikone normally.
|
||||
|
||||
Executable
+54
@@ -0,0 +1,54 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
# Build database images from the dumps in .databasedumps/test and push them to
|
||||
# the Toska registry.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# shellcheck source-path=SCRIPTDIR
|
||||
# shellcheck source=lib/common.sh
|
||||
source "$SCRIPT_DIR"/lib/common.sh
|
||||
|
||||
usage() {
|
||||
cat >&2 <<EOF
|
||||
Usage: $(basename "$0") <database>... | all
|
||||
|
||||
Builds an image of each given database and pushes it to $REGISTRY.
|
||||
The dump is read from .databasedumps/test/<database>.sql, so run
|
||||
dump_local_db.sh or download_dump_from_s3.sh first.
|
||||
|
||||
Databases: ${DATABASES[*]}
|
||||
EOF
|
||||
}
|
||||
|
||||
if [[ "${1-}" == "-h" || "${1-}" == "--help" ]]; then
|
||||
usage
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [[ $# -eq 0 ]]; then
|
||||
usage
|
||||
die "No database given"
|
||||
fi
|
||||
|
||||
build_and_push_image() {
|
||||
local database="$1"
|
||||
local image="$REGISTRY/$database:latest"
|
||||
local dump_file="$TEST_DUMP_DIR/$database.sql"
|
||||
|
||||
[[ -f "$dump_file" ]] || die "No dump found at $dump_file, run dump_local_db.sh or download_dump_from_s3.sh first"
|
||||
|
||||
infomsg "Building $image"
|
||||
docker build "$PROJECT_ROOT" -f "$PROJECT_ROOT/db.Dockerfile" -t "$image" --build-arg DB_NAME="$database"
|
||||
|
||||
infomsg "Pushing $image"
|
||||
docker push "$image"
|
||||
successmsg "Pushed $image"
|
||||
}
|
||||
|
||||
resolve_databases "$@"
|
||||
|
||||
for database in "${selected_databases[@]}"; do
|
||||
build_and_push_image "$database"
|
||||
done
|
||||
Executable
+36
@@ -0,0 +1,36 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
# Create the s3 bucket used for the test database dumps, if it doesn't exist yet.
|
||||
# Doing nothing when the bucket is already there, so this is safe to run every time.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# shellcheck source-path=SCRIPTDIR
|
||||
# shellcheck source=lib/common.sh
|
||||
source "$SCRIPT_DIR"/lib/common.sh
|
||||
|
||||
usage() {
|
||||
cat >&2 <<EOF
|
||||
Usage: $(basename "$0")
|
||||
|
||||
Creates the bucket $S3_BUCKET if it doesn't exist yet, and does nothing if it does.
|
||||
EOF
|
||||
}
|
||||
|
||||
if [[ "${1-}" == "-h" || "${1-}" == "--help" ]]; then
|
||||
usage
|
||||
exit 0
|
||||
fi
|
||||
|
||||
require_s3_config
|
||||
|
||||
# s3cmd info exits with an error when the bucket doesn't exist
|
||||
if s3 info "$S3_BUCKET" > /dev/null 2>&1; then
|
||||
infomsg "Bucket $S3_BUCKET already exists"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
warningmsg "Bucket $S3_BUCKET doesn't exist, creating it"
|
||||
s3 mb "$S3_BUCKET"
|
||||
successmsg "Created bucket $S3_BUCKET"
|
||||
@@ -1,235 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
# === Config ===
|
||||
|
||||
# Quit if failing
|
||||
set -euo pipefail
|
||||
|
||||
# Print messages and logs that are not script output to stderr
|
||||
msg() {
|
||||
echo >&2 -e "${1-}"
|
||||
}
|
||||
|
||||
# Some special types of messages with colours
|
||||
successmsg() {
|
||||
msg "${GREEN}$1${NOFORMAT}"
|
||||
}
|
||||
errormsg() {
|
||||
msg "${RED}$1${NOFORMAT}"
|
||||
}
|
||||
infomsg() {
|
||||
msg "${BLUE}$1${NOFORMAT}"
|
||||
}
|
||||
warningmsg() {
|
||||
msg "${ORANGE}$1${NOFORMAT}"
|
||||
}
|
||||
|
||||
# Setup colors for messages if running interactive shell
|
||||
if [[ -t 2 ]] && [[ "${TERM-}" != "dumb" ]]; then
|
||||
NOFORMAT='\033[0m' RED='\033[0;31m' GREEN='\033[0;32m' ORANGE='\033[0;33m' BLUE='\033[0;34m'
|
||||
else
|
||||
NOFORMAT='' RED='' GREEN='' ORANGE='' BLUE=''
|
||||
fi
|
||||
|
||||
|
||||
# === #
|
||||
|
||||
PROJECT_ROOT="$(git rev-parse --show-toplevel)"
|
||||
|
||||
get_project_relative_path () {
|
||||
local absolute_path="$1"
|
||||
local root="$2"
|
||||
realpath -s --relative-to="${root:-$PROJECT_ROOT}" "$absolute_path"
|
||||
}
|
||||
|
||||
LOCAL_DUMP_DIR="$(get_project_relative_path ".databasedumps/local_containers" "$PWD")" # Dumps from running docker containers
|
||||
S3_DUMP_DIR="$(get_project_relative_path ".databasedumps/s3" "$PWD")" # Dumps from s3
|
||||
TEST_DUMP_DIR="$(get_project_relative_path ".databasedumps/test" "$PWD")" # Location where .sql files are pulled while building the new image
|
||||
|
||||
S3_CONFIG_FILE=~/.s3cfg
|
||||
S3_BUCKET="s3://oodikone-test"
|
||||
|
||||
REGISTRY="registry-toska.ext.ocp-prod-0.k8s.it.helsinki.fi"
|
||||
|
||||
## Following the naming convention in docker compose, these are names for database
|
||||
## services. The real data databases inside service will have names suffixed by "-real".
|
||||
KONE_DB_NAME="kone-db"
|
||||
SIS_DB_NAME="sis-db"
|
||||
SIS_IMPORTER_DB_NAME="sis-importer-db"
|
||||
USER_DB_NAME="user-db"
|
||||
JAMI_DB_NAME="jami-db"
|
||||
DATABASES=("$USER_DB_NAME" "$KONE_DB_NAME" "$SIS_DB_NAME" "$SIS_IMPORTER_DB_NAME" "$JAMI_DB_NAME")
|
||||
|
||||
mkdir -p "$TEST_DUMP_DIR"
|
||||
mkdir -p "$S3_DUMP_DIR"
|
||||
mkdir -p "$LOCAL_DUMP_DIR"
|
||||
|
||||
|
||||
# === Handle s3 bucket ===
|
||||
# Use s3 bucket as the dump save/pull location
|
||||
download_s3_db () {
|
||||
local database="$1"
|
||||
|
||||
# TODO: Decompress compressed files
|
||||
s3cmd -c "$S3_CONFIG_FILE" get "$S3_BUCKET/$database.sql" "$S3_DUMP_DIR/$database.sql"
|
||||
cp --remove-destination -r "$S3_DUMP_DIR" "$TEST_DUMP_DIR" # Move dumps to the place where docker build pulls .sql files from
|
||||
}
|
||||
|
||||
upload_s3_db () {
|
||||
local database="$1"
|
||||
|
||||
if [[ $(s3cmd ls s3:// | grep -q oodikone-test; echo $?) -eq 0 ]]
|
||||
then
|
||||
infomsg "OODIKONE-TEST bucket exists, skipping creation";
|
||||
else
|
||||
warningmsg "OODIKONE-TEST bucket doesn't exist, creating new bucket...";
|
||||
s3cmd -c "$S3_CONFIG_FILE" mb "s3://oodikone-test"
|
||||
fi
|
||||
|
||||
infomsg "Pushing $TEST_DUMP_DIR/$database.sql to $S3_BUCKET"
|
||||
# TODO: Also compress files
|
||||
s3cmd -c "$S3_CONFIG_FILE" put "$TEST_DUMP_DIR/$database.sql" "$S3_BUCKET/$database.sql"
|
||||
}
|
||||
|
||||
|
||||
# === Build images ===
|
||||
# Update registry with an existing dump located in .databasedumps/test/$1.sql
|
||||
build_and_push_image () {
|
||||
local database="$1"
|
||||
|
||||
infomsg "Building docker image for $database\n"
|
||||
docker build "$(get_project_relative_path "$PROJECT_ROOT" "$PWD")" -t "$REGISTRY/$database:latest" -f db.Dockerfile --build-arg DB_NAME="$database"
|
||||
successmsg "$database docker image built"
|
||||
infomsg "Pushing $database to $REGISTRY\n"
|
||||
infomsg "$REGISTRY/$database"
|
||||
docker push "$REGISTRY/$database"
|
||||
successmsg "Image $REGISTRY/$database pushed successfully"
|
||||
}
|
||||
|
||||
|
||||
# === Handle dumping local dbs ===
|
||||
# To conveniently update all dbs, build all with docker-compose.db.yml and then run dump_all_local_dbs
|
||||
dump_local_db () {
|
||||
local database="$1"
|
||||
local container_name="oodikone-$1"
|
||||
local database_name="$database"
|
||||
|
||||
if [[ $database == "jami-db" ]]; then database_name="postgres"; fi
|
||||
|
||||
infomsg "Trying to dump $database to $LOCAL_DUMP_DIR/$database.sql"
|
||||
docker exec -i "$container_name" pg_dump -Fp -U postgres -d "$database_name" > "$LOCAL_DUMP_DIR/$database.sql"
|
||||
successmsg "Dumped $database to $LOCAL_DUMP_DIR/$database.sql"
|
||||
cp --remove-destination -r "$LOCAL_DUMP_DIR/$database.sql" "$TEST_DUMP_DIR" # Move dump to the place where docker build pulls the .sql file from
|
||||
}
|
||||
|
||||
|
||||
# === Helpers ===
|
||||
do_for_all () {
|
||||
local cmd="$1"
|
||||
shift # Removes $1 from "$@"
|
||||
|
||||
infomsg "Running $cmd for all databases"
|
||||
for database in "${DATABASES[@]}"; do
|
||||
"$cmd" "$database" "$@"
|
||||
done
|
||||
successmsg "$cmd ran successfully"
|
||||
}
|
||||
|
||||
ask_for_input () {
|
||||
local cmd="$1"
|
||||
shift # Removes $1 from "$@"
|
||||
|
||||
echo
|
||||
|
||||
init_database_menu
|
||||
select database in "${options[@]}"; do
|
||||
"$cmd" "$database" "$@"
|
||||
break
|
||||
done
|
||||
}
|
||||
|
||||
|
||||
init_action_menu () {
|
||||
PS3="Please enter your choice: "
|
||||
|
||||
options=(
|
||||
"Pull dumps from s3"
|
||||
"Dump from running docker images"
|
||||
"Update dumps in s3"
|
||||
"Push new images to registry"
|
||||
)
|
||||
}
|
||||
|
||||
init_mode_menu () {
|
||||
PS3="Select command mode: "
|
||||
options=(
|
||||
"Run command for all databases"
|
||||
"Select single database"
|
||||
)
|
||||
}
|
||||
|
||||
init_database_menu () {
|
||||
PS3="Select database: "
|
||||
options=("${DATABASES[@]}")
|
||||
}
|
||||
|
||||
|
||||
# === RUN ===
|
||||
|
||||
# For running as non-interactive CI-mode
|
||||
# $1: Command to be run
|
||||
# $2: Database name
|
||||
case "$1" in
|
||||
"local")
|
||||
if [ "$#" -lt 2 ]; then echo "Missing database name"; exit 1; fi
|
||||
dump_local_db "$2"
|
||||
build_and_push_image "$2"
|
||||
exit 0;;
|
||||
|
||||
# TODO: NOT YET IMPLEMENTED, should use s3 storage
|
||||
"ci")
|
||||
exit 1
|
||||
# if [ "$#" -lt 2 ]; then echo "Missing database name"; exit 1; fi
|
||||
# download_s3_db "$2"
|
||||
# build_and_push_image "$2"
|
||||
# exit 0;;
|
||||
esac
|
||||
|
||||
# For interactive running
|
||||
while true; do
|
||||
init_mode_menu
|
||||
select mode in "${options[@]}"; do
|
||||
case "$mode" in
|
||||
"Run command for all databases")
|
||||
cmd="do_for_all"
|
||||
break;;
|
||||
"Select single database")
|
||||
cmd="ask_for_input"
|
||||
break;;
|
||||
esac
|
||||
done
|
||||
|
||||
echo
|
||||
|
||||
echo
|
||||
|
||||
init_action_menu
|
||||
select opt in "${options[@]}"; do
|
||||
case "$opt" in
|
||||
"Pull dumps from s3")
|
||||
"$cmd" download_s3_db "$S3_DUMP_DIR"
|
||||
break;;
|
||||
"Dump from running docker images")
|
||||
"$cmd" dump_local_db "$LOCAL_DUMP_DIR"
|
||||
break;;
|
||||
"Update dumps in s3")
|
||||
"$cmd" upload_s3_db
|
||||
break;;
|
||||
"Push new images to registry")
|
||||
"$cmd" build_and_push_image
|
||||
break;;
|
||||
esac
|
||||
done
|
||||
|
||||
echo
|
||||
done
|
||||
Executable
+93
@@ -0,0 +1,93 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
# Download database dumps from s3 into .databasedumps/s3, and copy them to
|
||||
# .databasedumps/test where db.Dockerfile reads them from.
|
||||
#
|
||||
# Dumps are uploaded as <db_name>-<git_branch>-<datetime>.sql (see
|
||||
# upload_dump_to_s3.sh), so the newest matching dump is the one downloaded.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# shellcheck source-path=SCRIPTDIR
|
||||
# shellcheck source=lib/common.sh
|
||||
source "$SCRIPT_DIR"/lib/common.sh
|
||||
|
||||
usage() {
|
||||
cat >&2 <<EOF
|
||||
Usage: $(basename "$0") [-b|--branch <branch>] <database>... | all
|
||||
|
||||
Downloads the newest dump of each given database from $S3_TARGET.
|
||||
Databases: ${DATABASES[*]}
|
||||
|
||||
Options:
|
||||
-b, --branch <branch> Only consider dumps uploaded from this git branch
|
||||
-h, --help Show this help
|
||||
EOF
|
||||
}
|
||||
|
||||
branch=""
|
||||
arguments=()
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
-b | --branch)
|
||||
branch="${2-}"
|
||||
[[ -n "$branch" ]] || {
|
||||
usage
|
||||
die "Missing value for $1"
|
||||
}
|
||||
shift 2
|
||||
;;
|
||||
-h | --help)
|
||||
usage
|
||||
exit 0
|
||||
;;
|
||||
-*)
|
||||
usage
|
||||
die "Unknown option: $1"
|
||||
;;
|
||||
*)
|
||||
arguments+=("$1")
|
||||
shift
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
# Newest object in s3 for the given database, or empty if there are none
|
||||
newest_dump_in_s3() {
|
||||
local database="$1"
|
||||
local prefix="$S3_TARGET/$database-"
|
||||
if [[ -n "$branch" ]]; then prefix="$prefix$(sanitize_branch_name "$branch")-"; fi
|
||||
|
||||
# s3cmd ls prints "<date> <time> <size> <object>", so sorting by the first two
|
||||
# columns in reverse puts the newest dump first
|
||||
s3 ls "$prefix" | sort -rk1,2 | awk 'NR == 1 { print $4 }'
|
||||
}
|
||||
|
||||
download_database() {
|
||||
local database="$1"
|
||||
local dump_file="$S3_DUMP_DIR/$database.sql"
|
||||
local newest_dump
|
||||
newest_dump="$(newest_dump_in_s3 "$database")"
|
||||
|
||||
[[ -n "$newest_dump" ]] || die "No dumps found for $database in $S3_TARGET${branch:+ (branch $branch)}"
|
||||
|
||||
infomsg "Downloading $newest_dump"
|
||||
s3 get --force "$newest_dump" "$dump_file"
|
||||
cp --remove-destination "$dump_file" "$TEST_DUMP_DIR/$database.sql"
|
||||
successmsg "Downloaded $newest_dump to $dump_file"
|
||||
}
|
||||
|
||||
if [[ ${#arguments[@]} -eq 0 ]]; then
|
||||
usage
|
||||
die "No database given"
|
||||
fi
|
||||
|
||||
resolve_databases "${arguments[@]}"
|
||||
require_s3_config
|
||||
make_dump_dirs
|
||||
|
||||
for database in "${selected_databases[@]}"; do
|
||||
download_database "$database"
|
||||
done
|
||||
Executable
+53
@@ -0,0 +1,53 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
# Dump one or more databases from the running docker containers into
|
||||
# .databasedumps/local_containers, and copy them to .databasedumps/test where
|
||||
# db.Dockerfile reads them from.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# shellcheck source-path=SCRIPTDIR
|
||||
# shellcheck source=lib/common.sh
|
||||
source "$SCRIPT_DIR"/lib/common.sh
|
||||
|
||||
usage() {
|
||||
cat >&2 <<EOF
|
||||
Usage: $(basename "$0") <database>... | all
|
||||
|
||||
Dumps the given databases from their running containers.
|
||||
Databases: ${DATABASES[*]}
|
||||
EOF
|
||||
}
|
||||
|
||||
if [[ "${1-}" == "-h" || "${1-}" == "--help" ]]; then
|
||||
usage
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [[ $# -eq 0 ]]; then
|
||||
usage
|
||||
die "No database given"
|
||||
fi
|
||||
|
||||
dump_database() {
|
||||
local database="$1"
|
||||
local container_name="oodikone-$database"
|
||||
local database_name
|
||||
database_name="$(postgres_database_name "$database")"
|
||||
local dump_file="$LOCAL_DUMP_DIR/$database.sql"
|
||||
|
||||
infomsg "Dumping $database from container $container_name"
|
||||
# Dump to a temporary file first, so that a failed dump doesn't overwrite the previous one
|
||||
docker exec -i "$container_name" pg_dump -Fp -U postgres -d "$database_name" > "$dump_file.tmp"
|
||||
mv "$dump_file.tmp" "$dump_file"
|
||||
cp --remove-destination "$dump_file" "$TEST_DUMP_DIR/$database.sql"
|
||||
successmsg "Dumped $database to $dump_file"
|
||||
}
|
||||
|
||||
resolve_databases "$@"
|
||||
make_dump_dirs
|
||||
|
||||
for database in "${selected_databases[@]}"; do
|
||||
dump_database "$database"
|
||||
done
|
||||
@@ -0,0 +1,131 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
# Shared configuration and helpers for the anonyymioodi database scripts.
|
||||
# Meant to be sourced by the other scripts in this folder, not run on its own.
|
||||
|
||||
# The configuration below is used by the sourcing scripts, not by this file
|
||||
# shellcheck disable=SC2034
|
||||
|
||||
PROJECT_ROOT="$(git rev-parse --show-toplevel)"
|
||||
source "$PROJECT_ROOT"/scripts/utils.sh
|
||||
|
||||
# Dumps taken from running docker containers
|
||||
LOCAL_DUMP_DIR="$PROJECT_ROOT/.databasedumps/local_containers"
|
||||
# Dumps downloaded from s3
|
||||
S3_DUMP_DIR="$PROJECT_ROOT/.databasedumps/s3"
|
||||
# Where db.Dockerfile copies the .sql files from when an image is built
|
||||
TEST_DUMP_DIR="$PROJECT_ROOT/.databasedumps/test"
|
||||
|
||||
# Used as a fallback when ACCESS_KEY and SECRET_KEY aren't set in the environment
|
||||
S3_CONFIG_FILE=~/.s3cfg
|
||||
S3_HOST="s3.datacloud.helsinki.fi"
|
||||
S3_BUCKET="s3://oodikone"
|
||||
# Dumps live under a prefix of their own inside the bucket
|
||||
S3_TARGET="$S3_BUCKET/test-dbs"
|
||||
|
||||
REGISTRY="registry-toska.ext.ocp-prod-0.k8s.it.helsinki.fi"
|
||||
|
||||
# Databases, named after their docker compose services
|
||||
DATABASES=("user-db" "kone-db" "sis-db" "sis-importer-db" "jami-db")
|
||||
|
||||
# Set by resolve_databases, read by the scripts sourcing this file
|
||||
selected_databases=()
|
||||
|
||||
# The name of the postgres database inside the container. It matches the service
|
||||
# name for every database except jami-db.
|
||||
postgres_database_name() {
|
||||
local database="$1"
|
||||
if [[ "$database" == "jami-db" ]]; then
|
||||
echo "postgres"
|
||||
else
|
||||
echo "$database"
|
||||
fi
|
||||
}
|
||||
|
||||
# Validates the given database names and expands "all" to every database.
|
||||
# The result is set to the global array $selected_databases, because dying
|
||||
# inside a subshell wouldn't stop the calling script.
|
||||
resolve_databases() {
|
||||
selected_databases=()
|
||||
|
||||
if [[ $# -eq 0 ]]; then
|
||||
die "No database given, expected one or more of: ${DATABASES[*]} all"
|
||||
fi
|
||||
|
||||
for argument in "$@"; do
|
||||
if [[ "$argument" == "all" ]]; then
|
||||
selected_databases=("${DATABASES[@]}")
|
||||
return
|
||||
fi
|
||||
|
||||
# Check if given databases are correct ones
|
||||
local known=false
|
||||
for database in "${DATABASES[@]}"; do
|
||||
if [[ "$argument" == "$database" ]]; then known=true; fi
|
||||
done
|
||||
$known || die "Unknown database: $argument. Expected one or more of: ${DATABASES[*]} all"
|
||||
|
||||
selected_databases+=("$argument")
|
||||
done
|
||||
}
|
||||
|
||||
# Dumps in s3 are named after the branch they were uploaded from, so the branch
|
||||
# name has to be usable as a part of a file name. E.g. feature/foo -> feature-foo.
|
||||
# feature/foo would result in a new bucket "feature" with a sub-bucket "foo"
|
||||
sanitize_branch_name() {
|
||||
echo "${1//[^A-Za-z0-9._-]/-}"
|
||||
}
|
||||
|
||||
current_branch_name() {
|
||||
local branch
|
||||
branch="$(git rev-parse --abbrev-ref HEAD)"
|
||||
if [[ "$branch" == "HEAD" ]]; then # Detached head, use the commit instead
|
||||
branch="$(git rev-parse --short HEAD)"
|
||||
fi
|
||||
sanitize_branch_name "$branch"
|
||||
}
|
||||
|
||||
has_s3_env_credentials() {
|
||||
[[ -n "${ACCESS_KEY-}" && -n "${SECRET_KEY-}" ]]
|
||||
}
|
||||
|
||||
# Credentials come from ACCESS_KEY and SECRET_KEY when both are set, and from
|
||||
# $S3_CONFIG_FILE when neither is
|
||||
require_s3_config() {
|
||||
command -v s3cmd > /dev/null || die "s3cmd not found, please install it first"
|
||||
|
||||
if [[ -n "${ACCESS_KEY-}" || -n "${SECRET_KEY-}" ]]; then
|
||||
has_s3_env_credentials || die "Only one of ACCESS_KEY and SECRET_KEY is set, set both or neither"
|
||||
return
|
||||
fi
|
||||
|
||||
[[ -f "$S3_CONFIG_FILE" ]] || die "Set ACCESS_KEY and SECRET_KEY, or add $S3_CONFIG_FILE. The \".s3cfg\" config file can be found on toska/dokumentaatio repo"
|
||||
}
|
||||
|
||||
s3_env_config() {
|
||||
cat <<EOF
|
||||
[default]
|
||||
access_key = $ACCESS_KEY
|
||||
secret_key = $SECRET_KEY
|
||||
host_base = $S3_HOST
|
||||
host_bucket = $S3_HOST
|
||||
use_https = True
|
||||
check_ssl_certificate = True
|
||||
EOF
|
||||
}
|
||||
|
||||
# Runs s3cmd with the credentials chosen by require_s3_config. The environment
|
||||
# credentials are passed through process substitution, so that they aren't
|
||||
# written to disk or visible in the process list.
|
||||
s3() {
|
||||
if has_s3_env_credentials; then
|
||||
s3cmd -c <(s3_env_config) "$@"
|
||||
else
|
||||
s3cmd -c "$S3_CONFIG_FILE" "$@"
|
||||
fi
|
||||
}
|
||||
|
||||
# The dump directories are created lazily by the scripts that write into them
|
||||
make_dump_dirs() {
|
||||
mkdir -p "$LOCAL_DUMP_DIR" "$S3_DUMP_DIR" "$TEST_DUMP_DIR"
|
||||
}
|
||||
Executable
+145
@@ -0,0 +1,145 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
# Front end for the test database scripts in this folder. Run it without
|
||||
# arguments for an interactive menu, or give it an action and a database to run
|
||||
# it directly, e.g. in CI.
|
||||
#
|
||||
# Each action is just a call to one of the scripts next to this one, so they can
|
||||
# also be used on their own.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# Here BASH_SOURCE[0] is used instead of $0 because it works
|
||||
# consistently when ran normally and when sourced
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# shellcheck source-path=SCRIPTDIR
|
||||
# shellcheck source=lib/common.sh
|
||||
source "$SCRIPT_DIR"/lib/common.sh
|
||||
|
||||
ACTIONS=("download" "dump" "upload" "push" "release" "rebuild")
|
||||
|
||||
usage() {
|
||||
cat >&2 <<EOF
|
||||
Usage: $(basename "$0") [<action> <database>... | all]
|
||||
|
||||
Runs without arguments for an interactive menu.
|
||||
|
||||
Actions:
|
||||
download Download the newest dumps from s3
|
||||
dump Dump the databases from the running docker containers
|
||||
|
||||
upload Upload the current dumps to s3
|
||||
push Build images from the current dumps and push them to the registry
|
||||
|
||||
release Dump from the running containers, then upload and push (shorthand)
|
||||
rebuild Download the dumps from s3, then push (shorthand)
|
||||
|
||||
Databases: ${DATABASES[*]} all
|
||||
EOF
|
||||
}
|
||||
|
||||
# === Actions ===
|
||||
|
||||
download_dump() {
|
||||
"$SCRIPT_DIR"/download_dump_from_s3.sh "$1"
|
||||
}
|
||||
|
||||
dump_database() {
|
||||
"$SCRIPT_DIR"/dump_local_db.sh "$1"
|
||||
}
|
||||
|
||||
upload_dump() {
|
||||
"$SCRIPT_DIR"/upload_dump_to_s3.sh "$1"
|
||||
}
|
||||
|
||||
push_image() {
|
||||
"$SCRIPT_DIR"/build_and_push_image.sh "$1"
|
||||
}
|
||||
|
||||
run_action() {
|
||||
local action="$1"
|
||||
local database="$2"
|
||||
|
||||
case "$action" in
|
||||
download) download_dump "$database" ;;
|
||||
dump) dump_database "$database" ;;
|
||||
upload) upload_dump "$database" ;;
|
||||
push) push_image "$database" ;;
|
||||
release)
|
||||
dump_database "$database"
|
||||
upload_dump "$database"
|
||||
push_image "$database"
|
||||
;;
|
||||
rebuild)
|
||||
download_dump "$database"
|
||||
push_image "$database"
|
||||
;;
|
||||
*)
|
||||
usage
|
||||
die "Unknown action: $action"
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
run_action_for_all() {
|
||||
local action="$1"
|
||||
shift
|
||||
|
||||
# Check if command was a proper command
|
||||
local known_action=false
|
||||
for known in "${ACTIONS[@]}"; do
|
||||
if [[ "$action" == "$known" ]]; then known_action=true; fi
|
||||
done
|
||||
if ! $known_action; then
|
||||
usage
|
||||
die "Unknown action: $action"
|
||||
fi
|
||||
|
||||
resolve_databases "$@"
|
||||
for database in "${selected_databases[@]}"; do
|
||||
run_action "$action" "$database"
|
||||
done
|
||||
successmsg "Action $action done for: ${selected_databases[*]}"
|
||||
}
|
||||
|
||||
# === Interactive menu ===
|
||||
|
||||
ask_and_run() {
|
||||
local action database
|
||||
|
||||
PS3="Select action: "
|
||||
select action in "${ACTIONS[@]}" "quit"; do
|
||||
[[ -n "${action-}" ]] || continue
|
||||
if [[ "$action" == "quit" ]]; then exit 0; fi
|
||||
break
|
||||
done
|
||||
|
||||
PS3="Select database: "
|
||||
select database in "${DATABASES[@]}" "all"; do
|
||||
[[ -n "${database-}" ]] || continue
|
||||
break
|
||||
done
|
||||
|
||||
run_action_for_all "$action" "$database"
|
||||
}
|
||||
|
||||
# === Run ===
|
||||
|
||||
if [[ "${1-}" == "-h" || "${1-}" == "--help" ]]; then
|
||||
usage
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [[ $# -eq 0 ]]; then
|
||||
while true; do
|
||||
ask_and_run
|
||||
echo
|
||||
done
|
||||
fi
|
||||
|
||||
if [[ $# -lt 2 ]]; then
|
||||
usage
|
||||
die "Both an action and a database are required"
|
||||
fi
|
||||
|
||||
run_action_for_all "$@"
|
||||
Executable
+90
@@ -0,0 +1,90 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
# Upload database dumps from .databasedumps/test to s3, naming the objects
|
||||
# <database>-<git_branch>-<datetime>.sql so that dumps don't overwrite each other.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# shellcheck source-path=SCRIPTDIR
|
||||
# shellcheck source=lib/common.sh
|
||||
source "$SCRIPT_DIR"/lib/common.sh
|
||||
|
||||
usage() {
|
||||
cat >&2 <<EOF
|
||||
Usage: $(basename "$0") [-n|--dry-run] <database>... | all
|
||||
|
||||
Uploads the dumps in .databasedumps/test to $S3_TARGET
|
||||
as <database>-<git_branch>-<datetime>.sql. Run dump_local_db.sh first to
|
||||
create the dumps.
|
||||
|
||||
Databases: ${DATABASES[*]}
|
||||
|
||||
Options:
|
||||
-n, --dry-run Print the s3cmd command instead of running it
|
||||
-h, --help Show this help
|
||||
EOF
|
||||
}
|
||||
|
||||
dry_run=false
|
||||
arguments=()
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
-n | --dry-run)
|
||||
dry_run=true
|
||||
shift
|
||||
;;
|
||||
-h | --help)
|
||||
usage
|
||||
exit 0
|
||||
;;
|
||||
-*)
|
||||
usage
|
||||
die "Unknown option: $1"
|
||||
;;
|
||||
*)
|
||||
arguments+=("$1")
|
||||
shift
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
if [[ ${#arguments[@]} -eq 0 ]]; then
|
||||
usage
|
||||
die "No database given"
|
||||
fi
|
||||
|
||||
upload_database() {
|
||||
local database="$1"
|
||||
local dump_file="$TEST_DUMP_DIR/$database.sql"
|
||||
|
||||
[[ -f "$dump_file" ]] || die "No dump found at $dump_file, run dump_local_db.sh first"
|
||||
[[ -s "$dump_file" ]] || die "Dump is empty: $dump_file"
|
||||
|
||||
local datetime
|
||||
datetime="$(date +%Y%m%d-%H%M%S)"
|
||||
local target="$S3_TARGET/$database-$branch-$datetime.sql"
|
||||
|
||||
if [[ "$dry_run" == true ]]; then
|
||||
infomsg "Dry run, would run: s3cmd put $dump_file $target"
|
||||
return
|
||||
fi
|
||||
|
||||
infomsg "Pushing $dump_file to $target"
|
||||
s3 put "$dump_file" "$target"
|
||||
successmsg "Uploaded $dump_file to $target"
|
||||
}
|
||||
|
||||
resolve_databases "${arguments[@]}"
|
||||
require_s3_config
|
||||
|
||||
if [[ "$dry_run" == false ]]; then
|
||||
"$SCRIPT_DIR"/create_s3_bucket.sh
|
||||
fi
|
||||
|
||||
branch="$(current_branch_name)"
|
||||
|
||||
for database in "${selected_databases[@]}"; do
|
||||
upload_database "$database"
|
||||
done
|
||||
Reference in New Issue
Block a user