Separate studyprogamme updates into their own file

Easier to use in the cronjobs now
Add indication of loading to the new update-view
2026-10-13 03:13:40 +03:00 · 2022-01-18 10:08:43 +02:00 · 2022-01-18 09:39:04 +02:00 · 2022-01-18 09:35:06 +02:00 · 2022-01-17 10:09:13 +02:00 · 2022-01-17 09:03:06 +02:00
79 changed files with 1494 additions and 1432 deletions
@@ -9,6 +9,7 @@ on:
      - 'cypress/**'
      - 'cypress.json'
      - 'services/**'
+      - '!services/backend/src/rapodiff.js'
      - 'docker-compose.ci.yml'
      - 'docker-compose.test.yml'
      - 'nginx.conf'
@@ -2,11 +2,15 @@

 (name anonyymioodi for :culture: reasons)

-This repository contains a guide and scripts to create docker images from database dumps and/or push database images to Toska Hub docker registry. Images are mostly used by oodikone.
+This folder contains a guide and scripts to create anonymous database images for oodikone, mainly used for testing.

-## How to update database images
+## How to create / modify database images

-Updating images varies a bit depending on the db you're trying to update. Complete steps as defined below and run `./commit_and_push_to_toska_hub.sh <dbname>` to publish the image you've created.
+Creating / modifying images varies a bit depending on the db. Generally, to modify images, you can either use `psql` directly in docker by running `docker exec -it <dbname> psql -U postgres <dbname>` or start adminer with oodikone's docker-compose. 
+
+By default, previous db images from Toska Hub are used as a base. If needed, comment out Toska hub image and uncomment corresponding clean postgres image in docker-compose before running other steps.
+
+See more details on how to create / modify databases below.

 ### Sis-importer-db

@@ -15,11 +19,11 @@ The following steps are for building a clean sis-importer-db image from scratch.
 - See [sis-importer/importer-db-staging-sampletaker](https://github.com/UniversityOfHelsinkiCS/sis-importer/tree/master/importer-db-staging-sampletaker) and create `sis-importer-db.sqz` dump
 - Move `sis-importer-db.sqz` to this folder
 - Run `./create_container_from_dump.sh sis-importer-db`
- Enrich data according to what's in `enrich_data.sql` file
+- Enrich db with command in file `enrich_data.sql` either by piping command to container or by running command inside container.

 ### Sis-db:

-The following steps are for building a clean sis-importer-db image from scratch using the sis-updater. If you want to just modify the existing image a bit, follow the steps defined for user-db / kone-db.
+The following steps are for building a clean sis-db image from scratch using oodikone's updater. If you want to just modify the existing image a bit, follow the steps defined for user-db / kone-db.

 - Temporarily replace sis-db -service in oodikone's docker-compose.yml with sis-db -service from this repo's docker-compose.yml
 - Ensure the newest sis-importer-db image is pulled from toska hub in oodikone repo
@@ -32,25 +36,21 @@ curl --request GET --url 'http://localhost:8082/v1/meta?token=dev' && \
 curl --request GET --url 'http://localhost:8082/v1/programmes?token=dev' && \
 curl --request GET --url 'http://localhost:8082/v1/students?token=dev'
 ```
-
-(It might be a good idea to follow sis-updater-worker logs while updater is running)
-
+(or run these from frontend. You might also want to follow sis-updater-workers logs to ensure things are running smoothly.)
 - Clean up any extra stuff you don't want to keep in sis-db.
- (Bonus: check that cypress tests run okay)
+- Bonus: check that cypress tests run okay

 ### User-db / kone-db

- Temporarily replace user-db/kone-db -service in oodikone's docker-compose.yml with the same service from this repo's docker-compose.yml
- Ensure the newest user-db/kone-db image is pulled from the registry with `docker pull`
- Start oodikone, let it run migrations
+- Temporarily replace the db service in oodikone's docker-compose.yml with the same db service from this folder's docker-compose.yml
+- Ensure the newest user-db/kone-db image is pulled from the registry with `docker-compose pull`
 - Modify db to wanted state. Tips:
-  - Easy way to create new users is to modify dev user details in the frontend API config. A new user is created during login if the given uid is not present in user-db.
+  - Run oodikone if you want to upgrade something from frontend (e.g. tags in kone-db, courses in sis-db etc.).
+  - Easy way to create new users to user-db is to modify dev user details in the frontend API config. A new user is created during login if the given uid is not present in user-db.

-By default, previous user-db/kone-db images are used as a base. If you want to build the image from a clean Postgres image, just comment out toska hub image and uncomment the clean Postgres image.
+## Publishing the image

-## How to modify database images
-
-If you need to clean up / modify / do whatever with dbs, either use `psql` directly in docker by running `docker exec -it <dbname> psql -U postgres <dbname>` or start adminer with oodikone's docker-compose.
+After completing the steps above and making sure your local version has the correct data, run `./commit_and_push_to_toska_hub.sh <dbname>` to publish the db image to Toska Hub.

 ## How to update Postgres versions

@@ -186,7 +186,7 @@ describe('Course Statistics tests', () => {
      cy.contains('td', '010674989').siblings().contains('1')
    })

-    it.skip('Language distribution is correct', () => {
+    it('Language distribution is correct', () => {
      cy.get("input[placeholder='Search by a course code']").type('TKT20003')
      cy.contains(/^TKT20003$/).click()
      cy.contains('Fetch statistics').should('be.enabled').click()
@@ -317,7 +317,9 @@ describe('Course Statistics tests', () => {
    const attemptsTableContents = [
      // [time, passed, failed, passrate]
      ['Total*', 288, 213, 75, '73.96 %'],
+      // ['Total', 295, 216, 79, '73.22 %'],
      ['2020-2021', '5 or less students', 'NA', 'NA', 'NA'],
+      //['2020-2021', 5, 2, 3, '40.00 %'],
      ['2019-2020', 164, 121, 43, '73.78 %'],
      ['2018-2019', 85, 60, 25, '70.59 %'],
      ['2017-2018', 39, 32, 7, '82.05 %'],
@@ -22,11 +22,25 @@ const clearSingleDropdownSelection = dataCyAttribute => {
 // functions don't take any parameters and using global object for matching test step name seemed overcomplicated.
 const createRunTestStepWithPreAndPostPartsFunction = amountWithoutFiltering => {
  return (filterName, testStepFunctionToRun) => {
-    cy.get(`[data-cy="${filterName}-header"]`).click()
-    checkFilteringResult(amountWithoutFiltering)
-    testStepFunctionToRun()
-    checkFilteringResult(amountWithoutFiltering)
-    cy.get(`[data-cy="${filterName}-header"]`).click({ force: true })
+    const card = cy.cs(`${filterName}-filter-card`)
+
+    card.invoke('attr', 'data-open').then(open => {
+      console.log('open', open)
+
+      const getHeader = () => cy.cs(`${filterName}-header`)
+
+      if (open === 'false') {
+        getHeader().click({ force: true })
+      }
+
+      checkFilteringResult(amountWithoutFiltering)
+      testStepFunctionToRun()
+      checkFilteringResult(amountWithoutFiltering)
+
+      if (open === 'false') {
+        getHeader().click({ force: true })
+      }
+    })
  }
 }

@@ -42,33 +56,58 @@ describe('Population Statistics', () => {

  it('Transfer filter is on not transferred by default', () => {
    cy.contains('By default only students who have not transferred to this study programme are shown.')
-    const card = cy.cs('TransferredToProgramme-filter-card')
-    card.cs('TransferredToProgramme-header').click()
-    card.get('[data-cy="option-havenot"] input').should('be.checked')
-    card.cs('TransferredToProgramme-header').click()
+
+    runTestStepWithPreAndPostParts('TransferredToProgramme', () => {
+      const card = cy.cs('TransferredToProgramme-filter-card')
+      card.get('[data-cy="option-havenot"] input').should('be.checked')
+    })
+  })
+
+  it('Programme defaults to "Active Study Right" mode', () => {
+    runTestStepWithPreAndPostParts('Programme', () => {
+      cy.cs('Programme-filter-mode-selector').children().eq(0).should('contain', 'Active Study Right')
+    })
+  })
+
+  it('Programme filter works', () => {
+    runTestStepWithPreAndPostParts('Programme', () => {
+      const card = cy.cs('Programme-filter-card')
+      const programmeDropdown = card.cs('Programme-filter-dropdown').selectFromDropdown(0)
+      checkFilteringResult(5)
+      programmeDropdown.get('i.delete').click()
+    })
+  })
+
+  it('Study track filter works', () => {
+    runTestStepWithPreAndPostParts('StudyTrack', () => {
+      const card = cy.cs('StudyTrack-filter-card')
+      const programmeDropdown = card.cs('StudyTrack-filter-dropdown').selectFromDropdown(0)
+      checkFilteringResult(1)
+      programmeDropdown.get('i.delete').click()
+    })
  })

  it('Graduation filter works', () => {
    runTestStepWithPreAndPostParts('GraduatedFromProgramme', () => {
-      const card = cy.cs('GraduatedFromProgramme-filter-card')
+      const getCard = () => cy.cs('GraduatedFromProgramme-filter-card')

      const graduated = 11
-      card.cs('option-graduated-true').click()
+      getCard().cs('option-graduated-true').click()
      checkFilteringResult(graduated)
-      card.cs('option-graduated-false').click()
+      getCard().cs('option-graduated-false').click()
      checkFilteringResult(defaultAmountOfStudents - graduated)
-      card.get('[data-cy="GraduatedFromProgramme-filter-card"] [data-cy="option-all"]').click()
+      getCard().cs('option-all').click()
    })
  })

  it('Transfer filter works', () => {
    runTestStepWithPreAndPostParts('TransferredToProgramme', () => {
      const transferred = 1
-      cy.cs('option-have').click()
+      cy.cs('TransferredToProgramme-filter-card').cs('option-have').click()
      checkFilteringResult(transferred)
-      cy.cs('option-all').click()
+      cy.cs('TransferredToProgramme-filter-card').cs('option-all').click()
      checkFilteringResult(defaultAmountOfStudents + transferred)
-      cy.cs('option-havenot').click()
+      cy.cs('TransferredToProgramme-filter-card').cs('option-havenot').click()
    })
  })

@@ -95,7 +134,7 @@ describe('Population Statistics', () => {
    runTestStepWithPreAndPostParts('Age', () => {
      cy.cs('ageFilter-min').type('20')
      cy.cs('ageFilter-max').type('40')
-      checkFilteringResult(41)
+      checkFilteringResult(40)
      cy.cs('ageFilter-min').find('input').clear()
      cy.cs('ageFilter-max').find('input').clear()
    })
@@ -120,8 +159,8 @@ describe('Population Statistics', () => {
      cy.cs('startYearAtUni-dropdown').selectFromDropdown(0)
      checkFilteringResult(1)
      clearSemanticUIMultipleDropDownSelection('startYearAtUni-dropdown')
-      cy.cs('startYearAtUni-dropdown').selectFromDropdown(14)
-      checkFilteringResult(121)
+      cy.cs('startYearAtUni-dropdown').selectFromDropdown(7)
+      checkFilteringResult(141)
      clearSemanticUIMultipleDropDownSelection('startYearAtUni-dropdown')
    })
  })
@@ -159,14 +198,14 @@ describe('Population Statistics', () => {
  it('Filter combinations work', () => {
    runTestStepWithPreAndPostParts('GraduatedFromProgramme', () => {
      runTestStepWithPreAndPostParts('Age', () => {
-        const card = cy.cs('GraduatedFromProgramme-filter-card')
-        card.cs('option-graduated-true').click()
+        const getCard = () => cy.cs('GraduatedFromProgramme-filter-card')
+        getCard().cs('option-graduated-true').click()
        cy.cs('ageFilter-min').type('20')
        cy.cs('ageFilter-max').type('30')
        checkFilteringResult(11)
        cy.cs('ageFilter-min').find('input').clear()
        cy.cs('ageFilter-max').find('input').clear()
-        card.cs('option-all').click()
+        getCard().cs('option-all').click()
      })
    })
  })
@@ -193,11 +232,35 @@ describe('Course Statistics', () => {
    })
  })

+  it('Programme defaults to "Attainment" mode', () => {
+    runTestStepWithPreAndPostParts('Programme', () => {
+      cy.cs('Programme-filter-mode-selector').children().eq(0).should('contain', 'Attainment')
+    })
+  })
+
+  it('Programme filter works', () => {
+    runTestStepWithPreAndPostParts('Programme', () => {
+      const card = cy.cs('Programme-filter-card')
+      const programmeDropdown = card.cs('Programme-filter-dropdown').selectFromDropdown(1)
+      checkFilteringResult(116)
+      programmeDropdown.get('i.delete').click()
+    })
+  })
+
+  it('Study track filter works', () => {
+    runTestStepWithPreAndPostParts('StudyTrack', () => {
+      const card = cy.cs('StudyTrack-filter-card')
+      const programmeDropdown = card.cs('StudyTrack-filter-dropdown').selectFromDropdown(0)
+      checkFilteringResult(1)
+      programmeDropdown.get('i.delete').click()
+    })
+  })
+
  it('Age filter works', () => {
    runTestStepWithPreAndPostParts('Age', () => {
      cy.cs('ageFilter-min').type('20')
      cy.cs('ageFilter-max').type('40')
-      checkFilteringResult(32)
+      checkFilteringResult(31)
      cy.cs('ageFilter-min').find('input').clear()
      cy.cs('ageFilter-max').find('input').clear()
    })
@@ -222,8 +285,8 @@ describe('Course Statistics', () => {
      cy.cs('startYearAtUni-dropdown').selectFromDropdown([0])
      checkFilteringResult(1)
      clearSemanticUIMultipleDropDownSelection('startYearAtUni-dropdown')
-      cy.cs('startYearAtUni-dropdown').selectFromDropdown(12)
-      checkFilteringResult(29)
+      cy.cs('startYearAtUni-dropdown').selectFromDropdown(3)
+      checkFilteringResult(84)
      clearSemanticUIMultipleDropDownSelection('startYearAtUni-dropdown')
    })
  })
@@ -268,6 +331,21 @@ describe('Custom Population Statistics', () => {
    })
  })

+  it('Programme defaults to "Active Study Right" mode', () => {
+    runTestStepWithPreAndPostParts('Programme', () => {
+      cy.cs('Programme-filter-mode-selector').children().eq(0).should('contain', 'Active Study Right')
+    })
+  })
+
+  it('Programme filter works', () => {
+    runTestStepWithPreAndPostParts('Programme', () => {
+      const card = cy.cs('Programme-filter-card')
+      const programmeDropdown = card.cs('Programme-filter-dropdown').selectFromDropdown(0)
+      checkFilteringResult(5)
+      programmeDropdown.get('i.delete').click()
+    })
+  })
+
  it('Gender filter works', () => {
    runTestStepWithPreAndPostParts('Gender', () => {
      cy.cs('genderFilter-dropdown').selectFromDropdown(0)
@@ -290,7 +368,7 @@ describe('Custom Population Statistics', () => {
    })
  })

-  it('Courses filter works', () => {
+  it.skip('Courses filter works', () => {
    runTestStepWithPreAndPostParts('Courses', () => {
      const courses = ['MAT11001', 'TKT20004']
      cy.cs('courseFilter-course-dropdown')
@@ -80,7 +80,7 @@ describe('Population Statistics tests', () => {

      cy.intercept('/api/v3/courseyearlystats**').as('coursePage')
      // eslint-disable-next-line cypress/no-unnecessary-waiting
-      cy.wait(10000) // a bit hacky way, wait until ui is ready
+      cy.wait(15000) // a bit hacky way, wait until ui is ready
      cy.get('[data-cy=expand-TKT1]').click()
      cy.get('[data-cy=coursestats-link-TKT10002]').click()
      cy.wait('@coursePage')
@@ -38,7 +38,7 @@ describe('Student Statistics tests', () => {
    cy.url().should('include', '/students')
    cy.get('.prompt').type(student.studentnumber)
    cy.contains(student.studentnumber).click()
-    cy.contains('Started: 01.08.2011')
+    cy.contains('Started: 01.08.2017')
    cy.contains('Tietojenkäsittelytieteen kandiohjelma (01.08.2017 - 01.08.2025')
    cy.contains(student.lastname).should('not.exist')
    cy.contains(student.firstnames).should('not.exist')
@@ -7,7 +7,7 @@ const adminUserHeaders = {
  uid: 'admin',
  displayname: 'Admin User',
  'shib-session-id': 'mock-cypress-session',
-  hygroupcn: 'grp-oodikone-users;grp-oodikone-basic-users',
+  hygroupcn: 'grp-oodikone-users;grp-oodikone-basic-users;grp-toska',
  edupersonaffiliation: 'member;employee;faculty',
  mail: 'grp-toska+mockadminuser@helsinki.fi',
  hypersonsisuid: 'hy-hlo-6666666',
@@ -62,9 +62,13 @@ Cypress.Commands.add('init', (path = '', userId = 'basic') => {
 * Shorthand for using "Cypress Selectors" (CS), i.e., `data-cy` attributes.
 */
 Cypress.Commands.add('cs', { prevSubject: 'optional' }, (subject, name) => {
-  const p = subject === undefined ? cy : cy.wrap(subject)
+  const selector = `[data-cy="${name}"]`

-  return p.get(`[data-cy='${name}']`)
+  if (subject === undefined) {
+    return cy.get(selector)
+  } else {
+    return cy.wrap(subject).find(selector)
+  }
 })

 /**
@@ -77,7 +81,12 @@ Cypress.Commands.add('selectFromDropdown', { prevSubject: true }, (s, index) =>

  indexes.forEach(i => {
    subject.click()
-    subject.children('.menu').children().eq(i).click({ force: true })
+
+    if (typeof i === 'number') {
+      subject.children('.menu').children().eq(i).click({ force: true })
+    } else {
+      subject.children('.menu').containing(i).eq(0).click({ force: true })
+    }
  })

  // Close multiple selection so it does not block elements underneath it.
@@ -15,7 +15,8 @@
    "both": "./run.sh both anon up --detach",
    "both:real": "./run.sh both real up --detach",
    "docker:down": "./run.sh both anon down --remove-orphans; ./run.sh both real down --remove-orphans",
-    "prepare": "husky install"
+    "prepare": "husky install",
+    "rapodiff": "docker exec backend node /usr/src/app/src/rapodiff.js"
  },
  "repository": {
    "type": "git",
@@ -17,7 +17,7 @@ ENV SENTRY_ENVIRONMENT=$SENTRY_ENVIRONMENT
 ARG SENTRY_RELEASE
 ENV SENTRY_RELEASE=$SENTRY_RELEASE
 COPY --chown=node:node ./package* /usr/src/app/
-RUN npm ci --production
+RUN npm ci --production && apk add --no-cache vim curl nano
 COPY --chown=node:node . /usr/src/app/
 USER node
 CMD ["npm", "start"]
@@ -4,11 +4,7 @@
  "scripts": {
    "start": "node --max_old_space_size=4096 index.js",
    "dev": "nodemon --max_old_space_size=4096 index.js",
-    "diff:students": "node /usr/src/app/src/sisoodi_diff/students/index.js",
-    "diff:populations": "node /usr/src/app/src/sisoodi_diff/populations.js",
-    "diff:courses": "node /usr/src/app/src/sisoodi_diff/courses/index.js",
-    "diff:transfers": "node /usr/src/app/src/sisoodi_diff/transfers.js",
-    "diff:trends": "node /usr/src/app/src/sisoodi_diff/trends.js"
+    "rapodiff": "node /usr/src/app/src/rapodiff.js"
  },
  "dependencies": {
    "@sentry/node": "^5.30.0",
@@ -0,0 +1,9 @@
+module.exports = {
+  up: async queryInterface => {
+    await queryInterface.dropTable('user_hy_group')
+    await queryInterface.sequelize.query('DROP SEQUENCE IF EXISTS user_hy_group_id_seq')
+    await queryInterface.dropTable('hy_groups')
+    await queryInterface.sequelize.query('DROP SEQUENCE IF EXISTS hy_groups_id_seq')
+  },
+  down: async () => {},
+}
@@ -1,5 +1,6 @@
 const morgan = require('morgan')
 const logger = require('../util/logger')
+const _ = require('lodash')

 // So this appears to be a hack to get neatly formatted stats like response-time etc. from morgan
 // without actually using morgan what it's used for (LOGGING REQUESTS!).
@@ -7,7 +8,8 @@ const logger = require('../util/logger')
 // so don't e.g. make it into an async function or it'll print [object Promise] on every request
 const accessLogger = morgan((tokens, req, res) => {
  const fields = ['method', 'url', 'status', 'response-time', 'remote-addr', 'remote-user', 'user-agent', 'referrer']
-  const meta = { ...req.decodedToken }
+  const { user } = req
+  const meta = { ...user }
  fields.forEach(field => (meta[field] = tokens[field](req, res)))
  meta['time'] = tokens['date'](req, res, 'iso')
  if (req.route) {
@@ -15,7 +17,7 @@ const accessLogger = morgan((tokens, req, res) => {
  }

  const message = [
-    req.decodedToken.name,
+    user.name,
    ':',
    tokens['method'](req, res),
    tokens['url'](req, res),
@@ -26,7 +28,8 @@ const accessLogger = morgan((tokens, req, res) => {
  ].join(' ')

  logger.info(message, {
-    ...meta,
+    // don't log student list which might be huge
+    ..._.omit(meta, ['studentsUserCanAccess']),
    // pass this as a custom field so we can filter by it in graylog
    isUsageStats: true,
  })
@@ -1,6 +1,8 @@
 const roles = requiredRoles => async (req, res, next) => {
-  if (req.decodedToken) {
-    const { roles } = req
+  if (req.user) {
+    const {
+      user: { roles },
+    } = req

    if (requiredRoles.every(r => roles.indexOf(r) >= 0) || roles.includes('admin')) {
      return next()
@@ -1,6 +1,6 @@
 const Sentry = require('@sentry/node')
 const { ApplicationError } = require('../util/customErrors')
-const { getUser, getUserDataFor, getMockedUser } = require('../services/userService')
+const { getUser, getMockedUser } = require('../services/userService')
 const { requiredGroup } = require('../conf-backend')
 const _ = require('lodash')

@@ -8,47 +8,45 @@ const parseIamGroups = iamGroups => iamGroups?.split(';').filter(Boolean) ?? []

 const hasRequiredIamGroup = iamGroups => _.intersection(iamGroups, requiredGroup).length > 0

-const currentUserMiddleware = async (req, _, next) => {
-  const { uid: username, 'shib-session-id': sessionId, shib_logout_url: logoutUrl } = req.headers
+const currentUserMiddleware = async (req, _res, next) => {
+  const {
+    'shib-session-id': sessionId,
+    'x-show-as-user': showAsUser,
+    displayname: name,
+    hygroupcn,
+    hypersonsisuid: sisId,
+    mail: email,
+    shib_logout_url: logoutUrl,
+    uid: username,
+  } = req.headers

-  if (!sessionId) {
-    throw new ApplicationError('No session id found in request headers', 403, { logoutUrl })
+  if (!sessionId || !username) {
+    throw new ApplicationError('Not enough data in request headers', 403, { logoutUrl })
  }

-  if (!username) {
-    throw new ApplicationError('No username found in request headers', 403, { logoutUrl })
-  }
-  const { displayname: name, mail: email, hygroupcn: iamGroups, hypersonsisuid: sisId } = req.headers
-  const parsedIamGroups = parseIamGroups(iamGroups)
+  const iamGroups = parseIamGroups(hygroupcn)

-  req.decodedToken = await getUser({
+  if (!hasRequiredIamGroup(iamGroups)) {
+    throw new ApplicationError('User does not have required iam group', 403, { logoutUrl })
+  }
+
+  let user = await getUser({
    username,
    name,
    email,
-    iamGroups: parsedIamGroups,
+    iamGroups,
    sisId,
  })

-  if (!hasRequiredIamGroup(parsedIamGroups)) {
-    throw new ApplicationError('User is not enabled', 403, { logoutUrl })
+  if (showAsUser && user.isAdmin) {
+    user = await getMockedUser({ userToMock: showAsUser, mockedBy: username })
  }

-  const showAsUser = req.headers['x-show-as-user']
+  Sentry.setUser({ username: user.mockedBy ?? username })

-  if (showAsUser) {
-    const mockedUser = await getMockedUser(username, showAsUser)
-    if (mockedUser) {
-      req.decodedToken = mockedUser
-    }
-  }
+  req.user = user
  req.logoutUrl = logoutUrl

-  Sentry.setUser({ username: req.decodedToken.mockedBy || req.decodedToken.userId })
-
-  // TODO: remove this garbo
-  const userData = await getUserDataFor(req.decodedToken.userId)
-  Object.assign(req, userData)
-
  next()
 }

@@ -78,6 +78,9 @@ Credit.init(
    language: {
      type: STRING,
    },
+    is_open: {
+      type: BOOLEAN,
+    },
  },
  {
    underscored: false,
@@ -83,23 +83,6 @@ const User = sequelizeUser.define(
    timestamps: false,
  }
 )
-const HyGroup = sequelizeUser.define('hy_group', {
-  id: {
-    primaryKey: true,
-    type: Sequelize.BIGINT,
-    autoIncrement: true,
-  },
-
-  code: { type: Sequelize.STRING },
-
-  createdAt: {
-    type: Sequelize.DATE,
-  },
-
-  updatedAt: {
-    type: Sequelize.DATE,
-  },
-})

 User.hasMany(UserElementDetails, { as: 'programme' })
 UserElementDetails.belongsTo(User)
@@ -110,16 +93,12 @@ User.belongsToMany(AccessGroup, {
 })
 AccessGroup.belongsToMany(User, { through: 'user_accessgroup' })

-User.belongsToMany(HyGroup, { through: 'user_hy_group', as: 'hy_group' })
-HyGroup.belongsToMany(User, { through: 'user_hy_group' })
-
 User.hasMany(UserFaculties, { as: 'faculty', foreignKey: 'userId' })

 module.exports = {
  User,
  UserElementDetails,
  AccessGroup,
-  HyGroup,
  UserFaculties,
  sequelizeUser,
 }
@@ -0,0 +1,64 @@
+/* eslint no-console: 0 */
+
+// script needs rapo certs, so you need to run this in production pannu
+//
+// possible workflows:
+// - go to oodikone pannu, then "docker exec -it backend sh", then run this script with "npm
+// run rapodiff". Modify with vim / nano until you're happy, then copy changes to local version and commit.
+// - or bind mount a file to backend container in pannu, overriding this file. Modify that file
+// inside or outside container, however you want. Keep file in pannu, or put it to pannu's git repo (oodikone-server-setup).
+
+const axios = require('axios').default
+const fs = require('fs')
+const https = require('https')
+const _ = require('lodash')
+
+const { optimizedStatisticsOf } = require('./services/populations')
+
+const { RAPO_CERT_PATH, RAPO_KEY_PATH, RAPO_API_KEY } = process.env
+
+const agent = new https.Agent({
+  cert: fs.readFileSync(RAPO_CERT_PATH, 'utf8'),
+  key: fs.readFileSync(RAPO_KEY_PATH, 'utf8'),
+})
+
+const api = axios.create({
+  baseURL: 'https://gw.api.helsinki.fi/secure/rapo/report/',
+  headers: { 'X-Api-Key': RAPO_API_KEY },
+  httpsAgent: agent,
+})
+
+const main = async () => {
+  const facultyCode = 'H50'
+  const programme = 'KH50_005'
+  const studyrightStartDate = '2020-08-01'
+  const presence = 'LÄSNÄOLEVA'
+
+  const months = 12 // some amount of months, dunno
+  const year = 2020
+
+  const oodikoneQuery = {
+    semesters: ['FALL', 'SPRING'],
+    months,
+    studyRights: { programme },
+    year,
+  }
+
+  const oodikoneStuff = await optimizedStatisticsOf(oodikoneQuery)
+  const oodikoneStudents = oodikoneStuff.students.map(s => s.studentNumber)
+
+  const rapoUrl = encodeURI(
+    `studyrights?faculty=${facultyCode}&presence=${presence}&education=${programme}&studyrightStartDate=${studyrightStartDate}`
+  )
+
+  const { data: rapoStuff } = await api.get(rapoUrl)
+  const rapoStudents = rapoStuff.map(s => s.opiskelijanumero)
+
+  const onlyInOodikone = _.difference(oodikoneStudents, rapoStudents)
+  const onlyInRapo = _.difference(rapoStudents, oodikoneStudents)
+
+  console.log('Only in Oodikone:', onlyInOodikone)
+  console.log('Only in Rapo:', onlyInRapo)
+}
+
+main().then(() => process.exit(0))
@@ -16,7 +16,9 @@ router.get('/v2/coursesmulti', async (req, res) => {
 })

 router.get('/v3/courseyearlystats', async (req, res) => {
-  const { rights, roles } = req
+  const {
+    user: { rights, roles },
+  } = req

  const allowedRoles = roles && ['admin', 'courseStatistics'].find(role => roles.includes(role))

@@ -42,6 +44,4 @@ router.get('/v3/courseyearlystats', async (req, res) => {
  }
 })

-router.use('*', (req, res, next) => next())
-
 module.exports = router
@@ -8,7 +8,7 @@ const {

 router.get('/', async (req, res) => {
  const {
-    decodedToken: { id },
+    user: { id },
  } = req
  const customPopulationSearches = await getCustomPopulationSearchesByUser(id)
  res.json(customPopulationSearches)
@@ -16,7 +16,7 @@ router.get('/', async (req, res) => {

 router.post('/', async (req, res) => {
  const {
-    decodedToken: { id },
+    user: { id },
    body: { name, students },
  } = req

@@ -29,7 +29,7 @@ router.post('/', async (req, res) => {

 router.put('/:id', async (req, res) => {
  const {
-    decodedToken: { id: userId },
+    user: { id: userId },
    body: { students },
  } = req
  const { id } = req.params
@@ -46,7 +46,7 @@ router.put('/:id', async (req, res) => {

 router.delete('/:id', async (req, res) => {
  const {
-    decodedToken: { id: userId },
+    user: { id: userId },
  } = req
  const { id } = req.params

@@ -188,6 +188,4 @@ router.delete('/v2/studyprogrammes/:id/thesis/:course', async (req, res) => {
  }
 })

-router.use('*', (req, res, next) => next())
-
 module.exports = router
@@ -1,28 +1,19 @@
 const router = require('express').Router()
-const { getUserDataFor } = require('../services/userService')
 const { sendFeedbackToToska } = require('../services/mailservice')
 const logger = require('../util/logger')

-router.post('/email', async (req, res) => {
-  const { content } = req.body
+router.post('/email', async req => {
  const {
-    decodedToken: { userId },
+    body: { content },
+    user,
  } = req

-  const { email, full_name } = await getUserDataFor(userId)
-
-  const result = await sendFeedbackToToska({
+  await sendFeedbackToToska({
    feedbackContent: content,
-    userId,
-    userEmail: email,
-    userFullName: full_name,
+    user,
  })
-  if (result.error) {
-    return res.status(400).json(result).end()
-  }
-  logger.info(`${userId} succesfully sent some feedback mail to toska`)

-  res.status(200).json('success').end()
+  logger.info(`${user.userId} succesfully sent feedback to toska`)
 })

 module.exports = router
@@ -2,8 +2,10 @@ const router = require('express').Router()
 const userService = require('../services/userService')

 router.post('/language', async (req, res) => {
-  const { language } = req.body
-  const { userId } = req.decodedToken
+  const {
+    body: { language },
+    user: { userId },
+  } = req
  if (!['fi', 'sv', 'en'].includes(language)) {
    return res.status(400).json('invalid language')
  }
@@ -1,16 +1,19 @@
 const router = require('express').Router()
 const { ApplicationError } = require('../util/customErrors')
+const _ = require('lodash')

 router.get('/login', async (req, res) => {
-  const { decodedToken, logoutUrl } = req
+  const { user, logoutUrl } = req

-  if (!decodedToken) {
+  if (!user) {
    throw new ApplicationError('User not found', 404)
  }

  res.send({
-    token: decodedToken,
+    // don't send possibly huge list of students to frontend
+    user: _.omit(user, ['studentsUserCanAccess']),
    logoutUrl,
  })
 })
+
 module.exports = router
@@ -1,15 +1,14 @@
 const crypto = require('crypto')
 const Sentry = require('@sentry/node')
 const router = require('express').Router()
+const _ = require('lodash')
 const Population = require('../services/populations')
 const Student = require('../services/students')
 const StudyrightService = require('../services/studyrights')
-const UserService = require('../services/userService')
 const TagService = require('../services/tags')
 const CourseService = require('../services/courses')
 const StatMergeService = require('../services/statMerger')
-const { getAllStudentsUserHasInGroups } = require('../services/studyGuidanceGroups')
-const { logger } = require('../util/logger')
+const { ApplicationError } = require('../util/customErrors')

 const filterPersonalTags = (population, userId) => {
  return {
@@ -80,20 +79,13 @@ router.post('/v2/populationstatistics/coursesbycoursecode', async (req, res) =>
  if (Math.abs(to - from + 1) > maxYearsToCreatePopulationFrom) {
    return res.status(400).json({ error: `Max years to create population from is ${maxYearsToCreatePopulationFrom}` })
  }
-  let studentnumberlist
-  const studentnumbers = await Student.findByCourseAndSemesters(coursecodes, from, to)
+
  const {
-    decodedToken: { userId },
-    roles,
+    user: { isAdmin, studentsUserCanAccess },
  } = req

-  if (roles && roles.includes('admin')) {
-    studentnumberlist = studentnumbers
-  } else {
-    const unitsUserCanAccess = await UserService.getUnitsFromElementDetails(userId)
-    const codes = unitsUserCanAccess.map(unit => unit.id)
-    studentnumberlist = await Student.filterStudentnumbersByAccessrights(studentnumbers, codes)
-  }
+  const studentnumbers = await Student.findByCourseAndSemesters(coursecodes, from, to)
+  const studentnumberlist = isAdmin ? studentnumbers : _.intersection(studentnumbers, studentsUserCanAccess)

  const result = await Population.bottlenecksOf(
    {
@@ -120,19 +112,12 @@ router.post('/v2/populationstatistics/coursesbytag', async (req, res) => {
    res.status(400).json({ error: 'The body should have a tag defined' })
    return
  }
-  let studentnumberlist
-  const studentnumbers = await Student.findByTag(tag)
  const {
-    decodedToken: { userId },
-    roles,
+    user: { isAdmin, studentsUserCanAccess },
  } = req
-  if (roles && roles.includes('admin')) {
-    studentnumberlist = studentnumbers
-  } else {
-    const unitsUserCanAccess = await UserService.getUnitsFromElementDetails(userId)
-    const codes = unitsUserCanAccess.map(unit => unit.id)
-    studentnumberlist = await Student.filterStudentnumbersByAccessrights(studentnumbers, codes)
-  }
+  const studentnumbers = await Student.findByTag(tag)
+  const studentnumberlist = isAdmin ? studentnumbers : _.intersection(studentnumbers, studentsUserCanAccess)
+
  const result = await Population.bottlenecksOf(
    {
      year: 1900,
@@ -154,27 +139,17 @@ router.post('/v2/populationstatistics/coursesbytag', async (req, res) => {
 })

 router.post('/v2/populationstatistics/coursesbystudentnumberlist', async (req, res) => {
-  if (!req.body.studentnumberlist) {
-    res.status(400).json({ error: 'The body should have a studentnumberlist defined' })
-    return
-  }
-  let studentnumberlist
  const {
-    decodedToken: { userId, sisPersonId },
-    roles,
+    user: { isAdmin, studentsUserCanAccess },
+    body: { studentnumberlist: studentnumbersInReq },
  } = req

-  if (roles && roles.includes('admin')) {
-    studentnumberlist = req.body.studentnumberlist
-  } else if (req.body.usingStudyGuidanceGroups) {
-    const studentsUserCanAccess = await getAllStudentsUserHasInGroups(sisPersonId)
-    studentnumberlist = req.body.studentnumberlist.filter(s => studentsUserCanAccess.has(s))
-  } else {
-    const unitsUserCanAccess = await UserService.getUnitsFromElementDetails(userId)
-    const codes = unitsUserCanAccess.map(unit => unit.id)
-    studentnumberlist = await Student.filterStudentnumbersByAccessrights(req.body.studentnumberlist, codes)
+  if (!studentnumbersInReq) {
+    throw new ApplicationError('The body should have a studentnumberlist defined', 400)
  }

+  const studentnumberlist = isAdmin ? studentnumbersInReq : _.intersection(studentnumbersInReq, studentsUserCanAccess)
+
  const result = await Population.bottlenecksOf(
    {
      year: 1900,
@@ -195,7 +170,7 @@ router.post('/v2/populationstatistics/coursesbystudentnumberlist', async (req, r

 router.get('/v3/populationstatistics', async (req, res) => {
  const { year, semesters, studyRights: studyRightsJSON } = req.query
-  const { decodedToken } = req
+  const { user } = req

  if (!year || !semesters || !studyRightsJSON) {
    res.status(400).json({ error: 'The query should have a year, semester and studyRights defined' })
@@ -204,13 +179,13 @@ router.get('/v3/populationstatistics', async (req, res) => {
  let studyRights = null
  try {
    studyRights = JSON.parse(studyRightsJSON)
-    const { rights, roles } = req
+    const {
+      user: { rights, isAdmin },
+    } = req

-    if (!roles || !roles.includes('admin')) {
-      if (!rights.includes(studyRights.programme)) {
-        res.status(403).json([])
-        return
-      }
+    if (!isAdmin && !rights.includes(studyRights.programme)) {
+      res.status(403).json([])
+      return
    }
  } catch (e) {
    res.status(400).json({ error: 'The query had invalid studyRights' })
@@ -245,7 +220,7 @@ router.get('/v3/populationstatistics', async (req, res) => {
      return
    }

-    res.json(filterPersonalTags(result, decodedToken.id))
+    res.json(filterPersonalTags(result, user.id))
  } else {
    const result = await Population.optimizedStatisticsOf({ ...req.query, studyRights })
    if (result.error) {
@@ -254,32 +229,26 @@ router.get('/v3/populationstatistics', async (req, res) => {
      return
    }

-    res.json(filterPersonalTags(result, decodedToken.id))
+    res.json(filterPersonalTags(result, user.id))
  }
 })

 router.get('/v3/populationstatisticsbytag', async (req, res) => {
-  const { tag, studyRights: studyRightsJSON, months, year } = req.query
-  const { decodedToken } = req
+  const {
+    user: { id, isAdmin, studentsUserCanAccess },
+    query: { tag, studyRights: studyRightsJSON, months, year },
+  } = req

  if (!tag) return res.status(400).json({ error: 'The query should have a tag defined' })
  const foundTag = await TagService.findTagById(tag)
  if (!foundTag) return res.status(404).json({ error: 'Tag not found' })

  const semesters = ['FALL', 'SPRING']
-  let studentnumberlist
+
  const studentnumbers = await Student.findByTag(tag)
-  const {
-    decodedToken: { userId },
-    roles,
-  } = req
-  if (roles && roles.includes('admin')) {
-    studentnumberlist = studentnumbers
-  } else {
-    const unitsUserCanAccess = await UserService.getUnitsFromElementDetails(userId)
-    const codes = unitsUserCanAccess.map(unit => unit.id)
-    studentnumberlist = await Student.filterStudentnumbersByAccessrights(studentnumbers, codes)
-  }
+
+  const studentnumberlist = isAdmin ? studentnumbers : _.intersection(studentnumbers, studentsUserCanAccess)
+
  const studyRights = JSON.parse(studyRightsJSON)
  const newStartYear = await Population.getEarliestYear(studentnumberlist, studyRights)
  const yearDifference = Number(year) - Number(newStartYear)
@@ -301,12 +270,14 @@ router.get('/v3/populationstatisticsbytag', async (req, res) => {
    return
  }

-  res.json(filterPersonalTags(result, decodedToken.id))
+  res.json(filterPersonalTags(result, id))
 })

 router.get('/v3/populationstatisticsbycourse', async (req, res) => {
  const { coursecodes, from, to, separate: sep } = req.query
-  const { decodedToken, roles } = req
+  const {
+    user: { id, isAdmin, studentsUserCanAccess: allStudentsUserCanAccess },
+  } = req
  const separate = sep ? JSON.parse(sep) : false

  if (!coursecodes || !from || !to) {
@@ -335,14 +306,9 @@ router.get('/v3/populationstatisticsbycourse', async (req, res) => {
    studentnumbers
  )

-  let studentsUserCanAccess
-  if (roles && roles.includes('admin')) {
-    studentsUserCanAccess = new Set(studentnumbers)
-  } else {
-    const unitsUserCanAccess = await UserService.getUnitsFromElementDetails(decodedToken.userId)
-    const codes = unitsUserCanAccess.map(unit => unit.id)
-    studentsUserCanAccess = new Set(await Student.filterStudentnumbersByAccessrights(studentnumbers, codes))
-  }
+  const studentsUserCanAccess = isAdmin
+    ? new Set(studentnumbers)
+    : new Set(_.intersection(studentnumbers, allStudentsUserCanAccess))

  const randomHash = crypto.randomBytes(12).toString('hex')
  const obfuscateStudent = ({ studyrights, studentNumber, courses, gender_code }) => ({
@@ -368,26 +334,16 @@ router.get('/v3/populationstatisticsbycourse', async (req, res) => {
    return
  }

-  res.json(filterPersonalTags(result, decodedToken.id))
+  res.json(filterPersonalTags(result, id))
 })

 router.post('/v3/populationstatisticsbystudentnumbers', async (req, res) => {
-  const { studentnumberlist, usingStudyGuidanceGroups } = req.body
-  const { roles, decodedToken } = req
+  const { studentnumberlist } = req.body
+  const {
+    user: { isAdmin, id, studentsUserCanAccess },
+  } = req

-  if (usingStudyGuidanceGroups) {
-    if (!decodedToken.sisPersonId) {
-      logger.error(
-        `User ${decodedToken.userId} tried to get person groups but personId was ${decodedToken.sisPersonId} in header`
-      )
-      return res.status(400).json({ error: 'Not possible to get groups without personId' })
-    }
-  }
-
-  const studentsUserCanAccess = usingStudyGuidanceGroups
-    ? await getAllStudentsUserHasInGroups(decodedToken.sisPersonId)
-    : await UserService.getStudentsUserCanAccess(studentnumberlist, roles, decodedToken.userId)
-  const filteredStudentNumbers = studentnumberlist.filter(s => studentsUserCanAccess.has(s))
+  const filteredStudentNumbers = isAdmin ? studentnumberlist : _.intersection(studentnumberlist, studentsUserCanAccess)

  const result = await Population.optimizedStatisticsOf(
    {
@@ -402,12 +358,14 @@ router.post('/v3/populationstatisticsbystudentnumbers', async (req, res) => {
    Sentry.captureException(new Error(result.error))
    return
  }
-  res.status(200).json(filterPersonalTags(result, decodedToken.id))
+  res.status(200).json(filterPersonalTags(result, id))
 })

 router.get('/v3/populationstatistics/studyprogrammes', async (req, res) => {
-  const { rights, roles } = req
-  if (roles && roles.includes('admin')) {
+  const {
+    user: { rights, roles },
+  } = req
+  if (roles?.includes('admin')) {
    const studyrights = await StudyrightService.getAssociations()
    res.json(studyrights)
  } else {
@@ -427,6 +385,4 @@ router.get('/v3/populationstatistics/maxYearsToCreatePopulationFrom', async (req
  return res.json(maxYearsToCreatePopulationFrom)
 })

-router.use('*', (req, res, next) => next())
-
 module.exports = router
@@ -6,6 +6,4 @@ router.get('/semesters/codes', async (req, res) => {
  res.json(providers)
 })

-router.use('*', (req, res, next) => next())
-
 module.exports = router
@@ -1,8 +1,6 @@
 const router = require('express').Router()
 const Student = require('../services/students')
-const userService = require('../services/userService')
-const Unit = require('../services/units')
-const { getAllStudentsUserHasInGroups } = require('../services/studyGuidanceGroups')
+const { ApplicationError } = require('../util/customErrors')

 const filterStudentTags = (student, userId) => {
  return {
@@ -13,8 +11,7 @@ const filterStudentTags = (student, userId) => {

 router.get('/students', async (req, res) => {
  const {
-    roles,
-    decodedToken: { userId, sisPersonId },
+    user: { isAdmin, studentsUserCanAccess },
    query: { searchTerm },
  } = req

@@ -26,68 +23,30 @@ router.get('/students', async (req, res) => {
      .slice(0, 2)
      .find(t => t.length > 3)
  ) {
-    return res.status(400).json({ error: 'at least one search term must be longer than 3 characters' })
+    throw new ApplicationError('at least one search term must be longer than 3 characters', 400)
  }

-  if (roles && roles.includes('admin')) {
-    let results = []
-    if (trimmedSearchTerm) {
-      results = await Student.bySearchTerm(trimmedSearchTerm)
-    }
-    return res.status(200).json(results).end()
-  } else {
-    const unitsUserCanAccess = await userService.getUnitsFromElementDetails(userId)
-    const codes = unitsUserCanAccess.map(unit => unit.id)
-    let extraStudents = []
-    if (roles.includes('studyGuidanceGroups')) {
-      const matchingOnlyByTerm = await Student.bySearchTerm(trimmedSearchTerm)
-      const studentsUserCanAccess = await getAllStudentsUserHasInGroups(sisPersonId)
-      extraStudents.push(...matchingOnlyByTerm.filter(s => studentsUserCanAccess.has(s.studentNumber)))
-    }
-    const matchingStudents = await Student.bySearchTermAndElements(trimmedSearchTerm, codes)
-    res.json([...matchingStudents, ...extraStudents])
+  let results = []
+  if (trimmedSearchTerm) {
+    results = isAdmin
+      ? await Student.bySearchTerm(trimmedSearchTerm)
+      : await Student.bySearchTermAndStudentNumbers(trimmedSearchTerm, studentsUserCanAccess)
  }
+  res.json(results)
 })

 router.get('/students/:id', async (req, res) => {
  const { id: studentId } = req.params
-  const { roles, decodedToken } = req
+  const {
+    user: { id, isAdmin, studentsUserCanAccess },
+  } = req

-  if (roles?.includes('admin')) {
-    const results = await Student.withId(studentId)
-    return results.error
-      ? res.status(400).json({ error: 'error finding student' }).end()
-      : res.status(200).json(filterStudentTags(results, decodedToken.id)).end()
+  if (!isAdmin && !studentsUserCanAccess.includes(studentId)) {
+    throw new ApplicationError('Error finding student', 400)
  }

-  const uid = req.decodedToken.userId
  const student = await Student.withId(studentId)
-  if (student.error) {
-    return res.status(400).json({ error: 'error finding student' }).end()
-  }
-  const units = await userService.getUnitsFromElementDetails(uid)
-
-  const rights = await Promise.all(
-    units.map(async unit => {
-      const jtn = await Unit.hasStudent(unit.id, student.studentNumber)
-      return jtn
-    })
-  )
-
-  if (roles?.includes('studyGuidanceGroups')) {
-    const studentsUserCanAccess = await getAllStudentsUserHasInGroups(decodedToken.sisPersonId)
-    if (studentsUserCanAccess.has(studentId)) {
-      return res.status(200).json(filterStudentTags(student, decodedToken.id)).end()
-    }
-  }
-
-  if (rights.some(right => right !== null)) {
-    res.status(200).json(filterStudentTags(student, decodedToken.id)).end()
-  } else {
-    res.status(400).json({ error: 'error finding student' }).end()
-  }
+  res.json(filterStudentTags(student, id))
 })

-router.use('*', (req, res, next) => next())
-
 module.exports = router
@@ -4,7 +4,7 @@ const { getAllGroupsAndStudents, changeGroupTags } = require('../services/studyG

 router.get('/', async (req, res) => {
  const {
-    decodedToken: { uid, sisPersonId },
+    user: { uid, sisPersonId },
  } = req
  if (!sisPersonId) {
    logger.error(`User ${uid} tried to get person groups but personId was ${sisPersonId} in header`)
@@ -15,7 +15,7 @@ router.get('/', async (req, res) => {

 router.put('/:id/tags', async (req, res) => {
  const {
-    decodedToken: { uid, sisPersonId },
+    user: { uid, sisPersonId },
    body: tags,
    params: { id: groupId },
  } = req
@@ -5,7 +5,6 @@ const {
  getGraduationStatsForStudytrack,
 } = require('../services/studyprogrammeStats')
 const { getStudytrackStatsForStudyprogramme } = require('../services/studytrackStats')
-
 const {
  getBasicStats,
  setBasicStats,
@@ -16,6 +15,7 @@ const {
  getStudytrackStats,
  setStudytrackStats,
 } = require('../services/analyticsService')
+const { updateBasicView, updateStudytrackView } = require('../services/studyprogrammeUpdates')
 const logger = require('../util/logger')

 router.get('/v2/studyprogrammes/:id/basicstats', async (req, res) => {
@@ -126,6 +126,34 @@ router.get('/v2/studyprogrammes/:id/studytrackstats', async (req, res) => {
  }
 })

-router.use('*', (req, res, next) => next())
+router.get('/v2/studyprogrammes/:id/update_basicview', async (req, res) => {
+  const code = req.params.id
+  if (code) {
+    let result = null
+    try {
+      result = await updateBasicView(code)
+    } catch (e) {
+      logger.error(`Failed to update code ${code} basic stats: ${e}`)
+    }
+    return res.json(result)
+  } else {
+    res.status(422).end()
+  }
+})
+
+router.get('/v2/studyprogrammes/:id/update_studytrackview', async (req, res) => {
+  const code = req.params.id
+  if (code) {
+    let result = null
+    try {
+      result = await updateStudytrackView(code)
+    } catch (e) {
+      logger.error(`Failed to update code ${code} studytrack stats: ${e}`)
+    }
+    return res.json(result)
+  } else {
+    res.status(422).end()
+  }
+})

 module.exports = router
@@ -12,48 +12,50 @@ const filterRelevantStudentTags = (studentTags, userId) => {
  return studentTags.filter(({ tag }) => !tag.personal_user_id || tag.personal_user_id === userId)
 }

-router.get('/tags', async (req, res) => {
+router.get('/tags', async (_req, res) => {
  const tags = await Tags.findTags()
  res.status(200).json(tags)
 })

 router.get('/tags/:studytrack', async (req, res) => {
  const { studytrack } = req.params
-  const { rights, roles, decodedToken } = req
+  const {
+    user: { rights, roles, id },
+  } = req

-  if (!rights.includes(studytrack) && !(roles && roles.includes('admin'))) return res.status(403).end()
+  if (!rights.includes(studytrack) && !roles?.includes('admin')) return res.status(403).end()

  const tags = await Tags.findTagsByStudytrack(studytrack)
-  res.status(200).json(filterRelevantTags(tags, decodedToken.id))
+  res.status(200).json(filterRelevantTags(tags, id))
 })

 router.post('/tags', async (req, res) => {
  const {
    tag: { studytrack, tagname, year, personal_user_id },
  } = req.body
-  const { rights, roles, decodedToken } = req
+  const {
+    user: { rights, roles, id },
+  } = req

-  if (!rights.includes(studytrack) && !(roles && roles.includes('admin'))) return res.status(403).end()
+  if (!rights.includes(studytrack) && !roles?.includes('admin')) return res.status(403).end()

  await Tags.createNewTag({ studytrack, tagname, year, personal_user_id })
  const tags = await Tags.findTagsByStudytrack(studytrack)
-  res.status(200).json(filterRelevantTags(tags, decodedToken.id))
+  res.status(200).json(filterRelevantTags(tags, id))
 })

 router.delete('/tags', async (req, res) => {
  const { tag } = req.body
-  const { rights, roles, decodedToken } = req
+  const {
+    user: { rights, roles, id },
+  } = req

-  if (
-    !rights.includes(tag.studytrack) &&
-    !(roles && roles.includes('admin')) &&
-    !(tag.personal_user_id === decodedToken.id)
-  )
+  if (!rights.includes(tag.studytrack) && !roles?.includes('admin') && !(tag.personal_user_id === id))
    return res.status(403).end()

  await Tags.deleteTag(tag)
  const t = await Tags.findTagsByStudytrack(tag.studytrack)
-  res.status(200).json(filterRelevantTags(t, decodedToken.id))
+  res.status(200).json(filterRelevantTags(t, id))
 })

 router.get('/studenttags', async (req, res) => {
@@ -63,37 +65,43 @@ router.get('/studenttags', async (req, res) => {

 router.get('/studenttags/:studytrack', async (req, res) => {
  const { studytrack } = req.params
-  const { rights, roles, decodedToken } = req
+  const {
+    user: { rights, roles, id },
+  } = req

-  if (!rights.includes(studytrack) && !(roles && roles.includes('admin'))) return res.status(403).end()
+  if (!rights.includes(studytrack) && !roles?.includes('admin')) return res.status(403).end()

  const result = await TagStudent.getStudentTagsByStudytrack(studytrack)
-  res.status(200).json(filterRelevantStudentTags(result, decodedToken.id))
+  res.status(200).json(filterRelevantStudentTags(result, id))
 })

 router.get('/studenttags/:studentnumber', async (req, res) => {
  const { studentnumber } = req.params
-  const { decodedToken } = req
+  const { user } = req
  const result = await TagStudent.getStudentTagsByStudentnumber(studentnumber)
-  res.status(200).json(filterRelevantStudentTags(result, decodedToken.id))
+  res.status(200).json(filterRelevantStudentTags(result, user.id))
 })

 router.post('/studenttags/:studentnumber', async (req, res) => {
  const { tag, studytrack } = req.body
-  const { rights, roles, decodedToken } = req
+  const {
+    user: { roles, rights, id },
+  } = req

-  if (!rights.includes(studytrack) && !(roles && roles.includes('admin'))) return res.status(403).end()
+  if (!rights.includes(studytrack) && !roles?.includes('admin')) return res.status(403).end()

  await TagStudent.createStudentTag(tag)
  const result = await TagStudent.getStudentTagsByStudytrack(studytrack)
-  res.status(200).json(filterRelevantStudentTags(result, decodedToken.id))
+  res.status(200).json(filterRelevantStudentTags(result, id))
 })

 router.post('/studenttags', async (req, res) => {
  const { tags, studytrack } = req.body
-  const { rights, roles, decodedToken } = req
+  const {
+    user: { rights, roles, id },
+  } = req

-  if (!rights.includes(studytrack) && !(roles && roles.includes('admin'))) return res.status(403).end()
+  if (!rights.includes(studytrack) && !roles?.includes('admin')) return res.status(403).end()

  const existingtags = await Tags.findTagsByStudytrack(studytrack)
  const existingTagids = existingtags.map(t => t.tag_id)
@@ -110,37 +118,39 @@ router.post('/studenttags', async (req, res) => {

  await TagStudent.createMultipleStudentTags(tags)
  const result = await TagStudent.getStudentTagsByStudytrack(studytrack)
-  res.status(200).json(filterRelevantStudentTags(result, decodedToken.id))
+  res.status(200).json(filterRelevantStudentTags(result, id))
 })

 router.delete('/studenttags/delete_one', async (req, res) => {
  const { tag_id, studentnumber, studytrack } = req.body
-  const { rights, roles, decodedToken } = req
+  const {
+    user: { rights, roles, id },
+  } = req

-  if (!rights.includes(studytrack) && !(roles && roles.includes('admin'))) return res.status(403).end()
+  if (!rights.includes(studytrack) && !roles?.includes('admin')) return res.status(403).end()

  const tags = await Tags.findTagsFromStudytrackById(studytrack, [tag_id])
  if (tags.length === 0) return res.status(403).json({ error: 'The tag does not exist' })

  await TagStudent.deleteStudentTag(studentnumber, tag_id)
  const result = await TagStudent.getStudentTagsByStudytrack(studytrack)
-  res.status(200).json(filterRelevantStudentTags(result, decodedToken.id))
+  res.status(200).json(filterRelevantStudentTags(result, id))
 })

 router.delete('/studenttags/delete_many', async (req, res) => {
  const { tagId, studentnumbers, studytrack } = req.body
-  const { rights, roles, decodedToken } = req
+  const {
+    user: { rights, roles, id },
+  } = req

-  if (!rights.includes(studytrack) && !(roles && roles.includes('admin'))) return res.status(403).end()
+  if (!rights.includes(studytrack) && !roles?.includes('admin')) return res.status(403).end()

  const tags = await Tags.findTagsFromStudytrackById(studytrack, [tagId])
  if (tags.length === 0) return res.status(403).json({ error: 'The tag does not exist' })

  await TagStudent.deleteMultipleStudentTags(tagId, studentnumbers)
  const result = await TagStudent.getStudentTagsByStudytrack(studytrack)
-  res.status(200).json(filterRelevantStudentTags(result, decodedToken.id))
+  res.status(200).json(filterRelevantStudentTags(result, id))
 })

-router.use('*', (req, res, next) => next())
-
 module.exports = router
@@ -42,7 +42,9 @@ router.get('/top/categories', async (req, res) => {
 })

 router.get('/stats', async (req, res) => {
-  const { rights, roles } = req
+  const {
+    user: { rights, roles },
+  } = req

  const { providers, semesterStart, semesterEnd } = req.query
  if (!providers || !semesterStart) {
@@ -78,6 +80,4 @@ router.get('/:id', async (req, res) => {
  res.json(result)
 })

-router.use('*', (req, res, next) => next())
-
 module.exports = router
@@ -9,8 +9,12 @@ router.post('/event', (req, res) => {
    return res.status(400).json({ error: 'group, name and value are required' })
  }

+  const {
+    user: { userId },
+  } = req
+
  // don't await here because frontend doesn't care if it succeeds
-  tsaService.sendTsaEvent(req.decodedToken.userId, { group, name, label, value }).catch(e => {
+  tsaService.sendTsaEvent(userId, { group, name, label, value }).catch(e => {
    Sentry.withScope(scope => {
      // if error was caused by a 400, dig out the JSON response for Sentry
      if (e.isAxiosError && e.response && e.response.data) {
@@ -1,12 +0,0 @@
-const LRU = require('lru-cache')
-
-// Max 25 users can be stored in the cache
-const userDataCache = new LRU({
-  max: 25,
-  length: () => 1,
-  maxAge: 1000 * 60 * 60,
-})
-
-module.exports = {
-  userDataCache,
-}
@@ -13,7 +13,7 @@ const {
 const { parseCredit } = require('./parseCredits')
 const Op = Sequelize.Op
 const { CourseYearlyStatsCounter } = require('./course_yearly_stats_counter')
-const _ = require('lodash')
+const { sortMainCode, getSortRank } = require('../util/utils')

 const byNameOrCode = (searchTerm, language) =>
  Course.findAll({
@@ -59,7 +59,15 @@ const findOneByCode = code => {
  })
 }

-const creditsForCourses = async (codes, anonymizationSalt) => {
+const creditsForCourses = async (codes, anonymizationSalt, unifyOpenUniCourses) => {
+  let is_open = false
+
+  if (unifyOpenUniCourses) {
+    is_open = {
+      [Op.in]: [false, true],
+    }
+  }
+
  const credits = await Credit.findAll({
    include: [
      {
@@ -105,9 +113,11 @@ const creditsForCourses = async (codes, anonymizationSalt) => {
      student_studentnumber: {
        [Op.ne]: null,
      },
+      [Op.or]: [{ is_open }, { is_open: null }],
    },
    order: [['attainment_date', 'ASC']],
  })
+
  const parsedCredits = credits.map(credit => parseCredit(credit, anonymizationSalt))
  return parsedCredits
 }
@@ -144,47 +154,20 @@ const unifyOpenUniversity = code => {
  return regexresult[1]
 }

-const allCodeAltenatives = async code => {
-  let course = await Course.findAll({
-    raw: true,
-    attributes: ['id', 'code', 'substitutions'],
-    where: {
-      code: code,
-    },
+const allCodeAlternatives = async code => {
+  const course = await Course.findOne({
+    where: { code: code },
  })

-  const allSubstitutions = _.flatten(course.map(c => c.substitutions))
-
-  let subcodes = []
-  if (allSubstitutions) {
-    subcodes = await Course.findAll({
-      raw: true,
-      attributes: ['code'],
-      where: {
-        id: {
-          [Op.in]: allSubstitutions,
-        },
-      },
-    })
-  }
-
-  const courses = [...course, ...subcodes]
-  const temp = courses
-    .map(c => c.code)
-    .map(c => {
-      if (c.match(/^AY?/)) return [c, 4] // open university codes come last
-      if (c.match(/^\d/)) return [c, 2] // old numeric codes come second
-      if (c.match(/^[A-Za-z]/)) return [c, 1] // new letter based codes come first
-      return [c, 3] // unknown, comes before open uni?
-    })
-    .sort((a, b) => a[1] - b[1])
-    .map(c => c[0])
-
-  return allSubstitutions ? _.uniq(temp) : [code]
+  return sortMainCode([...course.substitutions, code])
 }

 const yearlyStatsOfNew = async (coursecode, separate, unifyOpenUniCourses, anonymizationSalt) => {
-  let codes = await allCodeAltenatives(coursecode)
+  const courseForSubs = await Course.findOne({
+    where: { code: coursecode },
+  })
+
+  let codes = courseForSubs.substitutions ? sortMainCode([...courseForSubs.substitutions, coursecode]) : [coursecode]

  if (isOpenUniCourseCode(coursecode) && !unifyOpenUniCourses) {
    codes = [coursecode]
@@ -195,7 +178,7 @@ const yearlyStatsOfNew = async (coursecode, separate, unifyOpenUniCourses, anony
  }

  const [credits, course] = await Promise.all([
-    creditsForCourses(codes, anonymizationSalt),
+    creditsForCourses(codes, anonymizationSalt, unifyOpenUniCourses),
    Course.findOne({
      where: {
        code: coursecode,
@@ -204,6 +187,7 @@ const yearlyStatsOfNew = async (coursecode, separate, unifyOpenUniCourses, anony
  ])

  const counter = new CourseYearlyStatsCounter()
+
  for (let credit of credits) {
    const {
      studentnumber,
@@ -357,10 +341,6 @@ const codeLikeTerm = code =>

 const byNameAndOrCodeLike = async (name, code) => {
  let rawCourses = await Course.findAll({
-    include: {
-      model: Organization,
-      required: true,
-    },
    where: {
      ...nameLikeTerm(name),
      ...codeLikeTerm(code),
@@ -369,9 +349,9 @@ const byNameAndOrCodeLike = async (name, code) => {

  const courses = rawCourses
    .map(course => {
-      return { ...course.dataValues, organizations: course.organizations.map(o => o.id) }
+      return { ...course.dataValues }
    })
-    .sort(a => (a.code.match(/^[A-Za-z]{3}[0-9]{1}/) ? -1 : 1))
+    .sort((x, y) => getSortRank(y.code) - getSortRank(x.code))

  let substitutionGroupIndex = 0
  const visited = []
@@ -379,7 +359,11 @@ const byNameAndOrCodeLike = async (name, code) => {
  const organizeSubgroups = course => {
    if (visited.includes(course.code)) return

-    let temp = courses.filter(c => (course.substitutions ? course.substitutions.includes(c.id) : false))
+    let temp = []
+    if (course.substitutions !== null) {
+      temp = courses.filter(c => course.substitutions.includes(c.code))
+    }
+
    temp.unshift(course)
    temp.forEach(cu => {
      if (visited.includes(course.code)) return
@@ -407,6 +391,6 @@ module.exports = {
  byCodes,
  maxYearsToCreatePopulationFrom,
  unifyOpenUniversity,
-  allCodeAltenatives,
+  allCodeAlternatives,
  findOneByCode,
 }
@@ -1,6 +1,6 @@
 const axios = require('axios')
 const { isProduction, pateToken } = require('../conf-backend')
-const logger = require('../util/logger')
+const { ApplicationError } = require('../util/customErrors')

 const pateClient = axios.create({
  baseURL: 'https://pate.toska.cs.helsinki.fi',
@@ -20,33 +20,22 @@ const baseSettings = {
 }

 const sendEmail = async (options = {}) => {
-  if (!isProduction) {
-    const errorMessage = 'Skipped sending email in non-production environment'
-    logger.error({ message: errorMessage, meta: options })
-    return { error: errorMessage }
-  }
-
-  if (!pateToken) {
-    const errorMessage = 'Skipped sending email since pate token was not available'
-    logger.error({ message: errorMessage, meta: options })
-    return { error: errorMessage }
-  }
-
-  const { data } = await pateClient.post('/', options)
-
-  return data
+  if (!isProduction) throw new ApplicationError('Email sending is disabled in development mode.')
+  if (!pateToken) throw new ApplicationError('Email sending failed because pate token is missing.')
+  await pateClient.post('/', options)
 }

-const sendFeedbackToToska = ({ feedbackContent, userId, userEmail, userFullName }) => {
-  const userDetails = `Sent by ${userFullName}, userid: ${userId}, email: ${userEmail}`
+const sendFeedbackToToska = async ({ feedbackContent, user }) => {
+  const { name, userId, email } = user
+  const userDetails = `Sent by ${name}, userid: ${userId}, email: ${email}`
  const text = [feedbackContent, userDetails].join('<br />')
-  return sendEmail({
+  await sendEmail({
    template,
    emails: [
      {
        to: 'Toska <grp-toska@helsinki.fi>',
-        replyTo: userEmail,
-        subject: `Oodikone feedback from ${userFullName}`,
+        replyTo: email,
+        subject: `Oodikone feedback from ${name}`,
        text,
      },
    ],
@@ -1,6 +1,7 @@
 const { Op } = require('sequelize')
 const moment = require('moment')
 const { sortBy } = require('lodash')
+const { sortMainCode } = require('../util/utils')

 const {
  Student,
@@ -19,7 +20,6 @@ const {
  dbConnections: { sequelize },
 } = require('../database/connection')
 const { Tag, TagStudent } = require('../models/models_kone')
-const { /*allCodeAltenatives, */ findOneByCode } = require('./courses')
 const { CourseStatsCounter } = require('./course_stats_counter')
 const { getPassingSemester, semesterEnd, semesterStart } = require('../util/semester')
 const { getAllProgrammes } = require('./studyrights')
@@ -819,10 +819,6 @@ const findCourses = async (studentnumbers, beforeDate) => {
          },
        },
      },
-      /* {
-        attributes: ['discipline_id', 'name'],
-        model: Discipline
-      }, */
      {
        attributes: ['coursetypecode', 'name'],
        model: CourseType,
@@ -845,12 +841,24 @@ const parseCreditInfo = credit => ({
  date: credit.attainment_date,
 })

-const isOpenUniCourseCode = code => code.match(/^AY?(.+?)(?:en|fi|sv)?$/)
+const findMainCourse = async code => {
+  const course = await Course.findOne({
+    where: { code: code },
+  })

-const unifyOpenUniversity = code => {
-  const regexresult = isOpenUniCourseCode(code)
-  if (!regexresult) return code
-  return regexresult[1]
+  let result = []
+
+  if (course.substitutions) {
+    const sortedCourses = sortMainCode([...course.substitutions, code])
+    result = await Course.findOne({
+      attributes: ['code', 'name'],
+      where: { code: sortedCourses[0] },
+    })
+  } else {
+    return course
+  }
+
+  return result
 }

 const bottlenecksOf = async (query, studentnumberlist) => {
@@ -921,6 +929,7 @@ const bottlenecksOf = async (query, studentnumberlist) => {
          transferredStudents,
          tag
        ))
+
      const allstudents = studentnumbers.reduce((numbers, num) => {
        numbers[num] = true
        return numbers
@@ -957,14 +966,14 @@ const bottlenecksOf = async (query, studentnumberlist) => {
  for (const course of courses) {
    let { course_type } = course
    let maincourse = course
-    if (isOpenUniCourseCode) {
+    const sortedMaincourse = await findMainCourse(course.code)
+    if (sortedMaincourse) maincourse = sortedMaincourse
+
+    /*  if (isOpenUniCourseCode) {
      const nonOpen = unifyOpenUniversity(course.code)
      const response = await findOneByCode(nonOpen)
+ */

-      if (response) {
-        maincourse = response
-      }
-    }
    if (!stats[maincourse.code]) {
      stats[maincourse.code] = new CourseStatsCounter(maincourse.code, maincourse.name, allstudentslength)
    }
@@ -985,6 +994,7 @@ const bottlenecksOf = async (query, studentnumberlist) => {

  bottlenecks.coursestatistics = Object.values(stats).map(coursestatistics => coursestatistics.getFinalStats())
  bottlenecks.allStudents = allstudentslength
+
  return bottlenecks
 }

@@ -288,56 +288,71 @@ const bySearchTerm = async searchterm => {
  return matches.map(formatStudent)
 }

-const bySearchTermAndElements = async (searchterm, codes) => {
+const bySearchTermAndStudentNumbers = async (searchterm, studentnumbers) => {
  const terms = splitByEmptySpace(searchterm)
  const matches = await Student.findAll({
-    include: {
-      model: StudyrightElement,
-      required: true,
-      where: {
-        code: {
-          [Op.in]: codes,
+    where: {
+      [Op.and]: {
+        [Op.or]: [nameLike(terms), studentnumberLike(terms)],
+        studentnumber: {
+          [Op.in]: studentnumbers,
        },
      },
    },
-    where: {
-      [Op.or]: [nameLike(terms), studentnumberLike(terms)],
-    },
  })
  return matches.map(formatStudent)
 }

-const filterStudentnumbersByAccessrights = async (studentnumbers, codes) => {
-  const students = await Student.findAll({
-    attributes: ['studentnumber'],
-    include: {
-      attributes: [],
-      model: StudyrightElement,
-      required: true,
-      where: {
-        code: {
-          [Op.in]: codes,
+const filterStudentnumbersByAccessrights = async (studentnumbers, codes) =>
+  (
+    await Student.findAll({
+      attributes: ['studentnumber'],
+      include: {
+        attributes: [],
+        model: StudyrightElement,
+        required: true,
+        where: {
+          code: {
+            [Op.in]: codes,
+          },
        },
      },
-    },
-    where: {
-      studentnumber: {
-        [Op.in]: studentnumbers,
+      where: {
+        studentnumber: {
+          [Op.in]: studentnumbers,
+        },
      },
-    },
-    raw: true,
-  })
-  return students.map(student => student.studentnumber)
-}
+      raw: true,
+    })
+  ).map(({ studentnumber }) => studentnumber)
+
+const getStudentnumbersByElementdetails = async codes =>
+  (
+    await Student.findAll({
+      attributes: ['studentnumber'],
+      include: {
+        attributes: [],
+        model: StudyrightElement,
+        required: true,
+        where: {
+          code: {
+            [Op.in]: codes,
+          },
+        },
+      },
+      raw: true,
+    })
+  ).map(({ studentnumber }) => studentnumber)

 module.exports = {
  withId,
  bySearchTerm,
  createStudent,
  updateStudent,
-  bySearchTermAndElements,
+  bySearchTermAndStudentNumbers,
  filterStudentnumbersByAccessrights,
  findByCourseAndSemesters,
  findByTag,
  splitByEmptySpace,
+  getStudentnumbersByElementdetails,
 }
@@ -2,42 +2,42 @@ const { getImporterClient } = require('../util/importerClient')
 const importerClient = getImporterClient()
 const { StudyGuidanceGroupTag } = require('../models/models_kone')
 const logger = require('../util/logger')
+const _ = require('lodash')

-const getAllStudentsUserHasInGroups = async sisPersonId => {
+const getGroupsFromImporter = async sisPersonId => {
  if (!importerClient) {
-    return new Set()
+    return []
  }
+  const answerTimeout = new Promise(resolve => setTimeout(resolve, 2000))
  try {
-    const { data } = await importerClient.get(`/person-groups/person/${sisPersonId}`)
-    return new Set(
-      Object.values(data)
-        .map(group => group.members)
-        .flat()
-        .map(member => member.personStudentNumber)
-    )
+    const result = await Promise.race([importerClient.get(`/person-groups/person/${sisPersonId}`), answerTimeout])
+    if (!result) return []
+    return Object.values(result.data)
  } catch (error) {
    logger.error("Couldn't fetch users studyguidance groups")
-    return new Set()
+    return []
  }
 }

 const getAllGroupsAndStudents = async sisPersonId => {
-  if (!importerClient) {
-    return []
-  }
-  try {
-    const { data } = await importerClient.get(`/person-groups/person/${sisPersonId}`)
-    const tagsByGroupId = (await StudyGuidanceGroupTag.findAll()).reduce((acc, curr) => {
-      const { studyProgramme, year } = curr
-      return { ...acc, [curr.studyGuidanceGroupId]: { studyProgramme, year } }
-    }, {})
-    return Object.values(data).map(group => ({ ...group, tags: tagsByGroupId[group.id] }))
-  } catch (error) {
-    logger.error("Couldn't fetch users studyguidance groups")
-    return []
-  }
+  const tagsByGroupId = (await StudyGuidanceGroupTag.findAll()).reduce((acc, curr) => {
+    const { studyProgramme, year, studyGuidanceGroupId } = curr
+    return { ...acc, [studyGuidanceGroupId]: { studyProgramme, year } }
+  }, {})
+  return (await getGroupsFromImporter(sisPersonId)).map(group => ({
+    ...group,
+    tags: tagsByGroupId[group.id],
+  }))
 }

+const getAllStudentsUserHasInGroups = async sisPersonId =>
+  _.uniqBy(
+    (await getGroupsFromImporter(sisPersonId))
+      .map(group => group.members)
+      .flat()
+      .map(member => member.personStudentNumber)
+  )
+
 const changeGroupTags = async ({ groupId, tags }) => {
  const { studyProgramme, year } = tags
  const tagToUpdate = studyProgramme ? { studyProgramme } : { year }
@@ -56,8 +56,11 @@ const changeGroupTags = async ({ groupId, tags }) => {
  return result
 }

+const checkStudyGuidanceGroupsAccess = async hyPersonSisuId => (await getGroupsFromImporter(hyPersonSisuId)).length > 0
+
 module.exports = {
  getAllStudentsUserHasInGroups,
  getAllGroupsAndStudents,
  changeGroupTags,
+  checkStudyGuidanceGroupsAccess,
 }
@@ -0,0 +1,66 @@
+const {
+  getBasicStatsForStudytrack,
+  getCreditStatsForStudytrack,
+  getGraduationStatsForStudytrack,
+} = require('../services/studyprogrammeStats')
+const { getStudytrackStatsForStudyprogramme } = require('../services/studytrackStats')
+const {
+  setBasicStats,
+  setCreditStats,
+  setGraduationStats,
+  setStudytrackStats,
+} = require('../services/analyticsService')
+
+const updateBasicView = async code => {
+  const academic = 'ACADEMIC_YEAR'
+  const calendar = 'CALENDAR_YEAR'
+
+  const basicStatsAcademic = await getBasicStatsForStudytrack({
+    studyprogramme: code,
+    yearType: academic,
+  })
+  await setBasicStats(basicStatsAcademic, academic)
+
+  const basicStatsCalendar = await getBasicStatsForStudytrack({
+    studyprogramme: code,
+    yearType: calendar,
+  })
+  await setBasicStats(basicStatsCalendar, calendar)
+
+  const creditStatsAcademic = await getCreditStatsForStudytrack({
+    studyprogramme: code,
+    yearType: academic,
+  })
+  await setCreditStats(creditStatsAcademic, academic)
+
+  const creditStatsCalendar = await getCreditStatsForStudytrack({
+    studyprogramme: code,
+    yearType: calendar,
+  })
+  await setCreditStats(creditStatsCalendar, calendar)
+
+  const graduationStatsAcademic = await getGraduationStatsForStudytrack({
+    studyprogramme: code,
+    yearType: academic,
+  })
+  await setGraduationStats(graduationStatsAcademic, academic)
+
+  const graduationStatsCalendar = await getGraduationStatsForStudytrack({
+    studyprogramme: code,
+    yearType: calendar,
+  })
+  await setGraduationStats(graduationStatsCalendar, calendar)
+
+  return 'OK'
+}
+
+const updateStudytrackView = async code => {
+  const result = await getStudytrackStatsForStudyprogramme({ studyprogramme: code })
+  await setStudytrackStats(result)
+  return 'OK'
+}
+
+module.exports = {
+  updateBasicView,
+  updateStudytrackView,
+}
@@ -1,41 +0,0 @@
-const Sequelize = require('sequelize')
-const { ElementDetail, StudyrightElement } = require('../models')
-const Op = Sequelize.Op
-
-const hasStudent = async (code, studentnumber) =>
-  StudyrightElement.findOne({
-    where: {
-      studentnumber,
-      code,
-    },
-  })
-
-const parseUnitFromElement = element => ({
-  id: element.code,
-  name: element.name.fi,
-  enabled: true,
-  type: element.type,
-})
-
-const getUnitsFromElementDetails = async () => {
-  const elementdetails = await ElementDetail.findAll({
-    where: {
-      type: {
-        [Op.or]: ['20'],
-      },
-    },
-  })
-  return elementdetails.map(parseUnitFromElement)
-}
-
-const getUnitFromElementDetail = async id => {
-  const element = await ElementDetail.findByPk(id)
-  return parseUnitFromElement(element)
-}
-
-module.exports = {
-  hasStudent,
-  getUnitsFromElementDetails,
-  getUnitFromElementDetail,
-  parseUnitFromElement,
-}
@@ -1,173 +1,280 @@
-const UnitService = require('./units')
-const elementDetailService = require('./elementdetails')
-const { filterStudentnumbersByAccessrights } = require('./students')
-const { userDataCache } = require('./cache')
-const { getImporterClient } = require('../util/importerClient')
-const logger = require('../util/logger')
+const Sequelize = require('sequelize')
+const { Op } = Sequelize
+const LRU = require('lru-cache')
+const { getStudentnumbersByElementdetails } = require('./students')
 const { facultiesAndProgrammesForTrends } = require('../services/organisations')
 const { sendNotificationAboutNewUser } = require('../services/mailservice')
-const User = require('./users/users')
-const AccessGroup = require('./users/accessgroups')
+const { checkStudyGuidanceGroupsAccess, getAllStudentsUserHasInGroups } = require('../services/studyGuidanceGroups')
 const _ = require('lodash')
+const { User, UserElementDetails, AccessGroup, UserFaculties, sequelizeUser } = require('../models/models_user')

-// Private helpers
-const enrichProgrammesFromFaculties = faculties =>
-  facultiesAndProgrammesForTrends.filter(f => faculties.includes(f.faculty_code)).map(f => f.programme_code)
+const courseStatisticsGroup = 'grp-oodikone-basic-users'
+const hyOneGroup = 'hy-one'
+const toskaGroup = 'grp-toska'

-const enrichWithProgrammes = toEnrich => {
-  if (Array.isArray(toEnrich)) {
-    return toEnrich.map(obj => enrichWithProgrammes(obj))
-  }
-  if (toEnrich.rights) {
-    return {
-      ...toEnrich,
-      rights: _.union(toEnrich.rights, enrichProgrammesFromFaculties(toEnrich.faculties)),
-    }
-  }
+// Max 25 users can be stored in the cache
+const userDataCache = new LRU({
+  max: 25,
+  length: () => 1,
+  maxAge: 1000 * 60 * 60,
+})

-  if (toEnrich.elementdetails) {
-    return {
-      ...toEnrich,
-      elementdetails: _.union(
-        toEnrich.elementdetails,
-        enrichProgrammesFromFaculties(toEnrich.faculty.map(f => f.faculty_code))
-      ),
-    }
+// Stuff to work with old userservice based user-db (separate microservice, now deprecated)
+const userIncludes = [
+  {
+    separate: true,
+    model: UserElementDetails,
+    as: 'programme',
+  },
+  {
+    model: AccessGroup,
+    as: 'accessgroup',
+    attributes: ['id', 'group_code', 'group_info'],
+  },
+  {
+    separate: true,
+    model: UserFaculties,
+    as: 'faculty',
+  },
+]
+
+const accessGroupsByCodes = codes =>
+  AccessGroup.findAll({
+    where: {
+      group_code: {
+        [Op.in]: codes,
+      },
+    },
+  })
+
+const byUsername = async username =>
+  await User.findOne({
+    where: {
+      username,
+    },
+    include: userIncludes,
+  })
+
+const byId = async id =>
+  await User.findOne({
+    where: {
+      id,
+    },
+    include: userIncludes,
+  })
+
+// Crud things from old userservice
+
+// some fields that frontend needs
+const formatUserForAdminView = user => ({
+  ...user.get(),
+  elementdetails: user.programme.map(p => p.elementDetailCode),
+  is_enabled: true, // this is probably not needed: is here mainly because old userservice created users for every logged in user, even if they hadn't correct iamgroups
+})
+
+const addProgrammes = async (id, codes) => {
+  for (const code of codes) {
+    await UserElementDetails.upsert({ userId: id, elementDetailCode: code })
  }
-  return toEnrich
-}
-const checkStudyGuidanceGroupsAccess = async hyPersonSisuId => {
-  if (!hyPersonSisuId) {
-    logger.error('Not possible to get groups without personId header')
-    return false
-  }
-  const importerClient = getImporterClient()
-  if (!importerClient) return false
-  const answerTimeout = new Promise(resolve => setTimeout(resolve, 2000))
-  const result = await Promise.race([importerClient.get(`/person-groups/person/${hyPersonSisuId}`), answerTimeout])
-  if (!result) return false
-  const { data } = result
-  return data && Object.values(data).length > 0
+  const user = await byId(id)
+  userDataCache.del(user.username)
 }

-const byUsernameData = async username => {
-  // OK, enriched
-  const user = await User.byUsernameMinified(username)
-  const roles = user.accessgroup.map(({ group_code }) => group_code)
-  const rights = User.getUserProgrammes(user)
-  const faculties = user.faculty.map(({ faculty_code }) => faculty_code)
-  const data = {
-    email: user.email,
-    full_name: user.full_name,
-    roles,
-    rights,
-    faculties,
+const removeProgrammes = async (id, codes) => {
+  for (const code of codes) {
+    await UserElementDetails.destroy({
+      where: { userId: id, elementDetailCode: code },
+    })
  }
-  return enrichWithProgrammes(data)
+  const user = await byId(id)
+  userDataCache.del(user.username)
 }

-// Exported helpers
-const getUserDataFor = async username => {
-  // no need to enrich
-  let userData = userDataCache.get(username)
-  if (!userData) {
-    userData = await byUsernameData(username)
-    userDataCache.set(username, userData)
-  }
+const modifyRights = async (username, rights) => {
+  const rightsToAdd = Object.entries(rights)
+    .map(([code, val]) => {
+      if (val === true) {
+        return code
+      }
+    })
+    .filter(code => code)
+  const rightsToRemove = Object.entries(rights)
+    .map(([code, val]) => {
+      if (val === false) {
+        return code
+      }
+    })
+    .filter(code => code)

-  return {
-    ...userData,
-    faculties: new Set(userData.faculties),
-  }
+  const user = await byUsername(username)
+  const accessGroupsToAdd = await accessGroupsByCodes(rightsToAdd)
+  const accessGroupsToRemove = await accessGroupsByCodes(rightsToRemove)
+
+  await user.addAccessgroup(accessGroupsToAdd)
+  await user.removeAccessgroup(accessGroupsToRemove)
+  userDataCache.del(username)
 }

-const getUnitsFromElementDetails = async username => {
-  // No need to enrich
-  let userData = userDataCache.get(username)
-  if (!userData) {
-    userData = await byUsernameData(username)
-    userDataCache.set(username, userData)
-  }
-  const elementDetails = await elementDetailService.byCodes(userData.rights)
-  return elementDetails.map(element => UnitService.parseUnitFromElement(element))
-}
-
-const getStudentsUserCanAccess = async (studentnumbers, roles, userId) => {
-  // No need to enrich
-  let studentsUserCanAccess
-  if (roles?.includes('admin')) {
-    studentsUserCanAccess = new Set(studentnumbers)
-  } else {
-    const unitsUserCanAccess = await getUnitsFromElementDetails(userId)
-    const codes = unitsUserCanAccess.map(unit => unit.id)
-    studentsUserCanAccess = new Set(await filterStudentnumbersByAccessrights(studentnumbers, codes))
-  }
-  return studentsUserCanAccess
-}
-
-// FOR ROUTES 1-to-1 mapping
 const modifyAccess = async body => {
  const { username, accessgroups } = body
-  await User.modifyRights(username, accessgroups)
-  const user = await User.byUsername(username)
-  return enrichWithProgrammes(User.getUserData(user))
+  await modifyRights(username, accessgroups)
+  userDataCache.del(username)
+  return formatUserForAdminView(await byUsername(username))
 }

-const getAccessGroups = async () => enrichWithProgrammes(await AccessGroup.findAll())
-
-const enableElementDetails = async (uid, codes) => {
-  await User.addProgrammes(uid, codes)
-  const user = await User.byId(uid)
-  return enrichWithProgrammes(User.getUserData(user))
+const enableElementDetails = async (id, codes) => {
+  await addProgrammes(id, codes)
+  const user = await byId(id)
+  userDataCache.del(user.username)
+  return formatUserForAdminView(user)
 }

-const findAll = async () => enrichWithProgrammes((await User.findAll()).map(User.getUserData))
-
-const setFaculties = async (uid, faculties) => {
-  await User.setFaculties(uid, faculties)
-  const user = await User.byId(uid)
-  return enrichWithProgrammes(User.getUserData(user))
+const setFaculties = async (id, faculties) => {
+  await sequelizeUser.transaction(async transaction => {
+    await UserFaculties.destroy({ where: { userId: id }, transaction })
+    for (const faculty of faculties) {
+      await UserFaculties.create({ userId: id, faculty_code: faculty }, { transaction })
+    }
+  })
+  const user = await byId(id)
+  userDataCache.del(user.username)
+  return formatUserForAdminView(user)
 }

-const removeElementDetails = async (uid, codes) => {
-  await User.removeProgrammes(uid, codes)
-  const user = await User.byId(uid)
-  return enrichWithProgrammes(User.getUserData(user))
+const removeElementDetails = async (id, codes) => {
+  await removeProgrammes(id, codes)
+  const user = await byId(id)
+  userDataCache.del(user.userId)
+  return formatUserForAdminView(user)
 }

 const updateUser = async (username, fields) => {
-  userDataCache.del(username)
-  const uid = username
-  const user = await User.byUsername(uid)
+  const user = await byUsername(username)
  if (!user) {
-    throw new Error(`User ${uid} not found`)
+    throw new Error(`User ${username} not found`)
  }
-  await User.updateUser(user, fields)
-  const returnedUser = await User.byUsername(username)
-  return enrichWithProgrammes(User.getUserData(returnedUser))
+  await User.upsert({ id: user.id, ...fields })
+  user.update(fields)
+
+  userDataCache.del(username)
+
+  return formatUserForAdminView(await byUsername(username))
 }

-const getLoginDataWithoutToken = async (uid, full_name, hyGroups, email, hyPersonSisuId) => {
-  const hasStudyGuidanceGroupAccess = await checkStudyGuidanceGroupsAccess(hyPersonSisuId)
-  const result = await User.loginWithoutToken(
-    uid,
-    full_name,
-    hyGroups,
-    email,
-    hyPersonSisuId,
-    hasStudyGuidanceGroupAccess
+const getAccessGroups = async () => await AccessGroup.findAll()
+
+// newer logic
+const enrichProgrammesFromFaculties = faculties =>
+  facultiesAndProgrammesForTrends.filter(f => faculties.includes(f.faculty_code)).map(f => f.programme_code)
+
+const findAll = async () =>
+  (
+    await User.findAll({
+      include: userIncludes,
+    })
+  ).map(user => ({
+    ...user.get(),
+    is_enabled: true,
+    elementdetails: _.uniqBy([
+      ...user.programme.map(p => p.elementDetailCode),
+      ...enrichProgrammesFromFaculties(user.faculty.map(f => f.faculty_code)),
+    ]),
+  }))
+
+const formatUser = async userFromDb => {
+  const [accessGroups, programmes, faculties] = [
+    ['accessgroup', 'group_code'],
+    ['programme', 'elementDetailCode'],
+    ['faculty', 'faculty_code'],
+  ].map(([field, code]) => userFromDb[field].map(entry => entry[code]))
+
+  const rights = _.uniqBy([...programmes, ...enrichProgrammesFromFaculties(faculties)])
+
+  const {
+    dataValues: { id, username: userId, full_name: name, email, language, sisu_person_id: sisPersonId },
+  } = userFromDb
+
+  const studentsUserCanAccess = _.uniqBy(
+    (await Promise.all([getStudentnumbersByElementdetails(rights), getAllStudentsUserHasInGroups(sisPersonId)])).flat()
  )
-  return result
+
+  // attribute naming is a bit confusing, but it's used so widely in oodikone, that it's not probably worth changing
+  return {
+    id, // our internal id, not the one from sis, (e.g. 101)
+    userId, // acually username (e.g. mluukkai)
+    name,
+    language,
+    sisPersonId, //
+    email,
+    is_enabled: true, // this is probably not needed: is here mainly because old userservice created users for every logged in user, even if they hadn't correct iamgroups
+    rights,
+    roles: accessGroups,
+    studentsUserCanAccess, // studentnumbers used in various parts of backend. For admin this is usually empty, since no programmes / faculties are set.
+    isAdmin: accessGroups.includes('admin'),
+  }
 }

-const getMockedUser = async (uid, asUser) => await User.superlogin(uid, asUser)
+const getMockedUser = async ({ userToMock, mockedBy }) => {
+  if (userDataCache.has(userToMock)) {
+    return userDataCache.get(userToMock)
+  }
+  const toReturn = {
+    ...(await formatUser(await byUsername(userToMock))),
+    mockedBy,
+  }
+  userDataCache.set(userToMock, toReturn)
+  return toReturn
+}

 const getUser = async ({ username, name, email, iamGroups, sisId }) => {
-  const { payload: user, isNew } = await getLoginDataWithoutToken(username, name, iamGroups, email, sisId)
-  if (isNew) {
-    await sendNotificationAboutNewUser({ userId: username, userFullName: name })
+  if (userDataCache.has(username)) {
+    return userDataCache.get(username)
  }
-  return user
+
+  const isNewUser = !(await User.findOne({ where: { username } }))
+  const language = 'fi'
+
+  await User.upsert({
+    full_name: name,
+    username,
+    email,
+    language,
+    sisu_person_id: sisId,
+    last_login: new Date(),
+  })
+
+  const userFromDb = await byUsername(username)
+  const formattedUser = await formatUser(userFromDb)
+  const { roles: currentAccessGroups } = formattedUser
+
+  // Modify accessgroups based on current groups and iamGroups
+  let newAccessGroups = []
+  if (await checkStudyGuidanceGroupsAccess(sisId)) newAccessGroups.push('studyGuidanceGroups')
+  if (iamGroups.includes(courseStatisticsGroup)) newAccessGroups.push('courseStatistics')
+  if (iamGroups.includes(hyOneGroup) || currentAccessGroups.includes('teachers')) newAccessGroups.push('teachers')
+  if (iamGroups.includes(toskaGroup) || currentAccessGroups.includes('admin')) newAccessGroups.push('admin')
+  if (_.difference(newAccessGroups, currentAccessGroups).length > 0) {
+    userFromDb.setAccessgroup(
+      (
+        await AccessGroup.findAll({
+          where: {
+            group_code: {
+              [Op.in]: newAccessGroups,
+            },
+          },
+        })
+      ).map(({ id }) => id)
+    )
+  }
+
+  if (isNewUser) await sendNotificationAboutNewUser({ userId: username, userFullName: name })
+
+  const toReturn = {
+    ...formattedUser,
+    roles: newAccessGroups,
+  }
+  userDataCache.set(username, toReturn)
+  return toReturn
 }

 module.exports = {
@@ -177,11 +284,7 @@ module.exports = {
  findAll,
  modifyAccess,
  getAccessGroups,
-  getUnitsFromElementDetails,
  setFaculties,
-  getUserDataFor,
-  getStudentsUserCanAccess,
-  getLoginDataWithoutToken,
  getUser,
  getMockedUser,
 }
@@ -1,17 +0,0 @@
-const { AccessGroup } = require('../../models/models_user')
-const { Op } = require('sequelize')
-
-const byId = id => AccessGroup.findByPk(id)
-
-const findAll = () => AccessGroup.findAll()
-
-const byCodes = codes =>
-  AccessGroup.findAll({
-    where: {
-      group_code: {
-        [Op.in]: codes,
-      },
-    },
-  })
-
-module.exports = { byId, byCodes, findAll }
@@ -1,24 +0,0 @@
-const { HyGroup } = require('../../models/models_user')
-const { Op } = require('sequelize')
-
-const byId = id => HyGroup.findByPk(id)
-
-const findAll = () => HyGroup.findAll()
-
-const byCodes = codes =>
-  HyGroup.findAll({
-    where: {
-      code: {
-        [Op.in]: codes,
-      },
-    },
-  })
-
-const create = code => HyGroup.create({ code })
-
-const addGroups = async (uid, codes) => {
-  const user = await byId(uid)
-  const groups = await byCodes(codes)
-  await user.addHygroups(groups)
-}
-module.exports = { byId, byCodes, findAll, create, addGroups }
@@ -1,339 +0,0 @@
-const Sequelize = require('sequelize')
-const _ = require('lodash')
-const moment = require('moment')
-const {
-  User,
-  UserElementDetails,
-  AccessGroup,
-  HyGroup,
-  UserFaculties,
-  sequelizeUser,
-} = require('../../models/models_user')
-const AccessService = require('./accessgroups')
-const HyGroupService = require('./hygroups')
-const Op = Sequelize.Op
-
-// move these to conf
-const courseStatisticsGroup = 'grp-oodikone-basic-users'
-const hyOneGroup = 'hy-one'
-const requiredGroup =
-  process.env.NODE_ENV === 'staging'
-    ? ['grp-oodikone-staging-users', 'grp-oodikone-basic-staging-users']
-    : ['grp-oodikone-users', 'grp-oodikone-basic-users']
-
-// help for refactoring away from token
-const generateTokenData = async (uid, mockedBy = null) => {
-  const user = await byUsername(uid)
-  const userData = getUserData(user)
-  const programmes = userData.elementdetails
-  const payload = {
-    id: user.id,
-    userId: uid, // username
-    name: user.full_name,
-    enabled: userData.is_enabled,
-    language: user.language,
-    mockedBy,
-    rights: programmes,
-    roles: user.accessgroup,
-    createdAt: moment().toISOString(),
-    sisPersonId: user.sisu_person_id,
-  }
-  return payload
-}
-
-const createMissingGroups = async (group, service) => {
-  const savedGroups = await service.findAll()
-  group.forEach(async code => {
-    if (!savedGroups.map(sg => sg.code).includes(code)) {
-      await service.create(code)
-    }
-  })
-}
-
-const updateGroups = async (user, hyGroups) => {
-  let hyGroupsToBeUpdated = (await user.getHy_group()).map(hg => hg.code)
-  let hyGroupsToAdd = []
-  let hyGroupsToDelete = []
-
-  hyGroups.forEach(async hyGroup => {
-    if (!hyGroupsToBeUpdated.includes(hyGroup)) {
-      hyGroupsToAdd = hyGroupsToAdd.concat(hyGroup)
-    }
-  })
-  hyGroupsToBeUpdated.forEach(async hyGroup => {
-    if (!hyGroups.includes(hyGroup)) {
-      hyGroupsToDelete = hyGroupsToDelete.concat(hyGroup)
-    }
-  })
-
-  await user.addHy_group(await HyGroupService.byCodes(hyGroupsToAdd))
-  await user.removeHy_group(await HyGroupService.byCodes(hyGroupsToDelete))
-}
-
-const loginBase = async (uid, full_name, hyGroups, mail, hyPersonSisuId, hasStudyGuidanceGroupAccess) => {
-  let user = await byUsername(uid)
-  let isNew = false
-  const lastLogin = new Date()
-  await createMissingGroups(hyGroups, HyGroupService)
-
-  if (!user) {
-    user = await createUser(uid, full_name, mail, hyPersonSisuId, lastLogin)
-
-    const userHyGroups = await HyGroupService.byCodes(hyGroups)
-    await user.addHy_group(userHyGroups)
-
-    isNew = true
-  } else {
-    user = await updateUser(user, { full_name, email: mail, sisu_person_id: hyPersonSisuId, last_login: lastLogin })
-    await updateGroups(user, hyGroups)
-  }
-
-  await determineAccessToCourseStats(user, hyGroups)
-  await determineAccessToTeachersForOne(user, hyGroups)
-  await determineAccessToStudyGuidanceGroups(user, hasStudyGuidanceGroupAccess)
-
-  return isNew
-}
-
-const loginWithoutToken = async (
-  uid,
-  full_name,
-  hyGroups,
-  affiliations,
-  mail,
-  hyPersonSisuId,
-  hasStudyGuidanceGroupAccess
-) => {
-  const isNew = await loginBase(
-    uid,
-    full_name,
-    hyGroups,
-    affiliations,
-    mail,
-    hyPersonSisuId,
-    hasStudyGuidanceGroupAccess
-  )
-  const payload = await generateTokenData(uid)
-  return { payload, isNew }
-}
-
-const superlogin = async (uid, asUser) => {
-  const user = await byUsername(uid)
-  if (user && user.accessgroup.map(r => r.group_code).includes('admin')) {
-    const token = await generateTokenData(asUser, uid)
-    return token
-  }
-  return undefined
-}
-
-const userIncludes = [
-  {
-    separate: true,
-    model: UserElementDetails,
-    as: 'programme',
-  },
-  {
-    model: AccessGroup,
-    as: 'accessgroup',
-    attributes: ['id', 'group_code', 'group_info'],
-  },
-  {
-    required: false,
-    model: HyGroup,
-    as: 'hy_group',
-    where: { code: requiredGroup },
-  },
-  {
-    separate: true,
-    model: UserFaculties,
-    as: 'faculty',
-  },
-]
-
-const getUserData = user => {
-  if (user == null) return null
-  const newuser = user.get()
-  newuser.elementdetails = getUserProgrammes(newuser)
-  const hyGroups = newuser.hy_group.map(e => e.code)
-  newuser.is_enabled = requiredGroup === null || _.intersection(hyGroups, requiredGroup).length > 0
-  newuser.hy_group = null
-  return newuser
-}
-
-const byUsernameMinified = async username => {
-  const userMinified = await User.findOne({
-    where: {
-      username,
-    },
-    include: [
-      {
-        separate: true,
-        model: UserElementDetails,
-        as: 'programme',
-        attributes: ['elementDetailCode'],
-      },
-      {
-        model: AccessGroup,
-        as: 'accessgroup',
-        attributes: ['group_code'],
-      },
-      {
-        separate: true,
-        model: UserFaculties,
-        as: 'faculty',
-        attributes: ['faculty_code'],
-      },
-    ],
-  })
-  return userMinified
-}
-
-const byUsername = async username => {
-  const user = await User.findOne({
-    where: {
-      username: {
-        [Op.eq]: username,
-      },
-    },
-    include: userIncludes,
-  })
-  return user
-}
-
-const createUser = async (username, fullname, email, hyPersonSisuId, lastLogin) => {
-  return User.create({
-    username: username,
-    full_name: fullname,
-    email,
-    sisu_person_id: hyPersonSisuId,
-    last_login: lastLogin,
-  })
-}
-
-const updateUser = async (userObject, values) => {
-  await User.upsert({ id: userObject.id, ...values })
-  return userObject.update(values)
-}
-
-const byId = async id => {
-  const user = await User.findOne({
-    where: {
-      id: {
-        [Op.eq]: id,
-      },
-    },
-    include: userIncludes,
-  })
-  return user
-}
-
-const getUserProgrammes = user => user.programme.map(p => p.elementDetailCode)
-
-const getUserAccessGroups = async username => {
-  const user = await byUsername(username)
-  return await user.getAccessgroup()
-}
-
-const findAll = async () => {
-  const users = await User.findAll({
-    include: userIncludes,
-  })
-  return users
-}
-
-const addProgrammes = async (uid, codes) => {
-  for (const code of codes) {
-    await UserElementDetails.upsert({ userId: uid, elementDetailCode: code })
-  }
-}
-
-const removeProgrammes = async (uid, codes) => {
-  for (const code of codes) {
-    await UserElementDetails.destroy({
-      where: { userId: uid, elementDetailCode: code },
-    })
-  }
-}
-
-const setFaculties = async (uid, faculties) => {
-  await sequelizeUser.transaction(async transaction => {
-    await UserFaculties.destroy({ where: { userId: uid }, transaction })
-    for (const faculty of faculties) {
-      await UserFaculties.create({ userId: uid, faculty_code: faculty }, { transaction })
-    }
-  })
-}
-
-const modifyRights = async (username, rights) => {
-  const rightsToAdd = Object.entries(rights)
-    .map(([code, val]) => {
-      if (val === true) {
-        return code
-      }
-    })
-    .filter(code => code)
-  const rightsToRemove = Object.entries(rights)
-    .map(([code, val]) => {
-      if (val === false) {
-        return code
-      }
-    })
-    .filter(code => code)
-
-  const user = await byUsername(username)
-  const accessGroupsToAdd = await AccessService.byCodes(rightsToAdd)
-  const accessGroupsToRemove = await AccessService.byCodes(rightsToRemove)
-
-  await user.addAccessgroup(accessGroupsToAdd)
-  await user.removeAccessgroup(accessGroupsToRemove)
-}
-const getRoles = async user => {
-  const foundUser = byUsername(user)
-  return foundUser.accessgroup
-}
-
-const determineAccessToCourseStats = async (user, hyGroups) => {
-  const accessGroups = (user && user.accessgroup) || []
-  const alreadyAccess = accessGroups.some(({ group_code }) => group_code === 'courseStatistics')
-  if (hyGroups.includes(courseStatisticsGroup) && !alreadyAccess) {
-    await modifyRights(user.username, { courseStatistics: true })
-  } else if (!hyGroups.includes(courseStatisticsGroup) && alreadyAccess) {
-    await modifyRights(user.username, { courseStatistics: false })
-  }
-}
-
-const determineAccessToTeachersForOne = async (user, hyGroups) => {
-  const accessGroups = (user && user.accessgroup) || []
-  const alreadyAccess = accessGroups.some(({ group_code }) => group_code === 'teachers')
-  if (hyGroups.includes(hyOneGroup) && !alreadyAccess) {
-    await modifyRights(user.username, { teachers: true })
-  }
-}
-
-const determineAccessToStudyGuidanceGroups = async (user, hasStudyGuidanceGroupAccess) => {
-  const accessGroups = (user && user.accessgroup) || []
-  const alreadyAccess = accessGroups.some(({ group_code }) => group_code === 'studyGuidanceGroups')
-  if (hasStudyGuidanceGroupAccess && !alreadyAccess) {
-    await modifyRights(user.username, { studyGuidanceGroups: true })
-  } else if (!hasStudyGuidanceGroupAccess && alreadyAccess) {
-    await modifyRights(user.username, { studyGuidanceGroups: false })
-  }
-}
-
-module.exports = {
-  byUsername,
-  byUsernameMinified,
-  updateUser,
-  findAll,
-  byId,
-  getUserProgrammes,
-  addProgrammes,
-  removeProgrammes,
-  superlogin,
-  modifyRights,
-  getUserAccessGroups,
-  getRoles,
-  getUserData,
-  setFaculties,
-  loginWithoutToken,
-}
@@ -11,6 +11,34 @@ const mapToProviders = elementDetails =>
    return r
  })

+// sort substitutions so that main code is first
+const newLetterBasedCode = /^[A-Za-z]/ // new letter based codes come first
+const oldNumericCode = /^\d/ // old numeric codes come second
+const openUniCode = /^AY?(.+?)(?:en|fi|sv)?$/ // open university codes come last
+const openUniCodeA = /A\d/ // open university with just A come last
+const digi = /DIGI-A?(.+?)(?:en|fi|sv)?$/ // digi-a goes on top courses goes third
+const bscsCode = /BSCS??/
+
+const codeRegexes = [openUniCodeA, openUniCode, bscsCode, oldNumericCode, newLetterBasedCode, digi]
+
+const getSortRank = code => {
+  for (let i = 0; i < codeRegexes.length; i++) {
+    if (codeRegexes[i].test(code)) {
+      return i
+    }
+  }
+  return 3 // if no hit, put before open uni courses
+}
+
+const sortMainCode = codeArray => {
+  if (!codeArray) return []
+  return codeArray.sort(function (x, y) {
+    return getSortRank(y) - getSortRank(x)
+  })
+}
+
 module.exports = {
  mapToProviders,
+  sortMainCode,
+  getSortRank,
 }
@@ -10,7 +10,7 @@ export const getHeaders = () => {
    uid: 'mluukkai',
    displayName: 'Matti Luukkainen',
    'shib-session-id': 'mock-session',
-    hyGroupCn: 'grp-oodikone-users;grp-oodikone-basic-users',
+    hyGroupCn: 'grp-oodikone-users;grp-oodikone-basic-users;grp-toska',
    mail: 'grp-toska+mockmluukkai@helsinki.fi',
    hyPersonSisuId: 'hy-hlo-1441871',
    shib_logout_url: 'https://helsinki.fi/shibboleth-sp/Logout',
@@ -71,6 +71,10 @@ export default {
  - **Study programme:** haluttu koulutusohjelma.
  - **Select tag (Optional):** itsetehty tai valmis tägi.
  - **Study track (Optional):** opintosuunta.`,
+  TagAndTrackMovedIntoFilters: `
+  Tägien ja opintosuuntien valinnat ovat siirtyneet tästä omiksi filtereikseen.
+  Voit tehdä valinnat siirryttyäsi tarkastelemaan ohjelman tilastoja.
+  `,
  Students: `**Students**

  Taulukko näyttää oletusarvoisesti vain opiskelijanumerot, mutta harmaata liukukytkintä klikkaamalla saa näkyviin opiskelijoiden nimet.
@@ -133,4 +137,5 @@ export default {
  ProgrammeDistributionCoursePopulation: `Näyttää sen koulutusohjelman, jossa opiskelija oli suorittaessaan kurssin`,
  GradeDistributionCoursePopulation: `Näyttää korkeimman arvosanan, jonka opiskelija on saanut valitulla aikavälillä.`,
  CreditDistributionCoursePopulation: `Näyttää tiedekunta-/ohjelmankohtaisen opintopistejakauman.`,
+  LanguageDistributionCoursePopulation: `Näyttää kielijakauman kurssin suorituskielistä.`,
 }
@@ -18,8 +18,8 @@ export const images = {
  irtomikko,
 }

-export const checkUserAccess = (requiredRoles, userRoles) => {
-  return intersection(userRoles, requiredRoles).length > 0
+export const checkUserAccess = (requiredRoles, roles) => {
+  return intersection(requiredRoles, roles).length > 0
 }

 export const containsOnlyNumbers = str => str.match('^\\d+$')
@@ -160,7 +160,7 @@ export const getNewestProgramme = (studyrights, studentNumber, studentToTargetCo
        elementDetails[srE.code] &&
        elementDetails[srE.code].type === 20 &&
        (studentToTargetCourseDateMap && studentNumber
-          ? moment(studentToTargetCourseDateMap[studentNumber]).isBetween(moment(srE.startdate), moment(srE.enddate))
+          ? moment(studentToTargetCourseDateMap[studentNumber]).isBetween(srE.startdate, srE.enddate, 'day', '[]')
          : true)
    )
    if (studyrightElements.length > 0) {
@@ -179,6 +179,15 @@ export const getNewestProgramme = (studyrights, studentNumber, studentToTargetCo
  if (programme) {
    return programme
  }
+
+  if (studentToTargetCourseDateMap) {
+    return {
+      name: { en: 'No programme at the time of attainment', fi: 'Ei ohjelmaa suorituksen hetkellä' },
+      startdate: '',
+      code: '00000',
+    }
+  }
+
  return { name: { en: 'No programme', fi: 'Ei ohjelmaa' }, startdate: '', code: '00000' }
 }

@@ -1,6 +1,8 @@
 import React, { useEffect, useState, useMemo, useRef } from 'react'
 import { withRouter } from 'react-router-dom'
 import { connect } from 'react-redux'
+import moment from 'moment'
+import _ from 'lodash'
 import { Segment, Header, Accordion } from 'semantic-ui-react'
 import scrollToComponent from 'react-scroll-to-component'
 import InfoBox from 'components/Info/InfoBox'
@@ -28,6 +30,7 @@ import {
  creditsEarnedFilter,
  startYearAtUniFilter,
  programmeFilter,
+  studyTrackFilter,
 } from '../FilterView/filters'
 import FilterView from '../FilterView'
 import useLanguage from '../LanguagePicker/useLanguage'
@@ -39,7 +42,6 @@ const CoursePopulation = ({
  getElementDetails,
  studentData,
  pending,
-  elementDetails,
  history,
  courseData,
  getSemestersDispatch,
@@ -190,7 +192,7 @@ const CoursePopulation = ({
      content: {
        content: (
          <div ref={languageDistRef}>
-            <InfoBox content={infotooltips.PopulationStatistics.GradeDistributionCoursePopulation} />
+            <InfoBox content={infotooltips.PopulationStatistics.LanguageDistributionCoursePopulation} />
            <CoursePopulationLanguageDist from={dateFrom} to={dateTo} samples={studentData.students} codes={codes} />
          </div>
        ),
@@ -291,6 +293,17 @@ const CoursePopulation = ({

  const courses = JSON.parse(queryParamsFromUrl(history.location).coursecodes)

+  const studyRightPredicate = (student, sre) => {
+    const date = _.chain(student)
+      .get('courses')
+      .filter(c => courses.includes(c.course_code))
+      .map('date')
+      .max()
+      .value()
+
+    return moment(date).isBetween(sre.startdate, sre.enddate, 'day', '[]')
+  }
+
  return (
    <FilterView
      name="CoursePopulation"
@@ -300,13 +313,28 @@ const CoursePopulation = ({
        courseFilter({ courses: courseStatistics?.coursestatistics }),
        creditsEarnedFilter,
        startYearAtUniFilter,
-        programmeFilter({ courses, elementDetails }),
+        programmeFilter({
+          additionalModes: [
+            {
+              key: 'attainment',
+              label: 'Attainment',
+              predicate: studyRightPredicate,
+              description: 'Student had an active study right at the time of course attainment.',
+            },
+          ],
+        }),
        gradeFilter({
          courseCodes: courses,
          from: dateFrom,
          to: dateTo,
        }),
+        studyTrackFilter({
+          activeAt: dateFrom,
+        }),
      ]}
+      initialOptions={{
+        [programmeFilter.key]: { mode: 'attainment', selectedProgrammes: [] },
+      }}
      students={studentData.students ?? []}
    >
      {filtered => (
@@ -1,8 +1,7 @@
 import { orderBy } from 'lodash'

 // handle special case where the course's actual code starts with "A" so it is mistakenly taken as Open Uni course
-// const isAvoin = code => !!code.match(/^[A][0-9]|^AY/)
-const isAvoinTwo = course => course.organizations.some(c => c === 'hy-org-48645785')
+const isAvoin = course => !!course.code.match(/^[A][0-9]|^AY/)

 const sortAlternatives = alternatives => {
  const result = orderBy(
@@ -31,7 +30,7 @@ const filterCourseSearchResults = (courses, unifyOpenUniCourses) => {
  const mergedCourses = {}
  const sortedCourses = sortAlternatives(courses)
  sortedCourses.forEach(course => {
-    const groupId = isAvoinTwo(course) && !unifyOpenUniCourses ? course.code : course.subsId
+    const groupId = isAvoin(course) && !unifyOpenUniCourses ? course.code : course.subsId

    if (!(course.max_attainment_date && course.min_attainment_date)) {
      return
@@ -32,8 +32,6 @@ const getActiveYears = course => {
  return `${startYearText} — ${endYearText}`
 }

-const userHasAccessToAllCourseStats = (roles, rights) => {
-  return (roles && ['admin'].some(role => roles.includes(role))) || rights.length > 0
-}
+const userHasAccessToAllCourseStats = (roles, rights) => roles.includes('admin') || rights.length > 0

 export { getActiveYears, getYearText, userHasAccessToAllCourseStats }
@@ -28,7 +28,7 @@ const MENU = {
 const CourseStatistics = () => {
  const history = useHistory()
  const dispatch = useDispatch()
-  const { rights, userRoles } = useGetAuthorizedUserQuery()
+  const { rights, roles } = useGetAuthorizedUserQuery()
  const courseStats = useSelector(({ courseStats }) => courseStats)
  const { pending: loading } = courseStats
  const courses = Object.keys(courseStats.data)
@@ -57,7 +57,7 @@ const CourseStatistics = () => {
    setSelected(coursecode)
  }

-  const userHasAccessToAllStats = userHasAccessToAllCourseStats(userRoles, rights)
+  const userHasAccessToAllStats = userHasAccessToAllCourseStats(roles, rights)

  const getPanes = () => {
    let panes = [
@@ -109,7 +109,8 @@ const CourseStatistics = () => {
      setActiveIndex(activeIndex)
    }
  }
-  if (!checkUserAccess(['courseStatistics', 'admin'], userRoles) && rights.length < 1)
+
+  if (!checkUserAccess(['courseStatistics', 'admin'], roles) && rights.length < 1)
    return (
      <div className="segmentContainer">
        <Message
@@ -27,6 +27,7 @@ import {
  tagsFilter,
  creditDateFilter,
  enrollmentStatusFilter,
+  studyTrackFilter,
 } from '../FilterView/filters'
 import useLanguage from '../LanguagePicker/useLanguage'
 import CustomPopulationSearch from './CustomPopulationSearch'
@@ -37,7 +38,6 @@ const selectCustomPopulationData = createSelector(
  state => state.populations.data,
  (semesters, populations) => ({
    allSemesters: semesters?.semesters ?? [],
-    elementDetails: populations?.elementDetails?.data ?? [],
    custompop: populations?.students ?? [],
  })
 )
@@ -46,7 +46,7 @@ const CustomPopulation = () => {
  const { language } = useLanguage()
  const dispatch = useDispatch()

-  const { allSemesters, elementDetails, custompop } = useSelector(selectCustomPopulationData)
+  const { allSemesters, custompop } = useSelector(selectCustomPopulationData)

  const { data: courseStats } = useGetStudentListCourseStatisticsQuery({
    studentNumbers: custompop.map(s => s.studentNumber),
@@ -68,17 +68,15 @@ const CustomPopulation = () => {
      transferredToProgrammeFilter,
      startYearAtUniFilter,
      tagsFilter,
-      programmeFilter({
-        courses: courseStats?.coursestatistics ? courseStats?.coursestatistics.map(c => c.course.code) : [],
-        elementDetails,
-      }),
+      programmeFilter,
      creditDateFilter,
      enrollmentStatusFilter({
        allSemesters: allSemesters ?? [],
        language,
      }),
+      studyTrackFilter,
    ],
-    [courseStats, elementDetails, allSemesters, language]
+    [courseStats, allSemesters, language]
  )

  return (
@@ -20,31 +20,33 @@ const FilterTray = () => {

  const haveOptionsBeenChanged = filters.some(({ key }) => areOptionsDirty(key))

-  const filterSet = filters.map(filter => {
-    const { key, title, isActive, render, info } = filter
-    const ctx = getContextByKey(key)
+  const filterSet = filters
+    .sort((a, b) => (a.title ?? a.key).localeCompare(b.title ?? b.key))
+    .map(filter => {
+      const { key, title, isActive, render, info } = filter
+      const ctx = getContextByKey(key)

-    const props = {
-      options: ctx.options,
-      onOptionsChange: options => setFilterOptions(key, options),
-      withoutSelf: () => withoutFilter(key),
-    }
+      const props = {
+        options: ctx.options,
+        onOptionsChange: options => setFilterOptions(key, options),
+        withoutSelf: () => withoutFilter(key),
+      }

-    return (
-      <div key={key}>
-        <FilterCard
-          title={title ?? key}
-          active={isActive(filterOptions[key])}
-          onClear={() => resetFilter(key)}
-          info={info}
-          filter={filter}
-          options={ctx.options}
-        >
-          {render(props, ctx)}
-        </FilterCard>
-      </div>
-    )
-  })
+      return (
+        <div key={key}>
+          <FilterCard
+            title={title ?? key}
+            active={isActive(filterOptions[key])}
+            onClear={() => resetFilter(key)}
+            info={info}
+            filter={filter}
+            options={ctx.options}
+          >
+            {render(props, ctx)}
+          </FilterCard>
+        </div>
+      )
+    })

  return (
    <>
@@ -1,15 +1,39 @@
-import React, { useState } from 'react'
+import React, { useState, useRef } from 'react'
 import { Icon, Header } from 'semantic-ui-react'
+import _ from 'lodash'
 import WithHelpTooltip from '../../../Info/InfoWithHelpTooltip'

 import './FilterCard.css'

+const useChange = value => {
+  const prevValue = useRef()
+
+  if (prevValue.current === undefined) {
+    prevValue.current = value
+    return true
+  }
+
+  const change = !_.isEqual(value, prevValue.current)
+
+  prevValue.current = value
+
+  return change
+}
+
 const FilterCard = ({ filter, options, children, onClear }) => {
  const title = filter.title ?? filter.key
  const active = filter.isActive(options)
  const { info, key } = filter

-  const [open, setOpen] = useState(false)
+  const hasChanged = useChange(options)
+
+  const [manuallyOpened, setManuallyOpened] = useState(null)
+
+  if (hasChanged && manuallyOpened === false) {
+    setManuallyOpened(null)
+  }
+
+  const open = manuallyOpened !== null ? manuallyOpened : active

  let header = (
    <div
@@ -20,7 +44,7 @@ const FilterCard = ({ filter, options, children, onClear }) => {
        flexGrow: 1,
      }}
      className="filter-card-header"
-      onClick={() => setOpen(!open)}
+      onClick={() => setManuallyOpened(!open)}
      data-cy={`${key}-header`}
    >
      <Icon name={open ? 'caret down' : 'caret right'} style={{ color: 'black', flexShrink: 0 }} />
@@ -63,9 +87,9 @@ const FilterCard = ({ filter, options, children, onClear }) => {
  }

  return (
-    <div style={{ margin: '1rem 0' }} data-cy={`${key}-filter-card`}>
+    <div style={{ margin: '1rem 0' }} data-cy={`${key}-filter-card`} data-open={open}>
      <div style={{ marginBottom: '1rem' }}>{header}</div>
-      {open && children}
+      {open && <div onClick={() => setManuallyOpened(true)}>{children}</div>}
    </div>
  )
 }
@@ -84,6 +84,7 @@ export default createFilter({
      )
    ), */
      fp.map(([sn, courses]) => [sn, getHighestGradeOfCourseBetweenRange(courses, args.from, args.to)]),
+      fp.filter(([, grade]) => grade !== undefined),
      fp.groupBy(([, { grade }]) => grade),
      fp.mapValues(fp.map(([sn]) => sn)),
      grades => ({ grades })
@@ -11,6 +11,7 @@ import admissionType from './admissionType'
 import tags from './tags'
 import date from './date'
 import enrollmentStatus from './enrollmentStatus'
+import studyTrack from './studyTrack'

 export const genderFilter = gender
 export const ageFilter = age
@@ -25,3 +26,4 @@ export const admissionTypeFilter = admissionType
 export const tagsFilter = tags
 export const creditDateFilter = date
 export const enrollmentStatusFilter = enrollmentStatus
+export const studyTrackFilter = studyTrack
@@ -1,76 +1,186 @@
 import React, { useMemo } from 'react'
 import { Dropdown } from 'semantic-ui-react'
 import fp from 'lodash/fp'
-import { getTextIn, getStudentToTargetCourseDateMap, getNewestProgramme } from '../../../common'
+import _ from 'lodash'
+import moment from 'moment'
+import { getTextIn } from '../../../common'
 import useLanguage from '../../LanguagePicker/useLanguage'
 import createFilter from './createFilter'

-const ProgrammeFilterCard = ({ options, onOptionsChange, programmes }) => {
+const ProgrammeFilterCard = ({
+  options,
+  onOptionsChange,
+  programmes,
+  withoutSelf,
+  studyRightPredicate,
+  studentToProgrammeMap,
+  additionalModes,
+}) => {
  const { language } = useLanguage()
  const { selectedProgrammes } = options
  const name = 'programmeFilterCard'

+  const visibleProgrammes = fp.flow(
+    fp.flatMap(student =>
+      _.get(studentToProgrammeMap, student.studentNumber, []).map(programme => ({ student, programme }))
+    ),
+    fp.groupBy('student.studentNumber'),
+    fp.pickBy(fp.some(({ programme }) => options.selectedProgrammes.every(pcode => programme.code === pcode))),
+    fp.values,
+    fp.flatten,
+    fp.filter(({ student, programme }) => studyRightPredicate(student, programme)),
+    fp.map('programme'),
+    fp.reduce((acc, details) => {
+      if (!acc[details.code]) {
+        acc[details.code] = { ...details, studentCount: 0 }
+      }
+
+      acc[details.code].studentCount += 1
+
+      return acc
+    }, {}),
+    fp.values
+  )(withoutSelf())
+
  const dropdownOptions = useMemo(
    () =>
-      programmes
+      _.chain(visibleProgrammes)
+        .concat(selectedProgrammes.map(code => programmes.find(p => p.code === code)))
        .map(program => ({
          key: `programme-filter-value-${program.code}`,
          text: getTextIn(program.name, language),
          value: program.code,
+          content: (
+            <>
+              {getTextIn(program.name, language)}{' '}
+              <span style={{ color: 'rgb(136, 136, 136)', whiteSpace: 'nowrap' }}>
+                ({program.studentCount} students)
+              </span>
+            </>
+          ),
        }))
-        .sort((a, b) => a.text.localeCompare(b.text)),
+        .uniqBy('value')
+        .sort((a, b) => a.text.localeCompare(b.text))
+        .value(),
    [programmes, language]
  )

  const handleChange = (_, { value }) => {
    onOptionsChange({
+      ...options,
      selectedProgrammes: value,
    })
  }

+  const setMode = mode => {
+    onOptionsChange({
+      ...options,
+      mode,
+    })
+  }
+
+  const builtInModes = [
+    {
+      key: 'any',
+      value: 'any',
+      label: 'Past or Present',
+      description: 'Student has had a study right at any point in time.',
+    },
+    {
+      key: 'active',
+      value: 'active',
+      label: 'Active Study Right',
+      description: 'Student has a currently active study right.',
+    },
+  ]
+
  return (
-    <Dropdown
-      options={dropdownOptions}
-      placeholder="Select Programme"
-      onChange={handleChange}
-      button
-      value={selectedProgrammes}
-      name={name}
-      closeOnChange
-      multiple
-    />
+    <>
+      <Dropdown
+        multiple
+        closeOnChange
+        fluid
+        name={name}
+        onChange={handleChange}
+        options={dropdownOptions}
+        placeholder="Select Programme"
+        search
+        selection
+        value={selectedProgrammes}
+        data-cy="Programme-filter-dropdown"
+      />
+      <div style={{ marginTop: '0.5em' }}>
+        Mode:{' '}
+        <Dropdown
+          data-cy="Programme-filter-mode-selector"
+          compact
+          inline
+          placeholder="Type"
+          value={options.mode ?? 'active'}
+          onChange={(_, { value }) => setMode(value)}
+          options={[...builtInModes, ...additionalModes].map(mode => ({
+            key: mode.key,
+            value: mode.key,
+            text: mode.label,
+            content: (
+              <>
+                {mode.label}
+                {mode.description && (
+                  <>
+                    <br />
+                    <span
+                      style={{
+                        fontWeight: 'normal',
+                        marginTop: '0.4em',
+                        color: '#5b5b5b',
+                        maxWidth: '13em',
+                        whiteSpace: 'normal',
+                        display: 'inline-block',
+                        fontSize: '0.9em',
+                      }}
+                    >
+                      {mode.description}
+                    </span>
+                  </>
+                )}
+              </>
+            ),
+          }))}
+        />
+      </div>
+    </>
  )
 }

-const createStudentToProgrammeMap = (courses, students, elementDetails) => {
-  if (!elementDetails) {
-    return {}
-  }
+const getStudentProgrammes = fp.flow(
+  fp.get('studyrights'),
+  fp.flatMap('studyright_elements'),
+  fp.filter(['element_detail.type', 20])
+)

-  const studentToTargetCourseDateMap = getStudentToTargetCourseDateMap(students, courses)
+const createStudentToProgrammeMap = (students, studyRightPredicate) => {
+  const studentProgrammePairs = []

-  return fp.flow(
-    fp.map(student => [
-      student,
-      getNewestProgramme(student.studyrights, student.studentNumber, studentToTargetCourseDateMap, elementDetails),
-    ]),
-    fp.reduce(
-      ({ programmeMap, studentToProgrammeMap }, [student, programme]) => {
-        return {
-          programmeMap: { ...programmeMap, [programme.code]: programme },
-          studentToProgrammeMap: {
-            ...studentToProgrammeMap,
-            [student.studentNumber]: programme.code,
-          },
-        }
-      },
-      { programmeMap: {}, studentToProgrammeMap: {} }
-    ),
-    ({ programmeMap, studentToProgrammeMap }) => ({
-      programmes: Object.values(programmeMap),
-      studentToProgrammeMap,
+  students.forEach(student => {
+    getStudentProgrammes(student).forEach(programme => {
+      studentProgrammePairs.push({ student, programme })
    })
-  )(students)
+  })
+
+  const programmes = fp.flow(fp.map('programme.element_detail'), fp.uniqBy('code'))(studentProgrammePairs)
+
+  const studentToProgrammeMap = fp.flow(
+    fp.filter(({ student, programme }) => studyRightPredicate(student, programme)),
+    fp.groupBy('student.studentNumber'),
+    fp.mapValues(fp.map('programme.element_detail'))
+  )(studentProgrammePairs)
+
+  return { programmes, studentToProgrammeMap }
+}
+
+const MODE_PREDICATES = {
+  any: () => true,
+  active: (_, sre) => moment().isBetween(sre.startdate, sre.enddate, 'day', '[]'),
 }

 const filter = createFilter({
@@ -78,14 +188,40 @@ const filter = createFilter({

  defaultOptions: {
    selectedProgrammes: [],
+    mode: 'active',
  },

-  precompute: ({ students, args }) => createStudentToProgrammeMap(args.courses, students, args.elementDetails ?? []),
+  precompute: ({ students, options, args }) => {
+    let predicate = () => true
+
+    if (args?.studyRightPredicate) {
+      predicate = args.studyRightPredicate
+    }
+
+    if (options.mode) {
+      let modePredicate = MODE_PREDICATES[options.mode]
+
+      const additional = _.get(args, 'additionalModes', []).find(mode => mode.key === options.mode)?.predicate
+
+      if (!modePredicate && additional) {
+        modePredicate = additional
+      }
+
+      if (modePredicate) {
+        const prevPredicate = predicate
+        predicate = (student, sre) => prevPredicate(student, sre) && modePredicate(student, sre)
+      }
+    }
+
+    return createStudentToProgrammeMap(students, predicate)
+  },

  isActive: ({ selectedProgrammes }) => selectedProgrammes.length > 0,

  filter({ studentNumber }, { selectedProgrammes }, { precomputed: { studentToProgrammeMap } }) {
-    return selectedProgrammes.some(pcode => pcode === studentToProgrammeMap[studentNumber])
+    return selectedProgrammes.every(pcode =>
+      _.get(studentToProgrammeMap, studentNumber, []).some(({ code }) => code === pcode)
+    )
  },

  selectors: {
@@ -104,7 +240,15 @@ const filter = createFilter({
    },
  },

-  render: (props, { precomputed }) => <ProgrammeFilterCard {...props} programmes={precomputed.programmes} />,
+  render: (props, { precomputed, args }) => (
+    <ProgrammeFilterCard
+      {...props}
+      programmes={precomputed.programmes}
+      studentToProgrammeMap={precomputed.studentToProgrammeMap}
+      additionalModes={args?.additionalModes ?? []}
+      studyRightPredicate={args?.studyRightPredicate ?? (() => true)}
+    />
+  ),
 })

 export default filter
@@ -0,0 +1,85 @@
+import React from 'react'
+import { Form, Dropdown } from 'semantic-ui-react'
+import { getTextIn } from 'common'
+import moment from 'moment'
+import _ from 'lodash'
+import createFilter from './createFilter'
+
+const StudyTrackFilterCard = ({ options, onOptionsChange, withoutSelf, activeAt }) => {
+  const { selected } = options
+
+  const activeAtMoment = activeAt ? moment(activeAt) : moment()
+
+  const dropdownOptions = _.chain(withoutSelf())
+    .flatMap(student => student.studyrights)
+    .flatMap(sr => sr.studyright_elements)
+    .filter(sre => sre.element_detail.type === 30 && activeAtMoment.isBetween(sre.startdate, sre.enddate, 'day', '[]'))
+    .map(sre => sre.element_detail)
+    .keyBy('code')
+    .values()
+    .map(({ code, name }) => ({
+      key: code,
+      value: code,
+      text: `${getTextIn(name)} (${code})`,
+      content: (
+        <>
+          {getTextIn(name)}{' '}
+          <span style={{ whiteSpace: 'nowrap', color: 'rgb(136, 136, 136)', fontSize: '0.8em' }}>({code})</span>
+        </>
+      ),
+    }))
+    .value()
+
+  const handleChange = (_, { value }) => {
+    onOptionsChange({
+      selected: value,
+    })
+  }
+
+  return (
+    <Form>
+      <Dropdown
+        options={dropdownOptions}
+        value={selected}
+        onChange={handleChange}
+        placeholder="Choose study track"
+        className="mini"
+        selection
+        clearable
+        button
+        search
+        multiple
+        fluid
+        data-cy="StudyTrack-filter-dropdown"
+      />
+    </Form>
+  )
+}
+
+export default createFilter({
+  key: 'StudyTrack',
+
+  title: 'Study Track',
+
+  defaultOptions: {
+    selected: [],
+  },
+
+  isActive: ({ selected }) => selected.length > 0,
+
+  filter: (student, { selected, args }) => {
+    const activeAt = _.get(args, 'activeAt', moment())
+
+    return student.studyrights
+      .flatMap(sr => sr.studyright_elements)
+      .filter(
+        sre =>
+          sre.element_detail.type === 30 &&
+          (!activeAt || moment(activeAt).isBetween(sre.startdate, sre.enddate, 'day', '[]'))
+      )
+      .map(sre => sre.element_detail.code)
+      .some(code => selected.includes(code))
+  },
+
+  render: (props, { args }) => <StudyTrackFilterCard {...props} activeAt={_.get(args, 'activeAt')} />,
+})
@@ -1,5 +1,5 @@
 import React from 'react'
-import { Form, Dropdown } from 'semantic-ui-react'
+import { Form, Dropdown, Message } from 'semantic-ui-react'
 import fp from 'lodash/fp'
 import createFilter from './createFilter'

@@ -14,7 +14,13 @@ const TagsFilterCard = ({ options, onOptionsChange, withoutSelf }) => {
    value: tag,
  }))

-  if (Object.entries(countsByTag).length === 0) return null
+  if (Object.entries(countsByTag).length === 0) {
+    return (
+      <Message color="orange" size="tiny">
+        No tags have been defined for any of the selected students.
+      </Message>
+    )
+  }

  return (
    <div className="card-content">
@@ -131,7 +131,9 @@ const FilterView = ({ children, name, filters: pFilters, students, displayTray:
  return (
    <FilterViewContext.Provider value={value}>
      <div style={{ display: 'flex', alignItems: 'start', justifyContent: 'center', gap: '0.75rem' }}>
-        <div style={{ alignSelf: 'flex-start', position: 'sticky', top: '1rem' }}>{displayTray && <FilterTray />}</div>
+        <div style={{ zIndex: 1, alignSelf: 'flex-start', position: 'sticky', top: '1rem' }}>
+          {displayTray && <FilterTray />}
+        </div>
        <div style={{ flexGrow: 1 }}>{typeof children === 'function' ? children(filteredStudents) : children}</div>
      </div>
    </FilterViewContext.Provider>
@@ -48,15 +48,15 @@ const OodiToOodikone = () => {
 }

 const FrontPage = () => {
-  const { rights, userRoles } = useGetAuthorizedUserQuery()
+  const { rights, roles } = useGetAuthorizedUserQuery()
  useTitle()

  const showItems = {
-    populations: userRoles.includes('admin') || rights.length > 0,
-    studyProgramme: userRoles.includes('admin') || rights.length > 0,
-    students: checkUserAccess(['studyGuidanceGroups', 'admin'], userRoles) || rights.length > 0,
-    courseStatistics: checkUserAccess(['courseStatistics', 'admin'], userRoles) || rights.length > 0,
-    teachers: checkUserAccess(['teachers', 'admin'], userRoles),
+    populations: roles.includes('admin') || rights.length > 0,
+    studyProgramme: roles.includes('admin') || rights.length > 0,
+    students: checkUserAccess(['studyGuidanceGroups', 'admin'], roles) || rights.length > 0,
+    courseStatistics: checkUserAccess(['courseStatistics', 'admin'], roles) || rights.length > 0,
+    teachers: checkUserAccess(['teachers', 'admin'], roles),
    trends: true,
    feedback: true,
  }
@@ -28,12 +28,17 @@ const allNavigationItems = {
    label: 'Study guidance groups',
    reqRights: ['studyGuidanceGroups'],
  },
+  customPopulations: {
+    path: '/custompopulation',
+    key: 'customPopulation',
+    label: 'Custom populations',
+  },
  updater: { path: '/updater', key: 'updater', label: 'Updater', reqRights: ['admin'] },
  feedback: { path: '/feedback', key: 'feedback', label: 'Give feedback' },
 }

 const NavigationBar = () => {
-  const { isLoading, rights, mockedBy, userId, userRoles } = useGetAuthorizedUserQuery()
+  const { isLoading, rights, mockedBy, userId, roles, isAdmin } = useGetAuthorizedUserQuery()
  const showAsUser = useShowAsUser()
  const [logout] = useLogoutMutation()

@@ -41,15 +46,16 @@ const NavigationBar = () => {
    const visibleNavigationItems = {}
    if (isLoading) return visibleNavigationItems
    Object.keys(allNavigationItems).forEach(key => {
-      if (key === 'populations' || key === 'students') {
-        if (!checkUserAccess(['admin', 'studyGuidanceGroups'], userRoles) && rights.length === 0) {
-          return
-        }
+      if (key === 'populations') {
+        if (!isAdmin && rights.length === 0) return
+      }
+      if (key === 'students' || key === 'customPopulations') {
+        if (!checkUserAccess(['admin', 'studyGuidanceGroups'], roles) && rights.length === 0) return
      } else if (key === 'courseStatistics') {
-        if (!checkUserAccess(['courseStatistics', 'admin'], userRoles) && rights.length < 1) return
+        if (!checkUserAccess(['courseStatistics', 'admin'], roles) && rights.length === 0) return
      }
      const { reqRights } = allNavigationItems[key]
-      if (!reqRights || reqRights.every(r => userRoles.includes(r))) {
+      if (!reqRights || reqRights.every(r => roles.includes(r))) {
        visibleNavigationItems[key] = allNavigationItems[key]
      }
    })
@@ -93,22 +93,19 @@ const useDelayedMemo = (fn, watch) => {

 const PopulationCourseStats = props => {
  const { language } = useLanguage()
-
  const [filterFields, setFilterFields] = useState({ codeFilter: '', nameFilter: '' })
  const [modules, setModules] = useState([])
-
  const [state, setState] = useState(initialState(props))
+  const [expandedGroups, setExpandedGroups] = useState(new Set())
+  const mandatoryCourses = useSelector(({ populationMandatoryCourses }) => populationMandatoryCourses.data)
+
+  const { handleTabChange } = useTabChangeAnalytics('Population statistics', 'Courses of Population tab changed')

  const courseStatistics = useDelayedMemo(
-    () => updateCourseStatisticsCriteria(props.courses?.coursestatistics, language, state),
+    () => updateCourseStatisticsCriteria(props.courses?.coursestatistics, language, state, mandatoryCourses),
    [props.courses, state, language]
  )

-  const [expandedGroups, setExpandedGroups] = useState(new Set())
-
-  const mandatoryCourses = useSelector(({ populationMandatoryCourses }) => populationMandatoryCourses.data)
-  const { handleTabChange } = useTabChangeAnalytics('Population statistics', 'Courses of Population tab changed')
-
  useEffect(() => {
    if (state && props.courses) {
      const studentAmountLimit =
@@ -8,6 +8,7 @@ import Datetime from 'react-datetime'
 import { sortBy, isEqual } from 'lodash'
 import moment from 'moment'

+import infoTooltips from 'common/InfoToolTips'
 import { getPopulationStatistics, clearPopulations } from '../../redux/populations'
 import { getPopulationCourses } from '../../redux/populationCourses'
 import { getPopulationSelectedStudentCourses, clearSelected } from '../../redux/populationSelectedStudentCourses'
@@ -15,7 +16,6 @@ import { getMandatoryCourses } from '../../redux/populationMandatoryCourses'
 import { getSemesters } from '../../redux/semesters'

 import { getProgrammes } from '../../redux/populationProgrammes'
-import { getTagsByStudytrackAction } from '../../redux/tags'
 import { momentFromFormat, reformatDate, textAndDescriptionSearch, getTextIn, cancelablePromise } from '../../common'
 import { useSearchHistory } from '../../common/hooks'
 import { setLoading } from '../../redux/graphSpinner'
@@ -53,7 +53,7 @@ const PopulationSearchForm = props => {

  const { query, isLoading, momentYear } = totalState

-  const { studyProgrammes, location, semesters, queries, history, tags, language, onProgress } = props
+  const { studyProgrammes, location, semesters, queries, history, language, onProgress } = props

  const parseQueryFromUrl = () => {
    const initial = initialQuery()
@@ -111,7 +111,6 @@ const PopulationSearchForm = props => {
        props.getPopulationCourses(request),
        [],
        props.getMandatoryCourses(formattedQueryParams.studyRights.programme),
-        props.getTagsByStudytrackAction(query.studyRights.programme),
      ])
    )
    const success = await fetchPopulationPromises.current.promise
@@ -171,9 +170,8 @@ const PopulationSearchForm = props => {
    })
  }

-  const handleProgrammeChange = (e, { value }) => {
+  const handleProgrammeChange = (_e, { value }) => {
    const programme = value
-    props.getTagsByStudytrackAction(value)
    if (programme === '') {
      handleClear('programme')
      return
@@ -206,7 +204,7 @@ const PopulationSearchForm = props => {
  }

  const getSearchHistoryTextFromQuery = () => {
-    const { studyRights, semesters, months, year, studentStatuses, tag } = query
+    const { studyRights, semesters, months, year, studentStatuses } = query
    const studyRightsText = `${getTextIn(studyProgrammes[studyRights.programme].name, language)} ${Object.values(
      studyRights
    )
@@ -215,9 +213,8 @@ const PopulationSearchForm = props => {
    const timeText = `${semesters.join(', ')}/${year}-${parseInt(year, 10) + 1}, ${months} months`
    const studentStatusesText =
      studentStatuses.length > 0 ? `includes ${studentStatuses.map(s => s.toLowerCase()).join(', ')} students` : null
-    const tagText = !tag ? null : `Tag: ${tags.find(t => t.tag_id === tag).tagname}`

-    return [studyRightsText, timeText, studentStatusesText, tagText].filter(t => t).join(' - ')
+    return [studyRightsText, timeText, studentStatusesText].filter(t => t).join(' - ')
  }

  const handleSubmit = () => {
@@ -257,12 +254,11 @@ const PopulationSearchForm = props => {
        }
      }
    }
-    const tag = tags.find(t => t.tag_id === query.tag)
+
    setState({
      momentYear,
      query: {
        ...query,
-        tag: query.tag && tag.year ? null : query.tag,
        year: reformatDate(momentYear, YEAR_DATE_FORMAT),
        months: months(
          reformatDate(momentYear, YEAR_DATE_FORMAT),
@@ -276,15 +272,6 @@ const PopulationSearchForm = props => {
    })
  }

-  const getMonths = (year, end, term) => {
-    if (moment.isMoment(end)) {
-      const lastDayOfMonth = moment(end).endOf('month')
-      const start = term === 'FALL' ? `${year}-08-01` : `${year}-01-01`
-      return Math.round(moment.duration(moment(lastDayOfMonth).diff(moment(start))).asMonths())
-    }
-    return -1
-  }
-
  const addYear = () => {
    const { year } = query
    const nextYear = momentFromFormat(year, YEAR_DATE_FORMAT).add(1, 'year')
@@ -297,23 +284,6 @@ const PopulationSearchForm = props => {
    handleYearSelection(previousYear)
  }

-  const handleStudyTrackChange = (e, { value }) => {
-    const studyTrack = value
-    if (studyTrack === '') {
-      handleClear('studyTrack')
-      return
-    }
-    setState({
-      query: {
-        ...query,
-        studyRights: {
-          ...query.studyRights,
-          studyTrack,
-        },
-      },
-    })
-  }
-
  const validYearCheck = momentYear => {
    if (!moment.isMoment(momentYear)) {
      return false
@@ -324,31 +294,6 @@ const PopulationSearchForm = props => {
    return props.studyProgrammes[query.studyRights.programme].enrollmentStartYears[momentYear.year()] != null
  }

-  const handleTagSelection = (e, { value }) => {
-    const tag = tags.find(t => t.tag_id === value)
-    if (!tag) {
-      setState({
-        query: {
-          ...query,
-          tag: null,
-          months: query.months,
-        },
-      })
-    } else {
-      const months = tag.year
-        ? getMonths(reformatDate(moment(`${tag.year}-01-01`), YEAR_DATE_FORMAT), moment(), 'FALL')
-        : query.months
-      setState({
-        query: {
-          ...query,
-          tag: tag.tag_id,
-          year: tag.year || query.year,
-          months,
-        },
-      })
-    }
-  }
-
  const renderableList = list =>
    list.map(sp => {
      const { type, name, code } = sp
@@ -418,57 +363,15 @@ const PopulationSearchForm = props => {
    </Form.Field>
  )

-  const renderTagDropdown = (tagOptions, chosenTag) => (
-    <Form.Field>
-      <label>Select tag (Optional)</label>
-      <Form.Dropdown
-        placeholder="select tag"
-        selection
-        value={chosenTag.tag_id}
-        options={tagOptions}
-        onChange={handleTagSelection}
-        selectOnBlur={false}
-        selectOnNavigation={false}
-      />
-    </Form.Field>
-  )
-
-  const renderAdditionalStudyTrackOrTagDropdown = (
-    studyRights,
-    studyTracksToRender,
-    shouldRenderTags,
-    tagOptions,
-    chosenTag
-  ) => {
-    const renderableTracks = () => (
-      <>
-        <label>Study Track (Optional)</label>
-        <Form.Dropdown
-          placeholder="Select study track"
-          search={textAndDescriptionSearch}
-          floating
-          selection
-          noResultsMessage="No selectable study track"
-          value={studyRights.studyTrack}
-          options={studyTracksToRender}
-          onChange={handleStudyTrackChange}
-          closeOnChange
-          clearable
-          selectOnBlur={false}
-          selectOnNavigation={false}
-        />
-      </>
-    )
+  const renderAdditionalStudyTrackOrTagDropdown = studyRights => {
    if (studyRights.programme) {
      return (
-        <Form.Group>
-          <Form.Field width={8}>{shouldRenderTags ? renderTagDropdown(tagOptions, chosenTag) : null}</Form.Field>
-          <Form.Field width={8}>
-            {studyTracksToRender && studyTracksToRender.length > 0 ? renderableTracks() : null}
-          </Form.Field>
-        </Form.Group>
+        <Message size="tiny" style={{ maxWidth: '25em' }}>
+          {infoTooltips.PopulationStatistics.TagAndTrackMovedIntoFilters}
+        </Message>
      )
    }
+
    return null
  }

@@ -492,29 +395,11 @@ const PopulationSearchForm = props => {
      const sortedStudyProgrammes = sortBy(studyProgrammes, s => getTextIn(s.name, language))
      programmesToRender = renderableList(sortedStudyProgrammes)
    }
-    let studyTracksToRender
-    if (studyRights.programme && validYearCheck(momentYear)) {
-      const associations = studyProgrammes[studyRights.programme].enrollmentStartYears[momentYear.year()]
-      if (associations) {
-        const sortedStudyTracks = sortBy(associations.studyTracks, s => getTextIn(s.name, language))
-        studyTracksToRender = renderableList(sortedStudyTracks)
-      }
-    }
-    const tagOptions = [{ key: 0, text: '', value: '' }]
-    tags.forEach(tag => tagOptions.push({ key: tag.tag_id, text: tag.tagname, value: tag.tag_id }))
-    const chosenTag = tags.find(tag => tag.tag_id === query.tag) || { tagname: '', tag_id: '' }
-    const shouldRenderTags = tags.filter(tag => tag.studytrack === query.studyRights.programme).length > 0

    return (
      <div>
        {renderStudyProgrammeDropdown(studyRights, programmesToRender)}
-        {renderAdditionalStudyTrackOrTagDropdown(
-          studyRights,
-          studyTracksToRender,
-          shouldRenderTags,
-          tagOptions,
-          chosenTag
-        )}
+        {renderAdditionalStudyTrackOrTagDropdown(studyRights)}
      </div>
    )
  }
@@ -566,7 +451,7 @@ const PopulationSearchForm = props => {
  )
 }

-const mapStateToProps = ({ semesters, settings, populations, populationProgrammes, tags }) => {
+const mapStateToProps = ({ semesters, settings, populations, populationProgrammes }) => {
  const { language } = settings
  const { pending } = populationProgrammes
  return {
@@ -575,7 +460,6 @@ const mapStateToProps = ({ semesters, settings, populations, populationProgramme
    queries: populations.query || {},
    studyProgrammes: populationProgrammes.data.programmes || {},
    pending,
-    tags: tags.data,
  }
 }

@@ -588,7 +472,6 @@ export default withRouter(
    getProgrammes,
    setLoading,
    getSemesters,
-    getTagsByStudytrackAction,
    clearSelected,
    clearPopulations,
  })(PopulationSearchForm)
@@ -20,6 +20,8 @@ import {
  tagsFilter,
  creditDateFilter,
  enrollmentStatusFilter,
+  studyTrackFilter,
+  programmeFilter,
 } from '../FilterView/filters'

 const selectPopulations = createSelector(
@@ -58,6 +60,8 @@ const PopulationStatistics = () => {
      allSemesters: allSemesters ?? [],
      language,
    }),
+    studyTrackFilter,
+    programmeFilter,
  ]

  if (parseInt(query?.year, 10) >= 2020) {
@@ -55,8 +55,12 @@ const PopulationStudents = ({
  useEffect(() => {
    if (tags && tags.length > 0) return
    const queryStudyright = queryStudyrights[0]
-    dispatch(getTagsByStudytrackAction(queryStudyright))
-    dispatch(getStudentTagsByStudytrackAction(queryStudyright))
+
+    if (queryStudyright) {
+      dispatch(getTagsByStudytrackAction(queryStudyright))
+      dispatch(getStudentTagsByStudytrackAction(queryStudyright))
+    }
+
    setState({ ...state, admin })
  }, [])

@@ -68,6 +72,7 @@ const PopulationStudents = ({

  const mandatoryCodes = mandatoryCourses.filter(course => course.visible && course.visible.visibility).map(c => c.code)
  let mandatoryPassed = {}
+
  if (populationCourses.coursestatistics) {
    const mandatorycodesMapCourseCodeToCourseID = new Map()
    mandatoryCourses
@@ -17,10 +17,10 @@ const NoAccessToPageBanner = () => (
 )

 const ProtectedRoute = ({ requiredRoles = [], requireUserHasRights = false, ...rest }) => {
-  const { rights, isAdmin, userRoles } = useGetAuthorizedUserQuery()
+  const { rights, isAdmin, roles } = useGetAuthorizedUserQuery()
  const hasAccessToRoute = () => {
    if (isAdmin) return true
-    const hasRequiredRoles = requiredRoles.length > 0 ? checkUserAccess(requiredRoles, userRoles) : true
+    const hasRequiredRoles = requiredRoles.length > 0 ? checkUserAccess(requiredRoles, roles) : true
    const hasRequiredRights = requireUserHasRights ? rights.length > 0 : true
    if (requiredRoles.includes('courseStatistics')) {
      return hasRequiredRoles || hasRequiredRights
@@ -206,6 +206,26 @@ const SingleStudyGroupFilterView = props => {
      courses: props.courses?.coursestatistics ?? [],
    }),
    filters.creditDateFilter,
+    filters.studyTrackFilter,
+    filters.programmeFilter({
+      additionalModes: props.group?.tags?.year
+        ? [
+            {
+              key: 'assoc-year',
+              label: 'Since Assoc. Year',
+              description:
+                'Student has had a study right since the start year associated with this study guidance group.',
+              predicate: (_student, sre) =>
+                moment(createAcademicYearStartDate(props.group.tags?.year)).isBetween(
+                  sre.startdate,
+                  sre.enddate,
+                  'day',
+                  '[]'
+                ),
+            },
+          ]
+        : [],
+    }),
  ]

  if (props.group?.tags?.studyProgramme && props.group?.tags?.year && parseInt(props.group.tags.year, 10) >= 2020) {
@@ -6,7 +6,7 @@ const ReactHighcharts = require('react-highcharts')
 const colors = ['#333737', '#bdc2c7', '#1392c2', '#195f8a', '#54997b', '#036415']

 const BarChart = ({ data, track }) => {
-  if (!data || !data.creditGraphStats[track])
+  if (!data || !data.creditGraphStats || !data.creditGraphStats[track])
    return <NoDataMessage message="No progress data for the studytrack found. Try with another studytrack" />

  const correctData = data?.creditGraphStats[track]
@@ -0,0 +1,38 @@
+import React, { useState } from 'react'
+import { Button, Icon, Loader } from 'semantic-ui-react'
+import { useUpdateBasicViewQuery, useUpdateStudytrackViewQuery } from '../../../redux/studyProgramme'
+
+const getStatusIcon = stats => {
+  if (stats.isLoading) return <Loader active />
+  if (stats.isSuccess) return <Icon name="check" color="green" />
+  if (stats.isError) return <Icon name="close" color="red" />
+  return ''
+}
+
+const NewUpdateView = ({ studyprogramme }) => {
+  const [skipBasic, setSkipBasic] = useState(true)
+  const [skipStudytrack, setSkipStudytrack] = useState(true)
+  const basicstats = useUpdateBasicViewQuery({ id: studyprogramme }, { skip: skipBasic })
+  const studytrackstats = useUpdateStudytrackViewQuery({ id: studyprogramme }, { skip: skipStudytrack })
+
+  return (
+    <div className="update-view">
+      <div className="button-container">
+        <h4>Update data on Basic Information -view</h4>
+        <Button disabled={basicstats.isLoading} color="blue" onClick={() => setSkipBasic(false)}>
+          Update
+        </Button>
+        {getStatusIcon(basicstats)}
+      </div>
+      <div className="button-container">
+        <h4>Update data on Populations and Studytracks -view</h4>
+        <Button disabled={studytrackstats.isLoading} color="blue" onClick={() => setSkipStudytrack()}>
+          Update
+        </Button>
+        {getStatusIcon(studytrackstats)}
+      </div>
+    </div>
+  )
+}
+
+export default NewUpdateView
@@ -9,6 +9,7 @@ import StudyProgrammeSelector from './StudyProgrammeSelector'
 import Overview from './Overview'
 import NewOverview from './NewOverview'
 import NewStudytrackOverview from './NewStudytrackOverview'
+import NewUpdateView from './NewUpdateView'
 import StudyTrackOverview from './StudyTrackOverview'
 import ThesisCourses from './ThesisCourses'
 import '../PopulationQueryCard/populationQueryCard.css'
@@ -62,7 +63,7 @@ const StudyProgramme = props => {
      })
  }

-  const SHOW_NEW_OVERVIEW = isAdmin && roles?.find(r => r.group_code === 'teachers')
+  const SHOW_NEW_OVERVIEW = isAdmin && roles.includes('teachers')

  const getPanes = () => {
    const { match, programmes } = props
@@ -96,6 +97,17 @@ const StudyProgramme = props => {
      menuItem: 'Tags',
      render: () => <Tags studyprogramme={studyProgrammeId} />,
    })
+    if (isAdmin) {
+      panes.push({
+        menuItem: 'Admin',
+        render: () => (
+          <>
+            <Button onClick={() => refreshThroughput()}>recalculate throughput</Button>
+            <Button onClick={() => refreshProductivity()}>recalculate productivity</Button>
+          </>
+        ),
+      })
+    }
    if (SHOW_NEW_OVERVIEW) {
      panes.push({
        menuItem: 'Basic information (NEW)',
@@ -108,15 +120,10 @@ const StudyProgramme = props => {
        render: () => <NewStudytrackOverview studyprogramme={studyProgrammeId} history={props.history} />,
      })
    }
-    if (isAdmin) {
+    if (SHOW_NEW_OVERVIEW) {
      panes.push({
-        menuItem: 'Admin',
-        render: () => (
-          <>
-            <Button onClick={() => refreshThroughput()}>recalculate throughput</Button>
-            <Button onClick={() => refreshProductivity()}>recalculate productivity</Button>
-          </>
-        ),
+        menuItem: 'Update statistics (NEW)',
+        render: () => <NewUpdateView studyprogramme={studyProgrammeId} />,
      })
    }
    return panes
@@ -50,3 +50,11 @@
 .studytrack-overview .no-data-message {
  margin: 20px 0;
 }
+
+.update-view {
+  margin: 2em 0;
+}
+
+.update-view .button-container {
+  margin: 1em 0;
+}
@@ -36,14 +36,12 @@ export const useGetAuthorizedUserQuery = () => {

  if (isLoading || error) return { data, isLoading, error, ...rest }

-  const { token } = data
-  const userRoles = token?.roles?.map(r => r.group_code) ?? []
+  const { user } = data

  return {
-    ...token,
-    userRoles,
-    isAdmin: userRoles.includes('admin'),
+    ...user,
    isLoading,
+    error,
    ...rest,
  }
 }
@@ -14,9 +14,21 @@ const studyprogrammeApi = RTKApi.injectEndpoints({
    getStudytrackStats: builder.query({
      query: ({ id }) => `/v2/studyprogrammes/${id}/studytrackstats`,
    }),
+    updateBasicView: builder.query({
+      query: ({ id }) => `/v2/studyprogrammes/${id}/update_basicview`,
+    }),
+    updateStudytrackView: builder.query({
+      query: ({ id }) => `/v2/studyprogrammes/${id}/update_studytrackview`,
+    }),
  }),
  overrideExisting: false,
 })

-export const { useGetBasicStatsQuery, useGetCreditStatsQuery, useGetGraduationStatsQuery, useGetStudytrackStatsQuery } =
-  studyprogrammeApi
+export const {
+  useGetBasicStatsQuery,
+  useGetCreditStatsQuery,
+  useGetGraduationStatsQuery,
+  useGetStudytrackStatsQuery,
+  useUpdateBasicViewQuery,
+  useUpdateStudytrackViewQuery,
+} = studyprogrammeApi
@@ -0,0 +1,12 @@
+const { BOOLEAN } = require('sequelize')
+
+module.exports = {
+  up: async queryInterface => {
+    await queryInterface.addColumn('credit', 'is_open', {
+      type: BOOLEAN,
+    })
+  },
+  down: async queryInterface => {
+    await queryInterface.deleteColumn('credit', 'is_open')
+  },
+}
@@ -78,6 +78,9 @@ Credit.init(
    language: {
      type: STRING,
    },
+    is_open: {
+      type: BOOLEAN,
+    },
  },
  {
    underscored: false,
@@ -131,8 +131,7 @@ const creditMapper =
    courseUnitIdToCourseGroupId,
    moduleGroupIdToModuleCode,
    courseGroupIdToCourseCode,
-    studyrightIdToOrganisationsName,
-    courseCodeToAyCodelessId
+    studyrightIdToOrganisationsName
  ) =>
  attainment => {
    const {
@@ -165,15 +164,17 @@ const creditMapper =

    let course_id = !isModule(type) ? courseUnitIdToCourseGroupId[course_unit_id] : module_group_id

-    // if ay studyright and no ay code
+    // check if ay code or no ay code but ay studyright
+    let is_open = false
    if (study_right_id !== null && course_code) {
      if (!isModule(type)) {
-        const organisationName = studyrightIdToOrganisationsName[study_right_id]
-        if (organisationName) {
-          if (organisationName['fi'].startsWith('Avoin yliopisto')) {
-            if (!course_code.startsWith('AY')) {
-              course_code = 'AY'.concat(course_code)
-              course_id = courseCodeToAyCodelessId.get(course_code)
+        if (course_code.match(/^AY?(.+?)(?:en|fi|sv)?$/)) {
+          is_open = true
+        } else {
+          const organisationName = studyrightIdToOrganisationsName[study_right_id]
+          if (organisationName) {
+            if (organisationName['fi'].startsWith('Avoin yliopisto')) {
+              is_open = true
            }
          }
        }
@@ -183,6 +184,8 @@ const creditMapper =
    // These are leaf attainments that have no other attainments attached to them
    const isStudyModule = nodes && nodes[0] !== undefined

+    // console.log('isOpen: ', is_open)
+
    return {
      id: id,
      grade: getGrade(grade_scale_id, grade_id).value,
@@ -198,6 +201,7 @@ const creditMapper =
      isStudyModule,
      org: attainmentUniOrg,
      language: language,
+      is_open: is_open,
    }
  }

@@ -62,6 +62,21 @@ const updateCourses = async (courseIdToAttainments, groupIdToCourse) => {
    return mapCourse(groupedCourse, substitutions)
  })

+  // change substitutions ids to course codes and update
+
+  for (const course of courses) {
+    const newSubstitutions = []
+    for (const sub of course.substitutions) {
+      const subs = await Course.findOne({
+        attributes: ['code'],
+        where: { id: sub },
+      })
+
+      if (subs) newSubstitutions.push(subs.dataValues.code)
+    }
+    course.substitutions = newSubstitutions
+  }
+
  await bulkCreate(Course, courses)
  await bulkCreate(
    CourseProvider,
@@ -293,101 +293,6 @@ const updateAttainments = async (attainments, personIdToStudentNumber, attainmen
  const coursesToBeUpdated = new Map()
  const courseProvidersToBeCreated = []

-  // because of AY-codeless open uni courses
-  const fixAyCodelessOpenUniCourses = async (attainments, sisDbCoursesForStudentAttainments) => {
-    const groupIdsWithOpenUnStudyright = attainments.reduce((res, curr) => {
-      if (curr.study_right_id !== null && curr.type === 'CourseUnitAttainment') {
-        const organisationName = studyrightIdToOrganisationsName[curr.study_right_id]
-        if (organisationName) {
-          if (organisationName['fi'].startsWith('Avoin yliopisto')) {
-            res.push(courseUnitIdToCourseGroupId[curr.course_unit_id])
-          }
-        }
-      }
-      return res
-    }, [])
-
-    const coursesWithNoAyCodeAndOpenUniStudyright = groupIdsWithOpenUnStudyright.reduce((res, curr) => {
-      const foundCourse = sisDbCoursesForStudentAttainments.find(c => c.id === curr)
-      if (foundCourse) {
-        if (!foundCourse.code.startsWith('AY')) {
-          res.push(foundCourse)
-        }
-      }
-      return res
-    }, [])
-
-    const coursesWithAyCodeAlreadyExist = await Course.findAll({
-      raw: true,
-      where: {
-        code: {
-          [Op.in]: Object.values(coursesWithNoAyCodeAndOpenUniStudyright.map(c => 'AY'.concat(c.code))),
-        },
-      },
-    })
-
-    for (const course of coursesWithNoAyCodeAndOpenUniStudyright) {
-      const courseWithIdForAyCodeExist = coursesWithAyCodeAlreadyExist.find(
-        c => c.code === 'AY'.concat(course.code) && !c.id.endsWith('-ay')
-      )
-
-      if (courseWithIdForAyCodeExist) {
-        const latest_instance_date = getFirstDateIfIsAfter(
-          course.latest_instance_date,
-          courseWithIdForAyCodeExist.latest_instance_date
-        )
-        const max_attainment_date = getFirstDateIfIsAfter(
-          course.max_attainment_date,
-          courseWithIdForAyCodeExist.max_attainment_date
-        )
-        const min_attainment_date = getFirstDateIfIsBefore(
-          course.min_attainment_date,
-          courseWithIdForAyCodeExist.min_attainment_date
-        )
-
-        if (latest_instance_date || max_attainment_date || min_attainment_date) {
-          coursesToBeUpdated.set(courseWithIdForAyCodeExist.code, {
-            ...courseWithIdForAyCodeExist,
-            latest_instance_date: latest_instance_date,
-            max_attainment_date: max_attainment_date,
-            min_attainment_date: min_attainment_date,
-          })
-        }
-
-        courseCodeToAyCodelessId.set('AY'.concat(course.code), courseWithIdForAyCodeExist.id)
-      } else {
-        coursesToBeCreated.set('AY'.concat(course.code), {
-          ...course.dataValues,
-          id: course.id.concat('-ay'),
-          code: 'AY'.concat(course.code),
-          substitutions: course[course.id],
-        })
-
-        if (!course.substitutions.includes(course.id.concat('-ay'))) {
-          const subs = [...course.substitutions, course.id.concat('-ay')]
-          coursesToBeUpdated.set(course.code, {
-            ...course.dataValues,
-            substitutions: subs,
-          })
-        }
-
-        courseCodeToAyCodelessId.set('AY'.concat(course.code), course.id.concat('-ay'))
-        const courseProvider = await CourseProvider.findOne({
-          where: {
-            coursecode: `${course.id}-ay`,
-          },
-        })
-
-        if (!courseProvider) {
-          const mapCourseProvider = courseProviderMapper(course.id.concat('-ay'))
-          courseProvidersToBeCreated.push(mapCourseProvider({ organisationId: 'hy-org-48645785' }))
-        }
-      }
-    }
-  }
-
-  await fixAyCodelessOpenUniCourses(attainments, sisDbCoursesForStudentAttainments)
-
  // This mayhem fixes missing course_unit references for CustomCourseUnitAttainments.
  const fixCustomCourseUnitAttainments = async attainments => {
    const addCourseUnitToCustomCourseUnitAttainments = (courses, attIdToCourseCode) => async att => {
@@ -637,24 +542,6 @@ const updateTermRegistrations = async (termRegistrations, personIdToStudentNumbe
  await bulkCreate(SemesterEnrollment, semesterEnrollments)
 }

-const getFirstDateIfIsAfter = (dateOne, dateTwo) => {
-  const isAfter = new Date(dateOne) > new Date(dateTwo) ? dateOne : dateTwo
-  if (isAfter) {
-    return dateOne
-  } else {
-    return null
-  }
-}
-
-const getFirstDateIfIsBefore = (dateOne, dateTwo) => {
-  const isAfter = new Date(dateOne) < new Date(dateTwo) ? dateOne : dateTwo
-  if (isAfter) {
-    return dateOne
-  } else {
-    return null
-  }
-}
-
 module.exports = {
  updateStudents,
 }
Author	SHA1	Message	Date
Saara Satokangas	f506989700	Separate studyprogamme updates into their own file Easier to use in the cronjobs now	2022-01-18 10:08:43 +02:00
Saara Satokangas	f75957f0d3	Add indication of loading to the new update-view	2022-01-18 09:39:04 +02:00
Saara Satokangas	f58df8336b	Add a way to manually update new studyprogramme overviews' stats	2022-01-18 09:35:06 +02:00
Petri Palmu	bfc83ae5b5	test fix	2022-01-17 10:09:13 +02:00
Petri Palmu	3366da3016	test tests	2022-01-17 09:03:06 +02:00
Petri Palmu	649f9bd821	more fix to the tests	2022-01-14 16:35:01 +02:00
Petri Palmu	b7e1a10e4d	test fix	2022-01-13 14:35:59 +02:00
Petri Palmu	7d2776661f	Merge branch 'master' of github.com:UniversityOfHelsinkiCS/oodikone	2022-01-13 13:05:13 +02:00
Petri Palmu	bc133c8a76	few minor changes to the tests for new anon data	2022-01-13 13:05:02 +02:00
Otto Ahoniemi	dac2892749	Add working rapodiff	2022-01-13 12:31:43 +02:00
Otto Ahoniemi	f004c458a1	Add some debugging tools to backend image	2022-01-13 10:58:00 +02:00
Otto Ahoniemi	17b6592d2a	Fix query typo	2022-01-13 10:46:22 +02:00
Otto Ahoniemi	83486d2719	Don't run tests when changing rapodiff script	2022-01-13 10:32:06 +02:00
Otto Ahoniemi	2652180f4b	Add rapo script base	2022-01-13 10:23:09 +02:00
Otto Ahoniemi	055492b76b	Add admin user to always have admin rights	2022-01-13 08:56:22 +02:00
Otto Ahoniemi	c761899962	Fix populations failing because of invalid admin role check	2022-01-13 08:53:55 +02:00
Otto Ahoniemi	998bb3a453	Promote custom populations to navbar & fix normal populations being show to studyguidancegroup users	2022-01-13 08:06:53 +02:00
Otto Ahoniemi	b9e9c89376	Fix embarassing bug where I forgot to return mocked user from function	2022-01-13 07:55:46 +02:00
Otto Ahoniemi	943a2145e9	Add toska group for mluukkai dev user	2022-01-13 07:35:27 +02:00
Otto Ahoniemi	4fef6a8582	Cleanup bunch of extra logic from userservice	2022-01-13 07:35:27 +02:00
Otto Ahoniemi	4cf3003dec	Race when waiting for studyguidancegroup answer from importer	2022-01-13 07:35:27 +02:00
Otto Ahoniemi	ec47fab422	Cleanup user and error handling for feedback	2022-01-13 07:35:27 +02:00
Otto Ahoniemi	c391d7786b	Merge roles and rights to be part of user object in request	2022-01-13 07:35:27 +02:00
Mitja Karhusaari	cb0e4eae16	Hyvää synttäriä, I guess 🤷‍♂️ 🍰	2022-01-11 15:23:28 +02:00
Mitja Karhusaari	3b10c19a81	Fix tests after FilterCard open/close logic change	2022-01-11 15:23:28 +02:00
Mitja Karhusaari	c60955c133	Fix chaining of the .cs cypress command	2022-01-11 15:23:28 +02:00
Mitja Karhusaari	14fba8d8ba	Open filter options automatically if changed	2022-01-11 15:23:28 +02:00
Petri Palmu	9843e9fafa	courses of population students are great again	2022-01-11 14:33:08 +02:00
Petri Palmu	08a51b05b1	Merge branch 'master' of github.com:UniversityOfHelsinkiCS/oodikone	2022-01-11 11:46:47 +02:00
Mitja Karhusaari	1a7df991e8	Remove tag and study track selectors from population search view	2022-01-11 11:14:43 +02:00
Mitja Karhusaari	2391062e71	Do not fetch tags for study track if query does not specify a study track	2022-01-11 11:09:13 +02:00
Mitja Karhusaari	f0c179e23d	Fix usage of the isBetween method in the context of study rights	2022-01-11 11:09:13 +02:00
Petri Palmu	b183c438bb	added bscs regex in code sorting order	2022-01-06 20:21:34 +02:00
Petri Palmu	60db28f4bb	fix with null check for subs in services population	2022-01-05 10:09:00 +02:00
Mitja Karhusaari	e8be70bf9f	Make linter happy	2022-01-05 02:06:08 +02:00
Mitja Karhusaari	7b60bc220c	Improve grade filter fault tolerance	2022-01-05 01:59:15 +02:00
Mitja Karhusaari	44dfaf20a6	Adjust tests	2022-01-05 01:50:43 +02:00
Mitja Karhusaari	009a9f5397	Implement tests for the modified filters	2022-01-05 01:49:45 +02:00
Mitja Karhusaari	5ac2db44e3	Change study track filter to consider only the currently active study rights by default	2022-01-05 01:20:50 +02:00
Mitja Karhusaari	337b1f05e4	Generalize the implementation of programme filter and add it to all views	2022-01-05 01:20:50 +02:00
Mitja Karhusaari	537d9ecc33	Display a message when opening the tags filter and no tags are available	2022-01-05 01:20:50 +02:00
Mitja Karhusaari	4e9d1cd190	Display filters in alphabetical order for consistency's sake	2022-01-05 01:20:50 +02:00
Mitja Karhusaari	fbe1cc93fb	Implement study track filter	2022-01-05 01:20:50 +02:00
Petri Palmu	12c3c9430d	little correction to the test	2022-01-05 01:05:00 +02:00
Petri Palmu	6251b69feb	lint fixes	2022-01-04 23:33:03 +02:00
Petri Palmu	6960b67bfb	changed anon oodi sis-db data	2022-01-04 23:26:05 +02:00
Otto AhoniemiandGitHub	cb3da3d1bc	Update anon db guide + fix some typos Some of these things are now part of CI + there were some possible places for misunderstandings.	2022-01-04 17:49:17 +02:00
Otto Ahoniemi	79df421846	Remove old forwarders used for sisu api transition	2022-01-04 16:35:43 +02:00
Otto Ahoniemi	d32c3c9bd9	Remove references to decodedToken in backend	2022-01-04 16:35:43 +02:00
Otto Ahoniemi	af38c9fc97	Transform roles and admin thingys already in backend	2022-01-04 16:35:43 +02:00
Otto Ahoniemi	843fc4c8ec	Merge usage of userRoles and roles	2022-01-04 16:35:43 +02:00
Petri Palmu	97105588df	restore tests	2022-01-04 14:22:30 +02:00
Petri Palmu	6a68dd7817	Merge branch 'master' of github.com:UniversityOfHelsinkiCS/oodikone	2022-01-04 13:04:34 +02:00
Petri Palmu	acfd918d80	removed more temp ay code	2022-01-04 13:03:31 +02:00
Petri Palmu	73c29b2edf	temporary ay codes removed	2022-01-03 17:42:13 +02:00
Petri Palmu	5102ba4ab3	little change to the tests	2022-01-03 12:18:14 +02:00
Petri Palmu	87746582c3	changed language distribution info text	2022-01-03 12:07:39 +02:00
Petri Palmu	2b739e9ff3	course statistics find courses works	2022-01-03 12:07:12 +02:00
Petri Palmu	3d32244238	added migration for the new avoin studyright	2022-01-03 11:45:12 +02:00
Petri Palmu	f86dce3fb8	Fixes to the codeless avoin	2021-12-29 16:04:44 +02:00
Petri Palmu	229904187e	course statistics find courses works	2021-12-23 08:54:52 +02:00
Petri Palmu	88a14f235b	Merge branch 'master' into feature/newAvoin	2021-12-21 10:45:40 +02:00
Petri Palmu	67736b295b	added migration for the new avoin studyright	2021-12-21 10:44:31 +02:00