This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Procedures

A set of procedures releated to the Vertex software

1 - Release procedure

This is the full release procedure for the Vertex application in a kubernetes environment

Release procedure

This procedure consists of a list of phases. Some of the phases are optional, some are required for every release. Each step will provide a objective of the step, a bit of background information and a checklist that needs to be checked at the end of the phase. Finally there is a NO-GO scenario described in the step if applicable. It will describe what to do in case the checklist cannot be completed because of complications. If we abort a phase and execute the NO-GO scenario we will always return to phase 0 to plan for a new release at another moment.

A diagram of the flow between the phases can be found here:

graph TD
    subgraph Normal release procedure
    P0(Phase 0: Preparation)
    P1(Phase 1: Snapshotting / Backup)
    P2(Optional: Phase 2: Environment Cleaning)
    P3(Phase 3: Software Upgrade)
    P4(Phase 4: Applying Required Business Rules Changes)
    P5(Optional: Phase 5: Create New Snapshot)
    P6(Optional: Phase 6: System and Integration Testing)
    P2R(Phase 2 repeat: Environment Cleaning)
    P7(Phase 7: Post-Release Monitoring)
    end
    Z{Rollback}

    P0 --> P1

    P1 -->|Clean environment requested| P2
    P1 --> P3

    P2 --> P3

    P3 --> P4

    P4 -->|Snapshot needed for next release| P5
    P4 -->|SIT requested| P6
    P4 --> P7

    P5 -->|SIT requested| P6
    P5 --> P7

    P6 -->P2R

    P2R --> P7

    P1 -->|Critical issues during Snapshotting| Z
    P3 -->|Critical issues during Software Upgrade| Z
    P6 -->|Optional: Critical issues during Testing| Z

    Z --> |Back to the drawing board| P0

    classDef required fill:#f9f,stroke:#333,stroke-width:2px;
    classDef optional fill:#ccf,stroke:#333,stroke-width:2px;
    class P2,P5,P6,Z optional
    class P0,P1,P3,P4,P7 required

Release documentation

Throughout this document we will make a reference to the release documentation. How this is captured can be decided by the release team, this can be a confluence, a ticket in a ticketing system or a editable part of a release channel in some collaboration tool. The most important requirement is that it can be seen and updated by all people involved by the release and having multiple people to work on the same text without conflicts.

This is an example of a release documentation template:

Date and time: 01-12-2023 09:00-09:30
Version: 23.24.1.25
Environment: ACC
Phases: 0, 1, 4
Expected duration: 30 minutes
Channel: Vertex DTAP Release 
Coordinators:

Changes in this version:
(link to the location of the release notes)
And remarks that are different from normal releases that are found during the preparation or are raised from the release notes for explicit emphasis.

External dependencies:
- Inform team A that they can now enable new feature X after the release
- Ask team B if they received export C before going ahead with the release

Quality gate:
Release is ok on DEV environment by EE
Release is ok by test team on TST environment

GitOps commit for this version:
Commit c916644c: deployment 20231201.52

Release procedure:
(link back to this release procedure)

Phase 0: Preparation

Objective: Check all prerequisites and align the people involved in the actual release.

During the preparation of a release we start by creating by filling the release documentation with all information we already know before actually doing the release. The preparations will ensure that everyone is aligned on the task. This could be done by creating the documentation and having communication over a teams channel or by having a pre-release meeting with the key people involved.

Steps:

  1. Set Version and Environment

    • Role: Project Manager (in project phase) / Vertex Application Administrator(during normal operations)
    • Action: Determine the version of Vertex that needs to be deployed
    • Action: Determine to which environment the release will be deployed
    • Documentation: Capture the release version and environment
    • Details: This step assures that we release the correct version to the correct environment

  2. Provide release notes

    • Role: Software Delivery Manager
    • Action: Collect/write the release notes and publish them in a place where the applicable roles in the release procedure can access them.
    • Documentation: Add a link to the release notes and attach the release notes if applicable
    • Details: With the release notes important information of the new version of the software is communicated. With this documentation different people can prepare for the release.

  3. Decide Release Type

    • Role: Project Manager (in project phase) / Vertex Application Administrator(during normal operations)
    • Action: Determine the nature of the release, considering which phases are optional based on complexity and impact.
    • Documentation: Mark the type of release and the phases that will be executed during the release
    • Details: There are several phases that can be done at the same time as a Vertex software release since we have a maintenance window. These phases are marked as optional phases. Make sure to determine and communicate what phases need to be done in this maintenance window.

  4. Check for external dependencies

    • Role: Vertex Application Administrator
    • Action: Identify external dependencies specific to the release.
    • Action: Check if there are no maintenance windows planned in the broader period that the release is planned to be done.
    • Action: If external dependencies are applicable: capture these dependencies and make sure the extra roles/teams/people required to this specific release are included in the documentation, communication and time reservations for the release window.
    • Documentation: Capture any external dependency if found or mark that external dependencies have been checked but none have been found
    • Details: All normal dependencies are captured in the normal release procedure. This step is to check and identify any additional external dependencies we might have during the release.

  5. Greenlight from Testing Phases

    • Role: Test Team Lead / Key Users (depending on the environment)
    • Action: Ensure that the software version is approved by all necessary testing phases.
    • Documentation: Document that the ok is given to release the determined version to the determined environment
    • Details: Make sure that the version of the software is found to be ok to be released to the determined environment

  6. Determine Release Period

    • Role: Project Manager (in project phase) / Vertex Application Administrator(during normal operations)
    • Action: Consider factors such as resource availability, user impact, and maintenance schedules.
    • Action: Choose an appropriate time frame for the release.
    • Action: Make sure the expected time needed for the release is known
    • Documentation: Add the expected duration of each phase to the documentation
    • Details: This involves coordinating with different teams to find a suitable window that minimizes disruption. The goal is to have as little external factors at the same time of a release as possible. So if a large maintenance is planned to the infrastructure, or the remote desktop used by external parties is undergoing maintenance or if key integrations are having a maintenance window it better to find another window since it would be hard to find an integration error directly after the release if they are down for maintenance at that moment.

  7. Ensure People/Roles Availability

    • Role: Project Manager (in project phase) / Vertex Application Administrator(during normal operations)
    • Action: Confirm that all necessary roles are available during the selected release period.
    • Documentation: Document the people involved in the release and their backups
    • Details: Ensuring that all roles are available during the determined release period and making sure they reserve that time for that point on is crucial for a smooth release. Since the release is a large sequence of small and large actions a wait for one of the roles to become available can have a cascading effect.

  8. Release communication

    • Role: Project Manager (in project phase) / Vertex Application Administrator(during normal operations)
    • Action: Establish primary channels for communication during the release.
    • Action: Identify coordinators for each involved team.
    • Details: Choose a channel in which to communicate and identify the roles that will do the primary communication for a given team. This way we prevent a maze of direct peer-to-peer communication and the process centralized.

  9. User communication

    • Role: Vertex Application Administrator
    • Action: Send out a communication to the users of Vertex to announce the maintenance window in which a new release will be done.
    • Details: Announce to the users when and for how long they expect downtime of the environment. Communicate to the users that on the moment of the release they will receive an additional message that the release is starting. It would be a good moment to highlight some of the big changes in this release to the users and if they can expect changes to their processes and/or new features. Make sure that if the environment is cleaned in phase 3 this is also communicated to the users.

  10. Prepare GitOps Commit

    • Role: Software Delivery Manager
    • Action: Ready a commit in the GitOps repository with the correct Vertex version and application configuration.
    • Action: Ensure the application configuration is updated and accurate, aligning with the new version requirements.
    • Details: This step involves preparing the necessary changes in the GitOps repository so the Vertex components and application configurations are available in as a commit in the repository.

  11. Software Mandated Business Rules Configuration Package

    • Role: Software Delivery Manager & Business Rule Delivery Manager
    • Action: Identify all changes that must be applied together with the software release.
    • Action: Ensure that all software mandated changes are prepared for a fast application after the new software version is online.
    • Details: Note that there is a strong distinction with point 3: these changes need to be done together with the software release because they are breaking changes or a functional migration is required for the software to keep on functioning. This step is optional for most releases.

  12. Business Rules Configuration Package

    • Role: Business Rule Delivery Manager
    • Action: Prepare the business rules configuration release package.
    • Action: Ensure that all changes are included and are in line with the new software features.
    • Details: Here we collect the definitive set of change to the business rules in a single configuration package.

  13. Prepare Rollback Plan

    • Role: Software Delivery Manager
    • Action: Identify the need for a snapshot restore in case of a rollback, or prepare the rollback scripts if sufficient
    • Details: Most of the time rollback scripts are available to rollback to the previous software version but in rare occasions a snapshot restore would be required. If a rollback scenario with scripts is available these scripts need to be prepared by the Software Delivery Manager. The restore from snapshot scenario is always available as backup in case the rollback scripts fail or cause unintended side-effects.

Preparation Phase Checklist

  • Software version and environment to release to is known.
  • Release notes are available
  • Release period is known and resources are reserved for the given period.
  • No conflicting maintenance schedules during the release period.
  • The phases that are done during this release are known.
  • Communication channels and coordinators established.
  • Greenlight from all relevant testing (ex. test team / UAT).
  • All information is captured and available to all people involved.
  • GitOps commit prepared and reviewed.
  • Rollback plan prepared
  • Mandatory Business rules changes are identified and prepared if applicable.
  • Business rules configuration package ready and verified.

NO-GO action

Since we are only in the preparation phase and no change has been done yet we can postpone the release and look for another release window.

Phase 1: Snapshotting / Backup

Objective: Create reliable backup points to safeguard the system, ensuring the ability to restore to a previous state if needed.

Steps:

  1. Notify Stakeholders

    • Role: Project Manager (in project phase) / Vertex Application Administrator(during normal operations)
    • Action: Inform relevant teams and stakeholders about the start of the snapshot process.
    • Action: Make sure to notify anyone that any data done after the snapshot moment will not be available in case of a restore-scenario.
    • Details: Make sure that users are aware of the snapshot moment, because in the worst case when a snapshot needs to be restored that data might not be available anymore.

  2. Create Point-in-Time Snapshots

    • Role: Database Administrator
    • Action: Make comprehensive snapshots of the sql server
    • Documentation: Log all relevant backup and snapshot activities including database names, snapshot names and timestamps of when they where created
    • Communication: Announce when the snapshots are created
    • Details: If continues transactional backups are running for the database and point in time recovery is always available it might not be needed to create a specific snapshot of the database.

Snapshotting / Backup Phase Checklist

  • Stakeholders informed about the snapshot moment.
  • Database snapshots created, verified and logged in the release documentation.

Phase 2: Environment Cleaning (Optional)

Objective: Clean the environment of any residual data from previous testing or operations, ensuring a pristine state for the new release.

The primary users of the environment request a cleanup of their environment. Their reasons might vary but it might be because of:

  • from the ground up testing where they start form a clean environment
  • have clean test cases without interference of old test cases
  • the process has an important time element (time is not easy to fake, especially if there are external systems involved)
  • have completeness testing with external systems, where both systems are cleaned at the same time

Database test data state overview

The clean process is designed to always have a copy of the database that contains only business configuration without any test data.

Steps:

  1. Restore Clean State

    • Role: Database Administrator
    • Action: Use the pre-test data snapshot to restore the database of the environment to a clean state.
    • Action: Confirm the successful and complete restoration of the environment.
    • Communication: Announce the start and end of the restore of the clean state procedure
    • Details: In this step the clean snapshot of a previous release is restored to remove all test data from the current active database. The database administrator makes sure a backup/copy of the current database is still available in the exceptional case that a user was not aware that the data would be cleaned there is still a way to recover that data.

  2. Clean cluster state

    • Role: DevOps engineer
    • Action: Clean the state stores in the cluster
    • Communication: Announce the start and end of the restore of the clean state procedure
    • Details: The cluster itself also has a few state stores like caches and specialized databases. These can be cleaned of their state and rebuild after the restore of the database.

  3. Verify Environment State

    • Role: Vertex Application Administrator
    • Action: Check the database and/or application to verify that the database is restored to the correct version and the configuration that is expected is in the environment
    • Action: Ensure no residual test data or configurations remain from previous testing activities.
    • Communication: Announce when environment is checked and validated to be clean and ok for further steps
    • Details: Verifying that this step is done correctly enables us to make sure the environment is in a correct state before we start any next phases.

Environment Cleaning Phase Checklist

  • Environment restored to a clean state using a pre-test data snapshot.
  • State in the cluster is cleared
  • Environment state verified for release readiness.

NO-GO action

Retry the restore of the clean environment. If not possible roll back to the old state of the application using the created snapshot/copy of the old database.

Phase 3: Software upgrade

Objective: Deploy the new version of the software from a technical point of view.

In this step we will deploy the new version of the Vertex software and application configuration through a GitOps approach. This will include any technical migrations/upgrades to any state store.

Steps:

  1. Execute Deployment

    • Role: Vertex Application Administrator
    • Action: Deploy the new version of Vertex to the designated environment by moving the tag corresponding to the environment to the predefined commit from the release documentation.
    • Action: Monitor the deployment process closely through ArgoCD and Seq logs for any issues or anomalies.
    • Action: Monitor the upgrade jobs in the ArgoCD cluster that do the automated upgrade of the software
    • Communication: Announce the move of the tag as the start point of this step. The end of this step can be announced after the next step.
    • Details: The actual deployment should be executed as per the plan, with continuous monitoring to ensure successful completion.

  2. Post-Deployment Verification

    • Role: Vertex Application Administrator
    • Action: Conduct thorough testing to verify that the deployment was successful.
    • Action: Confirm that the application is functioning correctly in the production environment.
    • Communication: The software release can be communicated from a technical point of view after this verification step is finished.
    • Details: Post-deployment verification is key to ensuring that the new version operates as expected and without disruptions to users.

  3. Apply software mandated business rules changes (if applicable)

    • Role: Business Delivery Manager / Key user
    • Action: Apply the prepared business rule configuration changes that have been prepared as in the release documentation.
    • Details: They are rare, but sometimes we need to do business rules changes to make sure we are compatible with the new software version. Without these changes the software processes will be impacted that is why they are an integral part of the software release phase.

  4. User Notification

    • Role: Project Manager (in project phase) / Vertex Application Administer (during normal operations)
    • Action: Notify users about the completion of the deployment.
    • Action: Provide information on any significant changes or new features in the updated version of Vertex.
    • Details: User communication is important for adoption and adaptation to the new version, ensuring a smooth transition for end-users.

Release Execution Phase Checklist

  • New version of Vertex successfully deployed.
  • Post-deployment verification conducted.
  • Software version mandatory business rules changes are applied.
  • Users notified about the new version.

NO-GO action

If we have to abort a software release we can go to the rollback plan that was planned for the release. These will consist of:

  • Re-applying the old commit that marked the previous version of the software and application configuration in the GitOps repository
  • Running rollback scripts for the database

If this pre-defined rollback scenario fails we will need to do a restore from the snapshot / backup created at phase 1.

Phase 4: Applying Required Business Rules Changes

Objective: Implement the necessary business rules changes that are tightly coupled with the new software release, ensuring seamless integration and functionality.

Note: This includes the business rules changes that are not not mandatory by a software deployment. These can be planned in anytime after phase 3.

  • The environment does not have to be exclusive to the people configuring during this business configuration phase:
    • Almost all releases are soft releases: the software release enables a new feature and after that release we can start using that feature in the business configuration but we don’t have to immediately.
  • Most of the time the business configuration can be fully detached from a software release: the configuration items can be applied to an environment without the need for new software is most of the cases.
  • During initial delivery, software and business configuration are often bundled because at that point a lot of business configuration does need new software features and because a lot of business configuration is delivered at once.
  • Business configuration is documented in deployment configuration documentation where all changes in the business configuration are documented.
  • All business configuration is done through the front-end with named accounts: by design there is no sql injection script that will introduce business configuration into the system anonymously.
  • Import and export features are being created during roadmap development over the next time to package up business configuration in portable configuration files that can be exported from one environment and imported into the next. The person importing the file will be the user accountable for the business configuration.

Steps:

  1. Asses business changes

    • Role: Business Delivery Manager / Key user
    • Action: Asses the business changes and the risk related to the changes, make a decision if you need a maintenance window and a backup as described in phase 2.
    • Communication: At the end of this step communicate that the software release is finished from a business rules configuration point of view.
    • Details: Even if this is not part of a regular release: if you are doing large scale changes with a risk higher than normal operational tasks it is always go to work with a maintenance window and request a backup as described in phase 2 to make sure a rollback scenario is availible if needed.

  2. Implement Business Rule Changes

    • Role: Business Delivery Manager / Key user
    • Action: Apply the prepared business rule configuration changes.
    • Action: Conduct this process carefully to minimize disruption to ongoing operations. Use activation timestamps where possible for a planned go-live of the new configurations where possible.
    • Documentation: Capture the old version of the business rules of any configuration item in the software that is not versioned from the front-end to have a rollback version available.
    • Communication: At the end of this step communicate that the software release is finished from a business rules configuration point of view.
    • Details: The implementation should be smooth and efficient, ensuring that the new business rules integrate well with the new software version and do not adversely affect existing operations.

  3. Snapshot new configuration and run optional integration tests

    • (required if we did phase 3)
    • Communication: Communicate the fact that we need a snapshot and if we are going to run the integration tests.
    • Details: If we did a clean of the environment and we plan to move in the future it is important to do phase 5, possibly continued with 6 and 7.

  4. Monitor Application Functionality

    • Role: Business Delivery Manager / Key user
    • Action: Monitor the output of the processes after applying the changes to see if the changes have the expected results
    • Details: This verification step is essential to ensure that the application, post-update, meets all functional requirements.

  5. Notify End-Users of business rule updates

    • Role: Business Delivery Manager / Vertex Application Administrator / Key user
    • Action: Communicate the changes to the end-users if there are significant changes to the process due to business rules changes.
    • Details: Effective communication helps users prepare for changes in the processes and help them expect different output. It also helps in logging tickets after the changes because users know what functionality has been altered and could help in the analysis of anomalies.

Business Rules Changes Phase Checklist

  • Assessment has been made if we need an explicit backup
  • Business rules changes are implemented in the system.
  • Required if we did step 3: Initiate phase 6
  • Application functionality with new rules verified.
  • End-users notified about the changes.

NO-GO action

Most configuration items in Vertex are versioned, making it possible to rollback to a version of the business rules from before the changes were applied by activating the old version. If there was any change to a business item that does not support versions we re-apply the old configuration that we captured. In case the configuration was not captured we can always look in historic event to find all events for a given business object. This requires a bit more work, but the data is not lost. Note that some items do not apply event sourcing like time series and documents, but we expect that these items are not holding any business configuration.

Phase 5: Create new snapshot after business configuration (optional)

Objective: Create a new snapshot of copy of the database with only configuration data and no test data

Because we want to keep a clean copy of the database with the new configuration but without any test data we want to create a snapshot or copy of the database before we start with testing on that environment. More details about why we want this can be found in phase 3.

  1. Create New Snapshots
    • Role: Database Administrator
    • Action: Take a snapshot of the database
    • Action: Ensure these snapshots represent the updated state without test data.
    • Documentation: Add the database name, snapshot name, datetime of the data in the snapshot to the release documentation.
    • Details: Creating new snapshots after cleaning and updating the environment ensures that there is a current and clean fallback point before any testing is done.

Business Rules Changes Phase Checklist

  • A snapshot of the database has been created, verified and documented.

NO-GO action

If we cannot create a new snapshot we have to make a decision to continue testing but that means we do not have a clean snapshot to start phase 2 with the next time. This means that we lose our snapshot without test data and to get a new environment without test data we need to start from a fresh database with a new business configuration.

Phase 6: System and integration testing (Optional)

Objective: Make sure that the new changes are working correctly by running a set of tests targeted at testing the critical paths of the software and as many integrations as possible.

Some configuration can only be finalized on the dev environment because it is the first environment where we do not use mocks but actual external applications for our integrations. Releasing into TST is the validation that the business configuration is correctly documented in the deployment configuration documentation.

  1. Run a systems and integration test
    • Role: Business Delivery Manager / Key user / Tester
    • Action: Run the test plan for the system and integration testing
    • Action: Verify as much of the critical process as possible
    • Action: Verify as many integrations with external systems as possible
    • Documentation: Collect all test findings and categorize them with a severity
    • Details: This step helps to check if the software version, the application configuration and the business rules configuration are correct.

Business Rules Changes Phase Checklist

  • The systems and integration test has been completed
  • There are no findings or the findings including their severity are not a showstopper for the release

NO-GO action

If there are business configuration error we can perform the NO-GO action of phase 6. If there are blocking software issues we can do the NO-GO action of phase 6 and phase 3.

Phase 7: Post-Release Monitoring

Objective: Monitor the newly released software for stability and performance issues, gather user feedback, and ensure a seamless transition to the new version.

Steps:

  1. Continuous System / Application Monitoring

    • Role: Vertex Application Administrator
    • Action: Use the monitoring to monitor application logs, system load and error rate
    • Details: Continuous monitoring is crucial to quickly identify and address any post-release issues that may impact system processes or user experience.

  2. Performance Reporting

    • Role: Vertex Application Administrator
    • Action: Compile performance reports based on monitoring data and user feedback.
    • Action: Present these reports to stakeholders to provide insights into the release’s success and areas for improvement.
    • Details: Performance reports help in evaluating the overall impact of the release and guide future development priorities.

Environments

This is a small appendix that explains the different environments and what their task is:

  • DEV: continually new software versions to prepare and test the new release with actual external systems
  • TST: validate the business configuration deployment configuration and do a single large regression test for the main processes
  • ACC: key users test new software functionality and business configuration before it hits the production environment

Release time

There are a few remarks when it comes to release time:

  • Business configuration effort will scale with the number of business configuration items that need to be configured
  • Software releases and business configuration can be separate most of the time. The dependency on new software features will decrease over time.
  • ‘A release’ can vary between environments because they serve different purposes.
  • Release to accept is most close to a production release
  • Depending on preference we can group things together to bring a large package of features to an environment together or we can do small steps over time.
  • Our typical production release at this moment is about 15 minutes in the evening where the environment would be performance impacted for a little bit during the release. Business configuration is then done the next business day during normal operation.

2 - Troubleshooting

This is a troubleshooting guide page to diagnose and fix some known issues in Vertex

Troubleshooting

This document outlines the steps to diagnose and resolve known issues in Vertex that may arise during regular use.

Session issue

Gathering information about problematic sessions is crucial for diagnosing and resolving issues. The collected data will be used to implement troubleshooting steps, helping us identify the root cause and find a solution.

Resolving steps

  1. Mitigation Step: Users should perform an explicit logout and re-login in Vertex to generate a new session identifier. Simply closing all tabs and reopening Vertex will attempt to resume the existing session, which might lead to the same issue if the session is problematic.
  2. Retrieve all available session and user information using the /gateway/action/userinfo endpoint from the problematic session.
  3. Check the user and user sessions at the front-end level.
  4. Session Monitoring: Monitoring sessions that are functioning correctly is valuable, as it helps us prevent session issues from occurring in the first place

3 - Seq procedure

This contains a set of procedures and know-how on how to perform certain tasks on Seq or to troubleshoot some common issues

Reset Seq authentication

When configuring authentication it’s possible to end up locked out, either because of Seq configuration problems, external provider issues, or loss of network connectivity.

To recover a Seq instance in this state, the command line can be used to switch back to username/password authentication, and set a password for an administrative account.

Delay the start of Seq

Because you need to modify the metadata store of Seq you need to delay the start of Seq. Seq will get an exclusive lock on the metadata files and changing them is no longer possible.

  1. Terminal into the Seq container
  2. Create a file sleep.sh in the /data/Init folder with the following content
#!/bin/bash
sleep 120
  1. Call the top
  2. Press k to kill a process
  3. Hit enter to kill PID 1, which is the Seq process. Press enter again to send the killsig to the process.
  4. The container will now restart and after the restart you can terminal into the container again and you have 3 minutes before Seq starts

Rest the user account

  1. Perform the Delay the start of Seq procedure
  2. Before Seq can start execute the following command:
seq auth reset -u <username> -p <new password>

Note: make sure that the username you reset is the username of the admin account, this could be the new email address you provided when configuring the new authentication provider, if you are recovering from that scenario. 3. After the sleep period is over, Seq will start and you can login with these new credentials. You do need to change the password on the first login.