State stores: databases and persisted volumes
Vertex uses a set of different backends to store its state.
Object store
The object store is used for storing (large) unstructured data, referred to as objects. These objects can contain any kind of data, referred to as BLOBs (Binary Large Object).
The object store can be used for long term storage of data partitions, can be used to store messages and documents and can be used for storing and serving models.
Microsoft SQL Server
Our current default read-model: because of high familiarity with most developers and BI departments Microsoft SQL Server is chosen as our default read model. The means that most of our domain state is projected into the SQL Server to be shown in screens or exposed to data tools of the customer.
Postgres
A database that is more suitable for running in cluster or provides a lower costs PaaS solution for relational databases.
Roadmap item
Research will be done if we can leverage more power of postgres since to support our read-models. Because we use a generic data layer switching to postgres could be done on a large scale with relative ease.Apache Druid
Apache Druid is an open-source, real-time analytics and OLAP (Online Analytical Processing) database. It serves as an important component for real-time data ingestion and analysis. This means that queries will include the very latest data.
Data can be ingested from various sources including the data streams that are provided by Vertex. Once ingested Druid helps with fast querying and exploration of large datasets. It can be used to do large scale calculations in a distributed performant way and it is a great datasource to provide the data for data driven dashboards.
Because Druid can also handle a lot of time series data it plays an essential role in data intensive components like portfolio management and analysis.
InfluxDB
A datastore that is specifically design to handle lots of time series.
Roadmap item
InfluxDB will be replaced by TimescaleDB in the future. Build as an extension on the proven PostgresDB it provides a more mature and flexible target for time series data.TimescaleDB
TimescaleDB is a time-series database extension for PostgreSQL. It’s designed for handling time-series data efficiently, making it suitable as a specialized time series store within Vertex. TimescaleDB offers advanced indexing and compression techniques, enabling high-performance data storage and retrieval for time-series data.
It can seamlessly scale vertically and horizontally to accommodate growing workloads. With features like automatic sharding and distributed hyper-tables, TimescaleDB efficiently distributes data across nodes or instances, ensuring optimal resource utilization and high availability. This scalability makes TimescaleDB well-suited for cloud-based applications that require flexible and efficient data handling.
Apache Kafka
Apache Kafka is a distributed streaming platform that’s used for building real-time data pipelines and streaming applications. It’s designed for high-throughput, fault-tolerant, and scalable data streaming, making it a good fit for use between the different components in the Vertex application.
Kafka is used to exchange events between the different components, it is used to give fire-and-forget / fire-once,consume-many pattern of events where event-driven asynchronous processes are created with a message on a Kafka Topic as the end of the responsibility of a process/service and a defined start point for a new process/service.
Kafka is also used in many enterprise integration patterns, where Vertex can have incoming and outgoing event-driven/asynchronous integrations with external applications/parties.
Because Kafka acts as a persisted ledger it can be used to store messages / events for a short period of time, meaning that producing an event and consuming an event can be split in time. This means that even when a component is not online or busy at a given point it can consume the event left by another service at a later point in time.
Redis
Redis is an open-source, in-memory data store and cache. It serves as a crucial component for high-speed data storage and retrieval. Redis is used by various components in the Vertex landscape to have a cluster-wide distributed key-value store for caching and storing live cluster state shared between components. Data in Redis is not intended for long term storage. A loss of data in Redis can be recovered by restarting Vertex.
Backup and Recovery
All infrastructure (IaC) and the EDP solution stack is code on git. We can launch a new environment relatively quickly. This excludes data obviously. Our strategy to prevent disaster and to recover from disaster is: Azure Region Pair for disaster recovery purposes. If one of the regions were to experience a disaster or failure, then the services in that region will automatically failover to that regions’ secondary region in the pair. Minimizing downtime with High Availability. Backups: PostgreSQL (i.e. TimescaleDB) comes with the tools you need to perform backups, incremental/continuous backups, and do point-in-time-recovery from backups. Archiving WAL (Write Ahead Log) files is a fundamental operation that facilitates these features. We put these files in an object store. Apache Druid metadata store snapshotting, segments in an object store. Persist Kafka messages in an object store.