What is Event Sourcing?

Software applications have historically relied on database tables to track their current state, keep it persistent, and share it with other applications. This approach has been successful for many years, although it comes with its disadvantages. Any changes that have been prior made to the current state of data, unless logged elsewhere, becomes lost. To address this, a carefully architected tracking database known as Event Sourcing has become the updated alternative to tracking and maintaining an application's state. It records state changes, corresponding changes, as well as a chronological sequence of events, while providing additional details on how and why the transition of data has occurred. In comparison to traditional methods that only keep the latest version of an application's state, Event Sourcing allows the developer to record and recreate previous states from the event logs.

Developers use Event Sourcing to obtain information about an application's current relationship with the outside world, a common requirement in software development. Furthermore, it is also useful to know how an application achieved its current state. Event Sourcing allows developers to change an application's disposition automatically to account for retroactive changes as opposed to traditional transaction logging. The traditional transaction method states the transition but is absent of why the transition occurred.

Overview

The primary aim of Event Sourcing over older methods of tracking state changes is that it retains every change with an event object which stores these events in the sequence in which they occurred. This in effect allows developers to understand the manner of how the state changed over the application's lifetime.

To illustrate the way traditional software handles state changes, I will be utilizing examples in this article provided by Martin Fowler. Fowler explains this by asking his audience to imagine an application that tracks movements of ships along the oceans. He suggests that the least complicated way to track these ships is if the shipping company used a tracking application in order to express the ships departing and arrivals. The simplest way to do this is to record when the shipping company’s ships leave or arrive a port as shown below:

Figure 1: Interface for tracking shipping movements

When the application is executed, it updates the current event information for each of the company's ships. For example, if the most recent event is a departure, then the ship may be bound for its next destination. The use of Event Sourcing adds a further step to this process by creating an event object to record the state changes for each ship, as shown in the following figure:

Figure 2: Capturing a state change with an event

The flowchart above seems to show an unnecessary level of indirection from a strictly processing point of view, but it records every state change instead of just the most recent event. Assume for the example that the application in figure 2 records the following events:

● The King Roy departs San Francisco

● The Prince Trevor arrives at Los Angeles

● The King Roy arrives in Hong Kong

In comparison, the application in figure 1 only shows the most recent events for each ship, which include the following:

● The Prince Trevor arrives at Los Angeles

● The King Roy arrives in Hong Kong

The figure below shows the information retained by an application that doesn't use Event Sourcing.

Figure 3: State after tracking without Event Sourcing

Notice how the figure above shows that the event of the King Roy departing San Francisco is no longer retained by the application. While ship objects are persistent for both types of applications, shipping events are only persistent with the application that uses Event Sourcing. The figure below shows how an application that uses Event Sourcing retains all shipping events:

Figure 4: State After Tracking with Event Sourcing

Retaining all the events instead of the most recent one allows the developer to see and log where the ship has been as well as where it's going. However, this difference isn't much of an advantage on its own. The developer could also attract the history of ports for each ship by recording that information in the ship object or a log file each time a ship moves or has moved. A benefit Event Sourcing initiates is that it can warrant event objects to initiate changes to the domain objects. This advantage provides an event log with additional capabilities, including event replay, temporal query and complete rebuild.

An event replay may be necessary when they recorded past event is incorrect. The application can correct the event log by removing subsequent event in reverse order, replacing the incorrect blog entry and re-creating the ensuing events. This technique can also correct an event log when the application receives events in the wrong sequence, which often occurs within asynchronous messaging.

The ability of Event Sourcing to determine an application's state at any point in its history allows developers to perform temporal queries which is vital in determining the applications current disposition. This is a process that begins with a blank application state and rerunning events up to any given desired point. Developers have the ability to expand on this capability by handling multiple timelines at once, which is similar to branching in a version control system.

Another facility that can be built on the event log would be a complete rebuild. A complete rebuild is when entire application state becomes discarded, and you start with an empty application. From this point, a developer can then recreate the application state history from scratch by using the events within the event log. In this case, Subversion would often perform a complete rebuild of an application state in order to dump and restore repository files.

Storing Application States

To simplify, understanding Event Sourcing usage would be to calculate the summoned application state when it starts from a blank state. Following the blank state, the following events would be applied in order to reach an optimal state. This process is known by developers to be time-consuming depending on the number of events that need to be applied. Because timing is often in high demand, applications tend to request the most recent application state. Another technique for minimizing the time that Event Sourcing requires, is to store the applications current state and in the meantime allowing developers to build additional capabilities in addition to the current state.

Since storage requires only the event log, developers can store application states on permanent storage media or memory. One common approach for applications in use during the business day is to begin the storage procedure at the beginning of the day from a snapshot taken during overnight processing. The Event Sourcing software can then maintain the current application state in memory and replay events from the overnight event log in the event of a crash. The system administrator can then take a new snapshot of the application stay at the end of the business day. Event Sourcing allows the magistrate to take multiple snapshots in parallel without bringing down a running application.

The system of record for Event Sourcing can be the application's current state or event logs. However, holding the logs in a database may result in them, but only being available for audits and other types of special processing. Administrators can also use the event logs as the official records and build database tables from the logs as needed.

Event Handling

Developers can implement the logic for handling events in a numerous variety of ways, with the most common choices being a domain model or transaction scripts. To elaborate, a domain model is better for complex logic while transaction scripts are better catered for simple cases. The general tendency among developers is to use transaction scripts for applications that use commands or events to drive changes. Often, developers believe that this way of structuring event handling is the only option, but that isn't necessarily the case.

The best way to view this issue is to consider Domain Modeling and Transaction Scripting as fulfilling separate responsibilities. Domain Modeling, a process that develops essentially a web of interconnectivity, provides the business logic that manipulates the application. In turn, Transaction Scripting is the organization of logic simply as a single procedure to perform for a particular incoming event. Developers can combine these two functions, but they can also separate them by placing the selection logic in the event processing. This logic can then call a method in the domain model to process the domain logic.

Once a developer has made the decision about implementing event handling logic, the next step is to decide whether to place the selection logic in the event object itself or a separate event processor object. The primary drawback of using a processor object is that the logic and executes depends on the event, which is the type of switching that object-oriented (00) programmers try very hard to avoid. Most programmers will therefore opt to place the selection logic in the event itself, assuming all other factors are equal.

One case in which the other factors aren't equal occurs when the event object is a Data Transfer Object (DTO) that's automatically serialized and deserialized. This situation prevents the developer from implementing code in the event itself, making it necessary to use a separate processor. The developer also needs to locate the appropriate selection logic for the event, which further complicates event handling. One solution to this problem is to treat the DTO as a hidden data folder, while still treating the event as a polymorphic object. This approach may require additional logic to match the DTO's serialized events to the actual events, typically through the use of appropriate naming conventions or configuration files.

A domain model doesn't need to have any knowledge of the event log if it doesn't need to reverse events. If the domain model does need to reverse events, it's far more likely that it will need to be aware of the event log. Reversing events complicates the domain model because it then needs to store and retrieve prior states.

This process is most straightforward when the event is in the form of a difference rather than a fixed value. For example, adding $100 to an account can be reversed by simply subtracting $100 from that account. On the other hand, setting the balance of the account to $100 is more difficult to reverse because you don't have the information needed to determine the account's past value. An application that doesn't follow the difference approach for input events must therefore store the information it needs to reverse the event during processing. This requirement generally involves storing previous values before changing them.

The consequences of storing previous values are particularly significant when the processing logic uses the domain model. The reason for this general rule is that the domain model can change its internal state in ways that aren't visible to the event object. The domain model therefore needs to be aware of events, so that it can use them to store previous values. It's also possible to reverse events by reverting to a previous snapshot and replaying the event string, so reversals are an absolute requirement in Event Sourcing. However, they can be more efficient in cases where reversal involves fewer events than playing events forward.

Updates

Currently, most systems are not using the Event Sourcing tool which can be particularly difficult when the users and non-users interact. Usually, the correspondence between the two systems becomes difficult as an event sourcing application when attempting to send modifier messages to other systems as well as receiving queries. The primary advantage of replaying events at will is largely negated when events send messages to external systems. In these cases, the messages can have a tendency to fail because the external system can't distinguish between a real event and a replay. In order to conduct a solution to this issue, the developer must undergo the task of wrapping the external systems within a gateway, which is typically a good idea in any case. However, this gateway must be more sophisticated than usual because it must deal with the event-sourcing system's replay processing.

The application can usually deal with rebuilds and temporal queries by simply disabling the gateway during replay processing. However, it must do this in a way that doesn't require changes to the domain logic. For example, the domain logic should always call gateway methods, even if it isn't processing a replay. The gateway can then handle the distinction between real events and replays by referring to the event processor to determine which type of events before calling a method in an external system. External updates become even more complicated in the case of retroactive events.

Another way that event-sourcing applications often deal with actual systems is buffering external notifications by time. In some cases, like end-of-month processing, the application can make an external notification at some point in the future instead of making it immediately. These applications have the luxury of reprocessing events more freely until the time that it must notify an external system of those events. Common ways of handling this scenario include using the gateway to store external messages until the release date. Another solution delaying actual updates is to trigger the generation of external messages with a notification domain event.

Queries

External Queries can return results that affect event handling, which presents a problem for Event Sourcing. For example, let’s assume that an application requests a particular exchange rate on January 1st from an external system which now constitutes and event. I then need that application on January 15th to replay the event, meaning I will need the exchange rate from January 1st.

Some external systems may be able to provide past data by requesting a value for a particular point in history. If this is possible, and you trust the data, then you can use it to ensure a consistent replay of events. We may also find it possible to use Event Collaboration amongst these collaborative components. This would essentially require the application to simply retain the changes in historical components.

The solutions to the problems caused by external queries become more complex when the above possibilities don't exist. One approach is to modify the gateway to the external system so that it records the responses to its queries, which you can then use during the replay events. If the data from the external system changes slowly, it may only be necessary to remember events that change values. Otherwise, the gateway will need to remember the response to every external query.

Queries and updates to external systems complicate Event Sourcing, especially when interactions with those systems involve both actions. This situation often occurs when an external call returns a query result and a state change for the external system. A common example of this scenario is submitting an order for delivery to the external system, which returns delivery information for that order.

Code Changes

This discussion of Event Sourcing has thus far assumed that the application processing events won’t change, which obviously isn't the case with real software. Code changes may be classified into three broad categories, including new features, fixes and changes to temporal logic.

New Features

New features are changes that add additional capabilities to the application without invalidating existing code. The procedure requires that application to process the vents again in order to provide new results. Developers can generally add these changes freely at any time, allowing users to take advantage of them with old events.

When reprocessing that involves new features occurs, the default behavior should typically initiate the external gateways to turn off unless new features involve these gateways anyways. New features that involve these gateways are an obvious exception to this general rule. Even in this case, it may not be desirable for the external system to notify the application of past events. If it is, the developer will need to provide special handling for the first reprocessing of old events. This action shouldn't be necessary for subsequent reprocessing.

Fixes

Bug fixes are changes to existing code to correct a problem. This becomes an ideal tool when considering internal fixes. In order to simply fix these bugs, you just initiate the fix and reprocess the events which correct the application state.

This is not to claim that gateways are perfect. In fact, they come with their own set of complex behaviors. The gateways themselves must trace the distinctions between events that take place with the bug and those that function without them. The outlook to this issue is similar to that used for retroactive events. It may be worthwhile to replace events by using the retroactive event mechanism, typically when there is a large amount of reprocessing to perform. However, this solution requires the event to reverse the bug event as well as the correct one.

Temporal Logic

The temporal logic of applications often changes over time, especially those that use Event Sourcing. This type of logic is related to time, such as “charge $10 before January 15 and $20 afterwards.” The domain model is the best place for temporal logic, since it should be able to execute events with the correct rules at any time. Developers can also write this logic in the application by using conditional statements, but the code quickly becomes complex with this approach as the amount of temporal logic increases. A better solution in this case is to associate strategy objects with temporal properties, as shown in the following pseudo code:

chargingRules.get (aDate).process(anEvent)

Fixing bugs and temporal logic may have some overlap in cases where old events need to be reprocessed. This situation could possibly require the application to engage in bi-temporal behavior. This means that the application must reverse events for a certain time period according to the rules in place at that time and replace them with according to the current rules. This approach gets complex very quickly, so developers shouldn't attempt it unless they have no other choice.

Developers can handle some of this complexity by using adaptive object models to embed code in the data. Another way to accomplish this is to use a directly executable programming language that doesn't require compilation. Embedding JRuby into a Java app is a common example of this approach, but it requires a greater effort to ensure proper configuration control over the resulting code. One way to do this is to handle changes to the processing scripts through an event.

Use Cases

Strong use cases for Event Sourcing include scalable architectures, accounting systems, as well as debugging.

Scalable Architecture

Event Sourcing is a requirement for parallel models and retroactive events, which are difficult to retrofit into a system not originally designed with Event Sourcing in mind. Therefore, you should build Event Sourcing into an application from the beginning if you think there's a reasonable chance it will eventually require Event Sourcing at some point in the future.

Scalable architectures initiate some interesting possibilities for Event Sourcing, which are becoming increasingly common cases for this capability. Event-driven scalable architecture includes many specific concepts, but they generally involve the communication of event messages. These systems can operate in a parallel style where events are loosely coupled to the transmission of messages. This design can provide effective horizontal scalability and a high resilience with respect to system failures.

One example of this would be an application with multiple readers and few writers. Developers could make such an application scalable by carrying out clusters of databases in memory in which the application could keep synchronized with event streams. It could then route updates to a single master system that could take the form of a small cluster of servers around a single message queue or database.

The application could then apply updates to their system of record before broadcasting the resulting events to a large cluster of readers. This architecture can be highly beneficial even when the system of record is an application state stored in the database. There are also possibilities for delivering high-performance when the system of record is an event log, which requires a minimal locking due to being a strictly additive structure.

The primary drawback of this scalable architecture is that the reader systems can easily become unsynchronized with each other due to differences in their timing of event propagation. Nevertheless, this type of architecture is still highly popular with scalability and is a critical requirement. This use of event streams also allows developers to easily add new applications by populating separate event models, which don't need to be identical. This strategy is particularly effective for using a messaging approach to system integration.

Accounting Systems and Events

Accounting systems provide some particularly good examples of Event Sourcing, as the synergy between them is strong. The auditing capability that Event Sourcing provides is especially important with accounting systems remains the primary reason for implementing Event Sourcing in these systems.

Developers can create code in which accounting entries create the accounting consequences for the domain events, which they can then link to the original event. This design provides an effective basis for tracking event changes and reversals in addition to simplifying other techniques for adjusting events. Developers who implement Event Sourcing for accounting systems should view account entries as a log of all events that change an account's value, making the account a type of event source.

However, many developers oppose the practice of packaging each application change as an event, as they find it awkward. As a result, developers typically require some type of return before they will consider this type of application design. One of the most significant returns is that it makes it easy for developers to make an audit log, which is so important in accounting systems.

Audit logs also allow customer service staff to correct accounts easily, since transaction history is now an integral part of the application. This capability makes the audit trail visible to support staff, who can now step through a customer's transactions. While developers can accomplish this by performing traditional logging mechanisms more frequently, it's easier to implement in Event Sourcing once they become familiar with it.

Debugging

The complete audit files that are possible with Event Sourcing are also helpful for debugging. Developers have traditionally used log files to debug production applications, but Event Sourcing goes further by allowing them to precisely create a test environment. They can then replay events into that environment to determine what happened, along with the ability to stop, rewind and replay events. This capability is similar to the way developers execute tests in a traditional debugger, which is especially useful when performing the parallel testing needed in order to implement an upgrade in production. Developers can thus use Event Sourcing to replay actual events on the test system in order to ensure it returns the expected results.

Examples

Examples of event-sourcing applications include the following:

● Tracking ships

● External queries

● Reversing an event

● Updating an external system

Tracking Ships

The shipping application previously discussed also provides more complex examples involving Event Sourcing. Recall that this domain model consists of ships carrying cargo between ports. The four types of events affecting this model include the following:

● A ship arriving at a port

● A ship leaving a port

● Loading cargo onto a ship

● Unloading cargo from a ship

Arrivals and Departures

Arrivals and departures are the most important event since they provide information about the ship's location. Each of these events has a simple process method that captures the required data for that event and forwards it to an appropriate domain object. In this example, the event performs the selection logic, while the ship performs the domain logic.

An arrival event sets the ship's port to the port it just arrived at, while a departure event sets the ship's port to a special case. The developer has a choice to make about how to pass event information to the domain object. One option is to pass only the data that the domain object needs to process the event, but the developer can also pass the event itself. Passing the event means that it doesn't need to know what data the domain logic actually requires. As a result, there's no need to update signatures if additional data is added to the event later. The disadvantage of this approach is that the domain logic must now have an awareness of the event.

Loading and Unloading Cargo

This example of using domain logic to handle the loading and unloading of cargo determines when cargo has passed through a particular country. Many reasons exist for tracking the countries that a particular cargo is passed through, as each country has its own regulations on importing and exporting products. Assume for this example that a virulent disease originating from Country X makes it important to know which cargo has passed through that country.

Ships don't always move cargo directly from its origin to its destination, so shippers must routinely unload cargo from one ship and load it onto another one to make the most efficient use of multiple ships. The cargo is only in danger of exposure to the disease when it's in port, as it's safe while the ship is traveling between ports. The arrival event must track the movement of cargo, which the Ship object will handle. It will then pass the arrival notification to the Cargo object, since the Ship object isn't responsible for tracking the cargo's movements.

An alternative approach to tracking cargo is to put the domain logic in the event processing method, requiring it to contain knowledge of the domain model. The domain object then passes the event to the appropriate objects, so they can perform their respective tasks with the event information. This option requires the developer to maintain the event processing method to reflect the increasing complexity of the domain logic, making it less desirable in cases where the domain logic will change frequently.

External Query

External queries are always awkward when building event-sourcing applications because rebuilding the application state requires the developer to use responses to external queries that were made in the past. Consider the case when a ship must determine its cargo's value when it enters a port, which is performed by an external service. Assume for this example that a ship enters port on January 1, which the tracking application processes immediately. The external service then determines the cargo's value as of January 1. Rebuilding the application state on February 1 requires the application to know the cargo's value on January 1, even if its value has changed by February 1.

One way of handling this problem is to maintain a log of external queries and the responses, which the developer can then use to obtain the desired values when reprocessing an event. This approach is similar to the one used to perform an external update, meaning the application converts the interaction into an event and uses the event log to determine the cargo's value at a particular point in history.

The next step is to pass the event to the appropriate domain object, just as in previous examples. In this case, the Cargo object calls the external system for the cargo's present value and saves that value in an event log. The application would then do something useful with that value at some point in the future. In many cases, the developer will encapsulate all access to an external system through a gateway that the application controls. That gateway typically has the sole function of translating the external call into the form needed to perform the interaction. The developer can also support event replays by wrapping the gateway within a class that logs the queries.

In the following figure, as provided by Fowler, the control flow is illustrated for a shipping process example:

Figure 5: Wrapping the pricing gateway within a logging gateway

In the above example, the logging gateway examines the outward call to observe if an older request will match the current request. However, if the requests don’t exist, the gateway will convert the request into an event object in effect, execute the external query, then store it in the log. The gateway can then retrieve query results by submitting the request type to a domain event. This process provides a result set that requires further observation, depending on the gateway. The request log for these queries must be persistent in the same way in the log for domain events, since it is needed to rebuild the application state.

Reversing an Event

The critical part of reversing an event is ensuring that you can accurately calculate the object's prior state when an event changes that state. The event itself is often a good place to store this data, which works particularly well in the shipping example. The approach in this example is to pass the event to domain objects, which can easily store information in that event.

The event for loading cargo illustrates this process. The ship object generates the cargo loading event, which it uses to store Event Sourcing data. It then hands the event off to the cargo object, which stores the cargo's prior port. Reversing this event requires the developer to add a reverse method that reflects the actions of the process method. The domain object can then call the reversal method to reverse an event.

This example shows that the event contains changeable data on prior events as part of its processing data. The fields in this example would be quite simple, although more realistic applications would require a more complex data structure. The cargo object tracks whether the cargo has been to Country X when it handles an arrival event by using a simple Boolean field that contains a true or false value within the entry. This equips properties that are interchangeable at any time depending on the need. However, storing the prior port on the event requires another data structure since an arrival can affect multiple cargoes. One possible solution to this requirement is a map that is indexed by the cargo.

This example of cargo tracking shows how the event's source data and its error handling affect the way in which the application performs an event reversal. The application must store the port that the cargo was in from the time it was last loaded within the load event, provided the event doesn't already include this information in its source data. Note how adding a small amount of source data to the event object eliminates the need to add data for prior events in this case. However, this benefit doesn't apply to all event handling. For example, the arrival event can't determine whether a cargo's prior port was in Country X.

The processes behind handling an event can affect the optimal possibility to perform an event reversal. The shipping example includes both arrival and departure events, which should always alternate for a given ship. In other words, it always arrives at the port and the parts from it before arriving at the next port.

It's therefore trivial to reverse an arrival event because all the application has to do is set the ship's port field to indicate that it's out to sea. Two consecutive arrival events shouldn't happen in this simple example, so the second arrival should be flagged as an error. In this case, it wouldn't be necessary to store the ship's prior port.

Updating an External System

The ability to easily reprocess events is considered one of the most ideal benefits of Event Sourcing. However, reprocessing events also increases an application's interaction with external systems, which can be bothersome to administrators when you start spamming their systems. The easiest way to avoid this outcome is to make external system calls through gateways that are configured to prevent messages from going out to external systems unless the application is processing a real event.

A simple example involving ships and ports illustrates this approach. Assume that a ship must notify the local customs authority when it enters a port, which is handled by the domain logic for the event processor. This code merely invokes the gateway object's notification method and shouldn't care whether it's processing an event replay or a real event. In general, domain logic shouldn’t be concerned about the context of the events it processes.

The example demonstrates the way the gateway is responsible for determining if it should actually send a message to the external system. It can accomplish this by creating a link to the event processor and checking to see if it's active. The event processor then activates itself when processing regular events.

This example may seem basic; however, it demonstrates fundamental principles of an event-sourcing application interacting with an external system. Gateways decide whether to send a message to the systems rather than the domain logic. This decision is primarily based on the information the gateway has about the context of the event processing. The Boolean state of true or false should be sufficient to accomplish this goal in this case.

Summary

The ability to architect a structure that is equipped with the ability to record state changes is the most significant benefit of Event Sourcing compared to traditional techniques for recording event logs. It is this modern tool that can help contribute to software maintenance and cater to changing market and technology conditions.The ability to track and implement the correspondence of events which prompts an application’s update, essentially provides a representative log of transactions in a chronological manner. As previously mentioned, developers can also use this programming technique to analyze an application's interaction with external systems, which is an increasingly important requirement in modern software development. By fostering an effective test environment, Event Sourcing has become an ideal tool for auditing logs to assist in debugging, analyzing retroactive events and parallel models, as well as other aforementioned modes of special processing. To sentiment on the earlier points of this article, the idea of Event Sourcing allows developers to manipulate or recreate an application's previous states. This process includes replaying events and correcting event errors, as well as structuring a chronological logging system that is able to track and organize multi ecosystems of tracking data.

About Baytech

Baytech is passionate about the technology we use to build custom business applications, especially enterprise solutions that optimize business processes. We’ve been delivering software solutions in a variety of technologies such as Event Sourcing since 1997. Our success is due to the skill and efficiency of our senior staff, which includes software engineers, project managers and DevOps experts. All of our engineers are onshore, salaried staff members.

We focus on the quality, usability and scalability of our software, and don’t believe in mitigating cost at the risk of quality. We manage project costs by implementing an efficient development process that’s completely transparent and uses the latest standards and practices to build software right the first time. Contact us today to learn more about Event Sourcing and how it can help your business. Find us online at https://www.baytechconsulting.com/contact.