Extending static entities — Part 2
In the first part of this series, we looked at how to create an OutSystems component that uses a static entity, but still allows a developer to extend it with custom values. For example, we were able to define a custom A3 paper size to be used with Ultimate PDF, without changing the component itself.
Today we continue from where we left in Part 1. If you haven’t read it yet, absolutely go read it first. In particular, you should be familiar with how we extended the PaperSize static entity by subtyping it.
In case you don’t know what subtyping means, and you have no time to read Part 1, then here’s a brief summary. You can set the identifier of a static entity to have a data type referencing another static entity, just like you would do to create a foreign key. This creates a relationship between the two identifier types, such that they are compatible and almost interchangeable. That’s how we created a new A3 PaperSize record without having to change the PaperSize static entity itself.
With this technique, however, Ultimate PDF doesn’t have any knowledge about this new paper size. If Ultimate PDF were to query the PaperSize static entity, for example to display a list of all supported paper sizes, the A3 record would not be found by the query.
So far we avoided this problem altogether, because one of the rules mentioned on Part 1 said that the static entity should never be queried by the component. But rules can be broken. In this article, we’ll finally lift this restriction, enabling a richer integration between a component and its consumers.
Designing an auditing component
For the remaining of this article, we’ll focus on an example that is more interesting and more challenging than the PaperSize extension.
Imagine we are building an e-commerce application, and we want to define an auditing component, which will keep track of several events that might happen in our app:
- Add product to cart
- Checkout initiated
- Checkout completed
Easy! We define these events in an EventType static entity, and a normal AuditLog entity to store all events.
In order to log a particular event, we simply define a LogAuditEvent action that receives the EventTypeId as input parameter, and this action can be used by the e-commerce application when a particular event happens.
Another requirement comes along. The auditing component should automatically delete old events, based on the following:
- Events of type “Add product to cart” or “Checkout initiated” should be deleted after 7 days.
- Events of type “Checkout completed” should never be deleted.
It’s not too difficult to add this retention information to each record, by creating two attributes, ShouldDeleteOldEvents and DaysToDeleteOldEvents, and use these attributes in a cleanup timer that executes daily.
A few weeks later, on a second project, the same need arises and you think of reusing this auditing component. But now the application is an inventory management, and the events that must be tracked are:
- Order placed
- Order fulfilled
To support these new event types, you change the static entity and add two more records. And with that, we have broken our architecture!
Can you guess what is wrong with this architecture?
We have introduced a new dependency, and now the auditing component depends on every application that uses it. Although this is not quite a traditional architecture violation, I claim that the auditing component is now strongly coupled with both applications. To implement a new event type, you have to modify both the component and the application. This is analogous to having a cyclic reference between the two modules. And even though the dependency we’re talking about is not visible in any tool such as Discovery or AI Mentor, it’s real and it affects the work of developers trying to understand and change the system.
Depending on how you view it, we can also argue that there is a side dependency between the e-commerce and inventory management applications. The inventory management application, by having a reference to EventType static entity, will be made aware of the event types from the e-commerce application. This shouldn’t happen, as both applications should be completely independent.
Finally, there’s also a scalability problem. As the number of consumer applications grows, this static entity will also grow, to the point that it will become unmanageable.
Partitioning a static entity across applications
Let’s experiment with defining the event types inside of their respective applications. This way, we will avoid all the architectural concerns that we have highlighted before.
We can define an EcommerceEventType static entity inside the e-commerce application, and an InventoryEventType inside the inventory management application. We can relate all of these static entities by having them subtype the EventType.
Since the LogAuditEvent action receives an EventType Identifier, it will also accept any of its subtypes. In the e-commerce application, we can invoke LogAuditEvent with the AddProductToCart value, even though the auditing module does not know anything about the AddProductToCart event type.
But recall that the auditing module needs to know about the event types, so that it can implement retention policies for each of them. Our solution so far hasn’t addressed this problem. But to solve this last remaining piece of the puzzle, we’ll need to think outside of the box.
Registering new records
Instead of storing the event types in static entities, let’s use a normal entity for that. The auditing component will expose an entity to store all event types, and consumer applications will register new records on it.
We won’t abandon static entities completely, though. We still have a place for them in our solution. The event types of each application will still be defined by static entities. However, we will guarantee that every event type is also stored in a central entity owned by the auditing component. Whenever the auditing component needs to query the event types, it will do so by querying this new entity.
First, let’s change all static entities to have a text identifier. We need to ensure that all identifiers defined in all applications are globally unique. We don’t want two event types to conflict just because they were accidentally created with the same identifier. The easiest way to guarantee a globally unique identifier is to use a GUID, and just to make it clear in our data model, let’s rename the Id attribute to GUID.
You may notice a warning when you use a text literal for the GUID values. Don’t worry, it’s just a warning, and as much as I would like it to be gone, I couldn’t find any way to get rid of it without compromising this solution. I have created a community idea to try to get rid of this warning. But while it is not implemented, just ignore or hide the warning.
Note that the EventType static entity doesn’t need any records or attributes. The event type information will be stored in another entity anyway. We will call this new entity RegisteredEventType.
The AuditLog entity should be updated to reference the RegisteredEventType entity, instead of EventType directly. The foreign key should enforce the referential integrity, so that a consumer application is forced to register the event type before using it. That’s how we guarantee that the auditing component has full knowledge of every possible event type.
To make it easy for a consumer application to register an event type, we can expose a server action RegisterEventType. Then a consumer application can simply invoke it inside of a timer that executes on publish. Whenever we change the static entity inside of the e-commerce application, and publish the change, it will execute the timer, which will propagate the changes into the RegisteredEventType entity.
Finally! Looks like we found a solution! We’re not quite done yet with the article, but let’s briefly recap where we arrived.
We created a generic auditing component, that stores a log of events. It can be used in many different applications, yet it has no specific knowledge of any of those applications. Each consumer application can define the types of events it needs to track in a static entity, and register these event types before tracking them. This can be accomplished without changing the auditing component, or any other application for that matter, ensuring that the parts of the system are isolated and can evolve independently.
Optimising database performance
So far we have used a GUID for all identifiers, which is typically a column of 36 characters. This means that the foreign key in the AuditLog entity will also be a column of 36 characters.
Does this feel to you like a waste of database space, using 36 characters to reference a table that only has 5 rows? It certainly does to me. Even worse, index lookups over the foreign key in AuditLog will not perform very well due to the random and sparse nature of GUIDs.
We can improve by noting that the RegisteredEventType entity doesn’t actually need to have a GUID identifier. It can still have an auto-number identifier, and store the GUID in a separate attribute. With this change, the foreign key in AuditLog becomes an integer, saving database space and improving index lookups.
We can still ensure that GUID values are unique, by creating a unique index over that separate attribute.
Advanced use cases
Filtering events by application
Let’s say we want to have a list of all events in the e-commerce application. Easy! All of that information is stored in AuditLog table, so we can just create a list screen based on that.
However, keep in mind that the AuditLog also contains information from other applications, namely the inventory management. We don’t want to present those events in the e-commerce application, right?
How can we filter AuditLog to only list the e-commerce events?
You can do that by joining 3 entities in a single aggregate. I’m sure you can figure out how that works from the following picture.
Joining the AuditLog with the static entity can also be useful to enhance the list in many ways. For example, we could store metadata such as colour or icon information as attributes in EcommerceEventType, and joining allows us to easily read and use that information in a list screen. All of this is achieved without modifying the auditing component.
Tracing back which module defines each event type
Depending on the complexity of your architecture, you might decide to split the event types between multiple modules. For example, if you have a module focused on a checkout process, you might define the CheckoutInitiated and CheckoutCompleted events directly on that module, separated from the AddProductToCart event.
This works fine without any modifications to our pattern. You can partition the event types however you like. Simply define a CheckoutEventType static entity, and just don’t forget the timer to register these event types.
However, at some point we might lose track of which module registered which event type. We’re not storing any information in the RegisteredEventType entity to be able to trace back to the module that defined each event type.
We can keep track of that by adding a new RegisteredFromEspaceId attribute in the RegisteredEventType entity. We can use the GetEntryEspaceId() function to automatically populate this information inside of the RegisterEventType action.
Having this information, we also enable a few nice use cases. We could produce automatic documentation that lists all event types per module or per application. We could also build a portal dashboard, that summarises how many events happened in the last 30 days, grouped by application, giving an overview of the system usage across applications.
And I’m sure you can come up with more interesting use cases, by taking advantage of the platform’s meta-model entities.
Managing business applications
This need can manifest itself in different ways, but it always involves managing some information related to each application, usually by an IT Administrator or similar role. A few examples that I have come across in recent times:
- Managing user access for each application. An administrator needs to allocate users to each business application on the factory, and also perform some use cases such as reset password, grant temporary access, and change privileges. Some applications can have their own administrator role, which would be able to access the user management console, but scoped only to the users allocated to that application.
- Defining policies for each application. On a factory that hosts multiple business applications, each of them containing its own privacy policy, terms and conditions, usage policy, etc, it might make sense to have a central backoffice to manage all of them.
- Licensing of features within each application. Each business application can define a set of features, and a central backoffice would grant licensing limits to each feature for a particular tenant. For example, an inventory management application might have a feature that is Create Orders, and a tenant might purchase a license limited to 100 orders per month.
In all of these examples, the concept of business application does not align very well with an OutSystems application. In fact, it is quite common of a business application to be composed of multiple OutSystems applications. This makes it difficult to rely on the OutSystems metamodel to accomplish these requirements.
We also don’t want to couple any of these solutions with each application being managed.
A much better solution is to define the concept of BusinessApplication, and allow each application to register itself once published, with all of the metadata required by the component. For example, the user management solution would require each registered application to also define which roles can be managed. The licensing management would require the application to also define which features are exposed for licensing.
A similar approach is taken by the Case Management Framework, which requires each application to register itself before it can use its functionality. We’ll see this in more detail in the next section.
Real-world examples
Webhooks Producer and OAuth2 Provider
These are the two forge components where I first implemented this pattern, back in 2019.
Webhooks Producer allows a consumer application to define events, and trigger these events by using the TriggerWebhookEvent action. The component includes a backoffice where you can define a webhook URL and decide which events it subscribes to. When the event is triggered by the consumer application, all webhooks that subscribe to that event are automatically invoked asynchronously.
OAuth2 Provider allows consumer applications to define scopes. The applications can check if a given authenticated user was granted access to a particular scope via the action CheckScope(). The component includes a backoffice where it’s possible to define clients, and which scopes are to be granted to each client upon successful authentication.
Case Management Framework
To my surprise, Case Management Framework uses a very similar pattern to the one we’ve been describing. It relies heavily on defining static entities on each consumer application, advocates for using GUIDs to globally identify each record, and has a timer that executes on publish that registers it all by invoking the SetupCaseManagementApplication service action.
There is one difference, though, which is that Case Management Framework doesn’t use subtyping on its static entities. This means that CaseDefinitionConfiguration Identifier, as defined by the picture above, is a new type defined in the consumer application, and is not compatible with any type known by Case Management Framework.
This leads to an overuse of type casting by having TextToIdentifier() almost everywhere. The problem could have been avoided by having the consumer application subtype the static entities of the component, which would create compatible types between the component and the consumer.
OutSystems incompatibility
Unfortunately, with platform versions 11.12 or higher, the static entity partitioning no longer works. Consumer modules will fail to publish with “object reference not set to an instance of an object”.
The issue seems to be related to using a literal string on a foreign key to a reference static entity. We partition the event type static entity to be able to define new records in the e-commerce application, that do not exist in the EventType static entity itself. This situation leads to this error during compilation. Presumably, the compiler is attempting to find the referenced record in EventType, and is not prepared to the situation where it might not exist.
As far as I can tell, this was a regression introduced in 11.12. I have reported the issue in many support cases on different projects, and it’s resolution being tracked by RPM-1480.
There is no ETA for resolution, and the current workaround is to not use subtyping, by removing the foreign key on the identifier of each partitioned static entity. However, this will lead to data type warnings appearing all over the place, because the consumer data types are no longer compatible with the component. But we have to live with this workaround until the issue is fixed.
The Webhooks Producer component and demo is currently broken due to this issue. Thankfully the OAuth2 component can still be published, because I hadn’t used the idea of subtyping at the time. We also have several affected components at the company I work for, Phoenix DX, including an auditing component that served as inspiration for this article.
Since Case Management Framework doesn’t use subtyping, it is not affected by this issue.
EDIT: OutSystems has confirmed that there’s no plan for fixing this issue. So the best we can do unfortunately is to use type casting by using TextToIdentifier(). I have raised an idea in the community to see if an alternative solution can be made available by OutSystems: https://www.outsystems.com/ideas/13170/create-type-safe-generic-components-similar-to-case-management-framework/
