Software Development

Portfolio Architecture Examples – Data Engineering Collection

For a few years now we’ve been working on a project we have named Portfolio Architectures. These are based on selecting a specific use case we are seeing used in the real world by customers and then finding implementations of that case using three or more products from the Red Hat portfolio.

Figure 1: The portfolio architecture process

This basic premise is used as the foundation, but many aspects of open source are included in both the process and the final product we have defined. There is a community, where we share the initial project kickoff with a group of architects and use their initial feedback from the start. We also present the architecture product we’ve created right at the end before we publish to ensure usability by architects in the field. The final publish product includes some internal only Content around the customer projects researched, but most of the Content is open and freely available through various open source channels. 

This article is sharing an overview of the product we’ve developed, what’s available to you today in our architecture center, and concludes by sharing a collection of architectures we’ve published.

Introduction

The basis of a portfolio architecture is a use case, two to three actual implementations that can be researched, and includes the use of a minimum of three products. This is the ideal foundation for a project to start, but we encountered a problem with use cases containing emerging technologies or emerging domains in the market. To account for these we’ve chosen to note the fact that these are opinionated architectures based on internal reference architectures. 

The product has been defined as complete for publishing when it contains the following Content:

  • Short use case definition
  • Diagrams – logical, schematic (physical), and detail diagrams
  • Public slide deck containing the use case story and architecture diagrams
  • Internal slide deck containing both the public deck Content and the confidential customer research
  • Video (short) explanation of the architecture
  • Either a technical brief document or one or more articles covering the solution architecture

Note that the items in italics are all available to anyone  in the Red Hat Portfolio Architecture Center or in the Portfolio Architecture Examples repository.

Figure 2: Logical diagram design template

Tooling and workshops

The progress towards our products required a good idea of how we wanted to diagram our architectures. We chose to keep them very generic and simple in style to facilitate all levels of conversation around a particular use case without getting bogged down in notational discussions. 

A simple three level design for our architectures was captured by using logical, schematic, and detail diagrams. All of these have been integrated in open source tooling with pre-defined templates and icons for easily getting started. Furthermore, we’ve developed a tooling workshop to quickly ramp up on the design methods and tooling we’ve made available. It’s called Designing Your Best Architectural Diagrams, has been featured in several conferences around the world.

Data engineering collection

The collection featured today is centred around architectures leveraging data engineering concepts and tooling. There are currently eight architectures in this collection and we’ll provide a short overview of each, leaving the in depth exploration as an exercise for the reader.

Figure 3: Healthcare architecture collection

In each of these architecture overviews you’ll find a table of Contents outlining the technologies used, several example schematic diagrams with descriptions, and a link in the last section to open the diagrams directly into the online tooling in your browser.

Cloud to edge

This architecture covers the use case of providing a consistent infrastructure experience from cloud to edge and enabling modern containerized applications at edge.

(Note: this project is a new architecture and currently in progress, so sharing one of the schematic architecture diagrams and you can monitor this project for updates as it progresses to completion.)

This use case is bringing cloud like capabilities to the edge locations.

Data center to edge

This architecture covers the use case around data center to edge where the energy (utility) infrastructure companies operate across vast geographical area that connects the upstream drilling operations with downstream fuel processing and delivery to customers. These companies need to monitor the condition of pipeline and other infrastructure for operational safety and optimization.

The use case is bringing computing closer to the edge by monitoring for potential issues with gas pipelines (edge).

Edge manufacturing efficiency

The manufacturing industry has consistently used technology to fuel innovation, production optimization and operations. Now, with the combination of edge computing and AI/ML, manufacturers can benefit from bringing processing power closer to data. This helps actions be taken faster on things like errors and predictive maintenance.

The use case is for boosting manufacturing efficiency and product quality with artificial intelligence and machine learning out to the edge.

Edge medical diagnosis

This architecture covers edge medical diagnosis in the healthcare industry. It Accelerates medical diagnosis using condition detection in medical imagery with AI/ML at medical facilities.

The use case is accelerating medical diagnosis using condition detection in medical imagery with AI/ML at medical facilities.

Intelligent Data as a Service (iDaaS)

Intelligent DaaS (Data as a Service) is about building and delivery of systems and platforms in a secure and scalable manner while driving data needs for moving towards consumerisation in healthcare. Feel free to explore this portfolio architecture by clicking on the diagram below.

The use case is Intelligent Data as a Service (iDaaS) is about building and delivery of systems and platforms in a secure and scalable manner while driving data needs for moving towards consumerisation in healthcare.

Payments

An offering of (near) real-time payments lets businesses, consumers, and even governments send and accept funds that provide both availability to the recipient and instant confirmation to the sender. Enabling real-time – or at least faster – payments that improve the speed of online payment experiences to customers has the potential to give banks a greater opportunity to win, serve, and retain their customers. By building solutions that capture real-time payment business, banks also can drive higher payment volumes, ideally at lower costs as well as engage new customer segments.

The use case examines financial institutions enabling customers with fast, easy to use, and safe payment services available anytime, anywhere.

Retail data framework

Retail is the process of selling consumer goods or services to customers through multiple channels of distribution to earn a profit. A data framework refers to the process of managing enterprise retail data. The framework or system sets the guidelines and rules of engagement for business and management activities, especially those that deal with or result in the creation and manipulation of data.

The use case is creating a framework for access to retail data from customers, stock, stores, and staff across multiple internal teams.

SCADA interface modernisation

This Portfolio Architecture targets energy providers in North America that need to be compliant with NERC regulations and in order to achieve this they decide to modernise the interfaces between their business applications and their SCADA systems also for a better consumption of information that can be used in combination with AI/ML and decision management tools to better address customer needs.

The use case is providing interfaces with SCADA systems that are compliant with NERC regulations, creating different layers of API gateways to protect business service depending on the network zones.

If you are interested in more architecture solutions like these, feel free to explore the Portfolio Architecture Examples repository. More architecture collections include:

Published on Java Code Geeks with permission by Eric Schabell, partner at our JCG program. See the original article here: Portfolio Architecture Examples – Data Engineering Collection

Opinions expressed by Java Code Geeks contributors are their own.

Eric Schabell

Eric is Chronosphere's Director Technical Marketing & Evangelism. He's renowned in the development community as a speaker, lecturer, author and baseball expert. His current role allows him to coach the next generation of technical marketers & evangelists helping the world to understand the challenges with cloud native observability. He brings a unique perspective to the stage with a professional life dedicated to sharing his deep expertise of open source technologies and organizations. Follow on https://www.schabell.org.
Subscribe
Notify of
guest

This site uses Akismet to reduce spam. Learn how your comment data is processed.

0 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Back to top button