Home > Aria Crescendo Documentation > Data Streaming > Consuming Aria Data Streams

Consuming Aria Data Streams

Using Aria's Data Streaming Web Service involves three key components of stream consumption:

  1. Getting your SSE client up and running
  2. Loading a snapshot of Aria data events via the Load Stream
  3. Consuming near-real-time Aria data events via the Change Stream
  4. Planning for data continuity

Getting Your SSE Client Up and Running

You must contact Aria to turn Data Streaming on. When that happens, you will have endpoint information and can obtain an authorization code to connect to the Change Data Stream. Then, you can begin writing your SSE client script. Several pieces of example code, written in Python, are available to get you started on this process.

When you are satisfied that you have the scripts necessary to consume Aria data in the desired manner, you are ready to consume the Load Data Stream.

Loading a Snapshot of Aria Data

Events in the Load Stream all have the event type "load" and represent a current snapshot of a specified subset of Aria client data, such as "all account data," "all financial data," or "all usage data." To receive the Load Stream, you must submit a request to Aria Customer Support to launch an "extraction job" to generate a Load Stream with the desired data. 

It is a good idea to coordinate both the Load Stream, and the Change Stream described below, to ensure that you capture all existing and modified Aria data.

Consuming Near-Real-Time Aria Data

Events in the Change Stream describe changes to Aria data in near-real-time. Each of these SSE events is identified as one of three event types: "create," "update" or "delete." These events map to an Aria data entity identified by the header (first) data line in the event. Update events include updated fields related to the data entity and may omit fields that have not changed. 

Planning for Data Continuity

If you plan to start using Aria's Data Streaming Web Service and already have a live instance of the Aria billing platform, your plan should ensure that you consume—

  • a snapshot of existing data via the Load Stream
  • live data events that occur while consuming the Load Stream (which can take up to several hours) via the Change Stream

Coordinating the Change and Load Streams

The Load Stream captures a snapshot of the data in your Aria instance, from the data existing when the Load Stream is started to the moment the latest event has been consumed. When run against a live environment, the Load Stream will—by definition—not capture events after that latest event (which, in a live environment, will likely occur after you initiate the Load Stream).

If you do not run your the Change Stream concurrently, there can be data gaps. As a best practice and to avoid these data gaps, Aria recommends following this procedure: 

  1. Open an SSE client for both streams.
  2. As you receive load events, save that data to a local database, together with the load event's timestamp and tick found in the event header (the first of the two data lines).
  3. As you receive change events, check if the entity described by that event has been loaded in a load event yet. If it has, update your local copy accordingly.
  4. Once the last load event is received, you will have a full copy of the Aria dataset that corresponds to the time of that event. 

coordinating_data_streams.png

How can I determine the order of events received in the two different streams?

  • You cannot compare times based on when your client app receives the events: Transmission speeds and the speed your client consumes events can vary.
  • Events must be ordered based on the timestamp and tick and their event header. If both have the same time stamp value, use the tick as the tiebreaker.

Outages

Both the Change and Load Streams may be rerun as many times as you like to respond to connection outages, or to consume the data in a different way. Rerunning a stream will provide the same events with the same data.

The only constraint of Aria's Data Stream is that, because events are retired from the stream when their time limit elapses, you can only re-consume events back to the retention limit of 48 hours. 

Last modified

Tags

This page has no custom tags.

Classifications

This page has no classifications.