Home > Aria Crescendo Documentation > Data Feed Web Service > Consuming Aria Data Feeds

Consuming Aria Data Feeds

Using Aria's Data Feed Web Service involves three key components of feed consumption:

  1. Getting your SSE client up and running
  2. Loading a snapshot of Aria data events via the Load Feed
  3. Consuming near-real-time Aria data events via the Change Feed
  4. Planning for data continuity

Getting Your SSE Client Up and Running

You must contact Aria to turn Data Feed on. When that happens, you will have endpoint information and can obtain an authorization code to connect to the Change Data Feed. Then, you can begin writing your SSE client script. Several pieces of example code, written in Python, are available to get you started on this process.

When you are satisfied that you have the scripts necessary to consume Aria data in the desired manner, you are ready to consume the Load Data Feed.

Loading a Snapshot of Aria Data

Events in the Load Feed all have the event type "load" and represent a current snapshot of a specified subset of Aria client data, such as "all account data," "all financial data," or "all usage data." To receive the Load Feed, you must submit a request to Aria Customer Support to launch an "extraction job" to generate a Load Feed with the desired data. 

It is a good idea to coordinate both the Load Feed, and the Change Feed described below, to ensure that you capture all existing and modified Aria data.

Consuming the Aria Data Feed

Events in the Change Feed describe changes to Aria data in near-real-time. Each of these SSE events is identified as one of three event types: "create," "update" or "delete." These events map to an Aria data entity identified by the header (first) data line in the event. Update events include updated fields related to the data entity and may omit fields that have not changed. 

Planning for Data Continuity

If you plan to start using Aria's Data Feed Web Service and already have a live instance of the Aria billing platform, your plan should ensure that you consume—

  • a snapshot of existing data via the Load Feed
  • live data events that occur while consuming the Load Feed (which can take up to several hours) via the Change Feed

Coordinating the Change and Load Feed

The Load Feed captures a snapshot of the data in your Aria instance, from the data existing when the Load Feed is started to the moment the latest event has been consumed. When run against a live environment, the Load Feed will—by definition—not capture events after that latest event (which, in a live environment, will likely occur after you initiate the Load Feed).

If you do not run your the Change Feed concurrently, there can be data gaps. As a best practice and to avoid these data gaps, Aria recommends following this procedure: 

  1. Open an SSE client for both streams.
  2. As you receive load events, save that data to a local database, together with the load event's timestamp and tick found in the event header (the first of the two data lines).
  3. As you receive change events, check if the entity described by that event has been loaded in a load event yet. If it has, update your local copy accordingly.
  4. Once the last load event is received, you will have a full copy of the Aria dataset that corresponds to the time of that event. 

coordinating_data_streams.png

How can I determine the order of events received in the two different feeds?

  • You cannot compare times based on when your client app receives the events: Transmission speeds and the speed your client consumes events can vary.
  • Events must be ordered based on the timestamp and tick and their event header. If both have the same time stamp value, use the tick as the tiebreaker.

Outages

Both the Change and Load Feeds may be rerun as many times as you like to respond to connection outages, or to consume the data in a different way. Rerunning a stream will provide the same events with the same data.

The only constraint of Aria's Data Feed is that, because events are retired from the feed when their time limit elapses, you can only re-consume events back to the retention limit of 48 hours. 

Last modified

Tags

This page has no custom tags.

Classifications

This page has no classifications.