Search the Aria Knowledgebase for
User Documentation, APIs, SDKs, and more!



 

Home > Aria Crescendo Documentation > Data Streaming > Data Streaming FAQ

Data Streaming FAQ

This article applies to:Aria Crescendo

 

 

  1. Do you turn on individual modules for the data stream or do you get all objects?

For the load stream, when you put in a customer support ticket, you can pick which objects you want. For the change stream, you currently get all the objects. An object subscription UI is in the roadmap for later.

  1. How is last id used for the data stream connection?

If there is a client side interruption for any reason, such as an internet failure or a client processing failure, the client script should use the last_id to reconnect as shown in the example code. More info on last event id here  https://www.w3.org/TR/eventsource/.

  1. How should the data be consumed and loaded into a database once received?

Clients should create the data model in their database based on our Logical Object Model. Once the data model is ready, the client should write a script to consume the data (sample perl script) and populate the database based on which Entity they are reading from the stream.

  1. How secure is the Streaming connection? It looks like an http connection and not an https connection.

The stream service URLs always uses HTTPS protocol. Data is encrypted over SSL in the normal way.

  1. Can you provide examples of the full JSON message? I see payload examples but no header detail. 

Here’s a sample full JSON payload. The first ‘data’ is the header and second ‘data’ is the payload.

data:{"ref":{"type":"Account","ids":{"account_no":21119957,"userid":"43739988"}},"tstamp":"2020-06-08T19:53:30+02:00","tick":29580691950}
data:{"account_no":21119957,"userid":"43739988","client_account_id":null,"balance":0,"status":{"type":"Account_Status","ids":{"status_no":1,"enum":1}},"status_tstamp":"2121-12-31T19:53:32+01:00","parent_account":null,"created_tstamp":"2121-12-31T19:53:32+01:00","updated_tstamp":"2121-12-31T19:53:32+01:00","secret_question":null,"secret_question_answer":null,"is_test_account":false,"is_invoice_approval_required":false,"currency":{"type":"Currency","ids":{"currency_cd":"usd","code":"usd"}},"updated_receipt_id":null,"created_receipt_id":null,"sequence_functional_group":null,"start_date":"2121-12-31","admin_contact":{"type":"Contact","ids":{"contact_no":23436896}},"is_consumer_account":false,"locale":{"type":"Locale","ids":{"locale_no":10001}},"legal_entity":null,"is_anonymized":null,"notify_method":{"type":"Notify_Method","ids":{"method_no":2,"enum":2}},"functional_account_groups":[{"type":"Functional_Account_Group","ids":{"group_no":10095894,"client_group_id":"FA_-_DE"}}],"collection_account_groups":[],"account_fields":[{"account_field":{"type":"Account_Field","ids":{"field_no":22776,"field_name":"supp_field_1587439747321"}},"field_values":["Account_Field_value_1"]}],"tax_exemption_level":null,"notification_overrides":[],"notification_template_group":null,"surcharges":[],"revrec_profile_number":null,"purchase_order_number":null,"cc_email_addresses":[],"bcc_email_addresses":[]}
event:create
id:H4sIAAAAAAAAAKtWMjRUsjI0MTA0Ma8FAG50BJUNAAAA
  1. Is tick the event id?

The timestamp and tick in the SSE event headers identify when the data change happened. The tick is simply a tie breaker if two events come in at the exact same timestamp. The smaller tick represents the earlier event.  

  1. How do you handle when the stream fails or times out? How do you handle monitoring?

The stream service simply drops the connection and waits for the next request. For client failures, the client script should used the last event id to reconnect. 

  1. How are supplemental fields handled in the data model?

Supplemental fields are included in the stream. Below are two samples.

Account Fields

{
  "account_no": 21119729,
  "userid": "uid_1591636971086",
  "client_account_id": "caccid_1591636971086",
  "account_fields": [
    {
      "account_field": {
        "type": "Account_Field",
        "ids": {
          "field_no": 22782,
          "field_name": "supp_field_1587439754010"
        }
      },
      "field_values": [
        "1"
      ]
    },
    {
      "account_field": {
        "type": "Account_Field",
        "ids": {
          "field_no": 22784,
          "field_name": "supp_field_1587439756560"
        }
      },
      "field_values": [
        "1"
      ]
    }
  ]
}

Transaction Supplemental Fields (known as Optional Transaction Qualifiers)

{
  "transaction_no": 159241559,
  "transaction_type": {
    "type": "Transaction_Type",
    "ids": {
      "type_no": 9,
      "enum": 9
    }
  },
  "created_tstamp": "2114-06-24",
  "source_no": 381465,
  "account": {
    "type": "Account",
    "ids": {
      "account_no": 21112250,
      "userid": "uid_1591377970950",
      "client_account_id": "caid_1591377970950"
    }
  },
  "currency": {
    "type": "Currency",
    "ids": {
      "currency_cd": "usd",
      "code": "usd"
    }
  },
  "amount": 10,
  "applied_amount": 0,
  "fully_applied_tstamp": null,
  "statement": null,
  "master_plan_instance": {
    "type": "Plan_Instance",
    "ids": {
      "plan_instance_no": 25211656,
      "client_plan_instance_id": "25211656"
    }
  },
  "api_receipt_id": null,
  "transaction_supp_fields": [
    {
      "field_name": "aapm_Qualifier_name_1591377973588",
      "field_value": "aapm_Qualifier_value_1591377973588"
    }
  ],
  "voided_transaction": null,
  "voiding_transaction": null,
  "applications": [],
  "comments": "Balance forward recorded during account creation for plan instance via API"
}
  1. How do you know the order of the events? Will the event order be guaranteed?

Event order is guaranteed only within an account. Please refer to our best practices if ordering matters to you.

  1. What is the retry window on a failure? Is there something that will resend a missed event?

When there is a client failure, the client application should use the last_event_id to start restreaming. The client app can reconnect and restream from a given event_id as often as it wants up to the 48 hour retention period.

  1. What is the unique ID or an object? Is it the object ID and tick?

The unique ID of an object is its ‘ref’.  Every Data Streaming SSE event has a header line in its data field. That header line contains the ‘ref’ of the object impacted by that event, right next to the timestamp and tick. SSE event_ids (last event ids) are solely for reconnecting to the SSE server is case of client or connection failure or if  you want to get the data again.  It plays no part at all in identifying data objects.

  1. What are some possible values for the ‘from’ query parameter?

The ‘from’ query parameter takes values like 1day, 2hour, 20minute etc.


Subtopics 

Data Streaming Overview
Data Stream Client & Server-Sent Events (SSE)
Consuming Aria Data Streams
Data Streaming Authentication
Data Streaming Logical Object Model
Data Streaming JSON Representation
Example Data Streaming Code
Data Streaming Frequently Asked Questions

Last modified

Tags

This page has no custom tags.

Classifications

This page has no classifications.