Data Streams

Documentation

Data Streamers Functionality

General Information

1NCE offers a service called “Data Streams”. The Data Stream service allows to subscribe to real-time Event and Usage Data for all SIM Cards by pushing data directly to the customer server or an already integrated cloud service such as AWS Kinesis, DataDog or Keen.io. In total up to ten different Data Streams can be configured per account.

 

Configuration of Data Streams

Setting up a Data Stream on the configuration tab in the 1NCE Customer Portal requires the customer to select the Stream Type to specify if just Event or Usage Data is desired, or both should be sent jointly.

The customer can select the transmission of historic data. If historic data is selected the 1NCE Platform will send all customers data up to seven days old, however it will take significantly longer until transmissions are caught up with live data.

An activated and configured stream will be enabled instantly, and the performance can be monitored. The current Remote Status is shown in the Customer Portal as HTTP status coded from the remote site. If a 200 (OK) is displayed, the Data Stream is operating normally. In case the receiving section is not reachable, or the configuration is invalid it will show a 500er error.

Customers can pause and resume the stream at any time by using the action buttons next to the stream details.

For setting up the Data Streams for a RestAPI or RestAPI in Bulk mode the specification of an API Callback is required. The 1NCE Data Stream acts as an HTTP Client and needs an HTTP Server to handle each Post. Every sent message includes an array of one or more objects.

 

Event Data

To have a better understanding of Event Data, the following steps will provide a detailed view on the properties of Event Data and some typical samples for better understanding on a practical level.

 

Properties of Event Data

Event Data are always generated by the SIM carrying device as soon as changes appear, like a change of location, or in case of any error or disturbances. This gives an indication to the customer of tentatively irregular events or the need to take any action. To provide an overview all possible event types and properties are listed as follows.

1. Overview of Event Types

 

Id Description
0 Generic
1 Update location
2 Update GPRS location
3 Create PDP Context
4 Update PDP Context
5 Delete PDP Context
6 User authentication failed
7 Application authentication failed
8 SIM activation
9 SIM suspension
10 SIM deletion
11 Endpoint blocked
12 Organization blocked
13 Support Access
14 Multi-factor Authentication
15 Purge Location
16 Purge GPRS Location
17 Self-Signup
18 Threshold reached
19 Quota used up
2. Generic Event Data

The following properties are included in all events sent:

Property_______ Format_______ Description
id Numeric Unique identifier of this event, if multiple events with same id are received (e.g. due to transmission errors) these should be treated by the receiver as duplicates
timestamp Timestamp Date/time when this event happened
event_type Nested Object Type of the event, see below for details
event_severity Nested Object Severity of the event, see below for details
event_source Nested Object Source of the event, see below for details
organisation Nested Object Organization associated with this event, see below for details
alert Boolean Event is a candidate to be alerted to a user
description String Human readable description of the event
3. Optional Properties

Event types related to specific SIM Cards or User include the following additional properties:

Property_______ Format_______ Description
imsi Nested Object Details of IMSI, see below for details (in a multi-IMSI
sim Nested Object Details of SIM, see below for details (in case of multi-IMSI configuration multiple different IMSIs may be reported for the same SIM)
endpoint Nested Object Details of Endpoint, see below for details
4. Properties of Details Object
Property Format Description
id Numeric Unique identifier of the actual used mobile network operator
name String Name of the mobile network operator
country Nested Object Country of mobile network operator
country.id Numeric Unique identifier of the country
country.name String Name of country
country.country_code String Country code
country.mcc String Mobile Country Code (MCC)
country.iso_code String ISO code
pdp_context Nested Object PDP Context Details
volume Nested Object Volume consumed in PDP Context
volume.rx Number with up to 6 decimal places Downstream Volume in MB
volume.tx Number with up to 6 decimal places Upstream Volume in MB.
volume.total Number with up to 6 decimal places Total volume
5. Properties of PDP Context
Property Format_______ Description
pdp_context_id String Unique identifier of this PDP context
tunnel_created Timestamp Date/time when this PDP context was created
gtp_version String GTP Version, 1 or 2
ggsn_control_plane_ip_address String IP Address of GGSN/PGW Control Plane
ggsn_data_plane_ip_address String IP Address of GGSN/PGW Data Plane
sgsn_control_plane_ip_address String IP Address of SGSN/SGW Control Plane
sgsn_data_plane_ip_address String IP Address of SGSN/SGW Data Plane
region String Region where Data Plane is located
breakout_ip String IP Address used for Internet Breakout
apn String Access Point Name (APN)
nsapi Integer  Network Service Access Point Identifier (NSAPI)
ue_ip_address String IP address assigned to Endpoint
imeisv String IMEISV
mcc String Mobile Country Code (MCC)
mnc String Mobile Network Code (MNC)
lac Integer Location Area Code (LAC)
sac Integer Service Area code (SAC)
rac Integer Routing Area code (RAC)
ci Integer Cell Identification (CI)
rat_type Integer Radio Access Type (RAT) (1=3G, 2=2G, 3=WLAN, 4=GAN, 5=HSPA+, 6=4G, 8=NB-IoT)
6. Properties of IMSI Object
Property_______ Format_______ Description
id Numeric Unique identifier of this IMSI
imsi String International mobile subscriber identity (IMSI)
import_date Timestamp Date/Time this IMSI was provisioned
7. Properties of Organisation Object
Property Format Description
id Numeric Unique identifier of this organisation
name String Name of organisation
8. Properties of User Object
Property Format Description
id Numeric Unique identifier of this user
username String Username e.g. e-mail address
name String Realname of user
9. Properties of Endpoint Objects
Property Format Description
id Numeric Unique identifier of this endpoint
name String Configured name of this endpoint
ip_address String IP address assigned to this endpoint
tags String Tags assigned to this endpoint
imei String International mobile equipment identity (IMEI)
10. Properties of SIM Object
Property_______ Format_______ Description
id Numeric Unique identifier of this SIM
iccid String Integrated Circuit Card identifier (ICCID) without checksum digit
msisdn String MSISDN
production_date Timestamp Date/Time this SIM chip was produced
11. Severity Levels
Id Description
0 INFO
1 WARN
2 CRITICAL
12. Event Source
Id Description
0 Network
1 Policy Control
2 API

Event Data JSON Sample

1. Update Location Event Sample

{
“id”: 2013707XX,
“alert”: false,
“description”: “New location received from VLR for IMSI=’9014301234567891XXXX’, now attached to VLR=’4917200130XX’.”,
“timestamp”: “2017-10-26T07:28:00.000+0000”,
“event_type”: {
“id”: 1,
“description”: “Update location”
},
“event_source”: {
“id”: 0,
“description”: “Network”
},
“event_severity”: {
“id”: 0,
“description”: “Info”
},
“organisation”: {
“id”: 839921,
“name”: “Demo Company”
},
“endpoint”: {
“id”: 8638726,
“name”: “GPS Tracker”,
“ip_address”: “100.96.234.249”,
“tags”: null,
“imei”: “357762083301XXXX”
},
“imsi”: {
“id”: 205672,
“imsi”: “9014301234567891XXXX”,
“import_date”: “2016-12-27T10:09:23.000+0000”
},
“sim”: {
“id”: 274887,
“iccid”: “898830300123456XXXX”,
“production_date”: “2016-12-27T10:09:23.000+0000”
},
“detail”: {
“id”: 3,
“name”: “Telekom”,
“country”: {
“id”: 74,
“name”: “Germany”,
“country_code”: “49”,
“mcc”: “262”,
“iso_code”: “de”
},
“tapcode”: [{
“id”: 2,
“tapcode”: “DEUD2”
}],
“mnc”: [{
“id”: 3,
“mnc”: “02”
}]
}
}

2. Delete PDP Context Event Sample

{
“id”: 2013707XX,
“alert”: false,
“description”: “PDP Context deleted.”,
“timestamp”: “2017-10-26T07:27:59.000+0000”,
“event_type”: {
“id”: 5,
“description”: “Delete PDP Context”
},
“event_source”: {
“id”: 0,
“description”: “Network”
},
“event_severity”: {
“id”: 0,
“description”: “Info”
},
“organisation”: {
“id”: 839921,
“name”: “Demo Company”
},
“endpoint”: {
“id”: 8427408,
“name”: “GPS Tracker”,
“ip_address”: “10.194.50.XX”,
“tags”: null,
“imei”: “898830300123456XXXX”
},
“imsi”: {
“id”: 372566,
“imsi”: “9014301234567891XXXX”,
“import_date”: “2017-03-15T21:46:01.000+0000”
},
“sim”: {
“id”: 319318,
“iccid”: “898830300123456XXXX”,
“production_date”: “2017-03-15T21:46:01.000+0000”
},
“detail”: {
“id”: 48,
“name”: “SFR Cegetel”,
“volume”: {
“rx”: 0.012671,
“tx”: 0.01148,
“total”: 0.024151
},
“pdp_context”: {
“mcc”: “310”,
“tunnel_created”: “2017-12-11T05:49:29”,
“ggsn_control_plane_ip_address”: “185.57.216.XX”,
“pdp_context_id”: “162094787”,
“imeisv”: “898830300123456XXXX”,
“region”: “eu-west-1”,
“lac”: 40484,
“sac”: 61142,
“rat_type”: 1,
“gtp_version”: “1”,
“ue_ip_address”: “100.105.197.XX”,
“mnc”: “260”,
“sgsn_data_plane_ip_address”: “216.155.166.XXX”,
“ci”: null,
“apn”: null,
“tx_teid_control_plane”: 2667756875,
“rx_teid”: 2720724,
“rac”: null,
“imsi”: “9014301234567891XXXX”,
“sgsn_control_plane_ip_address”: “216.155.165.XXX”,
“nsapi”: 6,
“breakout_ip”: null,
“ggsn_data_plane_ip_address”: “185.57.216.XX”,
“tx_teid_data_plane”: 3095978
},
“country”: {
“id”: 68,
“name”: “France”,
“country_code”: “33”,
“mcc”: “208”,
“iso_code”: “fr”
}
}
}

3. User Authentification Failed Event Sample

{
“id”: 201388127,
“alert”: false,
“description”: “Failed authentication request from ‘user@company.com’, Reason: Invalid password from IP 9.9.9.9”,
“timestamp”: “2017-10-26T07:42:00.000+0000”,
“event_type”: {
“id”: 6,
“description”: “User authentication failed”
},
“event_source”: {
“id”: 2,
“description”: “API”
},
“event_severity”: {
“id”: 1,
“description”: “Warn”
},
“organisation”: {
“id”: 839921,
“name”: “Demo Company”
},
“user”: {
“id”: 84993,
“username”: “user@company.com”,
“name”: “Scott Tiger”
}
}

Usage Data

Usage Data provided by the 1NCE Data Streams enables the customer to closely monitor all SIM Cards and detect issues as well as misbehavior of devices.

1. Properties of Usage Data

 

Property Format Description
id Numeric Unique identifier of this transaction
cost Number with up to 6 decimal places Cost calculation of reported traffic volume
currency.id Numeric Unique identifier of currency of indicated cost
currency.code ISO 4217 Currency Code
start_timestamp UTC Timestamp Start time of traffic measurement
end_timestamp UTC Timestamp End time of traffic measurement
volume.rx Number with up to 6 decimal places Downstream traffic (MB) received by the endpoint
volume.tx Number with up to 6 decimal places Upstream traffic (MB) send by the endpoint
volume.total Number with up to 6 decimal places Total traffic consumed
imsi 15 digits numeric string Currently used IMSI
endpoint.id Numeric Unique identifier of endpoint
endpoint.name String The user-defined name set for this endpoint
endpoint.idp_address Numeric The IP address assigned to this endpoint
endpoint.tags String User-defined tags (if any) set for this endpoint
endpoint.imei Numeric The IMEI of the endpoint hardware
sim.id Numeric Unique identifier of SIM
sim.msisdn Numeric MSISDN of the associated SIM
sim.production_date Timestamp The production date of the associated SIM
sim.iccid 19 digits numeric string ICCID of SIM
organisation.id Numeric Unique identifier of organization
organisation.name String Name of organization
operator.id Numeric Unique identifier of visited operator
operator.mnc Numeric Mobile Network Code of the visited operator
operator.name String Name of that mobile operator
operator.country.id Numeric Unique identifier of visited country
operator.country.mnc Numeric Mobile Country Code of the visited operator
operator.country.name String Name of visited country
tariff.id Numeric Unique identifier of applied tariff
tariff.name String Name of Tariff
tariff.ratezone.id Numeric Unique identifier of applied rate zone
tariff.ratezone.name String Name of Rate zone
traffic_type.id Numeric Unique identifier of traffic type
traffic_type.name String Name of traffic type
2. Usage Data JSON Sample

{

“sim”:{

“production_date”:”2019-01-22 13:39:15″,

“msisdn”:”423111111111111″,

“id”:1337,

“iccid”:”1111111111111111111″

},

“traffic_type”:{

“id”:5,

“name”:”Data”

},

“cost”:0.0010142,

“volume”:{

“total”:0.005071,

“rx”:0.002418,

“tx”:0.002653

},

“imsi_id”:3506619,

“currency”:{

“symbol”:” “,

“code”:”EUR”,

“id”:1

},

“start_timestamp”:”2019-11-06 15:50:58″,

“imsi”:”111111111111111″,

“operator”:{

“mnc”:”01″,

“id”:2,

“country”:{

“mcc”:”262″,

“id”:74,

“name”:”Germany”

},

“name”:”Telekom”

},

“endpoint”:{

“ip_address”:”10.10.10.1″,

“imei”:”1111111111111111″,

“id”:9635467,

“tags”:null,

“name”:”My Endpoint”

},

“organisation”:{

“id”:1337,

“name”:”My Org”

},

“id”:85579,

“tariff”:{

“id”:166,

“ratezone”:{

“id”:276,

“name”:”Zone I”

},

“name”:”Global Roaming I – 1kb”

},

“end_timestamp”:”2019-11-06 16:06:42″

}

Integration with Cloud Services

AWS Kinesis

Integration with AWS Kinesis requires an AWS Access Key and Secret Key, with the right to write data to either all Kinesis Data Streams or the desired Data Stream. It is suggested to create a dedicated IAM user for connection.

Since Kinesis is a regional service, a specification of AWS region is required. You are required to specify the name of the Stream you want the data to be stored in.

 

AWS S3

Integration with AWS S3 requires an AWS Access Key and Secret Key, with the right to write data to either all S3 Buckets or the desired Bucket. It is suggested to create a dedicated IAM user for connection. Since S3 is a global service, a selection of AWS region is not required. You are required to specify the name of the S3 Bucket you want the data to be stored in.

The integration will deliver CSV files to the S3 Bucket in the following format:

  • Filename format events_YYYYMMDD_HHmmss.csv will be used for event data
  • Filename format cdr_YYYYMMDD_HHmmss.csv will be used for usage data

 

Keen.IO

Integration with keen.io offers the possibility to analyze and embed rich data about your SIM Cards.

The integration requires a keen.io account with a correct Project Key as well as the corresponding Write Key.

Shortly after configuration first data is sent to keen.io. Incoming data can be explored on keen.io on the “Streams” tab. Depending on the stream type the customer configured one or two streams appear with the following names: 1NCEEventData & 1NCEUsageData.

 

Data Dog

Integration with DataDog provides real-time performance monitoring for your SIMs. In conjunction with the 1NCE Platform, it enables the customer to collect and analyze metrics about the usage of your SIM Cards. The customer can create dashboards and trigger alerts on specific situations.

The Integration requires a DataDog account with an active Application Key and API Key.

Shortly after configuration the first data are sent to DataDog. Incoming data can be checked in the DataDog explorer and dashboards can be created.

The integration will deliver the following metrics:

  • volume
  • volume_tx
  • volume_rx
  • cost