Data Streamer | Documentation and manual

Data Streams

Documentation

Data Streamers Functionality

General Information

1NCE offers a service called “Data Streams”. The Data Stream service allows to subscribe to real-time Event and Usage Data for all SIM Cards by pushing data directly to the customer server or an already integrated cloud service such as AWS Kinesis, DataDog or Keen.io. In total up to ten different Data Streams can be configured per account.

Configuration of Data Streams

Setting up a Data Stream on the configuration tab in the 1NCE Customer Portal requires the customer to select the Stream Type to specify if just Event or Usage Data is desired, or both should be sent jointly.

The customer can select the transmission of historic data. If historic data is selected the 1NCE Platform will send all customers data up to seven days old, however it will take significantly longer until transmissions are caught up with live data.

An activated and configured stream will be enabled instantly, and the performance can be monitored. The current Remote Status is shown in the Customer Portal as HTTP status coded from the remote site. If a 200 (OK) is displayed, the Data Stream is operating normally. In case the receiving section is not reachable, or the configuration is invalid it will show a 500er error.

Customers can pause and resume the stream at any time by using the action buttons next to the stream details.

For setting up the Data Streams for a RestAPI or RestAPI in Bulk mode the specification of an API Callback is required. The 1NCE Data Stream acts as an HTTP Client and needs an HTTP Server to handle each Post. Every sent message includes an array of one or more objects.

Event Data

To have a better understanding of Event Data, the following steps will provide a detailed view on the properties of Event Data and some typical samples for better understanding on a practical level.

Properties of Event Data

Event Data are always generated by the SIM carrying device as soon as changes appear, like a change of location, or in case of any error or disturbances. This gives an indication to the customer of tentatively irregular events or the need to take any action. To provide an overview all possible event types and properties are listed as follows.

1. Overview of Event Types

Id	Description
0	Generic
1	Update location
2	Update GPRS location
3	Create PDP Context
4	Update PDP Context
5	Delete PDP Context
6	User authentication failed
7	Application authentication failed
8	SIM activation
9	SIM suspension
10	SIM deletion
11	Endpoint blocked
12	Organization blocked
13	Support Access
14	Multi-factor Authentication
15	Purge Location
16	Purge GPRS Location
17	Self-Signup
18	Threshold reached
19	Quota used up

2. Generic Event Data

The following properties are included in all events sent:

Property_______	Format_______	Description
id	Numeric	Unique identifier of this event, if multiple events with same id are received (e.g. due to transmission errors) these should be treated by the receiver as duplicates
timestamp	Timestamp	Date/time when this event happened
event_type	Nested Object	Type of the event, see below for details
event_severity	Nested Object	Severity of the event, see below for details
event_source	Nested Object	Source of the event, see below for details
organisation	Nested Object	Organization associated with this event, see below for details
alert	Boolean	Event is a candidate to be alerted to a user
description	String	Human readable description of the event

3. Optional Properties

Event types related to specific SIM Cards or User include the following additional properties:

Property_______	Format_______	Description
imsi	Nested Object	Details of IMSI, see below for details (in a multi-IMSI
sim	Nested Object	Details of SIM, see below for details (in case of multi-IMSI configuration multiple different IMSIs may be reported for the same SIM)
endpoint	Nested Object	Details of Endpoint, see below for details

4. Properties of Details Object

Property	Format	Description
id	Numeric	Unique identifier of the actual used mobile network operator
name	String	Name of the mobile network operator
country	Nested Object	Country of mobile network operator
country.id	Numeric	Unique identifier of the country
country.name	String	Name of country
country.country_code	String	Country code
country.mcc	String	Mobile Country Code (MCC)
country.iso_code	String	ISO code
pdp_context	Nested Object	PDP Context Details
volume	Nested Object	Volume consumed in PDP Context
volume.rx	Number with up to 6 decimal places	Downstream Volume in MB
volume.tx	Number with up to 6 decimal places	Upstream Volume in MB.
volume.total	Number with up to 6 decimal places	Total volume

5. Properties of PDP Context

Property	Format_______	Description
pdp_context_id	String	Unique identifier of this PDP context
tunnel_created	Timestamp	Date/time when this PDP context was created
gtp_version	String	GTP Version, 1 or 2
ggsn_control_plane_ip_address	String	IP Address of GGSN/PGW Control Plane
ggsn_data_plane_ip_address	String	IP Address of GGSN/PGW Data Plane
sgsn_control_plane_ip_address	String	IP Address of SGSN/SGW Control Plane
sgsn_data_plane_ip_address	String	IP Address of SGSN/SGW Data Plane
region	String	Region where Data Plane is located
breakout_ip	String	IP Address used for Internet Breakout
apn	String	Access Point Name (APN)
nsapi	Integer	Network Service Access Point Identifier (NSAPI)
ue_ip_address	String	IP address assigned to Endpoint
imeisv	String	IMEISV
mcc	String	Mobile Country Code (MCC)
mnc	String	Mobile Network Code (MNC)
lac	Integer	Location Area Code (LAC)
sac	Integer	Service Area code (SAC)
rac	Integer	Routing Area code (RAC)
ci	Integer	Cell Identification (CI)
rat_type	Integer	Radio Access Type (RAT) (1=3G, 2=2G, 3=WLAN, 4=GAN, 5=HSPA+, 6=4G, 8=NB-IoT)

6. Properties of IMSI Object

Property_______	Format_______	Description
id	Numeric	Unique identifier of this IMSI
imsi	String	International mobile subscriber identity (IMSI)
import_date	Timestamp	Date/Time this IMSI was provisioned

7. Properties of Organisation Object

Property	Format	Description
id	Numeric	Unique identifier of this organisation
name	String	Name of organisation

8. Properties of User Object

Property	Format	Description
id	Numeric	Unique identifier of this user
username	String	Username e.g. e-mail address
name	String	Realname of user

9. Properties of Endpoint Objects

Property	Format	Description
id	Numeric	Unique identifier of this endpoint
name	String	Configured name of this endpoint
ip_address	String	IP address assigned to this endpoint
tags	String	Tags assigned to this endpoint
imei	String	International mobile equipment identity (IMEI)

10. Properties of SIM Object

Property_______	Format_______	Description
id	Numeric	Unique identifier of this SIM
iccid	String	Integrated Circuit Card identifier (ICCID) without checksum digit
msisdn	String	MSISDN
production_date	Timestamp	Date/Time this SIM chip was produced

11. Severity Levels

Id	Description
0	INFO
1	WARN
2	CRITICAL

12. Event Source

Id	Description
0	Network
1	Policy Control
2	API

Event Data JSON Sample

1. Update Location Event Sample

{
“id”: 2013707XX,
“alert”: false,
“description”: “New location received from VLR for IMSI=’9014301234567891XXXX’, now attached to VLR=’4917200130XX’.”,
“timestamp”: “2017-10-26T07:28:00.000+0000”,
“event_type”: {
“id”: 1,
“description”: “Update location”
},
“event_source”: {
“id”: 0,
“description”: “Network”
},
“event_severity”: {
“id”: 0,
“description”: “Info”
},
“organisation”: {
“id”: 839921,
“name”: “Demo Company”
},
“endpoint”: {
“id”: 8638726,
“name”: “GPS Tracker”,
“ip_address”: “100.96.234.249”,
“tags”: null,
“imei”: “357762083301XXXX”
},
“imsi”: {
“id”: 205672,
“imsi”: “9014301234567891XXXX”,
“import_date”: “2016-12-27T10:09:23.000+0000”
},
“sim”: {
“id”: 274887,
“iccid”: “898830300123456XXXX”,
“production_date”: “2016-12-27T10:09:23.000+0000”
},
“detail”: {
“id”: 3,
“name”: “Telekom”,
“country”: {
“id”: 74,
“name”: “Germany”,
“country_code”: “49”,
“mcc”: “262”,
“iso_code”: “de”
},
“tapcode”: [{
“id”: 2,
“tapcode”: “DEUD2”
}],
“mnc”: [{
“id”: 3,
“mnc”: “02”
}]
}
}

2. Delete PDP Context Event Sample

{
“id”: 2013707XX,
“alert”: false,
“description”: “PDP Context deleted.”,
“timestamp”: “2017-10-26T07:27:59.000+0000”,
“event_type”: {
“id”: 5,
“description”: “Delete PDP Context”
},
“event_source”: {
“id”: 0,
“description”: “Network”
},
“event_severity”: {
“id”: 0,
“description”: “Info”
},
“organisation”: {
“id”: 839921,
“name”: “Demo Company”
},
“endpoint”: {
“id”: 8427408,
“name”: “GPS Tracker”,
“ip_address”: “10.194.50.XX”,
“tags”: null,
“imei”: “898830300123456XXXX”
},
“imsi”: {
“id”: 372566,
“imsi”: “9014301234567891XXXX”,
“import_date”: “2017-03-15T21:46:01.000+0000”
},
“sim”: {
“id”: 319318,
“iccid”: “898830300123456XXXX”,
“production_date”: “2017-03-15T21:46:01.000+0000”
},
“detail”: {
“id”: 48,
“name”: “SFR Cegetel”,
“volume”: {
“rx”: 0.012671,
“tx”: 0.01148,
“total”: 0.024151
},
“pdp_context”: {
“mcc”: “310”,
“tunnel_created”: “2017-12-11T05:49:29”,
“ggsn_control_plane_ip_address”: “185.57.216.XX”,
“pdp_context_id”: “162094787”,
“imeisv”: “898830300123456XXXX”,
“region”: “eu-west-1”,
“lac”: 40484,
“sac”: 61142,
“rat_type”: 1,
“gtp_version”: “1”,
“ue_ip_address”: “100.105.197.XX”,
“mnc”: “260”,
“sgsn_data_plane_ip_address”: “216.155.166.XXX”,
“ci”: null,
“apn”: null,
“tx_teid_control_plane”: 2667756875,
“rx_teid”: 2720724,
“rac”: null,
“imsi”: “9014301234567891XXXX”,
“sgsn_control_plane_ip_address”: “216.155.165.XXX”,
“nsapi”: 6,
“breakout_ip”: null,
“ggsn_data_plane_ip_address”: “185.57.216.XX”,
“tx_teid_data_plane”: 3095978
},
“country”: {
“id”: 68,
“name”: “France”,
“country_code”: “33”,
“mcc”: “208”,
“iso_code”: “fr”
}
}
}

3. User Authentification Failed Event Sample

{
“id”: 201388127,
“alert”: false,
“description”: “Failed authentication request from ‘user@company.com’, Reason: Invalid password from IP 9.9.9.9”,
“timestamp”: “2017-10-26T07:42:00.000+0000”,
“event_type”: {
“id”: 6,
“description”: “User authentication failed”
},
“event_source”: {
“id”: 2,
“description”: “API”
},
“event_severity”: {
“id”: 1,
“description”: “Warn”
},
“organisation”: {
“id”: 839921,
“name”: “Demo Company”
},
“user”: {
“id”: 84993,
“username”: “user@company.com”,
“name”: “Scott Tiger”
}
}

Usage Data

Usage Data provided by the 1NCE Data Streams enables the customer to closely monitor all SIM Cards and detect issues as well as misbehavior of devices.

1. Properties of Usage Data

Property	Format	Description
id	Numeric	Unique identifier of this transaction
cost	Number with up to 6 decimal places	Cost calculation of reported traffic volume
currency.id	Numeric	Unique identifier of currency of indicated cost
currency.code	ISO 4217	Currency Code
start_timestamp	UTC Timestamp	Start time of traffic measurement
end_timestamp	UTC Timestamp	End time of traffic measurement
volume.rx	Number with up to 6 decimal places	Downstream traffic (MB) received by the endpoint
volume.tx	Number with up to 6 decimal places	Upstream traffic (MB) send by the endpoint
volume.total	Number with up to 6 decimal places	Total traffic consumed
imsi	15 digits numeric string	Currently used IMSI
endpoint.id	Numeric	Unique identifier of endpoint
endpoint.name	String	The user-defined name set for this endpoint
endpoint.idp_address	Numeric	The IP address assigned to this endpoint
endpoint.tags	String	User-defined tags (if any) set for this endpoint
endpoint.imei	Numeric	The IMEI of the endpoint hardware
sim.id	Numeric	Unique identifier of SIM
sim.msisdn	Numeric	MSISDN of the associated SIM
sim.production_date	Timestamp	The production date of the associated SIM
sim.iccid	19 digits numeric string	ICCID of SIM
organisation.id	Numeric	Unique identifier of organization
organisation.name	String	Name of organization
operator.id	Numeric	Unique identifier of visited operator
operator.mnc	Numeric	Mobile Network Code of the visited operator
operator.name	String	Name of that mobile operator
operator.country.id	Numeric	Unique identifier of visited country
operator.country.mnc	Numeric	Mobile Country Code of the visited operator
operator.country.name	String	Name of visited country
tariff.id	Numeric	Unique identifier of applied tariff
tariff.name	String	Name of Tariff
tariff.ratezone.id	Numeric	Unique identifier of applied rate zone
tariff.ratezone.name	String	Name of Rate zone
traffic_type.id	Numeric	Unique identifier of traffic type
traffic_type.name	String	Name of traffic type

2. Usage Data JSON Sample

{

“sim”:{

“production_date”:”2019-01-22 13:39:15″,

“msisdn”:”423111111111111″,

“id”:1337,

“iccid”:”1111111111111111111″

“traffic_type”:{

“id”:5,

“name”:”Data”

“cost”:0.0010142,

“volume”:{

“total”:0.005071,

“rx”:0.002418,

“tx”:0.002653

“imsi_id”:3506619,

“currency”:{

“symbol”:” “,

“code”:”EUR”,

“id”:1

“start_timestamp”:”2019-11-06 15:50:58″,

“imsi”:”111111111111111″,

“operator”:{

“mnc”:”01″,

“id”:2,

“country”:{

“mcc”:”262″,

“id”:74,

“name”:”Germany”

“name”:”Telekom”

“endpoint”:{

“ip_address”:”10.10.10.1″,

“imei”:”1111111111111111″,

“id”:9635467,

“tags”:null,

“name”:”My Endpoint”

“organisation”:{

“id”:1337,

“name”:”My Org”

“id”:85579,

“tariff”:{

“id”:166,

“ratezone”:{

“id”:276,

“name”:”Zone I”

“name”:”Global Roaming I – 1kb”

“end_timestamp”:”2019-11-06 16:06:42″

}

Integration with Cloud Services

AWS Kinesis

Integration with AWS Kinesis requires an AWS Access Key and Secret Key, with the right to write data to either all Kinesis Data Streams or the desired Data Stream. It is suggested to create a dedicated IAM user for connection.

Since Kinesis is a regional service, a specification of AWS region is required. You are required to specify the name of the Stream you want the data to be stored in.

AWS S3

Integration with AWS S3 requires an AWS Access Key and Secret Key, with the right to write data to either all S3 Buckets or the desired Bucket. It is suggested to create a dedicated IAM user for connection. Since S3 is a global service, a selection of AWS region is not required. You are required to specify the name of the S3 Bucket you want the data to be stored in.

The integration will deliver CSV files to the S3 Bucket in the following format:

Filename format events_YYYYMMDD_HHmmss.csv will be used for event data
Filename format cdr_YYYYMMDD_HHmmss.csv will be used for usage data

Keen.IO

Integration with keen.io offers the possibility to analyze and embed rich data about your SIM Cards.

The integration requires a keen.io account with a correct Project Key as well as the corresponding Write Key.

Shortly after configuration first data is sent to keen.io. Incoming data can be explored on keen.io on the “Streams” tab. Depending on the stream type the customer configured one or two streams appear with the following names: 1NCEEventData & 1NCEUsageData.

Data Dog

Integration with DataDog provides real-time performance monitoring for your SIMs. In conjunction with the 1NCE Platform, it enables the customer to collect and analyze metrics about the usage of your SIM Cards. The customer can create dashboards and trigger alerts on specific situations.

The Integration requires a DataDog account with an active Application Key and API Key.

Shortly after configuration the first data are sent to DataDog. Incoming data can be checked in the DataDog explorer and dashboards can be created.

The integration will deliver the following metrics:

volume
volume_tx
volume_rx
cost

Download

Download the complete documentation of the Data Streamers Functionality in PDF-Format.

Buy the 1NCE IoT Flat Rate now!

Visit the 1NCE Shop and start connecting your IoT devices in the most easiest way. Simply pick and choose the required SIM card format and fill out all required forms. After the payment has been approved you get your cards within two to three business days.