Server Metas

The server_metas dataset is a JSONL-formatted listing of info on every DScope collection instance allocated during each day.

Location

This dataset has both public and private versions:

public: s3://dscope-datasets-public/metas/{date}.jsonl.zst
private: s3://dscope-datasets-private/metas/{date}.jsonl.zst

Format

For each DScope collection date, lines contain the following fields in JSON format:

Name	Description
`Id`	AWS EC2 Instance ID
`Zone`	AWS EC2 Zone name. Note that zone names vary across accounts, and are not guaranteed to be consistent in the dataset.
`ZoneId`	The Zone ID. These are consistent across accounts but not present in all meta records.
`LaunchTime`	Approximate instance launch time (from EC2 metadata service)
`RecordStartTime`	Exact time the instance started recording traffic.
`PublicIp`	EC2 Public IP assigned to the instance. Note: in the public version the last octet of this is deterministically randomized. The same IP is randomized the same way every time.
`TargetRecordDuration`	How long the instance was configured to record for, in seconds
`InstanceType`	The EC2 instance type of the recording instance
`BootTime`	The boot time of the instance as measured by the Linux kernel
`BuildVersion`	Git hash of the DScope collector version running.

Example Record (formatted) ↕

{
    "Id":"i-002736336124bab72",
    "Zone":"ap-northeast-1a",
    "ZoneId":"apne1-az4",
    "LaunchTime":"2023-06-01T21:29:50Z",
    "RecordStartTime":"2023-06-01T21:31:15.54268348Z",
    "PublicIp":"43.207.225.165",
    "TargetRecordDuration":600,
    "InstanceType":"t4g.nano",
    "BootTime":"2023-06-01T21:29:43.93266199Z",
    "BuildVersion":"3a7c9a7"
}

Caveats

Measurement Applications

EC2 spot instance preemption rates
AWS infrastructure responsiveness based on boot times
AWS IP Address Pool Reuse