Emerging in recent years, open edge computing platforms (OECPs) claim large-scale edge nodes, the extensive usage and adoption, as well as the openness to any third parties to join as edge nodes. For instance, OneThingCloud, a major OECP operated in China, advertises 5 million edge nodes, 70TB bandwidth, and 1,500PB storage. However, little information is publicly available for such OECPs with regards to their technical mechanisms and involvement in edge computing activities. Furthermore, different from known edge computing paradigms, OECPs feature an open ecosystem wherein any third party can participate as edge nodes and earn revenue for the contribution of computing and bandwidth resources, which, however, can introduce byzantine or even malicious edge nodes and thus break the traditional threat model for edge computing. In this study, we conduct the first empirical study on two representative OECPs, which is made possible through the deployment of edge nodes across locations, the efficient and semi-automatic analysis of edge traffic as well as the carefully designed security experiments. As the results, a set of novel findings and insights have been distilled with regards to their technical mechanisms, the landscape of edge nodes, the usage and adoption, and the practical security/privacy risks. Particularly, millions of daily active edge nodes have been observed, which feature a wide distribution in the network space and the extensive adoption in content delivery towards end users of 16 popular Internet services. Also, multiple practical and concerning security risks have been identified along with acknowledgements received from relevant parties, e.g., the exposure of long-term and cross-edge-node credentials, the co-location with malicious activities of diverse categories, the failures of TLS certificate verification, the extensive information leakage against end users, etc.
To figure out what purpose the OECP traffic flow is intended for, and what remote parties have communicated with our self-deployed edge nodes, and ultimately understand what edge computing activities have been conducted in OECPs, we pursue edge tasks through a combination of manual analysis and automatic measurements.
The manual analysis allows us to gain qualitative knowledge such as the categories of edge traffic flows, and the signatures to associate traffic flows with different categories or distinct remote parties. The automatic measurements are designed to generate quantitative measurement results, e.g., the volume and shares of different traffic categories.
As learned from edge traffic captured by ourselves, 22,214 edge node IPs have ever communicated with ones under our control, among which, 17,585 are YunFan CDN nodes (Tiptime edge nodes), and 2,818 are Xingyu CDN nodes (OneThingCloud nodes), and 1,817 are Bilibili CDN nodes that claim to be OneThingCloud nodes.
OECP | Node Source | Node IPs | /8 IPv4 | ASes |
---|---|---|---|---|
TipTime | YunFan CDN | 17,585 | 51 | 46 |
OneThingCloud | Bilibili CDN | 1,817 | 49 | 38 |
OneThingCloud | Xingyu CDN | 2,818 | 32 | 5 |
Both | All | 22,214 | 54 | 67 |
Through analyzing the edge traffic, several side channels have been successfully identified to gain an upper-bound approximation for edge nodes. For example, we observe that YunFan CDN assigns to each CDN node unique FQDNs (fully qualified domain names) and such FQDNs follow unified subdomain patterns. Therefore, querying passive DNS with these FQDN patterns can reveal historically active CDN node IPs, which provides another channel to upper-bound estimated edge nodes of TipTime.
OECP | Node FQDNs | Node IPs1 | IPv6 | /8 IPv4 | ASes2 |
---|---|---|---|---|---|
TipTime | 4,233,571,373 | 28,212,313 | 9,416,567 | 89 | 114 |
OneThingCloud | 100,492,251 | 7,383,677 | 4,654,242 | 255 | 182 |
Both | 4,334,063,624 | 34,364,400 | 14,070,775 | 255 | 237 |
1 Both IPv4 and IPv6 addresses. 2 Each platform has 500K IPs sampled to query IPinfo for autonomous systems (ASes). |
All edge computing tasks observed in our study are content delivery tasks which involve the collaboration between CDN services and the open edge computing platforms. Through analyzing the traffic flows of these CDN tasks, we have identified 16 upstream content providers that subscribe to one or more of these 6 CDN services and have their content payloads delivered through edge nodes of the two OECPs.
CDN | OECP | Content Provider |
---|---|---|
YunFan CDN | TipTime | KuaiShou, Douyin, Baidu Cloud, PPTV, Mogen Cloud, Jingdong Cloud, Zuiyou |
Wangsu CDN | TipTime | Toutiao |
Xingyu CDN | OneThing Cloud | Zuiyou, Wasu TV, Netease, Toutiao, GiTV, imoo, Xiaomi |
Bilibili CDN | OneThing Cloud | Bilibili |
Baidu CDN | OneThing Cloud | Haokan Video, Baidu Cloud |
Xunlei CDN | OneThing Cloud | Xunlei |
We find that edge nodes across platforms tend to share and locally store long-term TLS credentials, which renders a non-negligible Man-in-the-Middle(MITM) attacking surface for TLS traffic of content delivery.
Once edge nodes operated by the attacker gain access to TLS credentials, it could control the content delivery flow.
The Scenario of the Man-in-The-Middle Attacks
We looked into malicious traces of edge nodes as learned from the proprietary threat intelligence platform, which reveals that edge node IPs are concurrently involved in malicious activities that feature both a large scale and diverse categories.
Table below presents top 5 along with their contribution to MTFs and the involved edge nodes, including botnet, remote access trojan (RAT), illicit promotion, cryptojacking, and malicious downloads. Particularly, over 1.3 billion botnet traffic flows have been captured, which involve 11% of all the sampled edge node IPs. On the other hand, 55.90% edge node IPs are involved in MTFs of RAT which suggest that one or more machines attached to these IPs are compromised with RATs installed.
Category | MTFs | % MTFs | % Edge IPs | % CDN IPs |
---|---|---|---|---|
Botnet | 1.37B | 68.92% | 11.08% | 11.84% |
RAT1 | 312M | 15.69% | 55.90% | 59.59% |
Illicit promotion | 111M | 5.60% | 48.88% | 50.73% |
Cryptojacking | 67M | 3.38% | 17.17% | 18.09% |
Malicious downloads | 44M | 2.21% | 4.92% | 5.19% |
1 RAT stands for the remote access trojan. |
We observe and demonstrate that edge nodes of both OECPs fail to verify the server TLS certificate for part of the TLS traffic flows. A TipTime edge node is subject to this vulnerability for all the TLS traffic towards upstream servers, while only the logging traffic of OneThingCloud edge nodes shares this vulnerability.
Edge Type | Traffic Category | ||
---|---|---|---|
Control1 | Logging1 | Task Payload1 | |
TipTime | ✖ | ✖ | ✖ |
OneThingCloud | ✔ | ✖ | ✔ |
1. Task Payloads denotes flows for downloading deployment payloads of edge computing tasks. |
More security risks can be found in our paper.
@article{bi2024dissectingopenedgecomputing,
title={Dissecting Open Edge Computing Platforms: Ecosystem, Usage, and Security Risks},
author={Yu Bi and Mingshuo Yang and Yong Fang and Xianghang Mi and Shanqing Guo and Shujun Tang and Haixin Duan},
year={2024},
eprint={2404.09681},
archivePrefix={arXiv},
url={https://arxiv.org/abs/2404.09681}
}