如何从交互式图表中抓取JSON数据? [英] How to scrape json data from an interactive chart?
问题描述
我要从网站的特定部分中抓取数据，这是屏幕截图该部分的-
I have a specific section of a website that I want to scrape data from and here's the screenshot of the section -
我检查了该特定部分的元素，并注意到它在canvas标签内.但是，我还检查了网站的源代码，发现数据以我不熟悉的格式位于源代码中.这是该数据的示例
I inspected the elements of that particular section and noticed that it's within a canvas tag. However, I also checked the source code of the website and I found that the data lies within the source code in a format I'm not familiar with. Here's a sample of that data
JSON.parse('\x5B\x7B\x22id\x22\x3A\x2232522\x22,\x22minute\x22\x3A\x2222\x22,\x22result\x22\x3A\x22MissedShots\x22,
\x22X\x22\x3A\x220.7859999847412109\x22,\x22Y\x22\x3A\x220.52\x22,\x22xG\x22\x3A\x220.03867039829492569\x22,
\x22player\x22\x3A\x22Lionel\x20Messi\x22,
\x22h_a\x22\x3A\x22h\x22,
\x22player_id\x22\x3A\x222097\x22,\x22situation\x22\x3A\x22OpenPlay\x22,
\x22season\x22\x3A\x222014\x22,\x22shotType\x22\x3A\x22LeftFoot\x22,
\x22match_id\x22\x3A...);
我该如何解析这些数据，以获取屏幕快照中地图中每个镜头的x，y坐标?
How do I parse through this data to give me the x,y co-ordinates of every shot from the map in the screenshot?
推荐答案
是编码/解码问题.
您可以拉出该字符串，然后基本上需要忽略转义字符.完成此操作后，您可以使用json.loads()读取该内容，然后浏览json结构.
You can pull that string and then essentially need to ignore the escape charachters. Once you do that, you can use json.loads() to read that in and then can navigate the json structure.
现在，我只是快速浏览一下，但没有看到那里的数据来显示绘图在击球图上的位置.但是您可以看看是否可以找到它.但是，数据确实具有shotZones键.
Now I only looked quickly, but did not see the data in there to show where the plot is on the shot chart. But you can have a look to see if you can find it. The data does however have a shotZones key.
import requests
from bs4 import BeautifulSoup
import json
import codecs
url = 'https://understat.com/player/2097'
response = requests.get(url)
soup = BeautifulSoup(response.text, 'html.parser')
scripts = soup.find_all('script')
for script in scripts:
if 'var groupsData = JSON.parse' in script.text:
encoded_string = script.text
encoded_string = encoded_string .split("var groupsData = JSON.parse('")[-1]
encoded_string = encoded_string.rsplit("'),",1)[0]
jsonStr = codecs.getdecoder('unicode-escape')(encoded_string)[0]
jsonObj = json.loads(jsonStr)
修改
实际上我找到了.在这里，您去了:
Actually I found it. Here you go:
import requests
from bs4 import BeautifulSoup
import json
import codecs
from pandas.io.json import json_normalize
url = 'https://understat.com/player/2097'
response = requests.get(url)
soup = BeautifulSoup(response.text, 'html.parser')
scripts = soup.find_all('script')
# I noticed the data was imbedded in the script tag that started with `var shotsData`
for script in scripts:
if 'var shotsData' in script.text:
# I store that text, then trim off the string on the ends so that
# it's in a valid json format
encoded_string = script.text
encoded_string = encoded_string .split("JSON.parse('", 1)[-1]
encoded_string = encoded_string.rsplit("player_info =",1)[0]
encoded_string = encoded_string.rsplit("'),",1)[0]
# Have it ignore the escape characters so it can decode the ascii
# and be able to use json.loads
jsonStr = codecs.getdecoder('unicode-escape')(encoded_string)[0]
jsonObj = json.loads(jsonStr)
df = json_normalize(jsonObj)
输出:
print (df)
X ... xG
0 0.7859999847412109 ... 0.03867039829492569
1 0.8619999694824219 ... 0.06870150566101074
2 0.86 ... 0.15034306049346924
3 0.8180000305175781 ... 0.045503295958042145
4 0.8690000152587891 ... 0.06531666964292526
5 0.7230000305175781 ... 0.054804932326078415
6 0.9119999694824219 ... 0.0971858948469162
7 0.885 ... 0.11467907577753067
8 0.875999984741211 ... 0.10627452284097672
9 0.9540000152587891 ... 0.3100203275680542
10 0.8969999694824219 ... 0.12571729719638824
11 0.8959999847412109 ... 0.04122981056571007
12 0.8730000305175781 ... 0.09942527115345001
13 0.769000015258789 ... 0.025321772322058678
14 0.885 ... 0.7432776093482971
15 0.86 ... 0.4680374562740326
16 0.7619999694824219 ... 0.05699075385928154
17 0.919000015258789 ... 0.10647356510162354
18 0.9530000305175781 ... 0.571601390838623
19 0.8280000305175781 ... 0.07561512291431427
20 0.9030000305175782 ... 0.4600500166416168
21 0.9469999694824218 ... 0.3132372796535492
22 0.92 ... 0.2869703769683838
23 0.7659999847412109 ... 0.07576987147331238
24 0.9640000152587891 ... 0.3824153244495392
25 0.8590000152587891 ... 0.1282796859741211
26 0.9330000305175781 ... 0.42914989590644836
27 0.9230000305175782 ... 0.4968196153640747
28 0.8240000152587891 ... 0.08198583126068115
29 0.965999984741211 ... 0.4309735596179962
.. ... ... ...
843 0.9159999847412109 ... 0.4672183692455292
844 0.7430000305175781 ... 0.04068271815776825
845 0.815 ... 0.07300572842359543
846 0.8980000305175782 ... 0.06551901996135712
847 0.7680000305175781 ... 0.028392281383275986
848 0.885 ... 0.7432776093482971
849 0.875999984741211 ... 0.4060465097427368
850 0.7880000305175782 ... 0.09496577084064484
851 0.7190000152587891 ... 0.05071594566106796
852 0.7680000305175781 ... 0.090679831802845
853 0.7440000152587891 ... 0.06875557452440262
854 0.9069999694824219 ... 0.45824503898620605
855 0.850999984741211 ... 0.06454816460609436
856 0.935 ... 0.5926618576049805
857 0.9219999694824219 ... 0.16091874241828918
858 0.73 ... 0.05882067605853081
859 0.9080000305175782 ... 0.3522365391254425
860 0.8209999847412109 ... 0.1690768003463745
861 0.850999984741211 ... 0.11893663555383682
862 0.88 ... 0.11993970721960068
863 0.8119999694824219 ... 0.15579797327518463
864 0.7019999694824218 ... 0.011425728909671307
865 0.7530000305175781 ... 0.06945621967315674
866 0.850999984741211 ... 0.08273076266050339
867 0.8180000305175781 ... 0.06529481709003448
868 0.86 ... 0.10793478786945343
869 0.8190000152587891 ... 0.061923813074827194
870 0.8130000305175781 ... 0.05294585973024368
871 0.799000015258789 ... 0.06358513236045837
872 0.9019999694824219 ... 0.5841030478477478
[873 rows x 20 columns]
这篇关于如何从交互式图表中抓取JSON数据?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
