BeautifulSoup刮从JavaScript（EN codeD）变量 [英] BeautifulSoup scrape from javascript (encoded) variable
本文介绍了BeautifulSoup刮从JavaScript（EN codeD）变量的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！
问题描述
我刮一个网页，因为它是存储在一个javascript变量不能得到某一个领域。
I am scraping a page and cannot get a certain field because it is stored in a javascript variable.
我的问题是，我怎么能凑以下code，德code它，并保存＆LT;李＆GT; 标记（S）的含量？使用BeautifulSoup和其他任何蟒蛇。
My question is, how can I scrape the following code, decode it, and save the <li> tag(s) content? Using BeautifulSoup and anything else python.
下面是里面的code中的＆LT;脚本＆GT; 标签：
Here is the code inside the <script> tag:
<script type="text/javascript">
var html_audition_details_sidebar = ' \u003Cdiv id\u003D\u0022apply_wrapper\u0022\u003E \u003Cdiv class\u003D\u0022header\u0022\u003E \u003Cp\u003EAudition Information\u003C/p\u003E \u003C/div\u003E \u003Cdiv class\u003D\u0022text \u0022\u003E \u003Cdiv class\u003D\u0022roleContainer \u0022 style\u003D\u0022color: #999\u003B font\u002Dsize: 14px\u003B\u0022\u003E \u003Cp\u003EOnly official members can see audition information for this job\u003C/p\u003E \u003C/div\u003E \u003C/div\u003E \u003Cdiv class\u003D\u0022applyButton\u0022\u003E \u003Cp\u003E\u003Ca class\u003D\u0022applyLink\u0022 href\u003D\u0022/accounts/login/apply/41680/\u0022\u003ESubscribe Now \u003C/a\u003E\u003C/p\u003E \u003C/div\u003E \u003C/div\u003E';
var html_additional_requirements = '';
var html_role_listing = '\u003Cdiv class\u003D\u0022text callListing loggedout \u0022\u003E \u003Cp class\u003D\u0022title\u0022\u003E\u003Ca name\u003D\u0022roles\u0022\u003E\u003C/a\u003ESeeking Talent \u003Cspan class\u003D\u0022optional\u0022\u003ESelect a role below for more information and submission instructions.\u003C/span\u003E\u003C/p\u003E \u003Cdiv class\u003D\u0022castingRoles\u0022\u003E \u003Cul\u003E \u003Cli \u003E \u003Ca href\u003D\u0022/casting/untitled\u002Dcomedy\u002Dshort\u002D41680/martinique\u002D159296/\u0022\u003E Martinique (Lead): \u003Cspan class\u003D\u0022roletag\u0022\u003E Female, 18\u002D25, Caucasian \u003C/span\u003E \u003Cspan class\u003D\u0022applyNow\u0022\u003E \u003C/span\u003E \u003C/a\u003E \u003Cp class\u003D\u0022role\u002Ddesc\u0022 style\u003D\u0022border\u002Dbottom: none\u003B padding\u002Dbottom: 0px\u003B margin\u002Dbottom: 0px\u003B\u0022\u003E native French speaker. \u003C/p\u003E \u003C/li\u003E \u003Cli \u003E \u003Ca href\u003D\u0022/casting/untitled\u002Dcomedy\u002Dshort\u002D41680/justin\u002D159297/\u0022\u003E Justin (Lead): \u003Cspan class\u003D\u0022roletag\u0022\u003E Male, 20\u002D25, All Ethnicities \u003C/span\u003E \u003Cspan class\u003D\u0022applyNow\u0022\u003E \u003C/span\u003E \u003C/a\u003E \u003Cp class\u003D\u0022role\u002Ddesc\u0022 style\u003D\u0022border\u002Dbottom: none\u003B padding\u002Dbottom: 0px\u003B margin\u002Dbottom: 0px\u003B\u0022\u003E comedy and improv skills, hopeless romantic. \u003C/p\u003E \u003C/li\u003E \u003Cli \u003E \u003Ca href\u003D\u0022/casting/untitled\u002Dcomedy\u002Dshort\u002D41680/flower\u002Dshop\u002Dsalesperson\u002D159299/\u0022\u003E Flower Shop Salesperson : \u003Cspan class\u003D\u0022roletag\u0022\u003E Males \u0026amp\u003B Females, 30+, All Ethnicities \u003C/span\u003E \u003Cspan class\u003D\u0022applyNow\u0022\u003E \u003C/span\u003E \u003C/a\u003E \u003Cp class\u003D\u0022role\u002Ddesc\u0022 style\u003D\u0022border\u002Dbottom: none\u003B padding\u002Dbottom: 0px\u003B margin\u002Dbottom: 0px\u003B\u0022\u003E impatient. \u003C/p\u003E \u003C/li\u003E \u003Cli \u003E \u003Ca href\u003D\u0022/casting/untitled\u002Dcomedy\u002Dshort\u002D41680/models\u002D159300/\u0022\u003E Models (Supporting): \u003Cspan class\u003D\u0022roletag\u0022\u003E Female, 18\u002D35, All Ethnicities \u003C/span\u003E \u003Cspan class\u003D\u0022applyNow\u0022\u003E \u003C/span\u003E \u003C/a\u003E \u003Cp class\u003D\u0022role\u002Ddesc\u0022 style\u003D\u0022border\u002Dbottom: none\u003B padding\u002Dbottom: 0px\u003B margin\u002Dbottom: 0px\u003B\u0022\u003E small roles, under five lines. \u003C/p\u003E \u003C/li\u003E \u003C/ul\u003E \u003C/div\u003E\u003C/div\u003E';
</script>
这是可怕的格式化，我很抱歉。
It is horribly formatted, I apologize.
我需要保存里面所有的只为变量里标记 html_role_listing 的网址。
I need to save the urls inside of all the li tags only for the variable html_role_listing.
例如：
打开这样的：
\u003Ca href\u003D\u0022/casting/untitled\u002Dcomedy\u002Dshort\u002D41680/martinique\u002D159296/\u0022\u003E
进入这个：
/casting/untitled-comedy-short-41680/martinique-159296/
感谢您的帮助提前！
Thanks for your help ahead of time!
推荐答案
如果你想要做它在一个通用的方法，你需要一个分析的JavaScript库。在这个例子中，我将使用 SLIMIT 做到这一点。
If you want to do it in a generic way, you'll need a library that parses javascript. In this example, I'll use slimit to do that.
首先，负载数据：
from bs4 import BeautifulSoup as Soup
import slimit
from slimit.parser import Parser
from slimit.visitors import nodevisitor
a = """<script type="text/javascript">
var html_audition_details_sidebar = ' \u003Cdiv id\u003D\u0022apply_wrapper\u0022\u003E \u003Cdiv class\u003D\u0022header\u0022\u003E \u003Cp\u003EAudition Information\u003C/p\u003E \u003C/div\u003E \u003Cdiv class\u003D\u0022text \u0022\u003E \u003Cdiv class\u003D\u0022roleContainer \u0022 style\u003D\u0022color: #999\u003B font\u002Dsize: 14px\u003B\u0022\u003E \u003Cp\u003EOnly official members can see audition information for this job\u003C/p\u003E \u003C/div\u003E \u003C/div\u003E \u003Cdiv class\u003D\u0022applyButton\u0022\u003E \u003Cp\u003E\u003Ca class\u003D\u0022applyLink\u0022 href\u003D\u0022/accounts/login/apply/41680/\u0022\u003ESubscribe Now \u003C/a\u003E\u003C/p\u003E \u003C/div\u003E \u003C/div\u003E';
var html_additional_requirements = '';
var html_role_listing = '\u003Cdiv class\u003D\u0022text callListing loggedout \u0022\u003E \u003Cp class\u003D\u0022title\u0022\u003E\u003Ca name\u003D\u0022roles\u0022\u003E\u003C/a\u003ESeeking Talent \u003Cspan class\u003D\u0022optional\u0022\u003ESelect a role below for more information and submission instructions.\u003C/span\u003E\u003C/p\u003E \u003Cdiv class\u003D\u0022castingRoles\u0022\u003E \u003Cul\u003E \u003Cli \u003E \u003Ca href\u003D\u0022/casting/untitled\u002Dcomedy\u002Dshort\u002D41680/martinique\u002D159296/\u0022\u003E Martinique (Lead): \u003Cspan class\u003D\u0022roletag\u0022\u003E Female, 18\u002D25, Caucasian \u003C/span\u003E \u003Cspan class\u003D\u0022applyNow\u0022\u003E \u003C/span\u003E \u003C/a\u003E \u003Cp class\u003D\u0022role\u002Ddesc\u0022 style\u003D\u0022border\u002Dbottom: none\u003B padding\u002Dbottom: 0px\u003B margin\u002Dbottom: 0px\u003B\u0022\u003E native French speaker. \u003C/p\u003E \u003C/li\u003E \u003Cli \u003E \u003Ca href\u003D\u0022/casting/untitled\u002Dcomedy\u002Dshort\u002D41680/justin\u002D159297/\u0022\u003E Justin (Lead): \u003Cspan class\u003D\u0022roletag\u0022\u003E Male, 20\u002D25, All Ethnicities \u003C/span\u003E \u003Cspan class\u003D\u0022applyNow\u0022\u003E \u003C/span\u003E \u003C/a\u003E \u003Cp class\u003D\u0022role\u002Ddesc\u0022 style\u003D\u0022border\u002Dbottom: none\u003B padding\u002Dbottom: 0px\u003B margin\u002Dbottom: 0px\u003B\u0022\u003E comedy and improv skills, hopeless romantic. \u003C/p\u003E \u003C/li\u003E \u003Cli \u003E \u003Ca href\u003D\u0022/casting/untitled\u002Dcomedy\u002Dshort\u002D41680/flower\u002Dshop\u002Dsalesperson\u002D159299/\u0022\u003E Flower Shop Salesperson : \u003Cspan class\u003D\u0022roletag\u0022\u003E Males \u0026amp\u003B Females, 30+, All Ethnicities \u003C/span\u003E \u003Cspan class\u003D\u0022applyNow\u0022\u003E \u003C/span\u003E \u003C/a\u003E \u003Cp class\u003D\u0022role\u002Ddesc\u0022 style\u003D\u0022border\u002Dbottom: none\u003B padding\u002Dbottom: 0px\u003B margin\u002Dbottom: 0px\u003B\u0022\u003E impatient. \u003C/p\u003E \u003C/li\u003E \u003Cli \u003E \u003Ca href\u003D\u0022/casting/untitled\u002Dcomedy\u002Dshort\u002D41680/models\u002D159300/\u0022\u003E Models (Supporting): \u003Cspan class\u003D\u0022roletag\u0022\u003E Female, 18\u002D35, All Ethnicities \u003C/span\u003E \u003Cspan class\u003D\u0022applyNow\u0022\u003E \u003C/span\u003E \u003C/a\u003E \u003Cp class\u003D\u0022role\u002Ddesc\u0022 style\u003D\u0022border\u002Dbottom: none\u003B padding\u002Dbottom: 0px\u003B margin\u002Dbottom: 0px\u003B\u0022\u003E small roles, under five lines. \u003C/p\u003E \u003C/li\u003E \u003C/ul\u003E \u003C/div\u003E\u003C/div\u003E';
</script>"""
soup = Soup(a)
js_content = soup.findAll('script')[0].text
现在，你可以选择：解析 js_content 与常规的前pressions或使用词法分析器（SLIMIT在这种情况下）。
Now you can choose: parse the js_content with regular expressions or use a lexer (slimit in this case).
par = Parser()
tree = par.parse(js_content)
encoded_html = [n.value for n in nodevisitor.visit(tree) if isinstance(n, slimit.ast.String)][2]
最后，你可以很容易地去code该字符串，这是有效的HTML：
Finally, you can easily decode that string, which is valid HTML:
decoded_html = encoded_html.decode('unicode_escape')
print(decoded_html)
所以重新分析这个HTML：
So reparse this HTML:
role_listing = Soup(decoded_html)
output = [ anchor.attrs['href'] for anchor in role_listing.select('li a') ]
print('---')
print("\n".join(output))
输出看起来是这样的：
The output looks like this:
'<div class="text callListing loggedout "> <p class="title"><a name="roles"></a>Seeking Talent <span class="optional">Select a role below for more information and submission instructions.</span></p> <div class="castingRoles"> <ul> <li > <a href="/casting/untitled-comedy-short-41680/martinique-159296/"> Martinique (Lead): <span class="roletag"> Female, 18-25, Caucasian </span> <span class="applyNow"> </span> </a> <p class="role-desc" style="border-bottom: none; padding-bottom: 0px; margin-bottom: 0px;"> native French speaker. </p> </li> <li > <a href="/casting/untitled-comedy-short-41680/justin-159297/"> Justin (Lead): <span class="roletag"> Male, 20-25, All Ethnicities </span> <span class="applyNow"> </span> </a> <p class="role-desc" style="border-bottom: none; padding-bottom: 0px; margin-bottom: 0px;"> comedy and improv skills, hopeless romantic. </p> </li> <li > <a href="/casting/untitled-comedy-short-41680/flower-shop-salesperson-159299/"> Flower Shop Salesperson : <span class="roletag"> Males & Females, 30+, All Ethnicities </span> <span class="applyNow"> </span> </a> <p class="role-desc" style="border-bottom: none; padding-bottom: 0px; margin-bottom: 0px;"> impatient. </p> </li> <li > <a href="/casting/untitled-comedy-short-41680/models-159300/"> Models (Supporting): <span class="roletag"> Female, 18-35, All Ethnicities </span> <span class="applyNow"> </span> </a> <p class="role-desc" style="border-bottom: none; padding-bottom: 0px; margin-bottom: 0px;"> small roles, under five lines. </p> </li> </ul> </div></div>'
---
/casting/untitled-comedy-short-41680/martinique-159296/
/casting/untitled-comedy-short-41680/justin-159297/
/casting/untitled-comedy-short-41680/flower-shop-salesperson-159299/
/casting/untitled-comedy-short-41680/models-159300/
这篇关于BeautifulSoup刮从JavaScript（EN codeD）变量的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
查看全文
