Feedparser到数据框不会输出所有列 [英] Feedparser to dataframe doesnt ouput all columns
问题描述
我从feedparser解析URL并尝试获取所有列,但是我没有得到所有列作为输出,所以不确定问题出在哪里。如果执行以下内容。我没有得到几列的数据,但是数据确实存在,因为您可以在浏览器中检入。
I parsing a URL from feedparser and trying to get all columns, but i do not get all columns as a out put, not sure where the issue is. If you execute the below. I do not get data for few columns, but the data do exist as u can check in browser.
我的代码
import feedparser
import pandas as pd
xmldoc = feedparser.parse('http://www.ebay.com/rps/feed/v1.1/epnexcluded/EBAY-US')
df_cols = [
"title", "url", "endsAt", "image225","currency"
"price", "orginalPrice", "discountPercentage", "quantity", "shippingCost","dealUrl"
]
rows = []
for entry in xmldoc.entries:
s_title = entry.get("title","")
s_url = entry.get("url", "")
s_endsAt = entry.get("endsAt", "")
s_image225 = entry.get("image225", "")
s_currency = entry.get("currency", "")
s_price = entry.get("price","")
s_orginalPrice = entry.get("orginalPrice","")
s_discountPercentage = entry.get ("discountPercentage","")
s_quantity = entry.get("quantity","")
s_shippingCost = entry.get("shippingCost", "")
s_dealUrl = entry.get("dealUrl", "")#.replace('YOURUSERIDHERE','2427312')
rows.append({"title":s_title, "url": s_url, "endsAt": s_endsAt,
"image225": s_image225,"currency": s_currency,"price":s_price,
"orginalPrice": s_orginalPrice,"discountPercentage": s_discountPercentage,"quantity": s_quantity,
"shippingCost": s_shippingCost,"dealUrl": s_dealUrl})
out_df = pd.DataFrame(rows, columns=df_cols)
out_df
尝试过此操作,但这并不能给我任何数据,只有几列(我想是标题)
tried this, but this doesn't give me any data only few columns (headers i suppose)
import lxml.etree as ET
import urllib
response = urllib.request.urlopen('http://www.ebay.com/rps/feed/v1.1/epnexcluded/EBAY-US')
xml = response.read()
root = ET.fromstring(xml)
for item in root.findall('.*/item'):
df = pd.DataFrame([{item.tag: item.text if item.text.strip() != "" else item.find("*").text
for item in lnk.findall("*") if item is not None}
for lnk in root.findall('.//item')])
df
可以在一个排列如下,并将结果输出到PD。当我尝试此操作时,它确实可以部分解决问题(即,我缺少一些元素,导致此错误 AttributeError:对象没有属性价格,运输成本等),
如果元素为null,我们该如何处理?
Is is possible to iterate the URL offset within an array as below and result the output to a PD. When I try this it does work partially with issues (i.e) I have few elements missing, resulting in this error AttributeError: object has no attribute 'price', shipping cost etc.,
How do we handle if null for an element?
我的代码
import feedparser
import pandas as pd
#from simplified_scrapy import SimplifiedDoc, utils, req
getdeals = ['http://www.ebay.com/rps/feed/v1.1/epnexcluded/EBAY-US?limit=200',
'http://www.ebay.com/rps/feed/v1.1/epnexcluded/EBAY-US?limit=200&offset=200',
'http://www.ebay.com/rps/feed/v1.1/epnexcluded/EBAY-US?limit=200&offset=400']
posts=[]
for urls in getdeals:
feed = feedparser.parse(urls)
for deals in feed.entries:
print (deals)
posts.append((deals.title,deals.endsat,deals.image225,deals.price,deals.originalprice,deals.discountpercentage,deals.shippingcost,deals.dealurl))
df=pd.DataFrame(posts,columns=['title','endsat','image2255','price','originalprice','discountpercentage','shippingcost','dealurl'])
df.tail()
同样,如何循环多个JSON响应
Also , similarly how to loop multiple JSON responses
url= ["https://merchants.apis.com/v4/publisher/159663/offers?country=US&limit=2000",
"https://merchants.apis.com/v4/publisher/159663/offers?country=US&offset=2001&limit=2000"]
response = requests.request("GET", url, headers=headers, params=querystring)
response = response.json()
name = []
logo = []
date_added = []
description = []
for i in range(len(response['offers'])):
name.append(response['offers'][i]['merchant_details']['name'])
logo.append(response['offers'][i]['merchant_details']['metadata']['logo'])
date_added.append(response['offers'][i]['date_added'])
description.append(response['offers'][i]['description'])
try:
verticals.append(response['offers'][i]['merchant_details']['verticals'][0])
except IndexError:
verticals.append('NA')
pass
data1 = pd.DataFrame({'name':name,'logo':logo,'verticals':verticals, 'date_added':date_added,'description':description})
推荐答案
另一种方法。
import pandas as pd
from simplified_scrapy import SimplifiedDoc, utils, req
getdeals = ['http://www.ebay.com/rps/feed/v1.1/epnexcluded/EBAY-US?limit=200',
'http://www.ebay.com/rps/feed/v1.1/epnexcluded/EBAY-US?limit=200&offset=200',
'http://www.ebay.com/rps/feed/v1.1/epnexcluded/EBAY-US?limit=200&offset=400']
posts=[]
header = ['title','endsAt','image255','price','originalPrice','discountPercentage','shippingCost','dealUrl']
for url in getdeals:
try: # It's a good habit to have try and exception in your code.
feed = SimplifiedDoc(req.get(url))
for deals in feed.selects('item'):
row = []
for h in header: row.append(deals.select(h+">text()")) # Returns None when the element does not exist
posts.append(row)
except Exception as e:
print (e)
df=pd.DataFrame(posts,columns=header)
df.tail()
这篇关于Feedparser到数据框不会输出所有列的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!