改进在R中从Google获取股票新闻数据的功能 [英] Improving a function to get stock news data from google in R

问题描述

我已经编写了一个功能，可以从给定的股票代号获取并解析Google的新闻数据，但是我敢肯定，有一些方法可以对其进行改进.对于初学者来说，我的函数在GMT时区而不是用户当前时区中返回一个对象，并且如果传递的数字大于299，则它将失败(可能是因为google每只股票仅返回300个故事).这有点回答我自己的问题堆栈溢出，并且严重依赖

I've written a function to grab and parse news data from Google for a given stock symbol, but I'm sure there are ways it could be improved. For starters, my function returns an object in the GMT timezone, rather than the user's current timezone, and it fails if passed a number greater than 299 (probably because google only returns 300 stories per stock). This is somewhat in response to my own question on stack overflow, and relies heavily on this blog post.

tl; dr:如何改善此功能?

tl;dr: how can I improve this function?

 getNews <- function(symbol, number){

    # Warn about length
    if (number>300) {
        warning("May only get 300 stories from google")
    }

    # load libraries
    require(XML); require(plyr); require(stringr); require(lubridate);
    require(xts); require(RDSTK)

    # construct url to news feed rss and encode it correctly
    url.b1 = 'http://www.google.com/finance/company_news?q='
    url    = paste(url.b1, symbol, '&output=rss', "&start=", 1,
               "&num=", number, sep = '')
    url    = URLencode(url)

    # parse xml tree, get item nodes, extract data and return data frame
    doc   = xmlTreeParse(url, useInternalNodes = TRUE)
    nodes = getNodeSet(doc, "//item")
    mydf  = ldply(nodes, as.data.frame(xmlToList))

    # clean up names of data frame
    names(mydf) = str_replace_all(names(mydf), "value\\.", "")

    # convert pubDate to date-time object and convert time zone
    pubDate = strptime(mydf$pubDate, 
                     format = '%a, %d %b %Y %H:%M:%S', tz = 'GMT')
    pubDate = with_tz(pubDate, tz = 'America/New_york')
    mydf$pubDate = NULL

    #Parse the description field
    mydf$description <- as.character(mydf$description)
    parseDescription <- function(x) {
        out <- html2text(x)$text
        out <- strsplit(out,'\n|--')[[1]]

        #Find Lead
        TextLength <- sapply(out,nchar)
        Lead <- out[TextLength==max(TextLength)]

        #Find Site
        Site <- out[3]

        #Return cleaned fields
        out <- c(Site,Lead)
        names(out) <- c('Site','Lead')
        out
    }
    description <- lapply(mydf$description,parseDescription)
    description <- do.call(rbind,description)
    mydf <- cbind(mydf,description)

    #Format as XTS object
    mydf = xts(mydf,order.by=pubDate)

    # drop Extra attributes that we don't use yet
    mydf$guid.text = mydf$guid..attrs = mydf$description = mydf$link = NULL
    return(mydf) 

}

推荐答案

这是getNews函数的

  getNews2 <- function(symbol, number){

    # load libraries
    require(XML); require(plyr); require(stringr); require(lubridate);  

    # construct url to news feed rss and encode it correctly
    url.b1 = 'http://www.google.com/finance/company_news?q='
    url    = paste(url.b1, symbol, '&output=rss', "&start=", 1,
               "&num=", number, sep = '')
    url    = URLencode(url)

    # parse xml tree, get item nodes, extract data and return data frame
    doc   = xmlTreeParse(url, useInternalNodes = T);
    nodes = getNodeSet(doc, "//item");
    mydf  = ldply(nodes, as.data.frame(xmlToList))

    # clean up names of data frame
    names(mydf) = str_replace_all(names(mydf), "value\\.", "")

    # convert pubDate to date-time object and convert time zone
    mydf$pubDate = strptime(mydf$pubDate, 
                     format = '%a, %d %b %Y %H:%M:%S', tz = 'GMT')
    mydf$pubDate = with_tz(mydf$pubDate, tz = 'America/New_york')

    # drop guid.text and guid..attrs
    mydf$guid.text = mydf$guid..attrs = NULL

    return(mydf)    
}

此外，您在代码中可能存在错误，因为我尝试将其用于symbol = 'WMT'并返回了错误.我认为getNews2也适用于WMT.检查一下，让我知道它是否适合您.

Moreover, there might be a bug in your code, as I tried using it for symbol = 'WMT' and it returned an error. I think getNews2 works fine for WMT too. Check it out and let me know if it works for you.

PS. description列仍包含html代码.但是从中提取文本应该很容易.我会在有时间的时候发布更新

PS. The description column still contains html code. But it should be easy to extract the text from it. I will post an update when I find time

这篇关于改进在R中从Google获取股票新闻数据的功能的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！