R – Using xpathSApply to scrape XML attributes in R

rxmlxpath

I am scraping XML in R using xpathSApply (in the XML package) and having trouble pulling attributes out.

First, a relevant snippet of XML:

 <div class="offer-name">
        <a href="http://www.somesite.com" itemprop="name">Fancy Product</a>
      </div>

I have successfully pulled the 'Fancy Product' (i.e. element?) using:

Products <- xpathSApply(parsedHTML, "//div[@class='offer-name']", xmlValue)

That took some time (I'm a n00b), but the documentation is good and there are several answered questions here I was able to leverage. I can't figure out how to pull the "http://www.somesite.com" out though (attribute?). I've speculated that it involves changing the 3rd term from 'xmlValue' to 'xmlGetAttr' but I could be totally off.

FYI (1) There are 2 more parent < div> above the snippet I pasted and (2) here is the abbreviated complete-ish code (which I don't think is relevant but included for the sake of completeness) is:

library(XML)
library(httr)

content2 = paste(readLines(file.choose()), collapse = "\n") # User will select file.
parsedHTML = htmlParse(content2,asText=TRUE)

Products <- xpathSApply(parsedHTML, "//div[@class='offer-name']", xmlValue)

Best Solution

The href is an attribute. You can select the appropriate node //div/a and use the xmlGetAttr function with name = href:

'<div class="offer-name">
  <a href="http://www.somesite.com" itemprop="name">Fancy Product</a>
  </div>' -> xData
library(XML)
parsedHTML <- xmlParse(xData)
Products <- xpathSApply(parsedHTML, "//div[@class='offer-name']", xmlValue) 
hrefs <- xpathSApply(parsedHTML, "//div/a", xmlGetAttr, 'href')
> hrefs
[1] "http://www.somesite.com"

XML escape characters

There are only five:

"   &quot;
'   &apos;
<   &lt;
>   &gt;
&   &amp;

Escaping characters depends on where the special character is used.

The examples can be validated at the W3C Markup Validation Service.

Text

The safe way is to escape all five characters in text. However, the three characters ", ' and > needn't be escaped in text:

<?xml version="1.0"?>
<valid>"'></valid>

Attributes

The safe way is to escape all five characters in attributes. However, the > character needn't be escaped in attributes:

<?xml version="1.0"?>
<valid attribute=">"/>

The ' character needn't be escaped in attributes if the quotes are ":

<?xml version="1.0"?>
<valid attribute="'"/>

Likewise, the " needn't be escaped in attributes if the quotes are ':

<?xml version="1.0"?>
<valid attribute='"'/>

Comments

All five special characters must not be escaped in comments:

<?xml version="1.0"?>
<valid>
<!-- "'<>& -->
</valid>

CDATA

All five special characters must not be escaped in CDATA sections:

<?xml version="1.0"?>
<valid>
<![CDATA["'<>&]]>
</valid>

Processing instructions

All five special characters must not be escaped in XML processing instructions:

<?xml version="1.0"?>
<?process <"'&> ?>
<valid/>

XML vs. HTML

HTML has its own set of escape codes which cover a lot more characters.

Python – How to parse XML and count instances of a particular node attribute

I suggest ElementTree. There are other compatible implementations of the same API, such as lxml, and cElementTree in the Python standard library itself; but, in this context, what they chiefly add is even more speed -- the ease of programming part depends on the API, which ElementTree defines.

First build an Element instance root from the XML, e.g. with the XML function, or by parsing a file with something like:

import xml.etree.ElementTree as ET
root = ET.parse('thefile.xml').getroot()

Or any of the many other ways shown at ElementTree. Then do something like:

for type_tag in root.findall('bar/type'):
    value = type_tag.get('foobar')
    print(value)

And similar, usually pretty simple, code patterns.