在多线程应用程序中使用VTD [英] Using VTD in MultiThreaded App

问题描述

由于我有大量的xml，因此我有很多xpath可以从xml中提取条目，所以我试图产生多个线程，以便每个xpath可以在不同的线程中求值，但是我在下面遇到了错误是可以给出一个很不错的主意的代码段，为了简洁起见，我在这里使用了一个非常小的xml.我正在创建3个线程，并在10个任务中排队.

I have huge xmls as a result I have lot of xpaths to pull out the entries from the xml.So I am trying to spawn multiple threads so that each xpath can get evaluated in a different thread.But I am getting errors below is the code snippet which could give a fair idea, I have used a very small xml here for brevity purpose.I am creating 3 threads and queueing in 10 tasks.

import java.io.File;
import java.util.*;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;

import com.ximpleware.AutoPilot;
import com.ximpleware.EOFException;
import com.ximpleware.EncodingException;
import com.ximpleware.EntityException;
import com.ximpleware.ParseException;
import com.ximpleware.VTDGen;
import com.ximpleware.VTDNav;

public class MultiThread {
    public static void main(String args[]) throws InterruptedException, ExecutionException, EncodingException, EOFException, EntityException, ParseException
    {
        String str="<library><booked>book</booked> <book id=\"1\"> <title>Googled By God</title>  </book> </library>";
        File f = new File("/home/cloudera/wos.xml");
        byte[] ba =null;;
        ExecutorService executor = Executors.newFixedThreadPool(3);
        List<Task> extractorTasks = new ArrayList<Task>();
        VTDGen vg = new VTDGen();
        vg.setDoc(str.getBytes());
        vg.parse(false);


        //add 10 tasks
        for(int i=0;i<10;i++)
        {

            Task d = new Task(str.getBytes(),vg,"/library/book/title");
            extractorTasks.add(d);
        }

        List<Future<String>> output = executor.invokeAll(extractorTasks);
        executor.shutdown();    
    }
}
class Task implements Callable<String> {

    VTDGen vg = null;
    String xpath = "";
    byte [] ba=null;
    AutoPilot ap = null;
    Task(byte[] _ba,VTDGen _vg,String _xpath)
    {
        ba = _ba;
        vg = _vg;
        xpath = _xpath;
    }
    public String call() throws Exception 
    {

        String title = "";
        try 
        {
            /* if we uncomment below 3 lines, all works well, thats becuase we are reparsing the whole document*/
            //vg = new VTDGen();
            //vg.setDoc(ba);
            //vg.parse(false);

            VTDNav vn = vg.getNav();
            ap = new AutoPilot(vn);
            ap.selectXPath(xpath);

            //Get all the titles and print each of those
            while(ap.evalXPath() != -1)
            {
                //getText will return the index of  the VTDRecord
                int titleIndex = vn.getText();
                //Get the text of the VTDRecord
                title = vn.toNormalizedString(titleIndex);
                System.out.println("Title is "+title);
            }

            vn.toElement(VTDNav.ROOT);

        }  
        catch (Exception e) {
            e.printStackTrace();
        }

        return title;
    }

}

推荐答案

我能够找到一个修复程序.我将VTDNav存储在一个变量中，并将其重复项"传递给每个任务. GetNav()调用会清除内部状态，这也可能导致VTDnav失效，因此保留导航器的副本并传递导航器的副本就可以解决问题.

I was able to find a fix.I am Storing VTDNav in a variable and passing the "duplicate" of it to each task. The GetNav() call cleans internal state which perhaps also results in invalidating of VTDnav, so keeping a copy of the navigator and passing the duplicate of navigator did the trick.

import java.io.File;
import java.util.*;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;

import com.ximpleware.AutoPilot;
import com.ximpleware.EOFException;
import com.ximpleware.EncodingException;
import com.ximpleware.EntityException;
import com.ximpleware.ParseException;
import com.ximpleware.VTDGen;
import com.ximpleware.VTDNav;

public class MultiThread {
    public static void main(String args[]) throws InterruptedException, ExecutionException, EncodingException, EOFException, EntityException, ParseException
    {
        String str="<library><booked>book</booked> <book id=\"1\"> <title>Googled By God</title>  </book> </library>";
        File f = new File("/home/cloudera/wos.xml");
        byte[] ba =null;;
        ExecutorService executor = Executors.newFixedThreadPool(5);
        List<Task> extractorTasks = new ArrayList<Task>();
        VTDGen vg = new VTDGen();
        vg.setDoc(str.getBytes());
        vg.parse(false);

        //The GetNav() call cleans internal state , so keep a copy of VTDNav
        VTDNav vn = vg.getNav();

        for(int i=0;i<100;i++)
        {
            //pass the duplicates of navigator
            Task d = new Task(str.getBytes(),vn.duplicateNav(),"/library/book/title");
            extractorTasks.add(d);
        }

        List<Future<String>> output = executor.invokeAll(extractorTasks);
        executor.shutdown();    
    }
}
class Task implements Callable<String> {

    VTDGen vg = null;
    String xpath = "";
    byte [] ba=null;
    VTDNav vn = null;
    AutoPilot ap = null;
    Task(byte[] _ba,VTDNav _vn,String _xpath)
    {
        ba = _ba;
        vn = _vn;
        xpath = _xpath;
    }
    public String call() throws Exception 
    {

        String title = "";
        try 
        {
            ap = new AutoPilot(vn);
            //Thread.sleep(500);
            ap.selectXPath(xpath);

            //Get all the titles and print each of those
            while(ap.evalXPath() != -1)
            {
                //getText will return the index of  the VTDRecord
                int titleIndex = vn.getText();
                //Get the text of the VTDRecord
                title = vn.toNormalizedString(titleIndex);
                System.out.println("Title is "+title);
            }

            //if(vn.toElement(VTDNav.ROOT) == true)
            //  System.out.println("to element failed");

        }  
        catch (Exception e) {
            e.printStackTrace();
            System.out.println("Message is "+e.getMessage());
        }

        return title;
    }

}

这篇关于在多线程应用程序中使用VTD的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！