检查发送到网页的请求数 [英] Check the number of requests sent to a webpage

问题描述

我正在编写一个Java多线程应用程序，它可以访问不同Web服务器的数百万甚至数十亿个URL.想法是检查这些URL是否给出有效的200OK响应或404/某些其他代码.

I am writing a Java multithreaded application that hits millions and sometimes billions of URLs of different web servers. The idea is to check if those URLs gives a valid 200OK response or 404/some other code.

我怎么知道我的程序是否在服务器上没有引起高流量?我不希望发生DOS攻击.每台服务器都有将近800万个URL.

How can I know if my program is not causing high traffic on their servers? I don't want a DOS attack to happen.There are almost ~ 8 million URLs of each server .

为检查流量，我创建了一个简单的网页，托管在http://localhost:8089/上.我的应用程序一次访问此页面的次数不是200万次，而是一次一遍.在不使网络流量超载的意义上，我想了解代码的效率.

To check traffic this I created a simple webpage hosted at http://localhost:8089/ .My application is hitting this page 2 million times not all at once but one by one . I want to know efficiency of my code in the sense I don't overload network traffic.

netstat在TIME_WAIT中显示很多线程.

netstat shows a lot of threads in TIME_WAIT.

我如何知道此页面的访问量.我是否可以使用任何软件来跟踪此信息?或任何Linux命令或其他内容.我也乐于接受建议.

How can I know the traffic to this page. Is there any software I can use to track this?or any Linux command or something . I am open to suggestions too.

推荐答案

使用同步地图记住您访问每个服务器的时间.

Remember the time at which you accessed each server, using a synchronized Map.

一种方法是检查以前的访问时间，并在必要时休眠方法本身:

One approach is to check the previous access time, and if necessary, sleep in the method itself:

private static final Map<URL, Instant> lastAccessTimes = new HashMap<>();

private static final Duration COOLDOWN = Duration.ofSeconds(60);

public void contact(URL url)
throws IOException,
       InterruptedException {

    URL server = new URL(url.getProtocol() + ":" + url.getAuthority());

    Instant now = Instant.now();
    long delay = 0;

    synchronized (lastAccessTimes) {
        Instant newAccessTime = now;

        Instant lastAccess = lastAccessTimes.get(server);
        if (lastAccess != null) {
            Instant soonestAllowed = lastAccess.plus(COOLDOWN);
            if (now.isBefore(soonestAllowed)) {
                newAccessTime = soonestAllowed;
                delay = now.until(soonestAllowed, ChronoUnit.NANOS);
            }
        }

        lastAccessTimes.put(server, newAccessTime);
    }

    if (delay > 0) {
        TimeUnit.NANOSECONDS.sleep(delay);
    }

    URLConnection connection = url.openConnection();
    // etc.
}

如果您不想冒险占用线程池，可以在冷却时间过后使用执行程序重试.例如，使用默认线程池CompletableFuture:

If you don’t want to risk holding up your thread pool, you can use your executor to try again after the cooldown period. For example, using the default thread pool of CompletableFuture:

private static final Map<URL, Instant> lastAccessTimes = new HashMap<>();

private static final Duration COOLDOWN = Duration.ofSeconds(60);

public void contact(URL url) {
    URL server;
    try {
        server = new URL(url.getProtocol() + ":" + url.getAuthority());
    } catch (MalformedURLException e) {
        logger.log(Level.WARNING,
            "Could not extract server from " + url, e);
        return;
    }
    
    Instant now = Instant.now();

    synchronized (lastAccessTimes) {
        Instant lastAccess = lastAccessTimes.get(server);
        if (lastAccess != null) {
            Instant soonestAllowed = lastAccess.plus(COOLDOWN);
            if (now.isBefore(soonestAllowed)) {
                long delay = now.until(soonestAllowed, ChronoUnit.NANOS);

                CompletableFuture.runAsync(() -> contact(url),
                    CompletableFuture.delayedExecutor(
                        delay, TimeUnit.NANOSECONDS));
                return;
            }
        }

        lastAccessTimes.put(server, now);
    }

    try {
        URLConnection connection = url.openConnection();
        // etc.
    } catch (IOException e) {
        logger.log(Level.WARNING, "Could not contact " + url, e);
    }
}

这篇关于检查发送到网页的请求数的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！