解决Rest API调用中的竞争条件 [英] Solving race condition in rest api calls

问题描述

我有一个REST API，用于收集有关第三方应用程序使用的urls的信息.安装了一个嗅探器，该嗅探器正在寻找HTTP请求，然后调用API来提供有关所使用的URL和标头的信息.

I have an REST API which is ment to collect information about urls used by 3rd party applications. There is a sniffer installed which is looking for HTTP request and then call the API providing information about urls and headers used.

我希望存储的URL尽可能唯一，以便每次搜索用户:

I would like to store URLs as unique as possible so each search for user:

http://localhost/users/49a95b87-083e-475b-9278-bade6f24413b
http://localhost/users/508f2a55-fe5b-4b83-b853-7e829dd366b8
http://localhost/users/af48be64-ad48-4867-ac06-984ce064dbeb

将作为一个条目存储在数据库中(uuid在这里并不重要，因此只有最后或最后一个会保留).这样做的算法是这样的:

would be stored as one entry to database (uuid is not important here so it is oke that only last or first will stay). Algorithm to do so is like that:

1. 获取请求
2. 检查数据库是否包含类似的网址
3. 如果数据库不返回任何结果，则将URL存储到数据库

1. Get request
2. Check if database contains similiar url
3. If database return no findings store url to database

控制者在哪里:

    @PreAuthorize("hasAuthority('ROLE_API')")
    @PostMapping(value = "/api/webapp")
    public synchronized ResponseEntity<Status> getWebApp(@RequestBody ServiceDiscovery req) throws InterruptedException {
        return webAppService.processWebAppRequest(req.getWebApp());
    }

服务方法:webAppService.processWebAppRequest

private final static String UUID_PATTERN = "[a-zA-Z0-9]{8}-[a-zA-Z0-9]{4}-[a-zA-Z0-9]{4}-[a-zA-Z0-9]{4}-[a-zA-Z0-9]{12}";
    private final static String SIM_NUMBER_PATTERN = "[0-9]{19}";
    private final static String MSISDN_SHORT_PATTERN = "[0-9]{9}";
    private final static String MSISDN_LONG_PATTERN = "[0-9]{11}";

    @Transactional
    public ResponseEntity<Status> processScanWebAppRequest(ServiceDiscovery serviceDiscovery){
        try {
            Optional<WebApp> wa = checkRegexes(serviceDiscovery.getUrl());
            if (wa.isPresent()){
                updateExistingWebApplication(serviceDiscovery, wa.get());
            } else {
                saveNewWebApplication(serviceDiscovery);
            }
        } catch (IncorrectResultSizeDataAccessException ex) {
            return new ResponseEntity<Status>(new Status("Processing error"), HttpStatus.PRECONDITION_FAILED);
        } 
        return new ResponseEntity<Status>(new Status("OK"), HttpStatus.OK);
    }
    //Checkregex function which return Optional.empty it looks for regex in string and then replace finding with regex itself so it can be used in JPA query
    private Optional<WebApp> checkRegexes(String url) {
        String urlToLookFor = url.replaceAll(UUID_PATTERN,UUID_PATTERN);
        urlToLookFor=urlToLookFor.replaceAll(SIM_NUMBER_PATTERN,SIM_NUMBER_PATTERN);
        urlToLookFor=urlToLookFor.replaceAll(MSISDN_LONG_PATTERN,MSISDN_LONG_PATTERN);
        urlToLookFor=urlToLookFor.replaceAll(MSISDN_SHORT_PATTERN,MSISDN_SHORT_PATTERN);

        return waRepository.getWebAppByRegex(urlToLookFor+"$");
    }
    //JPA check for 
    @Query(value="select * from webapp wa where wa.url ~ :url", nativeQuery = true)
    Optional<WebApp> getWebAppByRegex(@Param("url") String url);

在我测试时一切正常.但是在生产环境中，我定期收到大量请求(几天之内一天一次〜5k请求)，在该请求中，有许多UUID用于同一端点发送，并且正在并行处理，因此在执行checkRegexes时没有找到，但1秒后有5.

And everything works good while I test it. But on the production environment I got periodically huge amount of requests (once a day ~5k requests in few seconds) where there are plenty of UUIDs for the same endpoint sent which are being processed parallel so when checkRegexes is being done there is no finding but 1 second later there are 5.

为避免这种情况，我尝试将API设置为synchronized，但没有成功.无论如何，是否可以在不更改客户端的情况下使它正常工作?

To avoid this I tried to set API to synchronized but no success. Is there anyway to make it work without changes to the client?

使用查询的示例输出是:

example output using query is:

db=# select inserted, url from webapp where url ~ 'https://api-gateway.intra.com/users/[a-zA-Z0-9]{8}-[a-zA-Z0-9]{4}-[a-zA-Z0-9]{4}-[a-zA-Z0-9]{4}-[a-zA-Z0-9]{12}$' order by inserted;
      inserted       |                                        url
---------------------+-----------------------------------------------------------------------------------
 2019-10-30 10:55:00 | https://api-gateway.intra.com/users/cd1da115-a2c1-4722-a381-6d524cbf5c95
 2019-10-30 10:55:00 | https://api-gateway.intra.com/users/6b9b2c76-f416-4c8c-a0b0-01d29e976c03
 2019-10-30 10:55:00 | https://api-gateway.intra.com/users/0f197568-0d2f-405d-8468-3bbea8b3a8ef
 2019-10-30 10:55:00 | https://api-gateway.intra.com/users/32e02581-02b4-4a99-9121-1592b0a67566
 2019-10-30 10:55:00 | https://api-gateway.intra.com/users/f1bbb4cf-1336-45ec-814f-77ecb9736c47
 2019-10-30 10:55:00 | https://api-gateway.intra.com/users/6b0969a1-cfd7-4cda-9041-14229ddb3c6f
 2019-10-30 10:55:00 | https://api-gateway.intra.com/users/1fa4e464-aae6-472e-9a4c-cf4c7d6b4e0f
 2019-10-30 10:55:00 | https://api-gateway.intra.com/users/a25cbb9c-49c8-4603-aeb6-68d5a065c58e
 2019-10-30 10:55:00 | https://api-gateway.intra.com/users/e9be9949-f866-4765-9d38-582193dd1839
 2019-10-30 11:10:43 | https://api-gateway.intra.com/users/ad46db32-82dd-4c8e-9e77-a67e59dfca9a
 2019-10-30 11:10:43 | https://api-gateway.intra.com/users/fed07e4a-42e4-44e3-b932-9d0e01a5b535
 2019-10-30 11:10:43 | https://api-gateway.intra.com/users/747ff6df-dea5-48db-a15b-b7f43bc48ecf
 2019-10-30 11:10:43 | https://api-gateway.intra.com/users/4b6341ad-2584-427d-898a-f24d3623eb32
 2019-10-30 11:10:43 | https://api-gateway.intra.com/users/1e75c8bb-fb27-4183-a993-6763fa796e79
 2019-10-30 11:10:43 | https://api-gateway.intra.com/users/72b19740-7f61-4a32-88f9-0f7196d47853
 2019-10-30 11:10:43 | https://api-gateway.intra.com/users/6c21da91-56eb-498b-91bd-4895be3cfdcc
 2019-10-30 11:10:43 | https://api-gateway.intra.com/users/a7d3dc5c-c5de-4fc4-9c56-9eefe7a67d80
 2019-10-30 11:10:43 | https://api-gateway.intra.com/users/b9658963-c1f5-45da-b78d-240bc7b2a225
 2019-10-30 11:10:43 | https://api-gateway.intra.com/users/e8be9c46-e7d0-4642-9c7b-981663aa552e
 2019-10-30 11:10:43 | https://api-gateway.intra.com/users/74886e61-150d-4f35-8af0-c911cfdbf009
 2019-10-30 11:10:43 | https://api-gateway.intra.com/users/175840ab-9f4c-43b7-a684-1adeb884af71
 2019-10-30 11:10:43 | https://api-gateway.intra.com/users/7655bf87-1f34-4a38-9ad0-11c5c1ddbb6b
 2019-10-30 11:10:43 | https://api-gateway.intra.com/users/34406b3d-319d-4ca4-9ca2-d53c104ad703
 2019-10-30 11:10:43 | https://api-gateway.intra.com/users/15f40dc6-5852-4f9f-9017-87b20dd326f1
 2019-10-30 11:10:43 | https://api-gateway.intra.com/users/de5caa58-cb7b-4dac-9b87-b6f4460072f3
 2019-10-30 11:10:43 | https://api-gateway.intra.com/users/b601b86b-ba8e-4768-b013-12d12f74362b
 2019-10-30 11:10:43 | https://api-gateway.intra.com/users/a64fd22b-f144-4b70-b3bb-86ee0e7a47c6
 2019-10-30 11:10:43 | https://api-gateway.intra.com/users/87672dac-ce7d-4a03-acfc-e694d229c4fe
 2019-10-30 11:10:43 | https://api-gateway.intra.com/users/ea6db1e8-461b-459a-b5d0-2f4aec872cdd
 2019-10-30 11:10:43 | https://api-gateway.intra.com/users/d4b7cd3c-68bd-42e4-99d0-0f5043428080
 2019-10-30 11:24:11 | https://api-gateway.intra.com/users/4dfd42cc-4029-46dc-8867-adafac249345
 2019-10-30 11:24:11 | https://api-gateway.intra.com/users/2be89a7a-cf26-4163-bd49-0549bba24cd0
 2019-10-30 11:24:11 | https://api-gateway.intra.com/users/3d1f3f34-a11d-4f8a-b5d5-7d8fc58e49f0
 2019-10-30 11:24:11 | https://api-gateway.intra.com/users/db1ec2c3-5571-4db0-a3fe-6e12a4b1ae46
 2019-10-30 11:24:11 | https://api-gateway.intra.com/users/75f964aa-db57-43d3-a424-99345c8c9997

推荐答案

我建议您同步对数据库的访问.因此select，save和update操作应位于同步块中.但是它将减少并发到单线程，并且性能下降将是巨大的.因此，您可以使用一些技巧来同步数据库访问，但可以一定程度地节省性能.

I would suggest you to synchronize access to DB. So select,save and update operations should be in synchronized block. But it will reduce concurrency to single thread and performance degradation will be enormous. So there is some trick you could use to synchronize the DB access but save the performance at some rate.

您可以通过混杂的url的哈希值同步数据库访问. 我使用url_hash %10000来限制锁定对象的数量. 当然，这会影响性能，但是比简单同步好10000.

You could synchronize DB access by hash of the mormilized url . I use url_hash %10000 to limit number of lock objects. Of course it will affect performance but it will be in 10000 better then simple synchronization.

看看代码:

private final static String UUID_PATTERN = "[a-zA-Z0-9]{8}-[a-zA-Z0-9]{4}-[a-zA-Z0-9]{4}-[a-zA-Z0-9]{4}-[a-zA-Z0-9]{12}";
private final static String SIM_NUMBER_PATTERN = "[0-9]{19}";
private final static String MSISDN_SHORT_PATTERN = "[0-9]{9}";
private final static String MSISDN_LONG_PATTERN = "[0-9]{11}";

private final Map<Integer, Object> locks = new ConcurrentHashMap<>();

@Transactional
public ResponseEntity<Status> processScanWebAppRequest(ServiceDiscovery serviceDiscovery){
    try {
        String urlToLookFor = normalizeUrl(serviceDiscovery.getUrl());
        int lockHash = urlToLookFor.hashCode() % 10000;
        synchronized (locks.computeIfAbsent(lockHash, integer -> new Object())) {
            Optional<WebApp> wa = checkRegexes(urlToLookFor);
            if (wa.isPresent()){
                updateExistingWebApplication(serviceDiscovery, wa.get());
            } else {
                saveNewWebApplication(serviceDiscovery);
            }
        }
    } catch (IncorrectResultSizeDataAccessException ex) {
        return new ResponseEntity<Status>(new Status("Processing error"), HttpStatus.PRECONDITION_FAILED);
    }
    return new ResponseEntity<Status>(new Status("OK"), HttpStatus.OK);
}

private String normalizeUrl(String url) {
    String urlToLookFor = url.replaceAll(UUID_PATTERN,UUID_PATTERN);
    urlToLookFor=urlToLookFor.replaceAll(SIM_NUMBER_PATTERN,SIM_NUMBER_PATTERN);
    urlToLookFor=urlToLookFor.replaceAll(MSISDN_LONG_PATTERN,MSISDN_LONG_PATTERN);
    urlToLookFor=urlToLookFor.replaceAll(MSISDN_SHORT_PATTERN,MSISDN_SHORT_PATTERN);
    return urlToLookFor;
}

private Optional<WebApp> checkRegexes(String url) {
    return waRepository.getWebAppByRegex(urlToLookFor+"$");
}

这篇关于解决Rest API调用中的竞争条件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！