添加了清漆禁令，但返回了旧对象 [英] Varnish ban is added but old object is returned
问题描述
我在图块服务器前使用清漆来缓存mapbox图块.为了删除旧的磁贴，我打算使用禁令来有效地删除大量缓存的磁贴.我的问题是，清漆仍然使用缓存的对象(至少响应中的 age 表明了这一点)，并且不与后端联系.
I'm using varnish in front of a tile server to cache mapbox tiles. To remove old tiles, I intended to use bans to effectively remove a large number of cached tiles. My problem is that varnish still uses the cached objects (at least the age in the response indicates this) and doesn't contact the backend.
我首先请求http://varnish/5/3/4.pbf，然后使用 curl -X BAN -H'X-Purge-Regex添加一个禁令:5/3/4.pbf'varnish 或 varnishadm ，然后 ban obj.http.url〜5/3/4.pbf ，然后请求http://varnish/5/再次是3/4.pbf.
I'm first requesting http://varnish/5/3/4.pbf, then adding a ban with curl -X BAN -H 'X-Purge-Regex: 5/3/4.pbf' varnish or alternatively varnishadm and then ban obj.http.url ~ 5/3/4.pbf and afterwards requesting http://varnish/5/3/4.pbf again.
一开始，我的禁令列表为空:
In the beginning my ban list is empty:
Present bans:
1610117471.434488 1 C
该禁令已成功添加 curl -X BAN -H'X-Purge-Regex:5/3/4.pbf'清漆
<!DOCTYPE html>
<html>
<head>
<title>200 Ban added</title>
</head>
<body>
<h1>Error 200 Ban added</h1>
<p>Ban added</p>
<h3>Guru Meditation:</h3>
<p>XID: 8</p>
<hr>
<p>Varnish cache server</p>
</body>
</html>
并显示在禁止列表中
Present bans:
1610117369.028870 0 - obj.http.url ~ 5/3/4.pbf
1610117307.220739 1 C
再次请求http://varnish/5/3/4.pbf之后，禁令列表表明该禁令已被使用
After requesting http://varnish/5/3/4.pbf again, the ban list indicates that the ban was used
Present bans:
1610117471.434488 1 - obj.http.url ~ 5/3/4.pbf
但是响应的年龄不为0，因为它仍然是第一个请求中的对象.
but the age of the response is not 0, because it's still the object from the first request.
不久后，该禁令被取消:
After a short time the ban is removed:
Present bans:
1610117471.434488 1 C
我的 vcl_recv 看起来像这样，但错误可能在其他地方，因为它也不适用于 varnishadm :
My vcl_recv looks like this but the error is probably somewhere else as it also doesn't work with varnishadm:
sub vcl_recv {
unset req.http.cookie;
# Allowing PURGE from localhost
if (req.method == "BAN"||req.method == "PURGE") {
if (!client.ip ~ purge) {
return(synth(405,"Not allowed."));
}
if (req.method == "BAN") {
ban("obj.http.url ~ " + req.http.X-Purge-Regex);
# Throw a synthetic page so the
# request won't go to the backend.
return(synth(200, "Ban added"));
}
if (req.method == "PURGE") {
return (purge);
}
}
}
我还尝试使用 https://stackoverflow.com/a/61507014 vcl_purge >但这似乎对禁令(?)没有帮助.
I also tried to use the vcl_purge from https://stackoverflow.com/a/61507014 but this doesn't seem to help for bans (?).
我使用的是 X-Purge-Regex 标头，不必担心必须转义特殊字符，例如https://stackoverflow.com/a/38526921 ，但是像 obj.http.url〜0 这样的禁令不起作用.
I'm using the X-Purge-Regex header to not worry about having to escape special characters like in https://stackoverflow.com/a/38526921 but just a ban like obj.http.url ~ 0 doesn't work.
我在vcl 4.0中使用清漆6.5.
I'm using varnish 6.5 with vcl 4.0.
* << Request >> 54
- Begin req 53 rxreq
- Timestamp Start: 1610121483.345437 0.000000 0.000000
- Timestamp Req: 1610121483.345437 0.000000 0.000000
- VCL_use boot
- ReqStart 192.168.48.2 50882 http
- ReqMethod BAN
- ReqURL /
- ReqProtocol HTTP/1.1
- ReqHeader Host: varnish-volatile
- ReqHeader User-Agent: curl/7.64.0
- ReqHeader Accept: */*
- ReqHeader X-Purge-Regex: 0
- ReqHeader X-Forwarded-For: 192.168.48.2
- VCL_call RECV
- VCL_acl MATCH purge "importer"
- VCL_return synth
- VCL_call HASH
- VCL_return lookup
- RespProtocol HTTP/1.1
- RespStatus 200
- RespReason Ban added
- RespHeader Date: Fri, 08 Jan 2021 15:58:03 GMT
- RespHeader Server: Varnish
- RespHeader X-Varnish: 54
- VCL_call SYNTH
- RespHeader Content-Type: text/html; charset=utf-8
- RespHeader Retry-After: 5
- VCL_return deliver
- Timestamp Process: 1610121483.347281 0.001844 0.001844
- RespHeader Content-Length: 246
- Storage malloc Transient
- Filters
- RespHeader Accept-Ranges: bytes
- RespHeader Connection: keep-alive
- Timestamp Resp: 1610121483.347557 0.002120 0.000276
- ReqAcct 98 0 98 218 246 464
- End
添加禁令后获取
* << Request >> 32806
- Begin req 32805 rxreq
- Timestamp Start: 1610121552.733872 0.000000 0.000000
- Timestamp Req: 1610121552.733872 0.000000 0.000000
- VCL_use boot
- ReqStart 192.168.48.1 55176 http
- ReqMethod GET
- ReqURL /public.snow_db/0/0/0.pbf
- ReqProtocol HTTP/1.1
- ReqHeader Host: localhost:8090
- ReqHeader User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:84.0) Gecko/20100101 Firefox/84.0
- ReqHeader Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
- ReqHeader Accept-Language: en-US,en;q=0.5
- ReqHeader Accept-Encoding: gzip, deflate
- ReqHeader DNT: 1
- ReqHeader Connection: keep-alive
- ReqHeader Upgrade-Insecure-Requests: 1
- ReqHeader Pragma: no-cache
- ReqHeader Cache-Control: no-cache
- ReqHeader X-Forwarded-For: 192.168.48.1
- VCL_call RECV
- ReqUnset Host: localhost:8090
- ReqHeader host: localhost:8090
- VCL_return hash
- ReqUnset Accept-Encoding: gzip, deflate
- ReqHeader Accept-Encoding: gzip
- VCL_call HASH
- VCL_return lookup
- Hit 28 601789.331504 10.000000 0.000000
- VCL_call HIT
- VCL_return deliver
- RespProtocol HTTP/1.1
- RespStatus 200
- RespReason OK
- RespHeader content-encoding: gzip
- RespHeader content-type: application/x-protobuf
- RespHeader date: Fri, 08 Jan 2021 15:09:02 GMT
- RespHeader Vary: Accept-Encoding
- RespHeader X-Varnish: 32806 28
- RespHeader Age: 3010
- RespHeader Via: 1.1 varnish (Varnish/6.5)
- VCL_call DELIVER
- VCL_return deliver
- Timestamp Process: 1610121552.734070 0.000197 0.000197
- Filters
- RespHeader Accept-Ranges: bytes
- RespHeader Content-Length: 295
- RespHeader Connection: keep-alive
- Timestamp Resp: 1610121552.734217 0.000345 0.000147
- ReqAcct 414 0 414 272 295 567
- End
重现问题
重现该错误:
-
git clone https://github.com/Baschdl/varnish-ban-setup.git&&cd varnish-ban-setup -
docker-compose up - 打开http://localhost:8092/5/3/1.pbf
-
docker-compose exec var varnishadm ban obj.http.url〜pbf - 再次打开http://localhost:8092/5/3/1.pbf，您将获得旧对象
推荐答案
您发出的 obj.http.url〜5/3/4.pbf 禁令与url 响应标头.
The obj.http.url ~ 5/3/4.pbf ban that you are issuing, is matching a url response header.
记住:该URL是请求标头，而不是响应标头.没有理由恐慌，您所做的事情完全可以理解，并且与所谓的 ban潜伏者的范围有关.
Remember: the URL is a request header, not a response header. No reason to panic, what you're doing makes perfect sense, and is related to the scope of the so-called ban lurker.
禁令潜伏者
禁令潜伏者是一个线程，该线程异步处理禁令列表上的禁令并将对象与禁令匹配，以便从缓存中删除对象模式.
Ban lurker
The ban lurker is a thread that asynchronously processes bans on the ban list and matches objects to the bans in order to remove patterns of objects from the cache.
禁令潜伏者不在请求范围内运行，而仅知道对象范围.
The ban lurker doesn't operate within a request scope, but only is aware of the object scope.
为了成功匹配请求信息，可以将请求上下文添加为响应头.这就是通过 obj.http.url
In order to successfully match request information, request context can be added as a response header. And that's what you're doing via obj.http.url
您的禁令无效的原因是，您没有在VCL文件中设置 obj.http.url .结果，禁令潜行者无法将任何对象与其匹配.
The reason why your ban isn't working, is because you didn't set obj.http.url in your VCL file. As a result, the ban lurker cannot match any objects to it.
解决方案很简单:在后端响应上下文中设置缺少的标头，如下所示:
The solution is simple: set the missing headers in the backend response context, as illustrated below:
sub vcl_backend_response {
set beresp.http.url = bereq.url;
set beresp.http.host = bereq.http.host;
}
当后端做出响应时，并且在将对象存储到缓存中之前，我们可以设置缺少的标头.
When the backend responds, and right before the object gets stored in cache, we can set the missing headers.
之后，禁令潜伏者将能够将禁令表达式与正确的对象进行匹配，并将其从缓存中删除.
After that, the ban lurker will be able to match the ban expression to the right objects, and remove them from cache.
Don't forget the objects aren't immediately matched: they are only removed when they reach the
ban_lurker_age, which is set to 1 minute by default.
这篇关于添加了清漆禁令，但返回了旧对象的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
