如何在两个goroutine之间共享一个HTTP请求实例? [英] How to share one HTTP request instance beween two goroutines?
问题描述
我有一些代码，现在需要3个请求来填充3个变量.两个请求是相同的.我想在两个不同的功能之间共享一个http请求(在现实世界中，这些功能分为两个不同的模块).
让我用比现实世界中简单得多的例子来描述我所遇到的问题.
目前，我具有以下主要功能和Post数据结构:
type Post struct {
ID int `json:"id"`
Title string `json:"title"`
UserID int `json:"userId"`
isCompleted bool `json:"completed"`
}
func main() {
var wg sync.WaitGroup
fmt.Println("Hello, world.")
wg.Add(3)
var firstPostID int
var secondPostID int
var secondPostName string
go func() {
firstPostID = getFirstPostID()
defer wg.Done()
}()
go func() {
secondPostID = getSecondPostID()
defer wg.Done()
}()
go func() {
secondPostName = getSecondPostName()
defer wg.Done()
}()
wg.Wait()
fmt.Println("first post id is", firstPostID)
fmt.Println("second post id is", secondPostID)
fmt.Println("second post title is", secondPostName)
}
有三个goroutine，所以我有3个并发请求，我使用sync.Workgroup同步所有内容.以下代码是请求的实现:
func makeRequest(url string) Post {
resp, err := http.Get(url)
if err != nil {
// handle error
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
var post Post
json.Unmarshal(body, &post)
return post
}
func makeFirstPostRequest() Post {
return makeRequest("https://jsonplaceholder.typicode.com/todos/1")
}
func makeSecondPostRequest() Post {
return makeRequest("https://jsonplaceholder.typicode.com/todos/2")
}
这里是功能的实现，该功能从获取的帖子中提取所需的信息:
func getFirstPostID() int {
var result = makeFirstPostRequest()
return result.ID
}
func getSecondPostID() int {
var result = makeSecondPostRequest()
return result.ID
}
func getSecondPostName() string {
var result = makeSecondPostRequest()
return result.Title
}
因此，目前我有3个并发请求，这很完美.问题是我不希望有2个绝对相同的单独HTTP请求来获取第二篇文章.一个就足够了.因此，我要实现的是对发布1和发布2的2个并发请求.我希望对makeSecondPostRequest的第二次调用不要创建新的HTTP请求，而是共享现有的请求(由第一次调用发送).
我该如何实现?
注意:例如，以下代码是如何使用JavaScript做到这一点的方法.
let promise = null;
function makeRequest() {
if (promise) {
return promise;
}
return promise = fetch('https://jsonplaceholder.typicode.com/todos/1')
.then(result => result.json())
// clean up cache variable, so any next request in the future will be performed again
.finally(() => (promise = null))
}
function main() {
makeRequest().then((post) => {
console.log(post.id);
});
makeRequest().then((post) => {
console.log(post.title);
});
}
main();
虽然您可以将诸如promises之类的东西放在一起，但在这种情况下，这不是必需的.
您的代码是以程序方式编写的.您已经编写了非常具体的函数，这些函数从Post中提取了特定的内容，而将其余的内容排除了.相反，请将您的Post放在一起.
package main
import(
"fmt"
"encoding/json"
"net/http"
"sync"
)
type Post struct {
ID int `json:"id"`
Title string `json:"title"`
UserID int `json:"userId"`
isCompleted bool `json:"completed"`
}
func fetchPost(id int) Post {
url := fmt.Sprintf("https://jsonplaceholder.typicode.com/todos/%v", id)
resp, err := http.Get(url)
if err != nil {
panic("HTTP error")
}
defer resp.Body.Close()
// It's more efficient to let json Decoder handle the IO.
var post Post
decoder := json.NewDecoder(resp.Body)
err = decoder.Decode(&post)
if err != nil {
panic("Decoding error")
}
return post
}
func main() {
var wg sync.WaitGroup
wg.Add(2)
var firstPost Post
var secondPost Post
go func() {
firstPost = fetchPost(1)
defer wg.Done()
}()
go func() {
secondPost = fetchPost(2)
defer wg.Done()
}()
wg.Wait()
fmt.Println("First post ID is", firstPost.ID)
fmt.Println("Second post ID is", secondPost.ID)
fmt.Println("Second post title is", secondPost.Title)
}
现在可以缓存帖子，而不是缓存响应.为此，我们可以添加PostManager来处理帖子的提取和缓存.
请注意，正常的map不能安全地同时使用，因此我们使用 sync.Map 作为我们的缓存.
type PostManager struct {
sync.Map
}
func (pc *PostManager) Fetch(id int) Post {
post, ok := pc.Load(id)
if ok {
return post.(Post)
}
post = pc.fetchPost(id)
pc.Store(id, post)
return post.(Post)
}
func (pc *PostManager) fetchPost(id int) Post {
url := fmt.Sprintf("https://jsonplaceholder.typicode.com/todos/%v", id)
resp, err := http.Get(url)
if err != nil {
panic("HTTP error")
}
defer resp.Body.Close()
// It's more efficient to let json Decoder handle the IO.
var post Post
decoder := json.NewDecoder(resp.Body)
err = decoder.Decode(&post)
if err != nil {
panic("Decoding error")
}
return post
}
PostManager方法必须使用指针接收器，以避免在sync.Map内部复制互斥锁.
我们使用PostManager，而不是直接获取帖子.
func main() {
var postManager PostManager
var wg sync.WaitGroup
wg.Add(2)
var firstPost Post
var secondPost Post
go func() {
firstPost = postManager.Fetch(1)
defer wg.Done()
}()
go func() {
secondPost = postManager.Fetch(2)
defer wg.Done()
}()
wg.Wait()
fmt.Println("First post ID is", firstPost.ID)
fmt.Println("Second post ID is", secondPost.ID)
fmt.Println("Second post title is", secondPost.Title)
}
通过使用条件请求检查是否已缓存，可以改善PostManager的缓存帖子是否已更改.
它的锁定也可以得到改善，因为它有可能在同一时间获取相同的Post.我们可以使用 singleflight 来解决此问题，从而只允许一次呼叫
type PostManager struct {
group singleflight.Group
cached sync.Map
}
func (pc *PostManager) Fetch(id int) Post {
post,ok := pc.cached.Load(id)
if !ok {
// Multiple calls with the same key at the same time will only run the code once, but all calls get the result.
post, _, _ = pc.group.Do(strconv.Itoa(id), func() (interface{}, error) {
post := pc.fetchPost(id)
pc.cached.Store(id, post)
return post, nil
})
}
return post.(Post)
}
I have some code that makes 3 requests to fill 3 variables now. Two requests are same. I want to share one http request between two different functions (in real world, these functions are splitted into two different modules).
Let me describe the problem what I have based on much simpler example than I have in real world.
At the moment, I have the following main function and Post data structure:
type Post struct {
ID int `json:"id"`
Title string `json:"title"`
UserID int `json:"userId"`
isCompleted bool `json:"completed"`
}
func main() {
var wg sync.WaitGroup
fmt.Println("Hello, world.")
wg.Add(3)
var firstPostID int
var secondPostID int
var secondPostName string
go func() {
firstPostID = getFirstPostID()
defer wg.Done()
}()
go func() {
secondPostID = getSecondPostID()
defer wg.Done()
}()
go func() {
secondPostName = getSecondPostName()
defer wg.Done()
}()
wg.Wait()
fmt.Println("first post id is", firstPostID)
fmt.Println("second post id is", secondPostID)
fmt.Println("second post title is", secondPostName)
}
There are three goroutines, so I have 3 concurrent requests, I sync everything using sync.Workgroup. The following code is implementation of the requests:
func makeRequest(url string) Post {
resp, err := http.Get(url)
if err != nil {
// handle error
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
var post Post
json.Unmarshal(body, &post)
return post
}
func makeFirstPostRequest() Post {
return makeRequest("https://jsonplaceholder.typicode.com/todos/1")
}
func makeSecondPostRequest() Post {
return makeRequest("https://jsonplaceholder.typicode.com/todos/2")
}
Here is implementation of functions which pulls needed information from fetched posts:
func getFirstPostID() int {
var result = makeFirstPostRequest()
return result.ID
}
func getSecondPostID() int {
var result = makeSecondPostRequest()
return result.ID
}
func getSecondPostName() string {
var result = makeSecondPostRequest()
return result.Title
}
So, at the moment I have 3 concurrent requests, this works perfectly. The problem is I don't want 2 absolutely same separate HTTP requests to fetch the second post. One would be enough. So, what I want to achieve is 2 concurrent requests for post 1 and post 2. I want second call to makeSecondPostRequest not to create new HTTP request, but share the existing one (which was sent by the first call).
How I can achieve this?
Note: the following code is how this can be done using JavaScript, for example.
let promise = null;
function makeRequest() {
if (promise) {
return promise;
}
return promise = fetch('https://jsonplaceholder.typicode.com/todos/1')
.then(result => result.json())
// clean up cache variable, so any next request in the future will be performed again
.finally(() => (promise = null))
}
function main() {
makeRequest().then((post) => {
console.log(post.id);
});
makeRequest().then((post) => {
console.log(post.title);
});
}
main();
While you could put something together like promises, in this case it's not necessary.
Your code is written in a procedural fashion. You've written very specific functions which pull specific little bits off the Post and throw out the rest. Instead, keep your Post together.
package main
import(
"fmt"
"encoding/json"
"net/http"
"sync"
)
type Post struct {
ID int `json:"id"`
Title string `json:"title"`
UserID int `json:"userId"`
isCompleted bool `json:"completed"`
}
func fetchPost(id int) Post {
url := fmt.Sprintf("https://jsonplaceholder.typicode.com/todos/%v", id)
resp, err := http.Get(url)
if err != nil {
panic("HTTP error")
}
defer resp.Body.Close()
// It's more efficient to let json Decoder handle the IO.
var post Post
decoder := json.NewDecoder(resp.Body)
err = decoder.Decode(&post)
if err != nil {
panic("Decoding error")
}
return post
}
func main() {
var wg sync.WaitGroup
wg.Add(2)
var firstPost Post
var secondPost Post
go func() {
firstPost = fetchPost(1)
defer wg.Done()
}()
go func() {
secondPost = fetchPost(2)
defer wg.Done()
}()
wg.Wait()
fmt.Println("First post ID is", firstPost.ID)
fmt.Println("Second post ID is", secondPost.ID)
fmt.Println("Second post title is", secondPost.Title)
}
Now instead of caching responses you can cache Posts. We can do this by adding a PostManager to handle fetching and caching Posts.
Note that normal map is not safe for concurrent use, so we use sync.Map for our cache.
type PostManager struct {
sync.Map
}
func (pc *PostManager) Fetch(id int) Post {
post, ok := pc.Load(id)
if ok {
return post.(Post)
}
post = pc.fetchPost(id)
pc.Store(id, post)
return post.(Post)
}
func (pc *PostManager) fetchPost(id int) Post {
url := fmt.Sprintf("https://jsonplaceholder.typicode.com/todos/%v", id)
resp, err := http.Get(url)
if err != nil {
panic("HTTP error")
}
defer resp.Body.Close()
// It's more efficient to let json Decoder handle the IO.
var post Post
decoder := json.NewDecoder(resp.Body)
err = decoder.Decode(&post)
if err != nil {
panic("Decoding error")
}
return post
}
PostManager methods must take a pointer receiver to avoid copying the mutex inside sync.Map.
And instead of fetching Posts directly, we use the PostManager.
func main() {
var postManager PostManager
var wg sync.WaitGroup
wg.Add(2)
var firstPost Post
var secondPost Post
go func() {
firstPost = postManager.Fetch(1)
defer wg.Done()
}()
go func() {
secondPost = postManager.Fetch(2)
defer wg.Done()
}()
wg.Wait()
fmt.Println("First post ID is", firstPost.ID)
fmt.Println("Second post ID is", secondPost.ID)
fmt.Println("Second post title is", secondPost.Title)
}
PostManager's caching would be improved by using conditional requests to check if the cached Post has changed or not.
Its locking can also be improved, as written its possible to fetch the same Post at the same time. We can fix this using singleflight to allow only one call to fetchPost with a given ID to happen at a time.
type PostManager struct {
group singleflight.Group
cached sync.Map
}
func (pc *PostManager) Fetch(id int) Post {
post,ok := pc.cached.Load(id)
if !ok {
// Multiple calls with the same key at the same time will only run the code once, but all calls get the result.
post, _, _ = pc.group.Do(strconv.Itoa(id), func() (interface{}, error) {
post := pc.fetchPost(id)
pc.cached.Store(id, post)
return post, nil
})
}
return post.(Post)
}
这篇关于如何在两个goroutine之间共享一个HTTP请求实例?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
