dimanche 24 septembre 2023

How to arbitrarily stop goroutines from operating on input?

Let's say i am building a little script that brute forces paths on a url for pentesting purposes (like legal stuff :)), and i am building it like the code snippet below, however i would like to make a change in this code such that when a url starts to respond with 429 status codes (too many requests) which means we're being rate limited, i would like it to stop sending requests to that specific host and resume bruteforcing the rest of the other hosts what is the optimal way of doing this such that i reduce (or better yet, completely eliminate) the number of requests send to a host once it returns 429 status code?

a possible solution i thought about would be setting an array or urls that i should stop bruteforcing and access that array from the gorountines via a mutex, and before each request in the goroutines loop, access that array and only send the request at hand if the url if not in that array.

The reason i don't like this approach is that given a url will start with responding with 429, it will allow many requests to be sent before we can access the array via the mutex and append it to it.

Is there a better approach ? I'm also fine with changing the design pattern used for the concurrency here if it leads to a better solution.

urls := make(chan string)
for i := 0; i < concurrency; i++ {
  wg.Add(1)
    go func() {
      for url := range fileNamesChan {
        sendRequest(url)
      }
      wg.Done()
    }()  
}

for _, url := range inputURLS {
  for _, fileName := range fileNames {
    urls <- url + fileName
  }
}

close(urls)
wg.Wait()

I tried the solution suggested above, but like i said given a url will start with responding with 429, it will allow many requests to be sent before we can access the array via the mutex and append it to it.

Aucun commentaire:

Enregistrer un commentaire