Lesson 14 · 14 of 20

Worker pools · Go

Share work among a fixed number of goroutines in Go with a jobs channel and a results channel, and sort the results before printing them.

Go · Worker pools in DMarlon Studio: the example's code and its output in the terminal

A worker pool limits how many tasks run at once: instead of one goroutine per invoice, three goroutines share them all. It is what you do when each task uses a scarce resource, such as database connections or calls to an API.

There are two buffered channels: jobs, with the invoice numbers still to do, and results, with one line of text per invoice. worker receives them with a direction: <-chan int for receiving only and chan<- string for sending only, so the compiler stops you using them the wrong way round.

Each worker runs for id := range jobs: it takes an invoice, adds 21% VAT, sends the result and goes back for another. The three compete for the same jobs, and which one does each job changes from run to run. close(jobs) says there is no more work, and their loops end.

main knows there will be exactly one result per invoice, so for range nets receives len(nets) times and needs no WaitGroup. Reading the nets map from several goroutines at once is safe because nobody writes to it.

The invoices are sent by looping over a map, whose order is random, and the workers finish whenever they finish: results arrive shuffled. `sort.Strings(lines)` puts them in order (every line starts with # and a three-digit number), so the output is always the same.

Code · main.go

package mainimport (	"fmt"	"sort")func worker(nets map[int]float64, jobs <-chan int, results chan<- string) {	for id := range jobs { // takes invoice IDs until jobs is closed		results <- fmt.Sprintf("#%d %7.2f -> %7.2f", id, nets[id], nets[id]*1.21)	}}func main() {	nets := map[int]float64{101: 120, 102: 80, 103: 45.5, 104: 300, 105: 19.99, 106: 64}	jobs := make(chan int, len(nets))	results := make(chan string, len(nets))	for w := 1; w <= 3; w++ { // a pool of three workers shares the jobs		go worker(nets, jobs, results)	}	for id := range nets { // map order is random, and so is the order of results		jobs <- id	}	close(jobs) // no more work: the workers' loops end	var lines []string	for range nets { // exactly one result per job		lines = append(lines, <-results)	}	sort.Strings(lines) // sort before printing: the same output every run	for _, line := range lines {		fmt.Println(line)	}	fmt.Println(len(lines), "invoices, VAT added by 3 workers")}

Output · go run main.go

#101  120.00 ->  145.20
#102   80.00 ->   96.80
#103   45.50 ->   55.05
#104  300.00 ->  363.00
#105   19.99 ->   24.19
#106   64.00 ->   77.44
6 invoices, VAT added by 3 workers

Key points

A fixed pool of workers limits how much runs at once.
Workers read from a jobs channel with range.
close(jobs) ends the workers' loops.
If you know how many results there will be, receive exactly that many.
Results arrive in any order: sort them before printing.

Exercise

Change the number of workers to 1 and then to 6, and check that the output does not change. Then pass a worker number to worker and add it at the end of each line: you will see the output stop being the same every time.