How to timeout a semaphore

Semaphore in Go is implemented with a channel:

An example is this:
https://sites.google.com/site/gopatterns/concurrency/semaphores

Context:

We have a few hundred servers and there are shared resources that we want to limit access to. So for a given resource, we want to use a semaphore to limit access to only 5 concurrent access by those servers. In order to do that, we are planning to use a lock server. When a machine accesses the resource, it will first register with the lock server that it is accessing the resource by a key. And then when it is done, it will send another request to the lock server to say that its done and release the semaphore. This ensures that we limit access to those resources to a maximal number of concurrent access.

Problem: Want to handle this gracefully if something goes wrong.

Question:

How do you go about implementing a timeout on the semaphore?

Example:

Let's say I have a semaphore size of 5. There are simultaneously 10 processes trying to acquire a lock in the semaphore so in this case only 5 will acquire it.

Sometimes, processes will die without responding (the real reason is a bit complicated to explain, but basically sometimes the process might not unlock it) so that causes a problem as a space in the semaphore is now permanently locked.

So I would like to have a timeout on this. Here are some issues:

The processes will run from anywhere between 2 seconds up to 60 minutes.

We have some race conditions, because if it times out and then the process tries to unlock it, then we have unlocked the semaphore twice instead of once. And vice versa, we unlock it first and then it times out.

How do I take the suggested pattern posted above and turn this into a thread-safe semaphore with timeouts?

It's a little difficult to figure out what you're trying to accomplish, but from what I can tell, you're trying to have concurrent goroutines access a shared resource and handle it gracefully if something doesn't go well. I have a couple suggestions on how you could handle this.

1. Use a WaitGroup from the sync package: http://golang.org/pkg/sync/#example_WaitGroup

With this strategy you basically add to a counter before each call to a new goroutine and use a defer to make sure it removes from the counter (so whether it times out or returns for another reason, it will still remove from the counter). Then you use a <code>wg.Wait()</code> command to make sure it doesn't go any further until all go routines have been returned. Here is an example: http://play.golang.org/p/wnm24TcBZg Note that without the <code>wg.Wait()</code> it will not wait on the go routines to finish before returning from main and terminating.

2. Use a time.Ticker to auto time out: http://golang.org/pkg/time/#Ticker

This approach will basically set a timer which will fire off at a set interval. You can use this timer to control time based events. Basically this has to run in a for loop that waits for the channel to be fed a tick, like in this example: http://play.golang.org/p/IHeqmiFBSS

Again, not entirely sure what you're trying to accomplish, but you may consider combining these two approaches so that if your process times out and sits in a loop the ticker will catch it and return after a set amount of time and call the defer <code>wg.Done()</code> function so that the part of the code thats waiting on it moves on. Hope this was at least a little helpful.

Since you are making a distributed lock service, I assume your lock server listens on a port, and when you accept() a connection you loop, waiting for commands in a goroutine per connection. And that goroutine exits when the socket is dropped (ie: remote node crash)

So, assuming that is true, you can do a couple things.

1. create a channel with a depth matching how many concurrent locks
2. when you lock, send a message to the channel (it will block if full)
3. when you unlock, just read a message from the channel
4. you can "defer release()" (where release consumes a message if you have already locked)

Here's a rough working example, all but the socket stuff.
Hopefully it makes sense.
http://play.golang.org/p/DLOX7m8m6q

package main

import "fmt"

import "time"

type Locker struct {
	ch chan int
	locked bool
}

func (l *Locker) lock(){
	l.ch <- 1
	l.locked=true
}
func (l *Locker) unlock() {
	if l.locked { // called directly or via defer, make sure we don't unlock if we don't have the lock
		l.locked = false // avoid unlocking twice if socket crashes after unlock
		<- l.ch
	}
}
	
func dostuff(name string, locker Locker) {
	locker.lock()
	defer locker.unlock()
	fmt.Println(name,"Doing stuff")
	time.Sleep(1 * time.Second)
}

func main() {
	ch := make(chan int, 2)
	go dostuff("1",Locker{ch,false})
	go dostuff("2",Locker{ch,false})
	go dostuff("3",Locker{ch,false})
	go dostuff("4",Locker{ch,false})
	time.Sleep(4 * time.Second)
}

ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
sem := semaphore.NewWeighted(int64(10))

if err := sem.Acquire(ctx, 1); err != nil {
 // - error means timeout else lock
}

Some assumptions:

• You need around 5 servers to get past the lock server at a time.
• Access to that resource are shortish and similar in length.

Use a quota server instead of a lock server. replenish the quota (a simple counter) at 5x the average (mean, 75th, etc) access/lock time. Only replenish the quota if it less than max. That way on average you'll maintain about 5 concurrent accesses/locks.

Some advanced features:

• If the shared resource can detect it's own load it could tell the quota server it can take more or fewer concurrent accesses.
• The servers can ping the quota server when they get done. This is not required, but frees up the resource sooner.

Maybe this helps, but I think this implementation is too expansive
I will appreciate any suggestions about the code.

package main
import (
"fmt"
"time"
"math/rand"
"strconv"
)
type Empty interface{}
type Semaphore struct {
dur time.Duration
ch  chan Empty
}
func NewSemaphore(max int, dur time.Duration) (sem *Semaphore) {
sem = new(Semaphore)
sem.dur = dur
sem.ch = make(chan Empty, max)
return
}
type Timeout struct{}
type Work struct{}
var null Empty
var timeout Timeout
var work Work
var global = time.Now()
func (sem *Semaphore) StartJob(id int, job func()) {
sem.ch <- null
go func() {
ch := make(chan interface{})
go func() {
time.Sleep(sem.dur)
ch <- timeout
}()
go func() {
fmt.Println("Job ", strconv.Itoa(id), " is started", time.Since(global))
job()
ch <- work
}()
switch (<-ch).(type) {
case Timeout:
fmt.Println("Timeout for job ", strconv.Itoa(id), time.Since(global))
case Work:
fmt.Println("Job ", strconv.Itoa(id), " is finished", time.Since(global))
}
<-sem.ch
}()
}
func main() {
rand.Seed(time.Now().Unix())
sem := NewSemaphore(3, 3*time.Second)
for i := 0; i < 10; i++ {
id := i
go sem.StartJob(i, func() {
seconds := 2 + rand.Intn(5)
fmt.Println("For job ", strconv.Itoa(id), " was allocated ", seconds, " secs")
time.Sleep(time.Duration(seconds) * time.Second)
})
}
time.Sleep(30 * time.Second)
}