Debugging extension state problems
The extension state, "NOT INUSE" or "INUSE" is managed by a pair of daemons on each of your servers. Even when having just a single server, there are still two daemons. One (devstatesender.php) is responsible of receiving events from asterisk, like call setup or call hnagup, the second one (devstatereceiver.php) receives the data from the sender, performs some computation and updates the Custom device states in Asterisk.
When there are problems with these softwares, regardless of the origin, you can experience an extension being in "INUSE" state while no one is on the phone, or having an extension in "NOT INUSE" state while is instead on the phone, so maybe receiving multiple calls at once.
If you have detecting problems, the best is to enable the collection of extensions events in Admin/Settings (Log Extension Events) and the enable logging (Device state logging). Once these settings are enabled, it is advisable to kill both devstatesender.php and devstatereceiver.php letting them to be restarted automatically.
In short, devstatesender.php receives the events from asterisk and communicates the change to devstatereceiver.php over port 19771. Devstatereceiver.php will update the database, table st_states and the Custom device state. Asterisk will use the Custom device state to update the Queue Member status.
When a problem arises, it is important to check where this process has failed.
Let's make an example. A client is calling saying he is not receiving calls on his phone, for example 100-DEVEL. You'll check the status, and find the phone in "INUSE" state while the client is not on call.
It is important to get all the info to be able to debug the issue, so take a note about the "server time" it is happening with "minute and second" precision and to get any useful info from the client about what kind of call has received or made in the time preceding the problem.
It can be useful to get the data of all three places where the status of the extension is stored and used. Let's make an example, the extension being reported "stuck" is 100-DEVEL, then in the database, get the status with
select * from st_states where st_extension='100-DEVEL'
Get the status of the custom device state
asterisk -rx 'devstate list' | grep 100-DEVEL
Get the status of the Queue member, if any
asterisk -rx 'queue show' | grep 100-DEVEL