C-Gate 2.9.5 on Linux - thread count

Discussion in 'C-Gate Developers' started by more-solutions, Sep 5, 2013.

  1. more-solutions

    more-solutions

    Joined:
    Apr 23, 2006
    Messages:
    263
    Likes Received:
    3
    Location:
    Peterborough, UK
    I had C-gate 2.8.0 running on a Linux box (32-bit) which I have just upgraded.

    To make the 2.8.0 install stable I had to restart C-Gate nightly - I don't recall exactly why now but I think CPU and memory usage crept up over time and restarting at 3am every morning was a quick fix. With the upgrade to 2.9.5 I've taken that restart out to see if the system can be stable without it.

    Checking this morning (it's been running less than 24 hours) C-Gate is now using 15% of available RAM (it was 8% last night), and the number of threads is growing (currently 1265, it was 1261 when I started writing this post).

    Suggestions as to what might be causing this?

    I have level 9 logs if needed.

    (Thread count now 1269...)
     
    more-solutions, Sep 5, 2013
    #1
    1. Advertisements

  2. more-solutions

    daniel C-Busser Moderator

    Joined:
    Jul 26, 2004
    Messages:
    761
    Likes Received:
    14
    Location:
    Adelaide
    Hmm, we'd have to take a look at those logs.

    Assuming C-Gate has already been running a while you could execute "get cgate *" (to dump memory & thread info to the log) every 30s for three minutes, then grab just that log snippet and send to us.
     
    daniel, Sep 6, 2013
    #2
    1. Advertisements

  3. more-solutions

    more-solutions

    Joined:
    Apr 23, 2006
    Messages:
    263
    Likes Received:
    3
    Location:
    Peterborough, UK
    Log sent to cis.errors@clipsal.com.au

    Looking at the log, however, it appears that the extra threads correlate to my application code opening connections on 20024/5 which never get logged as having closed. netstat confirms that I do not have 1000+ active threads (I have 19 in CLOSE_WAIT state at time of writing). I am obviously doing something wrong here and will investigate how these threads are being closed, but I suspect that C-Gate should still be killing those threads at some point?

    Mark
     
    more-solutions, Sep 6, 2013
    #3
  4. more-solutions

    more-solutions

    Joined:
    Apr 23, 2006
    Messages:
    263
    Likes Received:
    3
    Location:
    Peterborough, UK
    I was wrong on this point: the application code I thought we had running accessing C-Gate on 20024 hasn't been in use since 2011...

    However, what we do have is the "monit" monitoring application configured to check that C-Gate is functioning correctly. It tests that it can connect to 20023 (and receives "201 Service Ready" in response), and that it can connect to 20024/20025/20123. In other words it will make TCP connections to those ports, and then drop the connection without sending anything, as part of a monitoring program to ensure C-Gate is responding correctly.

    Is there a reason why this should cause a problem, or is this an issue in C-Gate?

    It does appear to be repeatable:
    Code:
    telnet localhost 20024
    Ctrl-]
    quit
    
    .. causes the thread count to increase by one. I don't know of any way to close these connections other than dropping the connection in this way?
     
    more-solutions, Sep 6, 2013
    #4
  5. more-solutions

    ashleigh Moderator

    Joined:
    Aug 4, 2004
    Messages:
    2,346
    Likes Received:
    3
    Location:
    Adelaide, South Australia
    Normally when a TCP connection is closed, the socket is left in a partially closed state, and then some time later it gets fully closed and destroyed. The operating system manages all this (and you can see that state using netstat or tcpstat depending on operating system).

    Threads in an program associated with sockets are a whole separate kettle of fishies.
     
    ashleigh, Sep 7, 2013
    #5
  6. more-solutions

    more-solutions

    Joined:
    Apr 23, 2006
    Messages:
    263
    Likes Received:
    3
    Location:
    Peterborough, UK
    Indeed...

    Looking at this on Windows now (to avoid any potential differences with Linux), I started with a thread count of 20 (from "get cgate *").

    After telnet to localhost on 20024 and quitting telnet, this was up to 21 threads. netstat reports the inbound state as FIN_WAIT_2 state and the outbound state as CLOSE_WAIT. If I wait until both of these have cleared from the netstat output C-Gate still reports a thread count of 21.

    This is repeatable. I'm pretty sure this is a resource leak in C-Gate.

    Interestingly if I run the same test with port 20023 (closing telnet rather than issuing a quit command for consistency) the TCP states are the same (as would be expected) but the C-Gate thread count (correctly) goes up to 22 on connection and returns to 21 immediately I kill the connection. I would expect the same to be true of connections to 20024/20025?

    Hope this helps!
     
    more-solutions, Sep 8, 2013
    #6
  7. more-solutions

    daniel C-Busser Moderator

    Joined:
    Jul 26, 2004
    Messages:
    761
    Likes Received:
    14
    Location:
    Adelaide
    Awesome info, thanks Mark! I agree with your conclusion that this looks like an issue on ports 20024/20025. I have logged this. CG-1565
     
    daniel, Sep 9, 2013
    #7
  8. more-solutions

    more-solutions

    Joined:
    Apr 23, 2006
    Messages:
    263
    Likes Received:
    3
    Location:
    Peterborough, UK
    That is quite probably the first time I've had a bug report described as "awesome", you've just made my day! Thanks for that!
     
    more-solutions, Sep 9, 2013
    #8
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.