Golang在服务器没有可用的后备队列时，TCP连接变慢。

huangapple
go评论116阅读模式
英文:

Golang TCP Connection Slow when Server Doesn't Have Backlog Available

问题

更新

通过添加了pthreaded C客户端，重新创建了问题，表明长连接时间是TCP协议的一部分，而不是特定实现的一部分。修改这些协议似乎不容易。

初始问题

我相信我的问题主要是：当使用Golang的net包尝试通过TCP连接到服务器时，以下情况会发生：

  1. 服务器没有可用的连接，即使在后台也是如此。
  2. 连接没有被拒绝/失败。

连接到服务器的连接似乎存在大量开销，服务器响应时间从5 ms增加到几秒钟。这在生产环境和下面的最小示例中都看到了。正确的解决方案是使用连接池连接到服务器，这将被实现。这主要是我的好奇心。

复现步骤：

  1. 使用backlog = 1运行服务器，运行client.go。
    • 所有50个goroutine同时触发，总完成时间接近2分钟。
  2. 使用backlog = 100运行服务器，运行client.go。
    • 所有50个goroutine同时触发，连接到服务器排队，并在~260 ms内完成。
  3. 运行三个使用50 us重试时间的C客户端，平均能够在12 ms内完成连接，因此没有看到这个问题。

backlog = 1的示例输出（第一个时间是拨号时间，第二个时间是完成时间）：

  1. user@computer ~/tcp-tests $ go run client.go 127.0.0.1:46999
  2. Long Elapsed Time: 216.579μs, 315.196μs
  3. Long Elapsed Time: 274.169μs, 5.970873ms
  4. Long Elapsed Time: 74.4μs, 10.753871ms
  5. Long Elapsed Time: 590.965μs, 205.851066ms
  6. Long Elapsed Time: 1.029287689s, 1.029574065s
  7. Long Elapsed Time: 1.02945649s, 1.035098229s
  8. ...
  9. Long Elapsed Time: 3.045881865s, 6.378597166s
  10. Long Elapsed Time: 3.045314838s, 6.383783688s
  11. Time taken stats: 2.85 +/- 1.59 s // 平均值 +/- STDEV
  12. Main Taken: 6.384677948s

backlog = 100的示例输出：

  1. ...
  2. Long Elapsed Time: 330.098μs, 251.004077ms
  3. Long Elapsed Time: 298.146μs, 256.187795ms
  4. Long Elapsed Time: 315.832μs, 261.523685ms
  5. Time taken stats: 0.13 +/- 0.08 s
  6. Main Taken: 263.186955ms

那么，在net.DialTCP的内部发生了什么（我们还使用了其他类型的拨号，没有明显的区别），导致拨号时间增加？

  • 在尝试建立连接之间的轮询时间？
  • RFC 5681全局拥塞控制（可能包括互斥锁？）变量，在所有初始失败的连接尝试上递增？
  • 其他原因？

我倾向于前两者，因为1s，3s，5s的值似乎是魔数。它们在我的普通本地机器和大规模生产环境中都出现过。

这是用C编写的最小服务器。感兴趣的配置值是listen的backlog参数。

  1. /*
  2.     Adapted from
  3.     https://www.geeksforgeeks.org/tcp-server-client-implementation-in-c/
  4.     
  5.     Compile and run with:
  6.         gcc server.c -o server; ./server
  7. */
  8. #include <stdio.h>
  9. #include <string.h>
  10. #include <sys/socket.h>
  11. #include <arpa/inet.h>
  12. #include <unistd.h>
  13. #include <sys/time.h>
  15. int main(void)
  16. {
  17.     int socket_desc, client_sock, client_size;
  18.     struct sockaddr_in server_addr, client_addr;
  19.     char server_message[2000], client_message[2000];
  21.     // Clean buffers:
  22.     memset(server_message, '
    /*
  23. Adapted from
  24. https://www.geeksforgeeks.org/tcp-server-client-implementation-in-c/
  25. Compile and run with:
  26. gcc server.c -o server; ./server
  27. */
  28. #include <stdio.h>
  29. #include <string.h>
  30. #include <sys/socket.h>
  31. #include <arpa/inet.h>
  32. #include <unistd.h>
  33. #include <sys/time.h>
  34. int main(void)
  35. {
  36. int socket_desc, client_sock, client_size;
  37. struct sockaddr_in server_addr, client_addr;
  38. char server_message[2000], client_message[2000];
  39. // Clean buffers:
  40. memset(server_message, '\0', sizeof(server_message));
  41. memset(client_message, '\0', sizeof(client_message));
  42. // Create socket:
  43. socket_desc = socket(AF_INET, SOCK_STREAM, 0);
  44. if(socket_desc < 0){
  45. printf("Error while creating socket\n");
  46. return -1;
  47. }
  48. printf("Socket created successfully\n");
  49. // Set port and IP:
  50. server_addr.sin_family = AF_INET;
  51. server_addr.sin_port = htons(46999);
  52. server_addr.sin_addr.s_addr = inet_addr("127.0.0.1");
  53. // Bind to the set port and IP:
  54. if(bind(socket_desc, (struct sockaddr*)&server_addr, sizeof(server_addr))<0){
  55. printf("Couldn't bind to the port\n");
  56. return -1;
  57. }
  58. printf("Done with binding\n");
  59. // Listen for clients:
  60. // Increasing the backlog allows the Go client to connect and wait
  61. // rather than poll/retry.
  62. if(listen(socket_desc, 100) < 0){
  63. printf("Error while listening\n");
  64. return -1;
  65. }
  66. printf("\nListening for incoming connections.....\n");
  67. // Accept an incoming connection:
  68. client_size = sizeof(client_addr);
  69. int server_run = 1;
  70. do
  71. {
  72. struct timeval start, end;
  73. double cpu_time_used;
  74. gettimeofday(&start, NULL);
  75. client_sock = accept(socket_desc, (struct sockaddr*)&client_addr, &client_size);
  76. if (client_sock < 0){
  77. printf("Can't accept\n");
  78. return -1;
  79. }
  80. // Receive client's message:
  81. if (recv(client_sock, client_message, sizeof(client_message), 0) < 0){
  82. printf("Couldn't receive\n");
  83. return -1;
  84. }
  85. if (strcmp(client_message, "stop") == 0)
  86. {
  87. server_run = 0;
  88. printf("Received stop message.\n");
  89. }
  90. // Respond to client:
  91. strcpy(server_message, "This is the server's message.");
  92. if (send(client_sock, server_message, strlen(server_message), 0) < 0){
  93. printf("Can't send\n");
  94. return -1;
  95. }
  96. // sleep for 5 ms
  97. usleep(5000);
  98. // Closing the socket:
  99. close(client_sock);
  100. gettimeofday(&end, NULL);
  101. cpu_time_used = (end.tv_usec - start.tv_usec) / 1000.0;
  102. if (cpu_time_used > 0.0) // overflow in tv_usec if negative
  103. printf("Server Time: %.4f ms\n", cpu_time_used);
  104. } while(server_run);
  105. close(socket_desc);
  106. return 0;
  107. }
  108. ', sizeof(server_message));
  109.     memset(client_message, '
    /*
  110. Adapted from
  111. https://www.geeksforgeeks.org/tcp-server-client-implementation-in-c/
  112. Compile and run with:
  113. gcc server.c -o server; ./server
  114. */
  115. #include <stdio.h>
  116. #include <string.h>
  117. #include <sys/socket.h>
  118. #include <arpa/inet.h>
  119. #include <unistd.h>
  120. #include <sys/time.h>
  121. int main(void)
  122. {
  123. int socket_desc, client_sock, client_size;
  124. struct sockaddr_in server_addr, client_addr;
  125. char server_message[2000], client_message[2000];
  126. // Clean buffers:
  127. memset(server_message, '\0', sizeof(server_message));
  128. memset(client_message, '\0', sizeof(client_message));
  129. // Create socket:
  130. socket_desc = socket(AF_INET, SOCK_STREAM, 0);
  131. if(socket_desc < 0){
  132. printf("Error while creating socket\n");
  133. return -1;
  134. }
  135. printf("Socket created successfully\n");
  136. // Set port and IP:
  137. server_addr.sin_family = AF_INET;
  138. server_addr.sin_port = htons(46999);
  139. server_addr.sin_addr.s_addr = inet_addr("127.0.0.1");
  140. // Bind to the set port and IP:
  141. if(bind(socket_desc, (struct sockaddr*)&server_addr, sizeof(server_addr))<0){
  142. printf("Couldn't bind to the port\n");
  143. return -1;
  144. }
  145. printf("Done with binding\n");
  146. // Listen for clients:
  147. // Increasing the backlog allows the Go client to connect and wait
  148. // rather than poll/retry.
  149. if(listen(socket_desc, 100) < 0){
  150. printf("Error while listening\n");
  151. return -1;
  152. }
  153. printf("\nListening for incoming connections.....\n");
  154. // Accept an incoming connection:
  155. client_size = sizeof(client_addr);
  156. int server_run = 1;
  157. do
  158. {
  159. struct timeval start, end;
  160. double cpu_time_used;
  161. gettimeofday(&start, NULL);
  162. client_sock = accept(socket_desc, (struct sockaddr*)&client_addr, &client_size);
  163. if (client_sock < 0){
  164. printf("Can't accept\n");
  165. return -1;
  166. }
  167. // Receive client's message:
  168. if (recv(client_sock, client_message, sizeof(client_message), 0) < 0){
  169. printf("Couldn't receive\n");
  170. return -1;
  171. }
  172. if (strcmp(client_message, "stop") == 0)
  173. {
  174. server_run = 0;
  175. printf("Received stop message.\n");
  176. }
  177. // Respond to client:
  178. strcpy(server_message, "This is the server's message.");
  179. if (send(client_sock, server_message, strlen(server_message), 0) < 0){
  180. printf("Can't send\n");
  181. return -1;
  182. }
  183. // sleep for 5 ms
  184. usleep(5000);
  185. // Closing the socket:
  186. close(client_sock);
  187. gettimeofday(&end, NULL);
  188. cpu_time_used = (end.tv_usec - start.tv_usec) / 1000.0;
  189. if (cpu_time_used > 0.0) // overflow in tv_usec if negative
  190. printf("Server Time: %.4f ms\n", cpu_time_used);
  191. } while(server_run);
  192. close(socket_desc);
  193. return 0;
  194. }
  195. ', sizeof(client_message));
  197.     // Create socket:
  198.     socket_desc = socket(AF_INET, SOCK_STREAM, 0);
  200.     if(socket_desc < 0){
  201.         printf("Error while creating socket\n");
  202.         return -1;
  203.     }
  204.     printf("Socket created successfully\n");
  206.     // Set port and IP:
  207.     server_addr.sin_family = AF_INET;
  208.     server_addr.sin_port = htons(46999);
  209.     server_addr.sin_addr.s_addr = inet_addr("127.0.0.1");
  211.     // Bind to the set port and IP:
  212.     if(bind(socket_desc, (struct sockaddr*)&server_addr, sizeof(server_addr))<0){
  213.         printf("Couldn't bind to the port\n");
  214.         return -1;
  215.     }
  216.     printf("Done with binding\n");
  218.     // Listen for clients:
  219.     // Increasing the backlog allows the Go client to connect and wait
  220.     // rather than poll/retry.
  221.     if(listen(socket_desc, 100) < 0){
  222.         printf("Error while listening\n");
  223.         return -1;
  224.     }
  225.     printf("\nListening for incoming connections.....\n");
  227.     // Accept an incoming connection:
  228.     client_size = sizeof(client_addr);
  229.     int server_run = 1;
  230.     do
  231.     {
  232.         struct timeval start, end;
  233.         double cpu_time_used;
  234.         gettimeofday(&start, NULL);
  235.         client_sock = accept(socket_desc, (struct sockaddr*)&client_addr, &client_size);
  237.         if (client_sock < 0){
  238.             printf("Can't accept\n");
  239.             return -1;
  240.         }
  242.         // Receive client's message:
  243.         if (recv(client_sock, client_message, sizeof(client_message), 0) < 0){
  244.             printf("Couldn't receive\n");
  245.             return -1;
  246.         }
  247.         if (strcmp(client_message, "stop") == 0)
  248.         {
  249.             server_run = 0;
  250.             printf("Received stop message.\n");
  251.         }
  253.         // Respond to client:
  254.         strcpy(server_message, "This is the server's message.");
  256.         if (send(client_sock, server_message, strlen(server_message), 0) < 0){
  257.             printf("Can't send\n");
  258.             return -1;
  259.         }
  261.         // sleep for 5 ms
  262.         usleep(5000);
  264.         // Closing the socket:
  265.         close(client_sock);
  266.         gettimeofday(&end, NULL);
  267.         cpu_time_used = (end.tv_usec - start.tv_usec) / 1000.0;
  268.         if (cpu_time_used > 0.0) // overflow in tv_usec if negative
  269.             printf("Server Time: %.4f ms\n", cpu_time_used);
  270.     } while(server_run);
  272.     close(socket_desc);
  274.     return 0;
  275. }

这是测试用的Go客户端

  1. /*
  2.     Adapted from
  3.     https://www.linode.com/docs/guides/developing-udp-and-tcp-clients-and-servers-in-go/
  5.     Run once the server.c is compiled and running with:
  6.         go run client.go 127.0.0.1:46999
  7. */
  8. package main
  10. import (
  11.     "fmt"
  12.     "net"
  13.     "os"
  14.     "time"
  15.     "github.com/montanaflynn/stats"
  16.     "sync"
  17. )
  18. func do_message(wg *sync.WaitGroup, connect string, time_taken *float64) {
  19.     defer wg.Done()
  20.     message := make([]byte, 128)
  21.     start_time := time.Now()
  22.     pAddr, err := net.ResolveTCPAddr("tcp", connect)
  23.     if err != nil {
  24.         return
  25.     }
  27.     c, err := net.DialTCP("tcp", nil, pAddr)
  28.     if err != nil {
  29.         fmt.Println(err)
  30.         return
  31.     }
  32.     c.SetLinger(0)
  33.     dialed_time := time.Since(start_time)
  35.     defer func() {
  36.         c.Close()
  37.         elapsed_time := time.Since(start_time)
  38.         if elapsed_time.Microseconds() > 60 { // microseconds
  39.             fmt.Println("Long Elapsed Time: " + dialed_time.String() + ", " + elapsed_time.String())
  40.         }
  41.         *time_taken = float64(elapsed_time.Microseconds())
  42.     }()
  44.     text := "{\"service\": \"magic_service_str\"}"
  45.     c.Write([]byte(text))
  46.     code, _ := c.Read(message) // Does not actually wait for response.
  47.     code = code
  48. }
  49. func main() {
  50.     main_start := time.Now()
  51.     arguments := os.Args
  52.     if len(arguments) == 1 {
  53.             fmt.Println("Please provide host:port.")
  54.             return
  55.     }
  56.     n_messages := 50
  57.     wg := new(sync.WaitGroup)
  58.     wg.Add(n_messages)
  59.     times := make([]float64, n_messages)
  60.     for i := 0; i < n_messages; i++ {
  61.         // Used to turn the goroutines into serial implementation
  62.         // time.Sleep(5500 * time.Microsecond)
  63.         go do_message(wg, arguments[1], &times[i])
  64.     }
  65.     wg.Wait()
  66.     avg, _ := stats.Mean(times)
  67.     std, _ := stats.StandardDeviation(times)
  68.     fmt.Println("Time taken stats: " + fmt.Sprintf("%.2f", avg / 1000000.0) + " +/- " + fmt.Sprintf("%.2f", std / 1000000.0) + " s")
  69.     main_taken := time.Since(main_start)
  70.     fmt.Println("Main Taken: " + main_taken.String())
  71. }

在C中更新了pthreaded客户端，并确认问题不是Golang实现：

  1. // gcc client_p.c -o pclient -lpthread
  2. #include <stdio.h>
  3. #include <string.h>
  4. #include <sys/socket.h>
  5. #include <arpa/inet.h>
  6. #include <unistd.h>
  7. #include <stdlib.h>
  8. #include<sys/time.h>
  11. #include <pthread.h>
  12. #include <errno.h>
  14. #ifndef THREAD_LOOP_COUNT
  15. #define THREAD_LOOP_COUNT 1
  16. #endif
  18. /* Subtract the ‘struct timeval’ values X and Y,
  19.    storing the result in RESULT.
  20.    Return 1 if the difference is negative, otherwise 0.
  21.    https://www.gnu.org/software/libc/manual/html_node/Calculating-Elapsed-Time.html
  22. */
  24. int
  25. timeval_subtract (struct timeval *result, struct timeval *x, struct timeval *y)
  26. {
  27.   /* Perform the carry for the later subtraction by updating y. */
  28.   if (x->tv_usec < y->tv_usec) {
  29.     int nsec = (y->tv_usec - x->tv_usec) / 1000000 + 1;
  30.     y->tv_usec -= 1000000 * nsec;
  31.     y->tv_sec += nsec;
  32.   }
  33.   if (x->tv_usec - y->tv_usec > 1000000) {
  34.     int nsec = (x->tv_usec - y->tv_usec) / 1000000;
  35.     y->tv_usec += 1000000 * nsec;
  36.     y->tv_sec -= nsec;
  37.   }
  39.   /* Compute the time remaining to wait.
  40.      tv_usec is certainly positive. */
  41.   result->tv_sec = x->tv_sec - y->tv_sec;
  42.   result->tv_usec = x->tv_usec - y->tv_usec;
  44.   /* Return 1 if result is negative. */
  45.   return x->tv_sec < y->tv_sec;
  46. }
  49. static void* workerThreadFunc(void* arg)
  50. {
  51.     int socket_desc;
  52.     struct sockaddr_in server_addr;
  53.     char server_message[2000], client_message[2000];
  54.     // Clean buffers:
  55.     memset(server_message,'
    // gcc client_p.c -o pclient -lpthread
  56. #include <stdio.h>
  57. #include <string.h>
  58. #include <sys/socket.h>
  59. #include <arpa/inet.h>
  60. #include <unistd.h>
  61. #include <stdlib.h>
  62. #include<sys/time.h>
  63. #include <pthread.h>
  64. #include <errno.h>
  65. #ifndef THREAD_LOOP_COUNT
  66. #define THREAD_LOOP_COUNT 1
  67. #endif
  68. /* Subtract the ‘struct timeval’ values X and Y,
  69. storing the result in RESULT.
  70. Return 1 if the difference is negative, otherwise 0.
  71. https://www.gnu.org/software/libc/manual/html_node/Calculating-Elapsed-Time.html
  72. */
  73. int
  74. timeval_subtract (struct timeval *result, struct timeval *x, struct timeval *y)
  75. {
  76. /* Perform the carry for the later subtraction by updating y. */
  77. if (x->tv_usec < y->tv_usec) {
  78. int nsec = (y->tv_usec - x->tv_usec) / 1000000 + 1;
  79. y->tv_usec -= 1000000 * nsec;
  80. y->tv_sec += nsec;
  81. }
  82. if (x->tv_usec - y->tv_usec > 1000000) {
  83. int nsec = (x->tv_usec - y->tv_usec) / 1000000;
  84. y->tv_usec += 1000000 * nsec;
  85. y->tv_sec -= nsec;
  86. }
  87. /* Compute the time remaining to wait.
  88. tv_usec is certainly positive. */
  89. result->tv_sec = x->tv_sec - y->tv_sec;
  90. result->tv_usec = x->tv_usec - y->tv_usec;
  91. /* Return 1 if result is negative. */
  92. return x->tv_sec < y->tv_sec;
  93. }
  94. static void* workerThreadFunc(void* arg)
  95. {
  96. int socket_desc;
  97. struct sockaddr_in server_addr;
  98. char server_message[2000], client_message[2000];
  99. // Clean buffers:
  100. memset(server_message,'\0',sizeof(server_message));
  101. memset(client_message,'\0',sizeof(client_message));
  102. // Set port and IP the same as server-side:
  103. server_addr.sin_family = AF_INET;
  104. server_addr.sin_port = htons(46999);
  105. server_addr.sin_addr.s_addr = inet_addr("127.0.0.1");
  106. int retries = 0;
  107. struct timeval start, end, difference;
  108. double cpu_time_used;
  109. for(int i = 0; i < THREAD_LOOP_COUNT; i++)
  110. {
  111. gettimeofday(&start, NULL);
  112. // Create socket:
  113. socket_desc = socket(AF_INET, SOCK_STREAM, 0);
  114. if(socket_desc < 0){
  115. printf("Unable to create socket\n");
  116. return;
  117. }
  118. // Send connection request to server:
  119. while(connect(socket_desc, (struct sockaddr*)&server_addr, sizeof(server_addr)) < 0){
  120. retries++;
  121. if (retries > 10)
  122. {
  123. printf("Unable to connect\n");
  124. retries = 0;
  125. }
  126. usleep(5);
  127. }
  128. int retries = 0;
  129. // Send the message to server:
  130. if(send(socket_desc, client_message, strlen("client message."), 0) < 0){
  131. printf("Unable to send message\n");
  132. close(socket_desc);
  133. return;
  134. }
  135. // Receive the server's response:
  136. if(recv(socket_desc, server_message, sizeof(server_message), 0) < 0){
  137. printf("Error while receiving server's msg\n");
  138. close(socket_desc);
  139. return;
  140. }
  141. // Close the socket:
  142. close(socket_desc);
  143. gettimeofday(&end, NULL);
  144. timeval_subtract (&difference, &end, &start);
  145. double cpu_time_used = (double)difference.tv_sec + (double)difference.tv_usec / 1000000.0;
  146. printf("Client Time: %.4e s\n", cpu_time_used);
  147. }
  148. }
  149. int main(int argc, char **argv)
  150. {
  151. int n_threads = 50;  // default value
  152. if (argc > 1)
  153. n_threads = atoi(argv[1]);
  154. pthread_t *threads = (pthread_t*)malloc(n_threads * sizeof(pthread_t));
  155. struct timeval start, end, difference;
  156. gettimeofday(&start, NULL);
  157. for(int i = 0; i < n_threads; i++)
  158. {
  159. int createRet = pthread_create(&threads[i], NULL, workerThreadFunc, NULL);
  160. if (createRet != 0)
  161. {
  162. printf("failed to create thread\n");
  163. }
  164. }
  165. for(int i = 0; i < n_threads; i++)
  166. pthread_join(threads[i], NULL);
  167. gettimeofday(&end, NULL);
  168. timeval_subtract (&difference, &end, &start);
  169. double cpu_time_used = (double)difference.tv_sec + (double)difference.tv_usec / 1000000.0;
  170. printf("Total Client Time: %.4e s\n", cpu_time_used);
  171. free(threads);
  172. return 0;
  173. }
  174. ',sizeof(server_message));
  175.     memset(client_message,'
    // gcc client_p.c -o pclient -lpthread
  176. #include <stdio.h>
  177. #include <string.h>
  178. #include <sys/socket.h>
  179. #include <arpa/inet.h>
  180. #include <unistd.h>
  181. #include <stdlib.h>
  182. #include<sys/time.h>
  183. #include <pthread.h>
  184. #include <errno.h>
  185. #ifndef THREAD_LOOP_COUNT
  186. #define THREAD_LOOP_COUNT 1
  187. #endif
  188. /* Subtract the ‘struct timeval’ values X and Y,
  189. storing the result in RESULT.
  190. Return 1 if the difference is negative, otherwise 0.
  191. https://www.gnu.org/software/libc/manual/html_node/Calculating-Elapsed-Time.html
  192. */
  193. int
  194. timeval_subtract (struct timeval *result, struct timeval *x, struct timeval *y)
  195. {
  196. /* Perform the carry for the later subtraction by updating y. */
  197. if (x->tv_usec < y->tv_usec) {
  198. int nsec = (y->tv_usec - x->tv_usec) / 1000000 + 1;
  199. y->tv_usec -= 1000000 * nsec;
  200. y->tv_sec += nsec;
  201. }
  202. if (x->tv_usec - y->tv_usec > 1000000) {
  203. int nsec = (x->tv_usec - y->tv_usec) / 1000000;
  204. y->tv_usec += 1000000 * nsec;
  205. y->tv_sec -= nsec;
  206. }
  207. /* Compute the time remaining to wait.
  208. tv_usec is certainly positive. */
  209. result->tv_sec = x->tv_sec - y->tv_sec;
  210. result->tv_usec = x->tv_usec - y->tv_usec;
  211. /* Return 1 if result is negative. */
  212. return x->tv_sec < y->tv_sec;
  213. }
  214. static void* workerThreadFunc(void* arg)
  215. {
  216. int socket_desc;
  217. struct sockaddr_in server_addr;
  218. char server_message[2000], client_message[2000];
  219. // Clean buffers:
  220. memset(server_message,'\0',sizeof(server_message));
  221. memset(client_message,'\0',sizeof(client_message));
  222. // Set port and IP the same as server-side:
  223. server_addr.sin_family = AF_INET;
  224. server_addr.sin_port = htons(46999);
  225. server_addr.sin_addr.s_addr = inet_addr("127.0.0.1");
  226. int retries = 0;
  227. struct timeval start, end, difference;
  228. double cpu_time_used;
  229. for(int i = 0; i < THREAD_LOOP_COUNT; i++)
  230. {
  231. gettimeofday(&start, NULL);
  232. // Create socket:
  233. socket_desc = socket(AF_INET, SOCK_STREAM, 0);
  234. if(socket_desc < 0){
  235. printf("Unable to create socket\n");
  236. return;
  237. }
  238. // Send connection request to server:
  239. while(connect(socket_desc, (struct sockaddr*)&server_addr, sizeof(server_addr)) < 0){
  240. retries++;
  241. if (retries > 10)
  242. {
  243. printf("Unable to connect\n");
  244. retries = 0;
  245. }
  246. usleep(5);
  247. }
  248. int retries = 0;
  249. // Send the message to server:
  250. if(send(socket_desc, client_message, strlen("client message."), 0) < 0){
  251. printf("Unable to send message\n");
  252. close(socket_desc);
  253. return;
  254. }
  255. // Receive the server's response:
  256. if(recv(socket_desc, server_message, sizeof(server_message), 0) < 0){
  257. printf("Error while receiving server's msg\n");
  258. close(socket_desc);
  259. return;
  260. }
  261. // Close the socket:
  262. close(socket_desc);
  263. gettimeofday(&end, NULL);
  264. timeval_subtract (&difference, &end, &start);
  265. double cpu_time_used = (double)difference.tv_sec + (double)difference.tv_usec / 1000000.0;
  266. printf("Client Time: %.4e s\n", cpu_time_used);
  267. }
  268. }
  269. int main(int argc, char **argv)
  270. {
  271. int n_threads = 50;  // default value
  272. if (argc > 1)
  273. n_threads = atoi(argv[1]);
  274. pthread_t *threads = (pthread_t*)malloc(n_threads * sizeof(pthread_t));
  275. struct timeval start, end, difference;
  276. gettimeofday(&start, NULL);
  277. for(int i = 0; i < n_threads; i++)
  278. {
  279. int createRet = pthread_create(&threads[i], NULL, workerThreadFunc, NULL);
  280. if (createRet != 0)
  281. {
  282. printf("failed to create thread\n");
  283. }
  284. }
  285. for(int i = 0; i < n_threads; i++)
  286. pthread_join(threads[i], NULL);
  287. gettimeofday(&end, NULL);
  288. timeval_subtract (&difference, &end, &start);
  289. double cpu_time_used = (double)difference.tv_sec + (double)difference.tv_usec / 1000000.0;
  290. printf("Total Client Time: %.4e s\n", cpu_time_used);
  291. free(threads);
  292. return 0;
  293. }
  294. ',sizeof(client_message));
  295.     // Set port and IP the same as server-side:
  296.     server_addr.sin_family = AF_INET;
  297.     server_addr.sin_port = htons(46999);
  298.     server_addr.sin_addr.s_addr = inet_addr("127.0.0.1");
  299.     int retries = 0;
  300.     struct timeval start, end, difference;
  301.     double cpu_time_used;
  302.     for(int i = 0; i < THREAD_LOOP_COUNT; i++)
  303.     {
  304.         gettimeofday(&start, NULL);
  305.         // Create socket:
  306.         socket_desc = socket(AF_INET, SOCK_STREAM, 0);
  307.         if(socket_desc < 0){
  308.             printf("Unable to create socket\n");
  309.             return;
  310.         }
  311.         // Send connection request to server:
  312.         while(connect(socket_desc, (struct sockaddr*)&server_addr, sizeof(server_addr)) < 0){
  313.             retries++;
  314.             if (retries > 10)
  315.             {
  316.                 printf("Unable to connect\n");
  317.                 retries = 0;
  318.             }
  319.             usleep(5);
  320.         }
  321.         int retries = 0;
  323.         // Send the message to server:
  324.         if(send(socket_desc, client_message, strlen("client message."), 0) < 0){
  325.             printf("Unable to send message\n");
  326.             close(socket_desc);
  327.             return;
  328.         }
  330.         // Receive the server's response:
  331.         if(recv(socket_desc, server_message, sizeof(server_message), 0) < 0){
  332.             printf("Error while receiving server's msg\n");
  333.             close(socket_desc);
  334.             return;
  335.         }
  337.         // Close the socket:
  338.         close(socket_desc);
  339.         gettimeofday(&end, NULL);
  340.         timeval_subtract (&difference, &end, &start);
  341.         double cpu_time_used = (double)difference.tv_sec + (double)difference.tv_usec / 1000000.0;
  342.         printf("Client Time: %.4e s\n", cpu_time_used);
  343.     }
  344. }
  346. int main(int argc, char **argv)
  347. {
  348.     int n_threads = 50;  // default value
  349.     if (argc > 1)
  350.         n_threads = atoi(argv[1]);
  352. 	pthread_t *threads = (pthread_t*)malloc(n_threads * sizeof(pthread_t));
  354.     struct timeval start, end, difference;
  355.     gettimeofday(&start, NULL);
  356.     for(int i = 0; i < n_threads; i++)
  357.     {
  358. 	    int createRet = pthread_create(&threads[i], NULL, workerThreadFunc, NULL);
  359.         if (createRet != 0)
  360.         {
  361. 		    printf("failed to create thread\n");
  362. 	    }
  363.     }
  364.     for(int i = 0; i < n_threads; i++)
  365.         pthread_join(threads[i], NULL);
  366.     gettimeofday(&end, NULL);
  367.     timeval_subtract (&difference, &end, &start);
  368.     double cpu_time_used = (double)difference.tv_sec + (double)difference.tv_usec / 1000000.0;
  369.     printf("Total Client Time: %.4e s\n", cpu_time_used);
  370.     free(threads);
  371.     return 0;
  372. }
英文:

Update

With the added pthreaded C client, the problem is recreated, indicating the long connection times are part of the TCP protocol, rather than specific implementations. Altering those protocols doesn't seem easily available.

Initial Question

I believe my question is largely: What does the Golang net package do when attempting to connect to a server over TCP and:

  1. The server has no connections available, even in backlog.
  2. The connection is not refused/failed.

There seems to be a large amount of overhead in that connection with server response times ramping up from 5 ms to several seconds. This was seen both in a production environment and in the minimal example below. The proper solution is to use connection pools to the server, which will be implemented. This is largely my curiosity.

To reproduce:

  1. Run server with backlog = 1, run client.go.
    • All 50 goroutines fire at once, with a total completion time of almost 2 minutes.
  2. Run server with backlog = 100, run client.go.
    • All 50 goroutines fire at once, queue up connected to the server, and complete in ~260 ms.
  3. Running three C clients utilizing 50 us retry times was able to complete connections within 12 ms on average, so didn't see this issue.

Example output for backlog = 1 (first time is time to dial, second is time to completion):

  1. user@computer ~/tcp-tests $ go run client.go 127.0.0.1:46999
  2. Long Elapsed Time: 216.579&#181;s, 315.196&#181;s
  3. Long Elapsed Time: 274.169&#181;s, 5.970873ms
  4. Long Elapsed Time: 74.4&#181;s, 10.753871ms
  5. Long Elapsed Time: 590.965&#181;s, 205.851066ms
  6. Long Elapsed Time: 1.029287689s, 1.029574065s
  7. Long Elapsed Time: 1.02945649s, 1.035098229s
  8. ...
  9. Long Elapsed Time: 3.045881865s, 6.378597166s
  10. Long Elapsed Time: 3.045314838s, 6.383783688s
  11. Time taken stats: 2.85 +/- 1.59 s // average +/- STDEV
  12. Main Taken: 6.384677948s

Example output for backlog = 100:

  1. ...
  2. Long Elapsed Time: 330.098&#181;s, 251.004077ms
  3. Long Elapsed Time: 298.146&#181;s, 256.187795ms
  4. Long Elapsed Time: 315.832&#181;s, 261.523685ms
  5. Time taken stats: 0.13 +/- 0.08 s
  6. Main Taken: 263.186955ms

So what's going on under the hood of net.DialTCP (we used other flavors of dial as well, with no discernible difference) that causes the dial time to grow?

  • Polling time between attempts to make a connection?
  • An RFC 5681 Global Congestion Control (likely including mutex lock?) variable that gets incremented on all the initial failed connection attempts?
  • Something else?

I'm leaning towards the first two, as the 1s, 3s, 5s values seem to be magic numbers. They show up both on my modest local machine, and a large scale production environment.

Here is the minimal server written in C. The configuration value of interest is the backlog argument to listen.

  1. /*
  2.     Adapted from
  3.     https://www.geeksforgeeks.org/tcp-server-client-implementation-in-c/
  4.     
  5.     Compile and run with:
  6.         gcc server.c -o server; ./server
  7. */
  8. #include &lt;stdio.h&gt;
  9. #include &lt;string.h&gt;
  10. #include &lt;sys/socket.h&gt;
  11. #include &lt;arpa/inet.h&gt;
  12. #include &lt;unistd.h&gt;
  13. #include &lt;sys/time.h&gt;
  15. int main(void)
  16. {
  17.     int socket_desc, client_sock, client_size;
  18.     struct sockaddr_in server_addr, client_addr;
  19.     char server_message[2000], client_message[2000];
  21.     // Clean buffers:
  22.     memset(server_message, &#39;
    /*
  23. Adapted from
  24. https://www.geeksforgeeks.org/tcp-server-client-implementation-in-c/
  25. Compile and run with:
  26. gcc server.c -o server; ./server
  27. */
  28. #include &lt;stdio.h&gt;
  29. #include &lt;string.h&gt;
  30. #include &lt;sys/socket.h&gt;
  31. #include &lt;arpa/inet.h&gt;
  32. #include &lt;unistd.h&gt;
  33. #include &lt;sys/time.h&gt;
  34. int main(void)
  35. {
  36. int socket_desc, client_sock, client_size;
  37. struct sockaddr_in server_addr, client_addr;
  38. char server_message[2000], client_message[2000];
  39. // Clean buffers:
  40. memset(server_message, &#39;\0&#39;, sizeof(server_message));
  41. memset(client_message, &#39;\0&#39;, sizeof(client_message));
  42. // Create socket:
  43. socket_desc = socket(AF_INET, SOCK_STREAM, 0);
  44. if(socket_desc &lt; 0){
  45. printf(&quot;Error while creating socket\n&quot;);
  46. return -1;
  47. }
  48. printf(&quot;Socket created successfully\n&quot;);
  49. // Set port and IP:
  50. server_addr.sin_family = AF_INET;
  51. server_addr.sin_port = htons(46999);
  52. server_addr.sin_addr.s_addr = inet_addr(&quot;127.0.0.1&quot;);
  53. // Bind to the set port and IP:
  54. if(bind(socket_desc, (struct sockaddr*)&amp;server_addr, sizeof(server_addr))&lt;0){
  55. printf(&quot;Couldn&#39;t bind to the port\n&quot;);
  56. return -1;
  57. }
  58. printf(&quot;Done with binding\n&quot;);
  59. // Listen for clients:
  60. // Increasing the backlog allows the Go client to connect and wait
  61. // rather than poll/retry.
  62. if(listen(socket_desc, 100) &lt; 0){
  63. printf(&quot;Error while listening\n&quot;);
  64. return -1;
  65. }
  66. printf(&quot;\nListening for incoming connections.....\n&quot;);
  67. // Accept an incoming connection:
  68. client_size = sizeof(client_addr);
  69. int server_run = 1;
  70. do
  71. {
  72. struct timeval start, end;
  73. double cpu_time_used;
  74. gettimeofday(&amp;start, NULL);
  75. client_sock = accept(socket_desc, (struct sockaddr*)&amp;client_addr, &amp;client_size);
  76. if (client_sock &lt; 0){
  77. printf(&quot;Can&#39;t accept\n&quot;);
  78. return -1;
  79. }
  80. // Receive client&#39;s message:
  81. if (recv(client_sock, client_message, sizeof(client_message), 0) &lt; 0){
  82. printf(&quot;Couldn&#39;t receive\n&quot;);
  83. return -1;
  84. }
  85. if (strcmp(client_message, &quot;stop&quot;) == 0)
  86. {
  87. server_run = 0;
  88. printf(&quot;Received stop message.\n&quot;);
  89. }
  90. // Respond to client:
  91. strcpy(server_message, &quot;This is the server&#39;s message.&quot;);
  92. if (send(client_sock, server_message, strlen(server_message), 0) &lt; 0){
  93. printf(&quot;Can&#39;t send\n&quot;);
  94. return -1;
  95. }
  96. // sleep for 5 ms
  97. usleep(5000);
  98. // Closing the socket:
  99. close(client_sock);
  100. gettimeofday(&amp;end, NULL);
  101. cpu_time_used = (end.tv_usec - start.tv_usec) / 1000.0;
  102. if (cpu_time_used &gt; 0.0) // overflow in tv_usec if negative
  103. printf(&quot;Server Time: %.4f ms\n&quot;, cpu_time_used);
  104. } while(server_run);
  105. close(socket_desc);
  106. return 0;
  107. }
  108. &#39;, sizeof(server_message));
  109.     memset(client_message, &#39;
    /*
  110. Adapted from
  111. https://www.geeksforgeeks.org/tcp-server-client-implementation-in-c/
  112. Compile and run with:
  113. gcc server.c -o server; ./server
  114. */
  115. #include &lt;stdio.h&gt;
  116. #include &lt;string.h&gt;
  117. #include &lt;sys/socket.h&gt;
  118. #include &lt;arpa/inet.h&gt;
  119. #include &lt;unistd.h&gt;
  120. #include &lt;sys/time.h&gt;
  121. int main(void)
  122. {
  123. int socket_desc, client_sock, client_size;
  124. struct sockaddr_in server_addr, client_addr;
  125. char server_message[2000], client_message[2000];
  126. // Clean buffers:
  127. memset(server_message, &#39;\0&#39;, sizeof(server_message));
  128. memset(client_message, &#39;\0&#39;, sizeof(client_message));
  129. // Create socket:
  130. socket_desc = socket(AF_INET, SOCK_STREAM, 0);
  131. if(socket_desc &lt; 0){
  132. printf(&quot;Error while creating socket\n&quot;);
  133. return -1;
  134. }
  135. printf(&quot;Socket created successfully\n&quot;);
  136. // Set port and IP:
  137. server_addr.sin_family = AF_INET;
  138. server_addr.sin_port = htons(46999);
  139. server_addr.sin_addr.s_addr = inet_addr(&quot;127.0.0.1&quot;);
  140. // Bind to the set port and IP:
  141. if(bind(socket_desc, (struct sockaddr*)&amp;server_addr, sizeof(server_addr))&lt;0){
  142. printf(&quot;Couldn&#39;t bind to the port\n&quot;);
  143. return -1;
  144. }
  145. printf(&quot;Done with binding\n&quot;);
  146. // Listen for clients:
  147. // Increasing the backlog allows the Go client to connect and wait
  148. // rather than poll/retry.
  149. if(listen(socket_desc, 100) &lt; 0){
  150. printf(&quot;Error while listening\n&quot;);
  151. return -1;
  152. }
  153. printf(&quot;\nListening for incoming connections.....\n&quot;);
  154. // Accept an incoming connection:
  155. client_size = sizeof(client_addr);
  156. int server_run = 1;
  157. do
  158. {
  159. struct timeval start, end;
  160. double cpu_time_used;
  161. gettimeofday(&amp;start, NULL);
  162. client_sock = accept(socket_desc, (struct sockaddr*)&amp;client_addr, &amp;client_size);
  163. if (client_sock &lt; 0){
  164. printf(&quot;Can&#39;t accept\n&quot;);
  165. return -1;
  166. }
  167. // Receive client&#39;s message:
  168. if (recv(client_sock, client_message, sizeof(client_message), 0) &lt; 0){
  169. printf(&quot;Couldn&#39;t receive\n&quot;);
  170. return -1;
  171. }
  172. if (strcmp(client_message, &quot;stop&quot;) == 0)
  173. {
  174. server_run = 0;
  175. printf(&quot;Received stop message.\n&quot;);
  176. }
  177. // Respond to client:
  178. strcpy(server_message, &quot;This is the server&#39;s message.&quot;);
  179. if (send(client_sock, server_message, strlen(server_message), 0) &lt; 0){
  180. printf(&quot;Can&#39;t send\n&quot;);
  181. return -1;
  182. }
  183. // sleep for 5 ms
  184. usleep(5000);
  185. // Closing the socket:
  186. close(client_sock);
  187. gettimeofday(&amp;end, NULL);
  188. cpu_time_used = (end.tv_usec - start.tv_usec) / 1000.0;
  189. if (cpu_time_used &gt; 0.0) // overflow in tv_usec if negative
  190. printf(&quot;Server Time: %.4f ms\n&quot;, cpu_time_used);
  191. } while(server_run);
  192. close(socket_desc);
  193. return 0;
  194. }
  195. &#39;, sizeof(client_message));
  197.     // Create socket:
  198.     socket_desc = socket(AF_INET, SOCK_STREAM, 0);
  200.     if(socket_desc &lt; 0){
  201.         printf(&quot;Error while creating socket\n&quot;);
  202.         return -1;
  203.     }
  204.     printf(&quot;Socket created successfully\n&quot;);
  206.     // Set port and IP:
  207.     server_addr.sin_family = AF_INET;
  208.     server_addr.sin_port = htons(46999);
  209.     server_addr.sin_addr.s_addr = inet_addr(&quot;127.0.0.1&quot;);
  211.     // Bind to the set port and IP:
  212.     if(bind(socket_desc, (struct sockaddr*)&amp;server_addr, sizeof(server_addr))&lt;0){
  213.         printf(&quot;Couldn&#39;t bind to the port\n&quot;);
  214.         return -1;
  215.     }
  216.     printf(&quot;Done with binding\n&quot;);
  218.     // Listen for clients:
  219.     // Increasing the backlog allows the Go client to connect and wait
  220.     // rather than poll/retry.
  221.     if(listen(socket_desc, 100) &lt; 0){
  222.         printf(&quot;Error while listening\n&quot;);
  223.         return -1;
  224.     }
  225.     printf(&quot;\nListening for incoming connections.....\n&quot;);
  227.     // Accept an incoming connection:
  228.     client_size = sizeof(client_addr);
  229.     int server_run = 1;
  230.     do
  231.     {
  232.         struct timeval start, end;
  233.         double cpu_time_used;
  234.         gettimeofday(&amp;start, NULL);
  235.         client_sock = accept(socket_desc, (struct sockaddr*)&amp;client_addr, &amp;client_size);
  237.         if (client_sock &lt; 0){
  238.             printf(&quot;Can&#39;t accept\n&quot;);
  239.             return -1;
  240.         }
  242.         // Receive client&#39;s message:
  243.         if (recv(client_sock, client_message, sizeof(client_message), 0) &lt; 0){
  244.             printf(&quot;Couldn&#39;t receive\n&quot;);
  245.             return -1;
  246.         }
  247.         if (strcmp(client_message, &quot;stop&quot;) == 0)
  248.         {
  249.             server_run = 0;
  250.             printf(&quot;Received stop message.\n&quot;);
  251.         }
  253.         // Respond to client:
  254.         strcpy(server_message, &quot;This is the server&#39;s message.&quot;);
  256.         if (send(client_sock, server_message, strlen(server_message), 0) &lt; 0){
  257.             printf(&quot;Can&#39;t send\n&quot;);
  258.             return -1;
  259.         }
  261.         // sleep for 5 ms
  262.         usleep(5000);
  264.         // Closing the socket:
  265.         close(client_sock);
  266.         gettimeofday(&amp;end, NULL);
  267.         cpu_time_used = (end.tv_usec - start.tv_usec) / 1000.0;
  268.         if (cpu_time_used &gt; 0.0) // overflow in tv_usec if negative
  269.             printf(&quot;Server Time: %.4f ms\n&quot;, cpu_time_used);
  270.     } while(server_run);
  272.     close(socket_desc);
  274.     return 0;
  275. }

Here is the testing Go client

  1. /*
  2.     Adapted from
  3.     https://www.linode.com/docs/guides/developing-udp-and-tcp-clients-and-servers-in-go/
  5.     Run once the server.c is compiled and running with:
  6.         go run client.go 127.0.0.1:46999
  7. */
  8. package main
  10. import (
  11.     &quot;fmt&quot;
  12.     &quot;net&quot;
  13.     &quot;os&quot;
  14.     &quot;time&quot;
  15.     &quot;github.com/montanaflynn/stats&quot;
  16.     &quot;sync&quot;
  17. )
  18. func do_message(wg *sync.WaitGroup, connect string, time_taken *float64) {
  19.     defer wg.Done()
  20.     message := make([]byte, 128)
  21.     start_time := time.Now()
  22.     pAddr, err := net.ResolveTCPAddr(&quot;tcp&quot;, connect)
  23.     if err != nil {
  24.         return
  25.     }
  27.     c, err := net.DialTCP(&quot;tcp&quot;, nil, pAddr)
  28.     if err != nil {
  29.         fmt.Println(err)
  30.         return
  31.     }
  32.     c.SetLinger(0)
  33.     dialed_time := time.Since(start_time)
  35.     defer func() {
  36.         c.Close()
  37.         elapsed_time := time.Since(start_time)
  38.         if elapsed_time.Microseconds() &gt; 60 { // microseconds
  39.             fmt.Println(&quot;Long Elapsed Time: &quot; + dialed_time.String() + &quot;, &quot; + elapsed_time.String())
  40.         }
  41.         *time_taken = float64(elapsed_time.Microseconds())
  42.     }()
  44.     text := &quot;{\&quot;service\&quot;: \&quot;magic_service_str\&quot;}&quot;
  45.     c.Write([]byte(text))
  46.     code, _ := c.Read(message) // Does not actually wait for response.
  47.     code = code
  48. }
  49. func main() {
  50.     main_start := time.Now()
  51.     arguments := os.Args
  52.     if len(arguments) == 1 {
  53.             fmt.Println(&quot;Please provide host:port.&quot;)
  54.             return
  55.     }
  56.     n_messages := 50
  57.     wg := new(sync.WaitGroup)
  58.     wg.Add(n_messages)
  59.     times := make([]float64, n_messages)
  60.     for i := 0; i &lt; n_messages; i++ {
  61.         // Used to turn the goroutines into serial implementation
  62.         // time.Sleep(5500 * time.Microsecond)
  63.         go do_message(wg, arguments[1], &amp;times[i])
  64.     }
  65.     wg.Wait()
  66.     avg, _ := stats.Mean(times)
  67.     std, _ := stats.StandardDeviation(times)
  68.     fmt.Println(&quot;Time taken stats: &quot; + fmt.Sprintf(&quot;%.2f&quot;, avg / 1000000.0) + &quot; +/- &quot; + fmt.Sprintf(&quot;%.2f&quot;, std / 1000000.0) + &quot; s&quot;)
  69.     main_taken := time.Since(main_start)
  70.     fmt.Println(&quot;Main Taken: &quot; + main_taken.String())
  71. }

Updated pthreaded client in C and confirmed the issue is not the Golang implementation:

  1. // gcc client_p.c -o pclient -lpthread
  2. #include &lt;stdio.h&gt;
  3. #include &lt;string.h&gt;
  4. #include &lt;sys/socket.h&gt;
  5. #include &lt;arpa/inet.h&gt;
  6. #include &lt;unistd.h&gt;
  7. #include &lt;stdlib.h&gt;
  8. #include&lt;sys/time.h&gt;
  11. #include &lt;pthread.h&gt;
  12. #include &lt;errno.h&gt;
  14. #ifndef THREAD_LOOP_COUNT
  15. #define THREAD_LOOP_COUNT 1
  16. #endif
  18. /* Subtract the ‘struct timeval’ values X and Y,
  19.    storing the result in RESULT.
  20.    Return 1 if the difference is negative, otherwise 0.
  21.    https://www.gnu.org/software/libc/manual/html_node/Calculating-Elapsed-Time.html
  22. */
  24. int
  25. timeval_subtract (struct timeval *result, struct timeval *x, struct timeval *y)
  26. {
  27.   /* Perform the carry for the later subtraction by updating y. */
  28.   if (x-&gt;tv_usec &lt; y-&gt;tv_usec) {
  29.     int nsec = (y-&gt;tv_usec - x-&gt;tv_usec) / 1000000 + 1;
  30.     y-&gt;tv_usec -= 1000000 * nsec;
  31.     y-&gt;tv_sec += nsec;
  32.   }
  33.   if (x-&gt;tv_usec - y-&gt;tv_usec &gt; 1000000) {
  34.     int nsec = (x-&gt;tv_usec - y-&gt;tv_usec) / 1000000;
  35.     y-&gt;tv_usec += 1000000 * nsec;
  36.     y-&gt;tv_sec -= nsec;
  37.   }
  39.   /* Compute the time remaining to wait.
  40.      tv_usec is certainly positive. */
  41.   result-&gt;tv_sec = x-&gt;tv_sec - y-&gt;tv_sec;
  42.   result-&gt;tv_usec = x-&gt;tv_usec - y-&gt;tv_usec;
  44.   /* Return 1 if result is negative. */
  45.   return x-&gt;tv_sec &lt; y-&gt;tv_sec;
  46. }
  49. static void* workerThreadFunc(void* arg)
  50. {
  51.     int socket_desc;
  52.     struct sockaddr_in server_addr;
  53.     char server_message[2000], client_message[2000];
  54.     // Clean buffers:
  55.     memset(server_message,&#39;
    // gcc client_p.c -o pclient -lpthread
  56. #include &lt;stdio.h&gt;
  57. #include &lt;string.h&gt;
  58. #include &lt;sys/socket.h&gt;
  59. #include &lt;arpa/inet.h&gt;
  60. #include &lt;unistd.h&gt;
  61. #include &lt;stdlib.h&gt;
  62. #include&lt;sys/time.h&gt;
  63. #include &lt;pthread.h&gt;
  64. #include &lt;errno.h&gt;
  65. #ifndef THREAD_LOOP_COUNT
  66. #define THREAD_LOOP_COUNT 1
  67. #endif
  68. /* Subtract the ‘struct timeval’ values X and Y,
  69. storing the result in RESULT.
  70. Return 1 if the difference is negative, otherwise 0.
  71. https://www.gnu.org/software/libc/manual/html_node/Calculating-Elapsed-Time.html
  72. */
  73. int
  74. timeval_subtract (struct timeval *result, struct timeval *x, struct timeval *y)
  75. {
  76. /* Perform the carry for the later subtraction by updating y. */
  77. if (x-&gt;tv_usec &lt; y-&gt;tv_usec) {
  78. int nsec = (y-&gt;tv_usec - x-&gt;tv_usec) / 1000000 + 1;
  79. y-&gt;tv_usec -= 1000000 * nsec;
  80. y-&gt;tv_sec += nsec;
  81. }
  82. if (x-&gt;tv_usec - y-&gt;tv_usec &gt; 1000000) {
  83. int nsec = (x-&gt;tv_usec - y-&gt;tv_usec) / 1000000;
  84. y-&gt;tv_usec += 1000000 * nsec;
  85. y-&gt;tv_sec -= nsec;
  86. }
  87. /* Compute the time remaining to wait.
  88. tv_usec is certainly positive. */
  89. result-&gt;tv_sec = x-&gt;tv_sec - y-&gt;tv_sec;
  90. result-&gt;tv_usec = x-&gt;tv_usec - y-&gt;tv_usec;
  91. /* Return 1 if result is negative. */
  92. return x-&gt;tv_sec &lt; y-&gt;tv_sec;
  93. }
  94. static void* workerThreadFunc(void* arg)
  95. {
  96. int socket_desc;
  97. struct sockaddr_in server_addr;
  98. char server_message[2000], client_message[2000];
  99. // Clean buffers:
  100. memset(server_message,&#39;\0&#39;,sizeof(server_message));
  101. memset(client_message,&#39;\0&#39;,sizeof(client_message));
  102. // Set port and IP the same as server-side:
  103. server_addr.sin_family = AF_INET;
  104. server_addr.sin_port = htons(46999);
  105. server_addr.sin_addr.s_addr = inet_addr(&quot;127.0.0.1&quot;);
  106. int retries = 0;
  107. struct timeval start, end, difference;
  108. double cpu_time_used;
  109. for(int i = 0; i &lt; THREAD_LOOP_COUNT; i++)
  110. {
  111. gettimeofday(&amp;start, NULL);
  112. // Create socket:
  113. socket_desc = socket(AF_INET, SOCK_STREAM, 0);
  114. if(socket_desc &lt; 0){
  115. printf(&quot;Unable to create socket\n&quot;);
  116. return;
  117. }
  118. // Send connection request to server:
  119. while(connect(socket_desc, (struct sockaddr*)&amp;server_addr, sizeof(server_addr)) &lt; 0){
  120. retries++;
  121. if (retries &gt; 10)
  122. {
  123. printf(&quot;Unable to connect\n&quot;);
  124. retries = 0;
  125. }
  126. usleep(5);
  127. }
  128. int retries = 0;
  129. // Send the message to server:
  130. if(send(socket_desc, client_message, strlen(&quot;client message.&quot;), 0) &lt; 0){
  131. printf(&quot;Unable to send message\n&quot;);
  132. close(socket_desc);
  133. return;
  134. }
  135. // Receive the server&#39;s response:
  136. if(recv(socket_desc, server_message, sizeof(server_message), 0) &lt; 0){
  137. printf(&quot;Error while receiving server&#39;s msg\n&quot;);
  138. close(socket_desc);
  139. return;
  140. }
  141. // Close the socket:
  142. close(socket_desc);
  143. gettimeofday(&amp;end, NULL);
  144. timeval_subtract (&amp;difference, &amp;end, &amp;start);
  145. double cpu_time_used = (double)difference.tv_sec + (double)difference.tv_usec / 1000000.0;
  146. printf(&quot;Client Time: %.4e s\n&quot;, cpu_time_used);
  147. }
  148. }
  149. int main(int argc, char **argv)
  150. {
  151. int n_threads = 50;  // default value
  152. if (argc &gt; 1)
  153. n_threads = atoi(argv[1]);
  154. pthread_t *threads = (pthread_t*)malloc(n_threads * sizeof(pthread_t));
  155. struct timeval start, end, difference;
  156. gettimeofday(&amp;start, NULL);
  157. for(int i = 0; i &lt; n_threads; i++)
  158. {
  159. int createRet = pthread_create(&amp;threads[i], NULL, workerThreadFunc, NULL);
  160. if (createRet != 0)
  161. {
  162. printf(&quot;failed to create thread\n&quot;);
  163. }
  164. }
  165. for(int i = 0; i &lt; n_threads; i++)
  166. pthread_join(threads[i], NULL);
  167. gettimeofday(&amp;end, NULL);
  168. timeval_subtract (&amp;difference, &amp;end, &amp;start);
  169. double cpu_time_used = (double)difference.tv_sec + (double)difference.tv_usec / 1000000.0;
  170. printf(&quot;Total Client Time: %.4e s\n&quot;, cpu_time_used);
  171. free(threads);
  172. return 0;
  173. }
  174. &#39;,sizeof(server_message));
  175.     memset(client_message,&#39;
    // gcc client_p.c -o pclient -lpthread
  176. #include &lt;stdio.h&gt;
  177. #include &lt;string.h&gt;
  178. #include &lt;sys/socket.h&gt;
  179. #include &lt;arpa/inet.h&gt;
  180. #include &lt;unistd.h&gt;
  181. #include &lt;stdlib.h&gt;
  182. #include&lt;sys/time.h&gt;
  183. #include &lt;pthread.h&gt;
  184. #include &lt;errno.h&gt;
  185. #ifndef THREAD_LOOP_COUNT
  186. #define THREAD_LOOP_COUNT 1
  187. #endif
  188. /* Subtract the ‘struct timeval’ values X and Y,
  189. storing the result in RESULT.
  190. Return 1 if the difference is negative, otherwise 0.
  191. https://www.gnu.org/software/libc/manual/html_node/Calculating-Elapsed-Time.html
  192. */
  193. int
  194. timeval_subtract (struct timeval *result, struct timeval *x, struct timeval *y)
  195. {
  196. /* Perform the carry for the later subtraction by updating y. */
  197. if (x-&gt;tv_usec &lt; y-&gt;tv_usec) {
  198. int nsec = (y-&gt;tv_usec - x-&gt;tv_usec) / 1000000 + 1;
  199. y-&gt;tv_usec -= 1000000 * nsec;
  200. y-&gt;tv_sec += nsec;
  201. }
  202. if (x-&gt;tv_usec - y-&gt;tv_usec &gt; 1000000) {
  203. int nsec = (x-&gt;tv_usec - y-&gt;tv_usec) / 1000000;
  204. y-&gt;tv_usec += 1000000 * nsec;
  205. y-&gt;tv_sec -= nsec;
  206. }
  207. /* Compute the time remaining to wait.
  208. tv_usec is certainly positive. */
  209. result-&gt;tv_sec = x-&gt;tv_sec - y-&gt;tv_sec;
  210. result-&gt;tv_usec = x-&gt;tv_usec - y-&gt;tv_usec;
  211. /* Return 1 if result is negative. */
  212. return x-&gt;tv_sec &lt; y-&gt;tv_sec;
  213. }
  214. static void* workerThreadFunc(void* arg)
  215. {
  216. int socket_desc;
  217. struct sockaddr_in server_addr;
  218. char server_message[2000], client_message[2000];
  219. // Clean buffers:
  220. memset(server_message,&#39;\0&#39;,sizeof(server_message));
  221. memset(client_message,&#39;\0&#39;,sizeof(client_message));
  222. // Set port and IP the same as server-side:
  223. server_addr.sin_family = AF_INET;
  224. server_addr.sin_port = htons(46999);
  225. server_addr.sin_addr.s_addr = inet_addr(&quot;127.0.0.1&quot;);
  226. int retries = 0;
  227. struct timeval start, end, difference;
  228. double cpu_time_used;
  229. for(int i = 0; i &lt; THREAD_LOOP_COUNT; i++)
  230. {
  231. gettimeofday(&amp;start, NULL);
  232. // Create socket:
  233. socket_desc = socket(AF_INET, SOCK_STREAM, 0);
  234. if(socket_desc &lt; 0){
  235. printf(&quot;Unable to create socket\n&quot;);
  236. return;
  237. }
  238. // Send connection request to server:
  239. while(connect(socket_desc, (struct sockaddr*)&amp;server_addr, sizeof(server_addr)) &lt; 0){
  240. retries++;
  241. if (retries &gt; 10)
  242. {
  243. printf(&quot;Unable to connect\n&quot;);
  244. retries = 0;
  245. }
  246. usleep(5);
  247. }
  248. int retries = 0;
  249. // Send the message to server:
  250. if(send(socket_desc, client_message, strlen(&quot;client message.&quot;), 0) &lt; 0){
  251. printf(&quot;Unable to send message\n&quot;);
  252. close(socket_desc);
  253. return;
  254. }
  255. // Receive the server&#39;s response:
  256. if(recv(socket_desc, server_message, sizeof(server_message), 0) &lt; 0){
  257. printf(&quot;Error while receiving server&#39;s msg\n&quot;);
  258. close(socket_desc);
  259. return;
  260. }
  261. // Close the socket:
  262. close(socket_desc);
  263. gettimeofday(&amp;end, NULL);
  264. timeval_subtract (&amp;difference, &amp;end, &amp;start);
  265. double cpu_time_used = (double)difference.tv_sec + (double)difference.tv_usec / 1000000.0;
  266. printf(&quot;Client Time: %.4e s\n&quot;, cpu_time_used);
  267. }
  268. }
  269. int main(int argc, char **argv)
  270. {
  271. int n_threads = 50;  // default value
  272. if (argc &gt; 1)
  273. n_threads = atoi(argv[1]);
  274. pthread_t *threads = (pthread_t*)malloc(n_threads * sizeof(pthread_t));
  275. struct timeval start, end, difference;
  276. gettimeofday(&amp;start, NULL);
  277. for(int i = 0; i &lt; n_threads; i++)
  278. {
  279. int createRet = pthread_create(&amp;threads[i], NULL, workerThreadFunc, NULL);
  280. if (createRet != 0)
  281. {
  282. printf(&quot;failed to create thread\n&quot;);
  283. }
  284. }
  285. for(int i = 0; i &lt; n_threads; i++)
  286. pthread_join(threads[i], NULL);
  287. gettimeofday(&amp;end, NULL);
  288. timeval_subtract (&amp;difference, &amp;end, &amp;start);
  289. double cpu_time_used = (double)difference.tv_sec + (double)difference.tv_usec / 1000000.0;
  290. printf(&quot;Total Client Time: %.4e s\n&quot;, cpu_time_used);
  291. free(threads);
  292. return 0;
  293. }
  294. &#39;,sizeof(client_message));
  295.     // Set port and IP the same as server-side:
  296.     server_addr.sin_family = AF_INET;
  297.     server_addr.sin_port = htons(46999);
  298.     server_addr.sin_addr.s_addr = inet_addr(&quot;127.0.0.1&quot;);
  299.     int retries = 0;
  300.     struct timeval start, end, difference;
  301.     double cpu_time_used;
  302.     for(int i = 0; i &lt; THREAD_LOOP_COUNT; i++)
  303.     {
  304.         gettimeofday(&amp;start, NULL);
  305.         // Create socket:
  306.         socket_desc = socket(AF_INET, SOCK_STREAM, 0);
  307.         if(socket_desc &lt; 0){
  308.             printf(&quot;Unable to create socket\n&quot;);
  309.             return;
  310.         }
  311.         // Send connection request to server:
  312.         while(connect(socket_desc, (struct sockaddr*)&amp;server_addr, sizeof(server_addr)) &lt; 0){
  313.             retries++;
  314.             if (retries &gt; 10)
  315.             {
  316.                 printf(&quot;Unable to connect\n&quot;);
  317.                 retries = 0;
  318.             }
  319.             usleep(5);
  320.         }
  321.         int retries = 0;
  323.         // Send the message to server:
  324.         if(send(socket_desc, client_message, strlen(&quot;client message.&quot;), 0) &lt; 0){
  325.             printf(&quot;Unable to send message\n&quot;);
  326.             close(socket_desc);
  327.             return;
  328.         }
  330.         // Receive the server&#39;s response:
  331.         if(recv(socket_desc, server_message, sizeof(server_message), 0) &lt; 0){
  332.             printf(&quot;Error while receiving server&#39;s msg\n&quot;);
  333.             close(socket_desc);
  334.             return;
  335.         }
  337.         // Close the socket:
  338.         close(socket_desc);
  339.         gettimeofday(&amp;end, NULL);
  340.         timeval_subtract (&amp;difference, &amp;end, &amp;start);
  341.         double cpu_time_used = (double)difference.tv_sec + (double)difference.tv_usec / 1000000.0;
  342.         printf(&quot;Client Time: %.4e s\n&quot;, cpu_time_used);
  343.     }
  344. }
  346. int main(int argc, char **argv)
  347. {
  348.     int n_threads = 50;  // default value
  349.     if (argc &gt; 1)
  350.         n_threads = atoi(argv[1]);
  352. 	pthread_t *threads = (pthread_t*)malloc(n_threads * sizeof(pthread_t));
  354.     struct timeval start, end, difference;
  355.     gettimeofday(&amp;start, NULL);
  356.     for(int i = 0; i &lt; n_threads; i++)
  357.     {
  358. 	    int createRet = pthread_create(&amp;threads[i], NULL, workerThreadFunc, NULL);
  359.         if (createRet != 0)
  360.         {
  361. 		    printf(&quot;failed to create thread\n&quot;);
  362. 	    }
  363.     }
  364.     for(int i = 0; i &lt; n_threads; i++)
  365.         pthread_join(threads[i], NULL);
  366.     gettimeofday(&amp;end, NULL);
  367.     timeval_subtract (&amp;difference, &amp;end, &amp;start);
  368.     double cpu_time_used = (double)difference.tv_sec + (double)difference.tv_usec / 1000000.0;
  369.     printf(&quot;Total Client Time: %.4e s\n&quot;, cpu_time_used);
  370.     free(threads);
  371.     return 0;
  372. }

答案1

得分: 0

根据@user207421的说法，问题出在TCP实现上，其中包括对重试的指数退避。无论是Golang还是C，似乎都没有一种简单的方法来改变这种行为。

答案是：如果你需要高吞吐量，不要频繁打开和关闭TCP连接，而是使用连接池。

有一些工作正在研究如何去除指数退避，下面是相关链接，但对于特定情况可能有更好的解决方案。对我来说确实有更好的解决方案。

ACM SIGCOMM计算机通信评论，“从TCP中去除指数退避”，第38卷，第5期，2008年10月。

英文:

As indicated by @user207421, the issue lies in the TCP implementation, which includes an exponential backoff on retries. Neither Golang nor C appear to have an easy way to alter this behavior.

The answer is: Don't open and close connections of TCP if you high throughput--use a connection pool.

There was some work looking at removing the exponential backoff, linked below, but there is likely a better solution for specific cases. There was for me.

ACM SIGCOMM Computer Communication Review, "Removing Exponential Backoff from TCP", Volume 38, Number 5, October 2008.

huangapple
  • 本文由 发表于 2022年4月12日 19:23:46
  • 转载请务必保留本文链接：https://go.coder-hub.com/71841771.html
  • c
  • go
  • tcp
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定