8000 php_network_getaddresses: getaddrinfo failed · Issue #2556 · phpredis/phpredis · GitHub
[go: up one dir, main page]
Skip to content

php_network_getaddresses: getaddrinfo failed #2556

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
passion-pioneer opened this issue Sep 26, 2024 · 6 comments
Closed

php_network_getaddresses: getaddrinfo failed #2556

passion-pioneer opened this issue Sep 26, 2024 · 6 comments

Comments

@passion-pioneer
Copy link

I'm seeing this behaviour on

  • EKS 1.29
  • OS: Amazon Linux 2
  • Redis: AWS Elasticache Redis
  • PHP: Pod (php-fpm 7.3)

Infrastructure situation

When a request comes in, this error intermittently occurs in the application using PHP-FPM.
In this case, the ELB encounters a 4xx error.

Error message

FastCGI sent in stderr: "PHP message: PHP Fatal error: Uncaught ErrorException: session_start(): php_network_getaddresses: getaddrinfo failed: Temporary failure in name resolution in /home/ubuntu/apps/xxxxxx/public/index.php:24
Stack trace:
#0 [internal function]: {closure}(2, 'session_start()...', '/home/ubuntu/ap...', 24, Array)
#1 /home/ubuntu/apps/xxxxxx/public/index.php(24): session_start()
#2 {main}
Next RedisClusterException: Couldn't map cluster keyspace using any provided seed in /home/ubuntu/apps/xxxxxx/public/index.php:24
Stack trace:
#0 /home/ubuntu/apps/xxxxxx/public/index.php(24): session_start()
#1 {main}
thrown in /home/ubuntu/apps/synctree-tool/public/index.php on line 24" while reading response header from upstream, client: x.x.x.x, server: _, request: "POST /v1/customer/verify HTTP/1.1", upstream: "fastcgi://unix:/run/php/php7.3-fpm.sock:", host: "api.mydata.xxxxxx.com"

Setting session.save_path

  • session.save_path = "seed[]=tcp://mydata.xxxxxx.clustercfg.apn2.cache.amazonaws.com:6379"

Please let me know how to solve this problem.
Help me.
@passion-pioneer
Copy link
Author

Is it possible that it is coredns load?

@passion-pioneer
Copy link
Author

The issue was resolved by applying NodeLocalDNSCache.

@jmarcher
Copy link
jmarcher commented Oct 1, 2024

The issue was resolved by applying NodeLocalDNSCache.

Can you explain a bit more what that means? We are having the same issue on our systems

@passion-pioneer
Copy link
Author
passion-pioneer commented Oct 6, 2024 •
edited
Loading

The issue was resolved by applying NodeLocalDNSCache.

Can you explain a bit more what that means? We are having the same issue on our systems

My Situation

During the night, as simultaneous requests are made, the PHP application (Pod) sends domain name resolution requests for Redis to a CoreDNS pod on another node, outside of the node it resides on.

The PHP application (Pod) attempts to forward the Redis domain name resolution request packet to its own node and send it out, but the large volume of simultaneous packets exceeds the Linux host's conntrack limit, causing packet drops in the node's eth0 outbound.

As a result, the PHP application does not receive a response and triggers an error. (PHP message: PHP Fatal error: Uncaught ErrorException: session_start(): php_network_getaddresses: getaddrinfo failed: Temporary failure in name resolution)

Additionally, if the pods are deployed redundantly within the same node, this phenomenon is further exacerbated.

To improve this, we adjusted the application (PHP) pods to not be redundantly deployed within a single node, increased the number of CoreDNS pods from 2 to 5, and configured NodeLocalDNSCache to ensure that domain queries receive responses within the node as much as possible. This resolved the error.

By checking conntrack on the server's eth0, you can confirm that packets have been dropped.

@michael-grunder
Copy link
Member

Should we look into caching DNS lookups in PhpRedis? It wouldn't be particularly complex to implement. We could use a hash table in PHP's persistent_list with the lookup and TTL.

Would that even be desirable?

@passion-pioneer
Copy link
Author

Should we look into caching DNS lookups in PhpRedis? It wouldn't be particularly complex to implement. We could use a hash table in PHP's persistent_list with the lookup and TTL.

Would that even be desirable?

The details seem to vary depending on the infrastructure situation.
However, the cause of the error I experienced was the part about the domain call in php or phpredis. If the server is Linux, check the status by using "ethtool -S eth0". Check the part related to conntrack.
Check the link below. https://aws.github.io/aws-eks-best-practices/networking/monitoring/

PHP message: PHP Fatal error: Uncaught ErrorException: session_start(): php_network_getaddresses: getaddrinfo failed: Temporary failure in name resolution

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@michael-grunder @jmarcher @passion-pioneer
0