8000 Segfault in cluster mode (3.0.0) · Issue #951 · phpredis/phpredis · GitHub
[go: up one dir, main page]
Skip to content

Segfault in cluster mode (3.0.0) #951

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
vkirovskiy opened this issue Sep 19, 2016 · 6 comments
Closed

Segfault in cluster mode (3.0.0) #951

vkirovskiy opened this issue Sep 19, 2016 · 6 comments

Comments

@vkirovskiy
Copy link
vkirovskiy commented Sep 19, 2016 •
edited
Loading

Hello everyone

We often receive segfaults like this

Sep 19 16:21:45 v-6-07-12-d5110-93 kernel: [10117415.046740] php-fpm7.0[33132]: segfault at 7fa76d53f498 ip 00007fa75abe1623 sp 00007ffd395eabd0 error 6 in redis.so[7fa75ab4e000+a3000]

Our enviroment:
ubuntu 14.04 LTS
php-fpm 7.0
php-redis 3.0.0 (compiled from pecl stable release)
Tried it on redis server 3.0.3, 3.2.0, 3.2.3

Redis is build in cluster mode, we have 2 setups, prod and stage.
Prod has 6 servers (3 masters and 3 slaves).
Stage has 6 redis instances on single server.
Segfaults appear on both of setups.

Backtrace

Program terminated with signal SIGSEGV, Segmentation fault.

0 0x00007fd26cbe1623 in cluster_shuffle_seeds (seeds=0x7fd266d85f88, len=0x7fffa6ae777c) at /vagrant/php-redis/cluster_library.c:862

862 /vagrant/php-redis/cluster_library.c: No such file or directory.
(gdb) bt full

0 0x00007fd26cbe1623 in cluster_shuffle_seeds (seeds=0x7fd266d85f88, len=0x7fffa6ae777c) at /vagrant/php-redis/cluster_library.c:862

   z_seeds = 0x7fd27bb094d0
   map = 0x7fd27bad3060
   i = 7
   count = 7
   index = 8

1 0x00007fd26cbe16bd in cluster_init_seeds (cluster=0x7fd27bb0f000, ht_seeds=0x7fd266d85f88) at /vagrant/php-redis/cluster_library.c:880

   redis_sock = 0x7fd27ba780e0
   str = 0x56201ee608b5 <destroy_op_array+757> "I�\204$�"
   psep = 0x7fd27ba15e40 "\030?�f�\177"
   key = "��\236}�\177\000\000`^�{�\177\000\000\230f�{�\177\000\000B��\036V\000\000@�$\037 V\000\000�D�{�\177\000\000\060^�{�\177\000\000\225��\036V\000\000�3�{�\177\000\000\060^�{�\177\000\000\002\000\000\000\000\000\000\000���\036 V\000\000�F#\037 V\000\000�F#\037 V\000\000 ]�{�\177", '\000' <repeats 18 times>, "Kt�\036 V\000\000\001\000\000\000\000\000\000\000�F#\037 V\000\000\000\000\000\000\000\000\000\000���\036 V\000\000\200\201�{�\177\000\000\220[�{�\177\000\000H", '\000' <repeats 15 times>...
   key_len = 0
   count = 0
   i = 0
   z_seeds = 0x7fd27ba15e40
   z_seed = 0x7fd27ba780e0

2 0x00007fd26cbc6a50 in redis_cluster_init (c=0x7fd27bb0f000, ht_seeds=0x7fd266d85f88, timeout=0, read_timeout=0, persistent=0) at /vagrant/php-redis/redis_cluster.c:353

No locals.

3 0x00007fd26cbc70a1 in zim_RedisCluster___construct (execute_data=0x7fd27ba15df0, return_value=0x7fd27ba15de0) at /vagrant/php-redis/redis_cluster.c:455

   object = 0x7fd27ba15e10
   z_seeds = 0x7fd27ba15e60
   name = 0x7fd2665d27f8 ""
   name_len = 0
   timeout = 0
   read_timeout = 0
   persistent = 0 '\000'
   context = 0x7fd27bb0f000

4 0x000056201ee5bafb in dtrace_execute_internal ()

No symbol table info available.

5 0x000056201eee4f64 in ?? ()

I guess it's a block of code where error is appeared (line 862)

/* Iterate over our source array and use our map to create a random list */
    for (zend_hash_internal_pointer_reset(seeds);
         zend_hash_has_more_elements(seeds) == SUCCESS;
         zend_hash_move_forward(seeds))
    {
        >> z_seeds[map[index++]] = zend_hash_get_current_data(seeds); <<
    }

Help please to determine the reason and resolve this issue.
Thanks!

Vladimir.
@vkirovskiy
Copy link
Author
vkirovskiy commented Nov 8, 2016 •
edited
Loading

Additional debug info
I hope it will help you
If any additional info is needed ask please
It's still urgent for us
Thanks

static zval **cluster_shuffle_seeds(HashTable *seeds, int *len) {
    zval **z_seeds;
    int *map, i, count, index=0;

    /* How many */
    count = zend_hash_num_elements(seeds);

    /* Allocate our return value and map */
    z_seeds = ecalloc(count, sizeof(zval*));
    map = emalloc(sizeof(int)*count);

    /* Fill in and shuffle our map */
    for (i = 0; i < count; i++) map[i] = i;
    fyshuffle(map, count);

    /* Iterate over our source array and use our map to create a random list */
    for (zend_hash_internal_pointer_reset(seeds);
         zend_hash_has_more_elements(seeds) == SUCCESS;
         zend_hash_move_forward(seeds))
    {
        z_seeds[map[index++]] = zend_hash_get_current_data(seeds);
    }

    efree(map);

    *len = count;
    return z_seeds;
}

(gdb) p z_seeds[map[index++]]
Cannot access memory at address 0x7ffc418c28c8

(gdb) p count
$13 = 7

(gdb) p index
$4 = 8

(gdb) p *map@9
$6 = {0, 3, 6, 5, 4, 1, 2, -16747776, 0}

(gdb) p *seeds
$34 = {gc = {refcount = 2, u = {v = {type = 7 '\a', flags = 2 '\002', gc_info = 0}, type_info = 519}}, u = {v = {flags = 28 '\034', nApplyCount = 0 '\000', nIteratorsCount = 0 '\000', reserve = 0 '\000'}, flags = 28},
nTableMask = 4294967294, arData = 0x7f6322877388, nNumUsed = 7, nNumOfElements = 7, nTableSize = 8, nInternalPointer = 4294967295, nNextFreeElement = 7, pDestructor = 0x55f8c218dca0 <_zval_ptr_dtor>}

(gdb) p *z_seeds@9
$10 = {0x7f6322877388, 0x7f6322877408, 0x7f6322877428, 0x7f63228773a8, 0x7f63228773e8, >0x7f63228773e8, 0x7f63228773c8, 0x700000001, 0xfffffffe00000012}

@vkirovskiy
Copy link
Author

Some investigation and workaround

Sometimes function zend_hash_has_more_elements(seeds) returns SUCCESS when no more elements are defined (or available). Unfortunately we have no resources and time to find the main reason.

Small patch.
We tested it and it's seems we didn't lose any functionality, all redis nodes are used, segfaults disappeared.

*** ./1/redis-3.0.0/cluster_library.c   2016-06-10 20:08:21.000000000 +0000
--- ./redis-3.0.0/cluster_library.c 2016-11-11 12:36:01.376468711 +0000
***************
*** 859,866 ****
--- 859,868 ----
           zend_hash_has_more_elements(seeds) == SUCCESS;
           zend_hash_move_forward(seeds))
      {
          z_seeds[map[index++]] = zend_hash_get_current_data(seeds);
+ 
+   if ( index == count ) break;
      }

      efree(map);

@yatsukhnenko
Copy link
Member

@vkirovskiy, could you try this code?

-    for (zend_hash_internal_pointer_reset(seeds);
-         zend_hash_has_more_elements(seeds) == SUCCESS;
-         zend_hash_move_forward(seeds))
-    {
-        z_seeds[map[index++]] = zend_hash_get_current_data(seeds);
-    }
+    ZEND_HASH_FOREACH_VAL(seeds, z_ele) {
+        z_seeds[map[index++]] = z_ele;
+    } ZEND_HASH_FOREACH_END();

@vkirovskiy
Copy link
Author

@yatsukhnenko sure! thanks for the answer.
Will compile and test it today

@yatsukhnenko
Copy link
Member

@vkirovskiy BTW develop brunch is also support php7 and some of bugs already fixed

@vkirovskiy
Copy link
Author

@yatsukhnenko Hi. Good patch, segfaults didn't appear after stress testing.
All nodes are used.
I found a bit failure connections, about 10 of 65000 connections was lost, but i guess it's cause of high connection rate.
Thank you for resolving this issue!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@yatsukhnenko @vkirovskiy
0