Posted on 2022-17-02
HAProxy is usually very reliable, that said my use case is not very common: I am using cutting edge version of the software on an aarch64 system and I may use code path that very few people uses, and so I may meet some bugs.
Note
HAProxy is Open Source and what I love the most about open source is that you can help the community making the product even better. Contributions are not only code, they may be a nice bug report that dev can leverage to quickly find the issue and make the software more reliable. This applies to any open source solution you use!
So today, we'll see how we can help the community when an unexpected event happens :)
Note
Keep in mind these instructions apply to Debian 12. They might be different for your own environment.
First, we need to raise the size for the core files. For this purpose, we create a new file in /etc/security/limits.d/core.conf:
* soft core unlimited
Then, we need to tell our kernel where to save coredump files. For this, we'll add a new file in /etc/sysctl.d/98-core.conf with the following content:
kernel.core_pattern=/tmp/core-%e-%p
Because our HAProxy is chrooted, we need to ensure that a /tmp folder exists in there:
sudo mkdir YOUR_HAPROXY_CHROOT_PATH/tmp sudo chmod a+w YOUR_HAPROXY_CHROOT_PATH/tmp
Last, we can also enable the set-dumpable parameter in HAProxy's global configuration.
global [...] set-dumpable
According to the doc, this parameter will perform the following:
will try hard to re-enable core dumps that were possibly disabled by file size limitations (ulimit -f), core size limitations (ulimit -c), or "dumpability" of a process after changing its UID/GID (such
It's basically a very nice helper, but does not prevent you from checking a core dump is well produced at the expected place: sudo kill -11 $(pgrep haproxy) will produce a core.
A quick troubleshooting / checklist if you can't get your core files: - ensure ulimit -f and ulimit -c are set properly for the user HAProxy drops privileges to - ensure the target core dump directory exists in the chroot - ensure the user running the HAProxy process can write into this directory - always try to generate a core to double check everything works as expected
For more information and use cases on this topic, you can read this excellent page
For the record, the "bug" I am tracking currently is not in HAProxy, but somewhere between OpenSSL and the libc on my aarch64 debian. Here is my backtrace from gdb:
#0 0x0000ffffa12ef690 in free () from /lib/aarch64-linux-gnu/libc.so.6
#1 0x0000aaaad8509cec in ossl_ecx_key_free ()
#2 0x0000aaaad8468f18 in ecx_freectx ()
#3 0x0000aaaad83b22c4 in evp_pkey_ctx_free_old_ops ()
#4 0x0000aaaad83b2330 in EVP_PKEY_CTX_free ()
#5 0x0000aaaad82a60dc in ssl_derive ()
#6 0x0000aaaad82edfe0 in tls_construct_stoc_key_share ()
#7 0x0000aaaad82e410c in tls_construct_extensions ()
#8 0x0000aaaad82feea4 in tls_construct_server_hello ()
#9 0x0000aaaad82ef888 in state_machine ()
#10 0x0000aaaad82b616c in SSL_do_handshake ()
#11 0x0000aaaad7f01670 in ssl_sock_handshake (conn=0xaaaadb76cb70, flag=134217728) at src/ssl_sock.c:6283
#12 0x0000aaaad7f02398 in ssl_sock_io_cb (t=0xffff8c02b7d0, context=0xffff8c02ba10, state=32960)
at src/ssl_sock.c:6626
#13 0x0000aaaad81ce300 in run_tasks_from_lists (budgets=0xffff9b7ec6d8) at src/task.c:596
#14 0x0000aaaad81cf090 in process_runnable_tasks () at src/task.c:876
#15 0x0000aaaad816e73c in run_poll_loop () at src/haproxy.c:3050
#16 0x0000aaaad816f0b4 in run_thread_poll_loop (data=0xaaaad8af3cc0 <ha_thread_info+192>) at src/haproxy.c:3252
Tweet