Nginx 缓存 DNS 解析问题
2022-03-03 tech nginx 29 mins 1 图 10228 字
我使用了容器化的oprenresty，一般都会使用 upstream 特性转发到多个后端。因为是高可用的架构，后端偶尔也会手动重启服务。但问题在于 nginx 原生不支持动态解析 dns，需要做额外的模块编译。这篇文章记录我手动编译 openresty 容器的过程。
一、缓存问题的解决方案
1 每次更改 DNS 解析都重载 Nginx
重载 Nginx 一定会刷新缓存。
2 使用 Nginx 的 resolver
我们在使用 Nginx 过程中，有时需要根据 Url 传值动态选择 host 进行代理转发，这种模式下，一开始是不会去进行 DNS 解析的，只有请求的时候才会进行 DNS 解析，并且要设置 resolver 指定 DNS 服务器 IP。
这个时候，我们就可以使用 resolver 语法来解决 DNS 缓存的问题，比如说，我在原来的 Nginx 配置里指定 DNS IP，并设置缓存 60 秒。
server {
listen 80;
server_name www.test.com;
resolver 127.0.0.1 valid=60s;
resolver_timeout 3s;
set $proxy_url "proxy.test.com";
location / {
proxy_set_header Host proxy.test.com;
proxy_pass http://$proxy_url;
}
}
但这种方法无法作用于 upstream 里的域名。
3 使用模块nginx-upstream-dynamic-servers
模块地址: nginx-upstream-dynamic-servers
该模块在第一次启动的时候会进行一次解析，解析完后，在 DNS 服务器设定的 TTL 过期时间内不会再次更新，过期后会再次发起解析请求
使用方法
http {
resolver 8.8.8.8;
upstream example {
server example.com resolve;
}
}
使用这种方法的时候，DNS TTL 时间需要设置短一些。
4 使用模块 ngx_upstream_jdomain
文档地址: domain_resolve
ngx_upstream_jdomain 模块是一个依赖 DNS 解析实现的 upstream 负载均衡，该模式下，允许使用域名来写 upstream 后端。该模块默认情况下，会每秒做一次 DNS 解析，使用方法如下
http {
resolver 8.8.8.8;
resolver_timeout 10s;
upstream backend {
jdomain www.baidu.com port=80 interval=5; # 每 5 秒解析一次
}
server {
listen 8080;
location / {
proxy_pass http://backend;
}
}
}
5 使用其他版本 Nginx
- Tengine Tengine 的
ngx_http_upstream_dynamic模块可以提供动态的 DNS 解析 - NGINX Plus Plus版本提供了动态解析的语法
在这里我才用了第三种方案，使用模块nginx-upstream-dynamic-servers。
二、容器化编译openresty
大致介绍一下注意点：
-
自定义编译参考官方文档：https://github.com/openresty/docker-openresty#building-non-rpm-based
-
系统我选用了alpine-3.12版本，RESTY版本1.17.8.2
-
在RESTY_CONFIG_OPTIONS中增加了自定义的编译命令：
--add-module=/tmp/nginx-upstream-dynamic-servers-0.4.0 \ -
增加这个包的下载解压命令：
&& cd /tmp \ && curl -fSL https://github.com/GUI/nginx-upstream-dynamic-servers/archive/refs/tags/v0.4.0.tar.gz -o nginx-upstream-dynamic-servers.v0.4.0.tar.gz \ && tar zxvf nginx-upstream-dynamic-servers.v0.4.0.tar.gz \ -
修改了一些其他组件的下载地址，有些链接失效了，例如pcre：
curl -fSL https://ftp.exim.org/pub/pcre/pcre-${RESTY_PCRE_VERSION}.tar.gz -o pcre-${RESTY_PCRE_VERSION}.tar.gz -
注意编译最后一步要把下载的东西给清空了，节省空间
编译文件下载。编译文件夹内容如下：
具体dockerfile如下：
# Dockerfile - alpine
# https://github.com/openresty/docker-openresty
ARG RESTY_IMAGE_BASE="alpine"
ARG RESTY_IMAGE_TAG="3.12"
FROM ${RESTY_IMAGE_BASE}:${RESTY_IMAGE_TAG}
LABEL maintainer="Evan Wies <evan@neomantra.net>"
# Docker Build Arguments
ARG RESTY_IMAGE_BASE="alpine"
ARG RESTY_IMAGE_TAG="3.12"
ARG RESTY_VERSION="1.17.8.2"
ARG RESTY_OPENSSL_VERSION="1.1.1g"
ARG RESTY_OPENSSL_PATCH_VERSION="1.1.1f"
ARG RESTY_OPENSSL_URL_BASE="https://www.openssl.org/source"
ARG RESTY_PCRE_VERSION="8.44"
ARG RESTY_VTS_VERSION="0.1.18"
ARG RESTY_J="1"
ARG RESTY_CONFIG_OPTIONS="\
--with-compat \
--with-file-aio \
--with-http_addition_module \
--with-http_auth_request_module \
--with-http_dav_module \
--with-http_flv_module \
--with-http_geoip_module=dynamic \
--with-http_gunzip_module \
--with-http_gzip_static_module \
--with-http_image_filter_module=dynamic \
--with-http_mp4_module \
--with-http_random_index_module \
--with-http_realip_module \
--with-http_secure_link_module \
--with-http_slice_module \
--with-http_ssl_module \
--with-http_stub_status_module \
--with-http_sub_module \
--with-http_v2_module \
--with-http_xslt_module=dynamic \
--with-ipv6 \
--with-mail \
--with-mail_ssl_module \
--with-md5-asm \
--with-pcre-jit \
--with-sha1-asm \
--with-stream \
--with-stream_ssl_module \
--with-threads \
--add-module=/tmp/nginx-module-vts-0.1.18 \
--add-module=/tmp/nginx-upstream-dynamic-servers-0.4.0 \
"
ARG RESTY_CONFIG_OPTIONS_MORE=""
ARG RESTY_LUAJIT_OPTIONS="--with-luajit-xcflags='-DLUAJIT_NUMMODE=2 -DLUAJIT_ENABLE_LUA52COMPAT'"
ARG RESTY_ADD_PACKAGE_BUILDDEPS=""
ARG RESTY_ADD_PACKAGE_RUNDEPS=""
ARG RESTY_EVAL_PRE_CONFIGURE=""
ARG RESTY_EVAL_POST_MAKE=""
# These are not intended to be user-specified
ARG _RESTY_CONFIG_DEPS="--with-pcre \
--with-cc-opt='-DNGX_LUA_ABORT_AT_PANIC -I/usr/local/openresty/pcre/include -I/usr/local/openresty/openssl/include' \
--with-ld-opt='-L/usr/local/openresty/pcre/lib -L/usr/local/openresty/openssl/lib -Wl,-rpath,/usr/local/openresty/pcre/lib:/usr/local/openresty/openssl/lib' \
"
LABEL resty_image_base="${RESTY_IMAGE_BASE}"
LABEL resty_image_tag="${RESTY_IMAGE_TAG}"
LABEL resty_version="${RESTY_VERSION}"
LABEL resty_openssl_version="${RESTY_OPENSSL_VERSION}"
LABEL resty_openssl_patch_version="${RESTY_OPENSSL_PATCH_VERSION}"
LABEL resty_openssl_url_base="${RESTY_OPENSSL_URL_BASE}"
LABEL resty_pcre_version="${RESTY_PCRE_VERSION}"
LABEL resty_config_options="${RESTY_CONFIG_OPTIONS}"
LABEL resty_config_options_more="${RESTY_CONFIG_OPTIONS_MORE}"
LABEL resty_config_deps="${_RESTY_CONFIG_DEPS}"
LABEL resty_add_package_builddeps="${RESTY_ADD_PACKAGE_BUILDDEPS}"
LABEL resty_add_package_rundeps="${RESTY_ADD_PACKAGE_RUNDEPS}"
LABEL resty_eval_pre_configure="${RESTY_EVAL_PRE_CONFIGURE}"
LABEL resty_eval_post_make="${RESTY_EVAL_POST_MAKE}"
RUN apk add --no-cache --virtual .build-deps \
build-base \
coreutils \
curl \
gd-dev \
geoip-dev \
libxslt-dev \
linux-headers \
make \
perl-dev \
readline-dev \
zlib-dev \
${RESTY_ADD_PACKAGE_BUILDDEPS} \
&& apk add --no-cache \
gd \
geoip \
libgcc \
libxslt \
zlib \
${RESTY_ADD_PACKAGE_RUNDEPS} \
&& cd /tmp \
&& if [ -n "${RESTY_EVAL_PRE_CONFIGURE}" ]; then eval $(echo ${RESTY_EVAL_PRE_CONFIGURE}); fi \
&& cd /tmp \
&& curl -fSL https://github.com/GUI/nginx-upstream-dynamic-servers/archive/refs/tags/v0.4.0.tar.gz -o nginx-upstream-dynamic-servers.v0.4.0.tar.gz \
&& tar zxvf nginx-upstream-dynamic-servers.v0.4.0.tar.gz \
&& cd /tmp \
&& curl -fSL https://github.com/vozlt/nginx-module-vts/archive/v${RESTY_VTS_VERSION}.tar.gz -o nginx-module-vts-${RESTY_VTS_VERSION}.tar.gz \
&& tar xzf nginx-module-vts-${RESTY_VTS_VERSION}.tar.gz \
&& cd /tmp \
&& curl -fSL https://github.com/vozlt/nginx-module-vts/archive/v${RESTY_VTS_VERSION}.tar.gz -o nginx-module-vts-${RESTY_VTS_VERSION}.tar.gz \
&& tar xzf nginx-module-vts-${RESTY_VTS_VERSION}.tar.gz \
&& curl -fSL "${RESTY_OPENSSL_URL_BASE}/openssl-${RESTY_OPENSSL_VERSION}.tar.gz" -o openssl-${RESTY_OPENSSL_VERSION}.tar.gz \
&& tar xzf openssl-${RESTY_OPENSSL_VERSION}.tar.gz \
&& cd openssl-${RESTY_OPENSSL_VERSION} \
&& if [ $(echo ${RESTY_OPENSSL_VERSION} | cut -c 1-5) = "1.1.1" ] ; then \
echo 'patching OpenSSL 1.1.1 for OpenResty' \
&& curl -s https://raw.githubusercontent.com/openresty/openresty/master/patches/openssl-${RESTY_OPENSSL_PATCH_VERSION}-sess_set_get_cb_yield.patch | patch -p1 ; \
fi \
&& if [ $(echo ${RESTY_OPENSSL_VERSION} | cut -c 1-5) = "1.1.0" ] ; then \
echo 'patching OpenSSL 1.1.0 for OpenResty' \
&& curl -s https://raw.githubusercontent.com/openresty/openresty/ed328977028c3ec3033bc25873ee360056e247cd/patches/openssl-1.1.0j-parallel_build_fix.patch | patch -p1 \
&& curl -s https://raw.githubusercontent.com/openresty/openresty/master/patches/openssl-${RESTY_OPENSSL_PATCH_VERSION}-sess_set_get_cb_yield.patch | patch -p1 ; \
fi \
&& ./config \
no-threads shared zlib -g \
enable-ssl3 enable-ssl3-method \
--prefix=/usr/local/openresty/openssl \
--libdir=lib \
-Wl,-rpath,/usr/local/openresty/openssl/lib \
&& make -j${RESTY_J} \
&& make -j${RESTY_J} install_sw \
&& cd /tmp \
&& curl -fSL https://ftp.exim.org/pub/pcre/pcre-${RESTY_PCRE_VERSION}.tar.gz -o pcre-${RESTY_PCRE_VERSION}.tar.gz \
&& tar xzf pcre-${RESTY_PCRE_VERSION}.tar.gz \
&& cd /tmp/pcre-${RESTY_PCRE_VERSION} \
&& ./configure \
--prefix=/usr/local/openresty/pcre \
--disable-cpp \
--enable-jit \
--enable-utf \
--enable-unicode-properties \
&& make -j${RESTY_J} \
&& make -j${RESTY_J} install \
&& cd /tmp \
&& curl -fSL https://openresty.org/download/openresty-${RESTY_VERSION}.tar.gz -o openresty-${RESTY_VERSION}.tar.gz \
&& tar xzf openresty-${RESTY_VERSION}.tar.gz \
&& cd /tmp/openresty-${RESTY_VERSION} \
&& eval ./configure -j${RESTY_J} ${_RESTY_CONFIG_DEPS} ${RESTY_CONFIG_OPTIONS} ${RESTY_CONFIG_OPTIONS_MORE} ${RESTY_LUAJIT_OPTIONS} \
&& make -j${RESTY_J} \
&& make -j${RESTY_J} install \
&& cd /tmp \
&& if [ -n "${RESTY_EVAL_POST_MAKE}" ]; then eval $(echo ${RESTY_EVAL_POST_MAKE}); fi \
&& rm -rf \
nginx-module-vts-${RESTY_VTS_VERSION}.tar.gz nginx-module-vts-${RESTY_VTS_VERSION} \
openssl-${RESTY_OPENSSL_VERSION}.tar.gz openssl-${RESTY_OPENSSL_VERSION} \
pcre-${RESTY_PCRE_VERSION}.tar.gz pcre-${RESTY_PCRE_VERSION} \
openresty-${RESTY_VERSION}.tar.gz openresty-${RESTY_VERSION} \
* \
&& apk del .build-deps \
&& mkdir -p /var/run/openresty \
&& ln -sf /dev/stdout /usr/local/openresty/nginx/logs/access.log \
&& ln -sf /dev/stderr /usr/local/openresty/nginx/logs/error.log
# Add additional binaries into PATH for convenience
ENV PATH=$PATH:/usr/local/openresty/luajit/bin:/usr/local/openresty/nginx/sbin:/usr/local/openresty/bin
# Copy nginx configuration files
COPY nginx.conf /usr/local/openresty/nginx/conf/nginx.conf
COPY nginx.vh.default.conf /etc/nginx/conf.d/default.conf
CMD ["/usr/local/openresty/bin/openresty", "-g", "daemon off;"]
# Use SIGQUIT instead of default SIGTERM to cleanly drain requests
# See https://github.com/openresty/docker-openresty/blob/master/README.md#tips--pitfalls
STOPSIGNAL SIGQUIT
编译命令：
docker build -t kelvinblood/openresty:v3.12.1 .
