Nginx 缓存 DNS 解析问题

2022-03-03     tech nginx 29 mins 1 图 10228 字

我使用了容器化的oprenresty，一般都会使用 upstream 特性转发到多个后端。因为是高可用的架构，后端偶尔也会手动重启服务。但问题在于 nginx 原生不支持动态解析 dns，需要做额外的模块编译。这篇文章记录我手动编译 openresty 容器的过程。

一、缓存问题的解决方案

1 每次更改 DNS 解析都重载 Nginx

重载 Nginx 一定会刷新缓存。

2 使用 Nginx 的 resolver

我们在使用 Nginx 过程中，有时需要根据 Url 传值动态选择 host 进行代理转发，这种模式下，一开始是不会去进行 DNS 解析的，只有请求的时候才会进行 DNS 解析，并且要设置 resolver 指定 DNS 服务器 IP。

这个时候，我们就可以使用 resolver 语法来解决 DNS 缓存的问题，比如说，我在原来的 Nginx 配置里指定 DNS IP，并设置缓存 60 秒。

server {
    listen 80;
    server_name www.test.com;

    resolver 127.0.0.1 valid=60s;
    resolver_timeout 3s;

    set $proxy_url "proxy.test.com";
    location / {
        proxy_set_header Host proxy.test.com;
        proxy_pass http://$proxy_url;
    }
}

但这种方法无法作用于 upstream 里的域名。

3 使用模块nginx-upstream-dynamic-servers

模块地址: nginx-upstream-dynamic-servers

该模块在第一次启动的时候会进行一次解析，解析完后，在 DNS 服务器设定的 TTL 过期时间内不会再次更新，过期后会再次发起解析请求

使用方法

http {
  resolver 8.8.8.8;

  upstream example {
    server example.com resolve;
  }
}

使用这种方法的时候，DNS TTL 时间需要设置短一些。

4 使用模块 ngx_upstream_jdomain

文档地址: domain_resolve

ngx_upstream_jdomain 模块是一个依赖 DNS 解析实现的 upstream 负载均衡，该模式下，允许使用域名来写 upstream 后端。该模块默认情况下，会每秒做一次 DNS 解析，使用方法如下

http {
    resolver 8.8.8.8;
    resolver_timeout 10s;

    upstream backend {
        jdomain www.baidu.com port=80 interval=5; # 每 5 秒解析一次
    }
    server {
        listen  8080;
    
        location / {
            proxy_pass http://backend;
        }
    }
}

5 使用其他版本 Nginx

  • Tengine Tengine 的 ngx_http_upstream_dynamic 模块可以提供动态的 DNS 解析
  • NGINX Plus Plus版本提供了动态解析的语法

在这里我才用了第三种方案，使用模块nginx-upstream-dynamic-servers。

二、容器化编译openresty

大致介绍一下注意点：

  1. 自定义编译参考官方文档：https://github.com/openresty/docker-openresty#building-non-rpm-based

  2. 系统我选用了alpine-3.12版本，RESTY版本1.17.8.2

  3. 在RESTY_CONFIG_OPTIONS中增加了自定义的编译命令：

        --add-module=/tmp/nginx-upstream-dynamic-servers-0.4.0 \

  4. 增加这个包的下载解压命令：

        && cd /tmp \
    && curl -fSL https://github.com/GUI/nginx-upstream-dynamic-servers/archive/refs/tags/v0.4.0.tar.gz -o  nginx-upstream-dynamic-servers.v0.4.0.tar.gz \
    && tar zxvf nginx-upstream-dynamic-servers.v0.4.0.tar.gz \

  5. 修改了一些其他组件的下载地址，有些链接失效了，例如pcre：

    curl -fSL https://ftp.exim.org/pub/pcre/pcre-${RESTY_PCRE_VERSION}.tar.gz -o pcre-${RESTY_PCRE_VERSION}.tar.gz

  6. 注意编译最后一步要把下载的东西给清空了，节省空间

编译文件下载。编译文件夹内容如下：

image-20220304121133258

具体dockerfile如下：

# Dockerfile - alpine
# https://github.com/openresty/docker-openresty

ARG RESTY_IMAGE_BASE="alpine"
ARG RESTY_IMAGE_TAG="3.12"

FROM ${RESTY_IMAGE_BASE}:${RESTY_IMAGE_TAG}

LABEL maintainer="Evan Wies <evan@neomantra.net>"

# Docker Build Arguments
ARG RESTY_IMAGE_BASE="alpine"
ARG RESTY_IMAGE_TAG="3.12"
ARG RESTY_VERSION="1.17.8.2"
ARG RESTY_OPENSSL_VERSION="1.1.1g"
ARG RESTY_OPENSSL_PATCH_VERSION="1.1.1f"
ARG RESTY_OPENSSL_URL_BASE="https://www.openssl.org/source"
ARG RESTY_PCRE_VERSION="8.44"
ARG RESTY_VTS_VERSION="0.1.18"
ARG RESTY_J="1"
ARG RESTY_CONFIG_OPTIONS="\
    --with-compat \
    --with-file-aio \
    --with-http_addition_module \
    --with-http_auth_request_module \
    --with-http_dav_module \
    --with-http_flv_module \
    --with-http_geoip_module=dynamic \
    --with-http_gunzip_module \
    --with-http_gzip_static_module \
    --with-http_image_filter_module=dynamic \
    --with-http_mp4_module \
    --with-http_random_index_module \
    --with-http_realip_module \
    --with-http_secure_link_module \
    --with-http_slice_module \
    --with-http_ssl_module \
    --with-http_stub_status_module \
    --with-http_sub_module \
    --with-http_v2_module \
    --with-http_xslt_module=dynamic \
    --with-ipv6 \
    --with-mail \
    --with-mail_ssl_module \
    --with-md5-asm \
    --with-pcre-jit \
    --with-sha1-asm \
    --with-stream \
    --with-stream_ssl_module \
    --with-threads \
    --add-module=/tmp/nginx-module-vts-0.1.18 \
    --add-module=/tmp/nginx-upstream-dynamic-servers-0.4.0 \
    "
ARG RESTY_CONFIG_OPTIONS_MORE=""
ARG RESTY_LUAJIT_OPTIONS="--with-luajit-xcflags='-DLUAJIT_NUMMODE=2 -DLUAJIT_ENABLE_LUA52COMPAT'"

ARG RESTY_ADD_PACKAGE_BUILDDEPS=""
ARG RESTY_ADD_PACKAGE_RUNDEPS=""
ARG RESTY_EVAL_PRE_CONFIGURE=""
ARG RESTY_EVAL_POST_MAKE=""

# These are not intended to be user-specified
ARG _RESTY_CONFIG_DEPS="--with-pcre \
    --with-cc-opt='-DNGX_LUA_ABORT_AT_PANIC -I/usr/local/openresty/pcre/include -I/usr/local/openresty/openssl/include' \
    --with-ld-opt='-L/usr/local/openresty/pcre/lib -L/usr/local/openresty/openssl/lib -Wl,-rpath,/usr/local/openresty/pcre/lib:/usr/local/openresty/openssl/lib' \
    "

LABEL resty_image_base="${RESTY_IMAGE_BASE}"
LABEL resty_image_tag="${RESTY_IMAGE_TAG}"
LABEL resty_version="${RESTY_VERSION}"
LABEL resty_openssl_version="${RESTY_OPENSSL_VERSION}"
LABEL resty_openssl_patch_version="${RESTY_OPENSSL_PATCH_VERSION}"
LABEL resty_openssl_url_base="${RESTY_OPENSSL_URL_BASE}"
LABEL resty_pcre_version="${RESTY_PCRE_VERSION}"
LABEL resty_config_options="${RESTY_CONFIG_OPTIONS}"
LABEL resty_config_options_more="${RESTY_CONFIG_OPTIONS_MORE}"
LABEL resty_config_deps="${_RESTY_CONFIG_DEPS}"
LABEL resty_add_package_builddeps="${RESTY_ADD_PACKAGE_BUILDDEPS}"
LABEL resty_add_package_rundeps="${RESTY_ADD_PACKAGE_RUNDEPS}"
LABEL resty_eval_pre_configure="${RESTY_EVAL_PRE_CONFIGURE}"
LABEL resty_eval_post_make="${RESTY_EVAL_POST_MAKE}"

RUN apk add --no-cache --virtual .build-deps \
        build-base \
        coreutils \
        curl \
        gd-dev \
        geoip-dev \
        libxslt-dev \
        linux-headers \
        make \
        perl-dev \
        readline-dev \
        zlib-dev \
        ${RESTY_ADD_PACKAGE_BUILDDEPS} \
    && apk add --no-cache \
        gd \
        geoip \
        libgcc \
        libxslt \
        zlib \
        ${RESTY_ADD_PACKAGE_RUNDEPS} \
    && cd /tmp \
    && if [ -n "${RESTY_EVAL_PRE_CONFIGURE}" ]; then eval $(echo ${RESTY_EVAL_PRE_CONFIGURE}); fi \
    && cd /tmp \
    && curl -fSL https://github.com/GUI/nginx-upstream-dynamic-servers/archive/refs/tags/v0.4.0.tar.gz -o  nginx-upstream-dynamic-servers.v0.4.0.tar.gz \
    && tar zxvf nginx-upstream-dynamic-servers.v0.4.0.tar.gz \
    && cd /tmp \
    && curl -fSL https://github.com/vozlt/nginx-module-vts/archive/v${RESTY_VTS_VERSION}.tar.gz -o  nginx-module-vts-${RESTY_VTS_VERSION}.tar.gz \
    && tar xzf nginx-module-vts-${RESTY_VTS_VERSION}.tar.gz \
    && cd /tmp \
    && curl -fSL https://github.com/vozlt/nginx-module-vts/archive/v${RESTY_VTS_VERSION}.tar.gz -o  nginx-module-vts-${RESTY_VTS_VERSION}.tar.gz \
    && tar xzf nginx-module-vts-${RESTY_VTS_VERSION}.tar.gz \
    && curl -fSL "${RESTY_OPENSSL_URL_BASE}/openssl-${RESTY_OPENSSL_VERSION}.tar.gz" -o openssl-${RESTY_OPENSSL_VERSION}.tar.gz \
    && tar xzf openssl-${RESTY_OPENSSL_VERSION}.tar.gz \
    && cd openssl-${RESTY_OPENSSL_VERSION} \
    && if [ $(echo ${RESTY_OPENSSL_VERSION} | cut -c 1-5) = "1.1.1" ] ; then \
        echo 'patching OpenSSL 1.1.1 for OpenResty' \
        && curl -s https://raw.githubusercontent.com/openresty/openresty/master/patches/openssl-${RESTY_OPENSSL_PATCH_VERSION}-sess_set_get_cb_yield.patch | patch -p1 ; \
    fi \
    && if [ $(echo ${RESTY_OPENSSL_VERSION} | cut -c 1-5) = "1.1.0" ] ; then \
        echo 'patching OpenSSL 1.1.0 for OpenResty' \
        && curl -s https://raw.githubusercontent.com/openresty/openresty/ed328977028c3ec3033bc25873ee360056e247cd/patches/openssl-1.1.0j-parallel_build_fix.patch | patch -p1 \
        && curl -s https://raw.githubusercontent.com/openresty/openresty/master/patches/openssl-${RESTY_OPENSSL_PATCH_VERSION}-sess_set_get_cb_yield.patch | patch -p1 ; \
    fi \
    && ./config \
      no-threads shared zlib -g \
      enable-ssl3 enable-ssl3-method \
      --prefix=/usr/local/openresty/openssl \
      --libdir=lib \
      -Wl,-rpath,/usr/local/openresty/openssl/lib \
    && make -j${RESTY_J} \
    && make -j${RESTY_J} install_sw \
    && cd /tmp \
    && curl -fSL https://ftp.exim.org/pub/pcre/pcre-${RESTY_PCRE_VERSION}.tar.gz -o pcre-${RESTY_PCRE_VERSION}.tar.gz \
    && tar xzf pcre-${RESTY_PCRE_VERSION}.tar.gz \
    && cd /tmp/pcre-${RESTY_PCRE_VERSION} \
    && ./configure \
        --prefix=/usr/local/openresty/pcre \
        --disable-cpp \
        --enable-jit \
        --enable-utf \
        --enable-unicode-properties \
    && make -j${RESTY_J} \
    && make -j${RESTY_J} install \
    && cd /tmp \
    && curl -fSL https://openresty.org/download/openresty-${RESTY_VERSION}.tar.gz -o openresty-${RESTY_VERSION}.tar.gz \
    && tar xzf openresty-${RESTY_VERSION}.tar.gz \
    && cd /tmp/openresty-${RESTY_VERSION} \
    && eval ./configure -j${RESTY_J} ${_RESTY_CONFIG_DEPS} ${RESTY_CONFIG_OPTIONS} ${RESTY_CONFIG_OPTIONS_MORE} ${RESTY_LUAJIT_OPTIONS} \
    && make -j${RESTY_J} \
    && make -j${RESTY_J} install \
    && cd /tmp \
    && if [ -n "${RESTY_EVAL_POST_MAKE}" ]; then eval $(echo ${RESTY_EVAL_POST_MAKE}); fi \
    && rm -rf \
        nginx-module-vts-${RESTY_VTS_VERSION}.tar.gz nginx-module-vts-${RESTY_VTS_VERSION} \
        openssl-${RESTY_OPENSSL_VERSION}.tar.gz openssl-${RESTY_OPENSSL_VERSION} \
        pcre-${RESTY_PCRE_VERSION}.tar.gz pcre-${RESTY_PCRE_VERSION} \
        openresty-${RESTY_VERSION}.tar.gz openresty-${RESTY_VERSION} \
        * \
    && apk del .build-deps \
    && mkdir -p /var/run/openresty \
    && ln -sf /dev/stdout /usr/local/openresty/nginx/logs/access.log \
    && ln -sf /dev/stderr /usr/local/openresty/nginx/logs/error.log

# Add additional binaries into PATH for convenience
ENV PATH=$PATH:/usr/local/openresty/luajit/bin:/usr/local/openresty/nginx/sbin:/usr/local/openresty/bin

# Copy nginx configuration files
COPY nginx.conf /usr/local/openresty/nginx/conf/nginx.conf
COPY nginx.vh.default.conf /etc/nginx/conf.d/default.conf

CMD ["/usr/local/openresty/bin/openresty", "-g", "daemon off;"]

# Use SIGQUIT instead of default SIGTERM to cleanly drain requests
# See https://github.com/openresty/docker-openresty/blob/master/README.md#tips--pitfalls
STOPSIGNAL SIGQUIT

编译命令：

docker build -t kelvinblood/openresty:v3.12.1 .

参考资料

阶级跨越 pandoc 转换 markdown 为 pdf