一、概述
NGINX 代理是非常常見的,內網生成環境一般不讓在本機直接訪問,這時候就得利用代理,但是一般代理就得脫敏或去掉頁面上的高危操作。這裡就拿 YARN 代理來講解。這裡也會涉及到 subs_filter 講解,如果對 subs_filter 不理解的小夥伴,可以先查閱我這篇文章:NGINX sub_filter 和 subs_filter 指令講解
![](https://news.xinpengboligang.com/upload/keji/1eabb4e54d0e3e10019dccd07313304f.jpeg)
二、部署 Hadoop 環境
Hadoop 的部署可以參考我之前的以下幾篇文章:
- 大數據Hadoop原理介紹 安裝 實戰操作(HDFS YARN MapReduce)
- 大數據Hadoop之——Hadoop 3.3.4 HA(高可用)原理與實現(QJM)
- Hadoop on k8s 編排部署進階篇
- Hadoop HA on k8s 編排部署進階篇
- 通過 docker-compose 快速部署 Hadoop 集群極簡教程
三、部署 yarn-proxy on docker
這裡部署的是基於GitHub上的一個項目:https://github.com/HBigdata/yarn-reverse-proxy
1)部署 docker
# 安裝yum-config-manager配置工具
yum -y install yum-utils
# 建議使用阿裡雲yum源:(推薦)
#yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
# 安裝docker-ce版本
yum install -y docker-ce
# 啟動並開機啟動
systemctl enable --now docker
docker --version
2)部署 docker-compose
curl -SL https://github.com/docker/compose/releases/download/v2.16.0/docker-compose-linux-x86_64 -o /usr/local/bin/docker-compose
chmod x /usr/local/bin/docker-compose
docker-compose --version
3)部署 dnsmasq
![](https://news.xinpengboligang.com/upload/keji/4294898497de3a7632b49198564aae16.jpeg)
- dnsmasq 先去解析 /etc/hosts 文件。
- 再去解析 /etc/dnsmasq.d/ 下的 *.conf 文件,並且這些文件的優先級要高於 /etc/dnsmasq.conf。
- 我們自定義的 resolv.dnsmasq.conf中的 DNS 也被稱為上遊 DNS ,這是最後去查詢解析的。
yum install -y dnsmasq
systemctl start dnsmasq.service
把 hadoop 相關的主機名映射都提前配置到 /etc/hosts 文件中。
想了解更多,可以參考我這篇文章:dnsmasq( DNS和DHCP)服務
4)下載 yarn-proxy
git clone https://github.com/HBigdata/yarn-reverse-proxy.git
cd yarn-reverse-proxy-master
cp docker-compose.example.yml docker-compose.yml
5)修改配置
version: '3'
services:
app:
build: .
#expose:
# - 80
## WARNING: uncomment the following only if you are
## not directly exposing a non-kerberized cluster
## to the Internet
# ports:
# - "5555:80"
environment:
TZ : 'Asia/Shanghai'
dns:
- localhost
ports:
- "80:80"
# 指定 DNS 服務
environment:
# Because your primary resource-manager is at /rm1
# access your main status page at $SERVER_URL_BASE/rm1
SERVERS: "hostname-rm-110:8088@rm110
hostname-rm-111:8088@rm111
hostname-nm-112:8042@nm112
hostname-nm-113:8042@nm113
hostname-nm-114:8042@nm114
hostname-nm-115:8042@nm115
hostname-nm-116:8042@nm116
hostname-nm-117:8042@nm117
hostname-nm-118:8042@nm118
hostname-jn-119:19888@jh119"
RESOLVER: YOUR_NETWORKS_DNS_SERVER # or 127.0.0.11 for the system resolver
SERVER_URL_BASE: http://REVERSE_PROXY_HOSTNAME:80
需要修改的地方:
- SERVERS、REVERSE_PROXY_HOSTNAME
- dns 地址記得更好成自己的 dns 機器。
6)開始部署
docker-compose build
docker-compose up -d
7)通過 subs_filter 進行改進
如果nm節點太多了,都遍歷代理的話,就導致超過NGINX配置的長度的,太長配置也影響閱讀,所以這裡就通過 nginx subs_filter 進行改進。
subs_filter 的部署與用法可以參考我這篇文章:NGINX sub_filter 和 subs_filter 指令講解
通過正則表達式修改 run.sh
# 將這段內容替換成正則
cat <<EOF
# Rewrite references without the scheme, e.g. //google.com
sub_filter 'src="//$REWRITE_NAME:$REWRITE_PORT' 'src="$SERVER_URL_BASE/$REWRITE_ALIAS';
sub_filter 'href="//$REWRITE_NAME:$REWRITE_PORT' 'href="$SERVER_URL_BASE/$REWRITE_ALIAS';
sub_filter 'href=\'//$REWRITE_NAME:$REWRITE_PORT' 'href=\'$SERVER_URL_BASE/$REWRITE_ALIAS';
# Rewrite all absolute references, regardless of where
# they appear in the page
sub_filter 'http://$REWRITE_NAME:$REWRITE_PORT/' '$SERVER_URL_BASE/$REWRITE_ALIAS/';
sub_filter 'http://$REWRITE_NAME:$REWRITE_PORT' '$SERVER_URL_BASE/$REWRITE_ALIAS';
proxy_redirect http://$REWRITE_NAME:$REWRITE_PORT/ $SERVER_URL_BASE/$REWRITE_ALIAS/;
EOF
# 示例替換
cat <<EOF
# Rewrite references without the scheme, e.g. //google.com
sub_filter 'src="//local-168-(\w )-(\d ):$REWRITE_PORT' 'src="$SERVER_URL_BASE/\$1\$2' g r;
sub_filter 'href="//local-168-(\w )-(\d ):$REWRITE_PORT' 'href="$SERVER_URL_BASE/\$1\$2' g r;
sub_filter 'href=\'//local-168-(\w )-(\d ):$REWRITE_PORT' 'href=\'$SERVER_URL_BASE/\$1\$2' g r;
# Rewrite all absolute references, regardless of where
# they appear in the page
sub_filter 'http://local-168-(\w )-(\d ):$REWRITE_PORT/' '$SERVER_URL_BASE/\$1\$2/' g r;
sub_filter 'http://local-168-(\w )-(\d ):$REWRITE_PORT' '$SERVER_URL_BASE/\$1\$2' g r;
proxy_redirect http://local-168-(\w )-(\d ):$REWRITE_PORT/ $SERVER_URL_BASE/\$1\$2/ g r;
EOF
四、部署 yarn-proxy on k8s
yarn-proxy on k8s 部署跟 docker-compose 差不多,這裡就不再重復講解了,有任何疑問歡迎給我留言或私信。
NGINX 實戰操作(yarn-proxy)講解就先到這裡了,有任何疑問也可關註我公眾號:大數據與雲原生技術分享,進行技術交流,如本篇文章對您有所幫助,麻煩幫忙一鍵三連(點贊、轉發、收藏)~