你的浏览器禁用了JavaScript, 请开启后刷新浏览器获得更好的体验!
首页
热门
推荐
精选
登录
|
注册
Apache Nifi在Windows环境下搭建伪群集及证书登录
立即下载
用AI写一个
该例子支持:好用才打赏哦
现在下载学习
发布时间:2017-11-23
5人
|
浏览:4546次
|
收藏
|
分享
技术:Nifi
运行环境:windows8.1
概述
关于Windows上搭建个伪分布式集群的分享,同时通过另外一种方式实现Apache Nifi的授权认证
详细
前些时间做了关于`Apache Nifi`分布式集群的搭建分享,但很多时候要搭建分布式集群机器资源是个问题,而现在的单机的配置还是相当不错的,故现在就做个关于Windows上搭建个伪分布式集群的分享,同时通过另外一种方式实现**Apache Nifi**的授权认证。 # 系统环境及软件版本 - Windows8.1 - JDK1.8.0_131 - Nifi-1.4.0 | Nifi安装目录 | WEB端口 | | :----------------: | :---: | | xxx\nifi-ncm | 9443 | | xxx\nifi-cluster01 | 9444 | | xxx\nifi-cluster02 | 9445 | > (其它版本可参考此篇文章) > 另在测试中发个问题,使用`Apache Nifi`内嵌的`Zookeeper`搭建伪集群里启动总是提示端口占用的问题,故放弃只采用了单结点启动。 # **Nifi**的服务证书 ## 生成本地**Nifi**服务证书 解压`nifi-toolkit-1.4.0-bin.tar.gz`文件后,通过**CMD**进入`bin`目录,执行以下的命令: ```shell D:\DevelopTools\nifi-toolkit-1.4.0\bin>tls-toolkit.bat standalone -n "localhost( 3)" -C "CN=Admin, OU=ApacheNIFI" -o "..\target" 2017/10/26 18:21:32 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolki tStandaloneCommandLine: No nifiPropertiesFile specified, using embedded one. 2017/10/26 18:21:32 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolki tStandalone: Running standalone certificate generation with output directory ..\ target ****************************************************************************** 2017/10/26 18:21:34 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolki tStandalone: Successfully generated client certificate ..\target\CN=Admin_OU=Apa cheNIFI.p12 2017/10/26 18:21:34 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolki tStandalone: tls-toolkit standalone completed successfully ``` 生成后的目录结构如下: ```Shell Folder PATH listing for volume senhui.li Volume serial number is 000000F0 FA46:A0EB D:. │ CN=Admin_OU=ApacheNIFI.p12 │ CN=Admin_OU=ApacheNIFI.password │ nifi-cert.pem │ nifi-key.key │ ├─localhost │ keystore.jks │ nifi.properties │ truststore.jks │ ├─localhost_2 │ keystore.jks │ nifi.properties │ truststore.jks │ └─localhost_3 keystore.jks nifi.properties truststore.jks ``` > **特意注意:** *-C "CN=Admin, OU=ApacheNIFI"* 中间的空格必须保留 ## 拷贝**Nifi**服务证书 - 将`localhost`目录下的文件拷贝到`nifi-ncm`目录下替换所有的文件 - 将`localhost_2`目录下的文件拷贝到`nifi-cluster01`目录下替换所有的文件 - 将`localhost_3`目录下的文件拷贝到`nifi-cluster02`目录下替换所有的文件 - 将`CN=Admin_OU=ApacheNIFI.p12`和`CN=Admin_OU=ApacheNIFI.password`拷贝到桌面备用,后续登录需要使用 # 配置单点Zookeeper相关 ## 创建目录及id 进入`nifi-ncm`的目录,创建woker目录,并把server id写到文件中,命令如下: ```shell D:\DevelopTools\nifi-ncm>mkdir -p state\zookeeper D:\DevelopTools\nifi-ncm>echo -n '1' > state/zookeeper/myid ``` ## 更新**ZK**配置 进入`nifi-ncm`的conf目录,打开`zookeeper.properties`文件,内容更新参考如下: ```properties clientPort=2181 initLimit=10 autopurge.purgeInterval=24 syncLimit=5 tickTime=2000 dataDir=./state/zookeeper autopurge.snapRetainCount=30 # 只需要配置端口服务 server.1=localhost:2181 ``` ## 更新Nifi配置 进入`nifi-ncm`的conf目录,打开`nifi.properties`文件,更新如下的配置属性: ```properties nifi.state.management.embedded.zookeeper.start=true # zookeeper properties, used for cluster management # # 另外两个节点,只要编辑此字段即可 nifi.zookeeper.connect.string=localhost:2181 ``` ## 更新State配置 进入`nifi-ncm`的conf目录,打开`state-management.xml`文件,更新**zookeeper**配置,如下: ```xml
zk-provider
org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider
localhost:2181
/nifi
10 seconds
Open
``` > 然后把此文件拷贝到`nifi-cluster01`和`nifi-cluster02`相同的目录下 # 配置Nifi Admin ## 添加Admin用户 进入`nifi-ncm`的conf目录,打开`authorizers.xml`文件,找到**file-provider**添加如下配置: ```xml
file-provider
org.apache.nifi.authorization.FileAuthorizer
./conf/authorizations.xml
./conf/users.xml
CN=Admin, OU=ApacheNifi
CN=localhost, OU=NIFI
CN=localhost_2, OU=NIFI
CN=localhost_3, OU=NIFI
``` 然后把此文件同时拷贝到别外两个节点目录。 > **注:** 在`Node Identity x`中的OU要写成***NIFI***,尝试过用别的名称好像不成功,具体的原因未知,感兴趣的可以自行探究一二。 ## 安装证书 打开谷歌浏览器,在设置中找到安全选项中找到**管理证书**,点击Import开始导入上面生成的证书:`CN=Admin_OU=ApacheNIFI.p12`,密码在后缀名为`.password`的文件中,如下图所示: ![WIN-NIFI-CLUSTER-00](http://7xi700.com1.z0.glb.clouddn.com/WIN-NIFI-CLUSTER-00.png-alias) ![WIN-NIFI-CLUSTER-01](http://7xi700.com1.z0.glb.clouddn.com/WIN-NIFI-CLUSTER-01.png-alias) # 启动Nifi服务 进入到Nifi安装目录,然后在bin目录中找到run-nifi.bat文件并双击运行,注意启动的顺序: nifi-ncm-->nifi-cluster01/2,等待片刻后(可能会有点久,需要一个选举的过程)打开浏览器输入`https://localhost:9443/nifi`,选择刚刚导入的证书,如看到下面的画面表示启动成功: ![WIN-NIFI-CLUSTER-02](http://7xi700.com1.z0.glb.clouddn.com/WIN-NIFI-CLUSTER-02.png-alias) ![WIN-NIFI-CLUSTER-03](http://7xi700.com1.z0.glb.clouddn.com/WIN-NIFI-CLUSTER-03.png-alias) # 用户策略 刚登录NIFI页面时,你会发现图标都是灰色的,需要赋予相应的权限才可以开始编辑权限才可以开始编辑。点击页面左侧面板上的钥匙图标,会弹出访问策略的窗口,如下图所示: 在此会看到用户列表为空,那么就要给相应的行为添加用户,点击**Create**链接即可开始添加,如下图所示: ![WIN-NIFI-CLUSTER-04](http://7xi700.com1.z0.glb.clouddn.com/WIN-NIFI-CLUSTER-04.png-alias) ![WIN-NIFI-CLUSTER-05](http://7xi700.com1.z0.glb.clouddn.com/WIN-NIFI-CLUSTER-05.png-alias) ![WIN-NIFI-CLUSTER-044](http://7xi700.com1.z0.glb.clouddn.com/WIN-NIFI-CLUSTER-044.png-alias) 待所有的权限添加完成后,便可看到NIFI页面的按钮已经点亮,可以开始创建流程。 # 示例演示 ## 模板上传 下载**WordCountDemo.zip**压缩包,解压出来有个`WordCountDemo.xml`文件。然后打开浏览器输入NIFI访问地址: `https://localhost:9443/nifi/`,点击左侧面板中的上传按钮上传模板,如下图所示: ![WIN-NIFI-CLUSTER-06](http://7xi700.com1.z0.glb.clouddn.com/WIN-NIFI-CLUSTER-06.png-alias) ## 模板代码 `Nifi`所有的流程可以导出为模板保存,输出的模板为XML文档,关键部分都加了注释说明,如下所示: ```xml
Local word count demo
326f6452-015f-1000-99be-1d670a0ae923
WordCountDemo
cf1fdc67-56e0-3629-0000-000000000000
d6c1b1d9-24fa-3e40-0000-000000000000
0.0
0.0
Local Word Count Demo
adf2a3c8-b97d-38b4-0000-000000000000
cf1fdc67-56e0-3629-0000-000000000000
1 GB
10000
cf1fdc67-56e0-3629-0000-000000000000
be4176ae-781f-3eef-0000-000000000000
PROCESSOR
0 sec
1
success
cf1fdc67-56e0-3629-0000-000000000000
5d756769-97bb-3dde-0000-000000000000
PROCESSOR
0
2870ac2c-9995-371f-0000-000000000000
cf1fdc67-56e0-3629-0000-000000000000
1 GB
10000
cf1fdc67-56e0-3629-0000-000000000000
f257102e-8389-3fc4-0000-000000000000
PROCESSOR
0 sec
1
success
cf1fdc67-56e0-3629-0000-000000000000
be4176ae-781f-3eef-0000-000000000000
PROCESSOR
0
de134a82-8649-373d-0000-000000000000
cf1fdc67-56e0-3629-0000-000000000000
808.7726989746093
39.81819076538085
426.0
WordCountDemo
938.0
be4176ae-781f-3eef-0000-000000000000
cf1fdc67-56e0-3629-0000-000000000000
1165.7726989746093
294.8181945800781
nifi-scripting-nar
org.apache.nifi
1.4.0
INFO
1
Script Engine
Script Engine
Script File
Script File
Script Body
Script Body
Module Directory
Module Directory
ALL
false
30 sec
Script Engine
Groovy
Script File
Script Body
import org.apache.commons.io.IOUtils import java.nio.charset.* def flowFile = session.get() if(!flowFile) return flowFile = session.write(flowFile, {inputStream, outputStream -> def wordCount = [:] def tellTaleHeart = IOUtils.toString(inputStream, StandardCharsets.UTF_8) def words = tellTaleHeart.split(/(!|\?|-|\.|\"|:|;|,|\s)+/)*.toLowerCase() words.each { word -> def currentWordCount = wordCount.get(word) if(!currentWordCount) { wordCount.put(word, 1) } else { wordCount.put(word, currentWordCount + 1) } } def outputMapString = wordCount.inject("", {k,v -> k += "${v.key}: ${v.value}\n"}) outputStream.write(outputMapString.getBytes(StandardCharsets.UTF_8)) } as StreamCallback) flowFile = session.putAttribute(flowFile, 'filename', 'telltale_heart_wordcount') session.transfer(flowFile, REL_SUCCESS)
Module Directory
0
0 sec
TIMER_DRIVEN
1 sec
ExecuteScript
true
failure
false
success
STOPPED
org.apache.nifi.processors.script.ExecuteScript
f257102e-8389-3fc4-0000-000000000000
cf1fdc67-56e0-3629-0000-000000000000
1354.7726989746093
75.81820983886718
nifi-standard-nar
org.apache.nifi
1.4.0
WARN
1
Directory
Directory
Conflict Resolution Strategy
Conflict Resolution Strategy
Create Missing Directories
Create Missing Directories
Maximum File Count
Maximum File Count
Last Modified Time
Last Modified Time
Permissions
Permissions
Owner
Owner
Group
Group
ALL
false
30 sec
Directory
/data/tmp/
Conflict Resolution Strategy
fail
Create Missing Directories
true
Maximum File Count
Last Modified Time
Permissions
Owner
Group
0
0 sec
TIMER_DRIVEN
1 sec
PutFile
true
failure
true
success
STOPPED
org.apache.nifi.processors.standard.PutFile
5d756769-97bb-3dde-0000-000000000000
cf1fdc67-56e0-3629-0000-000000000000
834.7726989746093
81.31820983886718
nifi-standard-nar
org.apache.nifi
1.4.0
WARN
Generate File Source
1
File Size
File Size
Batch Size
Batch Size
Data Format
Data Format
Unique FlowFiles
Unique FlowFiles
generate-ff-custom-text
generate-ff-custom-text
character-set
character-set
ALL
false
30 sec
File Size
0B
Batch Size
1
Data Format
Text
Unique FlowFiles
false
generate-ff-custom-text
Put simply NiFi was built to automate the flow of data between systems. While the term dataflow is used in a variety of contexts, we use it here to mean the automated and managed flow of information between systems. This problem space has been around ever since enterprises had more than one system, where some of the systems created data and some of the systems consumed data. The problems and solution patterns that emerged have been discussed and articulated extensively. A comprehensive and readily consumed form is found in the Enterprise Integration Patterns [eip]. Some of the high-level challenges of dataflow include: Systems fail Networks fail, disks fail, software crashes, people make mistakes. Data access exceeds capacity to consume Sometimes a given data source can outpace some part of the processing or delivery chain - it only takes one weak-link to have an issue. Boundary conditions are mere suggestions You will invariably get data that is too big, too small, too fast, too slow, corrupt, wrong, or in the wrong format. What is noise one day becomes signal the next Priorities of an organization change - rapidly. Enabling new flows and changing existing ones must be fast.
character-set
UTF-8
2000
0 sec
TIMER_DRIVEN
1 sec
GenerateFlowFile
false
success
STOPPED
org.apache.nifi.processors.standard.GenerateFlowFile
WordCountDemo
10/24/2017 10:46:13 CST
``` ## 创建流程 拖动NIFI页面顶部的模板按钮到画板空白处,点击*ADD*按钮即可,然后双击打开`WordCountDemo`组找到`PutFile`组件,修改目录地址为你机器的实际可访问路径,如下图所示: ![WIN-NIFI-CLUSTER-07](http://7xi700.com1.z0.glb.clouddn.com/WIN-NIFI-CLUSTER-07.png-alias) ![WIN-NIFI-CLUSTER-08](http://7xi700.com1.z0.glb.clouddn.com/WIN-NIFI-CLUSTER-08.png-alias) ## 启动流程 点击NIFI页面左下角的`NiFi Flow`链接返回到主面板,点击`WordCountDemo`组,然后点击左侧面板中的开始按钮启动流程,如下图所示: ![WIN-NIFI-CLUSTER-09](http://7xi700.com1.z0.glb.clouddn.com/WIN-NIFI-CLUSTER-09.png-alias) 如无异常那么此时你可在目录下找到名为**telltale_heart_wordcount**的文件,打开便可看到如下图的统计内容: ![WIN-NIFI-CLUSTER-10](http://7xi700.com1.z0.glb.clouddn.com/WIN-NIFI-CLUSTER-10.png-alias) 至此在本地搭建NIFI伪集群就完成了,有问题欢迎留言。
本实例支付的费用只是购买源码的费用,如有疑问欢迎在文末留言交流,如需作者在线代码指导、定制等,在作者开启付费服务后,可以点击“购买服务”进行实时联系,请知悉,谢谢
感谢
0
手机上随时阅读、收藏该文章 ?请扫下方二维码
相似例子推荐
评论
作者
凡梦星尘
2
例子数量
19
帮助
1
感谢
评分详细
可运行:
4.5
分
代码质量:
4.5
分
文章描述详细:
4.5
分
代码注释:
4.5
分
综合:
4.5
分
作者例子
Apache Nifi在Windows环境下搭建伪群集及证书登录
用Python实现邮件发送Hive明细数据