1

I have some key value pairs in a text file (sftp server) and would like those values to be stored somewhere in memory or as global variables so that I can use it in all my ExecuteScript processors. I have like more than 20 ExecuteScript routed based on different types of data and I need to use these key value pairs inside these processors. I cannot store these values as NiFi global variables since the file content is not the same for each run of my NiFi Data flow.

The way Im expecting is when my NiFi flow starts these file contents has to stored somewhere(memory/variables) so I can refer it in all my scripts.

Can this be done? Any suggestions or recommendations for this would be much appreciated.

Leibnitz
  • 355
  • 5
  • 19
  • What about using the PutDistributedMapCache with a Redis backing store and then in your scripts retrieving the previously saved entries? – Kolban Jan 15 '20 at 04:40
  • Can you please point to an example or how to go about that? Never tried it before – Leibnitz Jan 15 '20 at 04:49

1 Answers1

1

There are multiple ways to do this. E.g.:

Variable registry

Use the variable registry. You already pointed out the caveat, that the variable registry cannot be updated dynamically.

YAML/JSON config file

Put your configuration into a file (as you already have). Use ExecuteScript to merge the key value pairs into attributes.

Web service

Wrap you configuration file in a web service and use InvokeHttp to query it. Evaluate the response and merge values into attributes onto the FlowFile.

DistributedMapCache

Setup a DistributedMapCacheServer and Client. Use PutDistributedMapCache to store values and FetchDistributedMapCache with Put Cache Value In Attribute set to true to fetch values and put them into attributes.

How to setup DistributedMapCache: https://stackoverflow.com/a/44591909/401025

DarkLeafyGreen
  • 69,338
  • 131
  • 383
  • 601