WLOADCTL Kettle Plugin: Direct-Core Invocation

Previously, when WLOADCTL called Kettle jobs, it actually invoked them through the pan and kitchen commands. Every single call would re-initialize the entire Kettle runtime environment and it consumed a significant amount of time. On top of that, each instance launched was effectively a new JVM process, and every JVM would consume several hundred MB of memory by default. So running several jobs concurrently could easily eat up a large amount of memory. In some extreme cases, this could even trigger an out-of-memory error and cause a fatal crash.

After technical upgrades, WLOADCTL now calls the Kettle core directly to run jobs. This has dramatically improved scheduling efficiency and reduced resource consumption (excluding the resources consumed by the job itself). Running multiple jobs in parallel is now much more easier.

---

I. Installation & Deployment

1. download WLOADCTL in our official website
2. Make sure the installation environment already has a Kettle program directory in place.
3. Download the Kettle plugin installation package and extract it to a directory of your choice.
4. Navigate to the installation directory and run the install command. Note: on Windows, this must be run as Administrator.
5. Follow the prompts from the install program to confirm the Kettle installation directory, IP address and port, repository information, etc.
6. Once installation is complete, cprunsoapkjb.sh (for calling jobs) and cprunsoapktr.sh (for calling transformations) will be generated. Following the on-screen install instructions, place them into WLOADCTL's plugin directory respectively.
7. Run the startup command to start the plugin service. Installation complete!

---

II. Usage Guide

1. In the desktop application Admin, configure the Kettle plugin (linking it to the cprunsoap*.sh plugin scripts), as shown below:

1. Use Designer to design a Kettle scheduling workflow, as in the example below:

<ktrjob>
  <name>ktr2</name>
  <progname>$HOME/mykettle/myktr.ktr</progname>
  <para>filename=output,line=$(msystime)</para>
  <jobdesc>Direct-core transformation calls are fast</jobdesc>
</ktrjob>
<kjbjob>
  <name>kjb5</name>
  <progname>$HOME/mykettle/mykjb.kjb</progname>
  <para>filename=output,line=$(msystime)</para>
  <jobdesc>Linux file system</jobdesc>
</kjbjob>

1. Unzip the sample mykettle.zip and copy its contents into the mykettle/ directory under the plugin's installation environment (on Windows, place it under d:\mykettle).

2. Compile the workflow and run the job. You'll be able to find the output files generated by the Kettle job inside the mykettle directory.

---

III. FAQ

Q: I previously installed plugin-kettle-plugin version 1.0.x. How do I upgrade?
A: Simply reinstall plugin-kettle 1.2.0. There's no need to reconfigure the job type in Admin.

Q: Can I call jobs stored in a Kettle repository?
A: Yes, both database repositories and file-based repositories are supported. During plugin installation, choose the repository type:

DB (database repository) or FILE (file repository)

and follow the prompts accordingly.

Q: When calling a Kettle job on a Windows platform, do I still need to install WLOADCTL's Windows agent?
A: No. This plugin uses the SOAP protocol for access. You only need to install the plugin in the Windows environment itself.

Q: I'm getting the error "Can not Access the http://xxx.xxx.xxx.xxx:xxxx/ SoapAPIService"
A: You need to run the startup command to start the plugin service, or check whether port 8099 (default) is open.

Q: How do I pass parameters to a job?
A: Use the format para1=value1,para2=value2. The named parameters para1 and para2 need to be predefined within the job itself.

Q: How do I adjust the job's log output level?
A: In the plugin's conf directory, configure the ctlconf.int file. Set PLUGIN.KETTLE.LOGLEVEL=[Kettle log level], then restart the plugin.

Q: How long are plugin logs retained?
A: 30 days.

If you encounter other issues, please contact [e-mail]