DEV Community: Aceld

(Golang Triad)-II-Comprehensive Analysis of Go's Hybrid Write Barrier Garbage Collection

Aceld — Tue, 06 Aug 2024 08:05:22 +0000

Author:
discord: https://discord.gg/xQ8Xxfyfcz
zinx: https://github.com/aceld/zinx
github: https://github.com/aceld
aceld's home: https://yuque.com/aceld
Twitter: https://twitter.com/Aceld_

(Golang Triad)-I-Understanding the Golang Goroutine Scheduler GPM Model
(Golang Triad)-II-Comprehensive Analysis of Go's Hybrid Write Barrier Garbage Collection

Garbage Collection (GC) is an automatic memory management mechanism provided in programming languages, capable of automatically releasing unnecessary memory objects and relinquishing storage resources without the need for manual intervention by programmers. The support for GC mechanisms is widespread in modern programming languages, and the performance of GC is often considered one of the comparative metrics between different languages.

Throughout its evolution in GC, Golang has undergone several significant changes. Up to Go V1.8, the modifications to Golang's GC have been substantial. The specific major changes are as follows:

Mark and Sweep method before Go V1.3.
Three-color concurrent marking method in Go V1.5.
"Strong-Weak" three-color invariant, insertion barrier, and deletion barrier in Go V1.5.
Hybrid write barrier mechanism in Go V1.8.

This chapter will introduce the algorithmic models of each GC method, their respective advantages and disadvantages, and how Golang's GC has evolved step by step from logical requirements to the hybrid write barrier mode.

1 Mark-Sweep Algorithm in Go V1.3

Next, let's take a look at the conventional mark-sweep algorithm primarily used in Golang before version 1.3. This algorithm consists of two main steps:
(1) Mark Phase.
(2) Sweep Phase.
The process is quite straightforward: identify memory data to be cleared and then sweep them all at once. Below is a detailed explanation of the process.

1.1 Detailed Process of Mark and Sweep Algorithm

(1) In the first step, pause the program's business logic, categorize objects into reachable and unreachable, and then mark them accordingly, as shown in Figure 1.1.

Figure 1.1

The diagram represents the reachability relationship between the program and objects. Currently, the objects reachable by the program are objects 1, 2, 3, and objects 4 and 7, totaling five objects.

(2) In the second step, marking begins. The program identifies all the objects it can reach and marks them, as shown in Figure 1.2.

Figure 1.2

Therefore, objects 1, 2, 3, and objects 4 and 7, a total of five objects, are marked.

(3) In the third step, after marking is completed, the unmarked objects are then cleared, as shown in Figure 1.3.

Figure 1.3

The operation is quite simple, but there is one important point to note. The Mark and Sweep algorithm requires the program to pause during execution, known as STW (Stop The World). During STW, the CPU does not execute user code and is entirely dedicated to garbage collection. This process has a significant impact, making STW one of the biggest challenges and focal points for optimization in some garbage collection mechanisms. Therefore, during the third step, the program will temporarily halt any work and wait for the garbage collection to complete.

(4) In the fourth step, the pause stops, allowing the program to resume running. This process is then repeated in a loop until the program's lifecycle ends. This describes the Mark and Sweep algorithm.

1.2 Disadvantages of the Mark and Sweep Algorithm

The Mark and Sweep algorithm is straightforward and clear in its process, but it has some significant issues. The first is STW (Stop The World), which causes the program to pause and may lead to noticeable stuttering, an important concern. The second issue is that marking requires scanning the entire heap. The third issue is that clearing data can create heap fragmentation.

Prior to version 1.3, Go implemented garbage collection using this approach. The basic GC process involved first initiating an STW pause, then performing marking, followed by data collection, and finally stopping the STW, as shown in Figure 1.4.

From the above figure, it can be seen that the entire GC time is encapsulated within the STW period, which seems to cause the program pause to be too long, affecting the program's runtime performance. Therefore, in Go version 1.3, a simple optimization was made to advance the STW step, reducing the duration of the STW pause. As shown in Figure 1.5.

The figure mainly illustrates that the STW step was advanced because during the Sweep phase, an STW pause is not necessary. These objects are already unreachable, so there won't be any conflicts or issues during garbage collection. However, regardless of the optimization, Go version 1.3 still faces a significant issue: the Mark and Sweep algorithm pauses the entire program.

How does Go address this problem? In the next version, Go 1.5, the tricolor concurrent marking algorithm was introduced to optimize this issue.

2 Tricolor Marking in Go V1.5

In Golang, garbage collection mainly employs the tricolor marking algorithm. The GC process can run concurrently with other user goroutines, but a certain amount of STW time is still required. The tricolor marking algorithm determines which objects need to be cleared through three marking phases. This section will introduce the specific process.

2.1 The Process of Tricolor Marking

First, every newly created object is initially marked as "white," as shown in Figure 1.6.

Figure 1.6

As shown in the figure, the memory object relationships reachable by the program are illustrated on the left. The marking table on the right is used to record the current marking color classification of each object. It is important to note that the term "program" refers to a collection of objects that are root nodes. Therefore, if you expand the "program," you will get a representation similar to what is shown in Figure 1.7.

Figure 1.7

In the second step, at the start of each GC collection, the process begins by traversing all objects from the root nodes. Objects encountered during this traversal are moved from the white set to the "gray" set, as shown in Figure 1.8.

Figure 1.8

It is important to note that this traversal is a single-level, non-recursive process. It traverses one level of objects reachable from the program. As shown in the figure, if the currently reachable objects are object 1 and object 4, then by the end of this round of traversal, objects 1 and 4 will be marked as gray, and the gray marking table will include these two objects.

In the third step, the gray set is traversed. Objects referenced by gray objects are moved from the white set to the gray set. Afterward, the original gray objects are moved to the black set, as shown in Figure 1.9.

Figure 1.9

This traversal only scans gray objects. The first layer of objects reachable from the gray objects is moved from the white set to the gray set, such as objects 2 and 7. Meanwhile, the previously gray objects 1 and 4 are marked as black and moved from the gray marking table to the black marking table.

In the fourth step, repeat the third step until there are no objects left in the gray set, as shown in Figures 1.10.1 and 1.10.2.

Figure 1.10.1

Figure 1.10.2

Once all reachable objects have been traversed, the gray marking table will no longer contain any gray objects. At this point, all memory data will only have two colors: black and white. Black objects are the ones that are logically reachable (needed) by the program. These are the valid and useful data that support the normal operation of the program and cannot be deleted. White objects, on the other hand, are unreachable by the current program logic and are considered garbage data in memory, which needs to be cleared.

In the fifth step, all objects in the white marking table are reclaimed, which means collecting the garbage, as shown in Figure 1.11.

Figure 1.11

All white objects are deleted and reclaimed, leaving only the black objects, which are all the necessary dependencies.

This is the tricolor concurrent marking algorithm. It clearly demonstrates the characteristics of the tricolor marking. However, many concurrent processes might be scanned, and the memory of these concurrent processes might be interdependent. To ensure data safety during the GC process, an STW is added before starting the tricolor marking. After scanning and determining the black and white objects, the STW is lifted. However, it is evident that the performance of such a GC scan is quite low.

So, how does Go address the stuttering issue in the Mark and Sweep algorithm?

2.2 Tricolor Marking Without STW

To start with an example, if there is no STW, performance issues would be eliminated. But what would happen if the tricolor marking algorithm did not include STW? Let's explore this process in detail.

Based on the aforementioned tricolor concurrent marking algorithm, it is indeed dependent on STW. Without pausing the program, any changes in object reference relationships during the marking phase could affect the correctness of the marking results. Consider a scenario where the tricolor marking process does not use STW and what might occur.

Assume the initial state has already undergone the first round of scanning. Currently, objects 1 and 4 are black, objects 2 and 7 are gray, and the rest are white. Object 2 is pointing to object 3 through a pointer p, as shown in Figure 1.12.

Figure 1.12

Now, if the tricolor marking process does not initiate STW, any object may undergo read or write operations during the GC scan. As shown in Figure 1.13, before object 2 has been scanned, the already black-marked object 4 creates a pointer q and points to the white object 3.

Figure 1.13

Meanwhile, the gray object 2 removes the pointer p. As a result, the white object 3 is now effectively attached to the already scanned black object 4, as shown in Figure 1.14.

Figure 1.14

Then, following the normal logic of the tricolor marking algorithm, all gray objects are marked as black. Consequently, objects 2 and 7 are marked as black, as shown in Figure 1.15.

Figure 1.15

Next, the final step of the tricolor marking process is executed, where all white objects are treated as garbage and collected, as shown in Figure 1.16.

Figure 1.16

However, the final result is that object 3, which was legitimately referenced by object 4, is mistakenly collected by the GC. This is not the desired outcome for Golang's garbage collection or for developers.

2.3 Necessary Conditions for Triggering Unsafe Tricolor Marking

It can be seen that there are two situations in the tricolor marking algorithm that are undesirable:

Condition 1: A white object is referenced by a black object (the white object is attached under the black object).
Condition 2: The reachable relationship between a gray object and a white object it references is broken (the gray object loses the reference to the white object).
If both of the above conditions are met simultaneously, it results in object loss. Furthermore, in the scenario shown above, if the white object 3 has many downstream objects, they will all be cleared as well.

To prevent this phenomenon, the simplest method is STW, which directly prohibits other user programs from interfering with object reference relationships. However, the STW process leads to significant resource waste and greatly impacts all user programs. Is it possible to maximize GC efficiency and minimize STW time while ensuring no objects are lost? The answer is yes. By using a mechanism that attempts to break the above two necessary conditions, this can be achieved.

3 Barrier Mechanism in Go V1.5

The GC collector ensures that objects are not lost when one of the following two conditions is met: the "strong tricolor invariant" and the "weak tricolor invariant."

3.1 "Strong-Weak" Tricolor Invariants

（1）Strong Tricolor Invariant

No black object should reference a white object, as shown in Figure 1.17.

Figure 1.17

The Strong Tricolor Invariant essentially enforces that black objects are not allowed to reference white objects, preventing the accidental deletion of white objects.

（2）Weak Tricolor Invariant

All white objects referenced by black objects are in a gray protected state, as shown in Figure 1.18.

Figure 1.18

The Weak Tricolor Invariant emphasizes that while black objects can reference white objects, those white objects must have references from other gray objects or have a chain of gray objects upstream in their reachable path. This means that although a white object referenced by a black object is in a potentially dangerous state of being deleted, the references from upstream gray objects protect it and ensure its safety.

To adhere to these two invariants, the GC algorithm has evolved to include two types of barriers: insertion barriers and deletion barriers.

3.2 Insertion Barrier

The specific operation of an insertion barrier is as follows: when object A references object B, object B is marked as gray (object B must be marked as gray when it is attached downstream of object A).

The insertion barrier effectively satisfies the strong tricolor invariant (preventing the situation where a black object references a white object, because the white object will be forcibly marked as gray). The pseudocode for the insertion barrier is as follows:



Add downstream object (current downstream slot, new downstream object ptr) {
  // Step 1
  Mark gray (new downstream object ptr)

  // Step 2
  current downstream slot = new downstream object ptr
}

The pseudocode for inserting a write barrier in different scenarios is as follows:



// A previously had no downstream, now add a downstream object B, and B is marked gray
A.Add downstream object(nil, B)

// A replaces the downstream object C with B, and B is marked gray
A.Add downstream object(C, B)

This pseudocode logic represents the write barrier. Memory slots for black objects can be located in two places: the stack and the heap. The stack is characterized by its small capacity but requires fast access due to frequent function calls. Therefore, the "insertion barrier" mechanism is not used for objects in stack space but is applied to objects in heap space. Next, a series of process diagrams will be used to simulate a detailed process to provide a clearer view of the overall flow for readers.

Currently, let's assume a program is newly created. In the stack space, there are objects 1, 2, 3, and 5. Object 1 references object 2, object 2 references object 3, object 3 has no downstream objects, and object 5 references object 2. In the heap space, object 4 references object 7, object 7 has no downstream objects, and object 6 references no objects and is not referenced by any objects. All these memory objects are marked as white, so the white marking table includes all these objects, as shown in Figure 1.19.

Figure 1.19

Following the tricolor marking process, traverse the Root Set (root node set) in a non-recursive manner, scanning only once. This traversal identifies the first layer of gray nodes, which are object 1 and object 4. These gray nodes are also added to the gray marking table, as shown in Figure 1.20.

Figure 1.20

Following the tricolor marking method, the next step is to traverse the objects in the gray marking table, which are objects 1 and 4. The reachable objects are marked from white to gray. Meanwhile, the traversed gray objects are marked as black, as shown in Figure 2.21.

Figure 2.21

Due to the nature of concurrency, at this moment, the external environment adds the white object 8 to the already black-marked object 4, and the white object 9 to the already black-marked object 1, as shown in Figure 1.22. Object 1 is in stack space, and according to the characteristics of the insertion barrier, to ensure performance, the creation of objects in stack space does not trigger the insertion barrier. However, object 4 is in heap space, so it will trigger the insertion barrier mechanism.

Figure 1.22

Due to the insertion write barrier mechanism (when a black object adds a white object, the white object is changed to gray), when object 4 in the heap adds object 8, object 8 is marked as gray, while object 9 remains white, as shown in Figure 1.23.

Figure 1.23

Next, the normal tricolor marking process continues, cycling through the steps until there are no gray nodes left. The resulting object states are shown in Figure 1.24: objects 1, 2, and 3 in the stack space are marked black, and objects 4, 7, and 8 in the heap space are marked black. The other objects, 9, 5, and 6, remain white.

Figure 1.24

At this point, the insertion barrier will not immediately trigger garbage collection but will instead perform an additional process. However, if the stack is not scanned, there may still be white objects referenced in the stack after the tricolor marking is complete (such as object 9 in the previous figure). Therefore, the stack needs to be re-scanned with the tricolor marking to ensure no objects are lost. This re-scan requires initiating an STW (Stop-The-World) pause until the tricolor marking of the stack space is finished.

All memory objects in the stack space are re-marked as white (objects 1, 2, 3, 9, and 5). During this process, STW is activated to protect these white objects, preventing any read or write operations on these protected objects by intercepting and blocking them, thus avoiding external interference (e.g., new white objects being added by black objects).

Meanwhile, other heap space objects will not trigger STW, ensuring the performance of GC collection in the heap space, as shown in Figure 1.25.

Figure 1.25

Next, within the STW-protected area, the tricolor marking process continues until all reachable white objects are scanned and no gray nodes remain. The final state is achieved where objects 1, 2, 3, and 9 are marked black. Since object 5 is still not scanned, it remains white, as shown in Figure 1.26.

Figure 1.26

This time, when all memory objects are either white or black, the STW will be stopped, and the protection layer will be released, as shown in Figure 1.27.

Figure 1.27

Finally, all remaining white nodes in the stack and heap space (objects 5 and 6) are scanned and collected. Typically, the STW duration for this stack space operation is around 10 to 100 milliseconds. After the final scan, all objects in memory will be black, as shown in Figure 1.28. This concludes the explanation of the tricolor marking and collection mechanism based on the insertion barrier.

Figure 1.28

The purpose of the insertion barrier is to ensure that when a black object is inserted, there is a gray object to protect it, or the inserted object is turned gray. The insertion barrier essentially satisfies the strong tricolor invariant, preventing mistakenly deleted white objects.

3.3 Deletion Barrier

The specific operation of a deletion barrier is as follows: if a deleted object is gray or white, it is marked as gray.

The deletion barrier essentially satisfies the weak tricolor invariant, aiming to ensure that the path from gray objects to white objects remains intact. The pseudocode for the deletion barrier is as follows:



Add downstream object (current downstream slot, new downstream object ptr) {
  // Step 1
  if (current downstream slot is gray || current downstream slot is white) {
    // The slot being removed, mark it as gray
    Mark gray (current downstream slot)
  }

  // Step 2
  current downstream slot = new downstream object ptr
}

The pseudocode for the deletion barrier in different scenarios is as follows:



// Object A removes the reference to object B. B is removed by A and marked gray (if B was previously white)
Add downstream object(B, nil)

// Object A replaces downstream B with C. B is removed by A and marked gray (if B was previously white)
Add downstream object(B, C)

The deletion barrier ensures that when an object's reference is removed or replaced by an upstream object, the removed object is marked as gray. The purpose of marking it as gray is to prevent a white object from being mistakenly collected if it is later referenced by another black object. This mechanism ensures that the deleted object, which was not intended to be collected, is properly protected. The following diagrams will simulate a detailed process to provide a clearer view of the overall workflow.

As shown in Figure 1.29, before executing the deletion barrier tricolor marking, the current memory situation is as follows: In the stack space, object 1 references object 5, object 5 references object 2, object 2 references object 3, and object 3 has no downstream objects. In the heap space, object 4 references object 7, and object 7 has no downstream objects. Object 6 is not referenced by anyone and has no downstream objects. All these objects are marked as white and added to the white marking table.

Figure 1.29

Following the tricolor marking process, the Root Set is traversed in a non-recursive manner, marking the first layer of gray nodes: objects 1 and 4. These gray nodes are also added to the gray marking table, as shown in Figure 1.30.

Figure 1.30

If gray object 1 deletes the white object 5 without triggering the deletion barrier mechanism, white object 5, along with its downstream objects 2 and 3, will be disconnected from the main chain and eventually be collected, as shown in Figure 1.31.

Figure 1.31

However, with the current tricolor marking algorithm using the deletion barrier mechanism, deleted objects are marked as gray. This is to protect object 5 and its downstream objects (consider why protection is necessary—what unexpected issues could arise if object 5 were not marked as gray? For example, as shown in Figure 2.31, if object 1 deletes object 5 but object 5 remains white, and if the process does not include STW protection, there is a high chance that, during the deletion process, an already black-marked object might reference object 5. Object 5 is still a valid memory object needed by the program, but it could be mistakenly collected by the GC as a white object. This happens because the downstream objects of black objects are not protected). Therefore, object 5 is marked as gray, as shown in Figure 1.32.

Figure 1.32

Following the tricolor marking process, the next step is to traverse the gray marking table, which includes objects 1, 4, and 5. During this traversal, their reachable objects are marked gray from white. Additionally, the gray objects being traversed are marked as black. As shown in Figure 1.33, after this round of processing, objects 1, 4, and 5 are marked as black, objects 2 and 7 are marked as gray, and objects 3 and 6 remain white.

Figure 1.33

Continue the tricolor marking process until there are no more gray nodes. The final state, as shown in Figure 1.34, indicates that all nodes except object 6 are marked as black.

Figure 1.34

Finally, execute the collection and cleanup process, removing all white objects through GC, as shown in Figure 1.35.

Figure 1.35

The above describes the tricolor marking process using the deletion barrier. The deletion barrier still allows for garbage collection in a parallel state, but this method has lower precision. Even if an object is deleted, the last pointer pointing to it can still "survive" this round of garbage collection and will only be cleared in the next GC cycle.

4 Hybrid Write Barrier in Go V1.8

While both insertion and deletion write barriers can address the issue of non-parallel processing caused by STW to some extent, each has its own limitations:

Insertion Write Barrier: Requires STW at the end to re-scan the stack and mark the live white objects referenced on the stack.
Deletion Write Barrier: Has lower accuracy in garbage collection. At the start of GC, STW scans the heap to record an initial snapshot, which protects all live objects at that moment.

Go V1.8 introduces the Hybrid Write Barrier mechanism, which avoids the need for stack re-scanning, significantly reducing STW time. This mechanism combines the advantages of both insertion and deletion write barriers. This section will discuss the rules of the hybrid write barrier and analyze some scenarios that trigger this barrier.

4.1 Hybrid Write Barrier Rules

The specific operations of the hybrid write barrier generally need to follow the following conditions:

At the start of GC, all objects on the stack are scanned and marked as black (there is no need for a second scan or STW).
Any new objects created on the stack during GC are marked as black.
Deleted objects are marked as gray.
Added objects are marked as gray.

The hybrid write barrier effectively satisfies a variant of the weak tricolor invariant. The pseudocode is as follows:



Add downstream object(current downstream slot, new downstream ptr) {
    // 1
    Mark gray(current downstream slot)    // Mark the current downstream object as gray when it is removed

    // 2
    Mark gray(new downstream ptr)

    // 3
    current downstream slot = new downstream ptr
}

Note: The barrier technique is not applied to stack objects to ensure stack operation efficiency. The hybrid write barrier is a GC barrier mechanism and is triggered only during GC execution.

Next, let's simulate the detailed process of the hybrid write barrier to provide a clearer understanding of the overall flow.

Assume the following scenario at the start of GC. The initial memory object structure is as follows: In the stack space, the Root Set references object 1, object 2 references object 3, and object 3 has no downstream references. Object 5 has no upstream objects but references object 8, which has no downstream references. In the heap space, the Root Set references object 4, which references object 7, and object 7 has no downstream references. Object 6 has no upstream objects and no downstream references. All these memory objects are marked as white and placed in the white marking table, as shown in Figure 1.36.

Figure 1.36

As GC begins, following the steps of the hybrid write barrier, the first step is to scan the stack area and mark all reachable objects as black. Therefore, when the stack scan is complete, objects 1, 2, and 3, which are reachable, are marked as black and added to the black marking table, as shown in Figure 1.37.

Figure 1.37

Next, we will analyze several scenarios involving the hybrid write barrier. This section will outline four scenarios, all starting from the point shown in Figure 2.37, where the stack space has been scanned and reachable objects are marked as black. The four scenarios are:

Heap Deletion of Reference, Becoming a Stack Downstream
Stack Deletion of Reference, Becoming a Stack Downstream
Heap Deletion of Reference, Becoming a Heap Downstream
Stack Deletion of Reference, Becoming a Heap Downstream

4.2 Scenario 1: Heap Deletion of Reference, Becoming a Stack Downstream

Scenario 1 mainly describes the case where an object is deleted from a heap object's references and becomes a downstream object of a stack object. The pseudocode is as follows:



// Pseudocode explanation:
// For example: HeapObject4 -> Object7 = Object7, meaning: Object7 is referenced by HeapObject4

StackObject1 -> Object7 = HeapObject7;  // Attach heap object 7 to stack object 1 as a downstream
HeapObject4 -> Object7 = null;          // HeapObject4 deletes reference to Object7

Now execute the first line of code from the above scenario: StackObject1 -> Object7 = HeapObject7, which adds the white object 7 to the downstream of the black stack object 1. It is important to note that because the stack does not trigger the write barrier mechanism, the white object 7 is directly attached under the black object 1, and object 7 remains white. Now, scan object 4, marking object 4 as gray, as shown in Figure 1.38.

Figure 1.38

Then execute the second line of code from the above scenario: HeapObject4 -> Object7 = null. The gray object 4 deletes the white object 7 (deletion means reassigning to null). Since object 4 is in the heap space, the write barrier is triggered, and the deleted object 7 is marked as gray, as shown in Figure 1.39.

Figure 1.39

Thus, from the scenario of Scene One, object 7 ends up being attached to the downstream of object 1. Since object 7 is gray, it won't be collected as garbage, thus it is protected. In the hybrid write barrier of Scene One, STW (Stop The World) will not be triggered again to re-scan the stack space objects. The subsequent process continues to follow the hybrid write barrier's tricolor marking logic. Eventually, objects 4 and 7 will be marked as black, and the GC will ultimately reclaim objects 5, 8, and 6.

4.3 Scenario Two: Stack Deletes Reference, Becomes Stack Downstream

Scenario Two primarily describes a situation where an object is deleted by one stack object and becomes the downstream of another stack object. The pseudocode is as follows:



// Pseudocode explanation:
// For example: stack object 2->object 3 = null, meaning: object 3 is unreferenced by object 2

new stack object 9;               // Create a new object 9 in the stack space
object 9->object 3 = object 3;    // Attach stack object 3 to the downstream of stack object 9
object 2->object 3 = null;        // Object 2 deletes the reference to object 3

Now, execute the first line of code in the above scenario: "new stack object 9". According to the hybrid write barrier's rules, any new memory object created in the stack space will be marked as black. Here, we inherit the memory layout scenario described in Figure 1.37, so object 9 is currently a black object referenced by the program's root set, as shown in Figure 1.40.

Figure 1.40

Next, execute the second line of code in the above scenario: object 9->object 3 = object 3. Object 9 adds downstream object 3. Since object 9 is in the stack space, this addition does not trigger the write barrier. Object 3 is directly attached to the downstream of object 9, as shown in Figure 1.41.

Figure 1.41

Finally, execute the third line of code in the above scenario: "object 2->object 3 = null". Object 2 will remove downstream object 3. Since object 2 is within the stack space, the write barrier mechanism is not triggered. Object 2 directly detaches object 3 from its downstream, as shown in Figure 1.42.

Figure 1.42

From the above process, we can see that in the hybrid write barrier mechanism, an object transferring from the downstream of one stack object to another stack object's downstream ensures the object's safety without the need to trigger the write barrier or the STW mechanism, since stack objects are all black. This highlights the ingenious design of the hybrid write barrier.

4.3 Scenario Three: Heap Reference Deletion, Becoming a Heap Downstream

Scenario three mainly describes a situation where an object is removed from the reference of one heap object and becomes the downstream of another heap object. The pseudocode is as follows:



// Pseudocode explanation:
// For example: heapObject4->object7 = object7, meaning object7 is referenced by heapObject4

heapObject10->object7 = heapObject7;   // Attach heap object 7 to the downstream of heap object 10
heapObject4->object7 = null;           // heap object 4 removes the reference to object 7

Now, let's restore the memory layout for scenario three. In the heap space, there is a black object 10 (whether it is black or not doesn't matter, because object 10 is a reachable heap object, and it will eventually be marked as black). Here, we do not consider the possibility of object 10 being any other color. Since black is a special case, if object 10 is white, its downstream will eventually be scanned and thus safe. Similarly, if object 10 is gray, its downstream is also safe. Only when object 10 is black, there is a risk of its downstream memory being unsafe. So, the current memory layout is as shown in Figure 1.43.

Figure 1.43

Now, executing the first line of code in the above scenario: heapObject10->object7 = heapObject7, heapObject10 adds a downstream reference to the white heap object 7. Since object 10 is within the heap space, this write operation will trigger the barrier mechanism. According to the conditions of the hybrid write barrier, the added object will be marked as gray. Thus, the white object 7 will be marked as gray, indirectly protecting the white object 6, as shown in Figure 1.44.

Figure 1.44

Next, executing the second line of code in the above scenario: heapObject4->object7 = null, the gray heap object 4 removes the downstream reference to heap object 7. Since object 4 is within the heap space, this operation triggers the barrier mechanism. According to the conditions of the hybrid write barrier, the removed object will be marked as gray. Therefore, object 7 will be marked as gray (even though it is already gray), as shown in Figure 1.45.

Figure 1.45

Through the processes described, the originally white object 7 has successfully been transferred from beneath heap object 4 to beneath black heap object 10. Both object 7 and its downstream object (object 6) are protected, and throughout this process, no STW was used, avoiding any delay in the program's performance.

4.4 Scenario Four: Stack Deletes Reference, Becomes Heap Downstream

Scenario Four primarily describes the case where an object is deleted from a stack object and becomes a downstream object of another heap object. The pseudocode is as follows:



// Pseudocode explanation:
// For example: HeapObject4 -> Object7 = Object7 means Object7 is referenced by HeapObject4

StackObject1 -> Object2 = null;        // Remove StackObject1's downstream reference to Object2
HeapObject4 -> Object7 = StackObject2; // HeapObject4 removes reference to Object7 and references new downstream Object2

Continuing from the memory layout scenario in Figure 1.37, executing the first line of code, StackObject1 -> Object2 = null, removes StackObject1's reference to StackObject2. Since StackObject1 is in the stack space, this operation does not trigger the write barrier mechanism. Consequently, StackObject1 directly deletes Object2 and all downstream objects associated with Object2, as shown in Figure 1.46.

Figure 1.46

Next, executing the second line of code in the scenario: HeapObject4 -> Object7 = StackObject2, actually performs two actions:

It removes the previous downstream white object, Object7, from HeapObject4.

It adds StackObject2 as the new downstream object for HeapObject4, as shown in Figure 1.47.

Figure 1.47

When HeapObject4 performs the two actions described, it triggers the write barrier mechanism since HeapObject4 is in the heap space. According to the rules of the hybrid write barrier, deleted objects are marked as gray, and newly added objects are also marked as gray. Consequently, Object7 is marked as gray, thereby protecting its downstream object, Object6. Object2, being newly added, will also go through the gray marking process. Since Object2 is already black (and thus safe), it will remain black, as shown in Figure 1.48.

Figure 1.48

Ultimately, Object2, which was originally referenced by a stack object, has been successfully moved to be a downstream object of HeapObject4. The memory dependencies and safety states are maintained. After several iterations, Object1, Object4, Object2, and Object3 will all be marked as black, while Object7 and Object6 will also be marked as black in this round of GC. The white memory objects reclaimed in this round of GC will be Object5 and Object8.

However, a question arises: Why were Object7 and Object6, which are already disconnected from the program's Root Set, not reclaimed? This is due to the delay issue with the hybrid write barrier. To avoid triggering STW, some memory may be delayed in being reclaimed for one cycle. In the next round of GC, if Object7 and Object6 are not referenced by external objects, they will eventually be reclaimed as white garbage memory.

5 Summary

This chapter has discussed the evolution of the garbage collection (GC) mechanism in Golang, highlighting the continuous optimization efforts aimed at improving GC performance. The garbage collection in Golang has seen significant advancements, and while this chapter covers up to Go 1.8, subsequent versions of Go have also introduced further performance improvements. However, the major changes discussed in this chapter are pivotal.

Currently, garbage collection employs a combination of the tri-color marking and barrier mechanisms, with the primary performance impact stemming from the Stop The World (STW) mechanism. STW is necessary for memory safety, but the hybrid write barrier mechanism significantly reduces the need for STW, allowing for concurrent garbage collection without pausing the program.

Readers should note the key milestones in the evolution of Golang's GC:

Go 1.3: The traditional mark-and-sweep approach required STW, resulting in low efficiency.
Go 1.5: Introduced tri-color marking combined with insertion and deletion write barriers, where write barriers were applied to heap space but not stack space. This method required an additional stack scan after the initial scan, leading to moderate efficiency.
Go 1.8: Implemented tri-color marking with a hybrid write barrier mechanism, where the stack space did not require barriers while the heap space did. This approach minimized the need for STW and achieved higher efficiency.

Each of these approaches had its limitations, but through comparison, it is evident that the hybrid write barrier mechanism in Go 1.8 offers a substantial improvement in efficiency by minimizing the use of STW.

(Golang Triad)-I-Understanding the Golang Goroutine Scheduler GPM Model

(Golang Triad)-II-Comprehensive Analysis of Go's Hybrid Write Barrier Garbage Collection

Author:
discord: https://discord.gg/xQ8Xxfyfcz
zinx: https://github.com/aceld/zinx
github: https://github.com/aceld
aceld's home: https://yuque.com/aceld

Case (IV) - KisFlow-Golang Stream Real- KisFlow in Message Queue (MQ) Applications

Aceld — Thu, 11 Jul 2024 02:23:33 +0000

Github: https://github.com/aceld/kis-flow
Document: https://github.com/aceld/kis-flow/wiki

Part1-OverView
Part2.1-Project Construction / Basic Modules
Part2.2-Project Construction / Basic Modules
Part3-Data Stream
Part4-Function Scheduling
Part5-Connector
Part6-Configuration Import and Export
Part7-KisFlow Action
Part8-Cache/Params Data Caching and Data Parameters
Part9-Multiple Copies of Flow
Part10-Prometheus Metrics Statistics
Part11-Adaptive Registration of FaaS Parameter Types Based on Reflection

Case1-Quick Start
Case2-Flow Parallel Operation
Case3-Application of KisFlow in Multi-Goroutines
Case4-KisFlow in Message Queue (MQ) Applications

Download KisFlow Source

$go get github.com/aceld/kis-flow

KisFlow Developer Documentation

KisFlow with Kafka

Sample source code

https://github.com/aceld/kis-flow-usage/tree/main/12-with_kafka

In this example, we use github.com/segmentio/kafka-go as the third-party Kafka Client SDK (developers can choose other Kafka Go tools).

package main

import (
    "context"
    "fmt"
    "github.com/aceld/kis-flow/file"
    "github.com/aceld/kis-flow/kis"
    "github.com/segmentio/kafka-go"
    "sync"
    "time"
)

func main() {
    ctx := context.Background()

    // Load Configuration from file
    if err := file.ConfigImportYaml("conf/"); err != nil {
        panic(err)
    }

    // Get the flow
    flowOrg := kis.Pool().GetFlow("CalStuAvgScore")
    if flowOrg == nil {
        panic("flowOrg is nil")
    }

    // Create a new Kafka reader
    reader := kafka.NewReader(kafka.ReaderConfig{
        Brokers:     []string{"localhost:9092"},
        Topic:       "SourceStuScore",
        GroupID:     "group1",
        MinBytes:    10e3,                   // 10KB
        MaxBytes:    10e6,                   // 10MB
        MaxWait:     500 * time.Millisecond, // Maximum wait time
        StartOffset: kafka.FirstOffset,
    })

    defer reader.Close()

    var wg sync.WaitGroup
    for i := 0; i < 5; i++ { // Use 5 consumers to consume in parallel
        wg.Add(1)
        go func() {
            // Fork a new flow for each consumer
            flowCopy := flowOrg.Fork(ctx)

            defer wg.Done()
            for {
                // Read a message from Kafka
                message, err := reader.ReadMessage(ctx)
                if err != nil {
                    fmt.Printf("error reading message: %v\n", err)
                    break
                }

                // Commit the message to the flow
                _ = flowCopy.CommitRow(string(message.Value))

                // Run the flow
                if err := flowCopy.Run(ctx); err != nil {
                    fmt.Println("err: ", err)
                    return
                }
            }
        }()
    }

    wg.Wait()

    return
}

func init() {
    // Register functions
    kis.Pool().FaaS("VerifyStu", VerifyStu)
    kis.Pool().FaaS("AvgStuScore", AvgStuScore)
    kis.Pool().FaaS("PrintStuAvgScore", PrintStuAvgScore)
}

KisFlow with Nsq

Sample source code:

https://github.com/aceld/kis-flow-usage/tree/main/13-with_nsq

This KisFlow consumer uses github.com/nsqio/go-nsq as the third-party SDK.

package main

import (
    "context"
    "fmt"
    "github.com/aceld/kis-flow/file"
    "github.com/aceld/kis-flow/kis"
    "github.com/nsqio/go-nsq"
)

func main() {
    ctx := context.Background()

    // Load Configuration from file
    if err := file.ConfigImportYaml("conf/"); err != nil {
        panic(err)
    }

    // Get the flow
    flowOrg := kis.Pool().GetFlow("CalStuAvgScore")
    if flowOrg == nil {
        panic("flowOrg is nil")
    }

    // Create a new NSQ consumer
    config := nsq.NewConfig()
    config.MaxInFlight = 5

    consumer, err := nsq.NewConsumer("SourceStuScore", "channel1", config)
    if err != nil {
        panic(err)
    }

    consumer.AddHandler(nsq.HandlerFunc(func(message *nsq.Message) error {
        // Fork a new flow for each message
        flowCopy := flowOrg.Fork(ctx)

        // Commit the message to the flow
        _ = flowCopy.CommitRow(string(message.Body))

        // Run the flow
        if err := flowCopy.Run(ctx); err != nil {
            fmt.Println("err: ", err)
            return err
        }

        return nil
    }))

    err = consumer.ConnectToNSQLookupd("localhost:4161")
    if err != nil {
        panic(err)
    }

    defer consumer.Stop()

    select {}
}

func init() {
    // Register functions
    kis.Pool().FaaS("VerifyStu", VerifyStu)
    kis.Pool().FaaS("AvgStuScore", AvgStuScore)
    kis.Pool().FaaS("PrintStuAvgScore", PrintStuAvgScore)
}

KisFlow with RocketMQ

Sample source code:

https://github.com/aceld/kis-flow-usage/tree/main/14-with_rocketmq

Using github.com/apache/rocketmq-client-go as the RocketMQ consumer SDK.

package main

import (
    "context"
    "fmt"
    "github.com/aceld/kis-flow/file"
    "github.com/aceld/kis-flow/kis"
    "github.com/apache/rocketmq-client-go/v2"
    "github.com/apache/rocketmq-client-go/v2/consumer"
    "github.com/apache/rocketmq-client-go/v2/primitive"
)

func main() {
    // Load Configuration from file
    if err := file.ConfigImportYaml("conf/"); err != nil {
        panic(err)
    }

    // Get the flow
    myFlow := kis.Pool().GetFlow("CalStuAvgScore")
    if myFlow == nil {
        panic("myFlow is nil")
    }

    // Create a new RocketMQ consumer
    c, err := rocketmq.NewPushConsumer(
        consumer.WithGroupName("group1"),
        consumer.WithNameServer([]string{"localhost:9876"}),
    )
    if err != nil {
        panic(err)
    }

    err = c.Subscribe("SourceStuScore", consumer.MessageSelector{}, func(ctx context.Context, msgs ...*primitive.MessageExt) (consumer.ConsumeResult, error) {

        for _, msg := range msgs {
            // Commit the message to the flow
            _ = myFlow.CommitRow(string(msg.Body))

        }

        // Run the flow
        if err := myFlow.Run(ctx); err != nil {
            fmt.Println("err: ", err)
            return consumer.ConsumeRetryLater, err
        }

        return consumer.ConsumeSuccess, nil
    })
    if err != nil {
        panic(err)
    }

    err = c.Start()
    if err != nil {
        panic(err)
    }

    defer c.Shutdown()

    select {}
}

Author: Aceld
GitHub: https://github.com/aceld

KisFlow Open Source Project Address: https://github.com/aceld/kis-flow

Document: https://github.com/aceld/kis-flow/wiki

Case1-Quick Start
Case2-Flow Parallel Operation
Case3-Application of KisFlow in Multi-Goroutines
Case4-KisFlow in Message Queue (MQ) Applications

Case (III) - KisFlow-Golang Stream Real- Application of KisFlow in Multi-Goroutines

Aceld — Tue, 09 Jul 2024 01:53:20 +0000

Github: https://github.com/aceld/kis-flow
Document: https://github.com/aceld/kis-flow/wiki

Case1-Quick Start
Case2-Flow Parallel Operation
Case3-Application of KisFlow in Multi-Goroutines
Case4-KisFlow in Message Queue (MQ) Applications

Download KisFlow Source

$go get github.com/aceld/kis-flow

KisFlow Developer Documentation

Source Code Example

https://github.com/aceld/kis-flow-usage/tree/main/6-flow_in_goroutines

If you need the same Flow to run concurrently in multiple Goroutines, you can use the flow.Fork() function to clone a Flow instance with isolated memory but the same configuration. Each Flow instance can then be executed in different Goroutines to compute their respective data streams.

package main

import (
    "context"
    "fmt"
    "github.com/aceld/kis-flow/file"
    "github.com/aceld/kis-flow/kis"
    "sync"
)

func main() {
    ctx := context.Background()
    // Get a WaitGroup
    var wg sync.WaitGroup

    // Load Configuration from file
    if err := file.ConfigImportYaml("conf/"); err != nil {
        panic(err)
    }

    // Get the flow
    flow1 := kis.Pool().GetFlow("CalStuAvgScore")
    if flow1 == nil {
        panic("flow1 is nil")
    }
    // Fork the flow
    flowClone1 := flow1.Fork(ctx)

    // Add to WaitGroup
    wg.Add(2)

    // Run Flow1
    go func() {
        defer wg.Done()
        // Submit a string
        _ = flow1.CommitRow(`{"stu_id":101, "score_1":100, "score_2":90, "score_3":80}`)
        // Submit a string
        _ = flow1.CommitRow(`{"stu_id":1001, "score_1":100, "score_2":70, "score_3":60}`)

        // Run the flow
        if err := flow1.Run(ctx); err != nil {
            fmt.Println("err: ", err)
        }
    }()

    // Run FlowClone1
    go func() {
        defer wg.Done()
        // Submit a string
        _ = flowClone1.CommitRow(`{"stu_id":201, "score_1":100, "score_2":90, "score_3":80}`)
        // Submit a string
        _ = flowClone1.CommitRow(`{"stu_id":2001, "score_1":100, "score_2":70, "score_3":60}`)

        if err := flowClone1.Run(ctx); err != nil {
            fmt.Println("err: ", err)
        }
    }()

    // Wait for Goroutines to finish
    wg.Wait()

    fmt.Println("All flows completed.")

    return
}

func init() {
    // Register functions
    kis.Pool().FaaS("VerifyStu", VerifyStu)
    kis.Pool().FaaS("AvgStuScore", AvgStuScore)
    kis.Pool().FaaS("PrintStuAvgScore", PrintStuAvgScore)
}

In this code snippet, we start two Goroutines to run Flow1 and its clone (FlowClone1) concurrently to calculate the final average scores for students 101, 1001, 201, and 2001.

Author: Aceld
GitHub: https://github.com/aceld

KisFlow Open Source Project Address: https://github.com/aceld/kis-flow

Document: https://github.com/aceld/kis-flow/wiki

Case1-Quick Start
Case2-Flow Parallel Operation
Case3-Application of KisFlow in Multi-Goroutines
Case4-KisFlow in Message Queue (MQ) Applications

Case (II) - KisFlow-Golang Stream Real-Time Computing - Flow Parallel Operation

Aceld — Mon, 08 Jul 2024 03:15:34 +0000

Github: https://github.com/aceld/kis-flow
Document: https://github.com/aceld/kis-flow/wiki

Case1-Quick Start
Case2-Flow Parallel Operation
Case3-Application of KisFlow in Multi-Goroutines
Case4-KisFlow in Message Queue (MQ) Applications

Download KisFlow Source

$go get github.com/aceld/kis-flow

KisFlow Developer Documentation

Source Code Example

https://github.com/aceld/kis-flow-usage/tree/main/8-connector

KisFlow Can Achieve the Combination of Two Flows via Connector

Using the combination of the following two flows, this introduction will cover the interface and usage of the Connector.

Data Flow Diagram

Case Introduction

Assume a student has four attributes:

Student ID: stu_id
Credit 1: score_1
Credit 2: score_2
Credit 3: score_3

Define Flow1: CalStuAvgScore-1-2 to calculate a student's average score of Credit 1 (score_1) and Credit 2 (score_2) (avg_score_1_2).
Define Flow2: CalStuAvgScore-3 to calculate a student's average score of Credit 3 (score_3) and avg_score_1_2, which is the average of Credit 1, Credit 2, and Credit 3. The average of Credit 1 and Credit 2 is provided by Flow1.

Flow1

Flow1 consists of 4 functions:

V (Function: VerifyStu) to verify the validity of StuId
C (Function: AvgStuScore12) to calculate the average score of Credit 1 and Credit 2
S (Function: SaveScoreAvg12) to store avg_score_1_2 in Redis
E (Function: PrintStuAvgScore) to print the average score of Credit 1 and Credit 2.

Flow2

Flow2 consists of 4 functions:

V (Function: VerifyStu) to verify the validity of StuId
L (Function: LoadScoreAvg12) to read the current student's average score of Credit 1 and Credit 2 (avg_score_1_2) calculated by Flow1
C (Function: AvgStuScore3) to calculate the average score of Credit 3 and the average score of Credit 1 and Credit 2
E (Function: PrintStuAvgScore) to print the average score of Credit 1, Credit 2, and Credit 3.

conf/func/func-AvgStuScore-3.yml

kistype: func
fname: AvgStuScore3
fmode: Calculate
source:
    name: SourceStuScore
    must:
        - stu_id

conf/func/func-LoadScoreAvg-1-2.yml

kistype: func
fname: LoadScoreAvg12
fmode: Load
source:
    name: SourceStuScore
    must:
        - stu_id
option:
    cname: Score12Cache

Basic Data Protocol

stu_proto.go

package main

type StuScore1_2 struct {
    StuId  int `json:"stu_id"`
    Score1 int `json:"score_1"`
    Score2 int `json:"score_2"`
}

type StuScoreAvg struct {
    StuId    int     `json:"stu_id"`
    AvgScore float64 `json:"avg_score"`
}

type StuScore3 struct {
    StuId      int     `json:"stu_id"`
    AvgScore12 float64 `json:"avg_score_1_2"` // score_1, score_2 avg
    Score3     int     `json:"score_3"`
}

Connector Init

The Connector defined in this project, Score12Cache, is a link resource associated with Redis. This Connector requires an initialization method for establishing a connection when KisFlow starts.

conn_init.go

package main

import (
    "context"
    "fmt"
    "github.com/aceld/kis-flow/kis"
    "github.com/aceld/kis-flow/log"
    "github.com/go-redis/redis/v8"
)

// type ConnInit func(conn Connector) error

func InitScore12Cache(connector kis.Connector) error {
    fmt.Println("===> Call Connector InitScore12Cache")

    // init Redis Conn Client
    rdb := redis.NewClient(&redis.Options{
        Addr:     connector.GetConfig().AddrString, // Redis-Server address
        Password: "",                               // password
        DB:       0,                                // select db
    })

    // Ping test
    pong, err := rdb.Ping(context.Background()).Result()
    if err != nil {
        log.Logger().ErrorF("Failed to connect to Redis: %v", err)
        return err
    }
    fmt.Println("Connected to Redis:", pong)

    // set rdb to connector
    connector.SetMetaData("rdb", rdb)

    return nil
}

Here, the successfully connected Redis instance is stored in the connector's cache variable "rdb."

    // set rdb to connector
    connector.SetMetaData("rdb", rdb)

FaaS Implementation

Function(V): VerifyStu

faas_stu_verify.go

package main

import (
    "context"
    "fmt"
    "github.com/aceld/kis-flow/kis"
    "github.com/aceld/kis-flow/serialize"
)

type VerifyStuIn struct {
    serialize.DefaultSerialize
    StuId int `json:"stu_id"`
}

func VerifyStu(ctx context.Context, flow kis.Flow, rows []*VerifyStuIn) error {
    fmt.Printf("->Call Func VerifyStu\n")

    for _, stu := range rows {
        // Filter out invalid data
        if stu.StuId < 0 || stu.StuId > 999 {
            // Terminate the current Flow process, subsequent functions of the current Flow will not be executed
            return flow.Next(kis.ActionAbort)
        }
    }

    return flow.Next(kis.ActionDataReuse)
}

VerifyStu() is used to validate data. If the data does not meet the requirements, the current data flow is terminated. Finally, the data is reused and passed to the next layer through flow.Next(kis.ActionDataReuse).

Function(C): AvgStuScore12

faas_avg_score_1_2.go

package main

import (
    "context"
    "fmt"
    "github.com/aceld/kis-flow/kis"
    "github.com/aceld/kis-flow/serialize"
)

type AvgStuScoreIn_1_2 struct {
    serialize.DefaultSerialize
    StuScore1_2
}

type AvgStuScoreOut_1_2 struct {
    serialize.DefaultSerialize
    StuScoreAvg
}

func AvgStuScore12(ctx context.Context, flow kis.Flow, rows []*AvgStuScoreIn_1_2) error {
    fmt.Printf("->Call Func AvgStuScore12\n")

    for _, row := range rows {

        out := AvgStuScoreOut_1_2{
            StuScoreAvg: StuScoreAvg{
                StuId:    row.StuId,
                AvgScore: float64(row.Score1+row.Score2) / 2,
            },
        }

        // Submit result data
        _ = flow.CommitRow(out)
    }

    return flow.Next()
}

AvgStuScore12() calculates the average score of score_1 and score_2, resulting in avg_score.

Function(S): SaveScoreAvg12

faas_save_score_avg_1_2.go

package main

import (
    "context"
    "fmt"
    "github.com/aceld/kis-flow/kis"
    "github.com/aceld/kis-flow/serialize"
    "github.com/go-redis/redis/v8"
    "strconv"
)

type SaveStuScoreIn struct {
    serialize.DefaultSerialize
    StuScoreAvg
}

func BatchSetStuScores(ctx context.Context, conn kis.Connector, rows []*SaveStuScoreIn) error {

    var rdb *redis.Client

    // Get Redis Client
    rdb = conn.GetMetaData("rdb").(*redis.Client)

    // Set data to redis
    pipe := rdb.Pipeline()

    for _, score := range rows {
        // make key
        key := conn.GetConfig().Key + strconv.Itoa(score.StuId)

        pipe.HMSet(context.Background(), key, map[string]interface{}{
            "avg_score": score.AvgScore,
        })
    }

    _, err := pipe.Exec(ctx)
    if err != nil {
        return err
    }

    return nil
}

func SaveScoreAvg12(ctx context.Context, flow kis.Flow, rows []*SaveStuScoreIn) error {
    fmt.Printf("->Call Func SaveScoreAvg12\n")

    conn, err := flow.GetConnector()
    if err != nil {
        fmt.Printf("SaveScoreAvg12(): GetConnector err = %s\n", err.Error())
        return err
    }

    if BatchSetStuScores(ctx, conn, rows) != nil {
        fmt.Printf("SaveScoreAvg12(): BatchSetStuScores err = %s\n", err.Error())
        return err
    }

    return flow.Next(kis.ActionDataReuse)
}

SaveScoreAvg12() stores the data in Redis through the bound Connector, using the key configured in the Connector. Finally, the source data is transparently transmitted to the next function.

Function(E): PrintStuAvgScore

faas_stu_score_avg_print.go

package main

import (
    "context"
    "fmt"
    "github.com/aceld/kis-flow/kis"
    "github.com/aceld/kis-flow/serialize"
)

type PrintStuAvgScoreIn struct {
    serialize.DefaultSerialize
    StuId    int     `json:"stu_id"`
    AvgScore float64 `json:"avg_score"`
}

func PrintStuAvgScore(ctx context.Context, flow kis.Flow, rows []*PrintStuAvgScoreIn) error {
    fmt.Printf("->Call Func PrintStuAvgScore, in Flow[%s]\n", flow.GetName())

    for _, row := range rows {
        fmt.Printf("stuid: [%+v], avg score: [%+v]\n", row.StuId, row.AvgScore)
    }

    return flow.Next()
}

PrintStuAvgScore() prints the average score of the current student.

Function(L): LoadScoreAvg12

faas_load_score_avg_1_2.go

package main

import (
    "context"
    "fmt"
    "github.com/aceld/kis-flow/kis"
    "github.com/aceld/kis-flow/serialize"
    "github.com/go-redis/redis/v8"
    "strconv"
)

type LoadStuScoreIn struct {
    serialize.DefaultSerialize
    StuScore3
}

type LoadStuScoreOut struct {
    serialize.DefaultSerialize
    StuScore3
}

func GetStuScoresByStuId(ctx context.Context, conn kis.Connector, stuId int) (float64, error) {

    var rdb *redis.Client

    // Get Redis Client
    rdb = conn.GetMetaData("rdb").(*redis.Client)

    // make key
    key := conn.GetConfig().Key + strconv.Itoa(stuId)

    // get data from redis
    result, err := rdb.HGetAll(ctx, key).Result()
    if err != nil {
        return 0, err
    }

    // get value
    avgScoreStr, ok := result["avg_score"]
    if !ok {
        return 0, fmt.Errorf("avg_score not found for stuId: %d", stuId)
    }

    // parse to float64
    avgScore, err := strconv.ParseFloat(avgScoreStr, 64)
    if err != nil {
        return 0, err
    }

    return avgScore, nil
}

func LoadScoreAvg12(ctx context.Context, flow kis.Flow, rows []*LoadStuScoreIn) error {
    fmt.Printf("->Call Func LoadScoreAvg12\n")

    conn, err := flow.GetConnector()
    if err != nil {
        fmt.Printf("LoadScoreAvg12(): GetConnector err = %s\n", err.Error())
        return err
    }

    for _, row := range rows {
        stuScoreAvg1_2, err := GetStuScoresByStuId(ctx, conn, row.StuId)
        if err != nil {
            fmt.Printf("LoadScoreAvg12(): GetStuScoresByStuId err = %s\n", err.Error())
            return err
        }

        out := LoadStuScoreOut{
            StuScore3: StuScore3{
                StuId:      row.StuId,
                Score3:     row.Score3,
                AvgScore12: stuScoreAvg1_2, // avg score of score1 and score2 (load from redis)
            },
        }

        // commit result
        _ = flow.CommitRow(out)
    }

    return flow.Next()
}

LoadScoreAvg12() reads the average score of score_1 and score_2 from Redis through the linked resource Redis of the bound Connector using the key configured in the Connector. It then sends the source data from upstream, along with the newly read average score of score1 and score2, to the next layer.

Function(C): AvgStuScore3

faas_stu_score_avg_3.go

package main

import (
    "context"
    "fmt"
    "github.com/aceld/kis-flow/kis"
    "github.com/aceld/kis-flow/serialize"
)

type AvgStuScore3In struct {
    serialize.DefaultSerialize
    StuScore3
}

type AvgStuScore3Out struct {
    serialize.DefaultSerialize
    StuScoreAvg
}

func AvgStuScore3(ctx context.Context, flow kis.Flow, rows []*AvgStuScore3In) error {
    fmt.Printf("->Call Func AvgStuScore3\n")

    for _, row := range rows {

        out := AvgStuScore3Out{
            StuScoreAvg: StuScoreAvg{
                StuId:    row.StuId,
                AvgScore: (float64(row.Score3) + row.AvgScore12*2) / 3,
            },
        }

        // Submit result data
        _ = flow.CommitRow(out)
    }

    return flow.Next()
}

AvgStuScore3() recalculates the average score of three scores by adding score_3 and the average score of score_1 and score_2, resulting in the final average score avg_score.

Register FaaS & CaaSInit/CaaS (Register Function/Connector)

main.go

func init() {
    // Register functions
    kis.Pool().FaaS("VerifyStu", VerifyStu)
    kis.Pool().FaaS("AvgStuScore12", AvgStuScore12)
    kis.Pool().FaaS("SaveScoreAvg12", SaveScoreAvg12)
    kis.Pool().FaaS("PrintStuAvgScore", PrintStuAvgScore)
    kis.Pool().FaaS("LoadScoreAvg12", LoadScoreAvg12)
    kis.Pool().FaaS("AvgStuScore3", AvgStuScore3)

    // Register connectors
    kis.Pool().CaaSInit("Score12Cache", InitScore12Cache)
}

Main Process

main.go

package main

import (
    "context"
    "github.com/aceld/kis-flow/file"
    "github.com/aceld/kis-flow/kis"
    "sync"
)

func RunFlowCalStuAvgScore12(ctx context.Context, flow kis.Flow) error {

    // Commit data
    _ = flow.CommitRow(`{"stu_id":101, "score_1":100, "score_2":90}`)
    _ = flow.CommitRow(`{"stu_id":102, "score_1":100, "score_2":80}`)

    // Run the flow
    if err := flow.Run(ctx); err != nil {
        return err
    }

    return nil
}

func RunFlowCalStuAvgScore3(ctx context.Context, flow kis.Flow) error {

    // Commit data
    _ = flow.CommitRow(`{"stu_id":101, "score_3": 80}`)
    _ = flow.CommitRow(`{"stu_id":102, "score_3": 70}`)

    // Run the flow
    if err := flow.Run(ctx); err != nil {
        return err
    }

    return nil
}

func main() {
    ctx := context.Background()

    // Load Configuration from file
    if err := file.ConfigImportYaml("conf/"); err != nil {
        panic(err)
    }

    var wg sync.WaitGroup
    wg.Add(2)

    go func() {
        // Run flow1 concurrently
        defer wg.Done()

        flow1 := kis.Pool().GetFlow("CalStuAvgScore12")
        if flow1 == nil {
            panic("flow1 is nil")
        }

        if err := RunFlowCalStuAvgScore12(ctx, flow1); err != nil {
            panic(err)
        }
    }()

    go func() {
        // Run flow2 concurrently
        defer wg.Done()

        flow2 := kis.Pool().GetFlow("CalStuAvgScore3")
        if flow2 == nil {
            panic("flow2 is nil")
        }

        if err := RunFlowCalStuAvgScore3(ctx, flow2); err != nil {
            panic(err)
        }
    }()

    wg.Wait()

    return
}

Two Goroutines are launched concurrently to execute Flow1 and Flow2, calculating the final average scores for student 101 and student 102.

Execution Results

===> Call Connector InitScore12Cache
Connected to Redis: PONG
Add FlowRouter FlowName=CalStuAvgScore12
===> Call Connector InitScore12Cache
Connected to Redis: PONG
Add FlowRouter FlowName=CalStuAvgScore3
->Call Func VerifyStu
->Call Func VerifyStu
->Call Func AvgStuScore12
->Call Func LoadScoreAvg12
->Call Func SaveScoreAvg12
->Call Func PrintStuAvgScore, in Flow[CalStuAvgScore12]
stuid: [101], avg score: [95]
stuid: [102], avg score: [90]
->Call Func AvgStuScore3
->Call Func PrintStuAvgScore, in Flow[CalStuAvgScore3]
stuid: [101], avg score: [90]
stuid: [102], avg score: [83.33333333333333]

In Flow[CalStuAvgScore3], we observe the final computed average scores for scores 1, 2, and 3.

Author: Aceld
GitHub: https://github.com/aceld

KisFlow Open Source Project Address: https://github.com/aceld/kis-flow

Document: https://github.com/aceld/kis-flow/wiki

Case1-Quick Start
Case2-Flow Parallel Operation
Case3-Application of KisFlow in Multi-Goroutines
Case4-KisFlow in Message Queue (MQ) Applications

Case (I) - KisFlow-Golang Stream Real-Time Computing - Quick Start Guide

Aceld — Tue, 02 Jul 2024 01:57:03 +0000

Github: https://github.com/aceld/kis-flow
Document: https://github.com/aceld/kis-flow/wiki

Case1-Quick Start
Case2-Flow Parallel Operation
Case3-Application of KisFlow in Multi-Goroutines
Case4-KisFlow in Message Queue (MQ) Applications

Download KisFlow Source

$go get github.com/aceld/kis-flow

KisFlow Developer Documentation

KisFlow Quick Start (Using Configuration Files) Source Code Example: kis-flow-usage/2-quick_start_with_config at main · aceld/kis-flow-usage

First, let's create a project with the following file structure:

Project Directory

├── Makefile
├── conf
│   ├── flow-CalStuAvgScore.yml
│   ├── func-AvgStuScore.yml
│   └── func-PrintStuAvgScore.yml
├── faas_stu_score_avg.go
├── faas_stu_score_avg_print.go
└── main.go

Flow

Define the current Flow. The current Flow is named "CalStuAvgScore", which is a data flow for calculating students' average scores.

Define two Functions. Function1 is Calculate, which is the logic for calculating students' average scores, and Function2 is Expand, which is for printing the final results.

Config

The configuration files for the Flow and Functions are as follows:

(1) Flow Config

conf/flow-CalStuAvgScore.yml

kistype: flow
status: 1
flow_name: CalStuAvgScore
flows:
 - fname: AvgStuScore
 - fname: PrintStuAvgScore

(2) Function1 Config

conf/func-AvgStuScore.yml

kistype: func
fname: AvgStuScore
fmode: Calculate
source:
 name: Student Scores
 must:
 - stu_id

(3) Function2 Config

conf/func-PrintStuAvgScore.yml

kistype: func
fname: PrintStuAvgScore
fmode: Expand
source:
 name: Student Scores
 must:
 - stu_id

Main

Next is the main logic, which is divided into three steps:

Load configuration files and get Flow instances.
Submit data.
Run the Flow.

main.go

package main

import (
    "context"
    "fmt"

    "github.com/aceld/kis-flow/file"
    "github.com/aceld/kis-flow/kis"
)

func main() {
    ctx := context.Background()

    // Load configuration from file
    if err := file.ConfigImportYaml("conf/"); err != nil {
        panic(err)
    }

    // Get the flow
    flow1 := kis.Pool().GetFlow("CalStuAvgScore")
    if flow1 == nil {
        panic("flow1 is nil")
    }

    // Submit a string
    _ = flow1.CommitRow(`{"stu_id":101, "score_1":100, "score_2":90, "score_3":80}`)
    // Submit a string
    _ = flow1.CommitRow(`{"stu_id":102, "score_1":100, "score_2":70, "score_3":60}`)

    // Run the flow
    if err := flow1.Run(ctx); err != nil {
        fmt.Println("err: ", err)
    }

    return
}

Function1

The implementation logic of the first calculation process is as follows. AvgStuScoreIn is the input data type, currently containing three scores, and AvgStuScoreOut is the output data type, which is the average score.

faas_stu_score_avg.go

package main
import (
    "context"
    "github.com/aceld/kis-flow/kis"
    "github.com/aceld/kis-flow/serialize"
)

type AvgStuScoreIn struct {
    serialize.DefaultSerialize
    StuId  int `json:"stu_id"`
    Score1 int `json:"score_1"`
    Score2 int `json:"score_2"`
    Score3 int `json:"score_3"`
}

type AvgStuScoreOut struct {
    serialize.DefaultSerialize
    StuId    int     `json:"stu_id"`
    AvgScore float64 `json:"avg_score"`
}

// AvgStuScore(FaaS) calculates students' average scores
func AvgStuScore(ctx context.Context, flow kis.Flow, rows []*AvgStuScoreIn) error {
    for _, row := range rows {
        out := AvgStuScoreOut{
            StuId:    row.StuId,
            AvgScore: float64(row.Score1+row.Score2+row.Score3) / 3,
        }
        // Submit result data
        _ = flow.CommitRow(out)
    }
    return nil
}

Function2

The logic for printing is to directly print the data as follows.

faas_stu_score_avg_print.go

package main

import (
    "context"
    "fmt"
    "github.com/aceld/kis-flow/kis"
    "github.com/aceld/kis-flow/serialize"
)

type PrintStuAvgScoreIn struct {
    serialize.DefaultSerialize
    StuId    int     `json:"stu_id"`
    AvgScore float64 `json:"avg_score"`
}

type PrintStuAvgScoreOut struct {
    serialize.DefaultSerialize
}

func PrintStuAvgScore(ctx context.Context, flow kis.Flow, rows []*PrintStuAvgScoreIn) error {
    for _, row := range rows {
        fmt.Printf("stuid: [%+v], avg score: [%+v]\n", row.StuId, row.AvgScore)
    }
    return nil
}

Output

Finally, run the program and get the following results:

Add KisPool FuncName=AvgStuScore
Add KisPool FuncName=PrintStuAvgScore
Add FlowRouter FlowName=CalStuAvgScore
stuid: [101], avg score: [90]
stuid: [102], avg score: [76.66666666666667]

2. KisFlow Quick Start (Using Native Interface, Dynamic Configuration)

Source Code Example: kis-flow-usage/1-quick_start at main · aceld/kis-flow-usage

Project Directory

├── faas_stu_score_avg.go
├── faas_stu_score_avg_print.go
└── main.go

Flow

Main

main.go

package main

import (
    "context"
    "fmt"
    "github.com/aceld/kis-flow/common"
    "github.com/aceld/kis-flow/config"
    "github.com/aceld/kis-flow/flow"
    "github.com/aceld/kis-flow/kis"
)

func main() {
    ctx := context.Background()

    // Create a new flow configuration
    myFlowConfig1 := config.NewFlowConfig("CalStuAvgScore", common.FlowEnable)

    // Create new function configuration
    avgStuScoreConfig := config.NewFuncConfig("AvgStuScore", common.C, nil, nil)
    printStuScoreConfig := config.NewFuncConfig("PrintStuAvgScore", common.E, nil, nil)

    // Create a new flow
    flow1 := flow.NewKisFlow(myFlowConfig1)

    // Link functions to the flow
    _ = flow1.Link(avgStuScoreConfig, nil)
    _ = flow1.Link(printStuScoreConfig, nil)

    // Submit a string
    _ = flow1.CommitRow(`{"stu_id":101, "score_1":100, "score_2":90, "score_3":80}`)
    // Submit a string
    _ = flow1.CommitRow(`{"stu_id":102, "score_1":100, "score_2":70, "score_3":60}`)

    // Run the flow
    if err := flow1.Run(ctx); err != nil {
        fmt.Println("err: ", err)
    }

    return
}

func init() {
    // Register functions
    kis.Pool().FaaS("AvgStuScore", AvgStuScore)
    kis.Pool().FaaS("PrintStuAvgScore", PrintStuAvgScore)
}

Function1

faas_stu_score_avg.go

package main

import (
    "context"
    "github.com/aceld/kis-flow/kis"
    "github.com/aceld/kis-flow/serialize"
)

type AvgStuScoreIn struct {
    serialize.DefaultSerialize
    StuId  int `json:"stu_id"`
    Score1 int `json:"score_1"`
    Score2 int `json:"score_2"`
    Score3 int `json:"score_3"`
}

type AvgStuScoreOut struct {
    serialize.DefaultSerialize
    StuId    int     `json:"stu_id"`
    AvgScore float64 `json:"avg_score"`
}

// AvgStuScore(FaaS) calculates students' average scores
func AvgStuScore(ctx context.Context, flow kis.Flow, rows []*AvgStuScoreIn) error {
    for _, row := range rows {

        out := AvgStuScoreOut{
            StuId:    row.StuId,
            AvgScore: float64(row.Score1+row.Score2+row.Score3) / 3,
        }

        // Submit result data
        _ = flow.CommitRow(out)
    }

    return nil
}

Function2

faas_stu_score_avg_print.go

package main

import (
    "context"
    "fmt"
    "github.com/aceld/kis-flow/kis"
    "github.com/aceld/kis-flow/serialize"
)

type PrintStuAvgScoreIn struct {
    serialize.DefaultSerialize
    StuId    int     `json:"stu_id"`
    AvgScore float64 `json:"avg_score"`
}

type PrintStuAvgScoreOut struct {
    serialize.DefaultSerialize
}

func PrintStuAvgScore(ctx context.Context, flow kis.Flow, rows []*PrintStuAvgScoreIn) error {
    for _, row := range rows {
        fmt.Printf("stuid: [%+v], avg score: [%+v]\n", row.StuId, row.AvgScore)
    }

    return nil
}

Output

Add KisPool FuncName=AvgStuScore
Add KisPool FuncName=PrintStuAvgScore
funcName NewConfig source is nil, funcName = AvgStuScore, use default unNamed Source.
funcName NewConfig source is nil, funcName = PrintStuAvgScore, use default unNamed Source.
stuid: [101], avg score: [90]
stuid: [102], avg score: [76.66666666666667]

Author: Aceld
GitHub: https://github.com/aceld

KisFlow Open Source Project Address: https://github.com/aceld/kis-flow

Document: https://github.com/aceld/kis-flow/wiki

Case1-Quick Start
Case2-Flow Parallel Operation
Case3-Application of KisFlow in Multi-Goroutines
Case4-KisFlow in Message Queue (MQ) Applications

(Part 11)Golang Framework Hands-on - Adaptive Registration of FaaS Parameter Types Based on Reflection

Aceld — Mon, 01 Jul 2024 02:54:05 +0000

Github: https://github.com/aceld/kis-flow
Document: https://github.com/aceld/kis-flow/wiki

Case1-Quick Start
Case2-Flow Parallel Operation
Case3-Application of KisFlow in Multi-Goroutines
Case4-KisFlow in Message Queue (MQ) Applications

Next, we will enhance the Function in KisFlow to better focus on processing business data. We will change the previous Function implementation:

func FuncDemo3Handler(ctx context.Context, flow kis.Flow) error {
    fmt.Println("---> Call funcName3Handler ----")
    fmt.Printf("Params = %+v\n", flow.GetFuncParamAll())

    for _, row := range flow.Input() {
        str := fmt.Sprintf("In FuncName = %s, FuncId = %s, row = %s", flow.GetThisFuncConf().FName, flow.GetThisFunction().GetId(), row)
        fmt.Println(str)
    }

    return nil
}

In this implementation, raw data is obtained from flow.Input(). We will modify it so that the business can directly obtain the specific data structure type it wants without assertions and type conversions. The modified Function extended parameter usage is roughly as follows:

proto

type StuScores struct {
    StuId  int `json:"stu_id"`
    Score1 int `json:"score_1"`
    Score2 int `json:"score_2"`
    Score3 int `json:"score_3"`
}

type StuAvgScore struct {
    StuId    int     `json:"stu_id"`
    AvgScore float64 `json:"avg_score"`
}

FaaS

type AvgStuScoreIn struct {
    serialize.DefaultSerialize
    proto.StuScores
}

type AvgStuScoreOut struct {
    serialize.DefaultSerialize
    proto.StuAvgScore
}

// AvgStuScore(FaaS) calculates the average score of students
func AvgStuScore(ctx context.Context, flow kis.Flow, rows []*AvgStuScoreIn) error {
    for _, row := range rows {
        avgScore := proto.StuAvgScore{
            StuId:    row.StuId,
            AvgScore: float64(row.Score1+row.Score2+row.Score3) / 3,
        }
        // Commit the result data
        _ = flow.CommitRow(AvgStuScoreOut{StuAvgScore: avgScore})
    }

    return nil
}

In this way, we can directly obtain the desired target output structure data through the third parameter rows, without needing assertions and conversions, thereby focusing more on the business-side development efficiency.

Of course, if you want to obtain the raw data, you can still get it from flow.Input().

This chapter will implement the above functionality in KisFlow.

11.1 Self-Describing FaaS Business Callback Functions

In this section, we will complete the conceptual transformation for self-describing FaaS. Previously, the FaaS callback was defined as:

type FaaS func(context.Context, Flow) error

We need a structure to describe this function's properties, including its name, address, number of parameters, parameter types, return type, etc.

11.1.1 FaaSDesc: Self-Describing Callback Type

Create a new file faas.go under kis-flow/kis/ and define the following structure:

kis-flow/kis/faas.go

// FaaS: Function as a Service

// Change
// type FaaS func(context.Context, Flow) error
// to
// type FaaS func(context.Context, Flow, ...interface{}) error
// to allow data transmission through arbitrary input types in variadic parameters
type FaaS interface{}

// FaaSDesc: Description of the FaaS callback business function
type FaaSDesc struct {
    FnName    string         // Function name
    f         interface{}    // FaaS function
    fName     string         // Function name
    ArgsType  []reflect.Type // List of function parameter types
    ArgNum    int            // Number of function parameters
    FuncType  reflect.Type   // Function type
    FuncValue reflect.Value  // Function value (function address)
}

The previous FaaS type is improved to interface{}, and FaaSDesc now has some attributes.

FnName: Indicates the name of the current function, such as "funcDemo1" in our previous examples. This is used to identify the function in KisFlow.
f: The defined FaaS function.
fName: The name of the function defined by f. ArgsType: The list of parameter types of the defined f function, which is a slice.
ArgNum: The number of input parameters of the defined f function.
FuncType: The data type of the defined f function.
FuncValue: The value of the defined f function (address of the schedulable function).

11.1.2 Create a New FaaSDesc Object

Below is a constructor function for creating a FaaSDesc object. The parameter types are the FunctionName in KisFlow and the defined FaaS function:

kis-flow/kis/faas.go

// NewFaaSDesc creates a FaaSDesc instance based on the registered FnName and FaaS callback function
func NewFaaSDesc(fnName string, f FaaS) (*FaaSDesc, error) {
    // The callback function FaaS, function value (function address)
    funcValue := reflect.ValueOf(f)

    // The type of the callback function FaaS
    funcType := funcValue.Type()

    // Check if the provided FaaS pointer is a function type
    if !isFuncType(funcType) {
        return nil, fmt.Errorf("provided FaaS type is %s, not a function", funcType.Name())
    }

    // Check if the provided FaaS function has exactly one return value of type error
    if funcType.NumOut() != 1 || funcType.Out(0) != reflect.TypeOf((*error)(nil)).Elem() {
        return nil, errors.New("function must have exactly one return value of type error")
    }

    // The list of parameter types of the FaaS function
    argsType := make([]reflect.Type, funcType.NumIn())

    // Get the name of the FaaS function
    fullName := runtime.FuncForPC(funcValue.Pointer()).Name()

    // Ensure the parameter list of FaaS func(context.Context, Flow, ...interface{}) error includes context.Context and kis.Flow

    // Check if the parameter list includes kis.Flow
    containsKisFlow := false
    // Check if the parameter list includes context.Context
    containsCtx := false

    // Iterate over the parameter types of the FaaS function
    for i := 0; i < funcType.NumIn(); i++ {

        // Get the type of the i-th parameter
        paramType := funcType.In(i)

        if isFlowType(paramType) {
            // Check if the parameter list includes kis.Flow
            containsKisFlow = true

        } else if isContextType(paramType) {
            // Check if the parameter list includes context.Context
            containsCtx = true

        } else if isSliceType(paramType) {
            // Check if the parameter list includes a slice type

            // Get the element type of the current slice parameter
            itemType := paramType.Elem()

            // If the current parameter is a pointer type, get the type of the structure it points to
            if itemType.Kind() == reflect.Ptr {
                itemType = itemType.Elem() // Get the type of the structure it points to
            }
        } else {
            // Other types are not supported...
        }

        // Append the current parameter type to the argsType list
        argsType[i] = paramType
    }

    if !containsKisFlow {
        // If the parameter list does not include kis.Flow, return an error
        return nil, errors.New("function parameters must have kis.Flow param, please use FaaS type like: [type FaaS func(context.Context, Flow, ...interface{}) error]")
    }

    if !containsCtx {
        // If the parameter list does not include context.Context, return an error
        return nil, errors.New("function parameters must have context, please use FaaS type like: [type FaaS func(context.Context, Flow, ...interface{}) error]")
    }

    // Return the FaaSDesc instance
    return &FaaSDesc{
        FnName:    fnName,
        f:         f,
        fName:     fullName,
        ArgsType:  argsType,
        ArgNum:    len(argsType),
        FuncType:  funcType,
        FuncValue: funcValue,
    }, nil
}

Here, we use reflection to get the related attribute values from the f function and store them in FaaSDesc. To ensure that the provided FaaS function meets the following format:

type FaaS func(context.Context, Flow, ...interface{}) error

We perform strict type checks on the context.Context and Flow parameters. The checking methods are as follows:

kis-flow/kis/faas.go

// isFuncType checks if the provided paramType is a function type
func isFuncType(paramType reflect.Type) bool {
    return paramType.Kind() == reflect.Func
}

// isFlowType checks if the provided paramType is of type kis.Flow
func isFlowType(paramType reflect.Type) bool {
    var flowInterfaceType = reflect.TypeOf((*Flow)(nil)).Elem()

    return paramType.Implements(flowInterfaceType)
}

// isContextType checks if the provided paramType is of type context.Context
func isContextType(paramType reflect.Type) bool {
    typeName := paramType.Name()

    return strings.Contains(typeName, "Context")
}

// isSliceType checks if the provided paramType is a slice type
func isSliceType(paramType reflect.Type) bool {
    return paramType.Kind() == reflect.Slice
}

In NewFaaSDesc(), we use two boolean variables, containsKisFlow and containsCtx, to check whether the parameter list includes Context and Flow types. The following code ensures compatibility when the parameter type is a structure pointer:

            // ... ... 

            // Get the current parameter type
            itemType := paramType.Elem()

            // If the current parameter is a pointer type, get the type of the structure it points to
            if itemType.Kind() == reflect.Ptr {
                itemType = itemType.Elem() // Get the type of the structure it points to
            }

            // ... ...

For example, the developer might define the FaaS function prototype as follows:

func MyFaaSDemo(ctx context.Context, flow Flow, []*A) error

or:

func MyFaaSDemo(ctx context.Context, flow Flow, []A) error

11.1.3 Registering FaaS Functions

Next, we will modify the method for registering FaaS functions in the kisPool module to register a FaaSDesc description. The modified registration method is as follows:

kis-flow/kis/pool.go

// FaaS registers a Function's business logic, indexed and registered by Function Name
func (pool *kisPool) FaaS(fnName string, f FaaS) {

    // When registering a FaaS logic callback, create a FaaSDesc description object
    faaSDesc, err := NewFaaSDesc(fnName, f)
    if err != nil {
        panic(err)
    }

    pool.fnLock.Lock() // Write lock
    defer pool.fnLock.Unlock()

    if _, ok := pool.fnRouter[fnName]; !ok {
        // Register the FaaSDesc description object into fnRouter
        pool.fnRouter[fnName] = faaSDesc
    } else {
        errString := fmt.Sprintf("KisPool FaaS Repeat FuncName=%s", fnName)
        panic(errString)
    }

    log.Logger().InfoF("Add KisPool FuncName=%s", fnName)
}

Now, the key in fnRouter remains the FunctionName, but the value is the FaaSDesc description object for the current FaaS function.

11.1.4 Dispatching FaaSDesc in kisPool

Finally, when scheduling a function, use FaaSDesc to retrieve the function address and parameter list for scheduling. The modified CallFunction() is as follows:

kis-flow/kis/pool.go

// CallFunction dispatches a Function
func (pool *kisPool) CallFunction(ctx context.Context, fnName string, flow Flow) error {

    if funcDesc, ok := pool.fnRouter[fnName]; ok {

        // Parameter list for the scheduled function
        params := make([]reflect.Value, 0, funcDesc.ArgNum)

        for _, argType := range funcDesc.ArgsType {

            // If it's a Flow type parameter, pass the value of flow
            if isFlowType(argType) {
                params = append(params, reflect.ValueOf(flow))
                continue
            }

            // If it's a Context type parameter, pass the value of ctx
            if isContextType(argType) {
                params = append(params, reflect.ValueOf(ctx))
                continue
            }

            // If it's a Slice type parameter, pass the value of flow.Input()
            if isSliceType(argType) {
                params = append(params, value)
                continue
            }

            // If the parameter is neither Flow, Context, nor Slice type, give it the zero value
            params = append(params, reflect.Zero(argType))
        }

        // Invoke the logic of the current function
        retValues := funcDesc.FuncValue.Call(params)

        // Retrieve the first return value; if nil, return nil
        ret := retValues[0].Interface()
        if ret == nil {
            return nil
        }

        // If the return value is of error type, return the error
        return retValues[0].Interface().(error)

    }

    log.Logger().ErrorFX(ctx, "FuncName: %s Cannot find in KisPool, Not Added.\n", fnName)

    return errors.New("FuncName: " + fnName + " Cannot find in KisPool, Not Added.")
}

The overall scheduling logic of the function is roughly as follows:
First, use fnName to route to the corresponding FaaSDesc from fnRouter. Iterate over the parameter list of FaaSDesc:
Extract the Context and Flow objects, extract the custom slice parameters passed in, and if the parameter is neither Flow, Context, nor Slice type, give it the zero value as shown below:

            params = append(params, reflect.Zero(argType))

Finally, execute the function scheduling:

        retValues := funcDesc.FuncValue.Call(params)

Obtain the value of the first return value error; if it is nil, return nil, otherwise return the error type.

In this way, we have successfully established the self-describing scheduling mode for FaaS. With this functionality, what can KisFlow do? In the next section, we can serialize the custom parameter data types passed in when scheduling FaaSDesc to obtain the data types expected by the developer.

11.2 Custom Data Type Serialization for FaaS Parameters

11.2.1 Serialize Interface

First, let's define a data serialization interface. Create a file named serialize.go under kis-flow/kis/ as follows:

kis-flow/kis/serialize.go

// Serialize Data Serialization Interface
type Serialize interface {
    // UnMarshal is used to deserialize KisRowArr into a specified type value.
    UnMarshal(common.KisRowArr, reflect.Type) (reflect.Value, error)
    // Marshal is used to serialize a specified type value into KisRowArr.
    Marshal(interface{}) (common.KisRowArr, error)
}

Here, KisRowArr is the data slice that we pass to each function in KisFlow, previously defined in kis-flow/common/data_type.go:

package common

// KisRow represents a row of data
type KisRow interface{}

// KisRowArr represents a batch of data for a single business process
type KisRowArr []KisRow

/*
    KisDataMap holds all data carried by the current flow
    key :  the Function ID where the data resides
    value: the corresponding KisRow
*/
type KisDataMap map[string]KisRowArr

The Serialize interface provides two methods:

UnMarshal: Used to deserialize KisRowArr into a specified type value.
Marshal: Used to serialize a specified type value into KisRowArr. KisFlow will provide a default Serialize implementation for each FaaS function, but developers can also customize their own Serialize implementations to perform custom data serialization actions on FaaS parameters.

11.2.2 Default Serialization in KisFlow

KisFlow provides a default Serialize implementation, primarily in JSON format. Create a serialize folder under kis-flow/, and then create a file named serialize_default.go under kis-flow/serialize/ with the following code for serialization and deserialization:

kis-flow/serialize/serialize_default.go

package serialize

import (
    "encoding/json"
    "fmt"
    "kis-flow/common"
    "reflect"
)

type DefaultSerialize struct{}

// UnMarshal is used to deserialize KisRowArr into a specified type value.
func (f *DefaultSerialize) UnMarshal(arr common.KisRowArr, r reflect.Type) (reflect.Value, error) {
    // Ensure the input type is a slice
    if r.Kind() != reflect.Slice {
        return reflect.Value{}, fmt.Errorf("r must be a slice")
    }

    slice := reflect.MakeSlice(r, 0, len(arr))

    // Iterate over each element and attempt deserialization
    for _, row := range arr {
        var elem reflect.Value
        var err error

        // Attempt to assert as a struct or pointer
        elem, err = unMarshalStruct(row, r.Elem())
        if err == nil {
            slice = reflect.Append(slice, elem)
            continue
        }

        // Attempt to directly deserialize a string
        elem, err = unMarshalJsonString(row, r.Elem())
        if err == nil {
            slice = reflect.Append(slice, elem)
            continue
        }

        // Attempt to serialize to JSON and then deserialize
        elem, err = unMarshalJsonStruct(row, r.Elem())
        if err == nil {
            slice = reflect.Append(slice, elem)
        } else {
            return reflect.Value{}, fmt.Errorf("failed to decode row: %v", err)
        }
    }

    return slice, nil
}

// Marshal is used to serialize a specified type value into KisRowArr (JSON serialization).
func (f *DefaultSerialize) Marshal(i interface{}) (common.KisRowArr, error) {
    var arr common.KisRowArr

    switch reflect.TypeOf(i).Kind() {
    case reflect.Slice, reflect.Array:
        slice := reflect.ValueOf(i)
        for i := 0; i < slice.Len(); i++ {
            // Serialize each element to a JSON string and add it to the slice.
            jsonBytes, err := json.Marshal(slice.Index(i).Interface())
            if err != nil {
                return nil, fmt.Errorf("failed to marshal element to JSON: %v", err)
            }
            arr = append(arr, string(jsonBytes))
        }
    default:
        // If not a slice or array type, serialize the entire structure to a JSON string.
        jsonBytes, err := json.Marshal(i)
        if err != nil {
            return nil, fmt.Errorf("failed to marshal element to JSON: %v", err)
        }
        arr = append(arr, string(jsonBytes))
    }

    return arr, nil
}

Some helper functions are defined as follows:

kis-flow/serialize/serialize_default.go

// Attempt to assert as a struct or pointer
func unMarshalStruct(row common.KisRow, elemType reflect.Type) (reflect.Value, error) {
    // Check if row is a struct or struct pointer type
    rowType := reflect.TypeOf(row)
    if rowType == nil {
        return reflect.Value{}, fmt.Errorf("row is nil pointer")
    }
    if rowType.Kind() != reflect.Struct && rowType.Kind() != reflect.Ptr {
        return reflect.Value{}, fmt.Errorf("row must be a struct or struct pointer type")
    }

    // If row is a pointer type, get its underlying type
    if rowType.Kind() == reflect.Ptr {
        // Null pointer
        if reflect.ValueOf(row).IsNil() {
            return reflect.Value{}, fmt.Errorf("row is nil pointer")
        }

        // Dereference
        row = reflect.ValueOf(row).Elem().Interface()

        // Get the type after dereferencing
        rowType = reflect.TypeOf(row)
    }

    // Check if row can be asserted to elemType (target type)
    if !rowType.AssignableTo(elemType) {
        return reflect.Value{}, fmt.Errorf("row type cannot be asserted to elemType")
    }

    // Convert row to reflect.Value and return
    return reflect.ValueOf(row), nil
}

// Attempt to directly deserialize a string (deserialize JSON string to struct)
func unMarshalJsonString(row common.KisRow, elemType reflect.Type) (reflect.Value, error) {
    // Check if the source data can be asserted as a string
    str, ok := row.(string)
    if !ok {
        return reflect.Value{}, fmt.Errorf("not a string")
    }

    // Create a new struct instance to store the deserialized value
    elem := reflect.New(elemType).Elem()

    // Attempt to deserialize the JSON string into the struct.
    if err := json.Unmarshal([]byte(str), elem.Addr().Interface()); err != nil {
        return reflect.Value{}, fmt.Errorf("failed to unmarshal string to struct: %v", err)
    }

    return elem, nil
}

// Attempt to serialize to JSON and then deserialize (convert struct to JSON string, then deserialize JSON string to struct)
func unMarshalJsonStruct(row common.KisRow, elemType reflect.Type) (reflect.Value, error) {
    // Serialize row to JSON string
    jsonBytes, err := json.Marshal(row)
    if err != nil {
        return reflect.Value{}, fmt.Errorf("failed to marshal row to JSON: %v", err)
    }

    // Create a new struct instance to store the deserialized value
    elem := reflect.New(elemType).Interface()

    // Deserialize the JSON string into the struct
    if err := json.Unmarshal(jsonBytes, elem); err != nil {
        return reflect.Value{}, fmt.Errorf("failed to unmarshal JSON to element: %v", err)
    }

    return reflect.ValueOf(elem).Elem(), nil
}

UnMarshal(): First checks if the parameter is a slice. If it is, it serializes each element in the slice. It first tries to deserialize using unMarshalStruct(), then unMarshalJsonString(), and finally unMarshalJsonStruct() if the previous attempts fail.
Marshal(): Serializes any type into a JSON binary string stored in KisRowArr.

Note: The current default serialization in KisFlow only implements JSON serialization. Developers can refer to DefaultSerialize{} to implement their own serialization and deserialization for other formats.

11.2.3 Default Serialize Instance

Define a global default serialization instance, defaultSerialize, in the serialize interface definition.

kis-flow/kis/serialize.go

// defaultSerialize is the default serialization implementation provided by KisFlow (developers can customize)
var defaultSerialize = &serialize.DefaultSerialize{}

Also, provide a method to check if a data type implements the Serialize interface:

kis-flow/kis/serialize.go

// isSerialize checks if the passed paramType implements the Serialize interface
func isSerialize(paramType reflect.Type) bool {
    return paramType.Implements(reflect.TypeOf((*Serialize)(nil)).Elem())
}

11.2.4 Implementing the Serialize Interface for FaaSDesc

Next, we will extend FaaSDesc to implement the Serialize interface. When scheduling a FaaSDesc, the input parameters passed to it will be serialized to obtain the corresponding specific type parameters. The definition is as follows:

kis-flow/kis/faas.go

// FaaSDesc describes the FaaS callback computation business function
type FaaSDesc struct {
    // +++++++
    Serialize                // Serialization implementation for the data input and output of the current function
    // +++++++
    FnName    string         // Function name
    f         interface{}    // FaaS function
    fName     string         // Function name
    ArgsType  []reflect.Type // Collection of function parameter types
    ArgNum    int            // Number of function parameters
    FuncType  reflect.Type   // Function type
    FuncValue reflect.Value  // Function value (function address)
}

Then, in the constructor method NewFaaSDesc(), add a check for custom parameters. Determine whether the passed custom parameters implement the two serialization interfaces of Serialize. If they do, use the custom serialization interface; if not, use the default DefaultSerialize{} instance.

kis-flow/kis/faas.go

// NewFaaSDesc creates an FaaSDesc description instance based on the registered FnName and FaaS callback function
func NewFaaSDesc(fnName string, f FaaS) (*FaaSDesc, error) {

    // ++++++++++
    // Input/output serialization instance
    var serializeImpl Serialize
    // ++++++++++

    // ... ...
    // ... ...

    // Iterate over the parameter types of the FaaS
    for i := 0; i < funcType.NumIn(); i++ {

        // Get the type of the i-th formal parameter
        paramType := funcType.In(i)

        if isFlowType(paramType) {
            // Check if it contains a parameter of type kis.Flow
            containsKisFlow = true

        } else if isContextType(paramType) {
            // Check if it contains a parameter of type context.Context
            containsCtx = true

        } else if isSliceType(paramType) {

            // Get the element type of the current parameter slice
            itemType := paramType.Elem()

            // If the current parameter is a pointer type, get the type pointed to by the pointer
            if itemType.Kind() == reflect.Ptr {
                itemType = itemType.Elem() // Get the type pointed to by the pointer
            }


            // +++++++++++++++++++++++++++++

            // Check if f implements the Serialize interface
            if isSerialize(itemType) {
                // If the current parameter implements the Serialize interface, use the serialization implementation of the current parameter
                serializeImpl = reflect.New(itemType).Interface().(Serialize)

            } else {
                // If the current parameter does not implement the Serialize interface, use the default serialization implementation
                serializeImpl = defaultSerialize // Use global default implementation
            }
            // +++++++++++++++++++++++++++++++

        } else {
            // Other types are not supported
        }

        // Append the current parameter type to the argsType collection
        argsType[i] = paramType
    }

    // ... ...
    // ... ...

    // Return the FaaSDesc description instance
    return &FaaSDesc{
        Serialize: serializeImpl,
        FnName:    fnName,
        f:         f,
        fName:     fullName,
        ArgsType:  argsType,
        ArgNum:    len(argsType),
        FuncType:  funcType,
        FuncValue: funcValue,
    }, nil
}

11.2.5 Completing FaaS Data Serialization During Scheduling

Finally, when scheduling FaaSDesc, if it is a custom slice parameter, deserialize the raw data of flow.Input() into the structure data required by the developer. Implement it as follows:

kis-flow/kis/pool.go

// CallFunction schedules the function
func (pool *kisPool) CallFunction(ctx context.Context, fnName string, flow Flow) error {

    if funcDesc, ok := pool.fnRouter[fnName]; ok {

        // List of parameters for the scheduled function
        params := make([]reflect.Value, 0, funcDesc.ArgNum)

        for _, argType := range funcDesc.ArgsType {

            // If it is a Flow type parameter, pass the value of flow
            if isFlowType(argType) {
                params = append(params, reflect.ValueOf(flow))
                continue
            }

            // If it is a Context type parameter, pass the value of ctx
            if isContextType(argType) {
                params = append(params, reflect.ValueOf(ctx))
                continue
            }

            // If it is a Slice type parameter, pass the value of flow.Input()
            if isSliceType(argType) {

                // +++++++++++++++++++
                // Deserialize the raw data in flow.Input() into the data of argType type
                value, err := funcDesc.Serialize.UnMarshal(flow.Input(), argType)
                if err != nil {
                    log.Logger().ErrorFX(ctx, "funcDesc.Serialize.DecodeParam err=%v", err)
                } else {
                    params = append(params, value)
                    continue
                }
                // +++++++++++++++++++
            }

            // If the passed parameter is neither Flow type, nor Context type, nor Slice type, give the default zero value
            params = append(params, reflect.Zero(argType))
        }

        // Call the computation logic of the current function
        retValues := funcDesc.FuncValue.Call(params)

        // Get the first return value; if it is nil, return nil
        ret := retValues[0].Interface()
        if ret == nil {
            return nil
        }

        // If the return value is of type error, return the error
        return retValues[0].Interface().(error)

    }

    log.Logger().ErrorFX(ctx, "FuncName: %s Can not find in KisPool, Not Added.\n", fnName)

    return errors.New("FuncName: " + fnName + " Can not find in NsPool, Not Added.")
}

This completes the integration of data serialization with the FaaSDesc module. Next, we will write a unit test to test this capability.

11.3 Unit Test for Custom Parameter Serialization

11.3.1 Definition of Flow and Function Configuration Files

For unit testing, we create two Function configurations as follows:

kis-flow/test/load_conf/func/func-avgStuScore.yml

kistype: func
fname: AvgStuScore
fmode: Calculate
source:
    name: Student Average Score
    must:
        - stu_id

kis-flow/test/load_conf/func/func-PrintStuAvgScore.yml

kistype: func
fname: PrintStuAvgScore
fmode: Expand
source:
    name: Student Average Score
    must:
        - stu_id

Next, we define a Flow to link the two functions together:

kis-flow/test/load_conf/flow/flow-StuAvg.yml

kistype: flow
status: 1
flow_name: StuAvg
flows:
    - fname: AvgStuScore
    - fname: PrintStuAvgScore

11.3.2 Definition of Custom Base Data Proto

In the kis-flow/test/ directory, create a proto/ folder and a custom base data proto for future data protocol reuse:

kis-flow/test/proto/stu_score.go

package proto

// Student Scores
type StuScores struct {
    StuId  int `json:"stu_id"`
    Score1 int `json:"score_1"`
    Score2 int `json:"score_2"`
    Score3 int `json:"score_3"`
}

// Student's Average Score
type StuAvgScore struct {
    StuId    int     `json:"stu_id"`
    AvgScore float64 `json:"avg_score"`
}

11.3.3 Define Two FaaS Callback Functions

Define two FaaS functions: one to calculate a student's average score and one to print the student's average score:

kis-flow/test/faas/faas_stu_score_avg.go

package faas

import (
    "context"
    "kis-flow/kis"
    "kis-flow/serialize"
    "kis-flow/test/proto"
)

type AvgStuScoreIn struct {
    serialize.DefaultSerialize
    proto.StuScores
}

type AvgStuScoreOut struct {
    serialize.DefaultSerialize
    proto.StuAvgScore
}

// AvgStuScore(FaaS) calculates the student's average score
func AvgStuScore(ctx context.Context, flow kis.Flow, rows []*AvgStuScoreIn) error {
    for _, row := range rows {
        avgScore := proto.StuAvgScore{
            StuId:    row.StuId,
            AvgScore: float64(row.Score1+row.Score2+row.Score3) / 3,
        }
        // Submit the result data
        _ = flow.CommitRow(AvgStuScoreOut{StuAvgScore: avgScore})
    }
    return nil
}

The AvgStuScore() function is our improved FaaS function, where the third parameter rows []*AvgStuScoreIn is a custom serialized parameter. Previously, we used flow.Input() to get the raw data and then traversed it. Although this method still works, it requires developers to manually assert and judge in the FaaS function, which increases development costs. Now, developers can describe a parameter's data through AvgStuScoreIn and use rows to get the already serialized structure, improving code readability and reducing development costs.

The implementation for printing the average score FaaS is as follows:

kis-flow/test/faas/faas_stu_score_avg_print.go

package faas

import (
    "context"
    "fmt"
    "kis-flow/kis"
    "kis-flow/serialize"
    "kis-flow/test/proto"
)

type PrintStuAvgScoreIn struct {
    serialize.DefaultSerialize
    proto.StuAvgScore
}

type PrintStuAvgScoreOut struct {
    serialize.DefaultSerialize
}

func PrintStuAvgScore(ctx context.Context, flow kis.Flow, rows []*PrintStuAvgScoreIn) error {
    for _, row := range rows {
        fmt.Printf("stuid: [%+v], avg score: [%+v]\n", row.StuId, row.AvgScore)
    }
    return nil
}

Similar to the previous function, we use custom input parameters for logic development.

11.3.4 Unit Test Case

Next, we write a test case for the above Flow:

kis-flow/test/kis_auto_inject_param_test.go

package test

import (
    "context"
    "kis-flow/common"
    "kis-flow/config"
    "kis-flow/file"
    "kis-flow/flow"
    "kis-flow/kis"
    "kis-flow/test/faas"
    "kis-flow/test/proto"
    "testing"
)

func TestAutoInjectParamWithConfig(t *testing.T) {
    ctx := context.Background()

    kis.Pool().FaaS("AvgStuScore", faas.AvgStuScore)
    kis.Pool().FaaS("PrintStuAvgScore", faas.PrintStuAvgScore)

    // 1. Load the configuration files and build the Flow
    if err := file.ConfigImportYaml("load_conf/"); err != nil {
        panic(err)
    }

    // 2. Get the Flow
    flow1 := kis.Pool().GetFlow("StuAvg")
    if flow1 == nil {
        panic("flow1 is nil")
    }

    // 3. Submit raw data
    _ = flow1.CommitRow(&faas.AvgStuScoreIn{
        StuScores: proto.StuScores{
            StuId:  100,
            Score1: 1,
            Score2: 2,
            Score3: 3,
        },
    })
    _ = flow1.CommitRow(faas.AvgStuScoreIn{
        StuScores: proto.StuScores{
            StuId:  100,
            Score1: 1,
            Score2: 2,
            Score3: 3,
        },
    })

    // Submit raw data (JSON string)
    _ = flow1.CommitRow(`{"stu_id":101}`)

    // 4. Execute flow1
    if err := flow1.Run(ctx); err != nil {
        panic(err)
    }
}

When submitting raw data, we use the default serialization method, which supports JSON deserialization. In CommitRow(), we submit three pieces of data: the first two are structure data, and the last one is a JSON string. All of them are supported.

Navigate to kis-flow/test/ and execute:

$ go test -test.v -test.paniconexit0 -test.run TestAutoInjectParamWithConfig

The result is as follows:

$ go test -test.v -test.paniconexit0 -test.run TestAutoInjectParamWithConfig
...
...
Add KisPool FuncName=AvgStuScore
Add KisPool FuncName=PrintStuAvgScore
...
...
Add FlowRouter FlowName=StuAvg
context.Background
====> After CommitSrcData, flow_name = StuAvg, flow_id = flow-1265702bc905400da1788c0354080ded
All Level Data =
 map[FunctionIdFirstVirtual:[0xc0002bab40 {DefaultSerialize:{} StuScores:{StuId:100 Score1:1 Score2:2 Score3:3}} {"stu_id":101}]]

KisFunctionC, flow = &{Id:flow-1265702bc905400da1788c0354080ded Name:StuAvg Conf:0xc000286100 Funcs:map[AvgStuScore:0xc00023af80 PrintStuAvgScore:0xc00023b000] FlowHead:0xc00023af80 FlowTail:0xc00023b000 flock:{w:{state:0 sema:0} writerSem:0 readerSem:0 readerCount:0 readerWait:0} ThisFunction:0xc00023af80 ThisFunctionId:func-12a05e62a12a45fdade8477a3bddd2fd PrevFunctionId:FunctionIdFirstVirtual funcParams:map[func-12a05e62a12a45fdade8477a3bddd2fd:map[] func-7f308d00f4fa49488760ff1dfb85dc46:map[]] fplock:{w:{state:0 sema:0} writerSem:0 readerSem:0 readerCount:0 readerWait:0} buffer:[] data:map[FunctionIdFirstVirtual:[0xc0002bab40 {DefaultSerialize:{} StuScores:{StuId:100 Score1:1 Score2:2 Score3:3}} {"stu_id":101}]] inPut:[0xc0002bab40 {DefaultSerialize:{} StuScores:{StuId:100 Score1:1 Score2:2 Score3:3}} {"stu_id":101}] abort:false action:{DataReuse:false ForceEntryNext:false JumpFunc: Abort:false} cache:0xc000210b88 metaData:map[] mLock:{w:{state:0 sema:0} writerSem:0 readerSem:0 readerCount:0 readerWait:0}}

context.Background
 ====> After commitCurData, flow_name = StuAvg, flow_id = flow-1265702bc905400da1788c0354080ded
All Level Data =
 map[FunctionIdFirstVirtual:[0xc0002bab40 {DefaultSerialize:{} StuScores:{StuId:100 Score1:1 Score2:2 Score3:3}} {"stu_id":101}] func-12a05e62a12a45fdade8477a3bddd2fd:[{DefaultSerialize:{} StuAvgScore:{StuId:100 AvgScore:2}} {DefaultSerialize:{} StuAvgScore:{StuId:100 AvgScore:2}} {DefaultSerialize:{} StuAvgScore:{StuId:101 AvgScore:0}}]]

KisFunctionE, flow = &{Id:flow-1265702bc905400da1788c0354080ded Name:StuAvg Conf:0xc000286100 Funcs:map[AvgStuScore:0xc00023af80 PrintStuAvgScore:0xc00023b000] FlowHead:0xc00023af80 FlowTail:0xc00023b000 flock:{w:{state:0 sema:0} writerSem:0 readerSem:0 readerCount:0 readerWait:0} ThisFunction:0xc00023b000 ThisFunctionId:func-7f308d00f4fa49488760ff1dfb85dc46 PrevFunctionId:func-12a05e62a12a45fdade8477a3bddd2fd funcParams:map[func-12a05e62a12a45fdade8477a3bddd2fd:map[] func-7f308d00f4fa49488760ff1dfb85dc46:map[]] fplock:{w:{state:0 sema:0} writerSem:0 readerSem:0 readerCount:0 readerWait:0} buffer:[] data:map[FunctionIdFirstVirtual:[0xc0002bab40 {DefaultSerialize:{} StuScores:{StuId:100 Score1:1 Score2:2 Score3:3}} {"stu_id":101}] func-12a05e62a12a45fdade8477a3bddd2fd:[{DefaultSerialize:{} StuAvgScore:{StuId:100 AvgScore:2}} {DefaultSerialize:{} StuAvgScore:{StuId:100 AvgScore:2}} {DefaultSerialize:{} StuAvgScore:{StuId:101 AvgScore:0}}]] inPut:[{DefaultSerialize:{} StuAvgScore:{StuId:100 AvgScore:2}} {DefaultSerialize:{} StuAvgScore:{StuId:100 AvgScore:2}} {DefaultSerialize:{} StuAvgScore:{StuId:101 AvgScore:0}}] abort:false action:{DataReuse:false ForceEntryNext:false JumpFunc: Abort:false} cache:0xc000210b88 metaData:map[] mLock:{w:{state:0 sema:0} writerSem:0 readerSem:0 readerCount:0 readerWait:0}}

stuid: [100], avg score: [2]
stuid: [100], avg score: [2]
stuid: [101], avg score: [0]
--- PASS: TestAutoInjectParamWithConfig (0.01s)
PASS
ok      kis-flow/test   0.030s

11.4 [V1.0] Source Code

https://github.com/aceld/kis-flow/releases/tag/v1.0

Author: Aceld
GitHub: https://github.com/aceld

KisFlow Open Source Project Address: https://github.com/aceld/kis-flow

Document: https://github.com/aceld/kis-flow/wiki

Case1-Quick Start
Case2-Flow Parallel Operation
Case3-Application of KisFlow in Multi-Goroutines
Case4-KisFlow in Message Queue (MQ) Applications

(Part 10)Golang Framework Hands-on - Prometheus Metrics Statistics

Aceld — Thu, 27 Jun 2024 07:11:34 +0000

Github: https://github.com/aceld/kis-flow
Document: https://github.com/aceld/kis-flow/wiki

Case1-Quick Start
Case2-Flow Parallel Operation
Case3-Application of KisFlow in Multi-Goroutines
Case4-KisFlow in Message Queue (MQ) Applications

Before diving into this chapter, let's introduce how to start the Prometheus Metrics service. For those unfamiliar with Prometheus, it's advisable to look up additional information. In simple terms, Prometheus is a system monitoring and metrics tool.

As KisFlow is a stream computing framework, metrics such as function scheduling time, total data volume, and algorithm speed are crucial for developers and project teams. These metrics can be recorded using Prometheus Metrics through KisFlow.

Next, we will configure the framework globally, allowing developers to enable Prometheus metrics collection if needed.

10.1 Prometheus Metrics Service

10.1.1 Prometheus Client SDK

First, add the necessary dependency in the kis-flow/go.mod file:

module kis-flow

go 1.18

require (
    github.com/google/uuid v1.5.0
    github.com/patrickmn/go-cache v2.1.0+incompatible
    github.com/prometheus/client_golang v1.14.0  //++++++++
    gopkg.in/yaml.v3 v3.0.1
)

We use the official Prometheus Golang client SDK. More details can be found in the official README documentation:

Next, let's write a simple Prometheus service that allows external access to KisFlow service metrics. Create a new directory kis-flow/metrics/ for the KisFlow metrics code.

kis-flow/metrics/kis_metrics.go
package metrics

import (
"github.com/prometheus/client_golang/prometheus/promhttp"
"kis-flow/common"
"kis-flow/log"
"net/http"
)

// RunMetricsService starts the Prometheus monitoring service
func RunMetricsService(serverAddr string) error {

// Register the Prometheus monitoring route path
http.Handle(common.METRICS_ROUTE, promhttp.Handler())

// Start the HTTP server
err := http.ListenAndServe(serverAddr, nil) // Multiple processes cannot listen on the same port
if err != nil {
    log.Logger().ErrorF("RunMetricsService err = %s\n", err)
}

return err

}


Define `METRICS_ROUTE` as the monitoring service HTTP route path in `kis-flow/common/const.go`:

> kis-flow/common/const.go
```go
// ... ...

// metrics
const (
    METRICS_ROUTE string = "/metrics"
)

// ... ...

Let's briefly explain the above code. RunMetricsService() starts the Prometheus monitoring HTTP service. The purpose of this service is to provide metrics for the current KisFlow process through HTTP requests. While we haven't collected specific metrics yet, Prometheus will provide default metrics such as the current Go version, GC garbage collection time, memory allocation, etc.

serverAddr parameter: This is the address for the Prometheus monitoring service, usually a local address with a port number like "0.0.0.0:20004".

http.Handle(common.METRICS_ROUTE, promhttp.Handler())

This line of code sets "0.0.0.0:20004/metrics" as the metrics entry point.

After writing the above code, remember to pull the relevant dependency package from Prometheus Golang Client SDK, https://github.com/prometheus/client_golang .

$ go mod tidy

After pulling, the current go.mod dependencies will look something like this (with version differences):

kis-flow/go.mod
module kis-flow

go 1.18

require (
github.com/google/uuid v1.5.0
github.com/patrickmn/go-cache v2.1.0+incompatible
github.com/prometheus/client_golang v1.14.0
gopkg.in/yaml.v3 v3.0.1
)

require (
github.com/beorn7/perks v1.0.1 // indirect
github.com/cespare/xxhash/v2 v2.1.2 // indirect
github.com/golang/protobuf v1.5.2 // indirect
github.com/matttproud/golang_protobuf_extensions v1.0.1 // indirect
github.com/prometheus/client_model v0.3.0 // indirect
github.com/prometheus/common v0.37.0 // indirect
github.com/prometheus/procfs v0.8.0 // indirect
golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a // indirect
google.golang.org/protobuf v1.28.1 // indirect
)


### 10.1.2 Unit Testing for Prometheus Server Service Startup

Next, let's perform a simple test to verify if the service can start.

Create a file named `prometheus_server_test.go` in the `kis-flow/test/` directory:

> kis-flow/test/prometheus_server_test.go
```go
package test

import (
    "kis-flow/metrics"
    "testing"
)

func TestPrometheusServer(t *testing.T) {
    err := metrics.RunMetricsService("0.0.0.0:20004")
    if err != nil {
        panic(err)
    }
}

Here, the monitoring address is "0.0.0.0:20004". Next, start this unit test case by opening terminal A and navigating to the kis-flow/test/ directory:

$ cd kis-flow/test/
$ go test -test.v -test.paniconexit0 -test.run TestPrometheusServer
=== RUN   TestPrometheusServer

Then, open another terminal B and enter the following command to simulate an HTTP client request:

$ curl http://0.0.0.0:20004/metrics

After that, we should see the monitoring metrics result in terminal B as follows:

# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0
go_gc_duration_seconds{quantile="0.25"} 0
go_gc_duration_seconds{quantile="0.5"} 0
go_gc_duration_seconds{quantile="0.75"} 0
go_gc_duration_seconds{quantile="1"} 0
go_gc_duration_seconds_sum 0
go_gc_duration_seconds_count 0
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 8
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.18.8"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 3.2364e+06
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 3.2364e+06
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.446507e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 0
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 3.561224e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 3.2364e+06
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 4.636672e+06
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 3.260416e+06
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 21294
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 4.636672e+06
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 7.897088e+06
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 0
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 0
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 21294
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 9600
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 15600
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 46376
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 48960
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 4.194304e+06
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 1.171301e+06
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 491520
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 491520
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 1.46322e+07
# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 7
# HELP promhttp_metric_handler_requests_in_flight Current number of scrapes being served.
# TYPE promhttp_metric_handler_requests_in_flight gauge
promhttp_metric_handler_requests_in_flight 1
# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.
# TYPE promhttp_metric_handler_requests_total counter
promhttp_metric_handler_requests_total{code="200"} 1
promhttp_metric_handler_requests_total{code="500"} 0
promhttp_metric_handler_requests_total{code="503"} 0

We have already provided configurations for Function, Flow, and Connector in KisFlow, distinguished by kistype. Next, we will implement a global configuration with kistype set to global. In this configuration, we will add settings to enable or disable Prometheus and Metrics collection.

Let's proceed to add global configuration properties to KisFlow.

10.2 KisFlow Global Configuration

10.2.1 Loading Global Configuration Files

The global configuration in YAML format is as follows:

# kistype Global for the global configuration of KisFlow
kistype: global
# Whether to enable Prometheus monitoring
prometheus_enable: true
# Whether KisFlow needs to start a separate port listener
prometheus_listen: true
# The address for Prometheus to listen for metrics
prometheus_serve: 0.0.0.0:20004

10.2.2 Struct Definition

Next, based on the configuration protocol above, we'll define the strategy configuration struct for KisFlow and provide some initialization methods. Create a file named kis_global_config.go in the project documentation. Here, we'll define the necessary configuration.

kis-flow/config/kis_global_config.go
package config

type KisGlobalConfig struct {
// kistype Global for the global configuration of KisFlow
KisType string yaml:"kistype"
// Whether to enable Prometheus monitoring
EnableProm bool yaml:"prometheus_enable"
// Whether KisFlow needs to start a separate port listener
PrometheusListen bool yaml:"prometheus_listen"
// The address for Prometheus to listen for metrics
PrometheusServe string yaml:"prometheus_serve"
}

// GlobalConfig is the default global configuration, all are turned off
var GlobalConfig = new(KisGlobalConfig)


Here, we provide a global `GlobalConfig` object, which is a public variable, making it convenient for other modules to share the global configuration.

### 10.2.3 Configuration File Parsing

Next, we'll parse the global configuration and import it. Add the following function in `kis-flow/file/config_import.go`:

> kis-flow/file/config_import.go
```go
// kisTypeGlobalConfigure parses the Global configuration file in YAML format
func kisTypeGlobalConfigure(confData []byte, fileName string, kisType interface{}) error {
    // Global configuration
    if err := yaml.Unmarshal(confData, config.GlobalConfig); err != nil {
        return fmt.Errorf("%s has wrong format kisType = %s", fileName, kisType)
    }

    // TODO Initialize Prometheus metrics

    // TODO Start Prometheus metrics service

    return nil
}

This function loads the global YAML configuration file. After loading, it determines whether to initialize Prometheus metrics monitoring, which we will add later.

Where is kisTypeGlobalConfigure() called? It is invoked during the loading and scanning of local configuration files, similar to other configuration files:

kis-flow/file/config_import.go

// parseConfigWalkYaml parses all configuration files in YAML format and loads the configuration information into allConfig
func parseConfigWalkYaml(loadPath string) (*allConfig, error) {
    // ... ...

err := filepath.Walk(loadPath, func(filePath string, info os.FileInfo, err error) error {
    // ... ... 

    // Check if kistype exists
    if kisType, ok := confMap["kistype"]; !ok {
        return fmt.Errorf("yaml file %s has no field [kistype]!", filePath)
    } else {
        switch kisType {
        case common.KisIdTypeFlow:
            return kisTypeFlowConfigure(all, confData, filePath, kisType)

        case common.KisIdTypeFunction:
            return kisTypeFuncConfigure(all, confData, filePath, kisType)

        case common.KisIdTypeConnector:
            return kisTypeConnConfigure(all, confData, filePath, kisType)

        // +++++++++++++++++++++++++++++++++
        case common.KisIdTypeGlobal:
            return kisTypeGlobalConfigure(confData, filePath, kisType)
        // +++++++++++++++++++++++++++++++++

        default:
            return fmt.Errorf("%s sets wrong kistype %s", filePath, kisType)
        }
    }
})

if err != nil {
    return nil, err
}

return all, nil

}


Here, we add a case for kistype: `KisIdTypeGlobal` to call `kisTypeGlobalConfigure()`.

Next, we will create the Metrics module. In this section, we will start by tracking a simple metric: the total amount of data processed by KisFlow (based on the number of source data processed).

## 10.3 Metrics - DataTotal Metric

### 10.3.1 KisMetrics

First, create a KisMetrics module by creating the directory `kis-flow/metrics/` and the file `kis_metrics.go`:

> kis-flow/metrics/kis_metrics.go
```go
package metrics

import (
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promhttp"
    "kis-flow/common"
    "kis-flow/log"
    "net/http"
)

// kisMetrics defines the Prometheus metrics for KisFlow
type kisMetrics struct {
    // Total data count
    DataTotal prometheus.Counter
}

var Metrics *kisMetrics

// RunMetricsService starts the Prometheus monitoring service
func RunMetricsService(serverAddr string) error {
    // Register Prometheus monitoring route
    http.Handle(common.METRICS_ROUTE, promhttp.Handler())

    // Start HTTP server
    err := http.ListenAndServe(serverAddr, nil) // Multiple processes cannot listen on the same port
    if err != nil {
        log.Logger().ErrorF("RunMetricsService err = %s\n", err)
    }

    return err
}

// InitMetrics initializes the metrics
func InitMetrics() {
    Metrics = new(kisMetrics)

    // Initialize the DataTotal counter
    Metrics.DataTotal = prometheus.NewCounter(prometheus.CounterOpts{
        Name: common.COUNTER_KISFLOW_DATA_TOTAL_NAME,
        Help: common.COUNTER_KISFLOW_DATA_TOTAL_HELP,
    })

    // Register the metrics
    prometheus.MustRegister(Metrics.DataTotal)
}

kisMetrics struct: This struct defines the metrics that KisFlow needs to track. Currently, it only includes one metric, DataTotal, which is of type prometheus.Counter (for more information on the prometheus.Counter type, please refer to the Prometheus documentation).
Metrics *kisMetrics: This is a global metrics tracking object for KisFlow, making it publicly accessible for other modules.
RunMetricsService(serverAddr string): This function starts the Prometheus service listener, which was already unit tested in previous chapters.
InitMetrics(): This function initializes the global object and sets up the metrics. It calls prometheus.MustRegister to register the metrics with Prometheus, which is a necessary step in Prometheus metrics programming.

There are two constants representing the metric display name and description. These are defined as follows:

kis-flow/common/const.go

// metrics
const (
    METRICS_ROUTE string = "/metrics"

COUNTER_KISFLOW_DATA_TOTAL_NAME string = "kisflow_data_total"
COUNTER_KISFLOW_DATA_TOTAL_HELP string = "Total data processed by all Flows in KisFlow"

)


### 10.3.2 DataTotal Metric Tracking

To track the total data processed by KisFlow, we need to add the metrics tracking code in the `commitSrcData()` method. This method submits the current Flow's source data, indicating the first time the original source data is submitted to the current Flow. The updated code is as follows:

> kis-flow/flow/kis_flow_data.go
```go
func (flow *KisFlow) commitSrcData(ctx context.Context) error {
    // Create a batch of data
    dataCnt := len(flow.buffer)
    batch := make(common.KisRowArr, 0, dataCnt)

    for _, row := range flow.buffer {
        batch = append(batch, row)
    }

    // Clear previous data
    flow.clearData(flow.data)

    // Record the original data for the first submission
    // Since this is the first submission, PrevFunctionId is FirstVirtual because there is no previous Function
    flow.data[common.FunctionIdFirstVirtual] = batch

    // Clear the buffer
    flow.buffer = flow.buffer[0:0]

    // +++++++++++++++++++++++++++++++
    // Track the total data count on the first submission
    if config.GlobalConfig.EnableProm == true {
        // Increment the DataTotal metric by the data count
        metrics.Metrics.DataTotal.Add(float64(dataCnt))
    }
    // ++++++++++++++++++++++++++++++

    log.Logger().DebugFX(ctx, "====> After CommitSrcData, flow_name = %s, flow_id = %s\nAll Level Data =\n %+v\n", flow.Name, flow.Id, flow.data)

    return nil
}

First, it checks the global configuration to determine if metrics tracking is enabled. If true, it increments the total data count metric with the following code:

metrics.Metrics.DataTotal.Add(float64(dataCnt))

Here, dataCnt is the number of data items being added to the total count.

10.3.3 Starting the Metrics Service

After importing the configuration, we need to start the metrics service. The configuration file is updated as follows:

kis-flow/file/config_import.go

// kisTypeGlobalConfigure parses the Global configuration file in YAML format
func kisTypeGlobalConfigure(confData []byte, fileName string, kisType interface{}) error {
    // Global configuration
    if ok := yaml.Unmarshal(confData, config.GlobalConfig); ok != nil {
        return errors.New(fmt.Sprintf("%s is wrong format kisType = %s", fileName, kisType))
    }

// ++++++++++++++++++++
// Start the Metrics service
metrics.RunMetrics()

return nil

}


The `RunMetrics()` function is implemented as follows:
> kis-flow/metrics/kis_metrics.go
```go
// RunMetrics starts the Prometheus metrics service
func RunMetrics() {
    // Initialize Prometheus metrics
    InitMetrics()

    if config.GlobalConfig.EnableProm == true && config.GlobalConfig.PrometheusListen == true {
        // Start the Prometheus metrics service
        go RunMetricsService(config.GlobalConfig.PrometheusServe)
    }
}

With this setup, after importing the global configuration, it checks whether metrics tracking is enabled. If it is, a new goroutine is started to launch the Prometheus server, listening on the IP and port specified in the configuration file.

Next, we will create a unit test for the DataTotal metric to validate our implementation.

10.4 KisMetrics Unit Testing

10.4.1 Create Global Configuration File

Create a global configuration file kis-flow.yml under kis-flow/test/load_conf/ with the following content:

kis-flow/test/load_conf/kis-flow.yml

# kistype Global for kisflow global configuration
kistype: global
# Enable prometheus monitoring
prometheus_enable: true
# Enable separate kisflow port listening
prometheus_listen: true
# Prometheus endpoint listening address
prometheus_serve: 0.0.0.0:20004

10.4.2 Create Unit Test

Next, create the test case code in kis-flow/test/, and create the kis_metrics_test.go file as follows:

kis-flow/test/kis_metrics_test.go
package test

import (
"context"
"kis-flow/common"
"kis-flow/file"
"kis-flow/kis"
"kis-flow/test/caas"
"kis-flow/test/faas"
"testing"
"time"
)

func TestMetricsDataTotal(t *testing.T) {
ctx := context.Background()

// 0. Register Function callback business logic
kis.Pool().FaaS("funcName1", faas.FuncDemo1Handler)
kis.Pool().FaaS("funcName2", faas.FuncDemo2Handler)
kis.Pool().FaaS("funcName3", faas.FuncDemo3Handler)

// 0. Register ConnectorInit and Connector callback business logic
kis.Pool().CaaSInit("ConnName1", caas.InitConnDemo1)
kis.Pool().CaaS("ConnName1", "funcName2", common.S, caas.CaasDemoHanler1)

// 1. Load configuration file and build Flow
if err := file.ConfigImportYaml("/Users/tal/gopath/src/kis-flow/test/load_conf/"); err != nil {
    panic(err)
}

// 2. Get Flow
flow1 := kis.Pool().GetFlow("flowName1")

n := 0

for n < 10 {
    // 3. Submit raw data
    _ = flow1.CommitRow("This is Data1 from Test")

    // 4. Execute flow1
    if err := flow1.Run(ctx); err != nil {
        panic(err)
    }

    time.Sleep(1 * time.Second)
    n++
}

select {}

}


This case works similarly to starting KisFlow usually, except that it includes a loop that starts a stream computation every second and submits a piece of data, looping 10 times. Afterward, we can check the total data amount through the Prometheus monitoring service. The select{} statement prevents the main goroutine from exiting, ensuring that the Prometheus monitoring goroutine continues running.

Run the unit test by navigating to `kis-flow/test/` and executing:

```bash
go test -test.v -test.paniconexit0 -test.run TestMetricsDataTotal

You will see a lot of log output. Wait for 10 seconds, then open another terminal and enter the following command:

$ curl http://0.0.0.0:20004/metrics

The result will be:

# ... ...
# HELP kisflow_data_total KisFlow total data amount of all Flows
# TYPE kisflow_data_total counter
kisflow_data_total 10
# ... ...

Here, you'll find that the kisflow_data_total metric appears with a result of 10, indicating that our metrics are correctly tracked. Next, we can add more complex metrics for KisFlow.

10.5 Additional Metrics

10.5.1 Metric: Flow Data Total

(1) Define Metric

First, define the metric type as follows:

kis-flow/metrics/kis_metrics.go

// kisMetrics Prometheus monitoring metrics for kisFlow
type kisMetrics struct {
    // Total data amount
    DataTotal prometheus.Counter
    // Data total per Flow
    FlowDataTotal *prometheus.GaugeVec
}

FlowDataTotal uses the prometheus.GaugeVec type to distinguish which Flow generates the data.

(2) Initialize and Register Metrics

kis-flow/metrics/kis_metrics.go

func InitMetrics() {
    Metrics = new(kisMetrics)

// Initialize DataTotal Counter
Metrics.DataTotal = prometheus.NewCounter(prometheus.CounterOpts{
    Name: common.COUNTER_KISFLOW_DATA_TOTAL_NAME,
    Help: common.COUNTER_KISFLOW_DATA_TOTAL_HELP,
})

// +++++++++++
// Initialize FlowDataTotal GaugeVec
Metrics.FlowDataTotal = prometheus.NewGaugeVec(
    prometheus.GaugeOpts{
        Name: common.GANGE_FLOW_DATA_TOTAL_NAME,
        Help: common.GANGE_FLOW_DATA_TOTAL_HELP,
    },
    // Label names
    []string{common.LABEL_FLOW_NAME},
)

// Register Metrics
prometheus.MustRegister(Metrics.DataTotal)
prometheus.MustRegister(Metrics.FlowDataTotal) // +++

}


Related constant definitions:

> kis-flow/common/const.go
```go

// metrics
const (
    METRICS_ROUTE string = "/metrics"

    // ++++++++
    LABEL_FLOW_NAME     string = "flow_name"
    LABEL_FLOW_ID       string = "flow_id"
    LABEL_FUNCTION_NAME string = "func_name"
    LABEL_FUNCTION_MODE string = "func_mode"

    COUNTER_KISFLOW_DATA_TOTAL_NAME string = "kisflow_data_total"
    COUNTER_KISFLOW_DATA_TOTAL_HELP string = "KisFlow total data amount of all Flows"

    // +++++++ 
    GANGE_FLOW_DATA_TOTAL_NAME string = "flow_data_total"
    GANGE_FLOW_DATA_TOTAL_HELP string = "KisFlow data total amount per Flow"
)

(3) Add Metric Tracking

We should track the total data amount when submitting raw data.

kis-flow/flow/kis_flow_data.go

func (flow *KisFlow) commitSrcData(ctx context.Context) error {

// Create batch data
dataCnt := len(flow.buffer)
batch := make(common.KisRowArr, 0, dataCnt)

for _, row := range flow.buffer {
    batch = append(batch, row)
}

// Clear previous data
flow.clearData(flow.data)

// First submission, record flow raw data
flow.data[common.FunctionIdFirstVirtual] = batch

// Clear buffer
flow.buffer = flow.buffer[0:0]

// First submission, track data total
if config.GlobalConfig.EnableProm == true {
    // Track data total Metrics.DataTotal
    metrics.Metrics.DataTotal.Add(float64(dataCnt))

    // ++++++++
    // Track current Flow data total
    metrics.Metrics.FlowDataTotal.WithLabelValues(flow.Name).Add(float64(dataCnt))
}

log.Logger().DebugFX(ctx, "====> After CommitSrcData, flow_name = %s, flow_id = %s\nAll Level Data =\n %+v\n", flow.Name, flow.Id, flow.data)

return nil

}


So the tracking point is in the same position as before, but we add a `flow.Name` label when accumulating the data.

### 10.5.2 Metric: Flow Scheduling Count

#### (1) Metric Definition

First, define the metric type as follows:

> kis-flow/metrics/kis_metrics.go

```go
// kisMetrics represents the Prometheus monitoring metrics for kisFlow
type kisMetrics struct {
    // Total data count
    DataTotal prometheus.Counter
    // Total data processed by each Flow
    FlowDataTotal *prometheus.GaugeVec
    // Total Flow scheduling count
    FlowScheduleCntsTotal *prometheus.GaugeVec //++++
}

FlowScheduleCntsTotal uses prometheus.GaugeVec type to distinguish which Flow produced the data.

(2) Metric Initialization and Registration

kis-flow/metrics/kis_metrics.go

func InitMetrics() {
    Metrics = new(kisMetrics)

    // Initialize DataTotal as a Counter
    Metrics.DataTotal = prometheus.NewCounter(prometheus.CounterOpts{
        Name: common.COUNTER_KISFLOW_DATA_TOTAL_NAME,
        Help: common.COUNTER_KISFLOW_DATA_TOTAL_HELP,
    })

    // Initialize FlowDataTotal as a GaugeVec
    Metrics.FlowDataTotal = prometheus.NewGaugeVec(
        prometheus.GaugeOpts{
            Name: common.GANGE_FLOW_DATA_TOTAL_NAME,
            Help: common.GANGE_FLOW_DATA_TOTAL_HELP,
        },
        // Label name
        []string{common.LABEL_FLOW_NAME},
    )

    // +++++++++++++
    // Initialize FlowScheduleCntsTotal as a GaugeVec
    Metrics.FlowScheduleCntsTotal = prometheus.NewGaugeVec(
        prometheus.GaugeOpts{
            Name: common.GANGE_FLOW_SCHE_CNTS_NAME,
            Help: common.GANGE_FLOW_SCHE_CNTS_HELP,
        },
        // Label name
        []string{common.LABEL_FLOW_NAME},
    )

    // Register metrics
    prometheus.MustRegister(Metrics.DataTotal)
    prometheus.MustRegister(Metrics.FlowDataTotal) 
    // +++++
    prometheus.MustRegister(Metrics.FlowScheduleCntsTotal)
}

Define the relevant constants:

kis-flow/common/const.go

// metrics
const (
METRICS_ROUTE string = "/metrics"

LABEL_FLOW_NAME     string = "flow_name"
LABEL_FLOW_ID       string = "flow_id"
LABEL_FUNCTION_NAME string = "func_name"
LABEL_FUNCTION_MODE string = "func_mode"

COUNTER_KISFLOW_DATA_TOTAL_NAME string = "kisflow_data_total"
COUNTER_KISFLOW_DATA_TOTAL_HELP string = "KisFlow total data count"

GANGE_FLOW_DATA_TOTAL_NAME string = "flow_data_total"
GANGE_FLOW_DATA_TOTAL_HELP string = "Total data count for each FlowID in KisFlow"

// +++++++
GANGE_FLOW_SCHE_CNTS_NAME string = "flow_schedule_cnts"
GANGE_FLOW_SCHE_CNTS_HELP string = "Scheduling count for each FlowID in KisFlow"

)


#### (3) Metric Data Collection

To collect the scheduling count for each Flow, we should collect data in the main entry point `flow.Run()`, as follows:

> kis-flow/flow/kis_flow.go
```go
// Run starts the stream computation of KisFlow and executes the flow from the initial Function
func (flow *KisFlow) Run(ctx context.Context) error {

    var fn kis.Function

    fn = flow.FlowHead
    flow.abort = false

    if flow.Conf.Status == int(common.FlowDisable) {
        // Flow is disabled in configuration
        return nil
    }

    // Since no Function has been executed yet, PrevFunctionId is set to FirstVirtual because there is no previous Function
    flow.PrevFunctionId = common.FunctionIdFirstVirtual

    // Commit the original stream data
    if err := flow.commitSrcData(ctx); err != nil {
        return err
    }

    // +++++++++++ Metrics
    if config.GlobalConfig.EnableProm == true {
        // Collect scheduling count for Flow
        metrics.Metrics.FlowScheduleCntsTotal.WithLabelValues(flow.Name).Inc()
    }
    // ++++++++++++++++++++

    // Chain-style stream invocation
    for fn != nil && flow.abort == false {

        // Record the current Function being executed in the Flow
        fid := fn.GetId()
        flow.ThisFunction = fn
        flow.ThisFunctionId = fid

        // Obtain the source data for the current Function to process
        if inputData, err := flow.getCurData(); err != nil {
            log.Logger().ErrorFX(ctx, "flow.Run(): getCurData err = %s\n", err.Error())
            return err
        } else {
            flow.inPut = inputData
        }

        if err := fn.Call(ctx, flow); err != nil {
            // Error
            return err
        } else {
            // Success
            fn, err = flow.dealAction(ctx, fn)
            if err != nil {
                return err
            }

        }
    }

    return nil
}

Here, the metric collection occurs before calling the fn.Call() method, and we increment the counter for each Flow execution, grouping by flow.Name.

10.5.3 Metric: Function Scheduling Count

(1) Metric Definition

First, define the metric type as follows:

kis-flow/metrics/kis_metrics.go

// kisMetrics represents the Prometheus monitoring metrics for kisFlow
type kisMetrics struct {
    // Total data count
    DataTotal prometheus.Counter
    // Total data processed by each Flow
    FlowDataTotal *prometheus.GaugeVec
    // Total Flow scheduling count
    FlowScheduleCntsTotal *prometheus.GaugeVec 
    // Total Function scheduling count
    FuncScheduleCntsTotal *prometheus.GaugeVec //++++
}

FuncScheduleCntsTotal uses prometheus.GaugeVec type to distinguish which Function produced the data.

(2) Metric Initialization and Registration

kis-flow/metrics/kis_metrics.go

func InitMetrics() {
    Metrics = new(kisMetrics)

// Initialize DataTotal as a Counter
Metrics.DataTotal = prometheus.NewCounter(prometheus.CounterOpts{
    Name: common.COUNTER_KISFLOW_DATA_TOTAL_NAME,
    Help: common.COUNTER_KISFLOW_DATA_TOTAL_HELP,
})

// Initialize FlowDataTotal as a GaugeVec
Metrics.FlowDataTotal = prometheus.NewGaugeVec(
    prometheus.GaugeOpts{
        Name: common.GANGE_FLOW_DATA_TOTAL_NAME,
        Help: common.GANGE_FLOW_DATA_TOTAL_HELP,
    },
    // Label name
    []string{common.LABEL_FLOW_NAME},
)

// Initialize FlowScheduleCntsTotal as a GaugeVec
Metrics.FlowScheduleCntsTotal = prometheus.NewGaugeVec(
    prometheus.GaugeOpts{
        Name: common.GANGE_FLOW_SCHE_CNTS_NAME,
        Help: common.GANGE_FLOW_SCHE_CNTS_HELP,
    },
    // Label name
    []string{common.LABEL_FLOW_NAME},
)

// ++++++++++
// Initialize FuncScheduleCntsTotal as a GaugeVec
Metrics.FuncScheduleCntsTotal = prometheus.NewGaugeVec(
    prometheus.GaugeOpts{
        Name: common.GANGE_FUNC_SCHE_CNTS_NAME,
        Help: common.GANGE_FUNC_SCHE_CNTS_HELP,
    },
    // Label names
    []string{common.LABEL_FUNCTION_NAME, common.LABEL_FUNCTION_MODE},
)

// Register metrics
prometheus.MustRegister(Metrics.DataTotal)
prometheus.MustRegister(Metrics.FlowDataTotal) 
prometheus.MustRegister(Metrics.FlowScheduleCntsTotal)
// +++++++
prometheus.MustRegister(Metrics.FuncScheduleCntsTotal)

}


Define the relevant constants:

> kis-flow/common/const.go
```go

// metrics
const (
    METRICS_ROUTE string = "/metrics"

    LABEL_FLOW_NAME     string = "flow_name"
    LABEL_FLOW_ID       string = "flow_id"
    LABEL_FUNCTION_NAME string = "func_name"
    LABEL_FUNCTION_MODE string = "func_mode"

    COUNTER_KISFLOW_DATA_TOTAL_NAME string = "kisflow_data_total"
    COUNTER_KISFLOW_DATA_TOTAL_HELP string = "KisFlow total data count"

    GANGE_FLOW_DATA_TOTAL_NAME string = "flow_data_total"
    GANGE_FLOW_DATA_TOTAL_HELP string = "Total data count for each FlowID in KisFlow"

    GANGE_FLOW_SCHE_CNTS_NAME string = "flow_schedule_cnts"
    GANGE_FLOW_SCHE_CNTS_HELP string = "Scheduling count for each FlowID in KisFlow"

    // +++++++++ 
    GANGE_FUNC_SCHE_CNTS_NAME string = "func_schedule_cnts"
    GANGE_FUNC_SCHE_CNTS_HELP string = "Scheduling count for each Function in KisFlow"
)

(3) Metric Data Collection

To collect the scheduling count for each Function, we should collect data in the main entry point flow.Run(), as follows:

kis-flow/flow/kis_flow.go

// Run starts the stream computation of KisFlow and executes the flow from the initial Function
func (flow *KisFlow) Run(ctx context.Context) error {

var fn kis.Function

fn = flow.FlowHead
flow.abort = false

if flow.Conf.Status == int(common.FlowDisable) {
    // Flow is disabled in configuration
    return nil
}

// Since no Function has been executed yet, PrevFunctionId is set to FirstVirtual because there is no previous Function
flow.PrevFunctionId = common.FunctionIdFirstVirtual

// Commit the original stream data
if err := flow.commitSrcData(ctx); err != nil {
    return err
}

if config.GlobalConfig.EnableProm == true {
    // Collect scheduling count for Flow
    metrics.Metrics.FlowScheduleCntsTotal.WithLabelValues(flow.Name).Inc()
}

// Chain-style stream invocation
for fn != nil && flow.abort == false {

    // Record the current Function being executed in the Flow
    fid := fn.GetId()
    flow.ThisFunction = fn
    flow.ThisFunctionId = fid

    // ++++++++++++
    fName := fn.GetConfig().FName
    fMode := fn.GetConfig().FMode

    // +++++++++++++++++++++++++++
    if config.GlobalConfig.EnableProm == true {
        // Collect scheduling count for Function
        metrics.Metrics.FuncScheduleCntsTotal.WithLabelValues(fName, fMode).Inc()
    }
    // ++++++++++++++++++++++++++++

    // Obtain the source data for the current Function to process
    if inputData, err := flow.getCurData(); err != nil {
        log.Logger().ErrorFX(ctx, "flow.Run(): getCurData err = %s\n", err.Error())
        return err
    } else {
        flow.inPut = inputData
    }

    if err := fn.Call(ctx, flow); err != nil {
        // Error
        return err
    } else {
        // Success
        fn, err = flow.dealAction(ctx, fn)
        if err != nil {
            return err
        }
    }
}

return nil

}


Here, the metric collection occurs before calling the `fn.Call()` method. Each time a Function is scheduled, the counter is incremented, grouped by `fName` and `fMode`.

### 10.5.4 Metric: Function Execution Time

#### (1) Metric Definition

Define the metric type as follows:

> kis-flow/metrics/kis_metrics.go
```go
// kisMetrics defines the Prometheus metrics for kisFlow
type kisMetrics struct {
    // Total data count
    DataTotal prometheus.Counter
    // Total data processed by each Flow
    FlowDataTotal *prometheus.GaugeVec
    // Flow schedule count
    FlowScheduleCntsTotal *prometheus.GaugeVec 
    // Function schedule count
    FuncScheduleCntsTotal *prometheus.GaugeVec 
    // Function execution time
    FunctionDuration *prometheus.HistogramVec //++++
}

FunctionDuration uses the prometheus.HistogramVec type. This type provides distribution statistics across different time intervals, with various buckets representing different time ranges.

(2) Metric Initialization and Registration

kis-flow/metrics/kis_metrics.go

func InitMetrics() {
    Metrics = new(kisMetrics)

// Initialize DataTotal Counter
Metrics.DataTotal = prometheus.NewCounter(prometheus.CounterOpts{
    Name: common.COUNTER_KISFLOW_DATA_TOTAL_NAME,
    Help: common.COUNTER_KISFLOW_DATA_TOTAL_HELP,
})

// Initialize FlowDataTotal GaugeVec
Metrics.FlowDataTotal = prometheus.NewGaugeVec(
    prometheus.GaugeOpts{
        Name: common.GAUGE_FLOW_DATA_TOTAL_NAME,
        Help: common.GAUGE_FLOW_DATA_TOTAL_HELP,
    },
    // Label name
    []string{common.LABEL_FLOW_NAME},
)

// Initialize FlowScheduleCntsTotal GaugeVec
Metrics.FlowScheduleCntsTotal = prometheus.NewGaugeVec(
    prometheus.GaugeOpts{
        Name: common.GAUGE_FLOW_SCHE_CNTS_NAME,
        Help: common.GAUGE_FLOW_SCHE_CNTS_HELP,
    },
    // Label name
    []string{common.LABEL_FLOW_NAME},
)

// Initialize FuncScheduleCntsTotal GaugeVec
Metrics.FuncScheduleCntsTotal = prometheus.NewGaugeVec(
    prometheus.GaugeOpts{
        Name: common.GAUGE_FUNC_SCHE_CNTS_NAME,
        Help: common.GAUGE_FUNC_SCHE_CNTS_HELP,
    },
    // Label names
    []string{common.LABEL_FUNCTION_NAME, common.LABEL_FUNCTION_MODE},
)

// ++++++++++++++++++++++++++
// Initialize FunctionDuration HistogramVec
Metrics.FunctionDuration = prometheus.NewHistogramVec(prometheus.HistogramOpts{
    Name:    common.HISTOGRAM_FUNCTION_DURATION_NAME,
    Help:    common.HISTOGRAM_FUNCTION_DURATION_HELP,
    Buckets: []float64{0.005, 0.01, 0.03, 0.08, 0.1, 0.5, 1.0, 5.0, 10, 100, 1000, 5000, 30000}, // unit ms, max half a minute
},
    []string{common.LABEL_FUNCTION_NAME, common.LABEL_FUNCTION_MODE},
)

// Register Metrics
prometheus.MustRegister(Metrics.DataTotal)
prometheus.MustRegister(Metrics.FlowDataTotal) 
prometheus.MustRegister(Metrics.FlowScheduleCntsTotal)
prometheus.MustRegister(Metrics.FuncScheduleCntsTotal)
// +++++++
prometheus.MustRegister(Metrics.FunctionDuration)

}


Related constant definitions:
> kis-flow/common/const.go
```go

// metrics
const (
    METRICS_ROUTE string = "/metrics"

    LABEL_FLOW_NAME     string = "flow_name"
    LABEL_FLOW_ID       string = "flow_id"
    LABEL_FUNCTION_NAME string = "func_name"
    LABEL_FUNCTION_MODE string = "func_mode"

    COUNTER_KISFLOW_DATA_TOTAL_NAME string = "kisflow_data_total"
    COUNTER_KISFLOW_DATA_TOTAL_HELP string = "Total data processed by KisFlow"

    GAUGE_FLOW_DATA_TOTAL_NAME string = "flow_data_total"
    GAUGE_FLOW_DATA_TOTAL_HELP string = "Total data processed by each FlowID in KisFlow"

    GAUGE_FLOW_SCHE_CNTS_NAME string = "flow_schedule_cnts"
    GAUGE_FLOW_SCHE_CNTS_HELP string = "Flow schedule counts for each FlowID in KisFlow"

    GAUGE_FUNC_SCHE_CNTS_NAME string = "func_schedule_cnts"
    GAUGE_FUNC_SCHE_CNTS_HELP string = "Function schedule counts for each Function in KisFlow"

    HISTOGRAM_FUNCTION_DURATION_NAME string = "func_run_duration"
    HISTOGRAM_FUNCTION_DURATION_HELP string = "Function execution duration"
)

(3) Metric Instrumentation

To measure the execution time of each Function, we should instrument the main entry point of the Flow, flow.Run(), as follows:

kis-flow/flow/kis_flow.go

// Run starts the stream processing of KisFlow, executing from the starting Function
func (flow *KisFlow) Run(ctx context.Context) error {

var fn kis.Function

fn = flow.FlowHead
flow.abort = false

if flow.Conf.Status == int(common.FlowDisable) {
    // Flow is configured to be disabled
    return nil
}

// ++++++++++ Metrics +++++++++
var funcStart time.Time

// Since no Function has been executed yet, set PrevFunctionId to FirstVirtual as there is no previous Function
flow.PrevFunctionId = common.FunctionIdFirstVirtual

// Commit the original stream data
if err := flow.commitSrcData(ctx); err != nil {
    return err
}

if config.GlobalConfig.EnableProm == true {
    // Record Flow schedule count
    metrics.Metrics.FlowScheduleCntsTotal.WithLabelValues(flow.Name).Inc()
}

// Chain-style stream invocation
for fn != nil && flow.abort == false {

    // Record the current Function being executed in the Flow
    fid := fn.GetId()
    flow.ThisFunction = fn
    flow.ThisFunctionId = fid

    fName := fn.GetConfig().FName
    fMode := fn.GetConfig().FMode

    if config.GlobalConfig.EnableProm == true {
        // Record Function schedule count
        metrics.Metrics.FuncScheduleCntsTotal.WithLabelValues(fName, fMode).Inc()

        // +++++++++++++++
        // Record Function execution time start
        funcStart = time.Now()
    }

    // Obtain the source data for the current Function to process
    if inputData, err := flow.getCurData(); err != nil {
        log.Logger().ErrorFX(ctx, "flow.Run(): getCurData err = %s\n", err.Error())
        return err
    } else {
        flow.inPut = inputData
    }

    if err := fn.Call(ctx, flow); err != nil {
        // Error
        return err
    } else {
        // Success
        fn, err = flow.dealAction(ctx, fn)
        if err != nil {
            return err
        }

        // +++++++++++++++
        // Record Function execution duration
        if config.GlobalConfig.EnableProm == true {
            // Function execution duration
            duration := time.Since(funcStart)

            // Record the current Function execution time metric
            metrics.Metrics.FunctionDuration.With(
                prometheus.Labels{
                    common.LABEL_FUNCTION_NAME: fName,
                    common.LABEL_FUNCTION_MODE: fMode}).Observe(duration.Seconds() * 1000)
        }
        // +++++++++++++++

    }
}

return nil

}


The instrumentation captures the start time before invoking the `Call()` method of the Function and calculates the execution duration after the Function completes. This duration is then recorded in the corresponding bucket of the `HistogramVec` for the Function execution time.

### 10.5.5 Metric: Flow Execution Time

#### (1) Metric Definition

Define the metric type as follows:

> kis-flow/metrics/kis_metrics.go
```go
// kisMetrics defines the Prometheus metrics for kisFlow
type kisMetrics struct {
    // Total data count
    DataTotal prometheus.Counter
    // Total data processed by each Flow
    FlowDataTotal *prometheus.GaugeVec
    // Flow schedule count
    FlowScheduleCntsTotal *prometheus.GaugeVec 
    // Function schedule count
    FuncScheduleCntsTotal *prometheus.GaugeVec 
    // Function execution time
    FunctionDuration *prometheus.HistogramVec
    // Flow execution time
    FlowDuration *prometheus.HistogramVec // ++++
}

FlowDuration uses the prometheus.HistogramVec type. This type provides distribution statistics across different time intervals, with various buckets representing different time ranges.

(2) Metric Initialization and Registration

kis-flow/metrics/kis_metrics.go

func InitMetrics() {
    Metrics = new(kisMetrics)

// Initialize DataTotal Counter
Metrics.DataTotal = prometheus.NewCounter(prometheus.CounterOpts{
    Name: common.COUNTER_KISFLOW_DATA_TOTAL_NAME,
    Help: common.COUNTER_KISFLOW_DATA_TOTAL_HELP,
})

// Initialize FlowDataTotal GaugeVec
Metrics.FlowDataTotal = prometheus.NewGaugeVec(
    prometheus.GaugeOpts{
        Name: common.GAUGE_FLOW_DATA_TOTAL_NAME,
        Help: common.GAUGE_FLOW_DATA_TOTAL_HELP,
    },
    // Label name
    []string{common.LABEL_FLOW_NAME},
)

// Initialize FlowScheduleCntsTotal GaugeVec
Metrics.FlowScheduleCntsTotal = prometheus.NewGaugeVec(
    prometheus.GaugeOpts{
        Name: common.GAUGE_FLOW_SCHE_CNTS_NAME,
        Help: common.GAUGE_FLOW_SCHE_CNTS_HELP,
    },
    // Label name
    []string{common.LABEL_FLOW_NAME},
)

// Initialize FuncScheduleCntsTotal GaugeVec
Metrics.FuncScheduleCntsTotal = prometheus.NewGaugeVec(
    prometheus.GaugeOpts{
        Name: common.GAUGE_FUNC_SCHE_CNTS_NAME,
        Help: common.GAUGE_FUNC_SCHE_CNTS_HELP,
    },
    // Label names
    []string{common.LABEL_FUNCTION_NAME, common.LABEL_FUNCTION_MODE},
)

// Initialize FunctionDuration HistogramVec
Metrics.FunctionDuration = prometheus.NewHistogramVec(prometheus.HistogramOpts{
    Name:    common.HISTOGRAM_FUNCTION_DURATION_NAME,
    Help:    common.HISTOGRAM_FUNCTION_DURATION_HELP,
    Buckets: []float64{0.005, 0.01, 0.03, 0.08, 0.1, 0.5, 1.0, 5.0, 10, 100, 1000, 5000, 30000}, // unit ms, max half a minute
},
    []string{common.LABEL_FUNCTION_NAME, common.LABEL_FUNCTION_MODE},
)

// ++++++++++++++++++++++++++
// Initialize FlowDuration HistogramVec
Metrics.FlowDuration = prometheus.NewHistogramVec(prometheus.HistogramOpts{
    Name:    common.HISTOGRAM_FLOW_DURATION_NAME,
    Help:    common.HISTOGRAM_FLOW_DURATION_HELP,
    Buckets: []float64{0.005, 0.01, 0.03, 0.08, 0.1, 0.5, 1.0, 5.0, 10, 100, 1000, 5000, 30000}, // unit ms, max half a minute
},
    []string{common.LABEL_FLOW_NAME},
)

// Register Metrics
prometheus.MustRegister(Metrics.DataTotal)
prometheus.MustRegister(Metrics.FlowDataTotal) 
prometheus.MustRegister(Metrics.FlowScheduleCntsTotal)
prometheus.MustRegister(Metrics.FuncScheduleCntsTotal)
prometheus.MustRegister(Metrics.FunctionDuration)
// +++++++
prometheus.MustRegister(Metrics.FlowDuration)

}


Related constant definitions:

> kis-flow/common/const.go
```go

// metrics
const (
    METRICS_ROUTE string = "/metrics"

    LABEL_FLOW_NAME     string = "flow_name"
    LABEL_FLOW_ID       string = "flow_id"
    LABEL_FUNCTION_NAME string = "func_name"
    LABEL_FUNCTION_MODE string = "func_mode"

    COUNTER_KISFLOW_DATA_TOTAL_NAME string = "kisflow_data_total"
    COUNTER_KISFLOW_DATA_TOTAL_HELP string = "Total data processed by KisFlow"

    GAUGE_FLOW_DATA_TOTAL_NAME string = "flow_data_total"
    GAUGE_FLOW_DATA_TOTAL_HELP string = "Total data processed by each FlowID in KisFlow"

    GAUGE_FLOW_SCHE_CNTS_NAME string = "flow_schedule_cnts"
    GAUGE_FLOW_SCHE_CNTS_HELP string = "Flow schedule counts for each FlowID in KisFlow"

    GAUGE_FUNC_SCHE_CNTS_NAME string = "func_schedule_cnts"
    GAUGE_FUNC_SCHE_CNTS_HELP string = "Function schedule counts for each Function in KisFlow"

    HISTOGRAM_FUNCTION_DURATION_NAME string = "func_run_duration"
    HISTOGRAM_FUNCTION_DURATION_HELP string = "Function execution duration"

    HISTOGRAM_FLOW_DURATION_NAME string = "flow_run_duration"
    HISTOGRAM_FLOW_DURATION_HELP string = "Flow execution duration"
)

(3) Metric Instrumentation

To measure the execution time of each Flow, we should instrument the main entry point of the Flow, flow.Run(), as follows:

kis-flow/flow/kis_flow.go

// Run starts the stream processing of KisFlow, executing from the starting Function
func (flow *KisFlow) Run(ctx context.Context) error {

var fn kis.Function

fn = flow.FlowHead
flow.abort = false

if flow.Conf.Status == int(common.FlowDisable) {
    // Flow is configured to be disabled
    return nil
}

// ++++++++++ Metrics +++++++++
var funcStart, flowStart time.Time

// Since no Function has been executed yet, set PrevFunctionId to FirstVirtual as there is no previous Function
flow.PrevFunctionId = common.FunctionIdFirstVirtual

// Commit the original stream data
if err := flow.commitSrcData(ctx); err != nil {
    return err
}

if config.GlobalConfig.EnableProm == true {
    // Record Flow schedule count
    metrics.Metrics.FlowScheduleCntsTotal.WithLabelValues(flow.Name).Inc()
    // Record Flow execution time start
    flowStart = time.Now()
}

// Chain-style stream invocation
for fn != nil && flow.abort == false {

    // Record the current Function being executed in the Flow
    fid := fn.GetId()
    flow.ThisFunction = fn
    flow.ThisFunctionId = fid

    fName := fn.GetConfig().FName
    fMode := fn.GetConfig().FMode

    if config.GlobalConfig.EnableProm == true {
        // Record Function schedule count
        metrics.Metrics.FuncScheduleCntsTotal.WithLabelValues(fName, fMode).Inc()

        // Record Function execution time start
        funcStart = time.Now()
    }

    // Obtain the source data for the current Function to process
    if inputData, err := flow.getCurData(); err != nil {
        log.Logger().ErrorFX(ctx, "flow.Run(): getCurData err = %s\n", err.Error())
        return err
    } else {
        flow.inPut = inputData
    }

    if err := fn.Call(ctx, flow); err != nil {
        // Error
        return err
    } else {
        // Success
        fn, err = flow.dealAction(ctx, fn)
        if err != nil {
            return err
        }

        // Record Function execution duration
        if config.GlobalConfig.EnableProm == true {
            // Function execution duration
            duration := time.Since(funcStart)

            // Record the current Function execution time metric
            metrics.Metrics.FunctionDuration.With(
                prometheus.Labels{
                    common.LABEL_FUNCTION_NAME: fName,
                    common.LABEL_FUNCTION_MODE: fMode}).Observe(duration.Seconds() * 1000)
        }
    }
}

// Record Flow execution duration
if config.GlobalConfig.EnableProm == true {
    // Flow execution duration
    duration := time.Since(flowStart)

    // Record the Flow execution time metric
    metrics.Metrics.FlowDuration.With(
        prometheus.Labels{
            common.LABEL_FLOW_NAME: flow.Name}).Observe(duration.Seconds() * 1000)
}

return nil

}


The instrumentation captures the start time before invoking the first Function and calculates the execution duration after the Flow completes. This duration is then recorded in the corresponding bucket of the `HistogramVec` for the Flow execution time.


## 10.6 KieMetrics Unit Testing (Other Metrics Indicators)

### 10.6.1 Creating Unit Tests
We can reuse the previous TestMetricsDataTotal() method for unit test cases, as shown below:

> kis-flow/test/kis_metrics_test.go
```go
package test

import (
    "context"
    "kis-flow/common"
    "kis-flow/file"
    "kis-flow/kis"
    "kis-flow/test/caas"
    "kis-flow/test/faas"
    "testing"
    "time"
)

func TestMetricsDataTotal(t *testing.T) {
    ctx := context.Background()

    // 0. Register Function callbacks
    kis.Pool().FaaS("funcName1", faas.FuncDemo1Handler)
    kis.Pool().FaaS("funcName2", faas.FuncDemo2Handler)
    kis.Pool().FaaS("funcName3", faas.FuncDemo3Handler)

    // 0. Register ConnectorInit and Connector callbacks
    kis.Pool().CaaSInit("ConnName1", caas.InitConnDemo1)
    kis.Pool().CaaS("ConnName1", "funcName2", common.S, caas.CaasDemoHandler1)

    // 1. Load configuration files and build Flow
    if err := file.ConfigImportYaml("/Users/tal/gopath/src/kis-flow/test/load_conf/"); err != nil {
        panic(err)
    }

    // 2. Get Flow
    flow1 := kis.Pool().GetFlow("flowName1")

    n := 0

    for n < 10 {
        // 3. Submit raw data
        _ = flow1.CommitRow("This is Data1 from Test")

        // 4. Execute flow1
        if err := flow1.Run(ctx); err != nil {
            panic(err)
        }

        time.Sleep(1 * time.Second)
        n++
    }

    select {}
}

Execute the unit test by navigating to kis-flow/test/ and running:

go test -test.v -test.paniconexit0 -test.run TestMetricsDataTotal

You will see many log outputs. After waiting for 10 seconds, open another terminal and input the following command:

curl http://0.0.0.0:20004/metrics

You will see the following results:

# HELP flow_data_total KisFlow data count for each FlowID
# TYPE flow_data_total gauge
flow_data_total{flow_name="flowName1"} 10
# HELP flow_run_duration Flow execution time
# TYPE flow_run_duration histogram
flow_run_duration_bucket{flow_name="flowName1",le="0.005"} 0
flow_run_duration_bucket{flow_name="flowName1",le="0.01"} 0
flow_run_duration_bucket{flow_name="flowName1",le="0.03"} 0
flow_run_duration_bucket{flow_name="flowName1",le="0.08"} 0
flow_run_duration_bucket{flow_name="flowName1",le="0.1"} 0
flow_run_duration_bucket{flow_name="flowName1",le="0.5"} 0
flow_run_duration_bucket{flow_name="flowName1",le="1"} 0
flow_run_duration_bucket{flow_name="flowName1",le="5"} 9
flow_run_duration_bucket{flow_name="flowName1",le="10"} 10
flow_run_duration_bucket{flow_name="flowName1",le="100"} 10
flow_run_duration_bucket{flow_name="flowName1",le="1000"} 10
flow_run_duration_bucket{flow_name="flowName1",le="5000"} 10
flow_run_duration_bucket{flow_name="flowName1",le="30000"} 10
flow_run_duration_bucket{flow_name="flowName1",le="60000"} 10
flow_run_duration_bucket{flow_name="flowName1",le="+Inf"} 10
flow_run_duration_sum{flow_name="flowName1"} 29.135023
flow_run_duration_count{flow_name="flowName1"} 10
# HELP flow_schedule_cnts Number of times each FlowID is scheduled in KisFlow
# TYPE flow_schedule_cnts gauge
flow_schedule_cnts{flow_name="flowName1"} 10
# HELP func_run_duration Function execution time
# TYPE func_run_duration histogram
func_run_duration_bucket{func_mode="Calculate",func_name="funcName3",le="0.005"} 0
func_run_duration_bucket{func_mode="Calculate",func_name="funcName3",le="0.01"} 0
func_run_duration_bucket{func_mode="Calculate",func_name="funcName3",le="0.03"} 0
func_run_duration_bucket{func_mode="Calculate",func_name="funcName3",le="0.08"} 0
func_run_duration_bucket{func_mode="Calculate",func_name="funcName3",le="0.1"} 0
func_run_duration_bucket{func_mode="Calculate",func_name="funcName3",le="0.5"} 0
func_run_duration_bucket{func_mode="Calculate",func_name="funcName3",le="1"} 0
func_run_duration_bucket{func_mode="Calculate",func_name="funcName3",le="5"} 9
func_run_duration_bucket{func_mode="Calculate",func_name="funcName3",le="10"} 10
func_run_duration_bucket{func_mode="Calculate",func_name="funcName3",le="100"} 10
func_run_duration_bucket{func_mode="Calculate",func_name="funcName3",le="1000"} 10
func_run_duration_bucket{func_mode="Calculate",func_name="funcName3",le="5000"} 10
func_run_duration_bucket{func_mode="Calculate",func_name="funcName3",le="30000"} 10
func_run_duration_bucket{func_mode="Calculate",func_name="funcName3",le="+Inf"} 10
func_run_duration_sum{func_mode="Calculate",func_name="funcName3"} 20.925857
func_run_duration_count{func_mode="Calculate",func_name="funcName3"} 10
func_run_duration_bucket{func_mode="Save",func_name="funcName2",le="0.005"} 0
func_run_duration_bucket{func_mode="Save",func_name="funcName2",le="0.01"} 0
func_run_duration_bucket{func_mode="Save",func_name="funcName2",le="0.03"} 0
func_run_duration_bucket{func_mode="Save",func_name="funcName2",le="0.08"} 0
func_run_duration_bucket{func_mode="Save",func_name="funcName2",le="0.1"} 0
func_run_duration_bucket{func_mode="Save",func_name="funcName2",le="0.5"} 0
func_run_duration_bucket{func_mode="Save",func_name="funcName2",le="1"} 1
func_run_duration_bucket{func_mode="Save",func_name="funcName2",le="5"} 10
func_run_duration_bucket{func_mode="Save",func_name="funcName2",le="10"} 10
func_run_duration_bucket{func_mode="Save",func_name="funcName2",le="100"} 10
func_run_duration_bucket{func_mode="Save",func_name="funcName2",le="1000"} 10
func_run_duration_bucket{func_mode="Save",func_name="funcName2",le="5000"} 10
func_run_duration_bucket{func_mode="Save",func_name="funcName2",le="30000"} 10
func_run_duration_bucket{func_mode="Save",func_name="funcName2",le="+Inf"} 10
func_run_duration_sum{func_mode="Save",func_name="funcName2"} 27.026124
func_run_duration_count{func_mode="Save",func_name="funcName2"} 10
func_run_duration_bucket{func_mode="Verify",func_name="funcName1",le="0.005"} 0
func_run_duration_bucket{func_mode="Verify",func_name="funcName1",le="0.01"} 0
func_run_duration_bucket{func_mode="Verify",func_name="funcName1",le="0.03"} 0
func_run_duration_bucket{func_mode="Verify",func_name="funcName1",le="0.08"} 0
func_run_duration_bucket{func_mode="Verify",func_name="funcName1",le="0.1"} 0
func_run_duration_bucket{func_mode="Verify",func_name="funcName1",le="0.5"} 5
func_run_duration_bucket{func_mode="Verify",func_name="funcName1",le="1"} 10
func_run_duration_bucket{func_mode="Verify",func_name="funcName1",le="5"} 10
func_run_duration_bucket{func_mode="Verify",func_name="funcName1",le="10"} 10
func_run_duration_bucket{func_mode="Verify",func_name="funcName1",le="100"} 10
func_run_duration_bucket{func_mode="Verify",func_name="funcName1",le="1000"} 10
func_run_duration_bucket{func_mode="Verify",func_name="funcName1",le="5000"} 10
func_run_duration_bucket{func_mode="Verify",func_name="funcName1",le="30000"} 10
func_run_duration_bucket{func_mode="Verify",func_name="funcName1",le="+Inf"} 10
func_run_duration_sum{func_mode="Verify",func_name="funcName1"} 13.858197
func_run_duration_count{func_mode="Verify",func_name="funcName1"} 10

This concludes the section on KieMetrics Unit Testing and other Metrics Indicators.

10.7 Grafana Dashboard Display for KisFlow Metrics

With Prometheus metrics collected, we can integrate Grafana to display dashboards for KisFlow stream processing programs. Since each developer's project metrics and dashboard requirements may vary, this document does not provide specific Grafana dashboard configuration files. Instead, here is a sample dashboard for a KisFlow project for demonstration purposes, as shown below:

10.8 [V0.9] Source Code

https://github.com/aceld/kis-flow/releases/tag/v0.9

Author: Aceld
GitHub: https://github.com/aceld

KisFlow Open Source Project Address: https://github.com/aceld/kis-flow

Document: https://github.com/aceld/kis-flow/wiki

Case1-Quick Start
Case2-Flow Parallel Operation
Case3-Application of KisFlow in Multi-Goroutines
Case4-KisFlow in Message Queue (MQ) Applications

(Part 9)Golang Framework Hands-on - Multiple Copies of Flow

Aceld — Mon, 24 Jun 2024 07:28:13 +0000

Github: https://github.com/aceld/kis-flow
Document: https://github.com/aceld/kis-flow/wiki

Case1-Quick Start
Case2-Flow Parallel Operation
Case3-Application of KisFlow in Multi-Goroutines
Case4-KisFlow in Message Queue (MQ) Applications

9.1 Multi-Replica Capability

If KisFlow needs to be used concurrently by multiple Goroutines while executing flows, it may require creating multiple flows with the same configuration to match multiple concurrent computation flows. Therefore, Flow needs the ability to create replicas. This chapter will implement this capability.

9.1.1 Adding Fork Interface to Flow

First, add a new interface Fork() to the Flow abstraction layer with the following prototype:

kis-flow/kis/flow.go

type Flow interface {
    // Run schedules the Flow, sequentially dispatching and executing the Functions within the Flow
    Run(ctx context.Context) error
    // Link connects the Functions within the Flow according to the configuration file
    Link(fConf *config.KisFuncConfig, fParams config.FParam) error
    // CommitRow submits data to be executed to the Function layer of the Flow
    CommitRow(row interface{}) error
    // Input gets the input source data of the currently executing Function in the Flow
    Input() common.KisRowArr
    // GetName gets the name of the Flow
    GetName() string
    // GetThisFunction gets the currently executing Function
    GetThisFunction() Function
    // GetThisFuncConf gets the configuration of the currently executing Function
    GetThisFuncConf() *config.KisFuncConfig
    // GetConnector gets the Connector of the currently executing Function
    GetConnector() (Connector, error)
    // GetConnConf gets the configuration of the Connector of the currently executing Function
    GetConnConf() (*config.KisConnConfig, error)
    // GetConfig gets the configuration of the current Flow
    GetConfig() *config.KisFlowConfig
    // GetFuncConfigByName gets the configuration of the Function by name in the current Flow
    GetFuncConfigByName(funcName string) *config.KisFuncConfig
    // Next performs the Action for the next Function in the Flow
    Next(acts ...ActionFunc) error
    // GetCacheData gets the cached data of the current Flow
    GetCacheData(key string) interface{}
    // SetCacheData sets the cached data of the current Flow
    SetCacheData(key string, value interface{}, Exp time.Duration)
    // GetMetaData gets the temporary data of the current Flow
    GetMetaData(key string) interface{}
    // SetMetaData sets the temporary data of the current Flow
    SetMetaData(key string, value interface{})
    // GetFuncParam gets a key-value pair of the default parameters for the currently executing Function in the Flow
    GetFuncParam(key string) string
    // GetFuncParamAll gets all key-value pairs of the default parameters for the currently executing Function in the Flow
    GetFuncParamAll() config.FParam

    // +++++++++++++++++++++++++
    // Fork gets a deep copy of the Flow
    Fork(ctx context.Context) Flow
}

Fork() will clone a new KisFlow instance based on an existing KisFlow instance, creating a resource-isolated but identically configured KisFlow instance. The implementation method is as follows:

kis-flow/flow/kis_flow.go

// Fork gets a deep copy of the Flow
func (flow *KisFlow) Fork(ctx context.Context) kis.Flow {

    config := flow.Conf

    // Create a new Flow using the previous configuration
    newFlow := NewKisFlow(config)

    for _, fp := range flow.Conf.Flows {
        if _, ok := flow.funcParams[flow.Funcs[fp.FuncName].GetId()]; !ok {
            // The current Function has no Params configured
            newFlow.Link(flow.Funcs[fp.FuncName].GetConfig(), nil)
        } else {
            // The current Function has Params configured
            newFlow.Link(flow.Funcs[fp.FuncName].GetConfig(), fp.Params)
        }
    }

    log.Logger().DebugFX(ctx, "=====>Flow Fork, oldFlow.funcParams = %+v\n", flow.funcParams)
    log.Logger().DebugFX(ctx, "=====>Flow Fork, newFlow.funcParams = %+v\n", newFlow.GetFuncParamsAllFuncs())

    return newFlow
}

In Fork(), a new KisFlow instance is created based on the configuration information of the existing flow, and the associated Params and other configuration information are copied along with it. Finally, the newly created Functions are linked to the new Flow using Link().

To facilitate debugging, a new interface GetFuncParamsAllFuncs() has been added to the Flow to print the information of all FuncParams. The implementation method is as follows:

kis-flow/kis/flow.go

type Flow interface {
    // ... ...
    // ... ...

    // GetFuncParamsAllFuncs retrieves all key-value pairs of the FuncParams for all Functions in the Flow
    GetFuncParamsAllFuncs() map[string]config.FParam

    // ... ...
}

kis-flow/flow/kis_flow_data.go

// GetFuncParamsAllFuncs retrieves all key-value pairs of the FuncParams for all Functions in the Flow
func (flow *KisFlow) GetFuncParamsAllFuncs() map[string]config.FParam {
    flow.fplock.RLock()
    defer flow.fplock.RUnlock()

    return flow.funcParams
}

9.2 Unit Testing

Next, we will test the Fork capability with the following unit test code:

kis-flow/test/kis_fork_test.go

func TestForkFlow(t *testing.T) {
    ctx := context.Background()

    // 0. Register Function callback businesses
    kis.Pool().FaaS("funcName1", faas.FuncDemo1Handler)
    kis.Pool().FaaS("funcName2", faas.FuncDemo2Handler)
    kis.Pool().FaaS("funcName3", faas.FuncDemo3Handler)

    // 0. Register ConnectorInit and Connector callback businesses
    kis.Pool().CaaSInit("ConnName1", caas.InitConnDemo1)
    kis.Pool().CaaS("ConnName1", "funcName2", common.S, caas.CaasDemoHanler1)

    // 1. Load configuration files and build Flow
    if err := file.ConfigImportYaml("/Users/tal/gopath/src/kis-flow/test/load_conf/"); err != nil {
        panic(err)
    }

    // 2. Get Flow
    flow1 := kis.Pool().GetFlow("flowName1")

    flow1Clone1 := flow1.Fork(ctx)

    // 3. Submit original data
    _ = flow1Clone1.CommitRow("This is Data1 from Test")
    _ = flow1Clone1.CommitRow("This is Data2 from Test")
    _ = flow1Clone1.CommitRow("This is Data3 from Test")

    // 4. Execute flow1
    if err := flow1Clone1.Run(ctx); err != nil {
        panic(err)
    }
}

First, we create an instance of the flow flowName1 and then get flowClone1 using Fork(). After that, we execute the scheduling process of flowClone1.

Navigate to the kis-flow/test/ directory and execute:

go test -test.v -test.paniconexit0 -test.run TestForkFlow

The result is as follows:

=== RUN   TestForkFlow
Add KisPool FuncName=funcName1
Add KisPool FuncName=funcName2
Add KisPool FuncName=funcName3
Add KisPool CaaSInit CName=ConnName1
Add KisPool CaaS CName=ConnName1, FName=funcName2, Mode=Save
===> Call Connector InitDemo1
&{conn ConnName1 0.0.0.0:9988,0.0.0.0:9999,0.0.0.0:9990 redis redis-key map[args1:value1 args2:value2] [] [funcName2 funcName2]}
Add FlowRouter FlowName=flowName1
Add FlowRouter FlowName=flowName2
Add FlowRouter FlowName=flowName3
Add FlowRouter FlowName=flowName4
===> Call Connector InitDemo1
&{conn ConnName1 0.0.0.0:9988,0.0.0.0:9999,0.0.0.0:9990 redis redis-key map[args1:value1 args2:value2] [] [funcName2 funcName2 funcName2]}
Add FlowRouter FlowName=flowName5
===> Call Connector InitDemo1
&{conn ConnName1 0.0.0.0:9988,0.0.0.0:9999,0.0.0.0:9990 redis redis-key map[args1:value1 args2:value2] [] [funcName2 funcName2 funcName2]}
context.Background
=====>Flow Fork, oldFlow.funcParams = map[func-6b00f430fe494302a384c2ae09eb019c:map[default1:funcName3_param1 default2:funcName3_param2 myKey1:flowValue3-1 myKey2:flowValue3-2] func-bf9df5fc16684200b78f32985d073012:map[default1:funcName2_param1 default2:funcName2_param2 myKey1:flowValue2-1 myKey2:flowValue2-2] func-c0f1ae9850174f81b994a2e98fb34109:map[default1:funcName1_param1 default2:funcName1_param2 myKey1:flowValue1-1 myKey2:flowValue1-2]]
=====>Flow Fork, newFlow.funcParams = map[func-6b00f430fe494302a384c2ae09eb019c:map[default1:funcName3_param1 default2:funcName3_param2 myKey1:flowValue3-1 myKey2:flowValue3-2] func-bf9df5fc16684200b78f32985d073012:map[default1:funcName2_param1 default2:funcName2_param2 myKey1:flowValue2-1 myKey2:flowValue2-2] func-c0f1ae9850174f81b994a2e98fb34109:map[default1:funcName1_param1 default2:funcName1_param2 myKey1:flowValue1-1 myKey2:flowValue1-2]]
===>FaaS Demo1, row.Data = This is Data1 from Test
===>FaaS Demo1, row.Data = This is Data2 from Test
===>FaaS Demo1, row.Data = This is Data3 from Test
===>FaaS Demo2, row.Data = This is Data1 from Test
===>FaaS Demo2, row.Data = This is Data2 from Test
===>FaaS Demo2, row.Data = This is Data3 from Test
===>FaaS Demo3, row.Data = This is Data1 from Test
===>FaaS Demo3, row.Data = This is Data2 from Test
===>FaaS Demo3, row.Data = This is Data3 from Test
--- PASS: TestForkFlow (0.01s)
PASS
ok      command-line-arguments  (cached)

Github: https://github.com/aceld/kis-flow
Document: https://github.com/aceld/kis-flow/wiki

To be continued.

8.1 Flow Cache - Data Stream Caching

KisFlow also provides shared caching in stream computing, using a simple local cache for developers to use as needed. For third-party local cache technology dependencies, refer to: https://github.com/patrickmn/go-cache.

8.1.1 go-cache

(1) Installation

go get github.com/patrickmn/go-cache

(2) Usage

import (
    "fmt"
    "github.com/patrickmn/go-cache"
    "time"
)

func main() {
    // Create a cache with a default expiration time of 5 minutes, and which
    // purges expired items every 10 minutes
    c := cache.New(5*time.Minute, 10*time.Minute)

    // Set the value of the key "foo" to "bar", with the default expiration time
    c.Set("foo", "bar", cache.DefaultExpiration)

    // Set the value of the key "baz" to 42, with no expiration time
    // (the item won't be removed until it is re-set, or removed using
    // c.Delete("baz")
    c.Set("baz", 42, cache.NoExpiration)

    // Get the string associated with the key "foo" from the cache
    foo, found := c.Get("foo")
    if found {
        fmt.Println(foo)
    }

    // Since Go is statically typed, and cache values can be anything, type
    // assertion is needed when values are being passed to functions that don't
    // take arbitrary types, (i.e. interface{}). The simplest way to do this for
    // values which will only be used once--e.g. for passing to another
    // function--is:
    foo, found := c.Get("foo")
    if found {
        MyFunction(foo.(string))
    }

    // This gets tedious if the value is used several times in the same function.
    // You might do either of the following instead:
    if x, found := c.Get("foo"); found {
        foo := x.(string)
        // ...
    }
    // or
    var foo string
    if x, found := c.Get("foo"); found {
        foo = x.(string)
    }
    // ...
    // foo can then be passed around freely as a string

    // Want performance? Store pointers!
    c.Set("foo", &MyStruct, cache.DefaultExpiration)
    if x, found := c.Get("foo"); found {
        foo := x.(*MyStruct)
            // ...
    }
}

For detailed reference: https://github.com/patrickmn/go-cache

8.1.2 KisFlow Integration with go-cache

(1) Flow Provides Abstract Interface

Flow provides interfaces for cache operations as follows:

kis-flow/kis/flow.go

type Flow interface {
    // Run schedules the Flow, sequentially scheduling and executing Functions within the Flow
    Run(ctx context.Context) error
    // Link connects Functions within the Flow according to the configuration file
    Link(fConf *config.KisFuncConfig, fParams config.FParam) error
    // CommitRow submits Flow data to the Function layer about to be executed
    CommitRow(row interface{}) error
    // Input gets the input source data for the currently executing Function in the Flow
    Input() common.KisRowArr
    // GetName gets the name of the Flow
    GetName() string
    // GetThisFunction gets the currently executing Function
    GetThisFunction() Function
    // GetThisFuncConf gets the configuration of the currently executing Function
    GetThisFuncConf() *config.KisFuncConfig
    // GetConnector gets the Connector of the currently executing Function
    GetConnector() (Connector, error)
    // GetConnConf gets the configuration of the Connector for the currently executing Function
    GetConnConf() (*config.KisConnConfig, error)
    // GetConfig gets the configuration of the current Flow
    GetConfig() *config.KisFlowConfig
    // GetFuncConfigByName gets the configuration of the Function by its name
    GetFuncConfigByName(funcName string) *config.KisFuncConfig
    // Next advances the currently executing Function to the next Function with specified Action
    Next(acts ...ActionFunc) error

    // ++++++++++++++++++++++++++++++++++++++++
    // GetCacheData gets the cache data of the current Flow
    GetCacheData(key string) interface{}
    // SetCacheData sets the cache data of the current Flow
    SetCacheData(key string, value interface{}, Exp time.Duration)
}

SetCacheData() sets the local cache, with Exp as the expiration time. If Exp is 0, it is permanent.
GetCacheData() reads the local cache.

(2) Providing Constants

Provide some constants related to cache expiration time.

kis-flow/common/const.go

// cache
const (
    // DeFaultFlowCacheCleanUp is the default cache cleanup interval for Flow objects in KisFlow, in minutes
    DeFaultFlowCacheCleanUp = 5 // in minutes
    // DefaultExpiration is the default GoCache time, permanently saved
    DefaultExpiration time.Duration = 0
)

(3) Adding and Initializing Members in KisFlow

kis-flow/flow/kis_flow.go

// KisFlow represents the context environment for stream computing
type KisFlow struct {

    // ... ...
    // ... ...

    // Local cache for the flow
    cache *cache.Cache // Temporary cache context environment for Flow
}

// NewKisFlow creates a new KisFlow
func NewKisFlow(conf *config.KisFlowConfig) kis.Flow {
    flow := new(KisFlow)

    // ... ...
    // ... ...

    // Initialize local cache
    flow.cache = cache.New(cache.NoExpiration, common.DeFaultFlowCacheCleanUp*time.Minute)

    return flow
}

(4) Implementing the Interface

Finally, implement the two interfaces for cache read and write operations as follows:

kis-flow/flow/kis_flow_data.go

func (flow *KisFlow) GetCacheData(key string) interface{} {
    if data, found := flow.cache.Get(key); found {
        return data
    }
    return nil
}

func (flow *KisFlow) SetCacheData(key string, value interface{}, Exp time.Duration) {
    if Exp == common.DefaultExpiration {
        flow.cache.Set(key, value, cache.DefaultExpiration)
    } else {
        flow.cache.Set(key, value, Exp)
    }
}

8.2 MetaData Temporary Cache Parameters

MetaData is defined as a map[string]interface{} structure available at each level of Flow, Function, and Connector to store temporary data. The lifespan of this data is consistent with the lifespan of each instance.

8.2.1 Adding MetaData to Flow

First, add the metaData map[string]interface{} member and corresponding read-write lock to KisFlow.

kis-flow/flow/kis_flow.go

// KisFlow represents the context environment throughout the entire stream computing
type KisFlow struct {
    // ... ...

    // ... ...

    // +++++++++++++++++++++++++++++++++++++++++++
    // metaData for the flow
    metaData map[string]interface{} // Custom temporary data for Flow
    mLock    sync.RWMutex           // Read-write lock to manage metaData
}

Also, initialize the metaData member in the KisFlow constructor as follows:

kis-flow/flow/kis_flow.go

// NewKisFlow creates a KisFlow
func NewKisFlow(conf *config.KisFlowConfig) kis.Flow {
    flow := new(KisFlow)

    // ... ...
    // ... ...

    // ++++++++++++++++++++++++++++++++++++++
    // Initialize temporary data
    flow.metaData = make(map[string]interface{})

    return flow
}

Next, add the read and write interfaces for MetaData to the Flow as follows:

kis-flow/kis/flow.go

type Flow interface {
    // Run schedules the Flow, sequentially scheduling and executing Functions within the Flow
    Run(ctx context.Context) error
    // Link connects Functions within the Flow according to the configuration file
    Link(fConf *config.KisFuncConfig, fParams config.FParam) error
    // CommitRow submits Flow data to the Function layer about to be executed
    CommitRow(row interface{}) error
    // Input gets the input source data for the currently executing Function in the Flow
    Input() common.KisRowArr
    // GetName gets the name of the Flow
    GetName() string
    // GetThisFunction gets the currently executing Function
    GetThisFunction() Function
    // GetThisFuncConf gets the configuration of the currently executing Function
    GetThisFuncConf() *config.KisFuncConfig
    // GetConnector gets the Connector of the currently executing Function
    GetConnector() (Connector, error)
    // GetConnConf gets the configuration of the Connector for the currently executing Function
    GetConnConf() (*config.KisConnConfig, error)
    // GetConfig gets the configuration of the current Flow
    GetConfig() *config.KisFlowConfig
    // GetFuncConfigByName gets the configuration of the Function by its name
    GetFuncConfigByName(funcName string) *config.KisFuncConfig
    // Next advances the currently executing Function to the next Function with specified Action
    Next(acts ...ActionFunc) error
    // GetCacheData gets the cache data of the current Flow
    GetCacheData(key string) interface{}
    // SetCacheData sets the cache data of the current Flow
    SetCacheData(key string, value interface{}, Exp time.Duration)

    // ++++++++++++++++++++++++++++
    // GetMetaData gets the temporary data of the current Flow
    GetMetaData(key string) interface{}
    // SetMetaData sets the temporary data of the current Flow
    SetMetaData(key string, value interface{})
}

Define the GetMetaData() and SetMetaData() interfaces for reading and writing respectively. Finally, implement these interfaces as follows:

kis-flow/flow/kis_flow_data.go

// GetMetaData retrieves the temporary data of the current Flow object
func (flow *KisFlow) GetMetaData(key string) interface{} {
    flow.mLock.RLock()
    defer flow.mLock.RUnlock()

    data, ok := flow.metaData[key]
    if !ok {
        return nil
    }

    return data
}

// SetMetaData sets the temporary data of the current Flow object
func (flow *KisFlow) SetMetaData(key string, value interface{}) {
    flow.mLock.Lock()
    defer flow.mLock.Unlock()

    flow.metaData[key] = value
}

8.2.2 Adding MetaData to Function

First, add the metaData member to BaseFunction as follows:

kis-flow/function/kis_base_function.go

type BaseFunction struct {
    // Id, KisFunction instance ID, used to distinguish different instance objects within KisFlow
    Id     string
    Config *config.KisFuncConfig

    // flow
    flow kis.Flow // Context environment KisFlow

    // connector
    connector kis.Connector

    // ++++++++++++++++++++++++
    // Custom temporary data for Function
    metaData map[string]interface{}
    // Read-write lock to manage metaData
    mLock sync.RWMutex

    // link
    N kis.Function // Next stream computing Function
    P kis.Function // Previous stream computing Function
}

In the Function constructor, each specific Function needs a constructor to initialize the metaData member. The changes are as follows:

kis-flow/function/kis_base_function.go

func NewKisFunction(flow kis.Flow, config *config.KisFuncConfig) kis.Function {
    var f kis.Function

    // Factory produces generalized objects
    // ++++++++++++++
    switch common.KisMode(config.FMode) {
    case common.V:
        f = NewKisFunctionV() // +++
    case common.S:
        f = NewKisFunctionS() // +++
    case common.L:
        f = NewKisFunctionL() // +++
    case common.C:
        f = NewKisFunctionC() // +++
    case common.E:
        f = NewKisFunctionE() // +++
    default:
        // LOG ERROR
        return nil
    }

    // Generate random unique instance ID
    f.CreateId()

    // Set basic information properties
    if err := f.SetConfig(config); err != nil {
        panic(err)
    }

    // Set Flow
    if err := f.SetFlow(flow); err != nil {
        panic(err)
    }

    return f
}

Each constructor is as follows:

kis-flow/function/kis_function_c.go

func NewKisFunctionC() kis.Function {
    f := new(KisFunctionC)

    // Initialize metaData
    f.metaData = make(map[string]interface{})

    return f
}

kis-flow/function/kis_function_v.go

func NewKisFunctionV() kis.Function {
    f := new(KisFunctionV)

    // Initialize metaData
    f.metaData = make(map[string]interface{})

    return f
}

kis-flow/function/kis_function_e.go

func NewKisFunctionE() kis.Function {
    f := new(KisFunctionE)

    // Initialize metaData
    f.metaData = make(map[string]interface{})

    return f
}

kis-flow/function/kis_function_s.go

func NewKisFunctionS() kis.Function {
    f := new(KisFunctionS)

    // Initialize metaData
    f.metaData = make(map[string]interface{})

    return f
}

kis-flow/function/kis_function_l.go

func NewKisFunctionL() kis.Function {
    f := new(KisFunctionL)

    // Initialize metaData
    f.metaData = make(map[string]interface{})

    return f
}

Next, add interfaces to access the metaData member in the Function abstraction layer as follows:

type Function interface {
    // Call executes the stream computing logic
    Call(ctx context.Context, flow Flow) error

    // SetConfig configures the strategy for the current Function instance
    SetConfig(s *config.KisFuncConfig) error
    // GetConfig retrieves the configuration strategy of the current Function instance
    GetConfig() *config.KisFuncConfig

    // SetFlow sets the Flow instance that the current Function instance depends on
    SetFlow(f Flow) error
    // GetFlow retrieves the Flow instance that the current Function instance depends on
    GetFlow() Flow

    // AddConnector adds a Connector to the current Function instance
    AddConnector(conn Connector) error
    // GetConnector retrieves the Connector associated with the current Function instance
    GetConnector() Connector

    // CreateId generates a random instance KisID for the current Function instance
    CreateId()
    // GetId retrieves the FID of the current Function
    GetId() string
    // GetPrevId retrieves the FID of the previous Function node of the current Function
    GetPrevId() string
    // GetNextId retrieves the FID of the next Function node of the current Function
    GetNextId() string

    // Next returns the next layer computing stream Function, or nil if it is the last layer
    Next() Function
    // Prev returns the previous layer computing stream Function, or nil if it is the last layer
    Prev() Function
    // SetN sets the next Function instance
    SetN(f Function)
    // SetP sets the previous Function instance
    SetP(f Function)

    // ++++++++++++++++++++++++++++++++++
    // GetMetaData retrieves the temporary data of the current Function
    GetMetaData(key string) interface{}
    // SetMetaData sets the temporary data of the current Function
    SetMetaData(key string, value interface{})
}

Implement the above two interfaces in the BaseFunction.

kis-flow/function/kis_base_function.go

// GetMetaData retrieves the temporary data of the current Function
func (base *BaseFunction) GetMetaData(key string) interface{} {
    base.mLock.RLock()
    defer base.mLock.RUnlock()

    data, ok := base.metaData[key]
    if !ok {
        return nil
    }

    return data
}

// SetMetaData sets the temporary data of the current Function
func (base *BaseFunction) SetMetaData(key string, value interface{}) {
    base.mLock.Lock()
    defer base.mLock.Unlock()

    base.metaData[key] = value
}

8.2.3 Adding MetaData to Connector

First, add the metaData member to KisConnector as follows:

kis-flow/conn/kis_connector.go

type KisConnector struct {
    // Connector ID
    CId string
    // Connector Name
    CName string
    // Connector Config
    Conf *config.KisConnConfig
    // Connector Init
    onceInit sync.Once

    // ++++++++++++++
    // Custom temporary data for KisConnector
    metaData map[string]interface{}
    // Read-write lock to manage metaData
    mLock sync.RWMutex
}

// NewKisConnector creates a KisConnector based on the configuration strategy
func NewKisConnector(config *config.KisConnConfig) *KisConnector {
    conn := new(KisConnector)
    conn.CId = id.KisID(common.KisIdTypeConnector)
    conn.CName = config.CName
    conn.Conf = config

    // +++++++++++++++++++++++++++++++++++
    conn.metaData = make(map[string]interface{})

    return conn
}

Initialize metaData in the constructor.

Next, add interfaces to access and set MetaData in the Connector abstraction layer as follows:

kis-flow/kis/connector.go

type Connector interface {
    // Init initializes the links of the storage engine associated with the Connector
    Init() error
    // Call invokes the read and write operations of the external storage logic of the Connector
    Call(ctx context.Context, flow Flow, args interface{}) error
    // GetId retrieves the ID of the Connector
    GetId() string
    // GetName retrieves the name of the Connector
    GetName() string
    // GetConfig retrieves the configuration information of the Connector
    GetConfig() *config.KisConnConfig
    // GetMetaData retrieves the temporary data of the current Connector

    // +++++++++++++++++++++++++++++++
    GetMetaData(key string) interface{}
    // SetMetaData sets the temporary data of the current Connector
    SetMetaData(key string, value interface{})
}

Finally, implement the above two interfaces in KisConnector as follows:

kis-flow/conn/kis_connector.go

// GetMetaData retrieves the temporary data of the current Connector
func (conn *KisConnector) GetMetaData(key string) interface{} {
    conn.mLock.RLock()
    defer conn.mLock.RUnlock()

    data, ok := conn.metaData[key]
    if !ok {
        return nil
    }

    return data
}

// SetMetaData sets the temporary data of the current Connector
func (conn *KisConnector) SetMetaData(key string, value interface{}) {
    conn.mLock.Lock()
    defer conn.mLock.Unlock()

    conn.metaData[key] = value
}

8.3 Configuration File Parameters

KisFlow allows developers to define default parameters (Params) for configuring Flow, Function, Connector, etc., in the configuration file. Here are some examples:

Function:

kistype: func
fname: funcName1
fmode: Verify
source:
  name: Official Account Douyin Mall Order Data
  must:
    - order_id
    - user_id
option:
  default_params:
    default1: funcName1_param1
    default2: funcName1_param2

Flow:

kistype: flow
status: 1
flow_name: flowName1
flows:
  - fname: funcName1
    params:
      myKey1: flowValue1-1
      myKey2: flowValue1-2
  - fname: funcName2
    params:
      myKey1: flowValue2-1
      myKey2: flowValue2-2
  - fname: funcName3
    params:
      myKey1: flowValue3-1
      myKey2: flowValue3-2

Connector:

kistype: conn
cname: ConnName1
addrs: '0.0.0.0:9988,0.0.0.0:9999,0.0.0.0:9990'
type: redis
key: redis-key
params:
  args1: value1
  args2: value2
load: null
save:
  - funcName2

Developers can provide Params for each defined module. Params provided in Flow will also be added to the Functions.

In the previous steps, we already read these parameters into each module's memory, but we did not expose an interface for developers.

8.3.1 Adding Param Retrieval Interface to Flow

First, we provide an interface for Flow to query Params:

kis-flow/kis/flow.go

type Flow interface {
    // ... ...
    // ... ...

    // GetFuncParam retrieves a key-value pair of the default parameters for the currently executing Function in the Flow
    GetFuncParam(key string) string
    // GetFuncParamAll retrieves all key-value pairs of the default parameters for the currently executing Function in the Flow
    GetFuncParamAll() config.FParam
}

Implementation:

kis-flow/flow/kis_flow_data.go

// GetFuncParam retrieves a key-value pair of the default parameters for the currently executing Function in the Flow
func (flow *KisFlow) GetFuncParam(key string) string {
    flow.fplock.RLock()
    defer flow.fplock.RUnlock()

    if param, ok := flow.funcParams[flow.ThisFunctionId]; ok {
        if value, vok := param[key]; vok {
            return value
        }
    }

    return ""
}

// GetFuncParamAll retrieves all key-value pairs of the default parameters for the currently executing Function in the Flow
func (flow *KisFlow) GetFuncParamAll() config.FParam {
    flow.fplock.RLock()
    defer flow.fplock.RUnlock()

    param, ok := flow.funcParams[flow.ThisFunctionId]
    if !ok {
        return nil
    }

    return param
}

GetFuncParam() and GetFuncParamAll() retrieve a single key or all parameters respectively, but both fetch the Params for the currently executing Function.

8.3.2 Unit Testing

We add some parameters to each Function in flowName1.

kis-flow/test/load_conf/flow-FlowName1.yml

kistype: flow
status: 1
flow_name: flowName1
flows:
  - fname: funcName1
    params:
      myKey1: flowValue1-1
      myKey2: flowValue1-2
  - fname: funcName2
    params:
      myKey1: flowValue2-1
      myKey2: flowValue2-2
  - fname: funcName3
    params:
      myKey1: flowValue3-1
      myKey2: flowValue3-2

Then configure some default custom parameters for each associated Function:

kis-flow/test/load_conf/func/func-FuncName1.yml

kistype: func
fname: funcName1
fmode: Verify
source:
  name: Official Account Douyin Mall Order Data
  must:
    - order_id
    - user_id
option:
  default_params:
    default1: funcName1_param1
    default2: funcName1_param2

kis-flow/test/load_conf/func/func-FuncName2.yml

kistype: func
fname: funcName2
fmode: Save
source:
  name: User Order Error Rate
  must:
    - order_id
    - user_id
option:
  cname: ConnName1
  default_params:
    default1: funcName2_param1
    default2: funcName2_param2

kis-flow/test/load_conf/func/func-FuncName3.yml

kistype: func
fname: funcName3
fmode: Calculate
source:
  name: User Order Error Rate
  must:
    - order_id
    - user_id
option:
  default_params:
    default1: funcName3_param1
    default2: funcName3_param2

We also configure some Param parameters for the Connector associated with FuncName2:

kis-flow/test/load_conf/conn/conn-ConnName1.yml

kistype: conn
cname: ConnName1
addrs: '0.0.0.0:9988,0.0.0.0:9999,0.0.0.0:9990'
type: redis
key: redis-key
params:
  args1: value1
  args2: value2
load: null
save:
  - funcName2

To verify that our configuration parameters can be correctly retrieved during the execution of Functions, we modified each Function and Connector business function to print their Params:

kis-flow/test/faas/faas_demo1.go

func FuncDemo1Handler(ctx context.Context, flow kis.Flow) error {
    fmt.Println("---> Call funcName1Handler ----")

    // ++++++++++++++++
    fmt.Printf("Params = %+v\n", flow.GetFuncParamAll())

    for index, row := range flow.Input() {
        // Print data
        str := fmt.Sprintf("In FuncName = %s, FuncId = %s, row = %s", flow.GetThisFuncConf().FName, flow.GetThisFunction().GetId(), row)
        fmt.Println(str)

        // Calculate result data
        resultStr := fmt.Sprintf("data from funcName[%s], index = %d", flow.GetThisFuncConf().FName, index)

        // Commit result data
        _ = flow.CommitRow(resultStr)
    }

    return nil
}

kis-flow/test/faas/faas_demo2.go

func FuncDemo2Handler(ctx context.Context, flow kis.Flow) error {
    fmt.Println("---> Call funcName2Handler ----")

    // ++++++++++++++++
    fmt.Printf("Params = %+v\n", flow.GetFuncParamAll())

    for index, row := range flow.Input() {
        str := fmt.Sprintf("In FuncName = %s, FuncId = %s, row = %s", flow.GetThisFuncConf().FName, flow.GetThisFunction().GetId(), row)
        fmt.Println(str)

        conn, err := flow.GetConnector()
        if err != nil {
            log.Logger().ErrorFX(ctx, "FuncDemo2Handler(): GetConnector err = %s\n", err.Error())
            return err
        }

        if conn.Call(ctx, flow, row) != nil {
            log.Logger().ErrorFX(ctx, "FuncDemo2Handler(): Call err = %s\n", err.Error())
            return err
        }

        // Calculate result data
        resultStr := fmt.Sprintf("data from funcName[%s], index = %d", flow.GetThisFuncConf().FName, index)

        // Commit result data
        _ = flow.CommitRow(resultStr)
    }

    return nil
}

kis-flow/test/faas/faas_demo3.go

func FuncDemo3Handler(ctx context.Context, flow kis.Flow) error {
    fmt.Println("---> Call funcName3Handler ----")

    // ++++++++++++++++
    fmt.Printf("Params = %+v\n", flow.GetFuncParamAll())

    for _, row := range flow.Input() {
        str := fmt.Sprintf("In FuncName = %s, FuncId = %s, row = %s", flow.GetThisFuncConf().FName, flow.GetThisFunction().GetId(), row)
        fmt.Println(str)
    }

    return nil
}

kis-flow/test/caas/caas_demo1.go

func CaasDemoHanler1(ctx context.Context, conn kis.Connector, fn kis.Function, flow kis.Flow, args interface{}) error {
    fmt.Printf("===> In CaasDemoHanler1: flowName: %s, cName:%s, fnName:%s, mode:%s\n",
        flow.GetName(), conn.GetName(), fn.GetConfig().FName, fn.GetConfig().FMode)

    // +++++++++++ 
    fmt.Printf("Params = %+v\n", conn.GetConfig().Params)

    fmt.Printf("===> Call Connector CaasDemoHanler1, args from funciton: %s\n", args)

    return nil
}

Finally, we write the unit test cases:

kis-flow/test/kis_params_test.go

package test

import (
    "context"
    "kis-flow/common"
    "kis-flow/file"
    "kis-flow/kis"
    "kis-flow/test/caas"
    "kis-flow/test/faas"
    "testing"
)

func TestParams(t *testing.T) {
    ctx := context.Background()

    // 0. Register Function callback businesses
    kis.Pool().FaaS("funcName1", faas.FuncDemo1Handler)
    kis.Pool().FaaS("funcName2", faas.FuncDemo2Handler)
    kis.Pool().FaaS("funcName3", faas.FuncDemo3Handler)

    // 0. Register ConnectorInit and Connector callback businesses
    kis.Pool().CaaSInit("ConnName1", caas.InitConnDemo1)
    kis.Pool().CaaS("ConnName1", "funcName2", common.S, caas.CaasDemoHanler1)

    // 1. Load configuration files and build Flow
    if err := file.ConfigImportYaml("/Users/tal/gopath/src/kis-flow/test/load_conf/"); err != nil {
        panic(err)
    }

    // 2. Get Flow
    flow1 := kis.Pool().GetFlow("flowName1")

    // 3. Submit original data
    _ = flow1.CommitRow("This is Data1 from Test")
    _ = flow1.CommitRow("This is Data2 from Test")
    _ = flow1.CommitRow("This is Data3 from Test")

    // 4. Execute flow1
    if err := flow1.Run(ctx); err != nil {
        panic(err)
    }
}

Navigate to the kis-flow/test/ directory and execute:

go test -test.v -test.paniconexit0 -test.run  TestParams

=== RUN   TestParams
....
....

---> Call funcName1Handler ----
Params = map[default1:funcName1_param1 default2:funcName1_param2 myKey1:flowValue1-1 myKey2:flowValue1-2]
...
...

---> Call funcName2Handler ----
Params = map[default1:funcName2_param1 default2:funcName2_param2 myKey1:flowValue2-1 myKey2:flowValue2-2]
...
...
===> In CaasDemoHanler1: flowName: flowName1, cName:ConnName1, fnName:funcName2, mode:Save
Params = map[args1:value1 args2:value2]
...
...
===> In CaasDemoHanler1: flowName: flowName1, cName:ConnName1, fnName:funcName2, mode:Save
Params = map[args1:value1 args2:value2]
...
...

===> In CaasDemoHanler1: flowName: flowName1, cName:ConnName1, fnName:funcName2, mode:Save
Params = map[args1:value1 args2:value2]

...
...

---> Call funcName3Handler ----
Params = map[default1:funcName3_param1 default2:funcName3_param2 myKey1:flowValue3-1 myKey2:flowValue3-2]

...
...
--- PASS: TestParams (0.01s)
PASS
ok      kis-flow/test   0.433s

As we can see, we can now correctly retrieve the Params configuration parameters at each level.

8.4 [V0.7] Source Code

https://github.com/aceld/kis-flow/releases/tag/v0.7

Author: Aceld
GitHub: https://github.com/aceld

KisFlow Open Source Project Address: https://github.com/aceld/kis-flow

Document: https://github.com/aceld/kis-flow/wiki

Case1-Quick Start
Case2-Flow Parallel Operation
Case3-Application of KisFlow in Multi-Goroutines
Case4-KisFlow in Message Queue (MQ) Applications

(Part 8)Golang Framework Hands-on - Cache/Params Data Caching and Data Parameters

Aceld — Wed, 19 Jun 2024 03:07:00 +0000

Github: https://github.com/aceld/kis-flow
Document: https://github.com/aceld/kis-flow/wiki

Case1-Quick Start
Case2-Flow Parallel Operation
Case3-Application of KisFlow in Multi-Goroutines
Case4-KisFlow in Message Queue (MQ) Applications

8.1 Flow Cache - Data Stream Caching

8.1.1 go-cache

(1) Installation

go get github.com/patrickmn/go-cache

(2) Usage

import (
    "fmt"
    "github.com/patrickmn/go-cache"
    "time"
)

func main() {
    // Create a cache with a default expiration time of 5 minutes, and which
    // purges expired items every 10 minutes
    c := cache.New(5*time.Minute, 10*time.Minute)

    // Set the value of the key "foo" to "bar", with the default expiration time
    c.Set("foo", "bar", cache.DefaultExpiration)

    // Set the value of the key "baz" to 42, with no expiration time
    // (the item won't be removed until it is re-set, or removed using
    // c.Delete("baz")
    c.Set("baz", 42, cache.NoExpiration)

    // Get the string associated with the key "foo" from the cache
    foo, found := c.Get("foo")
    if found {
        fmt.Println(foo)
    }

    // Since Go is statically typed, and cache values can be anything, type
    // assertion is needed when values are being passed to functions that don't
    // take arbitrary types, (i.e. interface{}). The simplest way to do this for
    // values which will only be used once--e.g. for passing to another
    // function--is:
    foo, found := c.Get("foo")
    if found {
        MyFunction(foo.(string))
    }

    // This gets tedious if the value is used several times in the same function.
    // You might do either of the following instead:
    if x, found := c.Get("foo"); found {
        foo := x.(string)
        // ...
    }
    // or
    var foo string
    if x, found := c.Get("foo"); found {
        foo = x.(string)
    }
    // ...
    // foo can then be passed around freely as a string

    // Want performance? Store pointers!
    c.Set("foo", &MyStruct, cache.DefaultExpiration)
    if x, found := c.Get("foo"); found {
        foo := x.(*MyStruct)
            // ...
    }
}

For detailed reference: https://github.com/patrickmn/go-cache

8.1.2 KisFlow Integration with go-cache

(1) Flow Provides Abstract Interface

Flow provides interfaces for cache operations as follows:

kis-flow/kis/flow.go

type Flow interface {
    // Run schedules the Flow, sequentially scheduling and executing Functions within the Flow
    Run(ctx context.Context) error
    // Link connects Functions within the Flow according to the configuration file
    Link(fConf *config.KisFuncConfig, fParams config.FParam) error
    // CommitRow submits Flow data to the Function layer about to be executed
    CommitRow(row interface{}) error
    // Input gets the input source data for the currently executing Function in the Flow
    Input() common.KisRowArr
    // GetName gets the name of the Flow
    GetName() string
    // GetThisFunction gets the currently executing Function
    GetThisFunction() Function
    // GetThisFuncConf gets the configuration of the currently executing Function
    GetThisFuncConf() *config.KisFuncConfig
    // GetConnector gets the Connector of the currently executing Function
    GetConnector() (Connector, error)
    // GetConnConf gets the configuration of the Connector for the currently executing Function
    GetConnConf() (*config.KisConnConfig, error)
    // GetConfig gets the configuration of the current Flow
    GetConfig() *config.KisFlowConfig
    // GetFuncConfigByName gets the configuration of the Function by its name
    GetFuncConfigByName(funcName string) *config.KisFuncConfig
    // Next advances the currently executing Function to the next Function with specified Action
    Next(acts ...ActionFunc) error

    // ++++++++++++++++++++++++++++++++++++++++
    // GetCacheData gets the cache data of the current Flow
    GetCacheData(key string) interface{}
    // SetCacheData sets the cache data of the current Flow
    SetCacheData(key string, value interface{}, Exp time.Duration)
}

SetCacheData() sets the local cache, with Exp as the expiration time. If Exp is 0, it is permanent.
GetCacheData() reads the local cache.

(2) Providing Constants

Provide some constants related to cache expiration time.

kis-flow/common/const.go

// cache
const (
    // DeFaultFlowCacheCleanUp is the default cache cleanup interval for Flow objects in KisFlow, in minutes
    DeFaultFlowCacheCleanUp = 5 // in minutes
    // DefaultExpiration is the default GoCache time, permanently saved
    DefaultExpiration time.Duration = 0
)

(3) Adding and Initializing Members in KisFlow

kis-flow/flow/kis_flow.go

// KisFlow represents the context environment for stream computing
type KisFlow struct {

    // ... ...
    // ... ...

    // Local cache for the flow
    cache *cache.Cache // Temporary cache context environment for Flow
}

// NewKisFlow creates a new KisFlow
func NewKisFlow(conf *config.KisFlowConfig) kis.Flow {
    flow := new(KisFlow)

    // ... ...
    // ... ...

    // Initialize local cache
    flow.cache = cache.New(cache.NoExpiration, common.DeFaultFlowCacheCleanUp*time.Minute)

    return flow
}

(4) Implementing the Interface

Finally, implement the two interfaces for cache read and write operations as follows:

kis-flow/flow/kis_flow_data.go

func (flow *KisFlow) GetCacheData(key string) interface{} {
    if data, found := flow.cache.Get(key); found {
        return data
    }
    return nil
}

func (flow *KisFlow) SetCacheData(key string, value interface{}, Exp time.Duration) {
    if Exp == common.DefaultExpiration {
        flow.cache.Set(key, value, cache.DefaultExpiration)
    } else {
        flow.cache.Set(key, value, Exp)
    }
}

8.2 MetaData Temporary Cache Parameters

8.2.1 Adding MetaData to Flow

First, add the metaData map[string]interface{} member and corresponding read-write lock to KisFlow.

kis-flow/flow/kis_flow.go

// KisFlow represents the context environment throughout the entire stream computing
type KisFlow struct {
    // ... ...

    // ... ...

    // +++++++++++++++++++++++++++++++++++++++++++
    // metaData for the flow
    metaData map[string]interface{} // Custom temporary data for Flow
    mLock    sync.RWMutex           // Read-write lock to manage metaData
}

Also, initialize the metaData member in the KisFlow constructor as follows:

kis-flow/flow/kis_flow.go

// NewKisFlow creates a KisFlow
func NewKisFlow(conf *config.KisFlowConfig) kis.Flow {
    flow := new(KisFlow)

    // ... ...
    // ... ...

    // ++++++++++++++++++++++++++++++++++++++
    // Initialize temporary data
    flow.metaData = make(map[string]interface{})

    return flow
}

Next, add the read and write interfaces for MetaData to the Flow as follows:

kis-flow/kis/flow.go

type Flow interface {
    // Run schedules the Flow, sequentially scheduling and executing Functions within the Flow
    Run(ctx context.Context) error
    // Link connects Functions within the Flow according to the configuration file
    Link(fConf *config.KisFuncConfig, fParams config.FParam) error
    // CommitRow submits Flow data to the Function layer about to be executed
    CommitRow(row interface{}) error
    // Input gets the input source data for the currently executing Function in the Flow
    Input() common.KisRowArr
    // GetName gets the name of the Flow
    GetName() string
    // GetThisFunction gets the currently executing Function
    GetThisFunction() Function
    // GetThisFuncConf gets the configuration of the currently executing Function
    GetThisFuncConf() *config.KisFuncConfig
    // GetConnector gets the Connector of the currently executing Function
    GetConnector() (Connector, error)
    // GetConnConf gets the configuration of the Connector for the currently executing Function
    GetConnConf() (*config.KisConnConfig, error)
    // GetConfig gets the configuration of the current Flow
    GetConfig() *config.KisFlowConfig
    // GetFuncConfigByName gets the configuration of the Function by its name
    GetFuncConfigByName(funcName string) *config.KisFuncConfig
    // Next advances the currently executing Function to the next Function with specified Action
    Next(acts ...ActionFunc) error
    // GetCacheData gets the cache data of the current Flow
    GetCacheData(key string) interface{}
    // SetCacheData sets the cache data of the current Flow
    SetCacheData(key string, value interface{}, Exp time.Duration)

    // ++++++++++++++++++++++++++++
    // GetMetaData gets the temporary data of the current Flow
    GetMetaData(key string) interface{}
    // SetMetaData sets the temporary data of the current Flow
    SetMetaData(key string, value interface{})
}

Define the GetMetaData() and SetMetaData() interfaces for reading and writing respectively. Finally, implement these interfaces as follows:

kis-flow/flow/kis_flow_data.go

// GetMetaData retrieves the temporary data of the current Flow object
func (flow *KisFlow) GetMetaData(key string) interface{} {
    flow.mLock.RLock()
    defer flow.mLock.RUnlock()

    data, ok := flow.metaData[key]
    if !ok {
        return nil
    }

    return data
}

// SetMetaData sets the temporary data of the current Flow object
func (flow *KisFlow) SetMetaData(key string, value interface{}) {
    flow.mLock.Lock()
    defer flow.mLock.Unlock()

    flow.metaData[key] = value
}

8.2.2 Adding MetaData to Function

First, add the metaData member to BaseFunction as follows:

kis-flow/function/kis_base_function.go

type BaseFunction struct {
    // Id, KisFunction instance ID, used to distinguish different instance objects within KisFlow
    Id     string
    Config *config.KisFuncConfig

    // flow
    flow kis.Flow // Context environment KisFlow

    // connector
    connector kis.Connector

    // ++++++++++++++++++++++++
    // Custom temporary data for Function
    metaData map[string]interface{}
    // Read-write lock to manage metaData
    mLock sync.RWMutex

    // link
    N kis.Function // Next stream computing Function
    P kis.Function // Previous stream computing Function
}

In the Function constructor, each specific Function needs a constructor to initialize the metaData member. The changes are as follows:

kis-flow/function/kis_base_function.go

func NewKisFunction(flow kis.Flow, config *config.KisFuncConfig) kis.Function {
    var f kis.Function

    // Factory produces generalized objects
    // ++++++++++++++
    switch common.KisMode(config.FMode) {
    case common.V:
        f = NewKisFunctionV() // +++
    case common.S:
        f = NewKisFunctionS() // +++
    case common.L:
        f = NewKisFunctionL() // +++
    case common.C:
        f = NewKisFunctionC() // +++
    case common.E:
        f = NewKisFunctionE() // +++
    default:
        // LOG ERROR
        return nil
    }

    // Generate random unique instance ID
    f.CreateId()

    // Set basic information properties
    if err := f.SetConfig(config); err != nil {
        panic(err)
    }

    // Set Flow
    if err := f.SetFlow(flow); err != nil {
        panic(err)
    }

    return f
}

Each constructor is as follows:

kis-flow/function/kis_function_c.go

func NewKisFunctionC() kis.Function {
    f := new(KisFunctionC)

    // Initialize metaData
    f.metaData = make(map[string]interface{})

    return f
}

kis-flow/function/kis_function_v.go

func NewKisFunctionV() kis.Function {
    f := new(KisFunctionV)

    // Initialize metaData
    f.metaData = make(map[string]interface{})

    return f
}

kis-flow/function/kis_function_e.go

func NewKisFunctionE() kis.Function {
    f := new(KisFunctionE)

    // Initialize metaData
    f.metaData = make(map[string]interface{})

    return f
}

kis-flow/function/kis_function_s.go

func NewKisFunctionS() kis.Function {
    f := new(KisFunctionS)

    // Initialize metaData
    f.metaData = make(map[string]interface{})

    return f
}

kis-flow/function/kis_function_l.go

func NewKisFunctionL() kis.Function {
    f := new(KisFunctionL)

    // Initialize metaData
    f.metaData = make(map[string]interface{})

    return f
}

Next, add interfaces to access the metaData member in the Function abstraction layer as follows:

type Function interface {
    // Call executes the stream computing logic
    Call(ctx context.Context, flow Flow) error

    // SetConfig configures the strategy for the current Function instance
    SetConfig(s *config.KisFuncConfig) error
    // GetConfig retrieves the configuration strategy of the current Function instance
    GetConfig() *config.KisFuncConfig

    // SetFlow sets the Flow instance that the current Function instance depends on
    SetFlow(f Flow) error
    // GetFlow retrieves the Flow instance that the current Function instance depends on
    GetFlow() Flow

    // AddConnector adds a Connector to the current Function instance
    AddConnector(conn Connector) error
    // GetConnector retrieves the Connector associated with the current Function instance
    GetConnector() Connector

    // CreateId generates a random instance KisID for the current Function instance
    CreateId()
    // GetId retrieves the FID of the current Function
    GetId() string
    // GetPrevId retrieves the FID of the previous Function node of the current Function
    GetPrevId() string
    // GetNextId retrieves the FID of the next Function node of the current Function
    GetNextId() string

    // Next returns the next layer computing stream Function, or nil if it is the last layer
    Next() Function
    // Prev returns the previous layer computing stream Function, or nil if it is the last layer
    Prev() Function
    // SetN sets the next Function instance
    SetN(f Function)
    // SetP sets the previous Function instance
    SetP(f Function)

    // ++++++++++++++++++++++++++++++++++
    // GetMetaData retrieves the temporary data of the current Function
    GetMetaData(key string) interface{}
    // SetMetaData sets the temporary data of the current Function
    SetMetaData(key string, value interface{})
}

Implement the above two interfaces in the BaseFunction.

kis-flow/function/kis_base_function.go

// GetMetaData retrieves the temporary data of the current Function
func (base *BaseFunction) GetMetaData(key string) interface{} {
    base.mLock.RLock()
    defer base.mLock.RUnlock()

    data, ok := base.metaData[key]
    if !ok {
        return nil
    }

    return data
}

// SetMetaData sets the temporary data of the current Function
func (base *BaseFunction) SetMetaData(key string, value interface{}) {
    base.mLock.Lock()
    defer base.mLock.Unlock()

    base.metaData[key] = value
}

8.2.3 Adding MetaData to Connector

First, add the metaData member to KisConnector as follows:

kis-flow/conn/kis_connector.go

type KisConnector struct {
    // Connector ID
    CId string
    // Connector Name
    CName string
    // Connector Config
    Conf *config.KisConnConfig
    // Connector Init
    onceInit sync.Once

    // ++++++++++++++
    // Custom temporary data for KisConnector
    metaData map[string]interface{}
    // Read-write lock to manage metaData
    mLock sync.RWMutex
}

// NewKisConnector creates a KisConnector based on the configuration strategy
func NewKisConnector(config *config.KisConnConfig) *KisConnector {
    conn := new(KisConnector)
    conn.CId = id.KisID(common.KisIdTypeConnector)
    conn.CName = config.CName
    conn.Conf = config

    // +++++++++++++++++++++++++++++++++++
    conn.metaData = make(map[string]interface{})

    return conn
}

Initialize metaData in the constructor.

Next, add interfaces to access and set MetaData in the Connector abstraction layer as follows:

kis-flow/kis/connector.go

type Connector interface {
    // Init initializes the links of the storage engine associated with the Connector
    Init() error
    // Call invokes the read and write operations of the external storage logic of the Connector
    Call(ctx context.Context, flow Flow, args interface{}) error
    // GetId retrieves the ID of the Connector
    GetId() string
    // GetName retrieves the name of the Connector
    GetName() string
    // GetConfig retrieves the configuration information of the Connector
    GetConfig() *config.KisConnConfig
    // GetMetaData retrieves the temporary data of the current Connector

    // +++++++++++++++++++++++++++++++
    GetMetaData(key string) interface{}
    // SetMetaData sets the temporary data of the current Connector
    SetMetaData(key string, value interface{})
}

Finally, implement the above two interfaces in KisConnector as follows:

kis-flow/conn/kis_connector.go

// GetMetaData retrieves the temporary data of the current Connector
func (conn *KisConnector) GetMetaData(key string) interface{} {
    conn.mLock.RLock()
    defer conn.mLock.RUnlock()

    data, ok := conn.metaData[key]
    if !ok {
        return nil
    }

    return data
}

// SetMetaData sets the temporary data of the current Connector
func (conn *KisConnector) SetMetaData(key string, value interface{}) {
    conn.mLock.Lock()
    defer conn.mLock.Unlock()

    conn.metaData[key] = value
}

8.3 Configuration File Parameters

KisFlow allows developers to define default parameters (Params) for configuring Flow, Function, Connector, etc., in the configuration file. Here are some examples:

Function:

kistype: func
fname: funcName1
fmode: Verify
source:
  name: Official Account Douyin Mall Order Data
  must:
    - order_id
    - user_id
option:
  default_params:
    default1: funcName1_param1
    default2: funcName1_param2

Flow:

kistype: flow
status: 1
flow_name: flowName1
flows:
  - fname: funcName1
    params:
      myKey1: flowValue1-1
      myKey2: flowValue1-2
  - fname: funcName2
    params:
      myKey1: flowValue2-1
      myKey2: flowValue2-2
  - fname: funcName3
    params:
      myKey1: flowValue3-1
      myKey2: flowValue3-2

Connector:

kistype: conn
cname: ConnName1
addrs: '0.0.0.0:9988,0.0.0.0:9999,0.0.0.0:9990'
type: redis
key: redis-key
params:
  args1: value1
  args2: value2
load: null
save:
  - funcName2

Developers can provide Params for each defined module. Params provided in Flow will also be added to the Functions.

In the previous steps, we already read these parameters into each module's memory, but we did not expose an interface for developers.

8.3.1 Adding Param Retrieval Interface to Flow

First, we provide an interface for Flow to query Params:

kis-flow/kis/flow.go

type Flow interface {
    // ... ...
    // ... ...

    // GetFuncParam retrieves a key-value pair of the default parameters for the currently executing Function in the Flow
    GetFuncParam(key string) string
    // GetFuncParamAll retrieves all key-value pairs of the default parameters for the currently executing Function in the Flow
    GetFuncParamAll() config.FParam
}

Implementation:

kis-flow/flow/kis_flow_data.go

// GetFuncParam retrieves a key-value pair of the default parameters for the currently executing Function in the Flow
func (flow *KisFlow) GetFuncParam(key string) string {
    flow.fplock.RLock()
    defer flow.fplock.RUnlock()

    if param, ok := flow.funcParams[flow.ThisFunctionId]; ok {
        if value, vok := param[key]; vok {
            return value
        }
    }

    return ""
}

// GetFuncParamAll retrieves all key-value pairs of the default parameters for the currently executing Function in the Flow
func (flow *KisFlow) GetFuncParamAll() config.FParam {
    flow.fplock.RLock()
    defer flow.fplock.RUnlock()

    param, ok := flow.funcParams[flow.ThisFunctionId]
    if !ok {
        return nil
    }

    return param
}

GetFuncParam() and GetFuncParamAll() retrieve a single key or all parameters respectively, but both fetch the Params for the currently executing Function.

8.3.2 Unit Testing

We add some parameters to each Function in flowName1.

kis-flow/test/load_conf/flow-FlowName1.yml

kistype: flow
status: 1
flow_name: flowName1
flows:
  - fname: funcName1
    params:
      myKey1: flowValue1-1
      myKey2: flowValue1-2
  - fname: funcName2
    params:
      myKey1: flowValue2-1
      myKey2: flowValue2-2
  - fname: funcName3
    params:
      myKey1: flowValue3-1
      myKey2: flowValue3-2

Then configure some default custom parameters for each associated Function:

kis-flow/test/load_conf/func/func-FuncName1.yml

kistype: func
fname: funcName1
fmode: Verify
source:
  name: Official Account Douyin Mall Order Data
  must:
    - order_id
    - user_id
option:
  default_params:
    default1: funcName1_param1
    default2: funcName1_param2

kis-flow/test/load_conf/func/func-FuncName2.yml

kistype: func
fname: funcName2
fmode: Save
source:
  name: User Order Error Rate
  must:
    - order_id
    - user_id
option:
  cname: ConnName1
  default_params:
    default1: funcName2_param1
    default2: funcName2_param2

kis-flow/test/load_conf/func/func-FuncName3.yml

kistype: func
fname: funcName3
fmode: Calculate
source:
  name: User Order Error Rate
  must:
    - order_id
    - user_id
option:
  default_params:
    default1: funcName3_param1
    default2: funcName3_param2

We also configure some Param parameters for the Connector associated with FuncName2:

kis-flow/test/load_conf/conn/conn-ConnName1.yml

kistype: conn
cname: ConnName1
addrs: '0.0.0.0:9988,0.0.0.0:9999,0.0.0.0:9990'
type: redis
key: redis-key
params:
  args1: value1
  args2: value2
load: null
save:
  - funcName2

To verify that our configuration parameters can be correctly retrieved during the execution of Functions, we modified each Function and Connector business function to print their Params:

kis-flow/test/faas/faas_demo1.go

func FuncDemo1Handler(ctx context.Context, flow kis.Flow) error {
    fmt.Println("---> Call funcName1Handler ----")

    // ++++++++++++++++
    fmt.Printf("Params = %+v\n", flow.GetFuncParamAll())

    for index, row := range flow.Input() {
        // Print data
        str := fmt.Sprintf("In FuncName = %s, FuncId = %s, row = %s", flow.GetThisFuncConf().FName, flow.GetThisFunction().GetId(), row)
        fmt.Println(str)

        // Calculate result data
        resultStr := fmt.Sprintf("data from funcName[%s], index = %d", flow.GetThisFuncConf().FName, index)

        // Commit result data
        _ = flow.CommitRow(resultStr)
    }

    return nil
}

kis-flow/test/faas/faas_demo2.go

func FuncDemo2Handler(ctx context.Context, flow kis.Flow) error {
    fmt.Println("---> Call funcName2Handler ----")

    // ++++++++++++++++
    fmt.Printf("Params = %+v\n", flow.GetFuncParamAll())

    for index, row := range flow.Input() {
        str := fmt.Sprintf("In FuncName = %s, FuncId = %s, row = %s", flow.GetThisFuncConf().FName, flow.GetThisFunction().GetId(), row)
        fmt.Println(str)

        conn, err := flow.GetConnector()
        if err != nil {
            log.Logger().ErrorFX(ctx, "FuncDemo2Handler(): GetConnector err = %s\n", err.Error())
            return err
        }

        if conn.Call(ctx, flow, row) != nil {
            log.Logger().ErrorFX(ctx, "FuncDemo2Handler(): Call err = %s\n", err.Error())
            return err
        }

        // Calculate result data
        resultStr := fmt.Sprintf("data from funcName[%s], index = %d", flow.GetThisFuncConf().FName, index)

        // Commit result data
        _ = flow.CommitRow(resultStr)
    }

    return nil
}

kis-flow/test/faas/faas_demo3.go

func FuncDemo3Handler(ctx context.Context, flow kis.Flow) error {
    fmt.Println("---> Call funcName3Handler ----")

    // ++++++++++++++++
    fmt.Printf("Params = %+v\n", flow.GetFuncParamAll())

    for _, row := range flow.Input() {
        str := fmt.Sprintf("In FuncName = %s, FuncId = %s, row = %s", flow.GetThisFuncConf().FName, flow.GetThisFunction().GetId(), row)
        fmt.Println(str)
    }

    return nil
}

kis-flow/test/caas/caas_demo1.go

func CaasDemoHanler1(ctx context.Context, conn kis.Connector, fn kis.Function, flow kis.Flow, args interface{}) error {
    fmt.Printf("===> In CaasDemoHanler1: flowName: %s, cName:%s, fnName:%s, mode:%s\n",
        flow.GetName(), conn.GetName(), fn.GetConfig().FName, fn.GetConfig().FMode)

    // +++++++++++ 
    fmt.Printf("Params = %+v\n", conn.GetConfig().Params)

    fmt.Printf("===> Call Connector CaasDemoHanler1, args from funciton: %s\n", args)

    return nil
}

Finally, we write the unit test cases:

kis-flow/test/kis_params_test.go

package test

import (
    "context"
    "kis-flow/common"
    "kis-flow/file"
    "kis-flow/kis"
    "kis-flow/test/caas"
    "kis-flow/test/faas"
    "testing"
)

func TestParams(t *testing.T) {
    ctx := context.Background()

    // 0. Register Function callback businesses
    kis.Pool().FaaS("funcName1", faas.FuncDemo1Handler)
    kis.Pool().FaaS("funcName2", faas.FuncDemo2Handler)
    kis.Pool().FaaS("funcName3", faas.FuncDemo3Handler)

    // 0. Register ConnectorInit and Connector callback businesses
    kis.Pool().CaaSInit("ConnName1", caas.InitConnDemo1)
    kis.Pool().CaaS("ConnName1", "funcName2", common.S, caas.CaasDemoHanler1)

    // 1. Load configuration files and build Flow
    if err := file.ConfigImportYaml("/Users/tal/gopath/src/kis-flow/test/load_conf/"); err != nil {
        panic(err)
    }

    // 2. Get Flow
    flow1 := kis.Pool().GetFlow("flowName1")

    // 3. Submit original data
    _ = flow1.CommitRow("This is Data1 from Test")
    _ = flow1.CommitRow("This is Data2 from Test")
    _ = flow1.CommitRow("This is Data3 from Test")

    // 4. Execute flow1
    if err := flow1.Run(ctx); err != nil {
        panic(err)
    }
}

Navigate to the kis-flow/test/ directory and execute:

go test -test.v -test.paniconexit0 -test.run  TestParams

=== RUN   TestParams
....
....

---> Call funcName1Handler ----
Params = map[default1:funcName1_param1 default2:funcName1_param2 myKey1:flowValue1-1 myKey2:flowValue1-2]
...
...

---> Call funcName2Handler ----
Params = map[default1:funcName2_param1 default2:funcName2_param2 myKey1:flowValue2-1 myKey2:flowValue2-2]
...
...
===> In CaasDemoHanler1: flowName: flowName1, cName:ConnName1, fnName:funcName2, mode:Save
Params = map[args1:value1 args2:value2]
...
...
===> In CaasDemoHanler1: flowName: flowName1, cName:ConnName1, fnName:funcName2, mode:Save
Params = map[args1:value1 args2:value2]
...
...

===> In CaasDemoHanler1: flowName: flowName1, cName:ConnName1, fnName:funcName2, mode:Save
Params = map[args1:value1 args2:value2]

...
...

---> Call funcName3Handler ----
Params = map[default1:funcName3_param1 default2:funcName3_param2 myKey1:flowValue3-1 myKey2:flowValue3-2]

...
...
--- PASS: TestParams (0.01s)
PASS
ok      kis-flow/test   0.433s

As we can see, we can now correctly retrieve the Params configuration parameters at each level.

8.4 [V0.7] Source Code

https://github.com/aceld/kis-flow/releases/tag/v0.7

Author: Aceld
GitHub: https://github.com/aceld

KisFlow Open Source Project Address: https://github.com/aceld/kis-flow

Document: https://github.com/aceld/kis-flow/wiki

Case1-Quick Start
Case2-Flow Parallel Operation
Case3-Application of KisFlow in Multi-Goroutines
Case4-KisFlow in Message Queue (MQ) Applications

(Part 7)Golang Framework Hands-on - KisFlow Stream Computing Framework-KisFlow Action

Aceld — Mon, 03 Jun 2024 11:35:52 +0000

Github: https://github.com/aceld/kis-flow
Document: https://github.com/aceld/kis-flow/wiki

Case1-Quick Start
Case2-Flow Parallel Operation
Case3-Application of KisFlow in Multi-Goroutines
Case4-KisFlow in Message Queue (MQ) Applications

7.1 Action Abort

KisFlow Action refers to controlling the flow's scheduling logic while executing a Function. KisFlow provides some Action options for developers to choose from. This section introduces the simplest Action, Abort, which terminates the current Flow.

The final usage of Abort is as follows:

func AbortFuncHandler(ctx context.Context, flow kis.Flow) error {
    fmt.Println("---> Call AbortFuncHandler ----")

    for _, row := range flow.Input() {
        str := fmt.Sprintf("In FuncName = %s, FuncId = %s, row = %s", flow.GetThisFuncConf().FName, flow.GetThisFunction().GetId(), row)
        fmt.Println(str)
    }

    return flow.Next(kis.ActionAbort)  // Terminate the Flow
}

AbortFuncHandler() is a business callback method of a Function, defined by the developer. After the current Function is executed, the normal situation is to continue to the next Function. However, if flow.Next(kis.ActionAbort) is returned as the current Function's return value, the next Function will not be executed. Instead, the scheduling computation flow of the current Flow is directly terminated.

Let's implement the Abort Action mode of KisFlow below.

7.1.1 Abort Interface Definition

First, let's define the Abort() interface for the Flow.

kis-flow/kis/flow.go

type Flow interface {
    // Run schedules the Flow, sequentially scheduling and executing Functions in the Flow
    Run(ctx context.Context) error
    // Link connects the Functions in the Flow according to the configuration file
    Link(fConf *config.KisFuncConfig, fParams config.FParam) error
    // CommitRow submits Flow data to the Function layer about to be executed
    CommitRow(row interface{}) error
    // Input obtains the input source data of the currently executing Function in the Flow
    Input() common.KisRowArr
    // GetName retrieves the name of the Flow
    GetName() string
    // GetThisFunction obtains the currently executing Function
    GetThisFunction() Function
    // GetThisFuncConf obtains the configuration of the currently executing Function
    GetThisFuncConf() *config.KisFuncConfig
    // GetConnector obtains the Connector of the currently executing Function
    GetConnector() (Connector, error)
    // GetConnConf obtains the configuration of the Connector of the currently executing Function
    GetConnConf() (*config.KisConnConfig, error)
    // GetConfig obtains the configuration of the current Flow
    GetConfig() *config.KisFlowConfig
    // GetFuncConfigByName retrieves the configuration of the specified Function in the Flow
    GetFuncConfigByName(funcName string) *config.KisFuncConfig

    // --- KisFlow Action ---
    // Next proceeds to the next Function in the Flow with the specified Action
    Next(acts ...ActionFunc) error
}

Here, an interface Next(acts ...ActionFunc) error is provided, where the parameter is a variadic parameter of type ActionFunc. This is the method related to Actions that we define for KisFlow. The module for the definition of Action is as follows:

7.1.2 Action Module Definition

Action is a configuration module used to control special behaviors in the Flow execution process through Functions. This includes the Abort behavior mentioned above, which is one of the Actions. The module definition for Action is as follows. Create a file action.go under kis-flow/kis/ and implement it:

kis-flow/kis/action.go

package kis

// Action represents the Actions to be taken during the execution of KisFlow
type Action struct {
    // Abort indicates whether to terminate the Flow execution
    Abort bool
}

// ActionFunc is a type for KisFlow Functional Option
type ActionFunc func(ops *Action)

// LoadActions loads Actions and sequentially executes the ActionFunc functions
func LoadActions(acts []ActionFunc) Action {
    action := Action{}

    if acts == nil {
        return action
    }

    for _, act := range acts {
        act(&action)
    }

    return action
}

// ActionAbort sets the Action to terminate the Flow execution
func ActionAbort(action *Action) {
    action.Abort = true
}

First, currently, Action has only one behavior, Abort, which is represented by a boolean type. If true, it indicates that the Flow needs to be terminated.

Next, type ActionFunc func(ops *Action) is a function type where the parameter is a pointer to an Action{}. The function func ActionAbort(action *Action) is a specific instance of this function type. The purpose of the ActionAbort() method is to set the Abort member of the Action struct to true.

Finally, let's look at the func LoadActions(acts []ActionFunc) Action method. The parameter is an array of ActionFunc functions. LoadActions() creates a new Action{}, then sequentially executes the functions in the []ActionFunc array to modify the members of Action{}. It finally returns the modified Action{} to the upper layer.

7.1.3 Implementation of the Next Method

Next, we need to implement this interface for the KisFlow module. First, we need to add an Action{} member to KisFlow, indicating the action to be taken after each Function execution.

kis-flow/flow/kis_flow.go

// KisFlow provides the context for the entire streaming computation
type KisFlow struct {
    // Basic information
    Id   string                // Distributed instance ID of the Flow (used internally in KisFlow to distinguish different instances)
    Name string                // Readable name of the Flow
    Conf *config.KisFlowConfig // Flow configuration strategy

    // List of Functions
    Funcs          map[string]kis.Function // All Function objects managed by the current flow, key: FunctionName
    FlowHead       kis.Function            // Head of the Function list owned by the current Flow
    FlowTail       kis.Function            // Tail of the Function list owned by the current Flow
    flock          sync.RWMutex            // Lock for managing list insertion and reading/writing
    ThisFunction   kis.Function            // The KisFunction object currently being executed by the Flow
    ThisFunctionId string                  // ID of the currently executing Function
    PrevFunctionId string                  // ID of the previous Function executed

    // Function list parameters
    funcParams map[string]config.FParam // Custom fixed configuration parameters of the current Function in the flow, key: Function instance KisID, value: FParam
    fplock     sync.RWMutex             // Lock for managing reading/writing of funcParams

    // Data
    buffer common.KisRowArr  // Internal buffer for temporarily storing input byte data, a single piece of data is an interface{}, multiple pieces of data are []interface{}, i.e., KisBatch
    data   common.KisDataMap // Data source for each layer of streaming computation
    inPut  common.KisRowArr  // Input data for the currently executing Function

    // +++++++++++++++++++++

    // KisFlow Action
    action kis.Action        // Action to be taken by the current Flow
}

Then implement the Next() interface for KisFlow as follows:

kis-flow/flow/kis_flow.go

// Next proceeds to the next Function in the Flow with the specified Action
func (flow *KisFlow) Next(acts ...kis.ActionFunc) error {

    // Load Actions passed by the Function FaaS
    flow.action = kis.LoadActions(acts)

    return nil
}

Each time a developer executes a custom business callback in a Function, flow.Next() is called to pass the Action at the end. Therefore, Next(acts ...kis.ActionFunc) error loads the passed Action properties and saves them in flow.action.

7.1.4 Abort to Control Flow Execution

Now that we have an Abort action to control the Flow, we need to add a member to KisFlow to represent this state.

kis-flow/flow/kis_flow.go

// KisFlow provides the context for the entire streaming computation
type KisFlow struct {
    // Basic information
    Id   string                // Distributed instance ID of the Flow (used internally in KisFlow to distinguish different instances)
    Name string                // Readable name of the Flow
    Conf *config.KisFlowConfig // Flow configuration strategy

    // List of Functions
    Funcs          map[string]kis.Function // All Function objects managed by the current flow, key: FunctionName
    FlowHead       kis.Function            // Head of the Function list owned by the current Flow
    FlowTail       kis.Function            // Tail of the Function list owned by the current Flow
    flock          sync.RWMutex            // Lock for managing list insertion and reading/writing
    ThisFunction   kis.Function            // The KisFunction object currently being executed by the Flow
    ThisFunctionId string                  // ID of the currently executing Function
    PrevFunctionId string                  // ID of the previous Function executed

    // Function list parameters
    funcParams map[string]config.FParam // Custom fixed configuration parameters of the current Function in the flow, key: Function instance KisID, value: FParam
    fplock     sync.RWMutex             // Lock for managing reading/writing of funcParams

    // Data
    buffer common.KisRowArr  // Internal buffer for temporarily storing input byte data, a single piece of data is an interface{}, multiple pieces of data are []interface{}, i.e., KisBatch
    data   common.KisDataMap // Data source for each layer of streaming computation
    inPut  common.KisRowArr  // Input data for the currently executing Function
    action kis.Action        // Action to be taken by the current Flow

    // +++++++++
    abort  bool              // Indicates whether to abort the Flow
}

Each time the flow.Run() method is executed, the abort variable needs to be reset. Additionally, the loop scheduling needs to check the flow.abort status.

kis-flow/flow/kis_flow.go

// Run starts the streaming computation of KisFlow, executing the stream from the starting Function
func (flow *KisFlow) Run(ctx context.Context) error {

    // +++++++++
    // Reset abort
    flow.abort = false  // Reset the abort state each time scheduling starts

    // ... ...

    // ... ...

    // Stream chain call
    for fn != nil && flow.abort != true { // ++++ Do not enter the next loop if abort is set

        // ... ...
        // ... ...

        if err := fn.Call(ctx, flow); err != nil {
            // Error
            return err
        } else {
            // Success

            // ... ...

            fn = fn.Next()
        }
    }

    return nil
}

When Call() schedules the custom method of the Function, if return flow.Next(ActionAbort) is called, it will change the Action state of the Flow, thereby controlling the termination of the Flow execution. Finally, the Abort state of the Action is transferred to the Abort state of KisFlow.

Since we have the Abort state, we can add a condition during the Flow execution. If the current Function does not submit its result data (i.e., flow.buffer is empty), the Flow will not proceed to the next layer and will directly exit the Run() call.

kis-flow/flow/kis_flow_data.go

// commitCurData submits the result data of the currently executing Function in the Flow
func (flow *KisFlow) commitCurData(ctx context.Context) error {

    // Check if there is result data for the current computation; if not, exit the current Flow Run loop
    if (len(flow.buffer) == 0) {
        // ++++++++++++
        flow.abort = true
        return nil
    }

    // ... ...
    // ... ...

    return nil
}

7.1.5 Capturing and Handling Actions

Next, we will implement a method specifically for handling Actions. This method will be defined in the kis-flow/flow/kis_flow_action.go file as follows:

kis-flow/flow/kis_flow_action.go

package flow

import (
    "context"
    "errors"
    "fmt"
    "kis-flow/kis"
)

// dealAction handles Actions and decides the subsequent flow direction
func (flow *KisFlow) dealAction(ctx context.Context, fn kis.Function) (kis.Function, error) {

    if err := flow.commitCurData(ctx); err != nil {
        return nil, err
    }

    // Update the previous FunctionId cursor
    flow.PrevFunctionId = flow.ThisFunctionId
    fn = fn.Next()

    // Abort Action forces termination
    if flow.action.Abort {
        flow.abort = true
    }

    // Clear Action
    flow.action = kis.Action{}

    return fn, nil
}

Next, we'll slightly modify the KisFlow Run() process to incorporate the dealAction() method.

kis-flow/flow/kis_flow.go

// Run starts the streaming computation of KisFlow, executing the stream from the starting Function
func (flow *KisFlow) Run(ctx context.Context) error {

    var fn kis.Function

    fn = flow.FlowHead
    flow.abort = false

    if flow.Conf.Status == int(common.FlowDisable) {
        // Flow is configured to be disabled
        return nil
    }

    // Since no Function has been executed at this point, PrevFunctionId is FirstVirtual as there is no previous Function
    flow.PrevFunctionId = common.FunctionIdFirstVirtual

    // Submit the original data stream
    if err := flow.commitSrcData(ctx); err != nil {
        return err
    }

    // Stream chain call
    for fn != nil && flow.abort == false {

        // Flow records the current executing Function
        fid := fn.GetId()
        flow.ThisFunction = fn
        flow.ThisFunctionId = fid

        // Get the source data to be processed by the current Function
        if inputData, err := flow.getCurData(); err != nil {
            log.Logger().ErrorFX(ctx, "flow.Run(): getCurData err = %s\n", err.Error())
            return err
        } else {
            flow.inPut = inputData
        }

        if err := fn.Call(ctx, flow); err != nil {
            // Error
            return err
        } else {
            // Success

            // +++++++++++++++++++++++++++++++
            fn, err = flow.dealAction(ctx, fn)
            if err != nil {
                return err
            }
            // +++++++++++++++++++++++++++++++
        }
    }

    return nil
}

7.1.6 Action Abort Unit Test

First, let's create a Function configuration file as follows:

kis-flow/test/load_conf/func/func-AbortFunc.yml

kistype: func
fname: abortFunc
fmode: Calculate
source:
  name: 用户订单错误率
  must:
    - order_id
    - user_id

The name of the current Function is abortFunc. Next, we implement its FaaS function as follows:

kis-flow/test/faas/faas_abort.go

package faas

import (
    "context"
    "fmt"
    "kis-flow/kis"
)

// type FaaS func(context.Context, Flow) error

func AbortFuncHandler(ctx context.Context, flow kis.Flow) error {
    fmt.Println("---> Call AbortFuncHandler ----")

    for _, row := range flow.Input() {
        str := fmt.Sprintf("In FuncName = %s, FuncId = %s, row = %s", flow.GetThisFuncConf().FName, flow.GetThisFunction().GetId(), row)
        fmt.Println(str)
    }

    return flow.Next(kis.ActionAbort)
}

This Function will eventually call flow.Next(kis.ActionAbort) to terminate the Flow. Next, we create a Flow that uses the above Function as an intermediate Function to test if it will terminate before executing subsequent Functions. The new flow configuration is as follows:

kis-flow/test/load_conf/flow/flow-FlowName2.yml

kistype: flow
status: 1
flow_name: flowName2
flows:
  - fname: funcName1
  - fname: abortFunc
  - fname: funcName3

The name of the current Flow is flowName2, which contains three Functions: funcName1, abortFunc, and funcName3. If the abort functionality works correctly, funcName3 should not be executed.

Next, we implement the unit test case.

kis-flow/test/kis_action_test.go

package test

import (
    "context"
    "kis-flow/common"
    "kis-flow/file"
    "kis-flow/kis"
    "kis-flow/test/caas"
    "kis-flow/test/faas"
    "testing"
)

func TestActionAbort(t *testing.T) {
    ctx := context.Background()

    // 0. Register Function callbacks
    kis.Pool().FaaS("funcName1", faas.FuncDemo1Handler)
    kis.Pool().FaaS("abortFunc", faas.AbortFuncHandler) // Add abortFunc handler
    kis.Pool().FaaS("funcName3", faas.FuncDemo3Handler)

    // 0. Register ConnectorInit and Connector callbacks
    kis.Pool().CaaSInit("ConnName1", caas.InitConnDemo1)
    kis.Pool().CaaS("ConnName1", "funcName2", common.S, caas.CaasDemoHanler1)

    // 1. Load configuration files and build Flow
    if err := file.ConfigImportYaml("/Users/gopath/src/kis-flow/test/load_conf/"); err != nil {
        panic(err)
    }

    // 2. Get Flow
    flow1 := kis.Pool().GetFlow("flowName2")

    // 3. Submit original data
    _ = flow1.CommitRow("This is Data1 from Test")
    _ = flow1.CommitRow("This is Data2 from Test")
    _ = flow1.CommitRow("This is Data3 from Test")

    // 4. Execute flow1
    if err := flow1.Run(ctx); err != nil {
        panic(err)
    }
}

The following code registers the initial callbacks. You can also write this code in another file to avoid repeating it each time:

// 0. Register Function callbacks
kis.Pool().FaaS("funcName1", faas.FuncDemo1Handler)
kis.Pool().FaaS("abortFunc", faas.AbortFuncHandler) // Add abortFunc handler
kis.Pool().FaaS("funcName3", faas.FuncDemo3Handler)

// 0. Register ConnectorInit and Connector callbacks
kis.Pool().CaaSInit("ConnName1", caas.InitConnDemo1)
kis.Pool().CaaS("ConnName1", "funcName2", common.S, caas.CaasDemoHanler1)

Change to the kis-flow/test/ directory and run the following command:

go test -test.v -test.paniconexit0 -test.run  TestActionAbort

The result is as follows:

=== RUN   TestActionAbort
Add KisPool FuncName=funcName1
Add KisPool FuncName=abortFunc
Add KisPool FuncName=funcName3
Add KisPool CaaSInit CName=ConnName1
Add KisPool CaaS CName=ConnName1, FName=funcName2, Mode =Save
===> Call Connector InitDemo1
&{conn ConnName1 0.0.0.0:9988,0.0.0.0:9999,0.0.0.0:9990 redis redis-key map[args1:value1 args2:value2] [] [funcName2 funcName2]}
Add FlowRouter FlowName=flowName1

Add FlowRouter FlowName=flowName2

context.Background
====> After CommitSrcData, flow_name = flowName2, flow_id = flow-b6b90eb4b7d7457fbf85b3299b625513
All Level Data =
 map[FunctionIdFirstVirtual:[This is Data1 from Test This is Data2 from Test This is Data3 from Test]]

KisFunctionV, flow = &{Id:flow-b6b90eb4b7d7457fbf85b3299b625513 Name:flowName2 Conf:0xc000092cc0 Funcs:map[abortFunc:0xc000094d20 funcName1:0xc000094cc0 funcName3:0xc000094d80] FlowHead:0xc000094cc0 FlowTail:0xc000094d80 flock:{w:{state:0 sema:0} writerSem:0 readerSem:0 readerCount:0 readerWait:0} ThisFunction:0xc000094cc0 ThisFunctionId:func-c435cf9f8e3346a1851f8c76375fce0f PrevFunctionId:FunctionIdFirstVirtual funcParams:map[func-7f5af1521fd64d08839d5bdd26de5254:map[] func-c435cf9f8e3346a1851f8c76375fce0f:map[] func-f0b80593fe2e4018a878f155b9c543b4:map[]] fplock:{w:{state:0 sema:0} writerSem:0 readerSem:0 readerCount:0 readerWait:0} buffer:[] data:map[FunctionIdFirstVirtual:[This is Data1 from Test This is Data2 from Test This is Data3 from Test]] inPut:[This is Data1 from Test This is Data2 from Test This is Data3 from Test] jumpFunc:NoJump abort:false nextOpt:<nil>}

---> Call funcName1Handler ----
In FuncName = funcName1, FuncId = func-c435cf9f8e3346a1851f8c76375fce0f, row = This is Data1 from Test
In FuncName = funcName1, FuncId = func-c435cf9f8e3346a1851f8c76375fce0f, row = This is Data2 from Test
In FuncName = funcName1, FuncId = func-c435cf9f8e3346a1851f8c76375fce0f, row = This is Data3 from Test
context.Background
====> After commitCurData, flow_name = flowName2, flow_id = flow-b6b90eb4b7d7457fbf85b3299b625513
All Level Data =
 map[FunctionIdFirstVirtual:[This is Data1 from Test This is Data2 from Test This is Data3 from Test] func-c435cf9f8e3346a1851f8c76375fce0f:[data from funcName[funcName1], index = 0 data from funcName[funcName1], index = 1 data from funcName[funcName1], index = 2]]

KisFunctionC, flow = &{Id:flow-b6b90eb4b7d7457fbf85b3299b625513 Name:flowName2 Conf:0xc000092cc0 Funcs:map[abortFunc:0xc000094d20 funcName1:0xc000094cc0 funcName3:0xc000094d80] FlowHead:0xc000094cc0 FlowTail:0xc000094d80 flock:{w:{state:0 sema:0} writerSem:0 readerSem:0 readerCount:0 readerWait:0} ThisFunction:0xc000094d20 ThisFunctionId:func-7f5af1521fd64d08839d5bdd26de5254 PrevFunctionId:func-c435cf9f8e3346a1851f8c76375fce0f funcParams:map[func-7f5af1521fd64d08839d5bdd26de5254:map[] func-c435cf9f8e3346a1851f8c76375fce0f:map[] func-f0b80593fe2e4018a878f155b9c543b4:map[]] fplock:{w:{state:0 sema:0} writerSem:0 readerSem:0 readerCount:0 readerWait:0} buffer:[] data:map[FunctionIdFirstVirtual:[This is Data1 from Test This is Data2 from Test This is Data3 from Test] func-c435cf9f8e3346a1851f8c76375fce0f:[data from funcName[funcName1], index = 0 data from funcName[funcName1], index = 1 data from funcName[funcName1], index = 2]] inPut:[data from funcName[funcName1], index = 0 data from funcName[funcName1], index = 1 data from funcName[funcName1], index = 2] jumpFunc:NoJump abort:false nextOpt:<nil>}

---> Call AbortFuncHandler ----
In FuncName = abortFunc, FuncId = func-7f5af1521fd64d08839d5bdd26de5254, row = data from funcName[funcName1], index = 0
In FuncName = abortFunc, FuncId = func-7f5af1521fd64d08839d5bdd26de5254, row = data from funcName[funcName1], index = 1
In FuncName = abortFunc, FuncId = func-7f5af1521fd64d08839d5bdd26de5254, row = data from funcName[funcName1], index = 2
--- PASS: TestActionAbort (0.00s)
PASS
ok      kis-flow/test   0.487s

From the result, we can see that after executing the AbortFuncHandler, it did not continue executing and exited the Flow's Run() method.

7.2 Action DataReuse (Reuse Upper-Level Data)

The Action DataReuse is designed to reuse data from the previous function, meaning the current function's submitted result will not be used. Instead, the result data from the previous function will be reused for the next function as its data source.

Let's implement the Action DataReuse functionality.

7.2.1 Adding DataReuse Action

Add a DataReuse member to the Action, which is of boolean type.

kis-flow/kis/action.go

// Action KisFlow execution process Actions
type Action struct {
    // +++++++++++++
    // DataReuse indicates whether to reuse upper-level function data
    DataReuse bool

    // Abort terminates the execution of the Flow
    Abort bool
}

// ActionDataReuse sets the DataReuse option to true
func ActionDataReuse(act *Action) {
    act.DataReuse = true
}

Then provide an Action function named ActionDataReuse, which sets the DataReuse status to true.

7.2.2 Reusing Upper-Level Data to the Next Layer

Here, we need to implement a method for submitting reused data. The logic is as follows:

kis-flow/flow/kis_flow_data.go

// commitReuseData submits reused data from the previous function
func (flow *KisFlow) commitReuseData(ctx context.Context) error {

    // Check if the previous layer has result data, if not, exit the current Flow Run loop
    if len(flow.data[flow.PrevFunctionId]) == 0 {
        flow.abort = true
        return nil
    }

    // The current layer's result data equals the previous layer's result data (reuse upper-level result data)
    flow.data[flow.ThisFunctionId] = flow.data[flow.PrevFunctionId]

    // Clear the buffer (if ReuseData is selected, all submitted data will not be carried to the next layer)
    flow.buffer = flow.buffer[0:0]

    log.Logger().DebugFX(ctx, " ====> After commitReuseData, flow_name = %s, flow_id = %s\nAll Level Data =\n %+v\n", flow.Name, flow.Id, flow.data)

    return nil
}

The logic is simple. Unlike commitCurData(), which submits flow.buffer data to flow.data[flow.ThisFunctionId], commitReuseData() submits the previous layer's result data to flow.data[flow.ThisFunctionId].

7.2.3 Handling the DataReuse Action

Then, add handling for the DataReuse action in the dealAction() method:

kis-flow/flow/kis_flow_action.go

// dealAction processes actions and determines the next steps for the Flow
func (flow *KisFlow) dealAction(ctx context.Context, fn kis.Function) (kis.Function, error) {

    // ++++++++++++++++
    // DataReuse Action
    if flow.action.DataReuse {
        if err := flow.commitReuseData(ctx); err != nil {
            return nil, err
        }
    } else {
        if err := flow.commitCurData(ctx); err != nil {
            return nil, err
        }
    }

    // Update the previous function ID cursor
    flow.PrevFunctionId = flow.ThisFunctionId
    fn = fn.Next()

    // Abort Action force termination
    if flow.action.Abort {
        flow.abort = true
    }

    // Clear the Action
    flow.action = kis.Action{}

    return fn, nil
}

This captures and processes the DataReuse action, deciding whether to reuse upper-level data or to commit current data based on the action settings.

7.2.4 Unit Testing

Let's proceed with unit testing for DataReuse. First, create a function named dataReuseFunc, and create its configuration file:

kis-flow/test/load_conf/func/func-dataReuseFunc.yml

kistype: func
fname: dataReuseFunc
fmode: Calculate
source:
  name: User Order Error Rate
  must:
    - order_id
    - user_id

Also, create a new Flow called flowName3 with the following configuration:

kis-flow/test/load_conf/flow/func-FlowName3.yml

kistype: flow
status: 1
flow_name: flowName3
flows:
  - fname: funcName1
  - fname: dataReuseFunc
  - fname: funcName3

For the logic of the dataReuseFunc function, here's the implementation:

kis-flow/test/faas/faas_data_reuse.go

package faas

import (
    "context"
    "fmt"
    "kis-flow/kis"
)

func DataReuseFuncHandler(ctx context.Context, flow kis.Flow) error {
    fmt.Println("---> Call DataReuseFuncHandler ----")

    for index, row := range flow.Input() {
        str := fmt.Sprintf("In FuncName = %s, FuncId = %s, row = %s", flow.GetThisFuncConf().FName, flow.GetThisFunction().GetId(), row)
        fmt.Println(str)

        // Calculate result data
        resultStr := fmt.Sprintf("data from funcName[%s], index = %d", flow.GetThisFuncConf().FName, index)

        // Commit result data
        _ = flow.CommitRow(resultStr)
    }

    return flow.Next(kis.ActionDataReuse)
}

Finally, implement the test case:

kis-flow/test/kis_action_test.go

func TestActionDataReuse(t *testing.T) {
    ctx := context.Background()

    // 0. Register Function callback business
    kis.Pool().FaaS("funcName1", faas.FuncDemo1Handler)
    kis.Pool().FaaS("dataReuseFunc", faas.DataReuseFuncHandler) // Adding dataReuseFunc business
    kis.Pool().FaaS("funcName3", faas.FuncDemo3Handler)

    // 0. Register ConnectorInit and Connector callback business
    kis.Pool().CaaSInit("ConnName1", caas.InitConnDemo1)
    kis.Pool().CaaS("ConnName1", "funcName2", common.S, caas.CaasDemoHanler1)

    // 1. Load configuration files and build Flow
    if err := file.ConfigImportYaml("/Users/tal/gopath/src/kis-flow/test/load_conf/"); err != nil {
        panic(err)
    }

    // 2. Get Flow
    flow1 := kis.Pool().GetFlow("flowName3")

    // 3. Submit raw data
    _ = flow1.CommitRow("This is Data1 from Test")
    _ = flow1.CommitRow("This is Data2 from Test")
    _ = flow1.CommitRow("This is Data3 from Test")

    // 4. Execute flow1
    if err := flow1.Run(ctx); err != nil {
        panic(err)
    }
}

Navigate to kis-flow/test/ and execute:

go test -test.v -test.paniconexit0 -test.run  TestActionDataReuse

=== RUN   TestActionDataReuse
Add KisPool FuncName=funcName1
Add KisPool FuncName=dataReuseFunc
Add KisPool FuncName=funcName3
Add KisPool CaaSInit CName=ConnName1
Add KisPool CaaS CName=ConnName1, FName=funcName2, Mode =Save
===> Call Connector InitDemo1
&{conn ConnName1 0.0.0.0:9988,0.0.0.0:9999,0.0.0.0:9990 redis redis-key map[args1:value1 args2:value2] [] [funcName2 funcName2]}
Add FlowRouter FlowName=flowName5
===> Call Connector InitDemo1
&{conn ConnName1 0.0.0.0:9988,0.0.0.0:9999,0.0.0.0:9990 redis redis-key map[args1:value1 args2:value2] [] [funcName2 funcName2 funcName2]}
Add FlowRouter FlowName=flowName1
Add FlowRouter FlowName=flowName2
Add FlowRouter FlowName=flowName3
Add FlowRouter FlowName=flowName4
context.Background
====> After CommitSrcData, flow_name = flowName3, flow_id = flow-2c1a23d9587842bebaeee490319de81f
All Level Data =
 map[FunctionIdFirstVirtual:[This is Data1 from Test This is Data2 from Test This is Data3 from Test]]

KisFunctionV, flow = &{Id:flow-2c1a23d9587842bebaeee490319de81f Name:flowName3 Conf:0xc000092dc0 Funcs:map[dataReuseFunc:0xc000095620 funcName1:0xc0000955c0 funcName3:0xc000095680] FlowHead:0xc0000955c0 FlowTail:0xc000095680 flock:{w:{state:0 sema:0} writerSem:0 readerSem:0 readerCount:0 readerWait:0} ThisFunction:0xc0000955c0 ThisFunctionId:func-7886178381634f05b302841141382e59 PrevFunctionId:FunctionIdFirstVirtual funcParams:map[func-7886178381634f05b302841141382e59:map[] func-cfe66e39aba54ff989d6764cc4edda20:map[] func-ef567879d0dd45b287ed709e549e9d32:map[]] fplock:{w:{state:0 sema:0} writerSem:0 readerSem:0 readerCount:0 readerWait:0} buffer:[] data:map[FunctionIdFirstVirtual:[This is Data1 from Test This is Data2 from Test This is Data3 from Test]] inPut:[This is Data1 from Test This is Data2 from Test This is Data3 from Test] abort:false action:{DataReuse:false ForceEntryNext:false JumpFunc: Abort:false}}

---> Call funcName1Handler ----
In FuncName = funcName1, FuncId = func-7886178381634f05b302841141382e59, row = This is Data1 from Test
In FuncName = funcName1, FuncId = func-7886178381634f05b302841141382e59, row = This is Data2 from Test
In FuncName = funcName1, FuncId = func-7886178381634f05b302841141382e59, row = This is Data3 from Test
context.Background

 ====> After commitCurData, flow_name = flowName3, flow_id = flow-2c1a23d9587842bebaeee490319de81f
All Level Data =
 map[FunctionIdFirstVirtual:[This is Data1 from Test This is Data2 from Test This is Data3 from Test] func-7886178381634f05b302841141382e59:[data from funcName[funcName1], index = 0 data from funcName[funcName1], index = 1 data from funcName[funcName1], index = 2]]

KisFunctionC, flow = &{Id:flow-2c1a23d9587842bebaeee490319de81f Name:flowName3 Conf:0xc000092dc0 Funcs:map[dataReuseFunc:0xc000095620 funcName1:0xc0000955c0 funcName3:0xc000095680] FlowHead:0xc0000955c0 FlowTail:0xc000095680 flock:{w:{state:0 sema:0} writerSem:0 readerSem:0 readerCount:0 readerWait:0} ThisFunction:0xc000095620 ThisFunctionId:func-ef567879d0dd45b287ed709e549e9d32 PrevFunctionId:func-7886178381634f05b302841141382e59 funcParams:map[func-7886178381634f05b302841141382e59:map[] func-cfe66e39aba54ff989d6764cc4edda20:map[] func-ef567879d0dd45b287ed709e549e9d32:map[]] fplock:{w:{state:0 sema:0} writerSem:0 readerSem:0 readerCount:0 readerWait:0} buffer:[] data:map[FunctionIdFirstVirtual:[This is Data1 from Test This is Data2 from Test This is Data3 from Test] func-7886178381634f05b302841141382e59:[data from funcName[funcName1], index = 0 data from funcName[funcName1], index = 1 data from funcName[funcName1], index = 2]] inPut:[data from funcName[funcName1], index = 0 data from funcName[funcName1], index = 1 data from funcName[funcName1], index = 2] abort:false action:{DataReuse:false ForceEntryNext:false JumpFunc: Abort:false}}

---> Call DataReuseFuncHandler ----
In FuncName = dataReuseFunc, FuncId = func-ef567879d0dd45b287ed709e549e9d32, row = data from funcName[funcName1], index = 0
In FuncName = dataReuseFunc, FuncId = func-ef567879d0dd45b287ed709e549e9d32, row = data from funcName[funcName1], index = 1
In FuncName = dataReuseFunc, FuncId = func-ef567879d0dd45b287ed709e549e9d32, row = data from funcName[funcName1], index = 2
context.Background
 ====> After commitReuseData, flow_name = flowName3, flow_id = flow-2c1a23d9587842bebaeee490319de81f
All Level Data =
 map[FunctionIdFirstVirtual:[This is Data1 from Test This is Data2 from Test This is Data3 from Test] func-7886178381634f05b302841141382e59:[data from funcName[funcName1], index = 0 data from funcName[funcName1], index = 1 data from funcName[funcName1], index = 2] func-ef567879d0dd45b287ed709e549e9d32:[data from funcName[funcName1], index = 0 data from funcName[funcName1], index = 1 data from funcName[funcName1], index = 2]]

KisFunctionC, flow = &{Id:flow-2c1a23d9587842bebaeee490319de81f Name:flowName3 Conf:0xc000092dc0 Funcs:map[dataReuseFunc:0xc000095620 funcName1:0xc0000955c0 funcName3:0xc000095680] FlowHead:0xc0000955c0 FlowTail:0xc000095680 flock:{w:{state:0 sema:0} writerSem:0 readerSem:0 readerCount:0 readerWait:0} ThisFunction:0xc000095680 ThisFunctionId:func-cfe66e39aba54ff989d6764cc4edda20 PrevFunctionId:func-ef567879d0dd45b287ed709e549e9d32 funcParams:map[func-7886178381634f05b302841141382e59:map[] func-cfe66e39aba54ff989d6764cc4edda20:map[] func-ef567879d0dd45b287ed709e549e9d32:map[]] fplock:{w:{state:0 sema:0} writerSem:0 readerSem:0 readerCount:0 readerWait:0} buffer:[] data:map[FunctionIdFirstVirtual:[This is Data1 from Test This is Data2 from Test This is Data3 from Test] func-7886178381634f05b302841141382e59:[data from funcName[funcName1], index = 0 data from funcName[funcName1], index = 1 data from funcName[funcName1], index = 2] func-ef567879d0dd45b287ed709e549e9d32:[data from funcName[funcName1], index = 0 data from funcName[funcName1], index = 1 data from funcName[funcName1], index = 2]] inPut:[data from funcName[funcName1], index = 0 data from funcName[funcName1], index = 1 data from funcName[funcName1], index = 2] abort:false action:{DataReuse:false ForceEntryNext:false JumpFunc: Abort:false}}

---> Call funcName3Handler ----
In FuncName = funcName3, FuncId = func-cfe66e39aba54ff989d6764cc4edda20, row = data from funcName[funcName1], index = 0
In FuncName = funcName3, FuncId = func-cfe66e39aba54ff989d6764cc4edda20, row = data from funcName[funcName1], index = 1
In FuncName = funcName3, FuncId = func-cfe66e39aba54ff989d6764cc4edda20, row = data from funcName[funcName1], index = 2
--- PASS: TestActionDataReuse (0.02s)
PASS
ok      kis-flow

-flow/test   0.523s

The test execution output provides valuable insights into the functioning of the DataReuse feature. Initially, the test sets up various functions and connectors and then loads the configurations. After submitting raw data to flowName3, the flow is executed.

During execution, the data flows through different functions, starting with funcName1, then dataReuseFunc, and finally funcName3. At each step, we observe the input data and the resulting data transformation.

For funcName1, we see the original data being processed, and then for dataReuseFunc, the data from funcName1 is reused. Finally, in funcName3, we observe the reused data being further processed.

This comprehensive unit test ensures that the DataReuse functionality behaves as expected, effectively passing data from one function to another without loss or corruption.

7.3 Action ForceEntryNext (Forcing Entry to the Next Layer)

7.3.1 ForceEntryNext Action Attribute

In the current KisFlow implementation, if the current Function does not commit any data (results data for this layer), the Flow will not proceed to the next layer of Functions after the current one finishes. However, in some streaming computations, it might be necessary to continue executing downward even if there is no data available. Therefore, we can introduce a ForceEntryNext action to trigger this behavior.

Firstly, we add a ForceEntryNext attribute to the Action:

kis-flow/kis/action.go

// Action KisFlow execution Actions
type Action struct {
    // DataReuse indicates whether to reuse data from the upper layer Function
    DataReuse bool

    // By default, if the current Function calculates 0 rows of data, subsequent Functions will not execute
    // ForceEntryNext overrides the above default rule and forces entry to the next layer of Functions even if there's no data
    ForceEntryNext bool

    // Abort terminates the execution of the Flow
    Abort bool
}

// ActionForceEntryNext sets the ForceEntryNext attribute to true
func ActionForceEntryNext(act *Action) {
    act.ForceEntryNext = true
}

We also provide a configuration function ActionForceEntryNext() to modify this attribute's status.

7.3.2 Capturing the Action

In the dealAction() method, which captures the Action, we add a check for this status. If set, the flow.abort status needs to be changed to false, allowing the flow to continue to the next layer.

kis-flow/flow/kis_flow_action.go

// dealAction processes the Action and determines the next steps of the Flow
func (flow *KisFlow) dealAction(ctx context.Context, fn kis.Function) (kis.Function, error) {

    // DataReuse Action
    if flow.action.DataReuse {
        if err := flow.commitReuseData(ctx); err != nil {
            return nil, err
        }
    } else {
        if err := flow.commitCurData(ctx); err != nil {
            return nil, err
        }
    }

    // ++++++++++++++++++++++++++++
    // ForceEntryNext Action
    if flow.action.ForceEntryNext {
        if err := flow.commitVoidData(ctx); err != nil {
            return nil, err
        }
        flow.abort = false
    }

    // Update the previous FunctionId cursor
    flow.PrevFunctionId = flow.ThisFunctionId
    fn = fn.Next()

    // Abort Action
    if flow.action.Abort {
        flow.abort = true
    }

    // Clear the Action
    flow.action = kis.Action{}

    return fn, nil
}

Here is a detail: we need to call a commitVoidData() method, which commits empty data. The reason is that if empty data is not committed, the flow.buffer remains empty, preventing the commit action. This would result in the key flow.data[flow.ThisFunctionId] not existing, causing a key-not-found exception and panic when flow.getCurData() is executed. Therefore, empty data needs to be committed to flow.data[flow.ThisFunctionId].

The specific implementation of commitVoidData() is as follows:

kis-flow/flow/kis_flow_data.go

func (flow *KisFlow) commitVoidData(ctx context.Context) error {
    if len(flow.buffer) != 0 {
        return nil
    }

    // Create empty data
    batch := make(common.KisRowArr, 0)

    // Commit the buffer data to the result data of this layer
    flow.data[flow.ThisFunctionId] = batch

    log.Logger().DebugFX(ctx, " ====> After commitVoidData, flow_name = %s, flow_id = %s\nAll Level Data =\n %+v\n", flow.Name, flow.Id, flow.data)

    return nil
}

7.3.3 Unit Test Without Setting ForceEntryNext

First, create a Function configuration for ·noResultFunc· and implement the corresponding callback business function.

kis-flow/test/load_conf/func/func-NoResultFunc.yml

kistype: func
fname: noResultFunc
fmode: Calculate
source:
  name: user_order_error_rate
  must:
    - order_id
    - user_id

kis-flow/test/faas/faas_no_result.go

package faas

import (
    "context"
    "fmt"
    "kis-flow/kis"
)

// type FaaS func(context.Context, Flow) error

func NoResultFuncHandler(ctx context.Context, flow kis.Flow) error {
    fmt.Println("---> Call NoResultFuncHandler ----")

    for _, row := range flow.Input() {
        str := fmt.Sprintf("In FuncName = %s, FuncId = %s, row = %s", flow.GetThisFuncConf().FName, flow.GetThisFunction().GetId(), row)
        fmt.Println(str)
    }

    return flow.Next()
}

In this Function, at the end, we only call flow.Next() without passing any Action.

Next, create a new Flow FlowName4 with the following configuration:

kis-flow/test/load_conf/flow-FlowName4.yml

kistype: flow
status: 1
flow_name: flowName4
flows:
  - fname: funcName1
  - fname: noResultFunc
  - fname: funcName3

Finally, we write a unit test case code with noResultFunc in the middle.

kis-flow/test/kis_action_test.go

func TestActionForceEntry(t *testing.T) {
    ctx := context.Background()

    // 0. Register Function callback business
    kis.Pool().FaaS("funcName1", faas.FuncDemo1Handler)
    kis.Pool().FaaS("noResultFunc", faas.NoResultFuncHandler) // Add noResultFunc business
    kis.Pool().FaaS("funcName3", faas.FuncDemo3Handler)

    // 0. Register ConnectorInit and Connector callback business
    kis.Pool().CaaSInit("ConnName1", caas.InitConnDemo1)
    kis.Pool().CaaS("ConnName1", "funcName2", common.S, caas.CaasDemoHanler1)

    // 1. Load configuration files and build the Flow
    if err := file.ConfigImportYaml("/Users/tal/gopath/src/kis-flow/test/load_conf/"); err != nil {
        panic(err)
    }

    // 2. Get the Flow
    flow1 := kis.Pool().GetFlow("flowName4")

    // 3. Submit original data
    _ = flow1.CommitRow("This is Data1 from Test")
    _ = flow1.CommitRow("This is Data2 from Test")
    _ = flow1.CommitRow("This is Data3 from Test")

    // 4. Execute flow1
    if err := flow1.Run(ctx); err != nil {
        panic(err)
    }
}

Navigate to kis-flow/test/ and execute:

go test -test.v -test.paniconexit0 -test.run  TestActionForceEntry

The results are as follows:

=== RUN   TestActionForceEntry
Add KisPool FuncName=funcName1
Add KisPool FuncName=noResultFunc
Add KisPool FuncName=funcName3
Add KisPool CaaSInit CName=ConnName1
Add KisPool CaaS CName=ConnName1, FName=funcName2, Mode =Save
===> Call Connector InitDemo1
&{conn ConnName1 0.0.0.0:9988,0.0.0.0:9999,0.0.0.0:9990 redis redis-key map[args1:value1 args2:value2] [] [funcName2 funcName2]}
Add FlowRouter FlowName=flowName1
Add FlowRouter FlowName=flowName2
Add FlowRouter FlowName=flowName3
Add FlowRouter FlowName=flowName4
===> Call Connector InitDemo1
&{conn ConnName1 0.0.0.0:9988,0.0.0.0:9999,0.0.0.0:9990 redis redis-key map[args1:value1 args2:value2] [] [funcName2 funcName2 funcName2]}
Add FlowRouter FlowName=flowName5
context.Background
====> After CommitSrcData, flow_name = flowName4, flow_id = flow-a496d02c79204e9a803fb5e1307523c9
All Level Data =
 map[FunctionIdFirstVirtual:[This is Data1 from Test This is Data2 from Test This is Data3 from Test]]

KisFunctionV, flow = &{Id:flow-a496d02c79204e9a803fb5e1307523c9 Name:flowName4 Conf:0xc000152e40 Funcs:map[funcName1:0xc00011d560 funcName3:0xc00011d620 noResultFunc:0xc00011d5c0] FlowHead:0xc00011d560 FlowTail:0xc00011d620 flock:{w:{state:0 sema:0} writerSem:0 readerSem:0 readerCount:0 readerWait:0} ThisFunction:0xc00011d560 ThisFunctionId:func-4d113d6a8e744d30a906db310f2d7818 PrevFunctionId:FunctionIdFirstVirtual funcParams:map[func-47cb6f9ae464484aa779c18284035705:map[] func-4d113d6a8e744d30a906db310f2d7818:map[] func-70011c7ccecf46be91c6993d143639bb:map[]] fplock:{w:{state:0 sema:0} writerSem:0 readerSem:0 readerCount:0 readerWait:0} buffer:[] data:map[FunctionIdFirstVirtual:[This is Data1 from Test This is Data2 from Test This is Data3 from Test]] inPut:[This is Data1 from Test This is Data2 from Test This is Data3 from Test] abort:false action:{DataReuse:false ForceEntryNext:false JumpFunc: Abort:false}}

---> Call funcName1Handler ----
In FuncName = funcName1, FuncId = func-4d113d6a8e744d30a906db310f2d7818, row = This is Data1 from Test
In FuncName = funcName1, FuncId = func-4d113d6a8e744d30a906db310f2d7818, row = This is Data2 from Test
In FuncName = funcName1, FuncId = func-4d113d6a8e744d30a906db310f2d7818, row = This is Data3 from Test
context.Background
 ====> After commitCurData, flow_name = flowName4, flow_id = flow-a496d02c79204e9a803fb5e1307523c9
All Level Data =
 map[FunctionIdFirstVirtual:[This is Data1 from Test This is Data2 from Test This is Data3 from Test] func-4d113d6a8e744d30a906db310f2d7818:[data from funcName[funcName1], index = 0 data from funcName[funcName1], index = 1 data from funcName[funcName1], index = 2]]

KisFunctionC, flow = &{Id:flow-a496d02c79204e9a803fb5e1307523c9 Name:flowName4 Conf:0xc000152e40 Funcs:map[funcName1:0xc00011d560 funcName3:0xc00011d620 noResultFunc:0xc00011d5c0] FlowHead:0xc00011d560 FlowTail:0xc00011d620 flock:{w:{state:0 sema:0} writerSem:0 readerSem:0 readerCount:0 readerWait:0} ThisFunction:0xc00011d5c0 ThisFunctionId:func-47cb6f9ae464484aa779c18284035705 PrevFunctionId:func-4d113d6a8e744d30a906db310f2d7818 funcParams:map[func-47cb6f9ae464484aa779c18284035705:map[] func-4d113d6a8e744d30a906db310f2d7818:map[] func-70011c7ccecf46be91c6993d143639bb:map[]] fplock:{w:{state:0 sema:0} writerSem:0 readerSem:0 readerCount:0 readerWait:0} buffer:[] data:map[FunctionIdFirstVirtual:[This is Data1 from Test This is Data2 from Test This is Data3 from Test] func-4d113d6a8e744d30a906db310f2d7818:[data from funcName[funcName1], index = 0 data from funcName[funcName1], index = 1 data from funcName[funcName1], index = 2]] inPut:[data from funcName[funcName1], index = 0 data from funcName[funcName1], index = 1 data from funcName[funcName1], index = 2] abort:false action:{DataReuse:false ForceEntryNext:false JumpFunc: Abort:false}}

---> Call NoResultFuncHandler ----
In FuncName = noResultFunc, FuncId = func-47cb6f9ae464484aa779c18284035705, row = data from funcName[funcName1], index = 0
In FuncName = noResultFunc, FuncId = func-47cb6f9ae464484aa779c18284035705, row = data from funcName[funcName1], index = 1
In FuncName = noResultFunc, FuncId = func-47cb6f9ae464484aa779c18284035705, row = data from funcName[funcName1], index = 2
--- PASS: TestActionForceEntry (0.02s)
PASS
ok      kis-flow/test   0.958s

Because noResultFunc does not generate any result data, the next Function will not be executed. The execution ends with:

---> Call NoResultFuncHandler ----

7.3.4 Unit Testing with ForceEntryNext

Next, we will add the ForceEntryNext action. In NoResultFuncHandler(), we add flow.Next(kis.ActionForceEntryNext) as shown below:

kis-flow/test/faas/faas_no_result.go

package faas

import (
    "context"
    "fmt"
    "kis-flow/kis"
)

func NoResultFuncHandler(ctx context.Context, flow kis.Flow) error {
    fmt.Println("---> Call NoResultFuncHandler ----")

    for _, row := range flow.Input() {
        str := fmt.Sprintf("In FuncName = %s, FuncId = %s, row = %s", flow.GetThisFuncConf().FName, flow.GetThisFunction().GetId(), row)
        fmt.Println(str)
    }

    return flow.Next(kis.ActionForceEntryNext)
}

Navigate to the kis-flow/test/ directory and execute:

go test -test.v -test.paniconexit0 -test.run TestActionForceEntry

The results are as follows:

=== RUN   TestActionForceEntry
Add KisPool FuncName=funcName1
Add KisPool FuncName=noResultFunc
Add KisPool FuncName=funcName3
Add KisPool CaaSInit CName=ConnName1
Add KisPool CaaS CName=ConnName1, FName=funcName2, Mode=Save
===> Call Connector InitDemo1
&{conn ConnName1 0.0.0.0:9988,0.0.0.0:9999,0.0.0.0:9990 redis redis-key map[args1:value1 args2:value2] [] [funcName2 funcName2]}
Add FlowRouter FlowName=flowName5
===> Call Connector InitDemo1
&{conn ConnName1 0.0.0.0:9988,0.0.0.0:9999,0.0.0.0:9990 redis redis-key map[args1:value1 args2:value2] [] [funcName2 funcName2 funcName2]}
Add FlowRouter FlowName=flowName1
Add FlowRouter FlowName=flowName2
Add FlowRouter FlowName=flowName3
Add FlowRouter FlowName=flowName4
context.Background
====> After CommitSrcData, flow_name = flowName4, flow_id = flow-7fb47f227c9f4b9d8fa69c28177fc7bb
All Level Data =
 map[FunctionIdFirstVirtual:[This is Data1 from Test This is Data2 from Test This is Data3 from Test]]

KisFunctionV, flow = &{Id:flow-7fb47f227c9f4b9d8fa69c28177fc7bb Name:flowName4 Conf:0xc000028e80 Funcs:map[funcName1:0xc0000136e0 funcName3:0xc0000137a0 noResultFunc:0xc000013740] FlowHead:0xc0000136e0 FlowTail:0xc0000137a0 flock:{w:{state:0 sema:0} writerSem:0 readerSem:0 readerCount:0 readerWait:0} ThisFunction:0xc0000136e0 ThisFunctionId:func-ecddaee7d7d447a9852d07088732f509 PrevFunctionId:FunctionIdFirstVirtual funcParams:map[func-5729600ae6ea4d6f879eb5832c638e1a:map[] func-c9817c7993894919b8463dea1757544e:map[] func-ecddaee7d7d447a9852d07088732f509:map[]] fplock:{w:{state:0 sema:0} writerSem:0 readerSem:0 readerCount:0 readerWait:0} buffer:[] data:map[FunctionIdFirstVirtual:[This is Data1 from Test This is Data2 from Test This is Data3 from Test]] inPut:[This is Data1 from Test This is Data2 from Test This is Data3 from Test] abort:false action:{DataReuse:false ForceEntryNext:false JumpFunc: Abort:false}}

---> Call funcName1Handler ----
In FuncName = funcName1, FuncId = func-ecddaee7d7d447a9852d07088732f509, row = This is Data1 from Test
In FuncName = funcName1, FuncId = func-ecddaee7d7d447a9852d07088732f509, row = This is Data2 from Test
In FuncName = funcName1, FuncId = func-ecddaee7d7d447a9852d07088732f509, row = This is Data3 from Test
context.Background
 ====> After commitCurData, flow_name = flowName4, flow_id = flow-7fb47f227c9f4b9d8fa69c28177fc7bb
All Level Data =
 map[FunctionIdFirstVirtual:[This is Data1 from Test This is Data2 from Test This is Data3 from Test] func-ecddaee7d7d447a9852d07088732f509:[data from funcName[funcName1], index = 0 data from funcName[funcName1], index = 1 data from funcName[funcName1], index = 2]]

KisFunctionC, flow = &{Id:flow-7fb47f227c9f4b9d8fa69c28177fc7bb Name:flowName4 Conf:0xc000028e80 Funcs:map[funcName1:0xc0000136e0 funcName3:0xc0000137a0 noResultFunc:0xc000013740] FlowHead:0xc0000136e0 FlowTail:0xc0000137a0 flock:{w:{state:0 sema:0} writerSem:0 readerSem:0 readerCount:0 readerWait:0} ThisFunction:0xc000013740 ThisFunctionId:func-c9817c7993894919b8463dea1757544e PrevFunctionId:func-ecddaee7d7d447a9852d07088732f509 funcParams:map[func-5729600ae6ea4d6f879eb5832c638e1a:map[] func-c9817c7993894919b8463dea1757544e:map[] func-ecddaee7d7d447a9852d07088732f509:map[]] fplock:{w:{state:0 sema:0} writerSem:0 readerSem:0 readerCount:0 readerWait:0} buffer:[] data:map[FunctionIdFirstVirtual:[This is Data1 from Test This is Data2 from Test This is Data3 from Test] func-ecddaee7d7d447a9852d07088732f509:[data from funcName[funcName1], index = 0 data from funcName[funcName1], index = 1 data from funcName[funcName1], index = 2]] inPut:[data from funcName[funcName1], index = 0 data from funcName[funcName1], index = 1 data from funcName[funcName1], index = 2] abort:false action:{DataReuse:false ForceEntryNext:false JumpFunc: Abort:false}}

---> Call NoResultFuncHandler ----
In FuncName = noResultFunc, FuncId = func-c9817c7993894919b8463dea1757544e, row = data from funcName[funcName1], index = 0
In FuncName = noResultFunc, FuncId = func-c9817c7993894919b8463dea1757544e, row = data from funcName[funcName1], index = 1
In FuncName = noResultFunc, FuncId = func-c9817c7993894919b8463dea1757544e, row = data from funcName[funcName1], index = 2
context.Background
 ====> After commitVoidData, flow_name = flowName4, flow_id = flow-7fb47f227c9f4b9d8fa69c28177fc7bb
All Level Data =
 map[FunctionIdFirstVirtual:[This is Data1 from Test This is Data2 from Test This is Data3 from Test] func-c9817c7993894919b8463dea1757544e:[] func-ecddaee7d7d447a9852d07088732f509:[data from funcName[funcName1], index = 0 data from funcName[funcName1], index = 1 data from funcName[funcName1], index = 2]]

KisFunctionC, flow = &{Id:flow-7fb47f227c9f4b9d8fa69c28177fc7bb Name:flowName4 Conf:0xc000028e80 Funcs:map[funcName1:0xc0000136e0 funcName3:0xc0000137a0 noResultFunc:0xc000013740] FlowHead:0xc0000136e0 FlowTail:0xc0000137a0 flock:{w:{state:0 sema:0} writerSem:0 readerSem:0 readerCount:0 readerWait:0} ThisFunction:0xc0000137a0 ThisFunctionId:func-5729600ae6ea4d6f879eb5832c638e1a PrevFunctionId:func-c9817c7993894919b8463dea1757544e funcParams:map[func-5729600ae6ea4d6f879eb5832c638e1a:map[] func-c9817c7993894919b8463dea1757544e:map[] func-ecddaee7d7d447a9852d07088732f509:map[]] fplock:{w:{state:0 sema:0} writerSem:0 readerSem:0 readerCount:0 readerWait:0} buffer:[] data:map[FunctionIdFirstVirtual:[This is Data1 from Test This is Data2 from Test This is Data3 from Test] func-c9817c7993894919b8463dea1757544e:[] func-ecddaee7d7d447a9852d07088732f509:[data from funcName[funcName1], index = 0 data from funcName[funcName1], index = 1 data from funcName[funcName1], index = 2]] inPut:[] abort:false action:{DataReuse:false ForceEntryNext:false JumpFunc: Abort:false}}

---> Call funcName3Handler ----
--- PASS: TestActionForceEntry (0.01s)
PASS
ok      kis-flow/test   0.348s

It is observed that the function in the third layer, funcName3Handler, is executed, but it has no data.

7.4 Action JumpFunc (Flow Jump)

Next, we will implement the JumpFunc Action. JumpFunc allows jumping to a specified FuncName within the current Flow and continuing execution (provided that the target FuncName exists in the current Flow).

Note: JumpFunc can lead to infinite loops, so use it cautiously in business logic.

7.4.1 Adding JumpFunc to Action

First, add a JumpFunc property to the Action. Note that JumpFunc is not a boolean state but a string representing the specific FunctionName to jump to.

kis-flow/kis/action.go

// Action KisFlow execution flow actions
type Action struct {
    // DataReuse indicates whether to reuse data from the previous function
    DataReuse bool

    // By default, Next() will not execute the subsequent function if the current function's result set is empty.
    // ForceEntryNext forces the next function to execute even if the current function's result set is empty.
    ForceEntryNext bool

    // ++++++++++
    // JumpFunc specifies the function to jump to for further execution
    JumpFunc string

    // Abort terminates the flow execution
    Abort bool
}

// ActionJumpFunc returns an ActionFunc function that sets the JumpFunc property in Action
// (Note: Can easily cause flow loops leading to infinite loops)
func ActionJumpFunc(funcName string) ActionFunc {
    return func(act *Action) {
        act.JumpFunc = funcName
    }
}

Then provide a method to modify the JumpFunc configuration ActionJumpFunc(). Note that this method differs from previous ones as it returns an anonymous function and executes it to modify the JumpFunc property in Action.

7.4.2 Capturing the Action

Next, we capture the JumpFunc action by checking if JumpFunc is an empty string.

kis-flow/flow/kis_flow_action.go

// dealAction processes the Action and determines the next step in the Flow
func (flow *KisFlow) dealAction(ctx context.Context, fn kis.Function) (kis.Function, error) {

    // DataReuse Action
    if flow.action.DataReuse {
        if err := flow.commitReuseData(ctx); err != nil {
            return nil, err
        }
    } else {
        if err := flow.commitCurData(ctx); err != nil {
            return nil, err
        }
    }

    // ForceEntryNext Action
    if flow.action.ForceEntryNext {
        if err := flow.commitVoidData(ctx); err != nil {
            return nil, err
        }
        flow.abort = false
    }

    // ++++++++++++++++++++++++++++++++
    // JumpFunc Action
    if flow.action.JumpFunc != "" {
        if _, ok := flow.Funcs[flow.action.JumpFunc]; !ok {
            // JumpFunc is not in the flow
            return nil, errors.New(fmt.Sprintf("Flow Jump -> %s is not in Flow", flow.action.JumpFunc))
        }

        jumpFunction := flow.Funcs[flow.action.JumpFunc]
        // Update the previous function
        flow.PrevFunctionId = jumpFunction.GetPrevId()
        fn = jumpFunction

        // If a jump is set, force the jump
        flow.abort = false
    // ++++++++++++++++++++++++++++++++

    } else {

        // Update the previous function ID cursor
        flow.PrevFunctionId = flow.ThisFunctionId
        fn = fn.Next()
    }

    // Abort Action forcibly terminates the flow
    if flow.action.Abort {
        flow.abort = true
    }

    // Clear the Action
    flow.action = kis.Action{}

    return fn, nil
}

If JumpFunc is set, the next function fn pointer needs to be updated accordingly. Otherwise, the normal address fn.Next() is used.

7.4.3 Unit Testing

Next, let's define a function with a jump action configuration as follows:

kis-flow/test/load_conf/func/func-jumpFunc.yml

kistype: func
fname: jumpFunc
fmode: Calculate
source:
  name: User Order Error Rate
  must:
    - order_id
    - user_id

And implement the related function business logic as follows:

kis-flow/test/faas/faas_jump.go

package faas

import (
    "context"
    "fmt"
    "kis-flow/kis"
)

// type FaaS func(context.Context, Flow) error

func JumpFuncHandler(ctx context.Context, flow kis.Flow) error {
    fmt.Println("---> Call JumpFuncHandler ----")

    for _, row := range flow.Input() {
        str := fmt.Sprintf("In FuncName = %s, FuncId = %s, row = %s", flow.GetThisFuncConf().FName, flow.GetThisFunction().GetId(), row)
        fmt.Println(str)
    }

    return flow.Next(kis.ActionJumpFunc("funcName1"))
}

Here, flow.Next(kis.ActionJumpFunc("funcName1")) specifies the jump to the function named funcName1.

Create a new flow named FlowName5 with the following configuration:

kis-flow/test/load_conf/flow/flow-FlowName5.yml

kistype: flow
status: 1
flow_name: flowName5
flows:
  - fname: funcName1
  - fname: funcName2
  - fname: jumpFunc

Next, implement the unit test case code as follows:

kis-flow/test/kis_action_test.go

func TestActionJumpFunc(t *testing.T) {
    ctx := context.Background()

    // 0. Register Function callback business logic
    kis.Pool().FaaS("funcName1", faas.FuncDemo1Handler)
    kis.Pool().FaaS("funcName2", faas.FuncDemo2Handler)
    kis.Pool().FaaS("jumpFunc", faas.JumpFuncHandler) // Add jumpFunc business logic

    // 0. Register ConnectorInit and Connector callback business logic
    kis.Pool().CaaSInit("ConnName1", caas.InitConnDemo1)
    kis.Pool().CaaS("ConnName1", "funcName2", common.S, caas.CaasDemoHanler1)

    // 1. Load configuration files and build the flow
    if err := file.ConfigImportYaml("/Users/tal/gopath/src/kis-flow/test/load_conf/"); err != nil {
        panic(err)
    }

    // 2. Get the flow
    flow1 := kis.Pool().GetFlow("flowName5")

    // 3. Commit raw data
    _ = flow1.CommitRow("This is Data1 from Test")
    _ = flow1.CommitRow("This is Data2 from Test")
    _ = flow1.CommitRow("This is Data3 from Test")

    // 4. Execute flow1
    if err := flow1.Run(ctx); err != nil {
        panic(err)
    }
}

Change the directory to kis-flow/test/ and execute:

go test -test.v -test.paniconexit0 -test.run TestActionJumpFunc

The result is as follows:

... 
...

---> Call funcName1Handler ----
In FuncName = funcName1, FuncId = func-f6ca8010d66744429bf6069c9897a928, row = This is Data1 from Test
In FuncName = funcName1, FuncId = func-f6ca8010d66744429bf6069c9897a928, row = This is Data2 from Test
In FuncName = funcName1, FuncId = func-f6ca8010d66744429bf6069c9897a928, row = This is Data3 from Test
context.Background
 ====> After commitCurData, flow_name = flowName5, flow_id = flow-5da80af989dc49648a001762fa08b866
All Level Data =
 map[FunctionIdFirstVirtual:[This is Data1 from Test This is Data2 from Test This is Data3 from Test] func-5800567c4cd842b6b377c2b0c0fd81c2:[data from funcName[funcName2], index = 0 data from funcName[funcName2], index = 1 data from funcName[funcName2], index = 2] func-f6ca8010d66744429bf6069c9897a928:[data from funcName[funcName1], index = 0 data from funcName[funcName1], index = 1 data from funcName[funcName1], index = 2]]

KisFunctionS, flow = &{Id:flow-5da80af989dc49648a001762fa08b866 Name:flowName5 Conf:0xc000028f80 Funcs:map[funcName1:0xc000013620 funcName2:0xc000013680 jumpFunc:0xc0000136e0] FlowHead:0xc000013620 FlowTail:0xc0000136e0 flock:{w:{state:0 sema:0} writerSem:0 readerSem:0 readerCount:0 readerWait:0} ThisFunction:0xc000013680 ThisFunctionId:func-5800567c4cd842b6b377c2b0c0fd81c2 PrevFunctionId:func-f6ca8010d66744429bf6069c9897a928 funcParams:map[func-4faf8f019f4a4a48b84ef27abfad53d1:map[] func-5800567c4cd842b6b377c2b0c0fd81c2:map[] func-f6ca8010d66744429bf6069c9897a928:map[]] fplock:{w:{state:0 sema:0} writerSem:0 readerSem:0 readerCount:0 readerWait:0} buffer:[] data:map[FunctionIdFirstVirtual:[This is Data1 from Test This is Data2 from Test This is Data3 from Test] func-5800567c4cd842b6b377c2b0c0fd81c2:[data from funcName[funcName2], index = 0 data from funcName[funcName2], index = 1 data from funcName[funcName2], index = 2] func-f6ca8010d66744429bf6069c9897a928:[data from funcName[funcName1], index = 0 data from funcName[funcName1], index = 1 data from funcName[funcName1], index = 2]] inPut:[data from funcName[funcName1], index = 0 data from funcName[funcName1], index = 1 data from funcName[funcName1], index = 2] abort:false action:{DataReuse:false ForceEntryNext:false JumpFunc: Abort:false}}

---> Call funcName2Handler ----
In FuncName = funcName2, FuncId = func-5800567c4cd842b6b377c2b0c0fd81c2, row = data from funcName[funcName1], index = 0
===> In CaasDemoHanler1: flowName: flowName5, cName:ConnName1, fnName:funcName2, mode:Save
===> Call Connector CaasDemoHanler1, args from funciton: data from funcName[funcName1], index = 0
In FuncName = funcName2, FuncId = func-5800567c4cd842b6b377c2b0c0fd81c2, row = data from funcName[funcName1], index = 1
===> In CaasDemoHanler1: flowName: flowName5, cName:ConnName1, fnName:funcName2, mode:Save
===> Call Connector CaasDemoHanler1, args from funciton: data from funcName[funcName1], index = 1
In FuncName = funcName2, FuncId = func-5800567c4cd842b6b377c2b0c0fd81c2, row = data from funcName[funcName1], index = 2
===> In CaasDemoHanler1: flowName: flowName5, cName:ConnName1, fnName:funcName2, mode:Save
===> Call Connector CaasDemoHanler1, args from funciton: data from funcName[funcName1], index = 2
context.Background
 ====> After commitCurData, flow_name = flowName5, flow_id = flow-5da80af989dc49648a001762fa08b866
All Level Data =
 map[FunctionIdFirstVirtual:[This is Data1 from Test This is Data2 from Test This is Data3 from Test] func-5800567c4cd842b6b377c2b0c0fd81c2:[data from funcName[funcName2], index = 0 data from funcName[funcName2], index = 1 data from funcName[funcName2], index = 2] func-f6ca8010d66744429bf6069c9897a928:[data from funcName[funcName1], index = 0 data from funcName[funcName1], index = 1 data from funcName[funcName1], index = 2]]

KisFunctionC, flow = &{Id:flow-5da80af989dc49648a001762fa08b866 Name:flowName5 Conf:0xc000028f80 Funcs:map[funcName1:0xc000013620 funcName2:0xc000013680 jumpFunc:0xc0000136e0] FlowHead:0xc000013620 FlowTail:0xc0000136e0 flock:{w:{state:0 sema:0} writerSem:0 readerSem:0 readerCount:0 readerWait:0} ThisFunction:0xc0000136e0 ThisFunctionId:func-4faf8f019f4a4a48b84ef27abfad53d1 PrevFunctionId
funcParams
func-4faf8f019f4a4a48b84ef27abfad53d1
func-5800567c4cd842b6b377c2b0c0fd81c2
func-f6ca8010d66744429bf6069c9897a928
fplock:{w:{state:0 sema:0} writerSem:0 readerSem:0 readerCount:0 readerWait:0} buffer:[] data
FunctionIdFirstVirtual:[This is Data1 from Test This is Data2 from Test This is Data3 from Test] func-5800567c4cd842b6b377c2b0c0fd81c2:[data from funcName[funcName2], index = 0 data from funcName[funcName2], index = 1 data from funcName[funcName2], index = 2] func-f6ca8010d66744429bf6069c9897a928:[data from funcName[funcName1], index = 0 data from funcName[funcName1], index = 1 data from funcName[funcName1], index = 2]
inPut:[data from funcName[funcName2], index = 0 data from funcName[funcName2], index = 1 data from funcName[funcName2], index = 2] abort
action:{DataReuse
ForceEntryNext
JumpFunc: Abort
}}

---> Call JumpFuncHandler ----
In FuncName = jumpFunc, FuncId = func-4faf8f019f4a4a48b84ef27abfad53d1, row = data from funcName[funcName2], index = 0
In FuncName = jumpFunc, FuncId = func-4faf8f019f4a4a48b84ef27abfad53d1, row = data from funcName[funcName2], index = 1
In FuncName = jumpFunc, FuncId = func-4faf8f019f4a4a48b84ef27abfad53d1, row = data from funcName[funcName2], index = 2
KisFunctionV, flow = &{Id
Name
Conf:0xc000028f80 Funcs
funcName1:0xc000013620 funcName2:0xc000013680 jumpFunc:0xc0000136e0
FlowHead:0xc000013620 FlowTail:0xc0000136e0 flock:{w:{state:0 sema:0} writerSem:0 readerSem:0 readerCount:0 readerWait:0} ThisFunction:0xc000013620 ThisFunctionId
PrevFunctionId
funcParams
func-4faf8f019f4a4a48b84ef27abfad53d1
func-5800567c4cd842b6b377c2b0c0fd81c2
func-f6ca8010d66744429bf6069c9897a928
fplock:{w:{state:0 sema:0} writerSem:0 readerSem:0 readerCount:0 readerWait:0} buffer:[] data
FunctionIdFirstVirtual:[This is Data1 from Test This is Data2 from Test This is Data3 from Test] func-5800567c4cd842b6b377c2b0c0fd81c2:[data from funcName[funcName2], index = 0 data from funcName[funcName2], index = 1 data from funcName[funcName2], index = 2] func-f6ca8010d66744429bf6069c9897a928:[data from funcName[funcName1], index = 0 data from funcName[funcName1], index = 1 data from funcName[funcName1], index = 2]
inPut:[This is Data1 from Test This is Data2 from Test This is Data3 from Test] abort
action:{DataReuse
ForceEntryNext
JumpFunc: Abort
}}

---> Call funcName1Handler ----

...
...

We observe that the Flow keeps looping, indicating that our JumpFunc Action has taken effect.

7.5 [V0.6] Source Code

You can find the source code for version 0.6 of the project at:

https://github.com/aceld/kis-flow/releases/tag/v0.6

Author: Aceld
GitHub: https://github.com/aceld

KisFlow Open Source Project Address: https://github.com/aceld/kis-flow

Document: https://github.com/aceld/kis-flow/wiki

Case1-Quick Start
Case2-Flow Parallel Operation
Case3-Application of KisFlow in Multi-Goroutines
Case4-KisFlow in Message Queue (MQ) Applications

(Part 6)Golang Framework Hands-on - KisFlow Stream Computing Framework-Configuration Import and Export

Aceld — Mon, 13 May 2024 07:35:50 +0000

Github: https://github.com/aceld/kis-flow
Document: https://github.com/aceld/kis-flow/wiki

Case1-Quick Start
Case2-Flow Parallel Operation
Case3-Application of KisFlow in Multi-Goroutines
Case4-KisFlow in Message Queue (MQ) Applications

6.1 Configuration Import

Currently, establishing a Flow and Functions requires a series of cumbersome additions each time, which is not very convenient. Next, we can construct the structure of KisFlow and also export the structure of KisFlow to local files by batch reading and writing configuration files. Currently, we will use file format for configuration persistence, but developers can also choose to do persistence with databases or remote configurations in the future.

6.1.1 Creating Configuration Files

First, we create the necessary configuration files for KisFlow business under kis-flow/test/load_conf/. Within kis-flow/test/load_conf/, we create three folders: conn/, flow/, and func/ to respectively store the configuration information for Connector, Flow, and Function.

├── conn
│   └── conn-ConnName1.yml
├── flow
│   └── flow-FlowName1.yml
└── func
    ├── func-FuncName1.yml
    ├── func-FuncName2.yml
    └── func-FuncName3.yml

Create some yml files within these folders. The specific content is as follows:

A. Function

kis-flow/test/load_conf/func/func-FuncNam1.yml

kistype: func
fname: funcName1
fmode: Verify
source:
  name: Public account Douyin mall user order data
  must:
    - order_id
    - user_id

kis-flow/test/load_conf/func/func-FuncNam2.yml

kistype: func
fname: funcName2
fmode: Save
source:
  name: User order error rate
  must:
    - order_id
    - user_id
option:
  cname: ConnName1

kis-flow/test/load_conf/func/func-FuncNam3.yml

kistype: func
fname: funcName3
fmode: Calculate
source:
  name: User order error rate
  must:
    - order_id
    - user_id

B. Connector

kis-flow/test/load_conf/func/func-ConnName1.yml

kistype: conn
cname: ConnName1
addrs: '0.0.0.0:9988,0.0.0.0:9999,0.0.0.0:9990'
type: redis
key: redis-key
params:
  args1: value1
  args2: value2
load: null
save:
  - funcName2

C. Flow

kis-flow/test/load_conf/func/func-FlowName1.yml

kistype: flow
status: 1
flow_name: flowName1
flows:
  - fname: funcName1
  - fname: funcName2
  - fname: funcName3

6.1.2 Configuration Parsing

Create the kis-flow/file/ directory and the kis-flow/file/config_import.go file within it.

First, define an interface allConfig that can hold all configurations:

kis-flow/file/config_import.go

type allConfig struct {
    Flows map[string]*config.KisFlowConfig
    Funcs map[string]*config.KisFuncConfig
    Conns map[string]*config.KisConnConfig
}

Where the key serves as the Name field for each module.
Next, define methods for parsing Flow, Function, and Connector configurations. For YAML parsing, we will use the "gopkg.in/yaml.v3" library.

A. Flow Configuration Parsing

kis-flow/file/config_import.go

// kisTypeFlowConfigure parses Flow configuration files in YAML format
func kisTypeFlowConfigure(all *allConfig, confData []byte, fileName string, kisType interface{}) error {
    flow := new(config.KisFlowConfig)
    if ok := yaml.Unmarshal(confData, flow); ok != nil {
        return errors.New(fmt.Sprintf("%s has wrong format kisType = %s", fileName, kisType))
    }

    // Do not load Flow configuration if the status is disabled
    if common.KisOnOff(flow.Status) == common.FlowDisable {
        return nil
    }

    if _, ok := all.Flows[flow.FlowName]; ok {
        return errors.New(fmt.Sprintf("%s set repeat flow_id:%s", fileName, flow.FlowName))
    }

    // Add the configuration to the collection
    all.Flows[flow.FlowName] = flow

    return nil
}

confData: the file's binary data
fileName: the file path
kistype: the configuration file type

kisTypeFlowConfigure will parse the configuration information into the Flows member of allConfig.
Similarly, the methods for Function and Connector parsing are as follows.

B. Function Configuration Parsing

kis-flow/file/config_import.go

// kisTypeFuncConfigure parses Function configuration files in YAML format
func kisTypeFuncConfigure(all *allConfig, confData []byte, fileName string, kisType interface{}) error {
    function := new(config.KisFuncConfig)
    if ok := yaml.Unmarshal(confData, function); ok != nil {
        return errors.New(fmt.Sprintf("%s has wrong format kisType = %s", fileName, kisType))
    }
    if _, ok := all.Funcs[function.FName]; ok {
        return errors.New(fmt.Sprintf("%s set repeat function_id:%s", fileName, function.FName))
    }

    // Add the configuration to the collection
    all.Funcs[function.FName] = function

    return nil
}

C. Connector Configuration Parsing

kis-flow/file/config_import.go

// kisTypeConnConfigure parses Connector configuration files in YAML format
func kisTypeConnConfigure(all *allConfig, confData []byte, fileName string, kisType interface{}) error {
    conn := new(config.KisConnConfig)
    if ok := yaml.Unmarshal(confData, conn); ok != nil {
        return errors.New(fmt.Sprintf("%s is wrong format nsType = %s", fileName, kisType))
    }

    if _, ok := all.Conns[conn.CName]; ok {
        return errors.New(fmt.Sprintf("%s set repeat conn_id:%s", fileName, conn.CName))
    }

    // Add the configuration to the collection
    all.Conns[conn.CName] = conn

    return nil
}

6.1.3 Traversing Files

Below is an implementation to traverse all yml and yaml type files under a given path loadPath and parse the configuration information into allConfig according to the kistype category.

kis-flow/file/config_import.go

// parseConfigWalkYaml parses all configuration files in YAML format and stores the configuration information in allConfig
func parseConfigWalkYaml(loadPath string) (*allConfig, error) {

    all := new(allConfig)

    all.Flows = make(map[string]*config.KisFlowConfig)
    all.Funcs = make(map[string]*config.KisFuncConfig)
    all.Conns = make(map[string]*config.KisConnConfig)

    err := filepath.Walk(loadPath, func(filePath string, info os.FileInfo, err error) error {
        // Validate file extension
        if suffix := path.Ext(filePath); suffix != ".yml" && suffix != ".yaml" {
            return nil
        }

        // Read file content
        confData, err := ioutil.ReadFile(filePath)
        if err != nil {
            return err
        }

        confMap := make(map[string]interface{})

        // Validate YAML format
        if err := yaml.Unmarshal(confData, confMap); err != nil {
            return err
        }

        // Check if kisType exists
        if kisType, ok := confMap["kistype"]; !ok {
            return errors.New(fmt.Sprintf("yaml file %s has no file [kistype]!", filePath))
        } else {
            switch kisType {
            case common.KisIdTypeFlow:
                return kisTypeFlowConfigure(all, confData, filePath, kisType)

            case common.KisIdTypeFunction:
                return kisTypeFuncConfigure(all, confData, filePath, kisType)

            case common.KisIdTypeConnnector:
                return kisTypeConnConfigure(all, confData, filePath, kisType)

            default:
                return errors.New(fmt.Sprintf("%s set wrong kistype %s", filePath, kisType))
            }
        }
    })

    if err != nil {
        return nil, err
    }

    return all, nil
}

6.1.4 Import Method

Below is a public method ConfigImportYaml that imports files from a given root path.

kis-flow/file/config_import.go

// ConfigImportYaml parses all configuration files in YAML format
func ConfigImportYaml(loadPath string) error {

    all, err := parseConfigWalkYaml(loadPath)
    if err != nil {
        return err
    }

    for flowName, flowConfig := range all.Flows {

        // Build a Flow
        newFlow := flow.NewKisFlow(flowConfig)

        for _, fp := range flowConfig.Flows {
            if err := buildFlow(all, fp, newFlow, flowName); err != nil {
                return err
            }
        }

        // Add the flow to the FlowPool
        kis.Pool().AddFlow(flowName, newFlow)
    }

    return nil
}

First, it calls parseConfigWalkYaml() to load all configuration information into memory.
Then, it traverses all Flows one by one to construct them. Finally, the flow is added to the Pool. The specific construction process is as follows:

kis-flow/file/config_import.go

func buildFlow(all *allConfig, fp config.KisFlowFunctionParam, newFlow kis.Flow, flowName string) error {
    // Load Functions that the current Flow depends on
    if funcConfig, ok := all.Funcs[fp.FuncName]; !ok {
        return errors.New(fmt.Sprintf("FlowName [%s] need FuncName [%s], But has No This FuncName Config", flowName, fp.FuncName))
    } else {
        // Link Flow to Functions
        if funcConfig.Option.CName != "" {
            // Load Connectors that the current Function depends on
            if connConf, ok := all.Conns[funcConfig.Option.CName]; !ok {
                return errors.New(fmt.Sprintf("FuncName [%s] need ConnName [%s], But has No This ConnName Config", fp.FuncName, funcConfig.Option.CName))
            } else {
                // Associate Function Configurations with Connector Configurations
                _ = funcConfig.AddConnConfig(connConf)
            }
        }

        // Link Flow to Functions
        if err := newFlow.Link(funcConfig, fp.Params); err != nil {
            return err
        }
    }

    return nil
}

6.2 Configure Import Unit Testing

Create a unit test file kis-flow/test/kis_config_import_test.go.

package test

import (
    "context"
    "kis-flow/common"
    "kis-flow/file"
    "kis-flow/kis"
    "kis-flow/test/caas"
    "kis-flow/test/faas"
    "testing"
)

func TestConfigImportYmal(t *testing.T) {
    ctx := context.Background()

    // 0. Register Function callbacks
    kis.Pool().FaaS("funcName1", faas.FuncDemo1Handler)
    kis.Pool().FaaS("funcName2", faas.FuncDemo2Handler)
    kis.Pool().FaaS("funcName3", faas.FuncDemo3Handler)

    // 0. Register ConnectorInit and Connector callbacks
    kis.Pool().CaaSInit("ConnName1", caas.InitConnDemo1)
    kis.Pool().CaaS("ConnName1", "funcName2", common.S, caas.CaasDemoHanler1)

    // 1. Load configuration file and build Flow
    if err := file.ConfigImportYaml("/Users/gopath/src/kis-flow/test/load_conf/"); err != nil {
        panic(err)
    }

    // 2. Get Flow
    flow1 := kis.Pool().GetFlow("flowName1")

    // 3. Commit raw data
    _ = flow1.CommitRow("This is Data1 from Test")
    _ = flow1.CommitRow("This is Data2 from Test")
    _ = flow1.CommitRow("This is Data3 from Test")

    // 4. Execute flow1
    if err := flow1.Run(ctx); err != nil {
        panic(err)
    }
}

First, register the business methods. Then load the configuration via ConfigImportYaml, get the flow instance from the pool, submit data, and run it.

Note that the absolute path to the configuration file is provided here.

Navigate to the kis-flow/test/ directory and execute the command:

go test -test.v -test.paniconexit0 -test.run TestConfigImportYmal

The expected result is as follows:

=== RUN   TestConfigImportYmal
Add KisPool FuncName=funcName1
Add KisPool FuncName=funcName2
Add KisPool FuncName=funcName3
Add KisPool CaaSInit CName=ConnName1
Add KisPool CaaS CName=ConnName1, FName=funcName2, Mode =Save
===> Call Connector InitDemo1
&{conn ConnName1 0.0.0.0:9988,0.0.0.0:9999,0.0.0.0:9990 redis redis-key map[args1:value1 args2:value2] [] [funcName2 funcName2]}
Add FlowRouter FlowName=flowName1

context.Background
====> After CommitSrcData, flow_name = flowName1, flow_id = flow-bcaaa02a8d4b4a80b2f2895d9cecf20b
All Level Data =
 map[FunctionIdFirstVirtual:[This is Data1 from Test This is Data2 from Test This is Data3 from Test]]

KisFunctionV, flow = &{Id:flow-bcaaa02a8d4b4a80b2f2895d9cecf20b Name:flowName1 Conf:0xc00014eb00 Funcs:map[funcName1:0xc000114960 funcName2:0xc0001149c0 funcName3:0xc000114a20] FlowHead:0xc000114960 FlowTail:0xc000114a20 flock:{w:{state:0 sema:0} writerSem:0 readerSem:0 readerCount:0 readerWait:0} ThisFunction:0xc000114960 ThisFunctionId:func-37c7070f45144529891d433ae9c4ebfc PrevFunctionId:FunctionIdFirstVirtual funcParams:map[func-37c7070f45144529891d433ae9c4ebfc:map[] func-5315301ffbbb4ae4be021729ddff1569:map[] func-89a6a662729b4a0895e849c40bf29892:map[]] fplock:{w:{state:0 sema:0} writerSem:0 readerSem:0 readerCount:0 readerWait:0} buffer:[] data:map[FunctionIdFirstVirtual:[This is Data1 from Test This is Data2 from Test This is Data3 from Test]] inPut:[This is Data1 from Test This is Data2 from Test This is Data3 from Test]}

---> Call funcName1Handler ----
In FuncName = funcName1, FuncId = func-37c7070f45144529891d433ae9c4ebfc, row = This is Data1 from Test
In FuncName = funcName1, FuncId = func-37c7070f45144529891d433ae9c4ebfc, row = This is Data2 from Test
In FuncName = funcName1, FuncId = func-37c7070f45144529891d433ae9c4ebfc, row = This is Data3 from Test
context.Background
 ====> After commitCurData, flow_name = flowName1, flow_id = flow-bcaaa02a8d4b4a80b2f2895d9cecf20b
All Level Data =
 map[FunctionIdFirstVirtual:[This is Data1 from Test This is Data2 from Test This is Data3 from Test] func-37c7070f45144529891d433ae9c4ebfc:[data from funcName[funcName1], index = 0 data from funcName[funcName1], index = 1 data from funcName[funcName1], index = 2]]

KisFunctionS, flow = &{Id:flow-bcaaa02a8d4b4a80b2f2895d9cecf20b Name:flowName1 Conf:0xc00014eb00 Funcs:map[funcName1:0xc000114960 funcName2:0xc0001149c0 funcName3:0xc000114a20] FlowHead:0xc000114960 FlowTail:0xc000114a20 flock:{w:{state:0 sema:0} writerSem:0 readerSem:0 readerCount:0 readerWait:0} ThisFunction:0xc0001149c0 ThisFunctionId:func-5315301ffbbb4ae4be021729ddff1569 PrevFunctionId:func-37c7070f45144529891d433ae9c4ebfc funcParams:map[func-37c7070f45144529891d433ae9c4ebfc:map[] func-5315301ffbbb4ae4be021729ddff1569:map[] func-89a6a662729b4a0895e849c40bf29892:map[]] fplock:{w:{state:0 sema:0} writerSem:0 readerSem:0 readerCount:0 readerWait:0} buffer:[] data:map[FunctionIdFirstVirtual:[This is Data1 from Test This is Data2 from Test This is Data3 from Test] func-37c7070f45144529891d433ae9c4ebfc:[data from funcName[funcName1], index = 0 data from funcName[funcName1], index = 1 data from funcName[funcName1], index = 2]] inPut:[data from funcName[funcName1], index = 0 data from funcName[funcName1], index = 1 data from funcName[funcName1], index = 2]}

---> Call funcName2Handler ----
In FuncName = funcName2, FuncId = func-5315301ffbbb4ae4be021729ddff1569, row = data from funcName[funcName1], index = 0
===> In CaasDemoHanler1: flowName: flowName1, cName:ConnName1, fnName:funcName2, mode:Save
===> Call Connector CaasDemoHanler1, args from funciton: data from funcName[funcName1], index = 0
In FuncName = funcName2, FuncId = func-5315301ffbbb4ae4be021729ddff1569, row = data from funcName[funcName1], index = 1
===> In CaasDemoHanler1: flowName: flowName1, cName:ConnName1, fnName:funcName2, mode:Save
===> Call Connector CaasDemoHanler1, args from funciton: data from funcName[funcName1], index = 1
In FuncName = funcName2, FuncId = func-5315301ffbbb4ae4be021729ddff1569, row = data from funcName[funcName1], index = 2
===> In CaasDemoHanler1: flowName: flowName1, cName:ConnName1, fnName:funcName2, mode:Save
===> Call Connector CaasDemoHanler1, args from funciton: data from funcName[funcName1], index = 2
context.Background
 ====> After commitCurData, flow_name = flowName1, flow_id = flow-bcaaa02a8d4b4a80b2f2895d9cecf20b
All Level Data =
 map[FunctionIdFirstVirtual:[This is Data1 from Test This is Data2 from Test This is Data3 from Test] func-37c7070f45144529891d433ae9c4ebfc:[data from funcName[funcName1], index = 0 data from funcName[funcName1], index = 1 data from funcName[funcName1], index = 2] func-5315301ffbbb4ae4be021729ddff1569:[data from funcName[funcName2], index = 0 data from funcName[funcName2], index = 1 data from funcName[funcName2], index = 2]]

KisFunctionC, flow = &{Id:flow-bcaaa02a8d4b4a80b2f2895d9cecf20b Name:flowName1 Conf:0xc00014eb00 Funcs:map[funcName1:0xc000114960 funcName2:0xc0001149c0 funcName3:0xc000114a20] FlowHead:0xc000114960 FlowTail:0xc000114a20 flock:{w:{state:0 sema:0} writerSem:0 readerSem:0 readerCount:0 readerWait:0} ThisFunction:0xc000114a20 ThisFunctionId:func-89a6a662729b4a0895e849c40bf29892 PrevFunctionId:func-5315301ffbbb4ae4be021729ddff1569 funcParams:map[func-37c7070f45144529891d433ae9c4ebfc:map[] func-5315301ffbbb4ae4be021729ddff1569:map[] func-89a6a662729b4a0895e849c40bf29892:map[]] fplock:{w:{state:0 sema:0} writerSem:0 readerSem:0 readerCount:0 readerWait:0} buffer:[] data:map[FunctionIdFirstVirtual:[This is Data1 from Test This is Data2 from Test This is Data3 from Test] func-37c7070f45144529891d433ae9c4ebfc:[data from funcName[funcName1], index = 0 data from funcName[funcName1], index = 1 data from funcName[funcName1], index = 2] func-5315301ffbbb4ae4be021729ddff1569:[data from funcName[funcName2], index = 0 data from funcName[funcName2], index = 1 data from funcName[funcName2], index = 2]] inPut:[data from funcName[funcName2], index = 0 data from funcName[funcName2], index = 1 data from funcName[funcName2], index = 2]}

---> Call funcName3Handler ----
In FuncName = funcName3, FuncId = func-89a6a662729b4a0895e849c40bf29892, row = data from funcName[funcName2], index = 0
In FuncName = funcName3, FuncId = func-89a6a662729b4a0895e849c40bf29892, row = data from funcName[funcName2], index = 1
In FuncName = funcName3, FuncId = func-89a6a662729b4a0895e849c40bf29892, row = data from funcName[funcName2], index = 2
--- PASS: TestConfigImportYmal (0.01s)
PASS
ok      kis-flow/test   0.517s

The expected result is consistent with our expectations. Now we can load and build KisFlow through configuration files.

6.3 Configure Export

6.3.1 Implementation of Export

kis-flow/file/config_export.go

package file

import (
    "errors"
    "fmt"
    "gopkg.in/yaml.v3"
    "io/ioutil"
    "kis-flow/common"
    "kis-flow/kis"
)

// ConfigExportYaml exports flow configurations and stores them locally
func ConfigExportYaml(flow kis.Flow, savePath string) error {

    if data, err := yaml.Marshal(flow.GetConfig()); err != nil {
        return err
    } else {
        // Flow
        err := ioutil.WriteFile(savePath+common.KisIdTypeFlow+"-"+flow.GetName()+".yaml", data, 0644)
        if err != nil {
            return err
        }

        // Function
        for _, fp := range flow.GetConfig().Flows {
            fConf := flow.GetFuncConfigByName(fp.FuncName)
            if fConf == nil {
                return errors.New(fmt.Sprintf("function name = %s config is nil ", fp.FuncName))
            }

            if fdata, err := yaml.Marshal(fConf); err != nil {
                return err
            } else {
                if err := ioutil.WriteFile(savePath+common.KisIdTypeFunction+"-"+fp.FuncName+".yaml", fdata, 0644); err != nil {
                    return err
                }
            }

            // Connector
            if fConf.Option.CName != "" {
                cConf, err := fConf.GetConnConfig()
                if err != nil {
                    return err
                }
                if cdata, err := yaml.Marshal(cConf); err != nil {
                    return err
                } else {
                    if err := ioutil.WriteFile(savePath+common.KisIdTypeConnnector+"-"+cConf.CName+".yaml", cdata, 0644); err != nil {
                        return err
                    }
                }
            }
        }
    }

    return nil
}

6.3.2 New Interfaces in Flow

kis-flow/kis/flow.go

package kis

import (
    "context"
    "kis-flow/common"
    "kis-flow/config"
)

type Flow interface {
    // Run schedules the Flow, sequentially dispatches and executes Functions in the Flow
    Run(ctx context.Context) error
    // Link connects the Functions in the Flow according to the configuration file
    Link(fConf *config.KisFuncConfig, fParams config.FParam) error
    // CommitRow submits Flow data to the upcoming Function layer
    CommitRow(row interface{}) error
    // Input gets the input source data for the currently executing Function of the flow
    Input() common.KisRowArr
    // GetName gets the name of the Flow
    GetName() string
    // GetThisFunction gets the currently executing Function of the Flow
    GetThisFunction() Function
    // GetThisFuncConf gets the configuration of the currently executing Function of the Flow
    GetThisFuncConf() *config.KisFuncConfig
    // GetConnector gets the Connector of the currently executing Function of the Flow
    GetConnector() (Connector, error)
    // GetConnConf gets the configuration of the Connector of the currently executing Function of the Flow
    GetConnConf() (*config.KisConnConfig, error)

    // +++++++++++++++++++++++++++++++
    // GetConfig gets the configuration of the current Flow
    GetConfig() *config.KisFlowConfig
    // GetFuncConfigByName gets the configuration of the current Flow by Function name
    GetFuncConfigByName(funcName string) *config.KisFuncConfig
    // +++++++++++++++++++++++++++++++
}

Flow's new interfaces are implemented as follows:

kis-flow/flow/kis_flow.go

func (flow *KisFlow) GetConfig() *config.KisFlowConfig {
    return flow.Conf
}

// GetFuncConfigByName gets the configuration of the current Flow by Function name
func (flow *KisFlow) GetFuncConfigByName(funcName string) *config.KisFuncConfig {
    if f, ok := flow.Funcs[funcName]; ok {
        return f.GetConfig()
    } else {
        log.Logger().ErrorF("GetFuncConfigByName(): Function %s not found", funcName)
        return nil
    }
}

6.3.3 Fix in KisFlow's Funcs

There was a typo here before.

kis-flow/flow/kis_flow.go

// KisFlow is used to pass the context environment for the entire stream computing
type KisFlow struct {
    // Basic information
    Id   string                // Flow's distributed instance ID (used by KisFlow internally to distinguish different instances)
    Name string                // Flow's readable name
    Conf *config.KisFlowConfig // Flow configuration strategy

    // Function list
    Funcs          map[string]kis.Function // All managed Function objects of the current flow, key: FunctionName
    FlowHead       kis.Function            // Function list head owned by the current Flow
    FlowTail       kis.Function            // Function list tail owned by the current Flow
    flock          sync.RWMutex            // Lock for managing linked list insertion and reading and writing
    ThisFunction   kis.Function            // Currently executing KisFunction object of the Flow
    ThisFunctionId string                  // ID of the currently executing Function
    PrevFunctionId string                  // ID of the Function above the currently executing Function

    // Function list parameters
    funcParams map[string]config.FParam // Custom fixed configuration parameters of the flow in the current Function, Key: KisID of the function instance, value: FParam
    fplock     sync.RWMutex             // Lock for managing funcParams reading and writing

    // Data
    buffer common.KisRowArr  // Internal buffer used to temporarily store input byte data, one data is interface{}, multiple data is []interface{}, i.e., KisBatch
    data   common.KisDataMap // Data sources at various levels of stream computing
    inPut  common.KisRowArr  // Input data for the current Function calculation
}

The Funcs member here, its key meaning, was previously defined as KisID, and now it should be corrected to mean FunctionName.
Below is a simple correction to the code where Funcs member is assigned:

// appendFunc adds Function to Flow, linked list operation
func (flow *KisFlow) appendFunc(function kis.Function, fParam config.FParam) error {
    // ... ... 


    // Adds the detailed hash relationship of Function Name to the flow object
    flow.Funcs[function.GetConfig().FName] = function

    // ... ... 
}

6.3.4 Addition in KisPool

kis-flow/kis/pool.go

// GetFlows gets all the Flows
func (pool *kisPool) GetFlows() []Flow {
    pool.flowLock.RLock() // Read lock
    defer pool.flowLock.RUnlock()

    var flows []Flow

    for _, flow := range pool.flowRouter {
        flows = append(flows, flow)
    }

    return flows
}

KisPool adds a method to get all Flows to support the export module.

6.4 Configuration Export Unit Testing

Create the kis_config_export_test.go file in kis-flow/test/:

package test

import (
    "kis-flow/common"
    "kis-flow/file"
    "kis-flow/kis"
    "kis-flow/test/caas"
    "kis-flow/test/faas"
    "testing"
)

func TestConfigExportYmal(t *testing.T) {
    // 0. Register Function callback business
    kis.Pool().FaaS("funcName1", faas.FuncDemo1Handler)
    kis.Pool().FaaS("funcName2", faas.FuncDemo2Handler)
    kis.Pool().FaaS("funcName3", faas.FuncDemo3Handler)

    // 0. Register ConnectorInit and Connector callback business
    kis.Pool().CaaSInit("ConnName1", caas.InitConnDemo1)
    kis.Pool().CaaS("ConnName1", "funcName2", common.S, caas.CaasDemoHanler1)

    // 1. Load configuration file and build Flow
    if err := file.ConfigImportYaml("/Users/gopath/src/kis-flow/test/load_conf/"); err != nil {
        panic(err)
    }

    // 2. Export the configured in-memory KisFlow structure to files
    flows := kis.Pool().GetFlows()
    for _, flow := range flows {
        if err := file.ConfigExportYaml(flow, "/Users/gopath/src/kis-flow/test/export_conf/"); err != nil {
            panic(err)
        }
    }
}

Navigate to kis-flow/test/ and execute:

go test -test.v -test.paniconexit0 -test.run  TestConfigExportYmal

This will generate exported configurations under kis-flow/test/export_conf/:

├── export_conf
│   ├── conn-ConnName1.yaml
│   ├── flow-flowName1.yaml
│   ├── func-funcName1.yaml
│   ├── func-funcName2.yaml
│   └── func-funcName3.yaml

6.5 [V0.5] Source Code

https://github.com/aceld/kis-flow/releases/tag/v0.5

Author: Aceld
GitHub: https://github.com/aceld

KisFlow Open Source Project Address: https://github.com/aceld/kis-flow

Document: https://github.com/aceld/kis-flow/wiki

Case1-Quick Start
Case2-Flow Parallel Operation
Case3-Application of KisFlow in Multi-Goroutines
Case4-KisFlow in Message Queue (MQ) Applications

【Notice】Zinx open source license has been changed from GPL 3.0 to MIT.

Aceld — Tue, 07 May 2024 12:31:12 +0000

After discussions among Zinx developers, we have decided to change Zinx's open-source license from GPL 3.0 (GNU General Public License) to MIT (Massachusetts Institute of Technology License). This change is aimed at allowing more enterprises and individual developers to freely use Zinx in production environments and to satisfy their freedom of choice regarding open-source intentions.

(Suitable for all versions of Zinx development)

In brief,

"Zinx no longer mandates that derivative works be open-sourced. Applications using Zinx in enterprise or individual projects can be closed-source and freely used. Enterprises/developers have the freedom to choose whether to open-source their projects."

Credits to all developers for their continuous support of Zinx!

Zinx v1.2.5 Release

https://github.com/aceld/zinx/releases/tag/v1.2.5

Updates

LICENSE: Changed from GPL3.0 to MIT 692cfcf
Added a new module called RequestPool. #312
Refactored verifyLogIsolation() for a more concise expression. #287
Changed SendBuffMsg() to SendToQueue(). #295
Added several configuration options for setting up KCP connections. #299
In an independent coroutine, addclosecallback() can add any number of callback methods to conn, and execute them in the order they were added when conn.stop() is called. #301
Updated the way of cyclically registering HeartBeat routes. #304
The management of Request objects is changed to be handled by a sync pool object pool, and the usage of Request context is enhanced. #305
When WebSocket starts, console prompts. #311
Removed unnecessary info logs. #314

Fixes

Fixed [Panic]: send on closed channel. #318, #319
Fixed routerSlice deep copy. #294
Fixed go protobuf v1.26 msg.pb.go compatibility. #306
Fixed Request read-write lock to the lock itself instead of a pointer, reducing null safety risks. #307
Fixed htlv MaxFrameLength. 879a33e

Zinx Github: https://github.com/aceld/zinx