DEV Community: Tsuyoshi Ushio

[Update] Creating Nostalgic AOP Using Source Generator

Tsuyoshi Ushio — Mon, 15 Jan 2024 06:00:29 +0000

Introduction to Roslyn Source Generator

Roslyn Source Generator is used in various repositories, such as dotnet-isolated-worker. Unfortunately, I have never worked with it and couldn't understand the code. So, I decided to write a simple sample application to try and understand it.

The challenge is to create a mock version of AOP (Aspect Oriented Programming). I wanted to see if I could write a sample that injects logs before and after the execution of a function based on defined attributes.

What are Roslyn APIs?

Source Generator is part of the Roslyn APIs (The .NET Compiler Platform SDK). The Roslyn APIs allow the evaluation of source code during compilation to create a model of the application. These APIs are used to manipulate that model. Specifically, the following can be done:

Static code analysis
Code fixes
Code refactoring
Code generation

By using these features, Visual Studio can provide code suggestions and assist with code refactoring. This feature enables support for maintaining coding conventions.

.NET executes a pipeline similar to the one shown above during compilation. APIs are provided to support this pipeline:

Syntax Tree API: Access the tree model created by parsing the source code.
Symbol API: Access the symbols created as a result of parsing the code.
Binding and Flow Analysis APIs: Bind symbols to identifiers in the code.
Emit API: Output assemblies.

Comparing what Visual Studio can do with Roslyn APIs makes it easier to understand the capabilities.

Source Generator

For this investigation, I want to focus on Source Generator.

It allows you to perform the following steps:

Access the Compilation object to access and analyze the structure of the code.
Generate source code and compile it to create an assembly.

Specifically, you can:

Improve performance by performing tasks that are typically done with reflection using code generation.
Manipulate the order of execution of tasks in MSBuild.
Convert dynamic routing into static routing using code generation (used by ASP.NET).

Some use cases I have come across are:

Creating a definition file for logging is cumbersome, so perform code analysis and generate patterns for all of them.
Improve performance by generating code for methods that should be executed dynamically, referring to attributes.

How to use it

Create a source generator project

Reference the following libraries. Note that it seems to work only with netstandard2.0 currently.

<Project Sdk="Microsoft.NET.Sdk">

    <PropertyGroup>
        <TargetFramework>netstandard2.0</TargetFramework>
    </PropertyGroup>

    <ItemGroup>
        <PackageReference Include="Microsoft.CodeAnalysis.CSharp" Version="4.7.0" PrivateAssets="all" />
        <PackageReference Include="Microsoft.CodeAnalysis.Analyzers" Version="3.3.4" PrivateAssets="all" />
    </ItemGroup>
</Project>

Create a generator

To create a generator, implement the ISourceGenerator interface. You need to implement the Initialize(GeneratorInitializationContext context) and Execute(GeneratorExecutionContext context) methods. You can access the current code model and perform code generation through the context objects in each method. You can access the existing model using the context.Compilation object, and perform code generation using the context.AddSource method.

You can read, analyze the model, generate code according to your preferences, and compile it. In the following example, a factory is automatically created.

namespace SourceGenerator
{
    [Generator]
    public class FactoryGenerator : ISourceGenerator
    {
        public void Execute(GeneratorExecutionContext context)
        {
            // How to access the CompilationObject
            INamedTypeSymbol attributeSymbol = context.Compilation.GetTypeByMetadataName("AOPLib.FunctionAttribute");
            // Create Factory
            context.AddSource("FunctionFactory.g.cs", SourceText.From($@"
namespace AOPLib{{
  public class FunctionFactory {{
    public static IFunction CreateFunction(string name) {{
          switch(name) {{
            default:
              return new SampleFunction();
          }}
    }}
  }}
}}
", Encoding.UTF8));
        }

        public void Initialize(GeneratorInitializationContext context)
        {

        }

The caller would look like the following. We haven't created the code for the FunctionFactory class anywhere, but there will be no complaints from the IDE about the missing class.

    public static void Main(string[] args)
    {
        var functionName = Console.ReadLine();
        IFunction function = AOPLib.FunctionFactory.CreateFunction(functionName);
        function.Execute(new FunctionContext() { Name = "hello", TraceId = Guid.NewGuid() });
    }

In other words, the project containing the original Program.cs file, which acts as the caller, would have the following project definition. By setting OutputItemType=Analyzer and ReferenceOutputAssembly="false", the SourceGenerator project's DLL is not imported, and the analyzer is executed. You can find detailed definitions in Common MSBuild Project Items.

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net6.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
  </PropertyGroup>

  <ItemGroup>
    <ProjectReference Include="..\SourceGenerator\SourceGenerator.csproj"
                      OutputItemType="Analyzer"
                      ReferenceOutputAssembly="false"/>
  </ItemGroup>

</Project>

This is the basic usage of Source Generator.

Debugging with Source Generator

Debugging

By default, it is not possible to debug Source Generators directly. However, you can enable debugging in Visual Studio by calling the following method in the Initialize method:

public void Initialize(GeneratorInitializationContext context)
{
    Debugger.Launch();
}

Viewing Generated Source

By default, you cannot view the source code generated by the generator. However, you can enable it by adding the following definition to the csproj file of the project where the Program.cs is located. Add it to the PropertyGroup section:

  <EmitCompilerGeneratedFiles>true</EmitCompilerGeneratedFiles>

Creating an AOP Sample

The previous example was a simple one, but it would be better to try a slightly more complex sample. So let's implement a fake AOP (Aspect-Oriented Programming) by defining an attribute called Function. If the Logging flag in the attribute is set to true, it automatically outputs logs when the method is executed.

For example, if we write a program like this:

namespace AOPSample
{
    [Function(isLogging: true)]
    public class LoggingFunction : IFunction
    {
        public void Execute(FunctionContext context)
        {
            Console.WriteLine($"[{nameof(LoggingFunction)}]: execute. Name: {context.Name}");
        }
    }
}

Since the isLogging attribute is set to true, the Source Generator will automatically find this file and inject the logging logic.

To make it easier, let's write an adapter like this:

public class LoggingAdapter : IFunction
{
    private readonly IFunction _function;
    public LoggingAdapter(IFunction function)
    {
        _function = function;
    }
    public void Execute(FunctionContext context)
    {
        Console.WriteLine($"[{DateTime.UtcNow}] {_function.GetType().Name}: before execute. TraceId: {context.TraceId}");
        _function.Execute(context);
        Console.WriteLine($"[{DateTime.UtcNow}] {_function.GetType().Name}: after execute. TraceId: {context.TraceId}");
    }
}

Then, you can generate the code by creating an instance of LoggingAdapter and passing it to new ***Function().

Code Generation Logic

namespace SourceGenerator
{
    [Generator]
    public class FactoryGenerator : ISourceGenerator
    {
        public void Execute(GeneratorExecutionContext context)
        {
            // Get the symbol for the FunctionAttribute
            INamedTypeSymbol attributeSymbol = context.Compilation.GetTypeByMetadataName("AOPLib.FunctionAttribute");
            // Get all classes from Compilation.SyntaxTrees
            var targetClasses = context.Compilation.SyntaxTrees
                .SelectMany(st => st.GetRoot().DescendantNodes().OfType<ClassDeclarationSyntax>());
            // Extract only the classes that have the FunctionAttribute
            var selectedTargetClasses = targetClasses.Where(p => p.AttributeLists.FirstOrDefault()?.Attributes.FirstOrDefault()?.Name.NormalizeWhitespace().ToFullString() == "Function");
            // Save information about the selected classes as metadata in a dictionary
            var registeredClasses = new Dictionary<string, FunctionMetadata>();
            // Check the information of the Attribute used in the class.
            // Store the metadata information of the Function in the dictionary, such as `isLogging`.
            foreach (var clazz in selectedTargetClasses)
            {
                var attribute = clazz.AttributeLists.First()?.Attributes.FirstOrDefault();
                var attributeArgument = attribute.ArgumentList.Arguments.FirstOrDefault();
                var isLogging = attributeArgument.Expression.NormalizeWhitespace().ToFullString().Contains("true");
                var typeSymbol = context.Compilation.GetSemanticModel(clazz.SyntaxTree).GetDeclaredSymbol(clazz);
                var className = typeSymbol.ToDisplayString(SymbolDisplayFormat.FullyQualifiedFormat);
                var classNameOnly = typeSymbol.ToDisplayString(SymbolDisplayFormat.MinimallyQualifiedFormat);
                registeredClasses.Add(classNameOnly, new FunctionMetadata { FullyQualifiedName = className, Name = classNameOnly, IsLogging = isLogging });
            }

            // Create the code generation template and generate the code
            context.AddSource("FunctionFactory.g.cs", SourceText.From($@"
namespace AOPLib{{
  public class FunctionFactory {{
    public static IFunction CreateFunction(string name) {{
          switch(name) {{
            {GetFunctionsSection(registeredClasses)}
            default:
              return new SampleFunction();
          }}
    }}
  }}
}}
", Encoding.UTF8));
        }

        public void Initialize(GeneratorInitializationContext context)
        {
            Debugger.Launch();
        }

        private string GetFunctionsSection(Dictionary<string, FunctionMetadata> metadata)
        {
            var sb = new StringBuilder();
            foreach (var item in metadata)
            {
                sb.AppendLine($"case \"{item.Key}\":");
                if (item.Value.IsLogging)
                {
                    sb.AppendLine($"return new LoggingAdapter(new {item.Value.FullyQualifiedName}());");
                }
                else
                {
                    sb.AppendLine($"return new {item.Value.FullyQualifiedName}();");
                }
            }
            return sb.ToString();
        }
    }

    class FunctionMetadata
    {
        public string Name { get; set; }
        public string FullyQualifiedName { get; set; }
        public bool IsLogging { get; set; }
    }
}

Execution Result

Although no code was written in the main program, logs are automatically outputted as expected with the fake AOP.

If isLogging is set to false, no logs will be outputted.

IncrementalGenerators

Now, the interface I used in the above example is an official method, but it seems to be an older method. I received feedback from someone with worldwide expertise. It's amazing to receive feedback from such people when I write a blog!

https://x.com/neuecc/status/1703657375394341330?s=20

You should probably read the article C# in 2022 (Incremental) Source Generator Development Method and the official documentation on Incremental Generators. Let's explain the new interface.

IIncrementalGenerator

Compared to ISourceGenerator, it only has

Porting the AOP Sample

Now that we have learned the new approach, let's rewrite the AOP sample with Incremental Generators.
Great! It has become much easier to understand! I will add explanations within the code. The process follows the same flow as explained earlier: we process the IncrementalValue(s)Provider<T> using a Linq-like interface, manipulate it, and pass the final result to RegisterOutput to generate the files. The key point is to realize that it may look like Linq, but it's not actually Linq. By correctly reading about Incremental Generators, you will understand that it's not about handling collections, so it can also return a single value. That's where IncrementalValueProvider<T> comes in for singular values, and IncrementalValuesProvider<T> for multiple values.

I will add comments as needed throughout the code.

namespace SourceGenerator
{
    [Generator(LanguageNames.CSharp)]
    public class FactoryGenerator : IIncrementalGenerator
    {
        public void Initialize(IncrementalGeneratorInitializationContext context)
        {
            // Uncomment this line if you want to debug.
            // Debugger.Launch();
            // ExtensionMethod ForAttributeWithMetadataName() is awesome!
            // It returns all classes with the specified attribute.
            var source = context.SyntaxProvider.ForAttributeWithMetadataName(
                "AOPLib.FunctionAttribute",
                (node, token) => true,
                (ctx, token) => ctx
                )
                // It can be written in a similar way to Linq. Here we analyze the classes and 
                // retrieve the values from the attributes attached to the classes and store 
                // them in FunctionMetadata.
                .Select((s, token) => { 
                    var typeSymbol = (INamedTypeSymbol)s.TargetSymbol;
                    var fullType = typeSymbol.ToDisplayString(SymbolDisplayFormat.FullyQualifiedFormat);
                    var name = typeSymbol.ToDisplayString(SymbolDisplayFormat.MinimallyQualifiedFormat);
                    var isLogging = (bool)(s.Attributes.FirstOrDefault()?.ConstructorArguments.FirstOrDefault().Value ?? false);
                    return new FunctionMetadata { FullyQualifiedName = fullType, Name = name, IsLogging = isLogging };
            // Finally, we perform Collect() to convert it to IncrementalValueProvider.
            // It was IncrementalValuesProvider<FunctionMetadata> before, but now it
            // has been converted to IncrementalValueProvider<ImmutableArray<FunctionMetadata>>.
            // We convert it from plural to singular to ensure the Action is only called once.
            }).Collect();

            context.RegisterSourceOutput(source, Emit);
        }

        // Since we have converted it to a singular form, this method will be called only once.
        // Without Collect(), it would be called multiple times.
        private void Emit(SourceProductionContext context, ImmutableArray<FunctionMetadata> source)
        {
            // Create Factory
            context.AddSource("FunctionFactory.g.cs", SourceText.From($@"
namespace AOPLib{{
  public class FunctionFactory {{
    public static IFunction CreateFunction(string name) {{
          switch(name) {{
            {GetFunctionsSection(source)}
            default:
              return new SampleFunction();
          }}
    }}
  }}
}}
", Encoding.UTF8));
        }

        private string GetFunctionsSection(ImmutableArray<FunctionMetadata> metadata)
        {
            var sb = new StringBuilder();
            foreach (var item in metadata)
            {
                sb.AppendLine($"case \"{item.Name}\":");
                if (item.IsLogging)
                {
                    sb.AppendLine($"return new LoggingAdapter(new {item.FullyQualifiedName}());");
                }
                else
                {
                    sb.AppendLine($"return new {item.FullyQualifiedName}();");
                }
            }
            return sb.ToString();
        }
    }

    class FunctionMetadata
    {
        public string Name { get; set; }
        public string FullyQualifiedName { get; set; }
        public bool IsLogging { get; set; }
    }
}

Compared to the previous version, it looks much cleaner. By the way, let's compare the case where IIncrementalGenerator is available and when it's not, like this:

Summary

This generator enables us to execute code that would normally be done using reflection in a static manner, or analyze the structure of the code. It is a very useful and interesting feature. However, I have only scratched the surface, and I think it may be difficult to detect bugs in the generated code. It seems that it would be necessary to write unit tests or other means to cover those cases.

~~However, since I am still new to this, I can't say that I fully understand it yet. I will continue to investigate and write more samples.~~

I have provided the sample used in this blog post here:

AOP Sample

Resources

This blog is translated from [更新] Source Generator を使って懐かしの AOP を作ってみる Powered by BlogBabel

Kafka ScaledJobs sample for KEDA with EventHubs

Tsuyoshi Ushio — Tue, 09 Feb 2021 07:38:51 +0000

I wanted to create a sample for testing Kafka Scaler with ScaledJob. It requires simple go client that will finish once the message has consumed. The reason why I use go is, KEDA Kafka Scaler using Sarama for implementing Kafka Scaler. So that I thought it might help to understand the kafka scaler implementation.

Kafka Broker

I use the Kafka Broker as EventHubs for easily to create a broker. We can use EventHubs with Kafka API.

Use Azure Event Hubs from Apache Kafka applications

Configuration

Sarama

Configuration for Sarama with EventHubs are like this. You need to pass EventHubs Connection String as config.Net.SASL.Password. Other consideration, is, config.Consumer.Group.Session.Timeout If you don't configure the value, the client is easy to returns panic: Error from consumer: read tcp 172.26.24.68:34288->40.78.250.71:9093: i/o timeout. By configuring this value, your client works fine.

EventHubs Kafka Trouble Shooting

config := sarama.NewConfig()
config.Version = sarama.V1_0_0_0
config.Consumer.Group.Rebalance.Strategy = sarama.BalanceStrategyRange
config.Net.SASL.Enable = true
config.Net.SASL.User = "$ConnectionString"
config.Net.SASL.Password = os.Getenv(saslPassword)
config.Consumer.Group.Session.Timeout = 60 * time.Second
config.Net.TLS.Enable = true

consumer := Consumer{
    ready: make(chan bool),
}

ctx, cancel := context.WithCancel(context.Background())
client, err := sarama.NewConsumerGroup(strings.Split(brokers, ","), group, config)
if err != nil {
    log.Panicf("Error creating consumer group client: %v", err)
}

For the Scaled Job, I'd like to finish the pod once the app receive a message. We need to implement callback Struct for processing message. It is rough implementation for sample, however, I notify the message consuming by channel.


// Consumer struct
type Consumer struct {
    ready chan bool
}

// Setup method
func (consumer *Consumer) Setup(sarama.ConsumerGroupSession) error {
    close(consumer.ready)
    return nil
}

// Cleanup method
func (consumer *Consumer) Cleanup(sarama.ConsumerGroupSession) error {
    return nil
}

// ConsumeClaim method
func (consumer *Consumer) ConsumeClaim(session sarama.ConsumerGroupSession, claim sarama.ConsumerGroupClaim) error {
    for message := range claim.Messages() {
        log.Printf("Message claimed: value = %s, timestamp = %v, topic = %s\n", string(message.Value), message.Timestamp, message.Topic)
        session.MarkMessage(message, "")
        consumed <- true
        break
    }
    return nil
}

Scaled Job

Secret

For the Kafka Scaled Job, I create two secrets. One for Authentication for the Kafka Scaler, the other is Environment Variables for the Pod. Replace the value of YOUR_EVENT_HUBS_CONNECTION_STRING_BASE64 with your EventHubs connection string.Don't forget to convert into base64.

secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: keda-kafka-secrets
  namespace: default
data:
  sasl: "plaintext as base64"
  username: "$ConnectionString as base64"
  password: "YOUR_EVENT_HUBS_CONNECTION_STRING_BASE64"
  tls: "enable as base64"

pod-secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: keda-kafka-pod-secrets
  namespace: default
data:
  SASL_PASSWORD: "YOUR_EVENT_HUBS_CONNECTION_STRING_BASE64"
  BROKER_LIST: "BROKER_LIST_BASE64"
  TOPICS: "TOPIC_BASE64"

ScaledJob

apiVersion: keda.sh/v1alpha1
kind: ScaledJob
metadata: 
  name: kafka-consumer
  namespace: default
spec:
  jobTargetRef:
    template:
      spec:
        containers:
        - name: kafka-consumer
          image: tsuyoshiushio/kafka-consumer:0.2
          imagePullPolicy: Always
          envFrom:
          - secretRef:
              name: keda-kafka-pod-secrets
        restartPolicy: Never
  pollingInterval: 5
  maxReplicaCount: 10
  successfulJobsHistoryLimit: 10
  failedJobsHistoryLimit: 10
  scalingStrategy:
    strategy: accurate
  triggers:
  - type: kafka
    metadata:
      bootstrapServers: MY_EVENT_HUB_NAMESPACE.servicebus.windows.net:9093
      consumerGroup: $Default
      topic: multi-topic
      lagThreshold: '5'
    authenticationRef:
      name: kafka-trigger-auth
---
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: kafka-trigger-auth
  namespace: default
spec:
  secretTargetRef:
  - parameter: sasl
    name: keda-kafka-secrets
    key: sasl
  - parameter: username
    name: keda-kafka-secrets
    key: username
  - parameter: password
    name: keda-kafka-secrets
    key: password    
  - parameter: tls
    name: keda-kafka-secrets
    key: tls

Run on kubernets

Install KEDA and make sure
two keda pods working. You can see the operator logs by kubectl logs -f keda-operator-6b546dc696-gzc72 in this case. It helps to understand how keda scale jobs works.

kubectl get pods -n keda
NAME                                      READY   STATUS    RESTARTS   AGE
keda-metrics-apiserver-6dff8c4f7f-z9szt   1/1     Running   0          5d1h
keda-operator-6b546dc696-gzc72            1/1     Running   0          51m

Clone and build and publish the image on samples. KafkaScaledJobWithGo

$ docker build -f tsuyoshiushio/kafka-consumer:0.2
$ docker push tsuyoshiushio/kafka-consumer:0.2

If you update the version or name, change the scaledjob.yaml on the image section.

Apply the Secret and ScaledJob.

$ kubectl apply -f secret.yaml
$ kubectl apply -f pod-secret.yaml
$ kubectl apply -f scaledjob.yaml

Then send message to the target EventHub, then you will see the job is created according to the number of the messages.

$ kubectl get pods
NAME                                   READY   STATUS      RESTARTS   AGE
func-deployment-http-7dffc56bc-6494h   1/1     Running     0          4d8h
kafka-consumer-5bmdd-dwb44             0/1     Completed   0          73s
kafka-consumer-n4vvx-gwrhf             0/1     Completed   0          82s
kafka-consumer-t2hth-cthlh             1/1     Running     0          64s
kafka-consumer-vzh4d-rjjt9             0/1     Completed   0          91s

Resource

Enabling KEDA debugging

Tsuyoshi Ushio — Wed, 03 Feb 2021 04:57:54 +0000

If you are contributing the KEDA, you might be wondering how we can debug it. I'd like to share the tips for debugging KEDA.

We have several ways to do it.

Remote Debugging

Remote debugging is not very recommended since it is relatively slow, however, sometimes, it requires. KEDA has two servers. one is keda-operator that has reconcile loop. It ask the infromation to the keda-metrics-apiserver that is only reside in the k8s cluster. In case you want to debug keda-metrics-apiserver we need the remote debugging.

Create a Dockerfile

Create a Dockerfile that enalbes us to debug. It is just adding delv and build with gcflags "-N -l". then start the execution file with wrapping with dlv command with exposing port.

ENTRYPOINT ["/dlv", "--listen=:4040", "--headless=true", "--api-version=1", "--continue", "--accept-multiclient", "exec", "/workspace/keda-adapter", "--", "--secure-port=6443", "--logtostderr=true", "--v=0"]

The parameters means following. -- means passing arguments to the executable (in this case, keda-adapter)

$ dlv --help
Delve is a source level debugger for Go programs.

Delve enables you to interact with your program by controlling the execution of the process,
evaluating variables, and providing information of thread / goroutine state, CPU register state and more.

The goal of this tool is to provide a simple yet powerful interface for debugging Go programs.

Pass flags to the program you are debugging using `--`, for example:

`dlv exec ./hello -- server --config conf/config.toml`

Usage:
  dlv [command]

Available Commands:
  attach      Attach to running process and begin debugging.
  connect     Connect to a headless debug server.
  core        Examine a core dump.
  dap         [EXPERIMENTAL] Starts a TCP server communicating via Debug Adaptor Protocol (DAP).
  debug       Compile and begin debugging main package in current directory, or the package specified.
  exec        Execute a precompiled binary, and begin a debug session.
  help        Help about any command
  run         Deprecated command. Use 'debug' instead.
  test        Compile test binary and begin debugging program.
  trace       Compile and begin tracing program.
  version     Prints version.

Flags:
      --accept-multiclient   Allows a headless server to accept multiple client connections.
      --api-version int      Selects API version when headless. New clients should use v2. Can be reset via RPCServer.SetApiVersion. See Documentation/api/json-rpc/README.md. (default 1)
      --backend string       Backend selection (see 'dlv help backend'). (default "default")
      --build-flags string   Build flags, to be passed to the compiler.
      --check-go-version     Checks that the version of Go in use is compatible with Delve. (default true)
      --headless             Run debug server only, in headless mode.
      --init string          Init file, executed by the terminal client.
  -l, --listen string        Debugging server listen address. (default "127.0.0.1:0")
      --log                  Enable debugging server logging.
      --log-dest string      Writes logs to the specified file or file descriptor (see 'dlv help log').
      --log-output string    Comma separated list of components that should produce debug output (see 'dlv help log')
      --only-same-user       Only connections from the same user that started this instance of Delve are allowed to connect. (default true)
      --wd string            Working directory for running the program. (default ".")

Additional help topics:
  dlv backend Help about the --backend flag.
  dlv log     Help about logging flags.

Use "dlv [command] --help" for more information about a command.

Docker.remote

FROM golang:1.15 as builder

WORKDIR /workspace

RUN go get -u github.com/go-delve/delve/cmd/dlv

# Copy the Go Modules manifests
COPY go.mod go.mod
COPY go.sum go.sum
# cache deps before building and copying source so that we don't need to re-download as much
# and so that source changes don't invalidate our downloaded layer
RUN go mod download

COPY Makefile Makefile

# Copy the go source
COPY hack/ hack/
COPY version/ version/
COPY main.go main.go
COPY adapter/ adapter/
COPY api/ api/
COPY controllers/ controllers/
COPY pkg/ pkg/

RUN mkdir -p /apiserver.local.config/certificates && chmod -R 777 /apiserver.local.config

# Build
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 GO111MODULE=on go build -gcflags "-N -l" -o keda-adapter adapter/main.go

FROM ubuntu:18.04

WORKDIR /
COPY --from=builder /apiserver.local.config  /apiserver.local.config
COPY --from=builder /workspace /workspace
COPY --from=builder /go/bin/dlv /dlv

ENTRYPOINT ["/dlv", "--listen=:4040", "--headless=true", "--api-version=1", "--continue", "--accept-multiclient", "exec", "/workspace/keda-adapter", "--", "--secure-port=6443", "--logtostderr=true", "--v=0"]

Then build and publish to your dockerhub account. In my case,

$ docker login
$ docker build -f Docker.remote -t tsuyoshiushio/adapter:1
$ docker push tsuyoshiushio/adapter:1

Update the deployment

We can see the two deployment is available on the k8s cluster.

ushio@DESKTOP-KIUTRHV:~/Code/keda$ kubectl get deployment -n keda
NAME                     READY   UP-TO-DATE   AVAILABLE   AGE
keda-metrics-apiserver   1/1     1            1           6h2m
keda-operator            1/1     1            1           6h2m

let's edit the keda-metrics-apiserver.

$ kubectl edit deployment keda-metrics-apiserver -n keda

let's change the image part.

    spec:
      containers:
      - args:
        - /usr/local/bin/keda-adapter
        - --secure-port=6443
        - --logtostderr=true
        - --v=0
        env:
        - name: WATCH_NAMESPACE
        image: docker.io/tsuyoshiushio/adapter:3    // <- Change
        imagePullPolicy: Always

then port forward it.

$ kubectl get pods -n keda
NAME                                      READY   STATUS    RESTARTS   AGE
keda-metrics-apiserver-6d5689599f-884x9   1/1     Running   0          68s
keda-operator-6b546dc696-cxdbq            1/1     Running   0          5h49m
$ kubectl port-forward keda-metrics-apiserver-6d5689599f-884x9  -n keda 4040:4040
Forwarding from 127.0.0.1:4040 -> 4040
Forwarding from [::1]:4040 -> 4040

For VS Code, you can configure launch.json settings to attach remote server.

        {
            "name": "Remote debug",
            "type": "go",
            "request": "attach",
            "mode": "remote",
            "remotePath": "/workspace",
            "port": 4040,
            "host": "127.0.0.1"
        }

However, it is sometimes, hard to wait very slow response. I recommend to go with Local debugging and adding log to see the remote container.

Local Debugging

Set the launch.json with VSCode. Just start the main.go on the root of the KEDA repo. We can not use keda-metrics-apiserver since it reside in the cluster. However, we can do the local debugging for the keda-operator.

        {
            "name": "Connect to server",
            "type": "go",
            "request": "launch",
            "mode": "auto",
            "program": "${workspaceFolder}/main.go",
            "env": {
                "WATCH_NAMESPACE":""
            }
        },

Build and publish KEDA to your kubernets cluster

make undeploy will remove the keda resources. make deploy deploy the keda resources to the cluster when you in development.

$ kubectl get all -n keda
NAME                                          READY   STATUS    RESTARTS   AGE
pod/keda-metrics-apiserver-6dff8c4f7f-8qc9v   1/1     Running   0          30s
pod/keda-operator-6b546dc696-pf5ck            1/1     Running   0          29s

NAME                             TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)          AGE
service/keda-metrics-apiserver   ClusterIP   10.0.16.80   <none>        443/TCP,80/TCP   31s

NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/keda-metrics-apiserver   1/1     1            1           30s
deployment.apps/keda-operator            1/1     1            1           29s

NAME                                                DESIRED   CURRENT   READY   AGE
replicaset.apps/keda-metrics-apiserver-6dff8c4f7f   1         1         1       31s
replicaset.apps/keda-operator-6b546dc696            1         1         1       30s

Using Fiddler for Application Insights codeless java agent with docker on WSL2

Tsuyoshi Ushio — Thu, 17 Dec 2020 09:24:36 +0000

I'm working on Application Insights - Java codeless agent working on a docker container on WSL2. It works on the other guys machine, however, on my laptop, it sometimes work, most of the times, it doesn't work.

My colleague gave me an advice to use Fiddler. I use it for watching network requests that is coming from a docker container on WSL2.

Let's see how to configure it.

Strategy Overview

We are configure the Java Application Insights agent for proxying to the Fiddler. We need to answer these questions.

How to extract a root certificate from Fiddler and import it to JVM KeyStore on Docker. (I use zulu)
How to send request from WSL2 to Windows 10
How to configure Application Insights agent to proxy to the Fiddler.
Is there any other configuration required?

Export a root certificate from Fiddler

Go to Tools > Options > HTTPS > Actions then select Export Root Certificate to Desktop. Then move the file to WSL2 side.

From the wsl2 side I do this after the moving the file to Download.

$ mv /mnt/c/Users/tsushi/Download/FiddlerRoot.crt

Import the certificate to the KeyStore on JVM

We can use keytool for import the certificate to the JVM KeyStore. In case of zulu, the KeyStore is /usr/lib/jvm/zre-8-azure-amd64/lib/security/cacerts by default. The default password of the KeyStore is changeit.

COPY FiddlerRoot.cer /
RUN /usr/lib/jvm/zre-8-azure-amd64/bin/keytool -import -noprompt -trustcacerts -alias FiddlerRoot -file /FiddlerRoot.cer -keystore /usr/lib/jvm/zre-8-azure-amd64/lib/security/cacerts -storepass changeit

Configure Application Insights Codeless Agent for Proxy

Find IPAddress for accessing from WSL2 to Windows 10

To accessing from WSL2 to Windows 10, you can open Network Connection from your Type here to search box on your Windows 10.
You can find a network adopter vEthernet (WSL), Find the IP address. The IP Address is the address that is seen it as Windows10 from WSL2. In my case it is, 172.20.64.1.

You can find the Fiddler port number in here.

Configure the codeless agent

Configure the proxy section with the IP Address and port number. Then the Java codeless agent will send request to the proxy (Fiddler).

{
  "proxy":{
    "host": "172.20.64.1",
    "port": 8888
  },
  "selfDiagnostics": {
    "destination": "file+console",
    "level": "INFO",
    "file": {
      "path": "/var/log/applicationinsights/applicationinsights.log",
      "maxSizeMb":  5,
      "maxHistory": 1
    }
  }
}

Configure Fiddler to accept the request from WSL2

Configure the Allow remote computers to connect From the Windows 10 Fiddler, WSL2 is different computer.

Start the Fiddler, and run the Docker container. You will find the two requests. These are the Application Insights telemetry.

rt.services.visualstudio.com
{location}-{number}.in.applicationinsights.azure.com

Resources

Solves 0 function found on Azure Function - Linux Consumption plan

Tsuyoshi Ushio — Fri, 11 Dec 2020 08:25:31 +0000

I encounter an issue related archive. I trip to Japan. I use a desktop machine in Seattle, however, now I'm using my old laptop. I replaced the laptop with the desktop as my main machine.

The behavior is, I use the same Azure Functions sample. I crone the repo, In this case, the app is Java. I build it with mvn clean package then I archive it to a zip file. like this.

# Create a zip file
$currentDir = pwd
$zipFile = join-path $currentDir sample-cold.zip

Compress-Archive -Path '.\target\azure-functions\sample\*' -DestinationPath $zipFile -Force

I use the same logic and create an archive. However, the archive I created on my desktop works, but not for a zip from laptop.

When I test both archives, extract these, it looks the same. I move the archive to WSL2 and extract these with unzip. It looks the same. According to the Azure Functions Host log, it says

0 functions found

It is the issue of the archive. I did docker exec and find the root cause. When I go to /home/site/wwwroot I found the files that is downloaded and extracted zip file.

root@658408192cc6:~/site/wwwroot# ls -l
total 57867
-rw-rw-r-- 1 app app      354 Dec 10 11:51 'HttpTrigger-Java\function.json'
-rw-rw-r-- 1 app app     5267 Dec 10 11:51  codeless-azure-function-Test-1.0-SNAPSHOT.jar
-rw-rw-r-- 1 app app       24 Dec 10 19:42  host.json
-rw-rw-r-- 1 app app   317195 Dec 10 11:51 'lib\FastInfoset-1.2.16.jar'
-rw-rw-r-- 1 app app    30035 Dec 10 11:51 'lib\accessors-smart-1.2.jar'
-rw-rw-r-- 1 app app    62983 Dec 10 11:51 'lib\activation-1.1.jar'
-rw-rw-r-- 1 app app    94485 Dec 10 11:51 'lib\adal4j-1.6.2.jar'
-rw-rw-r-- 1 app app    17079 Dec 10 11:51 'lib\adapter-rxjava-2.4.0.jar'
-rw-rw-r-- 1 app app   136324 Dec 10 11:51 'lib\aether-api-1.0.2.v20150114.jar'
-rw-rw-r-- 1 app app   172998 Dec 10 11:51 'lib\aether-impl-1.0.2.v20150114.jar'
-rw-rw-r-- 1 app app    30705 Dec 10 11:51 'lib\aether-spi-1.0.2.v20150114.jar'

It is weird, It was not the directly.

Solution

I found the issue was Compress-Archive command let version. The version I had issue is

> $Host.Version

Major  Minor  Build  Revision
-----  -----  -----  --------
5      1      19041  610

I update the version to

> $host.version

Major  Minor  Build  Revision
-----  -----  -----  --------
7      1      0      -1

This version looks work. Now I can see the directory.

The problem of the extract zip only happens with Azure Functions Host. Probably it is because of the library on C#.

I can see several Compress-Archive command let issue with Linux.
Compress-Archive on linux creates zip files with no file permissions.

Happy coding with Azure Functions!

Enabling Memory Profiling on KEDA v2

Tsuyoshi Ushio — Tue, 20 Oct 2020 18:57:43 +0000

I have a minor memory leak issue to solve. So that I'm wondering how can we enable profiler on KEDA. KEDA uses Operator SDK.

Go lang has a good profiler.

It is very simple to use it. Just add this line for memory profiling.

Configuration and Issue

According to the documentation, the configuration will be like this.

go get github.com/pkg/profile

then add following line to the main.go

import (
    //...
    "github.com/pkg/profile"
)
   :
defer profile.Start(profile.MemProfile).Stop()

Run the KEDA on my local machine. However, it doesn't work. The root cause is, we need to stop the operator sdk with ctr+c. In this case, defer method doesn't execute Stop() method.

ushio@DESKTOP-KIUTRHV:~/Code/keda$ make run ARGS="--zap-log-level=debug"
/home/ushio/go/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go run \
-ldflags "-X=github.com/kedacore/keda/version.GitCommit=d0788b943ecf94f7f7c5b1a09f777cc3af6b746e -X=github.com/kedacore/keda/version.Version=v2" \
./main.go --zap-log-level=debug
2020/10/20 09:19:49 profile: memory profiling enabled (rate 4096), /tmp/profile174831365/mem.pprof
                   :
^C2020/10/20 09:20:02 profile: caught interrupt, stopping profiles
Exit 0 for profiling
make: *** [Makefile:96: run] Interrupt

$ ls -l /tmp/profile287657069/mem.pprof
-rw-r--r-- 1 ushio ushio 0 Oct 20 09:18 /tmp/profile287657069/mem.pprof

Solution

If you want to handle the SIG interrupt, you can use signal.Notify method for handling the signal.

func SetupCloseHandler() {
    c := make(chan os.Signal)
    signal.Notify(c, os.Interrupt, syscall.SIGTERM)
    go func() {
        <-c
        fmt.Println("\r- Ctrl+C pressed in Terminal")
        DeleteFiles()
        os.Exit(0)
    }()
}

I read the main.go they are also having the signal handling part. Let's change it to call profile.Stop() method.

func main() {
    // Memory profiling
    f := profile.Start(profile.MemProfile, profile.NoShutdownHook)
    :
    //  if err := mgr.Start(ctrl.SetupSignalHandler()); err != nil {
    //      setupLog.Error(err, "problem running manager")
    //      os.Exit(1) // Profiling test
    //  }
    fmt.Println("-----SetupCloseHandler")
    setupSignalHandler := func() (stopCh <-chan struct{}) {
        stop := make(chan struct{})
        c := make(chan os.Signal, 2)
        signal.Notify(c, os.Interrupt, syscall.SIGTERM)
        go func() {
            <-c
            fmt.Println("Exit 0 for profiling")
            f.Stop()
            os.Exit(1) // second signal. Exit directly.
        }()

        return stop
    }

    if err := mgr.Start(setupSignalHandler()); err != nil {
        setupLog.Error(err, "problem running manager")
        os.Exit(1)
    }
}

Run it. Works!

$ ls -l /tmp/profile388737932/mem.pprof
-rw-r--r-- 1 ushio ushio 47888 Oct 20 09:31 /tmp/profile388737932/mem.pprof

Viewer

The next step will be the view the analysis report.
It looks requires graphviz. Install it.

$ go tool pprof --pdf /tmp/profile356144532/mem.pprof > file.pdf
failed to execute dot. Is Graphviz installed? Error: exec: "dot": executable file not found in $PATH
$ sudo apt install graphviz

Works!

$ go tool pprof --pdf /tmp/profile356144532/mem.pprof > file.pdf

I don't know if there is better way to enable profiling using Operator SDK. Once I know it, I'll update this blog.

Reference

Handle Ctrl+C (Signal Interrupt) Close in the Terminal

Debugging Java Language Worker on Azure Functions

Tsuyoshi Ushio — Sat, 15 Aug 2020 07:47:31 +0000

If you want to contribute Java Language worker, how to debug it? It works with Azure Functions Host.

There is three player for this debug story.

Architecture

1. Azure Functions Host (C#)

The main process of the Azure Functions execution. Under the food, it uses WebJobsSDK.

2. Binding Extensions (C# + wrapper)

Azure functions has a several Triggers and bindings. It will receive an event by the various measure. Http, Queue, Kafka, EventHubs, CosmosDB ... and so on. It also outputs messages. It is written by C#. Language implementation is an wrapper of the C# implementation.

3. Java Language Worker (Java)

Java Language Worker is a worker process. Except for C# precompile implementation, Azure Functions host delegate the function execution to each language workers via gRPC.

4. Azure Functions Java Library (Java)

That is mainly the definition of annotation that is used by the Binding Extensions wrapper. The library is referenced by User's Azure Functions, Java Language Worker, and Azure Functions Maven Plugin.

5. Application Insights Agent (Java)

It is injected with the Java Language Worker as an agent. It is automatically track the telemetries with correlation without changing any code. The automated dependency tracking is limited for some libraries. Please refer to it on this article.

Java codeless application monitoring Azure Monitor Application Insights - public preview

6. Azure Functions (Java)

Azure Functions that customers write. The Language worker class loader load the functions by the annotations.

How to debug it?

Since there is a lot of players, how can I debug the Java Worker part? We can easily debug the Azure Functions part. However, Java Worker is started by the Azure Functions Host.

Create an Azure Functions

For debugging the Java Language Worker, we need create a function to execute it. Please refer to Quickstart: Create a function in Azure that responds to HTTP requests

One point for the debugging is, how to manage the Binding extensions. Usually, we use Extension bundles. So we don't need to install the Bindings by setting the host.json. It will automatically install the extensions.

{
    "version": "2.0",
    "extensionBundle": {
        "id": "Microsoft.Azure.Functions.ExtensionBundle",
        "version": "[1.*, 2.0.0)"
    }
}

However, the local debug scenario, you might need to install the extension by your self. You need to have extension.csproj file under the, target/azure-functions/<YOUR_FUNCTION_APP_NAME> directory.

Install extension

This code is a sample for EventHubs

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>netcoreapp3.1</TargetFramework>
    <WarningsAsErrors></WarningsAsErrors>
    <DefaultItemExcludes>**</DefaultItemExcludes>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.Azure.WebJobs.Extensions.EventHubs" Version="4.1.1" />
    <PackageReference Include="Microsoft.Azure.WebJobs.Script.ExtensionsMetadataGenerator" Version="1.1.7" />
  </ItemGroup>
</Project>

Then go to the target/azure-functions/<FUNCTION_APP_NAME> dir.

PS > mvn clean package
PS > cp extension.csproj target\azure-functions\fabrikam-functions
PS > cd target\azure-functions\fabrikam-functions
PS > func extension install

You will see the bin directory that contains DLL files for the extensions. The DLL is libraries of the Extension.

Now ready to configure Azure Functions Host.

Configure Azure Functions Host for Java Worker Debugging

WebJobs.Script.WebHost as startup

Clone the Azure Functions Host. Then open Visual Studio and make WebJobs.Script.WebHost as startup project.

Configure Environment Variables

Right click WebJobs.Script.WebHost > Properties > Debug. Configure the environment variables.

Name	Value (sample)	Description
AzureWebJobsScriptRoot	C:\Users\tsushi\Code\java\azure-functions-samples\EventHubs\target\azure-functions\fabrikam-functions-20200814172147278	Point to the azure functions function app directory
ASPNETCORE_ENVIRONMENT	Development	Mode for AspNetCore
AZURE_FUNCTIONS_ENVIRONMENT	Development	Mode for Azure Functions Host
AzureWebJobsDashboard	UseDevelopmentStorage=true	Connection string of Storage Account or Storage Emulator for the functions host dashboard.
AzureWebJobsStorage	UseDevelopmentStorage=true	Connection string of Storage Account or Storage Emulator for the functions host.
FUNCTIONS_WORKER_RUNTIME	java	Worker Runtime
languageWorkers:java:arguments	-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005	Environment Variables for debugging

Also you need to add AppSettings that your functions need.
You might wondering why AZURE_FUNCTIONS_ENVIRONMENT and ASPNETCORE_ENVIRONMENT.

You'll want to use AZURE_FUNCTIONS_ENVIRONMENT. The Functions runtime that powers a Function app on Azure is the WebHost project in this repo. As the host is initializing, it looks for the AZURE_FUNCTIONS_ENVIRONMENT app setting (as EnvironmentSettingNames.EnvironmentNameKey) and passes it to the IWebHostBuilder. Using only ASPNETCORE_ENVIRONMENT can lead to desired behavior changes and telemetry being missed.

Update JavaWorker

If you modify your code on the Java worker, you need to build it and package it on the Java Worker's cloned repo.

PS > mvn clean package

You will find azure-functions-java-worker-<VERSION>.jar. Copy the jar file to the WebJobs.Script.WebHost\bin\Debug\netcoreapp3.1\workers\java on the Azure Functions Host project.

Now ready to debug. hit F5. Then debugging starts. One tips is, if you choose IIS Express you can not see a log. Choose 1WebJobs.Script.WebHost` instead.

Attach from an IntelliJ

Follow Debugging Azure Functions Host and Java Worker

Create a remote debugging.

Just click start the debugging on the IntelliJ. It attach to the 5005 port that is the Java worker. Just send an request http://localhost:5000/api/YOUR_FUNCTIONS_NAME. or sending queues for triggering your Azure Functions.

It works!

Resource

Automatically create and deploy Java Azure Functions sample app to a Function App.

Tsuyoshi Ushio — Thu, 13 Aug 2020 07:45:44 +0000

The main tooling for the Azure Functions with Java is maven. I wanted to automatically create an sample app and deploy to the Premium Linux App. How can we do it?

Create a sample project

You can follow the quick start. However, I wanted to automate it.

Quickstart: Create a function in Azure that responds to HTTP requests

To automate this part is quite easy. If you try the tutorial, you need to manually input some parameters. groupId, artifactId, version and package is the one. Using -D flag, we can pass it.

mvn archetype:generate \
  -DarchetypeGroupId=com.microsoft.azure \
  -DarchetypeArtifactId=azure-functions-archetype \
  -DinteractiveMode=false \
  -DgroupId=com.fabrikam \
  -DartifactId=fabrikam-functions \
  -Dversion=1.0-SNAPSHOT \
  -Dpackage=com.fabrikam

It will create an pom.xml file and sample project. It is ready to deploy.

The problem

However, the problem is, the generated pom.xml is like below.

Java version on properties

Obviously, it is fixed to the Java8.

        <java.version>1.8</java.version>

Azure Functions Maven Plugin Config for Function APp.

azure-functions-maven-plugin can create/deploy an app to function app, however, it is already configured. If we want to automate it, we should modify the pom.xml. It is windows, however, I want to use Linux. I want to modify a lot of parts.I can write some app to modify this pom.xml, however, it is obviously, not cool. How can I modify it?

            <plugin>
                <groupId>com.microsoft.azure</groupId>
                <artifactId>azure-functions-maven-plugin</artifactId>
                <version>${azure.functions.maven.plugin.version}</version>
                <configuration>
                    <!-- function app name -->
                    <appName>${functionAppName}</appName>
                    <!-- function app resource group -->
                    <resourceGroup>java-functions-group</resourceGroup>
                    <!-- function app service plan name -->
                    <appServicePlanName>java-functions-app-service-plan</appServicePlanName>
                    <!-- function app region-->
                    <!-- refers https://github.com/microsoft/azure-maven-plugins/wiki/Azure-Functions:-Configuration-Details#supported-regions for all valid values -->
                    <region>westus</region>
                    <!-- function pricingTier, default to be consumption if not specified -->
                    <!-- refers https://github.com/microsoft/azure-maven-plugins/wiki/Azure-Functions:-Configuration-Details#supported-pricing-tiers for all valid values -->
                    <!-- <pricingTier></pricingTier> -->

                    <!-- Whether to disable application insights, default is false -->
                    <!-- refers https://github.com/microsoft/azure-maven-plugins/wiki/Azure-Functions:-Configuration-Details for all valid configurations for application insights-->
                    <!-- <disableAppInsights></disableAppInsights> -->
                    <runtime>
                        <!-- runtime os, could be windows, linux or docker-->
                        <os>windows</os>
                        <javaVersion>8</javaVersion>
                        <!-- for docker function, please set the following parameters -->
                        <!-- <image>[hub-user/]repo-name[:tag]</image> -->
                        <!-- <serverId></serverId> -->
                        <!-- <registryUrl></registryUrl>  -->
                    </runtime>
                    <appSettings>
                        <property>
                            <name>FUNCTIONS_EXTENSION_VERSION</name>
                            <value>~3</value>
                        </property>
                    </appSettings>
                </configuration>

Generated pom.xml

<?xml version="1.0" encoding="UTF-8" ?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.fabrikam</groupId>
    <artifactId>fabrikam-functions</artifactId>
    <version>1.0-SNAPSHOT</version>
    <packaging>jar</packaging>

    <name>Azure Java Functions</name>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <java.version>1.8</java.version>
        <azure.functions.maven.plugin.version>1.6.0</azure.functions.maven.plugin.version>
        <azure.functions.java.library.version>1.3.1</azure.functions.java.library.version>
        <functionAppName>fabrikam-functions-20200812194325715</functionAppName>
        <stagingDirectory>${project.build.directory}/azure-functions/${functionAppName}</stagingDirectory>
    </properties>

    <dependencies>
        <dependency>
            <groupId>com.microsoft.azure.functions</groupId>
            <artifactId>azure-functions-java-library</artifactId>
            <version>${azure.functions.java.library.version}</version>
        </dependency>

        <!-- Test -->
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter</artifactId>
            <version>5.4.2</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.mockito</groupId>
            <artifactId>mockito-core</artifactId>
            <version>2.23.4</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.8.1</version>
                <configuration>
                    <source>${java.version}</source>
                    <target>${java.version}</target>
                    <encoding>${project.build.sourceEncoding}</encoding>
                </configuration>
            </plugin>
            <plugin>
                <groupId>com.microsoft.azure</groupId>
                <artifactId>azure-functions-maven-plugin</artifactId>
                <version>${azure.functions.maven.plugin.version}</version>
                <configuration>
                    <!-- function app name -->
                    <appName>${functionAppName}</appName>
                    <!-- function app resource group -->
                    <resourceGroup>java-functions-group</resourceGroup>
                    <!-- function app service plan name -->
                    <appServicePlanName>java-functions-app-service-plan</appServicePlanName>
                    <!-- function app region-->
                    <!-- refers https://github.com/microsoft/azure-maven-plugins/wiki/Azure-Functions:-Configuration-Details#supported-regions for all valid values -->
                    <region>westus</region>
                    <!-- function pricingTier, default to be consumption if not specified -->
                    <!-- refers https://github.com/microsoft/azure-maven-plugins/wiki/Azure-Functions:-Configuration-Details#supported-pricing-tiers for all valid values -->
                    <!-- <pricingTier></pricingTier> -->

                    <!-- Whether to disable application insights, default is false -->
                    <!-- refers https://github.com/microsoft/azure-maven-plugins/wiki/Azure-Functions:-Configuration-Details for all valid configurations for application insights-->
                    <!-- <disableAppInsights></disableAppInsights> -->
                    <runtime>
                        <!-- runtime os, could be windows, linux or docker-->
                        <os>windows</os>
                        <javaVersion>8</javaVersion>
                        <!-- for docker function, please set the following parameters -->
                        <!-- <image>[hub-user/]repo-name[:tag]</image> -->
                        <!-- <serverId></serverId> -->
                        <!-- <registryUrl></registryUrl>  -->
                    </runtime>
                    <appSettings>
                        <property>
                            <name>FUNCTIONS_EXTENSION_VERSION</name>
                            <value>~3</value>
                        </property>
                    </appSettings>
                </configuration>
                <executions>
                    <execution>
                        <id>package-functions</id>
                        <goals>
                            <goal>package</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-resources-plugin</artifactId>
                <version>3.1.0</version>
                <executions>
                    <execution>
                        <id>copy-resources</id>
                        <phase>package</phase>
                        <goals>
                            <goal>copy-resources</goal>
                        </goals>
                        <configuration>
                            <overwrite>true</overwrite>
                            <outputDirectory>${stagingDirectory}</outputDirectory>
                            <resources>
                                <resource>
                                    <directory>${project.basedir}</directory>
                                    <includes>
                                        <include>host.json</include>
                                        <include>local.settings.json</include>
                                    </includes>
                                </resource>
                            </resources>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-dependency-plugin</artifactId>
                <version>3.1.1</version>
                <executions>
                    <execution>
                        <id>copy-dependencies</id>
                        <phase>prepare-package</phase>
                        <goals>
                            <goal>copy-dependencies</goal>
                        </goals>
                        <configuration>
                            <outputDirectory>${stagingDirectory}/lib</outputDirectory>
                            <overWriteReleases>false</overWriteReleases>
                            <overWriteSnapshots>false</overWriteSnapshots>
                            <overWriteIfNewer>true</overWriteIfNewer>
                            <includeScope>runtime</includeScope>
                            <excludeArtifactIds>azure-functions-java-library</excludeArtifactIds>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
            <!--Remove obj folder generated by .NET SDK in maven clean-->
            <plugin>
                <artifactId>maven-clean-plugin</artifactId>
                <version>3.1.0</version>
                <configuration>
                    <filesets>
                        <fileset>
                            <directory>obj</directory>
                        </fileset>
                    </filesets>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

How to create a Zip file to deploy Azure?

I thought that, it is not impossible, however, difficult to modify the pom.xml it's self, can we modify it with command line?

As I tested, we can modify properties part, however, not for azure functions maven plugin configuration. I'm asking how to modify it. Once I've got an answer, I'll update this blog. (8/13/2020)

If we can create a Zip file to deploy to Azure, and forget about deployment using maven, it might work and much easy to modify the deployment of Function App. We can use ARM template and Terraform.

I prefer the approach of Each Program Does One Thing Well. I remove the responsibility of deployment from maven. It must work.

clean and package with modifying `properties`

To modify the properties are easy. just use -D flag with the property that you want to modify. It works.

cd fabrikam-functions
mvn clean package -DfunctionAppName=sample

You will see that structure is created.

Unfortunately, the maven plugin doesn't have a goal that create zip file. They have deploy goal, however, we can't modify pom.

Create a zip file only

Just archive the red line part. zip command might need to install. I'm using WSL2. I needed to install the command.
-r means recursive, -q means quiet.

# packaging as a zip
cd target/azure-functions/sample
zip -r -q sample .

Deploy to the azure

Instead of using maven I use Azure CLI command instead. I deploy the zip file with Zip deployment. It works both for Linux AppService Function App and Linux Premium!

# deploy the app to the function app
az functionapp deployment source config-zip -g ${resourceGroup} -n ${baseName}-app --src sample.zip

NOTE: The official document says

When deploying your functions to a Linux function app in a Premium plan, you should always run from the package file and publish your app using the Azure Functions Core Tools

However, it is old information. The az command is already updated and ready to go!

Conclusion

By the series of the experiment, I successfully automate my workflow that is deploy function app with complex configuration and deploy sample app. Next time, I'll use terratest to automate the endpoint testing.

TBH, I'm not an expert of maven. If you have an better way to do the same thing, could you leave your comment, please?

The sample repo

FunctionsDeploymentTestingSuite

Resources

azure-maven-plugins

Deploy Linux AppService Function App with ARM template

Tsuyoshi Ushio — Thu, 13 Aug 2020 06:08:04 +0000

I explain how to do it for Linux Premium Plan.

Compared with Linux Premium, Linux AppService is much easier. I'll add some resources for the best practices for the ARM template.

You can find a sample in here.

Azure Functions Deployment Testing suite

serverfarms

For the AppService Plan with Linux, kind as Linux and reserved as true is the point.

Resolving “The parameter LinuxFxVersion has an invalid value” Error

        {
            "type": "Microsoft.Web/serverfarms",
            "apiVersion": "2018-02-01",
            "name": "[parameters('hostingPlanName')]",
            "location": "[parameters('location')]",
            "kind": "Linux",
            "sku": {
                "tier": "[parameters('sku')]",
                "name": "[parameters('skuCode')]"
            },
            "properties": {
                "reserved": true
            }
        },

Web/sites

kind as functionapp,linux, linuxFxVersion is the point. For more details about the linuxFxVersion please refer to Deploy Premium Linux Function App with ARM template

For the premium and consumption plan, we DONT need WEBSITE_CONTENTAZUREFILECONNECTIONSTRING and WEBSITE_CONTENTSHARE.

        {
            "apiVersion": "2018-11-01",
            "name": "[parameters('name')]",
            "type": "Microsoft.Web/sites",
            "kind": "functionapp,linux",
            "location": "[parameters('location')]",
            "tags": {},
            "dependsOn": [
                "[resourceId('microsoft.insights/components', parameters('appInsightsName'))]",
                "[resourceId('Microsoft.Web/serverfarms', parameters('hostingPlanName'))]",
                "[resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName'))]"
            ],
            "properties": {
                "name": "[parameters('name')]",
                "siteConfig": {
                    "linuxFxVersion": "[parameters('linuxFxVersion')]",
                    "alwaysOn": "[parameters('alwaysOn')]",
                    "appSettings": [
                        {
                            "name": "FUNCTIONS_EXTENSION_VERSION",
                            "value": "~3"
                        },
                        {
                            "name": "FUNCTIONS_WORKER_RUNTIME",
                            "value": "[parameters('functionWorkerRuntime')]"
                        },
                        {
                            "name": "APPINSIGHTS_INSTRUMENTATIONKEY",
                            "value": "[reference(concat('microsoft.insights/components/', parameters('appInsightsName')), '2015-05-01').InstrumentationKey]"
                        },
                        {
                            "name": "APPLICATIONINSIGHTS_CONNECTION_STRING",
                            "value": "[reference(concat('microsoft.insights/components/', parameters('appInsightsName')), '2015-05-01').ConnectionString]"
                        },
                        {
                            "name": "AzureWebJobsStorage",
                            "value": "[concat('DefaultEndpointsProtocol=https;AccountName=',parameters('storageAccountName'),';AccountKey=',listKeys(resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName')), '2019-06-01').keys[0].value,';EndpointSuffix=','core.windows.net')]"
                        }
                    ]
                },
                "serverFarmId": "[concat('/subscriptions/', parameters('subscriptionId'),'/resourcegroups/', parameters('serverFarmResourceGroup'), '/providers/Microsoft.Web/serverfarms/', parameters('hostingPlanName'))]",
                "clientAffinityEnabled": false
            }
        },

APPLICATIONINSIGHTS_CONNECTION_STRING

What is that? As you can see, it is the same value. They want to move on CONNECTION_STRING that they can add other key/value info. Currently, we need to specify both.

key	value (sample)
APPINSIGHTS_INSTRUMENTATIONKEY	96cd2ffe-ef28-4809-b98a-8f1da9da4b5a
APPLICATIONINSIGHTS_CONNECTION_STRING	InstrumentationKey=96cd2ffe-ef28-4809-b98a-8f1da9da4b5a

NOTE: The key is removed already.

ARM template resources

Understand the structure

Understand the structure and syntax of ARM templates

I didn't know the functions section. It means, we can write custom functions with using existing build-in functions.

{
  "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
  "contentVersion": "",
  "apiProfile": "",
  "parameters": {  },
  "variables": {  },
  "functions": [  ],
  "resources": [  ],
  "outputs": {  }
}

name	required	description	sample
$schema	Yes	Location of the JSON schema file that describes the version of the template language. The version number you use depends on the scope of the deployment and your JSON editor.	If you're using VS Code with the Azure Resource Manager tools extension, use the latest version `https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#`
contentVersion	Yes	Version of the template	1.0.0.0
apiProfile	No	An API version that serves as a collection of API versions for resource types. Use this value to avoid having to specify API versions for each resource in the template. For more details refer to Track versions using API profiles	2018–03-01-hybrid
parameters	No	Values that are provided when deployment is executed to customize resource deployment.	See the sample
variables	No	Values that are used in the template. Usually modify it from parameters or default values.	Variables
functions	No	User-defined functions that are available within the template.	functions
resources	Yes	Resource types that are deployed or updated in a resource group or subscription.	resources
outputs	No	Values that are returned after deployment. It is useful to get `ConnectionString` for example	output

How to set the default value to the parameter?

defaultValue works. Also you can add description on the metadata section. It helps users to understand the meaning.

        "linuxFxVersion": {
            "type": "string",
            "defaultValue": "Java|8",
            "metadata": {
                "description": "linuxFxVersion that specify the language runtime for linux. (Java|8) for example"
            }               
        },

How to generate random characters

I want to add suffix for the storage account name. If you delete a storage account, then you want to use the same name, it won't work for a while (around 2 hours). Some resources requires random values. Terraform provide it, how about ARM template?

There is. However, it looks not great. It is uniquestring it says

The returned value isn't a random string, but rather the result of a hash function. The returned value is 13 characters long. It isn't globally unique.

I eventually, create the random string with bash script. I frequently reference this repo. DevOps Openhack Proctor Repository. This repo is like a treasure box of DevOps practices.

randomChar() {
    s=abcdefghijklmnopqrstuvxwyz0123456789
    p=$(( $RANDOM % 36))
    echo -n ${s:$p:1}
}

randomNum() {
    echo -n $(( $RANDOM % 10 ))
}

postfix="$(randomChar;randomChar;randomChar;randomNum;)"

How to deploy the template from Azure CLI

Create a resource group with az group create then use az deployment group create. I avoid the parameter files. I wanted to create resources with some base string. e.g. tsuyoshi then I'd like to post fix these. tsuyoshi012234sed, tsuyoshi-app, tsuyoshi-plan or something like that. It is enough through the parameters. Also, want to avoid the confusion when we both use parameter files and parameter.

if [ `az group exists -n $resourceGroup -o tsv` == false ]; then
    echo "Resource group with name" $resourceGroup "could not be found. Creating new resource group.."
    set -e
    (
        set -x
        echo "0-Provision Resource Group (az group create --name $resourceGroup --location $location)"
        az group create --name $resourceGroup --location "$location"
    )
else
    echo "Using existing resource group..."
fi

echo "SubscriptionId : ${subscriptionId}"

echo "Deploying Function App..."

echo "1-Provision FunctionApp (az group deployment create --name ${deploymentName} --resource-group $resourceGroupName --template-file $templateFile --parameters subscriptionId=$subscriptionId serverFarmResourceGroup=$resourceGroup name=$functionAppName location="${location}" hostingPlanName=$hostingPlanName storageAccountName=$storageAccountName appInsightsName=$appInsightsName linuxFxVersion=$linuxFxVersion  functionWorkerRuntime=$functionWorkerRuntime)"
az deployment group create \
    --name ${deploymentName} \
    --resource-group $resourceGroup \
    --template-file $templateFile \
    --parameters subscriptionId=$subscriptionId serverFarmResourceGroup=$resourceGroup name=$functionAppName location="${location}" hostingPlanName=$hostingPlanName storageAccountName=$storageAccountName appInsightsName=$appInsightsName linuxFxVersion="$linuxFxVersion" functionWorkerRuntime=$functionWorkerRuntime

Deploy Premium Linux Function App with ARM template

Tsuyoshi Ushio — Thu, 13 Aug 2020 04:34:07 +0000

I wanted to automate the deployment of the Premium Linux and AppService plan for Java application. I'd like to share what I learned. This repo is the repo I leave the samples.

Azure Functions Deployment Testing suite

Premimu Linux

There is an official document however, I can't find Premium Linux configuration.

Serverfarms

targetWorkerCount and targetWorkerSizeId are something we can't find on the official documentation. That is int properties. Important point is, kind as elastic and reserved as true.

        {
            "apiVersion": "2018-02-01",
            "name": "[parameters('hostingPlanName')]",
            "type": "Microsoft.Web/serverfarms",
            "location": "[parameters('location')]",
            "kind": "elastic",
            "tags": {},
            "dependsOn": [],
            "properties": {
                "targetWorkerCount": "[parameters('targetWorkerCount')]",
                "targetWorkerSizeId": "[parameters('targetWorkerSizeId')]",
                "reserved": true,
                "maximumElasticWorkerCount": "[parameters('maximumElasticWorkerCount')]"
            },
            "sku": {
                "tier": "[parameters('sku')]",
                "name": "[parameters('skuCode')]"
            }
        },

We have a several resouces, however, the reference looks the latest.

Microsoft.Web serverfarms template reference

targetWorkerSizeId

Probably, these meaning.

Small = 0,
Medium = 1,
Large = 2,
D1 = 3,
D2 = 4,
D3 = 5,
SmallV3 = 6,
MediumV3 = 7,
LargeV3 = 8,
NestedSmall = 9,

Web/sites

kind as functionapp,linux. For specifying languages, I set FUNCTIONS_WORKER_RUNTIME as java, linuxFxVersion as Java|8. The linuxFxVersion requires only on linux. It represents docker images that you found azure functions docker. You can find some possible value in here. It can be specify the generic one like Java|8 but also you can specify specific docker image version like DOCKER|mcr.microsoft.com/azure-functions/python:2.0.13351-python3.6-appservice. You can find it on azure functions docker repo.

For the premium and consumption plan, we need WEBSITE_CONTENTAZUREFILECONNECTIONSTRING and WEBSITE_CONTENTSHARE.

        {
            "apiVersion": "2018-11-01",
            "name": "[parameters('name')]",
            "type": "Microsoft.Web/sites",
            "kind": "functionapp,linux",
            "location": "[parameters('location')]",
            "tags": {},
            "dependsOn": [
                "[resourceId('microsoft.insights/components', parameters('appInsightsName'))]",
                "[resourceId('Microsoft.Web/serverfarms', parameters('hostingPlanName'))]",
                "[resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName'))]"
            ],
            "properties": {
                "name": "[parameters('name')]",
                "siteConfig": {
                    "appSettings": [
                        {
                            "name": "FUNCTIONS_EXTENSION_VERSION",
                            "value": "~3"
                        },
                        {
                            "name": "FUNCTIONS_WORKER_RUNTIME",
                            "value": "[parameters('functionWorkerRuntime')]"
                        },
                        {
                            "name": "APPINSIGHTS_INSTRUMENTATIONKEY",
                            "value": "[reference(concat('microsoft.insights/components/', parameters('appInsightsName')), '2015-05-01').InstrumentationKey]"
                        },
                        {
                            "name": "APPLICATIONINSIGHTS_CONNECTION_STRING",
                            "value": "[reference(concat('microsoft.insights/components/', parameters('appInsightsName')), '2015-05-01').ConnectionString]"
                        },
                        {
                            "name": "AzureWebJobsStorage",
                            "value": "[concat('DefaultEndpointsProtocol=https;AccountName=',parameters('storageAccountName'),';AccountKey=',listKeys(resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName')), '2019-06-01').keys[0].value,';EndpointSuffix=','core.windows.net')]"
                        },
                        {
                            "name": "WEBSITE_CONTENTAZUREFILECONNECTIONSTRING",
                            "value": "[concat('DefaultEndpointsProtocol=https;AccountName=',parameters('storageAccountName'),';AccountKey=',listKeys(resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName')), '2019-06-01').keys[0].value,';EndpointSuffix=','core.windows.net')]"
                        },
                        {
                            "name": "WEBSITE_CONTENTSHARE",
                            "value": "[concat(toLower(parameters('name')), '94ed')]"
                        }
                    ],
                    "linuxFxVersion": "[parameters('linuxFxVersion')]"
                },
                "serverFarmId": "[concat('/subscriptions/', parameters('subscriptionId'),'/resourcegroups/', parameters('serverFarmResourceGroup'), '/providers/Microsoft.Web/serverfarms/', parameters('hostingPlanName'))]",
                "clientAffinityEnabled": false
            }
        },

One of the question is Why do I need both APPINSIGHTS_INSTRUMENTATIONKEY and APPLICATIONINSIGHTS_CONNECTION_STRING in my Azure Function configuration?. You can find the answer on the link.

What did I google it?

Deployed not supported region

{
    "Code": "BadRequest",
    "Message": "Requested features are not supported in region. Please try another region.",
    "Target": null,
    "Details": [
        {
            "Message": "Requested features are not supported in region. Please try another region."
        },
        {
            "Code": "BadRequest"
        },
        {
            "ErrorEntity": {
                "ExtendedCode": "59911",
                "MessageTemplate": "Requested features are not supported in region. Please try another region.",
                "Parameters": [],
                "Code": "BadRequest",
                "Message": "Requested features are not supported in region. Please try another region."
            }
        }
    ],
    "Innererror": null
}

These command shows you the location that available to you for each resources.

az provider show --namespace Microsoft.Web | jq ".resourceTypes[] | select(.resourceType == \"serverFarms\") | .locations"

Products available by region
az account list-location find locations that you can use.

The serverFarm property has an incorrect value for App Service Plan

The error message was very confusing. However, it was an issue of this part. It was not the serverfarms resource. I forget to add value to serverFarmResourceGroup.

"serverFarmId": "[concat('/subscriptions/', parameters('subscriptionId'),'/resourcegroups/', parameters('serverFarmResourceGroup'), '/providers/Microsoft.Web/serverfarms/', parameters('hostingPlanName'))]",

The way I identify issue is portal. Even if I deploy from my local machine, if you go to portal and see the deployment history, then you can find which resources causes an issue.

Deployment failed. Correlation ID: 95b838bf-3c0d-45f0-9457-0b10c916109f. {
  "Code": "BadRequest",
  "Message": "The serverFarm property has an incorrect value for App Service Plan. Valid values are a plan name as string or of format  subscriptions/subscriptionId/resourceGroups/resourceGroupName/providers/Microsoft.web/serverfarms/planName where the subscription is the same as the site's subscription.",
  "Target": null,

serverframs can not found

I encounter the issue. The reason was, the template that I've got from donwload automated script was wrong. It has a section like this.

            "dependsOn": [
                "microsoft.insights/components/tsushijava8trace",
                "[concat('Microsoft.Web/serverfarms/', parameters('hostingPlanName'))]",
                "[concat('Microsoft.Storage/storageAccounts/', parameters('storageAccountName'))]"
            ],

dependsOn section requires resourceId function.

            "dependsOn": [
                "[resourceId('microsoft.insights/components', parameters('appInsightsName'))]",
                "[resourceId('Microsoft.Web/serverfarms', parameters('hostingPlanName'))]",
                "[resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName'))]"
            ],

Lessons Learned

This time, I have no sample, so I can't identify how we can deploy Premium Functions.

Deploy Premium Linux functions from portal and download automated script
Deploy Premium Linux functions from portal and see Export template
Offical Doc Automate resource deployment for your function app in Azure Functions
References Microsoft.Web serverfarms template reference, Microsoft.Web sites/config template reference

When I look back, the best way might be...

Learn the basic with the official document.
Deploy Premimu Linux functions from portal and see Export template
Understand the meaning of the parameters with Reference

The download automated script is something I used to use frequently, however, this time, the template does not work. Understand the proper format and check with the Export template is the truth.

I hope this helps.

Find Expired Service Principal

Tsuyoshi Ushio — Tue, 11 Aug 2020 00:49:53 +0000

My tenant seems to have a quota limit of the Service Principal.

$ az ad sp create-for-rbac
The directory object quota limit for the Principal has been exceeded. Please ask your administrator to increase the quota limit or delete objects to reduce the used quota.

Since I'm a DevOps guy, I frequently create a service principal. I'd like to find expired service principals on my tenant.

I struggled for a while with OData and JMESPath that is supported with az command. I eventually use jq instead.

Get the current date with the Azure CLI format.

We can use %N for nanosecond, however, it doesn't match the style of the Azure CLI output. I eventually use this command.

CURRENT_TIME=`date -u +'%Y-%m-%dT%H:%M:%S.024000+00:00'`

Get Expired Service Principal

The query is only list the service principal that you create with password credentials.

$ az ad sp list --show-mine | jq ".[] | select(.passwordCredentials[].endDate < \"$CURRENT_TIME\") | { objectId : .objectId, appId: .appId, passwordCredentials : .passwordCredentials, displayName: .displayName, tenantId : .appOwnerTenantId }"

az ad sp list --show-mine will show the service principals that you created. select will filter the json. The json start with array that has no name. so start with .[]. .[] returns each element of the array. After filter the result, it project some values.

The original structure of the `az ad sp list --show-time`

[{
  "appId": "YOUR_APP_ID_HERE",
  "appOwnerTenantId": "YOUR_TENANT_ID_HERE",
  "displayName": "procohapp10",
  "objectId": "YOUR_OBJECT_ID_HERE",
  "passwordCredentials": [
    {
      "additionalProperties": null,
      "customKeyIdentifier": null,
      "endDate": "2019-05-02T20:32:27.908964+00:00",
      "keyId": "YOUR_KEY_ID",
      "startDate": "2018-05-02T20:32:27.908964+00:00",
      "value": null
    }
  ],
             :
},
  :
}

Result

{
  "objectId": "YOUR_OBJECT_ID",
  "appId": "YOUR_APP_ID",
  "passwordCredentials": [
    {
      "additionalProperties": null,
      "customKeyIdentifier": null,
      "endDate": "2019-05-02T20:32:27.908964+00:00",
      "keyId": "YOUR_KEY_ID",
      "startDate": "2018-05-02T20:32:27.908964+00:00",
      "value": null
    }
  ],
  "displayName": "procohapp10",
  "tenantId": "YOUR_TENANT_ID"
}

Remove the expired service principal

Please double check if your script works before execute the second last line.

$ az ad sp list --show-mine | jq ".[] | select(.passwordCredentials[].endDate < \"$CURRENT_TIME\") | .objectId" > sp_list.txt
$ lines=`cat sp_list.txt`
$ for item in $lines; do  echo $item | xargs az ad sp delete --id ; done
$ rm sp_list.txt

Confirm removed

It returns zero result.

$ az ad sp list --show-mine | jq ".[] | select(.passwordCredentials[].endDate < \"$CURRENT_TIME\") | .objectId"

Resource

Azure Storage Queue SDK for Go Samples for Enqueue/Dequeue and Count

Tsuyoshi Ushio — Sat, 25 Jul 2020 05:22:09 +0000

I searched a working sample of the Azure Storage Queue with Azure Storage Queue SDK for Go to save my time. However, I couldn't. Only we can find on the GitHub repo is documentation. So I create a sample for sending/receiving queue. You can refer the sample repository below.

Sample Repository

Get the count of the messages

The behavior is probably that you expected. However, I noticed that one thing. If you want to get a count of the queue, we can use props.ApproximateMessagesCount() function on the QueueGetPropertiesResponse. As it is represent the name, the count is Approximate number. If you run the example, you will see this output on the receive.go. The log that shows Approximate... is the one that I use the method. For the Visible count of message, it is fetched by a function that I'll explain later. This receive sample Dequeue a message. Before the dequeue, the number of the message is 6. However, after the dequeue, Approximate one is still 6. I should be 5. When you delete the message, the value is the same. It looks the ApproximateMessagesCount() looks ignore the visibility of the message.

receive

1: Approximate number of messages in the queue=7
2: Visible count of messages in the queue=7
3: Dequeue [25009-77-70 254:00:00 +0000 UTC] : Message: Hello!
4: Approximate number of messages in the queue=7
5: Visible count of messages in the queue=6
6: Delete message bc3ae6cc-3222-40c5-b922-f0a2b4266532
7: Approximate number of messages in the queue=6
8: Visible count of messages in the queue=6

According to the official REST API documentation:

The approximate number of messages in the queue. This number is not lower than the actual number of messages in the queue, but could be higher.
Response Headers.

How to get the visible queue count?

Just Peek the queue then, count the number. This method is accurate, however, one downside is, maximum maxCount is 32. numofmessages

func getVisibleCount(messagesURL *azqueue.MessagesURL, maxCount int32) (int32, error) {
    ctx := context.Background()
    queue, err := messagesURL.Peek(ctx, maxCount)
    if err != nil {
        return 0, err
    }
    num := queue.NumMessages()
    return num, nil
}

I'm writing a scaler logic by these values. I'll combine two method to scale my app.

FYI, for the send message, I sent 100 messages. In this case, Visible count of message is 1.

1: Approximate number of messages in the queue=100
2: Visible count of messages in the queue=1
3: Enqueue Hello 99: ec979756-bbc2-42d0-a791-0a29bfb00d2b 
4: Approximate number of messages in the queue=101
5: Visible count of messages in the queue=1

For the Enqueue method, it has two parameters. The function signature is func (m MessagesURL) Enqueue(ctx context.Context, messageText string, visibilityTimeout time.Duration, timeToLive time.Duration) (*EnqueueMessageResponse, error) { vt := int32(visibilityTimeout.Seconds()) I set the visibilityTimeout as 30*times.Second That is why it takes 30 sec for able to see the message. timeToLive 0 means 7 days.

response, err := messageURL.Enqueue(ctx, "Hello!", 30*time.Second, 0)

If you want to visible immediately, let's make it 0.

response, err := messageURL.Enqueue(ctx, "Hello!", 0, 0)

Then it works as you expected.

1: Approximate number of messages in the queue=30
2: Visible count of messages in the queue=30
3: Enqueue Hello 30: 75909a2a-9d12-41c7-bb23-675c32dd8c88 
4: Approximate number of messages in the queue=31
5: Visible count of messages in the queue=31
1: Approximate number of messages in the queue=31
2: Visible count of messages in the queue=31
3: Enqueue Hello 31: 8898a793-d8d0-47d4-afaf-266fcacc3778 
4: Approximate number of messages in the queue=32
5: Visible count of messages in the queue=32
1: Approximate number of messages in the queue=32
2: Visible count of messages in the queue=32
3: Enqueue Hello 32: 7989faaf-3905-402d-9d3c-0caeebbdb705 
4: Approximate number of messages in the queue=33
5: Visible count of messages in the queue=32
1: Approximate number of messages in the queue=33
2: Visible count of messages in the queue=32

DEV Community: Tsuyoshi Ushio

[Update] Creating Nostalgic AOP Using Source Generator

Introduction to Roslyn Source Generator

What are Roslyn APIs?

Source Generator

How to use it

Create a source generator project

Create a generator

Debugging with Source Generator

Debugging

Viewing Generated Source

Creating an AOP Sample

Code Generation Logic

Execution Result

IncrementalGenerators

IIncrementalGenerator

Porting the AOP Sample

Summary

Resources

Kafka ScaledJobs sample for KEDA with EventHubs

Kafka Broker

Configuration

Sarama

Scaled Job

Secret

ScaledJob

Run on kubernets

Resource

Enabling KEDA debugging

Remote Debugging

Create a Dockerfile

Update the deployment

Local Debugging

Build and publish KEDA to your kubernets cluster

Using Fiddler for Application Insights codeless java agent with docker on WSL2

Strategy Overview

Export a root certificate from Fiddler

Import the certificate to the KeyStore on JVM

Configure Application Insights Codeless Agent for Proxy

Find IPAddress for accessing from WSL2 to Windows 10

Configure the codeless agent

Configure Fiddler to accept the request from WSL2

Resources

Solves 0 function found on Azure Function - Linux Consumption plan

Solution

Enabling Memory Profiling on KEDA v2

Configuration and Issue

Solution

Viewer

Reference

Debugging Java Language Worker on Azure Functions

Architecture

1. Azure Functions Host (C#)

2. Binding Extensions (C# + wrapper)

3. Java Language Worker (Java)

4. Azure Functions Java Library (Java)

5. Application Insights Agent (Java)

6. Azure Functions (Java)

How to debug it?

Create an Azure Functions

Install extension

Configure Azure Functions Host for Java Worker Debugging

WebJobs.Script.WebHost as startup

Configure Environment Variables

Update JavaWorker

Attach from an IntelliJ

Resource

Automatically create and deploy Java Azure Functions sample app to a Function App.

Create a sample project

The problem

Java version on properties

Azure Functions Maven Plugin Config for Function APp.

Generated pom.xml

How to create a Zip file to deploy Azure?

clean and package with modifying properties

Create a zip file only

Deploy to the azure

Conclusion

Resources

Deploy Linux AppService Function App with ARM template

clean and package with modifying `properties`

The original structure of the `az ad sp list --show-time`