DEV Community: Galina Melnik

Update Swashbuckle.AspNetCore Version from version"4.0.1" to "6.2.3"

Galina Melnik — Fri, 13 May 2022 06:51:16 +0000

Hi there.
I had an interesting task: to update the Swashbuckle.AspNetCore version from "4.0.1" to "6.2.3". It looked very simple but it was not what it looked like.
The main problem was breaking changes which happened when passing from version 4 to version 5. Swashbuckle.AspNetCore began to use Swagger/OpenAPI version v3 instead of OpenAPI v2. The project makes use of NSwag to generate httpClient for getting data from another microservice. Any attempt to regenerate auto generated code changed it after updating Swashbuckle.AspNetCore version. These caused a lot of build errors. I should have decreased differencies between the new code and old one.
Main difference which cause incompatibility in the project were:

Autogenerated method names were changed. The previous format was, for example, "AddPersonalTaskV2Async" (it was a controller class method name). New auto generated names became "V2Async". There were a lot of such changes, it seemed impossible to change all method calls by hand. A better decision was to add CustomOperationIds to services.AddSwaggerGen. In my case:

services.AddSwaggerGen(c =>
            {
                c.SwaggerDoc("v1", new OpenApiInfo {Title = "title", Version = "v1"});
                c.CustomOperationIds(apiDesc =>
                    apiDesc.TryGetMethodInfo(out MethodInfo methodInfo) ? methodInfo.Name : null);
            });

There were no changes in names of parameters with attribute [FromUri]. But all parameters with attribute [FromBody] changed their names to body. As described in the spec [FromUri] is aligned with Operation Object => parameters, but [FromBody] is Operation Object => requestBody. There is no field "name" in Request Body Object.
Parameter order was changed. Before the order was the same as in method declaration. By now [FromUri] goes earlier than [FromBody]. It doesn't matter what you have coded in method declaration.
If you use JObject as a request parameter, you will be in for an unpleasant surprise. NSWag changes JObject to System.Collections.Generic.Dictionary. If you are disagree, it will be better to code custom realization of ISchemaFilter like this:

public class JObjectSchemaFilter : ISchemaFilter
    {
        public void Apply(OpenApiSchema schema, SchemaFilterContext context)
        {
            if (context.Type != typeof(JObject)) return;
            schema.Type = "object";
            schema.AdditionalProperties = null;
        }
    }

Another change has affected Enums. Enum fields names don't be the same as in your code. They are changed to figures 1,2, ..., n. To fix this singularity it should be created another custom realization of ISchemaFilter:

public class EnumSchemaFilter : ISchemaFilter
    {
        public void Apply(OpenApiSchema schema, SchemaFilterContext context)
        {
            if (!context.Type.IsEnum) return;

            var fields = context.Type.GetFields(BindingFlags.Static | BindingFlags.Public);

            schema.Enum = fields.Select(field => new OpenApiString(field.Name)).Cast<IOpenApiAny>().ToList();
            schema.Type = "string";
            schema.Properties = null;
            schema.AllOf = null;
        }
    }

The last thing I should mention: some fields of custom request objects suddenly have an attribute [Required]. I'm not sure that I coded the best decision, but this worked:

public class SettingsSchemaFilter : ISchemaFilter
    {
        public void Apply(OpenApiSchema schema, SchemaFilterContext context)
        {
            if (context.Type != typeof(Settings)) return;
            if (schema.Required == null || schema.Required.Count == 0)
            {
                schema.Nullable = true;
            }
        }
    }

It's all changes that I faced.

Migrate C# project to Istio

Galina Melnik — Thu, 21 Apr 2022 11:02:39 +0000

My team has a C# project. This project is deployed using k8s. There was a task to migrate to corpotive Istio using a certificate, a gateways and a virtual service but without a service mesh.
If you have such type of a task there are my recomendations:

It's needed to create a file with a certificate:

apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  labels:
    istio-certificate: "true"            # if it would be used with gateway object, the value should be set to "true"
    app.kubernetes.io/managed-by: {{ .Release.Service }}
  name: {{ .Values.environment.servicename }}
spec:
  dnsNames:
  - {{ .Values.service.name }}.{{ .Release.Namespace }}.{{ .Values.environment.cluster }}
  issuerRef:
    kind: ClusterIssuer
    name: certificates-issuer
  secretName: {{ .Values.environment.servicename }} # the secretName should be equals to the certificate name

If you want to learn more about certificate resources, there is pretty documentaions here
The second thing is needed to do is to add a gateway description.
The istio gateway provides a description of ports & protocols which will be used. More information here

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: {{ .Values.service.name }}
spec:
  selector:
    company/istio-ingressgateway: common
  servers:
    - port:
        number: 443                                       
        name: https
        protocol: HTTPS                                   
      tls:
        mode: SIMPLE                                      
        credentialName: {{ .Values.environment.servicename }} 
      hosts:
        - {{ .Values.service.name }}.{{ .Release.Namespace }}.{{ .Values.environment.cluster }}               # use a correct hostname for your namespace

Istio VirtualService describes routing. It's needed to specify hosts, ports and other things. All documentation available here

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: {{ .Values.service.name }}
spec:
  hosts:
    - {{ .Values.service.name }}.{{ .Release.Namespace }}.{{ .Values.environment.cluster }}
  gateways:
    - {{ .Values.service.name }}                        # join the virtualService with the gateway
  http:
  - name: "default"
    route:
    - destination:
        port:
          number: {{ .Values.environment.serviceport }}       
        host: {{ .Values.environment.servicename }}-service

Add existing autotests to existing GitLab CI/CD

Galina Melnik — Thu, 17 Mar 2022 06:22:21 +0000

Hello. Our command has java-autotests in TeamCity for the C# project. It's our heritage, because our testers have java-background and they prefer to use java for their purposes.
The company standart was changed some time ago. We begin to use GitLab CI/CD instead of TeamCity. I have a task to transfer java-tests from TeamCity to GitLab.
The main idea of maven autotests was to deploy c# project into kubernetes and after that tried to call java-autotests using maven outside of deployed project.
I've added stages after deploy to existed .gitlab-ci.yaml. I don't describe build and deploy stages.

I should mention that "<<: *dp-auth-docker" is a common template that we use to auth in corporative instance of docker.

My collegue found that k8s sometimes scaled down replicas to count equaled 0. That is why the first added stage is called scale-replicas:

scale-replicas:
  <<: *dp-auth-docker
  stage: scale-replicas
  script:
    - theBranch=$(echo $CI_COMMIT_REF_NAME | tr '[:upper:]' '[:lower:]')
    - |
      echo 'Check replicas. Will be set value 1.'
      docker run --rm $CORPORATIVE_DOCKER_IMAGE_K8S kubectl scale --replicas=1 deployments/$theBranch
  only:
    - merge_requests

$CI_COMMIT_REF_NAME is a predefined GitLab variable (documentation)
theBranch is a local stage variable, which contains a name of a branch. We always use number of a task as the branch name. I should convert the branch name to lower case because k8s deployment names are case sensitive.
--rm flag define clean up container (documentation)

Time to scale a replica could be significant. That is why it's needed to delay between scale the replica and tests.

sleep:
  stage: sleep
  script:
    - sleep 120
  only:
    - merge_requests

GitLab measures time in seconds. 2 minutes delay is enough to scale a replica in the system.

Next step is running tests.

run-tests:
  <<: *dp-auth-docker
  stage: run-tests
  script:
    - docker run --rm $CORPORATIVE_DOCKER_IMAGE_DIND dockerd & >/dev/null
    - |
      while [docker run --rm $CORPORATIVE_DOCKER_IMAGE_DIND ! -S /var/run/docker.sock ];
      do
          echo "waiting";
          sleep 1;
      done
    - pathKube="./.kube"
    - mkdir $pathKube
    - echo -e
    - |
      config=$(docker run \
        --rm -t \
        -p 7000:7000 \
        $CORPORATIVE_DOCKER_IMAGE_K8S \
        cat /root/.kube/config)
    - file="$pathKube/config"
    - echo "$config">$file
    - echo -e
    - |
      if test -f "$file"; then
          echo "New config was saved to: $file"
      else
          echo"New config was not saved"
          exit -1
      fi
    - docker run --rm --volume $(pwd):/tests $CORPORATIVE_DOCKER_IMAGE_NEPTUNE mvn antrun:run -P !QA,featureBranch -f /tests/pom.xml -Dkubectl.config=/tests/.kube/config -Dfeature.branch.number=$CI_COMMIT_REF_NAME clean test -Dsuite=$SUITE_NAME
  cache:
    paths:
      - .m2/repository
  only:
    - merge_requests

dockerd runs daemon in docker (documentaion)
while an unix socket creates and openes, script waits.
copy /.kube/config from k8s image to local. I need it to run tests from k8s and to forward ports for MongoDB.
maven use cache in directory .m2/repository

An easier method to deserialize json from NASA InSight: Mars Weather Service API

Galina Melnik — Mon, 17 May 2021 10:17:33 +0000

NASA has a free service with Mars wheather. I'm trying to use it and create a pet project with beautiful Mars's photos and wheathers descriptions.
You can see demo data with this URL
Just this time the service doesn't work but I hope It will work soon.
I have saved json for unit tests and show it for clarity:

{
    "781": {
        "First_UTC": "2021-02-05T16:28:36Z",
        "Last_UTC": "2021-02-06T17:08:11Z",
        "Month_ordinal": 12,
        "Northern_season": "late winter",
        "PRE": {
            "av": 717.422,
            "ct": 120387,
            "mn": 695.3022,
            "mx": 742.7693
        },
        "Season": "winter",
        "Southern_season": "late summer",
        "WD": {
            "most_common": null
        }
    },
// and so forth, and so on
    "sol_keys": [
        "781"
// and so forth, and so on
    ]
}

Some monthes ago I've created a post with one of the methods to deserialize NASA data using custom JsonConverter.
But now I think that there is an more easier method to do it.
Let's begin.
I have a class with description.

public class MarsWheather
    {
        public int Sol { get; set; }
        [JsonPropertyName("First_UTC")]
        public DateTime FirstUTC { get; set; }
        [JsonPropertyName("Last_UTC")]
        public DateTime LastUTC { get; set; }
        [JsonPropertyName("Season")]
        [JsonConverter(typeof(JsonStringEnumConverter))]
        public Season MarsSeason { get; set; }
        [JsonPropertyName("PRE")]
        public DataDescription AtmosphericPressure { get; set; }

        [JsonIgnore]
        public HashSet<string> Photos { get; set; }
        [JsonIgnore]
        public List<RoverInfo> Rovers { get; set; }
    }

    public enum Season {
        winter,
        spring,
        summer,
        autumn
    }

    public class DataDescription{
        [JsonPropertyName("av")]
        public double Average { get; set; }
        [JsonPropertyName("ct")]
        public double TotalCount { get; set; }
        [JsonPropertyName("mn")]
        public double Minimum { get; set; }
        [JsonPropertyName("mx")]
        public double Maximum { get; set; }
    }
    public enum RoverName
    {
        Curiosity,
        Opportunity,
        Spirit
    }

    public class RoverInfo {
        public string Name { get; set; }
        public DateTime LandingDate { get; set; }
        public DateTime LaunchDate { get; set; }
        public string Status { get; set; }
    }

I've read a description of service not attentively, that was why I've missed this paragraph

Get a top-level Sol key from the top-level key sol_keys

Ha-ha, it was a big mistake! I should return to my server and change a method to get data.
I've added a new class to process json (CustomDeserializer.cs)

public static class CustomDeserializer
    {
        static string fieldName = "sol_keys";
        public static async Task<IEnumerable<MarsWheather>> GetAsync(Stream stream)
        {
            var result = new List<MarsWheather>();
            using (var document = JsonDocument.Parse(stream))
            {
                var root = document.RootElement;
                var keys = root.GetProperty(fieldName);
                foreach (var key in keys.EnumerateArray())
                {
                    if (root.TryGetProperty(key.GetString(), out var element))
                    {
                            var wheather = await JsonSerializer.DeserializeAsync<MarsWheather>(new MemoryStream(Encoding.ASCII.GetBytes(element.GetRawText())));
                            int.TryParse(key.GetString(), out var sol);
                            wheather.Sol = sol;
                            result.Add(wheather);                        
                    }
                }
            }
            return result;
        }
    }

And this works!

JsonConverter<T>

Galina Melnik — Sun, 28 Feb 2021 11:58:28 +0000

I want to create home pet project with Mars wheather. Thanks God NASA have a public API with data.
Test URL https://api.nasa.gov/insight_weather/?api_key=DEMO_KEY&feedtype=json&ver=1.0
And there is their json for today :
{ "782": { "First_UTC": "2021-02-06T17:08:11Z", "Last_UTC": "2021-02-07T17:47:46Z", "Month_ordinal": 12, "Northern_season": "late winter", "PRE": { "av": 721.77, "ct": 113450, "mn": 698.8193, "mx": 742.2686 }, "Season": "winter", "Southern_season": "late summer", "WD": { "most_common": null } }, "783": { "First_UTC": "2021-02-07T17:47:46Z", "Last_UTC": "2021-02-08T18:27:22Z", "Month_ordinal": 12, "Northern_season": "late winter", "PRE": { "av": 722.186, "ct": 107270, "mn": 698.7664, "mx": 743.1983 }, "Season": "winter", "Southern_season": "late summer", "WD": { "most_common": null } }, ... irrelevant "sol_keys": [... irrelevant], "validity_checks": { ...irrelevant } }

I've created a project using netcoreapp3.0 and wanted to use System.Text.Json library. My first wrong way to do it was :

var dict = await JsonSerializer.DeserializeAsync<Dictionary<string, MarsWheather>>(stream);

But I caught an exception
System.Text.Json.JsonException: The JSON value could not be converted to MarsWheather. Path: $.sol_keys | LineNumber: 120 | BytePositionInLine: 15.
BOOM!
That's why I've created custom JsonConverter. I hope this code will helpful for community.
UPDATE 03/03/2021 : There was a bug with my code. This is a new version without it.
Data classes is :

[JsonConverter(typeof(MarsWheatherRootObjectConverter))]
    public class MarsWheatherRootObject
    {
        public List<MarsWheather> MarsWheather { get; set}
    }
    public class MarsWheather {
        public int Sol { get; set; }
        [JsonPropertyName("First_UTC")]
        public DateTime FirstUTC { get; set; }
        [JsonPropertyName("Last_UTC")]
        public DateTime LastUTC { get; set; }
        [JsonPropertyName("Season")]
        [JsonConverter(typeof(JsonStringEnumConverter))]
        public Season MarsSeason { get; set; }
        [JsonPropertyName("PRE")]
        public DataDescription AtmosphericPressure { get; set; }
    }

I've created an class wrapper (MarsWheatherRootObject) to use class attribute.

internal class MarsWheatherRootObjectConverter : JsonConverter<MarsWheatherRootObject>
    {
        public override MarsWheatherRootObject Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options)
        {
            var root = new MarsWheatherRootObject();
            var list = new List<MarsWheather>();
            while (reader.Read())
            {
                switch (reader.TokenType)
                {
                    case JsonTokenType.PropertyName:
                    case JsonTokenType.String:
                        var keyStr = reader.GetString();
                        int key;
                        if (!int.TryParse(keyStr, out key))
                        {
                            reader.Skip();
                        }
                        else
                        {
                            var value = JsonSerializer.Deserialize<MarsWheather>(ref reader, options);
                            value.Sol = key;
                            list.Add(value);
                        }
                        break;
                    default: break;
                }
            }
            root.MarsWheather = list;
            return root;
        }

        public override void Write(Utf8JsonWriter writer, MarsWheatherRootObject value, JsonSerializerOptions options)
        {
            throw new NotImplementedException();
        }
    }

I don't need to serialize the object. That's why I don't write code for Write method by now.