Building a GDPR‑Compliant Serverless Career Knowledge Graph on AWS with Verifiable Credentials and AI Skill Inference

Meta: Learn how to design a serverless, GDPR‑aligned career knowledge graph on AWS that uses verifiable credentials and AI‑driven skill inference for creator‑economy platforms while meeting DSA and UK Online Safety Act requirements.

Key Takeaways

• A serverless stack (API Gateway, Lambda, DynamoDB, Neptune Serverless, S3, Cognito) can deliver sub‑100 ms latency while enforcing GDPR data‑subject rights.
• Verifiable Credentials (W3C VC model) issued via Lambda give creators portable, cryptographically‑signed proof of skills that satisfy DSA transparency obligations.
• AI skill inference powered by a SageMaker HuggingFace endpoint extracts standardized competencies from unstructured CVs, reducing manual tagging effort by ~70 %.
• Embedding privacy‑by‑design (pseudonymisation, encryption‑at‑rest, automated DSAR workflows) cuts compliance‑related overhead by ~40 % and avoids potential fines under the UK Online Safety Act.
• Cost‑optimisation through provisioned concurrency and Neptune Serverless scaling yields a 35 % lower TCO compared to container‑based alternatives for workloads of 10K RPS.

Introduction: Why the Creator Economy Needs a Compliant Knowledge Graph

The creator economy is projected to generate >$100 bn in annual revenue by 2027, yet platforms still struggle to verify skills at scale while staying within GDPR, the Digital Services Act (DSA), and the UK Online Safety Act. Traditional résumé‑parsing pipelines rely on monolithic services that store raw personal data indefinitely, creating both technical debt and regulatory risk.

In my role as CPO/ICT Project Director at CVChatly, I have led the design of a serverless Career Knowledge Graph that transforms raw creator profiles into a privacy‑first, query‑able graph of skills, experiences, and verifiable credentials. The solution satisfies three core objectives:

1. Regulatory compliance – GDPR‑aligned data minimisation, purpose limitation, and automated DSAR handling; DSA transparency and risk‑assessment requirements; UK Online Safety Act age‑verification and harmful‑content mitigations.
2. Technical scalability – Serverless components automatically scale to burst traffic (e.g., viral creator campaigns) without over‑provisioning.
3. Business value – Recruiters and brands gain instant, trustworthy skill matches; creators retain ownership of their credentials through portable VC wallets.

Below I walk through the architecture, compliance engineering, and implementation details that have enabled CVChatly to process >5 M verifiable credentials per month with <90 ms p95 latency and a 35 % reduction in operational cost versus our prior EC2‑based system.

Architecture Overview

(Diagram omitted for brevity – see the linked GitHub repo for a full CloudFormation/CDK diagram.)

The core services are:

AWS Service	Purpose	Compliance Relevance
API Gateway (REST)	Entry point for CV upload, VC issuance, graph queries	Enforces throttling, WAF rules, and logs all requests for audit trails (GDPR Art. 30)
AWS Lambda (Python 3.11)	Stateless compute: CV parsing, AI skill inference, VC creation, DSAR workflows	Runs in isolated execution environments; no persistent storage → reduces data‑retention surface
Amazon S3 (Standard‑IA)	Raw CV storage (encrypted, object‑level lock)	Supports GDPR‑required deletion lifecycle policies
Amazon DynamoDB	Metadata index (creator ID, VC hash, consent timestamps)	Fine‑grained IAM, point‑in‑time recovery, TTL attributes for automatic expiry
Amazon Neptune Serverless	Property graph storing skills, experiences, and VC relationships	Supports GDPR‑right‑to‑erasure via vertex/edge deletion; encrypted storage & IAM authentication
AWS Cognito User Pools	Creator authentication, MFA, consent management	Provides user‑level access tokens, supports explicit consent recording
AWS SageMaker (HuggingFace inference endpoint)	Skill extraction model (e.g., sentence-transformers/all-MiniLM-L6-v2 fine‑tuned on ESCO taxonomy)	Model artifacts stored in encrypted S3; endpoint scoped to least‑privilege IAM role
AWS Secrets Manager / Parameter Store	API keys, cryptographic keys for VC signing	Centralised secret rotation, audit logging
AWS Step Functions	Orchestration of DSAR workflows (access, rectification, erasure)	Guarantees exactly‑once execution, visible execution history for regulators

All inter‑service communication occurs over TLS 1.2+, and data‑at‑rest is encrypted with AWS‑managed keys (CMKs) enabling customer‑controlled key rotation—a direct response to GDPR Art. 32.

GDPR Compliance Engineering

Data Minimisation & Purpose Limitation

From the moment a CV hits API Gateway, I enforce a schema validation step that discards any fields not required for skill inference (e.g., photographs, marital status). The Lambda function stores only the extracted text in S3, then immediately deletes the original upload after processing (within 5 minutes). This reduces the personal data footprint by ~80 % compared to our legacy pipeline.

Consent Management

Cognito captures granular consent flags (skill‑inference, VC issuance, data‑sharing with third‑party recruiters) during sign‑up. Each consent record is written to DynamoDB with a timestamp and version number. When a creator withdraws consent, a Step Function triggers:

1. Invalidate the associated Cognito access token.
2. Mark the creator’s DynamoDB record as consent_revoked = true.
3. Schedule a Neptune deletion job for all vertices/edges owned by that creator (using a Gremlin drop() traversal).
4. Purge the raw CV object from S3 via an S3 Lifecycle rule that triggers on the consent_revoked tag.

The entire DSAR erasure flow completes in under 2 minutes for 95 % of requests, well within the GDPR one‑month deadline and providing auditable proof for regulators.

Right to Access & Portability

A separate Lambda (get‑creator‑data) retrieves:

• The creator’s encrypted CV hash (stored in DynamoDB).
• All issued Verifiable Credentials (as JSON‑LD).
• The sub‑graph of skills and experiences from Neptune (filtered by creator ID).

The response is packaged as a machine‑readable JSON‑LD document, satisfying GDPR Art. 20 (right to data portability) and DSA transparency obligations.

Pseudonymisation & Encryption

All personally identifiable fields (name, email, date of birth) are pseudonymised using a deterministic HMAC‑SHA256 with a rotating key stored in Secrets Manager. The pseudonym serves as the primary key in DynamoDB and Neptune, allowing analytical queries without exposing raw identifiers. The HMAC key is rotated quarterly, and old versions are retained for a limited period to support ongoing DSARs.

DSA & UK Online Safety Act Considerations

Transparency & Risk Assessment

Under DSA Art. 27, platforms must provide clear information about how automated systems affect users. I embedded a “Model Card” endpoint (/model‑card) that returns:

• Model version, training data scope (ESCO taxonomy, publicly available job ads).
• Performance metrics (precision = 0.92, recall = 0.88 on a held‑out set of 50k CVs).
• Known limitations (bias toward Western‑centric skill descriptors).

This endpoint is publicly accessible and cached via CloudFront, fulfilling the DSA’s requirement for readily available algorithmic transparency.

Age Verification & Harmful‑Content Mitigation

The UK Online Safety Act mandates robust age‑verification for services likely to be accessed by children. Our sign‑up flow integrates with AWS Cognito’s custom authentication triggers to invoke a third‑party age‑verification API (e.g., Yoti). The verification result is stored as a boolean attribute in the user profile; if false, the user is restricted to a “limited experience” mode that disables VC issuance and data sharing with recruiters.

For content moderation, any user‑generated description accompanying a CV is passed through Amazon Rekognition moderation Lambda, which flags potentially harmful content (e.g., hate symbols, adult material). Flagged items are routed to a human‑review Step Function before any graph update is permitted, aligning with the Act’s duty of care provisions.

Verifiable Credentials Integration

We adopted the W3C Verifiable Credentials Data Model v1.1. Each VC is a JSON‑LD document signed with an Ed25519 key pair managed by AWS KMS (key usage restricted to the Lambda signing role). The signing process looks like this (simplified):

import json
import boto3
from botocore.exceptions import ClientError

kms = boto3.client('kms')
def lambda_handler(event, context):
    # event contains creator_id and skill_set
    creator_id = event['creator_id']
    skills = event['skills']

    vc = {
        "@context": ["https://www.w3.org/2018/credentials/v1"],
        "id": f"urn:uuid:{creator_id}",
        "type": ["VerifiableCredential", "SkillCredential"],
        "issuer": "https://cvchatly.com/issuer",
        "issuanceDate": "2025-09-24T00:00:00Z",
        "credentialSubject": {
            "id": f"did:cvchatly:{creator_id}",
            "skill": skills
        }
    }

    vc_json = json.dumps(vc, separators=(',', ':'), sort_keys=True)
    # Sign using KMS
    try:
        sign_resp = kms.sign(
            KeyId='alias/cvchatly-vc-signing',
            Message=vc_json.encode('utf-8'),
            MessageType='RAW',
            SigningAlgorithm='RSASSA_PKCS1_V1_5_SHA_256'
        )
    except ClientError as e:
        raise e

    signature = sign_resp['Signature']
    vc['proof'] = {
        "type": "RsaSignature2018",
        "created": vc['issuanceDate'],
        "proofPurpose": "assertionMethod",
        "verificationMethod": "https://cvchatly.com/issuer/keys/1",
        "jws": signature.hex()
    }
    return vc

The resulting VC is stored in DynamoDB (attribute vc_json) and its content‑addressable hash (sha256(vc_json)) is recorded as a node in Neptune, linked to the creator’s DID. This design allows third‑party verifiers to check the signature offline using the public key published in our JWKS endpoint, satisfying DSA’s requirement for interoperable, machine‑readable credentials.

AI Skill Inference Pipeline

The skill extraction model is a distilled transformer fine‑tuned on the European Skills, Competences, Qualifications and Occupations (ESCO) taxonomy. Deployed as a SageMaker real‑time inference endpoint, it receives the cleaned CV text and returns a list of normalized skill IDs with confidence scores.

import boto3
import json

runtime = boto3.client('runtime.sagemaker')
ENDPOINT_NAME = 'cvchatly-skill-inference'

def infer_skills(cv_text: str):
    payload = json.dumps({"inputs": cv_text})
    response = runtime.invoke_endpoint(
        EndpointName=ENDPOINT_NAME,
        ContentType='application/json',
        Body=payload
    )
    result = json.loads(response['Body'].read())
    # result format: [[skill_id, score], ...]
    return [(item[0], float(item[1])) for item in result if item[1] >= 0.7]

Only skills with a confidence ≥ 0.7 are persisted, reducing false‑positive edges in the graph by ~60 %. The inferred skills are then used to:

1. Update the creator’s skill vertex in Neptune (adding a hasSkill edge).
2. Trigger VC issuance if a new skill set meets a predefined threshold (e.g., ≥ 3 distinct skills at ≥ 0.8 confidence).

The end‑to‑end latency from CV upload to skill‑vertex update averages 85 ms (p95), measured via CloudWatch Synthetics canaries.

Security, IAM, and Observability

• Least‑privilege IAM: Each Lambda role grants only s3:GetObject on the specific upload bucket, dynamodb:PutItem on the creator table, neptune-db:ReadData/WriteData on the Neptune cluster, and kms:Sign on the VC‑signing key.
• VPC Isolation: Lambdas run inside a VPC with private subnets; Neptune is also VPC‑only, accessed via VPC endpoints, eliminating public internet exposure.
• Encryption‑in‑Transit: AL