WebLogic VMs to Kubernetes: A UPI Migration Playbook

Your UPI issuer stack has run on the same four WebLogic VMs for six years. It works. Nobody touches it outside a Sunday-night change window, and the runbook lives in one architect’s head. Then three clocks converge in a single quarter: the support clock on your WebLogic version, the audit calendar, and a capacity plan that assumes UPI volumes keep doing what they have been doing. At that point “it works” stops being an argument, and someone asks you for a migration plan by Friday. This is that plan — a WebLogic to Kubernetes migration worked end to end, with the real commands, the real error strings, and the decisions that actually matter. The companion runbook, with every command executed end to end against a live domain, is on GitHub at prasadgujar22/weblogic-to-kubernetes-migration.

One qualifier before you invest the next twenty minutes: this is a plain-WLS path. If your domain is JRF/FMW — SOA, OIM/OIG, WebCenter, anything with an RCU schema — your route is Domain on PV and the packaging half of this guide does not apply to you. Read the decision table, then treat the rest as background.

Why Now: The Support Clock and the Capacity Wall

The business case writes itself, but only if you get the version story right. WebLogic Server 12.2.1.4 is the terminal patch set of the 12.2.1 line. Premier Support ends in December 2026 and Extended Support in December 2027. Oracle does revise its lifecycle tables, so re-check the current revision of the Oracle Lifetime Support Policy for Fusion Middleware PDF on the day you put the date in a steering-committee deck — but plan against those two dates now, because the first one is inside most banks’ next budget cycle.

The trap is the target version. 14.1.1.0 is not the LTS release of the 14.1.x line. Oracle’s positioning, as relayed through the Error Correction Policy, was that a subsequent 14.1.x release would start a grace-period clock on 14.1.1 — and that subsequent release is 14.1.2.0, the actual 14c LTS on JDK 17/21. If your migration lands on 14.1.1.0 you will have spent a year of change-management capital to arrive somewhere you have to leave again. Land on 14.1.2.0.

The capacity side is less negotiable than the support side. UPI cleared 22,716.07 million transactions worth ₹28.92 lakh crore in June 2026, after 23,201.93 million worth ₹29.90 lakh crore in May, per NPCI’s monthly product statistics. Note that June came in below May: the curve is not a straight line month to month, and a capacity plan built on the last single headline figure is a capacity plan built on noise. Take the trend, not the print. And note the totals exclude Credit Card on UPI and Credit Line on UPI, so actual throughput across your rails is higher than either number. Against that curve, scaling an issuer switch by cloning a VM, patching it, joining it to the cluster and updating the load balancer takes weeks of elapsed calendar time. Scaling a Kubernetes Deployment takes seconds. That gap, not fashion, is the argument.

Be equally clear about what containerisation does not buy you. A monolithic adapter EAR is still a monolithic adapter EAR inside a pod. This is a re-platform, not a re-architecture. If someone in the room is expecting microservices to fall out of the other end, correct that expectation now, in writing.

The Workload: What Actually Runs on WebLogic in a UPI Stack

On the issuer side — the payer’s bank — WebLogic typically hosts the UPI switch or issuer adapter that terminates NPCI’s XML API messages, the PIN and credential validation flow fronting the HSM, account and VPA resolution, and debit orchestration into the Core Banking System. On the acquirer side — the beneficiary bank, and often the PSP bank sponsoring an app — it hosts collect and merchant request handling, credit posting, QR and merchant-onboarding APIs, and the settlement and dispute harnesses that chew through NPCI files.

Mechanically, that means adapter EARs and WARs, ISO 8583 ↔ XML ↔ JSON transformation tiers (sometimes OSB, often hand-rolled JAX-WS/JAX-RS), Finacle/Flexcube/BaNCS connector modules, JMS queues carrying the async legs, EJB timers driving retries and reversals, and occasionally Coherence. None of that is exotic. What matters is which parts hold state. One exclusion up front: Coherence*Web and managed Coherence have their own operator path and are out of scope here — if you have them, that is a separate design exercise to finish before this playbook applies.

That right-hand column is the whole project — and the reason is that in UPI, the same logical transaction legitimately arrives more than once. If your dedupe key lives in a file on a VM’s local disk, containerisation does not break it — it exposes that it was already broken.

Phase 0: Discovery — You Cannot Migrate What You Have Not Inventoried

Run WebLogic Deploy Tooling against the running source domain. It produces three artefacts — a model YAML, a variables properties file, and an archive containing your deployable applications.

# Use the JDK the SOURCE domain runs on — JDK 8 for a 12.2.1.4 estate.
# Discovery runs offline WLST out of the source ORACLE_HOME; JDK 17/21
# belongs on the target/image-build side only.
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk
export ORACLE_HOME=/u01/app/oracle/middleware

weblogic-deploy/bin/discoverDomain.sh \
  -oracle_home $ORACLE_HOME \
  -domain_home /u01/domains/upiadapter \
  -model_file  /tmp/model/upiadapter.yaml \
  -variable_file /tmp/model/upiadapter.properties \
  -archive_file  /tmp/model/upiadapter.zip \
  -domain_type WLS

What discovery gives you is topology, resources, deployments and tuning. What it deliberately does not give you is secrets: every credential comes out as a blank @@PROP:...@@ token. That is correct behaviour, and it is also the source of the most common first-boot failure in the whole exercise.

Discovery is necessary but not sufficient. Walk the domain manually and write down: every datasource and the real database host behind it, every JMS module and its store type, the security realm and its providers, custom keystores, any T3/T3S usage, network channels, startup and shutdown classes, JVM arguments, and deployment plans. Then verify every application’s actual context root from its WAR — never from the artifact name:

unzip -p upi-issuer-adapter.war WEB-INF/weblogic.xml | grep -A2 "context-root"

The Decision That Actually Matters: Model in Image vs Domain on PV

WKO supports three domainHomeSourceType values, and they are not equivalent in 2026.

Oracle’s own framing is deliberately neutral — choose either Domain on PV or Model in Image, depending on your needs — and I will not overclaim past that. The genuinely defensible rule is the JRF split. For a plain-WLS UPI adapter domain, Model in Image with auxiliary images is the right answer, and the reason is audit as much as engineering: the image tag is the change record, the model diff is the evidence, and a rebuild is reproducible in a way a hand-edited domain home never is. For the authoritative list of which WebLogic features are supported in Kubernetes at all, cite My Oracle Support Doc ID 2349228.1 — it is maintained, and it changes.

The WebLogic to Kubernetes Migration Runbook, Part 1: Base Image, Operator, Tooling

Authenticate to Oracle Container Registry

docker login container-registry.oracle.com
# Username: your Oracle SSO email
# Password: the Auth Token generated on container-registry.oracle.com or in the OCI Console

Then pull the exact tag for your WLS version, JDK and OS from the registry’s repository page. Do not guess the tag string; the naming convention has changed across releases and a wrong guess costs you a multi-gigabyte pull.

docker pull container-registry.oracle.com/middleware/weblogic:14.1.2.0-generic-jdk17-ol9
docker images | grep weblogic

Install the WebLogic Kubernetes Operator 4.x

helm repo add weblogic-operator https://oracle.github.io/weblogic-kubernetes-operator/charts
helm repo update

kubectl create namespace wls-operator
kubectl create namespace wls-domain
kubectl label namespace wls-domain weblogic-operator=enabled

helm install weblogic-operator weblogic-operator/weblogic-operator \
  --namespace wls-operator \
  --set "domainNamespaceSelectionStrategy=LabelSelector" \
  --set "domainNamespaceLabelSelector=weblogic-operator=enabled"

kubectl get pods -n wls-operator
kubectl get crd | grep weblogic   # domains.weblogic.oracle AND clusters.weblogic.oracle

Keep the operator and the domains in separate namespaces, and use the LabelSelector strategy rather than watching everything — in a shared bank cluster you will eventually be asked to prove which namespaces this controller can touch. Verify both CRDs land; if clusters.weblogic.oracle is missing you are on a 3.x chart.

Fetch the Image Tool and WDT

mkdir -p ~/wls-image-build/model && cd ~/wls-image-build

curl -fsSL -o imagetool.zip https://github.com/oracle/weblogic-image-tool/releases/latest/download/imagetool.zip
curl -fsSL -o weblogic-deploy.zip https://github.com/oracle/weblogic-deploy-tooling/releases/latest/download/weblogic-deploy.zip
unzip -q imagetool.zip
unzip -q weblogic-deploy.zip

export JAVA_HOME=/usr/lib/jvm/java-21-openjdk-amd64
chmod +x imagetool/bin/imagetool.sh
./imagetool/bin/imagetool.sh cache addInstaller --type WDT --version latest --path weblogic-deploy.zip

The Runbook, Part 2: Surgery on the Discovered Model

Copy the discovered model, properties and archive into the build directory, then apply four edits. Each one below corresponds to a real first-boot failure. Skip any of them and you will reproduce it.

Edit 1 — Credentials: @@PROP@@ → @@SECRET@@

# BEFORE (from discoverDomain — fails at introspection: "username was not provided")
domainInfo:
    AdminUserName: '@@PROP:AdminUserName@@'
    AdminPassword: '@@PROP:AdminPassword@@'

# AFTER
domainInfo:
    AdminUserName: '@@SECRET:__weblogic-credentials__:username@@'
    AdminPassword: '@@SECRET:__weblogic-credentials__:password@@'

__weblogic-credentials__ is a special name — the operator supplies it automatically from domain.spec.webLogicCredentialsSecret. Every other credential needs a secret of your own, referenced by name. For the CBS/UPI schema datasource:

JDBCDriverParams:
    PasswordEncrypted: '@@SECRET:wlsdomain-jdbc-oracleds-secret:password@@'
    URL: jdbc:oracle:thin:@//<db-host>:1521/<service-name>
    Properties:
        user:
            Value: '@@SECRET:wlsdomain-jdbc-oracleds-secret:username@@'

Edit 2 — Delete the on-prem topology

The operator manages server lifecycle through pods. NodeManager processes bound to fixed host IPs are meaningless here.

# DELETE entirely:
topology:
    NMProperties: {...}
    UnixMachine: {...}
    Machine: {...}
    SecurityConfiguration:
        NodeManagerUsername: '@@PROP:...@@'
        NodeManagerPasswordEncrypted: '@@PROP:...@@'

# Remove "Machine: <name>" under each Server: entry.
# Blank out — do not delete — each server's ListenAddress:
Server:
    AdminServer:
        ListenAddress: ''      # was 192.168.x.x — the Operator owns pod networking
    ms1:
        ListenAddress: ''
        ListenPort: 7003       # keep explicit ListenPorts

Edit 3 — Give AdminServer an explicit ListenPort

Without one, the operator cannot resolve a real port for the default channel and the external NodePort Service creation fails outright with Service "wlsdomain-adminserver-ext" is invalid: spec.ports: Required value.

Server:
    AdminServer:
        ListenAddress: ''
        ListenPort: 7001

Edit 4 — Disable SecureMode at the domain level

In 14.1.x, ProductionModeEnabled: true turns secure mode on by default unless you set it explicitly — which is exactly what you are about to do. Inside a pod network without full PKI and hostname verification, AdminServer dies on boot with a BEA-000386 MultiException — a null ServerChannel.getPublicAddress() and a failed IIOPService post-construct. A per-server SSL.Enabled: false does not save you; domain-level SecureMode overrides it.

Note where this bites hardest: the issuer leg talks to an HSM and carries PIN-block traffic. Turning secure mode off to get the domain booting is defensible for a like-for-like lift; leaving it off once issuer traffic is on the platform is not. That is precisely why the follow-up below needs an owner and a date, not a backlog ticket.

topology:
    SecurityConfiguration:
        SecureMode:
            SecureModeEnabled: false

Build the image with the WebLogic Image Tool

cd ~/wls-image-build
export JAVA_HOME=/usr/lib/jvm/java-21-openjdk-amd64

# Build a small AUXILIARY image carrying only the WDT model, variables,
# archive and the WDT installer. The WebLogic binaries stay in Oracle's
# stock base image, untouched.
./imagetool/bin/imagetool.sh createAuxImage \
  --tag upi-model:1.0 \
  --wdtModel model/upiadapter.yaml \
  --wdtVariables model/upiadapter.properties \
  --wdtArchive model/upiadapter.zip

docker images | grep upi-model

Expect validateModel.sh completed successfully (exit code = 0) with Total: SEVERE: 0 WARNING: 0.

Bump the tag on every model change — 1.0 → 1.1 → and so on. Model in Image has no hot-patch path, and that is the feature, not the limitation.

The Runbook, Part 3: Secrets, Manifests, Rollout, Proof

Secrets, created by a human

kubectl -n wls-domain create secret generic wlsdomain-weblogic-credentials \
  --from-literal=username='<admin-username>' \
  --from-literal=password='<admin-password>'

kubectl -n wls-domain create secret generic wlsdomain-runtime-encryption-secret \
  --from-literal=password="$(openssl rand -base64 32)"

kubectl -n wls-domain create secret generic wlsdomain-jdbc-oracleds-secret \
  --from-literal=username='<db-app-user>' \
  --from-literal=password='<db-app-password>'

kubectl -n wls-domain create secret docker-registry ocr-secret \
  --docker-server=container-registry.oracle.com \
  --docker-username='<oracle-sso-email>' \
  --docker-password='<oracle-auth-token>'

The registry pull secret belongs in this section for the same reason as the rest: it carries a credential, so it is created by a person at a terminal and referenced by name from the manifest. Note it takes your Oracle Auth Token, not your SSO password — the same distinction that is already on the readiness checklist.

Run these yourself, at a terminal, as a named human with cluster access. Never through a build script, a CI job whose logs are retained, or an AI assistant session. In a payments environment the credential you paste into a chat window is a reportable incident.

Manifests — apply the Cluster resource before the Domain

WKO 4.x moved cluster configuration out of the Domain and into its own resource. Manifests copied from a 3.x-era guide fail with error: unknown field "spec.clusters[0].clusterName". Check the installed CRD before you write anything:

kubectl explain domain.spec.clusters
kubectl explain cluster.spec

# cluster.yaml
apiVersion: weblogic.oracle/v1
kind: Cluster
metadata:
  name: wlsdomain-wlscluster
  namespace: wls-domain
spec:
  clusterName: WLSCluster      # must match the cluster name in the WDT model exactly
  replicas: 2
  serverPod:
    resources:
      requests:
        cpu: "250m"
        memory: "1Gi"
      limits:
        cpu: "1"
        memory: "2Gi"

# domain.yaml
apiVersion: weblogic.oracle/v9
kind: Domain
metadata:
  name: wlsdomain
  namespace: wls-domain
  labels:
    weblogic.domainUID: wlsdomain
spec:
  domainHome: /u01/domains/upiadapter
  domainHomeSourceType: FromModel
  # Stock Oracle base image — your model does NOT live here any more.
  image: container-registry.oracle.com/middleware/weblogic:14.1.2.0-generic-jdk17-ol9
  imagePullPolicy: IfNotPresent
  imagePullSecrets:
    - name: ocr-secret
  webLogicCredentialsSecret:
    name: wlsdomain-weblogic-credentials
  includeServerOutInPodLog: true
  replicas: 2
  configuration:
    secrets:
      - wlsdomain-jdbc-oracleds-secret
    model:
      domainType: WLS
      runtimeEncryptionSecret: wlsdomain-runtime-encryption-secret
      # The auxiliary image supplies the model, variables and archive.
      auxiliaryImages:
        - image: upi-model:1.0
  adminServer:
    adminService:
      channels:
        - channelName: default
          nodePort: 30070      # lab convenience only — see the note below
  clusters:
    - name: wlsdomain-wlscluster
  serverPod:
    env:
      - name: USER_MEM_ARGS
        # First-boot sizing only. See the percentage-based settings
        # in the Day 2 section before you put real traffic on this.
        value: "-Xms256m -Xmx512m"

One caveat on that NodePort: 30070 is a lab convenience. In production the admin channel stays ClusterIP behind a jump host or a kubectl port-forward, and your firewall team never hears about it — the same argument as T3, for the same reason.

kubectl apply -f cluster.yaml
kubectl apply -f domain.yaml

kubectl get pods -n wls-domain -w
kubectl get domain wlsdomain -n wls-domain -o jsonpath='{.status.conditions}'

The sequence is introspector Job (runs once) → AdminServer pod → cluster member pods. Success is [{"status":"True","type":"Available"},{"status":"True","type":"Completed"}]. Anything less is not “nearly there”.

Proof of life, not proof of config

A green pod proves the JVM started. It does not prove the datasource reached the CBS schema. Ask the domain itself, over the WLS RESTful management interface — the same interface the Monitoring Exporter scrapes in the Day 2 section, so you are not introducing a second mechanism to maintain. There is no script to copy into the pod and nothing to install.

ADMIN_USER=$(kubectl -n wls-domain get secret wlsdomain-weblogic-credentials \
  -o jsonpath='{.data.username}' | base64 -d)
ADMIN_PASS=$(kubectl -n wls-domain get secret wlsdomain-weblogic-credentials \
  -o jsonpath='{.data.password}' | base64 -d)

# One datasource on one server.
kubectl run wls-rest --image=curlimages/curl:latest --rm -i --restart=Never -n wls-domain -- \
  curl -s --user "$ADMIN_USER:$ADMIN_PASS" \
    -H "X-Requested-By: MyClient" -H "Accept: application/json" \
    "http://wlsdomain-adminserver:7001/management/weblogic/latest/domainRuntime/serverRuntimes/ms1/JDBCServiceRuntime/JDBCDataSourceRuntimeMBeans/UPIDataSource?fields=name,state,currentCapacity,activeConnectionsCurrentCount&links=none"

That answers for ms1. The version worth keeping in the runbook is the search call, which walks every server runtime and every datasource under it in a single POST — so you cannot pass on ms1 and quietly miss ms3:

kubectl run wls-rest --image=curlimages/curl:latest --rm -i --restart=Never -n wls-domain -- \
  curl -s -X POST --user "$ADMIN_USER:$ADMIN_PASS" \
    -H "X-Requested-By: MyClient" \
    -H "Accept: application/json" \
    -H "Content-Type: application/json" \
    -d '{
          "links": [], "fields": [],
          "children": {
            "serverRuntimes": {
              "links": [], "fields": [ "name" ],
              "children": {
                "JDBCServiceRuntime": {
                  "links": [],
                  "children": {
                    "JDBCDataSourceRuntimeMBeans": {
                      "links": [],
                      "fields": [ "name", "state", "currentCapacity",
                                  "activeConnectionsCurrentCount" ]
                    }
                  }
                }
              }
            }
          }
        }' \
    http://wlsdomain-adminserver:7001/management/weblogic/latest/domainRuntime/search

You want state: Running on every datasource on every server, and a non-zero current capacity. Then hit the real context root — the one you read out of weblogic.xml, not the one you assumed. Same disposable-pod trick, for the same reason: managed-server images are minimal and ship without curl.

kubectl run curltest --image=curlimages/curl:latest --rm -i --restart=Never -n wls-domain -- \
  curl -s -o /dev/null -w "HTTP_%{http_code}\n" http://wlsdomain-cluster-wlscluster:7003/<actual-context-root>/

An HTTP 200 from inside the cluster, against the operator-generated cluster Service, on the real context root, with a datasource in Running state — that is the first honest checkpoint in this migration.

What Breaks in a Payments Workload (And How to Handle It)

The domain is up. Pods are 1/1 Running, the datasource has real capacity, your context roots return 200. That is where most teams declare victory — and where a UPI workload starts quietly accumulating reconciliation tickets.

Sticky sessions and unicast: two non-negotiables

Oracle’s operator documentation is unambiguous: the load balancer in front of a WebLogic cluster must make the HTTP flow sticky, and Oracle’s WebLogic-for-OKE documentation says you must add session affinity annotations to the ingress. Pre-flight item, not post-go-live tuning.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: upi-adapter-ingress
  namespace: wls-domain
  annotations:
    nginx.ingress.kubernetes.io/affinity: "cookie"
    nginx.ingress.kubernetes.io/affinity-mode: "persistent"
    nginx.ingress.kubernetes.io/session-cookie-name: "WLSSTICKY"
spec:
  tls:
    - hosts:
        - upi-adapter.internal.example
      secretName: upi-adapter-tls
  rules:
    - host: upi-adapter.internal.example
      http:
        paths:
          - path: /upiadapter
            pathType: Prefix
            backend:
              service:
                name: wlsdomain-cluster-wlscluster
                port: { number: 7003 }

The second constraint is cluster messaging: Oracle’s operator FAQ states that only unicast is supported on Kubernetes — most network fabrics do not carry multicast, and the operator FAQ lists Flannel and Calico as certified fabrics at the time of writing. If your VM-era domain still uses multicast, converting it is part of this migration.

JMS stores and transaction logs: the “done but not safe” failure

This is the most common way a WebLogic to Kubernetes migration passes UAT and fails in production. Reversals, deferred credit posting and recon drops sit in JMS queues with persistent stores, alongside the transaction logs that make two-phase commit recoverable. In a pod, the default filesystem is ephemeral. Kill the pod, lose the store, lose the recovery record. Decide explicitly: JDBC stores in a database that already survives your DR plan, or PV-backed file stores with node affinity so a restarted pod re-attaches to its own store. If your answer is neither, you are gambling with money movement.

Be specific about which queues you are protecting, because the two sides fail differently. Issuer-side queues carry reversals and timeout resolutions — losing one means a customer stays debited. Acquirer-side queues carry deferred credit posting and recon drops — losing one means a merchant is not paid and it surfaces in the next day’s reconciliation. Both are unacceptable; only the first shows up on a customer’s phone within minutes.

What that decision looks like in the model

Take the JDBC route. It is the one that survives a node evacuation without a conversation with your storage team. Three things change in the model: the store gets its own datasource, the JMS store moves from a file store to a JDBCStore, and every server gets an explicit transaction log store.

# model.yaml — JMS and TLOG persistence
resources:
    JDBCSystemResource:
        StoreDS:
            Target: 'WLSCluster'
            JdbcResource:
                JDBCDataSourceParams:
                    JNDIName: [ jdbc/StoreDS ]
                    GlobalTransactionsProtocol: None      # non-XA. Not negotiable — see below.
                JDBCDriverParams:
                    DriverName: oracle.jdbc.OracleDriver  # thin driver, NOT OracleXADataSource
                    URL: '@@SECRET:wlsdomain-jdbc-store-secret:url@@'
                    PasswordEncrypted: '@@SECRET:wlsdomain-jdbc-store-secret:password@@'
                    Properties:
                        user:
                            Value: '@@SECRET:wlsdomain-jdbc-store-secret:username@@'
                JDBCConnectionPoolParams:
                    TestTableName: SQL ISVALID
                    TestConnectionsOnReserve: true

    # WAS: FileStore: UPIJMSStore: { Directory: /u01/shared/jms, Target: ... }
    JDBCStore:
        UPIJMSStore:
            DataSource: StoreDS
            PrefixName: UPI_JMS_          # unique per store, or you corrupt both
            Target: 'WLSCluster'          # cluster-targeted, not per-server
            DistributionPolicy: Distributed
            MigrationPolicy: On-Failure   # default is Off. Read that again.

    JMSServer:
        UPIJMSServer:
            PersistentStore: UPIJMSStore
            Target: 'WLSCluster'

topology:
    Server:
        ms1:
            TransactionLogJDBCStore:
                Enabled: true
                DataSource: StoreDS
                PrefixName: TLOG_MS1_
        ms2:
            TransactionLogJDBCStore:
                Enabled: true
                DataSource: StoreDS
                PrefixName: TLOG_MS2_
# On a dynamic cluster this collapses to one ServerTemplate entry with
# PrefixName: 'TLOG_${serverName}_' — the macro does the per-server discipline for you.

The GlobalTransactionsProtocol: None line is the one reviewers try to delete because it looks like a downgrade. It is not. Oracle’s constraint is explicit: a JDBC store “must use a JDBC data source that uses a non-XA JDBC driver and has Supports Global Transactions disabled,” and for the TLOG store, “you cannot use a data source that is configured to use an XA JDBC driver or is configured to support global transactions.” The reason is circularity — the store is the thing that records how a global transaction ended, so it cannot be a participant in one. Your application loses nothing: WebLogic JMS stays fully XA-capable regardless of what its store is written on. Same database if you want; separate, non-XA datasource definition, always.

Prefix discipline is the other line that gets skipped. Two stores pointed at one table is not a warning in the log, it is silent corruption — Oracle’s wording is that if multiple JDBC stores share a table, “the behavior is undefined and data loss is likely.” A TLOG store cannot be shared at all: one per server, and no two servers may use the same one. TLOG_MS1_ and TLOG_MS2_ are not naming taste. They are what stops ms2’s recovery record from being written over ms1’s. And note the coupling you just bought: if the store database is unreachable at boot, the server does not start degraded, it fails to boot. That is the right behaviour for a payments domain, but say it out loud in the DR review rather than discovering it during one.

Migration policy, and why leasing has to be the database

MigrationPolicy defaults to Off. A cluster-targeted store on the default policy gives you distribution, not recovery: when ms2’s pod dies, its store instance dies with it and stays dead until ms2 is back. On-Failure is what actually hands the store to a surviving member. The JTA equivalent is a per-server JTAMigratableTarget with a failure-recovery policy, so someone claims the dead server’s transaction log instead of waiting politely for it.

Both of those need cluster leasing, and here the choice is already made for you. WebLogic offers two bases. Consensus leasing keeps the lease table in memory inside the cluster — and, per Oracle, “requires that all server instances in the cluster are started by Node Manager.” You deleted Node Manager in Edit 2, because the operator owns process lifecycle now. So consensus leasing is not a preference you rejected; it is an option you removed two sections ago. Database leasing is what is left, and Oracle notes it is “particularly useful if Node Manager is not running in the system,” which is an unusually direct description of a Kubernetes domain.

topology:
    Cluster:
        WLSCluster:
            MigrationBasis: database
            DataSourceForAutomaticMigration: LeasingDS   # must be non-XA, same rule as above
            AutoMigrationTableName: ACTIVE

    Server:
        ms1:
            JTAMigratableTarget:
                Cluster: 'WLSCluster'
                MigrationPolicy: failure-recovery
        ms2:
            JTAMigratableTarget:
                Cluster: 'WLSCluster'
                MigrationPolicy: failure-recovery

The test: kill ms2 with a message in flight

Configuration is a claim. The test is the evidence. Put a persistent message on a browsable queue, confirm it is on ms2, delete the pod, and check two things afterwards: the message is still countable somewhere in the cluster, and someone has taken ownership of ms2’s transaction log.

# 0. Credentials, as in the proof-of-life section.
ADMIN_USER=$(kubectl -n wls-domain get secret wlsdomain-weblogic-credentials \
  -o jsonpath='{.data.username}' | base64 -d)
ADMIN_PASS=$(kubectl -n wls-domain get secret wlsdomain-weblogic-credentials \
  -o jsonpath='{.data.password}' | base64 -d)

# 1. Baseline, then kill ms2 while the message is still queued.
kubectl delete pod wlsdomain-ms2 -n wls-domain

# 2. Ask the domain what survived. One call, both facts.
kubectl run wls-rest --image=curlimages/curl:latest --rm -i --restart=Never -n wls-domain -- \
  curl -s -X POST --user "$ADMIN_USER:$ADMIN_PASS" \
    -H "X-Requested-By: MyClient" \
    -H "Accept: application/json" \
    -H "Content-Type: application/json" \
    -d '{
          "links": [], "fields": [],
          "children": {
            "serverRuntimes": {
              "links": [], "fields": [ "name" ],
              "children": {
                "JMSRuntime": {
                  "links": [],
                  "children": {
                    "JMSServers": {
                      "links": [],
                      "fields": [ "name", "messagesCurrentCount" ]
                    }
                  }
                },
                "JTARuntime": {
                  "links": [],
                  "children": {
                    "recoveryRuntimeMBeans": {
                      "links": [],
                      "fields": [ "name", "active",
                                  "initialRecoveredTransactionTotalCount" ]
                    }
                  }
                }
              }
            }
          }
        }' \
    http://wlsdomain-adminserver:7001/management/weblogic/latest/domainRuntime/search

Trimmed to the two facts that matter, a passing run looks like this — ms1 is carrying the message that was on ms2, and ms1 has activated a recovery runtime named ms2:

{ "serverRuntimes": { "items": [
  { "name": "ms1",
    "JMSRuntime": { "JMSServers": { "items": [
      { "name": "UPIJMSServer@ms1", "messagesCurrentCount": 1 } ] } },
    "JTARuntime": { "recoveryRuntimeMBeans": { "items": [
      { "name": "ms1", "active": true, "initialRecoveredTransactionTotalCount": 0 },
      { "name": "ms2", "active": true, "initialRecoveredTransactionTotalCount": 3 } ] } } }
] } }

Two failure signatures to recognise. If messagesCurrentCount comes back 0 everywhere, your store was still ephemeral and you have just watched a reversal disappear in a lab instead of in production, which is the cheapest possible place to learn it. If the message survived but no surviving server shows an active recovery runtime named ms2, persistence is fixed and migration is not — check MigrationPolicy and the leasing datasource before you call it done.

This is what the checklist line means. “Recovery-tested by killing a pod mid-flight” is not a sentiment; it is the four commands above and a specific expected output. Run it before cutover, and run it again after the first patch cycle, because a model change that quietly reverts MigrationPolicy to Off is invisible until the day it is not.

T3, retry storms and the shutdown window

Keep T3/T3S internal to the cluster. Oracle’s preferred route for external RMI clients is tunnelling over HTTP through the load balancer; NodePorting a T3 channel in a payments environment hands you a privileged protocol on a port your firewall team has never heard of.

UPI retries are normal traffic, not exceptions — NPCI retries, PSP re-submissions and ReqChkTxn polling all mean the same logical transaction arrives more than once. Dedupe keyed on the NPCI transaction ID/RRN plus idempotent CBS posting is the correctness backbone of the platform. It is also a systemic risk: following the March 2025 UPI disruptions, NPCI’s post-incident analysis was reported (Economic Times, April 2025) to have pointed at PSP banks not waiting for responses and repeatedly flooding the system with Check Transaction calls. Take that as reported rather than settled — but take the lesson. That is a textbook retry storm, and containerisation makes one easier to generate, because scaling out is now a one-line change. Design exponential backoff with jitter, client-side rate limits on status polling, and circuit breakers around the CBS and NPCI legs before you scale anything.

Graceful shutdown matters for the same reason: a pod terminated mid-two-phase-commit is a reconciliation ticket with a customer’s money in it.

apiVersion: weblogic.oracle/v1
kind: Cluster
metadata:
  name: wlsdomain-wlscluster
  namespace: wls-domain
spec:
  clusterName: WLSCluster
  replicas: 4
  maxUnavailable: 1                      # explicit, not inherited from the Domain
  serverPod:
    terminationGracePeriodSeconds: 180   # > longest in-flight txn + drain
    resources:
      requests: { cpu: "2", memory: "4Gi" }   # requests == limits
      limits:   { cpu: "2", memory: "4Gi" }   # → Guaranteed QoS
    env:
      - name: TZ
        value: "Asia/Kolkata"
      - name: USER_MEM_ARGS
        value: "-XX:MaxRAMPercentage=70.0 -XX:InitialRAMPercentage=50.0 -XX:ActiveProcessorCount=2"

Requests equal limits on purpose. A CPU limit above the request puts the JVM in Burstable QoS and exposes it to CFS throttling, which shows up as exactly the kind of p99 excursion the 25% gate in the cutover table is watching for — and it will look like an application regression rather than a scheduling artefact, which is the worst way to spend a cutover window.

Heap there is a percentage of the container’s memory, not a fixed -Xmx guessed against the old VM’s RAM. A JVM that ignores its cgroup limit sizes itself against the node and gets OOMKilled under peak load — a bug that only surfaces when volume arrives. And plan egress early: your CBS and HSM firewall rules are keyed to the fixed source IPs of the old VMs, while pod IPs are ephemeral. Agree a dedicated node pool, an egress gateway or SNAT to a reserved range, and get the rules tested before the first transaction. This one bites during cutover, when nobody has time.

Cutover Without Losing a Transaction

Acquirer side first, issuer side last. The reason is risk asymmetry: collect and merchant flows tolerate a controlled retry far better than issuer-side debit authorisation. A retried collect is an annoyance. A mishandled debit is a customer with money deducted, a TAT-bound reversal clock, and a complaint that reaches the regulator. Put the forgiving flows on the new platform first and buy soak hours.

Run both estates live, split traffic at the load balancer, and hold each step across at least one genuine peak window. Off-peak green is not evidence.

Define the rollback trigger numerically, in writing, before cutover: a specific technical-decline delta against the VM baseline, a specific p99 threshold, a specific burn rate over a specific window. Argued-about-at-3am is not a trigger. Rollback must be a load-balancer weight change — seconds, reversible, no pipeline involved. If your rollback needs a redeploy, you do not have one.

Before the final switch, drain the JMS queues on the source domain and confirm dedupe continuity across both estates: a transaction deduped on the VM path must still be recognised as a duplicate when its retry lands on a pod. Keep the VM estate warm, patched and licensed until soak completes — decommissioning is a separate, later change record.

Day 2: Observability, Patching and Secrets

Before the dashboards, the probes — because the operator’s defaults assume a server that boots quickly. Left alone, the readiness probe first checks after 30 seconds and repeats every 5, and the liveness probe first checks after 30 seconds, repeats every 45, times out at 5 and fails on a single miss. A large EAR on a cold JVM does not respect that budget, and the result is a crash loop that reads like an application fault. Give first deploys headroom with serverPod.livenessProbe — raise initialDelaySeconds and set failureThreshold: 3 so one slow check is not fatal — and widen readinessProbe.timeoutSeconds before you widen anything else. Kubernetes startup probes are the cleaner idiom for this, but the operator manages the pod spec, so the supported route is probe tuning through the domain or cluster resource. Tighten it back once you know the real boot time; do not ship the loose values to production and forget them.

Use the WebLogic Monitoring Exporter, which scrapes the WLS RESTful management interface into Prometheus. It ships in lockstep with the operator — at the time of writing (August 2026), WKO 4.3.14 and WME 2.3.14 released the same day — so pin and upgrade them together.

The metrics that matter for an issuer switch are narrower than the default dashboard: heap used versus committed per server, JDBC datasource current capacity against maximum (a pool pinned at max means your CBS leg is backing up), JMS queue depth per destination, open session counts against your affinity assumptions, and p99 request latency. Alert on the datasource and queue depth first — those move before customers notice.

Weight them by side. On the issuer, datasource capacity against the CBS schema is the leading indicator — the debit leg is where latency turns into technical declines. On the acquirer, JMS queue depth matters more, because a backing-up credit-posting queue is invisible to the payer and shows up as a merchant complaint a day later.

Patching changes shape entirely. A quarterly Critical Patch Update means rebuild the image, bump the tag, roll the domain. Never patch a running container.

# Quarterly CPU: the MODEL does not change — only the base image does.
# Point the domain at the newly patched base and roll. The auxiliary
# image (upi-model:1.0) is untouched.
kubectl -n wls-domain patch domain wlsdomain --type=merge \
  -p '{"spec":{"image":"container-registry.oracle.com/middleware/weblogic:14.1.2.0-generic-jdk17-ol9-cpu-oct2026"}}'

# A CONFIG change is the mirror image: rebuild only the small aux image,
# bump its tag, and patch the auxiliaryImages entry instead.
./imagetool/bin/imagetool.sh createAuxImage \
  --tag upi-model:1.1 \
  --wdtModel model/upiadapter.yaml \
  --wdtVariables model/upiadapter.properties \
  --wdtArchive model/upiadapter.zip

# Secret rotation: the model never changes, only the secret and the restart marker.
kubectl -n wls-domain create secret generic wlsdomain-jdbc-oracleds-secret \
  --from-literal=username='<db-app-user>' \
  --from-literal=password='<new-password>' \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl -n wls-domain patch domain wlsdomain --type=merge \
  -p '{"spec":{"restartVersion":"'"$(date +%s)"'"}}'

# Scale via the Cluster resource, not the Domain.
kubectl -n wls-domain scale cluster wlsdomain-wlscluster --replicas=6

Note what rotation is under Model in Image: rotate the secret, bump the restart version, roll. The model and image are untouched, because the model only ever held a @@SECRET:@@ reference. That is a better rotation story than anything you had on the VMs.

Replicas live on the Cluster resource. Treat HPA with care rather than enthusiasm: WebLogic server startup is tens of seconds to minutes, so an autoscaler reacting to a UPI spike typically arrives after the spike has passed — and every node that can schedule a WebLogic pod is a node your licensing conversation must account for. Scheduled scale-ups ahead of known peaks beat reactive autoscaling here.

For backup and DR with an ephemeral domain home, back up four things: the WDT model and properties in git, the application archive, the secrets (via your secrets platform, not a YAML dump in a bucket), and the PVs behind your JMS stores and transaction logs. If the image is not rebuildable from the first two, your pipeline is the single point of failure — not the cluster.

Governance: Making the Auditor Comfortable

RBI’s Master Direction on IT Governance, Risk, Controls and Assurance Practices (RBI/2023-24/107, 7 November 2023) contains a section on Change and Patch Management and, explicitly, one titled Data Migration Controls. A re-platform of a UPI switch sits squarely inside both — which is good news, because Model in Image produces exactly the artefacts an auditor wants: the model diff, the image tags, the introspector logs, the datasource verification and the endpoint tests. Attach them to the change record rather than reconstructing them six months later.

If your Kubernetes is managed, the RBI Directions on Outsourcing of Information Technology Services, 2023 (RBI/2023-24/102) treat it as a third-party arrangement: due diligence, audit and inspection rights extending to the provider, a documented exit strategy with data portability, and concentration-risk management. “Can we leave this platform and take our data with us?” needs an answer with a signature under it.

Data localisation is where container migrations quietly create new exposure. RBI’s 2018 Storage of Payment System Data directive requires payment system data to be stored in systems located only in India, with narrowly defined carve-outs for the foreign leg of cross-border transactions, and RBI’s FAQ names payment credentials and transaction data as in scope. In a containerised stack that data leaks into logs, traces, metrics labels and sidecar buffers. Decide where Prometheus, your log store and any APM SaaS live — and what fields they retain — before you ship, with your compliance function.

On licensing, honesty beats confidence: I could not find an authoritative Oracle-published document covering WebLogic licensing specifics on Kubernetes. The widely-circulated interpretations come from consultancies that sell audit-defence services and therefore have a commercial interest in worst-case framing. They may well be right, but they are not Oracle documents. Node-count and processor questions belong in a written conversation with Oracle or an independent advisor before you scale a node pool. Architecturally, the defensible posture is a dedicated, fixed-size node pool with an enforced scheduling boundary — because it is easiest to describe and evidence, not because Oracle has blessed it.

Pitfalls: The Errors You Will Actually See

WebLogic to Kubernetes Migration Readiness Checklist

• ☐ Target database reachable from the cluster network on a static (not DHCP) address
• ☐ Oracle Container Registry login working with an Auth Token, not the SSO password; licence accepted
• ☐ Exact base image tag confirmed on the registry page — never guessed
• ☐ Operator installed, demonstrably watching the target namespace; both CRDs present
• ☐ Model: all @@PROP@@ credentials converted to @@SECRET@@
• ☐ Model: NodeManager / Machine topology removed, ListenAddress blanked
• ☐ Model: AdminServer has an explicit ListenPort
• ☐ Model: SecureModeEnabled: false at domain level, PKI hardening booked as a follow-up
• ☐ Cluster messaging confirmed unicast — no multicast anywhere in the model
• ☐ Every application’s real context-root read out of its WAR, not assumed
• ☐ Image built clean (0 SEVERE / 0 WARNING) and verified present node by node
• ☐ All secrets created via kubectl by a human — never a script, chat session or CI log
• ☐ Cluster resource applied before the Domain that references it
• ☐ Rollout verified: Available: True and Completed: True
• ☐ JDBC datasource confirmed Running with real capacity, not merely configured
• ☐ Every application endpoint tested and returning 200 from inside the cluster
• ☐ Sticky sessions configured and tested with a multi-request session, not assumed from the annotation
• ☐ JMS store and transaction log persistence decided, documented and recovery-tested by killing a pod mid-flight
• ☐ Timezone and locale set explicitly in serverPod env — not inherited from the base image
• ☐ maxUnavailable set explicitly on every Cluster resource
• ☐ Retry, backoff and circuit-breaker behaviour load-tested under a simulated status-check storm
• ☐ CBS and HSM egress agreed, firewall rules updated and connectivity proven from the new source addresses
• ☐ Rollback trigger defined numerically and signed off before the first percentage of traffic moves
• ☐ Change record filed with model diff, image tag and verification evidence attached

Key Takeaways

• The container is never the hard part. State, network assumptions and cutover discipline are. JMS stores, transaction logs and firewall rules keyed to VM IPs will cost you more days than the image build ever will.
• Two Oracle-confirmed constraints are pre-flight, not tuning: sticky sessions at the ingress, and unicast cluster messaging.
• Acquirer first, issuer last, ramp slowly, roll back at the load balancer — with the rollback number written down before go-live.
• Immutability is a feature. Rebuild and roll for every CPU, let the image tag be your audit trail, and rotate secrets without touching the model.
• Reliability is public. NPCI publishes per-bank technical-decline and uptime data monthly. A bad migration does not stay inside the bank.

WebLogic to Kubernetes Migration FAQ

Is Domain in Image deprecated?

Yes — Domain in Image (domainHomeSourceType: Image) was deprecated in WKO 4.0. Deprecated, not removed, so existing domains still run, but it is not where you start new work. The trap is the default. domainHomeSourceType defaults to Image — the deprecated path — unless configuration.model is set, in which case it defaults to FromModel. A Model in Image manifest therefore usually lands correctly by accident, and a Domain resource that loses its configuration.model stanza silently changes source type. Set the field explicitly in every manifest so the reviewer never has to work that out.

Model in Image or Domain on PV — which should I choose?

Oracle’s own framing is deliberately neutral: choose either, depending on your needs. The genuinely defensible rule is the JRF split. For a plain-WLS domain such as a UPI adapter, use Model in Image with auxiliary images — the image tag is the change record, the model diff is the evidence, and a rebuild is reproducible in a way a hand-edited domain home never is. JRF/FMW stacks (SOA, OIG, WebCenter) must go Domain on PV, which is also the choice if you need a mutable domain home where console or WLST changes persist. Note that Model in Image without auxiliary images was deprecated in 4.0.7, and Model in Image for JRF in 4.1.0.

Do WebLogic clusters need sticky sessions on Kubernetes?

Yes. Oracle’s operator documentation is unambiguous that the load balancer in front of a WebLogic cluster must make the HTTP flow sticky, and Oracle’s WebLogic-for-OKE documentation says you must add session affinity annotations to the ingress. On NGINX ingress that means affinity: "cookie", affinity-mode: "persistent" and a named session cookie. Treat it as a pre-flight item, not post-go-live tuning, and test it with a real multi-request session rather than assuming the annotation worked.

Does multicast clustering work on Kubernetes?

No. Oracle’s operator FAQ states that only unicast cluster messaging is supported on Kubernetes — most network fabrics do not carry multicast, and Oracle certified on Flannel and Calico. If your VM-era domain still uses multicast for cluster messaging, converting it to unicast is part of the migration, and “no multicast anywhere in the model” belongs on your readiness checklist.

Can you patch a running WebLogic container?

No — and under Model in Image there is no hot-patch path by design. A quarterly Critical Patch Update means rebuild the image on the patched base, bump the tag, then roll the domain (for example by patching spec.image on the Domain resource). Secret rotation is the exception that proves the rule: because the model only ever holds a @@SECRET:@@ reference, you rotate the Kubernetes secret and bump restartVersion — the model and image are untouched. Immutability is the feature, not the limitation.

Where the Real Project Usually Is

This is not really a containerisation project. It is a state-management project wearing a containerisation costume, followed by a cutover that lives or dies on whether someone wrote a number down beforehand. The tooling is mature and does its job; the failures are all in the parts nobody put on the plan — the file store, the firewall rule, the retry that was never load-tested.

So: if you are staring at a WebLogic estate and a support deadline right now, which part of your domain worries you most — the JMS stores, the T3 channels, or the cutover? That is almost always where the real project turns out to be.