, , ,

How to solve problems when using Open Liberty InstantOn

Sean Boyd Avatar
Sean Boyd

Share with

During my early attempts to create an Open Liberty InstantOn image I encountered many problems.

Initially this was due to my lack of skills and experience. Later this was due to the highly technical nature of building and using Open Liberty InstantOn apps. It is important to meet all the requirements, some of which are not obvious, to be successful. Get ready for a rewarding challenge.

Hopefully this post will help should you encounter similar issues that I encountered.

ProblemSolution
thread Checkpoint failed, exiting… (00000061) called the method java.lang.System.exit
It is not possible to build an Open Liberty InstantOn app using podman on windows.
The server checkpoint request failed with the following message: Could not dump the JVM processes, err=-70Additional parameters need to be specified when building an Open Liberty InstantOn app.

–cap-add=CHECKPOINT_RESTORE –cap-add=SYS_PTRACE –cap-add=SETPCAP –security-opt seccomp=unconfined
(criu/proc_parse.c:694): Can’t open 1023’s mapfile link 55a65bfe8000: Operation not permitted
:
CWWKE0963E: The server checkpoint request failed because netlink system calls were unsuccessful		To solve this error, you need to issue the following command using the root user before you issue the build command.

setsebool virt_sandbox_use_netlink 1
line 1351: /opt/criu/criu: Operation not permitted
CWWKE0961I: Restoring the checkpoint server process failed		PODMAN
When running an Open Liberty InstantOn app using podman, you need to pass additional parameters. If you forget to add these parameters, you’ll get this error.

-cap-add=CHECKPOINT_RESTORE –cap-add=SETPCAP –security-opt seccomp=unconfined
/opt/ol/wlp/bin/server: line 1373: /opt/criu/criu: Operation not permitted CWWKE0961I: Restoring the checkpoint server process failed.OPENSHIFT
In this example, the serviceAccount and securityContext is in the wrong location in the deployment yaml.
Error (criu/proc_parse.c:379): Failed to resolve mapping 558982528000 filename
Error (criu/proc_parse.c:694): Can’t open 1023’s mapfile link 558982528000: Operation not permitted		I encountered this error when I issued the “setsebool virt_sandbox_use_netlink 1” using root but I still tried to incorrectly build the image using a non-root user.
Error: trying to reuse blob sha256:ecf6a89969f55913ddb3946ec16ae6f081ea6da1bbbdd9405acc637c25409b91 at destination: unable to retrieve auth token: invalid username/password: authentication requiredThis error message may appear when pushing images to OpenShift. You may need to issue the “crc podman-env” settings or to log into the OpenShift registry.
User “developer” cannot get resource “securitycontextconstraints” in API group “security.openshift.io” at the cluster scopeA Security Context Constraint (SCC) cannot be created with a low privileged user. It needs to be created with a privileged user such as a cluster admin account.
Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandboxI sometimes encountered this error when deploying an Open Liberty InstantOn app to Open Shift Local. I thought this was related to an issue with the version of OpenShift. I have found that restarting OpenShift Local solved this issue – not sure why and what is happening.
Error creating: pods “liberty-to-openshift-instanton-85fbc954dd-” is forbidden: unable to validate against any security context constraint: … Invalid value: “CHECKPOINT_RESTORE”: capability may not be added, provider restricted-v2: .containers[0].capabilities.add: Invalid value: “SETPCAP”:The deployment yaml is missing the serviceAccount or securityConstraint

The server checkpoint request failed with the following message: Could not dump the JVM processes, err=-70

To replicate this error, issue the following command. This command is missing the required parameters when using podman to build and Open Liberty InstantOn app. 

podman build -t liberty-to-openshift-instanton:olp-java17-1.0 -f Containerfile.olp.slim.java17
Command output 
D:\Sean>podman build -t liberty-to-openshift-instanton:olp-java17-1.0 -f Containerfile.olp.slim.java17
STEP 1/9: FROM icr.io/appcafe/open-liberty:kernel-slim-java17-openj9-ubi
STEP 2/9: ARG VERSION=1.0
--> Using cache 03f674df8cedde83c6c01d0bd68032033c696e6ec9d3a14fde3e9524d044d982
--> 03f674df8ce
STEP 3/9: ARG REVISION=SNAPSHOT
--> Using cache 395d58791e35877d227085a2aae7f03e654ac79d88dba56ae83f94baefad1863
--> 395d58791e3
STEP 4/9: LABEL   org.opencontainers.image.authors="Sean Boyd"   org.opencontainers.image.url="local"   org.opencontainers.image.version="$VERSION"   org.opencontainers.image.revision="$REVISION"   name="LibertyToOpenShift"   version="$VERSION-$REVISION"   summary="Sample Open Liberty InstantOn app for deploying to OpenShift"   description="This servlet has been written to help test running an Open Liberty app in OpenShift"
--> bfb0a4c14b7
STEP 5/9: COPY --chown=1001:0 target/server.xml /config/
--> 3b8135ed937
STEP 6/9: RUN features.sh
+ SNIPPETS_SOURCE=/opt/ol/helpers/build/configuration_snippets
+ SNIPPETS_TARGET=/config/configDropins/overrides
+ SNIPPETS_TARGET_DEFAULTS=/config/configDropins/defaults
+ mkdir -p /config/configDropins/overrides
+ mkdir -p /config/configDropins/defaults
+ '[' -n '' ']'
+ '[' '' == client ']'
+ '[' '' == embedded ']'
+ [[ -n '' ]]
+ '[' '' == true ']'
+ '[' '' == true ']'
+ featureUtility installServerFeatures --acceptLicense defaultServer --noCache
+ find /opt/ol/wlp/lib /opt/ol/wlp/bin '!' -perm -g=rw -print0
+ xargs -0 -r chmod g+rw
--> 37584c81c4e
STEP 7/9: COPY --chown=1001:0 target/LibertyToOpenShiftInstantOn.war /config/apps/
--> 5e1b75f11e1
STEP 8/9: RUN configure.sh
+ main
+ WLP_INSTALL_DIR=/opt/ol/wlp
+ SHARED_CONFIG_DIR=/opt/ol/wlp/usr/shared/config
+ SHARED_RESOURCE_DIR=/opt/ol/wlp/usr/shared/resources
+ SNIPPETS_SOURCE=/opt/ol/helpers/build/configuration_snippets
+ SNIPPETS_TARGET=/config/configDropins/overrides
+ SNIPPETS_TARGET_DEFAULTS=/config/configDropins/defaults
+ mkdir -p /config/configDropins/overrides
+ mkdir -p /config/configDropins/defaults
+ [[ -n '' ]]
+ '[' '' == client ']'
+ '[' '' == embedded ']'
+ keystorePath=/config/configDropins/defaults/keystore.xml
+ '[' '' '!=' false ']'
+ '[' '' '!=' false ']'
+ '[' '!' -e /config/configDropins/defaults/keystore.xml ']'
++ openssl rand -base64 32
+ export KEYSTOREPWD=BXwJbKXb6VfabbLp94GJday/keFlJ/1QO7G8ltMsFe8=
+ KEYSTOREPWD=BXwJbKXb6VfabbLp94GJday/keFlJ/1QO7G8ltMsFe8=
+ sed 's|REPLACE|BXwJbKXb6VfabbLp94GJday/keFlJ/1QO7G8ltMsFe8=|g' /opt/ol/helpers/build/configuration_snippets/keystore.xml
+ chmod g+w /config/configDropins/defaults/keystore.xml
+ [[ -n '' ]]
+ find /opt/ol/fixes -type f -name '*.jar' -print0
+ sort -z
+ xargs -0 -n 1 -r -I '{}' java -jar '{}' --installLocation /opt/ol/wlp
+ touch /config/server.xml
+ '[' true == true ']'
+ cmd='populate_scc.sh -i 1'
+ '[' '' == false ']'
+ '[' '!' '' = '' ']'
+ '[' '' = false ']'
+ '[' '!' '' = '' ']'
+ '[' '' = false ']'
+ '[' '!' '' = '' ']'
+ eval populate_scc.sh -i 1
++ populate_scc.sh -i 1
+ SCC_SIZE=80m
+ ITERATIONS=2
+ TRIM_SCC=yes
+ WARM_ENDPOINT=true
+ WARM_ENDPOINT_URL=localhost:9080/
+ WARM_OPENAPI_ENDPOINT=true
+ WARM_OPENAPI_ENDPOINT_URL=localhost:9080/openapi
+ [[ -d /opt/java/.scc ]]
++ stat -L -c %a /opt/java/.scc
++ cut -c 1,2
+ [[ 77 == \7\7 ]]
+ SCC=-Xshareclasses:name=openj9_system_scc,cacheDir=/opt/java/.scc
+ export 'OPENJ9_JAVA_OPTIONS=-XX:+OriginalJDK8HeapSizeCompatibilityMode -XX:+IProfileDuringStartupPhase -Xshareclasses:name=openj9_system_scc,cacheDir=/opt/java/.scc'
+ OPENJ9_JAVA_OPTIONS='-XX:+OriginalJDK8HeapSizeCompatibilityMode -XX:+IProfileDuringStartupPhase -Xshareclasses:name=openj9_system_scc,cacheDir=/opt/java/.scc'
+ export 'IBM_JAVA_OPTIONS=-XX:+OriginalJDK8HeapSizeCompatibilityMode -XX:+IProfileDuringStartupPhase -Xshareclasses:name=openj9_system_scc,cacheDir=/opt/java/.scc'
+ IBM_JAVA_OPTIONS='-XX:+OriginalJDK8HeapSizeCompatibilityMode -XX:+IProfileDuringStartupPhase -Xshareclasses:name=openj9_system_scc,cacheDir=/opt/java/.scc'
+ CREATE_LAYER='-XX:+OriginalJDK8HeapSizeCompatibilityMode -XX:+IProfileDuringStartupPhase -Xshareclasses:name=openj9_system_scc,cacheDir=/opt/java/.scc,createLayer,groupAccess'
+ DESTROY_LAYER='-XX:+OriginalJDK8HeapSizeCompatibilityMode -XX:+IProfileDuringStartupPhase -Xshareclasses:name=openj9_system_scc,cacheDir=/opt/java/.scc,destroy'
+ PRINT_LAYER_STATS='-XX:+OriginalJDK8HeapSizeCompatibilityMode -XX:+IProfileDuringStartupPhase -Xshareclasses:name=openj9_system_scc,cacheDir=/opt/java/.scc,printTopLayerStats'
+ getopts :i:s:u:o:tdhwcml OPT
+ case "$OPT" in
+ ITERATIONS=1
+ getopts :i:s:u:o:tdhwcml OPT
++ umask
+ OLD_UMASK=0022
+ umask 002
+ java -XX:+OriginalJDK8HeapSizeCompatibilityMode -XX:+IProfileDuringStartupPhase -Xshareclasses:name=openj9_system_scc,cacheDir=/opt/java/.scc,createLayer,groupAccess -Xscmx80m -version
openjdk version "17.0.11" 2024-04-16
IBM Semeru Runtime Open Edition 17.0.11.0 (build 17.0.11+9)
Eclipse OpenJ9 VM 17.0.11.0 (build openj9-0.44.0, JRE 17 Linux amd64-64-Bit Compressed References 20240416_760 (JIT enabled, AOT enabled)
OpenJ9   - b0699311c7
OMR      - 254af5a04
JCL      - 5d7d758b682 based on jdk-17.0.11+9)
+ '[' yes == yes ']'
+ echo 'Calculating SCC layer upper bound, starting with initial size 80m.'
+ /opt/ol/wlp/bin/server start
+ '[' true == true ']'
+ curl --silent --output /dev/null --show-error --fail --max-time 5 localhost:9080/
+ '[' true == true ']'
+ curl --silent --output /dev/null --show-error --fail --max-time 5 localhost:9080/openapi
+ /opt/ol/wlp/bin/server stop
++ awk '/^Cache is [0-9.]*% .*full/ {print substr($3, 1, length($3)-1)}'
+ FULL=25
+ echo 'SCC layer is 25% full. Destroying layer.'
+ java -XX:+OriginalJDK8HeapSizeCompatibilityMode -XX:+IProfileDuringStartupPhase -Xshareclasses:name=openj9_system_scc,cacheDir=/opt/java/.scc,destroy
JVMSHRC806I Compressed references persistent shared cache "openj9_system_scc" has been destroyed. Use option -Xnocompressedrefs if you want to destroy a non-compressed references cache.
+ true
+ SCC_SIZE=80
++ awk 'BEGIN {print int(80 * 25 / 100.0 + 0.5)}'
+ SCC_SIZE=20
+ '[' 20 -eq 0 ']'
+ SCC_SIZE=20m
+ echo 'Re-creating layer with size 20m.'
+ java -XX:+OriginalJDK8HeapSizeCompatibilityMode -XX:+IProfileDuringStartupPhase -Xshareclasses:name=openj9_system_scc,cacheDir=/opt/java/.scc,createLayer,groupAccess -Xscmx20m -version
openjdk version "17.0.11" 2024-04-16
IBM Semeru Runtime Open Edition 17.0.11.0 (build 17.0.11+9)
Eclipse OpenJ9 VM 17.0.11.0 (build openj9-0.44.0, JRE 17 Linux amd64-64-Bit Compressed References 20240416_760 (JIT enabled, AOT enabled)
OpenJ9   - b0699311c7
OMR      - 254af5a04
JCL      - 5d7d758b682 based on jdk-17.0.11+9)
+ (( i=0 ))
+ (( i<1 ))
+ /opt/ol/wlp/bin/server start
+ '[' true == true ']'
+ curl --silent --output /dev/null --show-error --fail --max-time 5 localhost:9080/
+ '[' true == true ']'
+ curl --silent --output /dev/null --show-error --fail --max-time 5 localhost:9080/openapi
+ /opt/ol/wlp/bin/server stop
+ (( i++ ))
+ (( i<1 ))
+ umask 0022
+ rm -rf /output/messaging /logs/console.log /logs/messages_24.08.12_19.04.11.0.log /logs/messages.log /logs/verbosegc.001.log /logs/verbosegc.002.log /opt/ol/wlp/output/.classCache
+ chmod -R g+rwx /output/workarea
+ [[ -d /output/resources ]]
++ awk '/^Cache is [0-9.]*% .*full/ {print substr($3, 1, length($3)-1)}'
+ FULL=54
+ echo 'SCC layer is 54% full.'
--> 3bb93545eaa
STEP 9/9: RUN checkpoint.sh afterAppStart
Performing checkpoint --at=afterAppStart

Launching defaultServer (Open Liberty 24.0.0.6/wlp-1.0.90.cl240620240603-2001) on Eclipse OpenJ9 VM, version 17.0.11+9 (en_US)
[AUDIT   ] CWWKE0001I: The server defaultServer has been launched.
[AUDIT   ] CWWKG0093A: Processing configuration drop-ins resource: /opt/ol/wlp/usr/servers/defaultServer/configDropins/defaults/keystore.xml
[AUDIT   ] CWWKG0093A: Processing configuration drop-ins resource: /opt/ol/wlp/usr/servers/defaultServer/configDropins/defaults/open-default-port.xml
[AUDIT   ] CWWKZ0058I: Monitoring dropins for applications.
[AUDIT   ] CWWKZ0001I: Application LibertyToOpenShiftInstantOn started in 0.167 seconds.
[AUDIT   ] CWWKC0451I: A server checkpoint "afterAppStart" was requested. When the checkpoint completes, the server stops.
Can't exec criu swrk: Operation not permitted
Can't read request: Connection reset by peer
Can't receive response: Invalid argument
[ERROR   ] CWWKC0453E: The server checkpoint request failed with the following message: Could not dump the JVM processes, err=-70
[AUDIT   ] CWWKZ0001I: Application LibertyToOpenShiftInstantOn started in 0.794 seconds.
[AUDIT   ] CWWKE0084I: The server defaultServer is stopping because thread Checkpoint failed, exiting... (00000061) called the method java.lang.System.exit:
        at java.base/java.lang.System.exit(System.java:519)
        at io.openliberty.checkpoint.internal.CheckpointImpl.lambda$checkpointOrExitOnFailure$5(CheckpointImpl.java:318)
        at java.base/java.lang.Thread.run(Thread.java:857)

[AUDIT   ] CWWKE1100I: Waiting for up to 30 seconds for the server to quiesce.
[AUDIT   ] CWWKF0012I: The server installed the following features: [el-3.0, jsp-2.3, localConnector-1.0, servlet-4.0].
[AUDIT   ] CWWKF0011I: The defaultServer server is ready to run a smarter planet. The defaultServer server started in 0.826 seconds.
Error: error building at STEP "RUN checkpoint.sh afterAppStart": error while running runtime: exit status 74

The error message can be seen towards the bottom of the output.

[AUDIT   ] CWWKC0451I: A server checkpoint "afterAppStart" was requested. When the checkpoint completes, the server stops.
Can't exec criu swrk: Operation not permitted
Can't read request: Connection reset by peer
Can't receive response: Invalid argument
[ERROR   ] CWWKC0453E: The server checkpoint request failed with the following message: Could not dump the JVM processes, err=-70
[AUDIT   ] CWWKZ0001I: Application LibertyToOpenShiftInstantOn started in 0.794 seconds.
[AUDIT   ] CWWKE0084I: The server defaultServer is stopping because thread Checkpoint failed, exiting... (00000061) called the method java.lang.System.exit:
        at java.base/java.lang.System.exit(System.java:519)
        at io.openliberty.checkpoint.internal.CheckpointImpl.lambda$checkpointOrExitOnFailure$5(CheckpointImpl.java:318)
        at java.base/java.lang.Thread.run(Thread.java:857)

To build an Open Liberty InstantOn image the following extra options must be specified. For some reason, I often forgot to add the additional options.

--cap-add=CHECKPOINT_RESTORE --cap-add=SYS_PTRACE --cap-add=SETPCAP --security-opt seccomp=unconfined

The correct command to build an Open Liberty InstantOn image follows.

podman build -t liberty-to-openshift-instanton:olp-java17-1.0 --cap-add=CHECKPOINT_RESTORE --cap-add=SYS_PTRACE --cap-add=SETPCAP --security-opt seccomp=unconfined -f Containerfile.olp.slim.java17
Command output 
[root@localhost LibertyToOpenShiftInstantOn]# podman build -t liberty-to-openshift-instanton:olp-java17-1.0 --cap-add=CHECKPOINT_RESTORE --cap-add=SYS_PTRACE --cap-add=SETPCAP --security-opt seccomp=unconfined -f Containerfile.olp.slim.java17
STEP 1/9: FROM icr.io/appcafe/open-liberty:kernel-slim-java17-openj9-ubi
STEP 2/9: ARG VERSION=1.0
--> Using cache f4c74d7b98f42412a079fd9660c81ceadedf1cfec726ae57dd6c65cf79b2d5d5
--> f4c74d7b98f
STEP 3/9: ARG REVISION=SNAPSHOT
--> Using cache cf717c4202b51903b9cbbe5c7d980b220c84cf3b328b41033fd4b14afd05ceaf
--> cf717c4202b
STEP 4/9: LABEL   org.opencontainers.image.authors="Sean Boyd"   org.opencontainers.image.url="local"   org.opencontainers.image.version="$VERSION"   org.opencontainers.image.revision="$REVISION"   name="LibertyToOpenShift"   version="$VERSION-$REVISION"   summary="Sample Open Liberty InstantOn app for deploying to OpenShift"   description="This servlet has been written to help test running an Open Liberty app in OpenShift"
--> 150747e9c88
STEP 5/9: COPY --chown=1001:0 target/server.xml /config/
--> eeb3078ca8f
STEP 6/9: RUN features.sh
+ SNIPPETS_SOURCE=/opt/ol/helpers/build/configuration_snippets
+ SNIPPETS_TARGET=/config/configDropins/overrides
+ SNIPPETS_TARGET_DEFAULTS=/config/configDropins/defaults
+ mkdir -p /config/configDropins/overrides
+ mkdir -p /config/configDropins/defaults
+ '[' -n '' ']'
+ '[' '' == client ']'
+ '[' '' == embedded ']'
+ [[ -n '' ]]
+ '[' '' == true ']'
+ '[' '' == true ']'
+ featureUtility installServerFeatures --acceptLicense defaultServer --noCache
+ find /opt/ol/wlp/lib /opt/ol/wlp/bin '!' -perm -g=rw -print0
+ xargs -0 -r chmod g+rw
--> ab39c9b6e6f
STEP 7/9: COPY --chown=1001:0 target/LibertyToOpenShiftInstantOn.war /config/apps/
--> 66b86522b37
STEP 8/9: RUN configure.sh
--> 8e0a124ce02
STEP 9/9: RUN checkpoint.sh afterAppStart
Performing checkpoint --at=afterAppStart

Launching defaultServer (Open Liberty 24.0.0.2/wlp-1.0.86.cl240220240212-1928) on Eclipse OpenJ9 VM, version 17.0.10+7 (en_US)
[AUDIT   ] CWWKE0001I: The server defaultServer has been launched.
[AUDIT   ] CWWKG0093A: Processing configuration drop-ins resource: /opt/ol/wlp/usr/servers/defaultServer/configDropins/defaults/keystore.xml
[AUDIT   ] CWWKG0093A: Processing configuration drop-ins resource: /opt/ol/wlp/usr/servers/defaultServer/configDropins/defaults/open-default-port.xml
[AUDIT   ] CWWKZ0058I: Monitoring dropins for applications.
[AUDIT   ] CWWKZ0001I: Application LibertyToOpenShiftInstantOn started in 0.668 seconds.
[AUDIT   ] CWWKC0451I: A server checkpoint "afterAppStart" was requested. When the checkpoint completes, the server stops.
COMMIT liberty-to-openshift-instanton:olp-java17-1.0
--> fdeb66c9055
Successfully tagged localhost/liberty-to-openshift-instanton:olp-java17-1.0
fdeb66c9055741fdf705ef1a89d3d04671cd694ce402fba5eaf87c45e191f286
Notice the additional step towards the bottom of the build.

STEP 9/9: RUN checkpoint.sh afterAppStart
Performing checkpoint --at=afterAppStart

Launching defaultServer (Open Liberty 24.0.0.2/wlp-1.0.86.cl240220240212-1928) on Eclipse OpenJ9 VM, version 17.0.10+7 (en_US)
[AUDIT   ] CWWKE0001I: The server defaultServer has been launched.
[AUDIT   ] CWWKG0093A: Processing configuration drop-ins resource: /opt/ol/wlp/usr/servers/defaultServer/configDropins/defaults/keystore.xml
[AUDIT   ] CWWKG0093A: Processing configuration drop-ins resource: /opt/ol/wlp/usr/servers/defaultServer/configDropins/defaults/open-default-port.xml
[AUDIT   ] CWWKZ0058I: Monitoring dropins for applications.
[AUDIT   ] CWWKZ0001I: Application LibertyToOpenShiftInstantOn started in 0.668 seconds.
[AUDIT   ] CWWKC0451I: A server checkpoint "afterAppStart" was requested. When the checkpoint completes, the server stops.
COMMIT liberty-to-openshift-instanton:olp-java17-1.0
--> fdeb66c9055

Notice the additional step towards the bottom of the build. You can see the checkpoint step successfully completed.

STEP 9/9: RUN checkpoint.sh afterAppStart
Performing checkpoint --at=afterAppStart

Launching defaultServer (Open Liberty 24.0.0.2/wlp-1.0.86.cl240220240212-1928) on Eclipse OpenJ9 VM, version 17.0.10+7 (en_US)
[AUDIT   ] CWWKE0001I: The server defaultServer has been launched.
[AUDIT   ] CWWKG0093A: Processing configuration drop-ins resource: /opt/ol/wlp/usr/servers/defaultServer/configDropins/defaults/keystore.xml
[AUDIT   ] CWWKG0093A: Processing configuration drop-ins resource: /opt/ol/wlp/usr/servers/defaultServer/configDropins/defaults/open-default-port.xml
[AUDIT   ] CWWKZ0058I: Monitoring dropins for applications.
[AUDIT   ] CWWKZ0001I: Application LibertyToOpenShiftInstantOn started in 0.668 seconds.
[AUDIT   ] CWWKC0451I: A server checkpoint "afterAppStart" was requested. When the checkpoint completes, the server stops.
COMMIT liberty-to-openshift-instanton:olp-java17-1.0
--> fdeb66c9055

Build on Linux and not Windows

If you are testing Open Liberty InstantOn using OpenShift Local on your PC, you may encounter the following error.

Not understanding the importance of building Open Liberty InstantOn images on the correct platform, I issued the correct command but on Windows and not UNIX.

podman build -t liberty-to-openshift-instanton:olp-java17-1.0 --cap-add=CHECKPOINT_RESTORE --cap-add=SYS_PTRACE --cap-add=SETPCAP --security-opt seccomp=unconfined -f Containerfile.olp.slim.java17
Command output 
D:\Sean>podman build -t liberty-to-openshift-instanton:olp-java17-1.0 --cap-add=CHECKPOINT_RESTORE --cap-add=SYS_PTRACE --cap-add=SETPCAP --security-opt seccomp=unconfined -f Containerfile.olp.slim.java17
STEP 1/9: FROM icr.io/appcafe/open-liberty:kernel-slim-java17-openj9-ubi
STEP 2/9: ARG VERSION=1.0
--> Using cache 03f674df8cedde83c6c01d0bd68032033c696e6ec9d3a14fde3e9524d044d982
--> 03f674df8ce
STEP 3/9: ARG REVISION=SNAPSHOT
--> Using cache 395d58791e35877d227085a2aae7f03e654ac79d88dba56ae83f94baefad1863
--> 395d58791e3
STEP 4/9: LABEL   org.opencontainers.image.authors="Sean Boyd"   org.opencontainers.image.url="local"   org.opencontainers.image.version="$VERSION"   org.opencontainers.image.revision="$REVISION"   name="LibertyToOpenShift"   version="$VERSION-$REVISION"   summary="Sample Open Liberty InstantOn app for deploying to OpenShift"   description="This servlet has been written to help test running an Open Liberty app in OpenShift"
--> Using cache bfb0a4c14b7863e6bd275834c394a3d258273930f65c60b173e5bc152590d62e
--> bfb0a4c14b7
STEP 5/9: COPY --chown=1001:0 target/server.xml /config/
--> Using cache 3b8135ed937bb83e8617681abe6329b48abfa4ed32ff5ddbc6f67e30f94bc4d2
--> 3b8135ed937
STEP 6/9: RUN features.sh
--> Using cache 37584c81c4edf0b3c0c75c3b969df080399157218ab4e758e3435494e4052d05
--> 37584c81c4e
STEP 7/9: COPY --chown=1001:0 target/LibertyToOpenShiftInstantOn.war /config/apps/
--> Using cache 5e1b75f11e19be5b6dd3a1bdcb8c71d086b25f78343e7a4a49626ed015584458
--> 5e1b75f11e1
STEP 8/9: RUN configure.sh
--> Using cache 3bb93545eaac5cadbfa21f61591e239d7a0640a993a76a45c016f3113407c1d9
--> 3bb93545eaa
STEP 9/9: RUN checkpoint.sh afterAppStart
Performing checkpoint --at=afterAppStart

Launching defaultServer (Open Liberty 24.0.0.6/wlp-1.0.90.cl240620240603-2001) on Eclipse OpenJ9 VM, version 17.0.11+9 (en_US)
[AUDIT   ] CWWKE0001I: The server defaultServer has been launched.
[AUDIT   ] CWWKG0093A: Processing configuration drop-ins resource: /opt/ol/wlp/usr/servers/defaultServer/configDropins/defaults/keystore.xml
[AUDIT   ] CWWKG0093A: Processing configuration drop-ins resource: /opt/ol/wlp/usr/servers/defaultServer/configDropins/defaults/open-default-port.xml
[AUDIT   ] CWWKZ0058I: Monitoring dropins for applications.
[AUDIT   ] CWWKZ0001I: Application LibertyToOpenShiftInstantOn started in 0.190 seconds.
[AUDIT   ] CWWKC0451I: A server checkpoint "afterAppStart" was requested. When the checkpoint completes, the server stops.
[ERROR   ] CWWKC0453E: The server checkpoint request failed with the following message: Could not dump the JVM processes, err=-52
[AUDIT   ] CWWKZ0001I: Application LibertyToOpenShiftInstantOn started in 0.896 seconds.
[AUDIT   ] CWWKE0084I: The server defaultServer is stopping because thread Checkpoint failed, exiting... (00000061) called the method java.lang.System.exit:
        at java.base/java.lang.System.exit(System.java:519)
        at io.openliberty.checkpoint.internal.CheckpointImpl.lambda$checkpointOrExitOnFailure$5(CheckpointImpl.java:318)
        at java.base/java.lang.Thread.run(Thread.java:857)

[AUDIT   ] CWWKE1100I: Waiting for up to 30 seconds for the server to quiesce.
[AUDIT   ] CWWKF0012I: The server installed the following features: [el-3.0, jsp-2.3, localConnector-1.0, servlet-4.0].
[AUDIT   ] CWWKF0011I: The defaultServer server is ready to run a smarter planet. The defaultServer server started in 0.925 seconds.
CWWKE0962E: The server checkpoint request failed. The following output is from the CRIU /logs/checkpoint/checkpoint.log file that contains details on why the checkpoint failed.
Warn  (criu/kerndat.c:1153): $XDG_RUNTIME_DIR not set. Cannot find location for kerndat file
Warn  (criu/kerndat.c:1153): $XDG_RUNTIME_DIR not set. Cannot find location for kerndat file
Warn  (compel/src/lib/infect.c:133): Unable to interrupt task: 1098 (Operation not permitted)
Error (criu/proc_parse.c:379): Failed to resolve mapping 5593b80a5000 filename
Error (criu/proc_parse.c:693): Can't open 1021's mapfile link 5593b80a5000: Operation not permitted
Error (criu/cr-dump.c:1563): Collect mappings (pid: 1021) failed with -1
Error (criu/cr-dump.c:2098): Dumping FAILED.
Error: error building at STEP "RUN checkpoint.sh afterAppStart": error while running runtime: exit status 74

If you check towards the end of the build you’ll see the following error.

[AUDIT   ] CWWKE0084I: The server defaultServer is stopping because thread Checkpoint failed, exiting... (00000061) called the method java.lang.System.exit:
        at java.base/java.lang.System.exit(System.java:519)
        at io.openliberty.checkpoint.internal.CheckpointImpl.lambda$checkpointOrExitOnFailure$5(CheckpointImpl.java:318)
        at java.base/java.lang.Thread.run(Thread.java:857)

[AUDIT   ] CWWKE1100I: Waiting for up to 30 seconds for the server to quiesce.
[AUDIT   ] CWWKF0012I: The server installed the following features: [el-3.0, jsp-2.3, localConnector-1.0, servlet-4.0].
[AUDIT   ] CWWKF0011I: The defaultServer server is ready to run a smarter planet. The defaultServer server started in 0.925 seconds.
CWWKE0962E: The server checkpoint request failed. The following output is from the CRIU /logs/checkpoint/checkpoint.log file that contains details on why the checkpoint failed.
Warn  (criu/kerndat.c:1153): $XDG_RUNTIME_DIR not set. Cannot find location for kerndat file
Warn  (criu/kerndat.c:1153): $XDG_RUNTIME_DIR not set. Cannot find location for kerndat file
Warn  (compel/src/lib/infect.c:133): Unable to interrupt task: 1098 (Operation not permitted)
Error (criu/proc_parse.c:379): Failed to resolve mapping 5593b80a5000 filename
Error (criu/proc_parse.c:693): Can't open 1021's mapfile link 5593b80a5000: Operation not permitted
Error (criu/cr-dump.c:1563): Collect mappings (pid: 1021) failed with -1
Error (criu/cr-dump.c:2098): Dumping FAILED.
Error: error building at STEP "RUN checkpoint.sh afterAppStart": error while running runtime: exit status 74

It is not possible to build the Open Liberty InstantOn image on Windows. It needs access to the criu (Checkpoint/Restore in Userspace) feature found in Linux.

Build the Open Liberty InstantOn image on a Linux server.

The below command was used to build an Open liberty InstantOn image using the root user. It had all the required options.

podman build -t liberty-to-openshift-instanton:olp-java17-1.0 --cap-add=CHECKPOINT_RESTORE --cap-add=SYS_PTRACE --cap-add=SETPCAP --security-opt seccomp=unconfined -f Containerfile.olp.slim.java17

During the build process the following error message was displayed.

Error (criu/proc_parse.c:694): Can't open 1023's mapfile link 55a65bfe8000: Operation not permitted
Error (criu/cr-dump.c:1558): Collect mappings (pid: 1023) failed with -1
Error (criu/cr-dump.c:2093): Dumping FAILED.
CWWKE0963E: The server checkpoint request failed because netlink system calls were unsuccessful. If SELinux is enabled in enforcing mode, netlink system calls might be blocked by the SELinux "virt_sandbox_use_netlink" policy setting. Either disable SELinux or enable the netlink system calls with the "setsebool virt_sandbox_use_netlink 1" command.
Error: building at STEP "RUN checkpoint.sh afterAppStart": while running runtime: exit status 74

To fix this error, issue the following command under the root user id before you build the image.

setsebool virt_sandbox_use_netlink 1

Re-issue the build command.

podman build -t liberty-to-openshift-instanton:olp-java17-1.0 --cap-add=CHECKPOINT_RESTORE --cap-add=SYS_PTRACE --cap-add=SETPCAP --security-opt seccomp=unconfined -f Containerfile.olp.slim.java17
Command output – the build process should complete without any errors. 
[root@localhost LibertyToOpenShiftInstantOn]# podman build -t liberty-to-openshift-instanton:olp-java17-1.0 --cap-add=CHECKPOINT_RESTORE --cap-add=SYS_PTRACE --cap-add=SETPCAP --security-opt seccomp=unconfined -f Containerfile.olp.slim.java17
STEP 1/9: FROM icr.io/appcafe/open-liberty:kernel-slim-java17-openj9-ubi
STEP 2/9: ARG VERSION=1.0
--> Using cache f4c74d7b98f42412a079fd9660c81ceadedf1cfec726ae57dd6c65cf79b2d5d5
--> f4c74d7b98f
STEP 3/9: ARG REVISION=SNAPSHOT
--> Using cache cf717c4202b51903b9cbbe5c7d980b220c84cf3b328b41033fd4b14afd05ceaf
--> cf717c4202b
STEP 4/9: LABEL   org.opencontainers.image.authors="Sean Boyd"   org.opencontainers.image.url="local"   org.opencontainers.image.version="$VERSION"   org.opencontainers.image.revision="$REVISION"   name="LibertyToOpenShift"   version="$VERSION-$REVISION"   summary="Sample Open Liberty InstantOn app for deploying to OpenShift"   description="This servlet has been written to help test running an Open Liberty app in OpenShift"
--> 150747e9c88
STEP 5/9: COPY --chown=1001:0 target/server.xml /config/
--> eeb3078ca8f
STEP 6/9: RUN features.sh
+ SNIPPETS_SOURCE=/opt/ol/helpers/build/configuration_snippets
+ SNIPPETS_TARGET=/config/configDropins/overrides
+ SNIPPETS_TARGET_DEFAULTS=/config/configDropins/defaults
+ mkdir -p /config/configDropins/overrides
+ mkdir -p /config/configDropins/defaults
+ '[' -n '' ']'
+ '[' '' == client ']'
+ '[' '' == embedded ']'
+ [[ -n '' ]]
+ '[' '' == true ']'
+ '[' '' == true ']'
+ featureUtility installServerFeatures --acceptLicense defaultServer --noCache
+ find /opt/ol/wlp/lib /opt/ol/wlp/bin '!' -perm -g=rw -print0
+ xargs -0 -r chmod g+rw
--> ab39c9b6e6f
STEP 7/9: COPY --chown=1001:0 target/LibertyToOpenShiftInstantOn.war /config/apps/
--> 66b86522b37
STEP 8/9: RUN configure.sh
--> 8e0a124ce02
STEP 9/9: RUN checkpoint.sh afterAppStart
Performing checkpoint --at=afterAppStart

Launching defaultServer (Open Liberty 24.0.0.2/wlp-1.0.86.cl240220240212-1928) on Eclipse OpenJ9 VM, version 17.0.10+7 (en_US)
[AUDIT   ] CWWKE0001I: The server defaultServer has been launched.
[AUDIT   ] CWWKG0093A: Processing configuration drop-ins resource: /opt/ol/wlp/usr/servers/defaultServer/configDropins/defaults/keystore.xml
[AUDIT   ] CWWKG0093A: Processing configuration drop-ins resource: /opt/ol/wlp/usr/servers/defaultServer/configDropins/defaults/open-default-port.xml
[AUDIT   ] CWWKZ0058I: Monitoring dropins for applications.
[AUDIT   ] CWWKZ0001I: Application LibertyToOpenShiftInstantOn started in 0.668 seconds.
[AUDIT   ] CWWKC0451I: A server checkpoint "afterAppStart" was requested. When the checkpoint completes, the server stops.
COMMIT liberty-to-openshift-instanton:olp-java17-1.0
--> fdeb66c9055
Successfully tagged localhost/liberty-to-openshift-instanton:olp-java17-1.0
fdeb66c9055741fdf705ef1a89d3d04671cd694ce402fba5eaf87c45e191f286
Notice the additional step towards the bottom of the build.

STEP 9/9: RUN checkpoint.sh afterAppStart
Performing checkpoint --at=afterAppStart

Launching defaultServer (Open Liberty 24.0.0.2/wlp-1.0.86.cl240220240212-1928) on Eclipse OpenJ9 VM, version 17.0.10+7 (en_US)
[AUDIT   ] CWWKE0001I: The server defaultServer has been launched.
[AUDIT   ] CWWKG0093A: Processing configuration drop-ins resource: /opt/ol/wlp/usr/servers/defaultServer/configDropins/defaults/keystore.xml
[AUDIT   ] CWWKG0093A: Processing configuration drop-ins resource: /opt/ol/wlp/usr/servers/defaultServer/configDropins/defaults/open-default-port.xml
[AUDIT   ] CWWKZ0058I: Monitoring dropins for applications.
[AUDIT   ] CWWKZ0001I: Application LibertyToOpenShiftInstantOn started in 0.668 seconds.
[AUDIT   ] CWWKC0451I: A server checkpoint "afterAppStart" was requested. When the checkpoint completes, the server stops.
COMMIT liberty-to-openshift-instanton:olp-java17-1.0
--> fdeb66c9055

(criu/proc_parse.c:379): Failed to resolve mapping 558982528000 filename

This error is probably related to the “setsebool virt_sandbox_use_netlink” fix for the error “CWWKE0963E”.

I managed to encounter this error by issuing the “setsebool virt_sandbox_use_netlink” using the root user but building the image using a non-root user. This was during my early days struggling with building Open Liberty InstantOn images due to my lack of understanding of this process.

The following command was issued using the root user.

setsebool virt_sandbox_use_netlink 1

Followed up with the following command using my user (that is, not the root user).

podman build -t liberty-to-openshift-instanton:olp-java17-1.0 --cap-add=CHECKPOINT_RESTORE --cap-add=SYS_PTRACE --cap-add=SETPCAP --security-opt seccomp=unconfined -f Containerfile.olp.slim.java17

You will see a similar error message to the “CWWKE0963E” error described above, but you’ll see an additional mapping error message pointing to “criu/proc_parse.c:379”.

Error (criu/proc_parse.c:379): Failed to resolve mapping 558982528000 filename
Error (criu/proc_parse.c:694): Can't open 1023's mapfile link 558982528000: Operation not permitted
Error (criu/cr-dump.c:1558): Collect mappings (pid: 1023) failed with -1
Error (criu/cr-dump.c:2093): Dumping FAILED.
Error: building at STEP "RUN checkpoint.sh afterAppStart": while running runtime: exit status 74

To resolve this error, switch to the root user and re-issue both commands

setsebool virt_sandbox_use_netlink 1
podman build -t liberty-to-openshift-instanton:olp-java17-1.0 --cap-add=CHECKPOINT_RESTORE --cap-add=SYS_PTRACE --cap-add=SETPCAP --security-opt seccomp=unconfined -f Containerfile.olp.slim.java17
Command output – the build process should complete without any errors. 
[root@localhost LibertyToOpenShiftInstantOn]# podman build -t liberty-to-openshift-instanton:olp-java17-1.0 --cap-add=CHECKPOINT_RESTORE --cap-add=SYS_PTRACE --cap-add=SETPCAP --security-opt seccomp=unconfined -f Containerfile.olp.slim.java17
STEP 1/9: FROM icr.io/appcafe/open-liberty:kernel-slim-java17-openj9-ubi
STEP 2/9: ARG VERSION=1.0
--> Using cache f4c74d7b98f42412a079fd9660c81ceadedf1cfec726ae57dd6c65cf79b2d5d5
--> f4c74d7b98f
STEP 3/9: ARG REVISION=SNAPSHOT
--> Using cache cf717c4202b51903b9cbbe5c7d980b220c84cf3b328b41033fd4b14afd05ceaf
--> cf717c4202b
STEP 4/9: LABEL   org.opencontainers.image.authors="Sean Boyd"   org.opencontainers.image.url="local"   org.opencontainers.image.version="$VERSION"   org.opencontainers.image.revision="$REVISION"   name="LibertyToOpenShift"   version="$VERSION-$REVISION"   summary="Sample Open Liberty InstantOn app for deploying to OpenShift"   description="This servlet has been written to help test running an Open Liberty app in OpenShift"
--> 150747e9c88
STEP 5/9: COPY --chown=1001:0 target/server.xml /config/
--> eeb3078ca8f
STEP 6/9: RUN features.sh
+ SNIPPETS_SOURCE=/opt/ol/helpers/build/configuration_snippets
+ SNIPPETS_TARGET=/config/configDropins/overrides
+ SNIPPETS_TARGET_DEFAULTS=/config/configDropins/defaults
+ mkdir -p /config/configDropins/overrides
+ mkdir -p /config/configDropins/defaults
+ '[' -n '' ']'
+ '[' '' == client ']'
+ '[' '' == embedded ']'
+ [[ -n '' ]]
+ '[' '' == true ']'
+ '[' '' == true ']'
+ featureUtility installServerFeatures --acceptLicense defaultServer --noCache
+ find /opt/ol/wlp/lib /opt/ol/wlp/bin '!' -perm -g=rw -print0
+ xargs -0 -r chmod g+rw
--> ab39c9b6e6f
STEP 7/9: COPY --chown=1001:0 target/LibertyToOpenShiftInstantOn.war /config/apps/
--> 66b86522b37
STEP 8/9: RUN configure.sh
--> 8e0a124ce02
STEP 9/9: RUN checkpoint.sh afterAppStart
Performing checkpoint --at=afterAppStart

Launching defaultServer (Open Liberty 24.0.0.2/wlp-1.0.86.cl240220240212-1928) on Eclipse OpenJ9 VM, version 17.0.10+7 (en_US)
[AUDIT   ] CWWKE0001I: The server defaultServer has been launched.
[AUDIT   ] CWWKG0093A: Processing configuration drop-ins resource: /opt/ol/wlp/usr/servers/defaultServer/configDropins/defaults/keystore.xml
[AUDIT   ] CWWKG0093A: Processing configuration drop-ins resource: /opt/ol/wlp/usr/servers/defaultServer/configDropins/defaults/open-default-port.xml
[AUDIT   ] CWWKZ0058I: Monitoring dropins for applications.
[AUDIT   ] CWWKZ0001I: Application LibertyToOpenShiftInstantOn started in 0.668 seconds.
[AUDIT   ] CWWKC0451I: A server checkpoint "afterAppStart" was requested. When the checkpoint completes, the server stops.
COMMIT liberty-to-openshift-instanton:olp-java17-1.0
--> fdeb66c9055
Successfully tagged localhost/liberty-to-openshift-instanton:olp-java17-1.0
fdeb66c9055741fdf705ef1a89d3d04671cd694ce402fba5eaf87c45e191f286
Notice the additional step towards the bottom of the build.

STEP 9/9: RUN checkpoint.sh afterAppStart
Performing checkpoint --at=afterAppStart

Launching defaultServer (Open Liberty 24.0.0.2/wlp-1.0.86.cl240220240212-1928) on Eclipse OpenJ9 VM, version 17.0.10+7 (en_US)
[AUDIT   ] CWWKE0001I: The server defaultServer has been launched.
[AUDIT   ] CWWKG0093A: Processing configuration drop-ins resource: /opt/ol/wlp/usr/servers/defaultServer/configDropins/defaults/keystore.xml
[AUDIT   ] CWWKG0093A: Processing configuration drop-ins resource: /opt/ol/wlp/usr/servers/defaultServer/configDropins/defaults/open-default-port.xml
[AUDIT   ] CWWKZ0058I: Monitoring dropins for applications.
[AUDIT   ] CWWKZ0001I: Application LibertyToOpenShiftInstantOn started in 0.668 seconds.
[AUDIT   ] CWWKC0451I: A server checkpoint "afterAppStart" was requested. When the checkpoint completes, the server stops.
COMMIT liberty-to-openshift-instanton:olp-java17-1.0
--> fdeb66c9055

line 1351: /opt/criu/criu: Operation not permitted CWWKE0961I: Restoring the checkpoint server process failed

This section describes the problem when using podman.

This error can also appear when using OpenShift. In this case, further details can be found in the following URL.

When starting an Open Liberty InstantOn container using podman the following options are required.

--cap-add=CHECKPOINT_RESTORE --cap-add=SETPCAP --security-opt seccomp=unconfined

The below command, issued using the root user, didn’t add the required options.

podman run -d --name liberty-to-openshift-instanton -p 9080:9080 liberty-to-openshift-instanton:olp-java17-1.0

The pod will start fine. When you check the pod logs you will see the following error.

[root@localhost LibertyToOpenShiftInstantOn]# podman logs liberty-to-openshift-instanton

/opt/ol/wlp/bin/server: line 1351: /opt/criu/criu: Operation not permitted
CWWKE0961I: Restoring the checkpoint server process failed. Check the /logs/checkpoint/restore.log log to determine why the checkpoint process was not restored. Launching the server without using the checkpoint image.
Launching defaultServer (Open Liberty 24.0.0.2/wlp-1.0.86.cl240220240212-1928) on Eclipse OpenJ9 VM, version 17.0.10+7 (en_US)
[AUDIT   ] CWWKE0001I: The server defaultServer has been launched.
[AUDIT   ] CWWKG0093A: Processing configuration drop-ins resource: /opt/ol/wlp/usr/servers/defaultServer/configDropins/defaults/keystore.xml
[AUDIT   ] CWWKG0093A: Processing configuration drop-ins resource: /opt/ol/wlp/usr/servers/defaultServer/configDropins/defaults/open-default-port.xml
[AUDIT   ] CWWKZ0058I: Monitoring dropins for applications.
[AUDIT   ] CWWKT0016I: Web application available (default_host): http://0f8d146c7403:9080/
[AUDIT   ] CWWKZ0001I: Application LibertyToOpenShiftInstantOn started in 0.754 seconds.
[AUDIT   ] CWWKF0012I: The server installed the following features: [el-3.0, jsp-2.3, localConnector-1.0, servlet-4.0].
[AUDIT   ] CWWKF0011I: The defaultServer server is ready to run a smarter planet. The defaultServer server started in 4.501 seconds.

To resolve this error, add the missing options, and re-try.

podman run -d --name liberty-to-openshift-instanton --cap-add=CHECKPOINT_RESTORE --cap-add=SETPCAP --security-opt seccomp=unconfined  -p 9080:9080 liberty-to-openshift-instanton:olp-java17-1.0

If all goes well, you’ll see a successful start using a checkpoint restore. Look for message ‘resumed operation from a checkpoint’.

[root@localhost LibertyToOpenShiftInstantOn]# podman logs liberty-to-openshift-instanton

[AUDIT   ] Launching defaultServer (Open Liberty 24.0.0.2/wlp-1.0.86.cl240220240212-1928) on Eclipse OpenJ9 VM, version 17.0.10+7 (en_US)
[AUDIT   ] CWWKT0016I: Web application available (default_host): http://85a664df916b:9080/
[AUDIT   ] CWWKC0452I: The Liberty server process resumed operation from a checkpoint in 0.436 seconds.
[AUDIT   ] CWWKZ0001I: Application LibertyToOpenShiftInstantOn started in 0.440 seconds.
[AUDIT   ] CWWKF0012I: The server installed the following features: [el-3.0, jsp-2.3, localConnector-1.0, servlet-4.0].
[AUDIT   ] CWWKF0011I: The defaultServer server is ready to run a smarter planet. The defaultServer server started in 0.654 seconds.

Error: trying to reuse blob sha256:ecf6a89969f55913ddb3946ec16ae6f081ea6da1bbbdd9405acc637c25409b91 at destination: unable to retrieve auth token: invalid username/password: authentication required

You may experience this error when pushing an image to OpenShift Local.

I generally encountered this problem due to opening a new command window without re-setting the environment variables.

I have found two scenarios that caused the issue.

May need to reset the podman-env

Issue the following command and re-try,

crc podman-env
@FOR /f “tokens=*” %i IN (‘crc podman-env’) DO @call %i

crc podman-env
@FOR /f “tokens=*” %i IN (‘crc podman-env’) DO @call %i

Log into the OpenShift registry

This problem can appear if you didn’t log into the OpenShift registry or when the login timed out.

Issue the following command to log into the OpenShift registry.

oc registry login –insecure=true

User “developer” cannot get resource “securitycontextconstraints” in API group “security.openshift.io” at the cluster scope

When issuing a command to create a Security Context Constraint using a low-privileged user you will receive an error. The below yaml file contains the details to create an SCC.

oc apply -f scc-cap-cr-minmal.yaml

The following error will appear.

c:\ocp\LibertyToOpenShiftInstantOn>oc apply -f scc-cap-cr-minmal.yaml
Error from server (Forbidden): error when retrieving current configuration of:
Resource: "security.openshift.io/v1, Resource=securitycontextconstraints", GroupVersionKind: "security.openshift.io/v1, Kind=SecurityContextConstraints"
Name: "liberty-to-openshift-instanton-scc", Namespace: ""
from server for: "scc-cap-cr-minmal.yaml": securitycontextconstraints.security.openshift.io "liberty-to-openshift-instanton-scc" is forbidden: User "developer" cannot get resource "securitycontextconstraints" in API group "security.openshift.io" at the cluster scope

To solve the issue, switch to the cluster admin account or the “kubeadmin” account when using OpenShift Local.

Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox

I received this error message when deploying an Open Liberty InstantOn app to OpenShift Local.

I initially thought this was related to the version of CRC I was using:

CRC version: 2.41.0+e6495f
OpenShift version: 4.16.7

When I used an older version, I didn’t see the problem.

CRC version: 2.33.0+c43b17
OpenShift version: 4.14.12

That was until the problem happened with this version also. Hmmm.

The error can be seen when viewing the OpenShift Local events.

oc get events
Command output. 
24s         Warning   FailedCreatePodSandBox   pod/liberty-to-openshift-instanton-85fbc954dd-tx7jp    Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_liberty-to-openshift-instanton-85fbc954dd-tx7jp_liberty-to-openshift-instanton_ee56edbf-3e8a-48d0-a0f6-8e7c8afc5731_0(3fc5e47b2d1ae51b8ba64231c3b1d8e389846102fbf7e769ff7ca7eb0df4ff4d): error adding pod liberty-to-openshift-instanton_liberty-to-openshift-instanton-85fbc954dd-tx7jp to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: '&{ContainerID:3fc5e47b2d1ae51b8ba64231c3b1d8e389846102fbf7e769ff7ca7eb0df4ff4d Netns:/var/run/netns/3293dadf-1206-4dd9-9597-e214509afe2c IfName:eth0 Args:IgnoreUnknown=1;K8S_POD_NAMESPACE=liberty-to-openshift-instanton;K8S_POD_NAME=liberty-to-openshift-instanton-85fbc954dd-tx7jp;K8S_POD_INFRA_CONTAINER_ID=3fc5e47b2d1ae51b8ba64231c3b1d8e389846102fbf7e769ff7ca7eb0df4ff4d;K8S_POD_UID=ee56edbf-3e8a-48d0-a0f6-8e7c8afc5731 Path: StdinData:[123 34 98 105 110 68 105 114 34 58 34 47 118 97 114 47 108 105 98 47 99 110 105 47 98 105 110 34 44 34 99 104 114 111 111 116 68 105 114 34 58 34 47 104 111 115 116 114 111 111 116 34 44 34 99 108 117 115 116 101 114 78 101 116 119 111 114 107 34 58 34 47 104 111 115 116 47 114 117 110 47 109 117 108 116 117 115 47 99 110 105 47 110 101 116 46 100 47 56 48 45 111 112 101 110 115 104 105 102 116 45 110 101 116 119 111 114 107 46 99 111 110 102 34 44 34 99 110 105 67 111 110 102 105 103 68 105 114 34 58 34 47 104 111 115 116 47 101 116 99 47 99 110 105 47 110 101 116 46 100 34 44 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 49 34 44 34 100 97 101 109 111 110 83 111 99 107 101 116 68 105 114 34 58 34 47 114 117 110 47 109 117 108 116 117 115 47 115 111 99 107 101 116 34 44 34 103 108 111 98 97 108 78 97 109 101 115 112 97 99 101 115 34 58 34 100 101 102 97 117 108 116 44 111 112 101 110 115 104 105 102 116 45 109 117 108 116 117 115 44 111 112 101 110 115 104 105 102 116 45 115 114 105 111 118 45 110 101 116 119 111 114 107 45 111 112 101 114 97 116 111 114 34 44 34 108 111 103 76 101 118 101 108 34 58 34 118 101 114 98 111 115 101 34 44 34 108 111 103 84 111 83 116 100 101 114 114 34 58 116 114 117 101 44 34 109 117 108 116 117 115 65 117 116 111 99 111 110 102 105 103 68 105 114 34 58 34 47 104 111 115 116 47 114 117 110 47 109 117 108 116 117 115 47 99 110 105 47 110 101 116 46 100 34 44 34 109 117 108 116 117 115 67 111 110 102 105 103 70 105 108 101 34 58 34 97 117 116 111 34 44 34 110 97 109 101 34 58 34 109 117 108 116 117 115 45 99 110 105 45 110 101 116 119 111 114 107 34 44 34 110 97 109 101 115 112 97 99 101 73 115 111 108 97 116 105 111 110 34 58 116 114 117 101 44 34 112 101 114 78 111 100 101 67 101 114 116 105 102 105 99 97 116 101 34 58 123 34 98 111 111 116 115 116 114 97 112 75 117 98 101 99 111 110 102 105 103 34 58 34 47 118 97 114 47 108 105 98 47 107 117 98 101 108 101 116 47 107 117 98 101 99 111 110 102 105 103 34 44 34 99 101 114 116 68 105 114 34 58 34 47 101 116 99 47 99 110 105 47 109 117 108 116 117 115 47 99 101 114 116 115 34 44 34 99 101 114 116 68 117 114 97 116 105 111 110 34 58 34 50 52 104 34 44 34 101 110 97 98 108 101 100 34 58 116 114 117 101 125 44 34 115 111 99 107 101 116 68 105 114 34 58 34 47 104 111 115 116 47 114 117 110 47 109 117 108 116 117 115 47 115 111 99 107 101 116 34 44 34 116 121 112 101 34 58 34 109 117 108 116 117 115 45 115 104 105 109 34 125]} ContainerID:"3fc5e47b2d1ae51b8ba64231c3b1d8e389846102fbf7e769ff7ca7eb0df4ff4d" Netns:"/var/run/netns/3293dadf-1206-4dd9-9597-e214509afe2c" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=liberty-to-openshift-instanton;K8S_POD_NAME=liberty-to-openshift-instanton-85fbc954dd-tx7jp;K8S_POD_INFRA_CONTAINER_ID=3fc5e47b2d1ae51b8ba64231c3b1d8e389846102fbf7e769ff7ca7eb0df4ff4d;K8S_POD_UID=ee56edbf-3e8a-48d0-a0f6-8e7c8afc5731" Path:"" ERRORED: error configuring pod [liberty-to-openshift-instanton/liberty-to-openshift-instanton-85fbc954dd-tx7jp] networking: Multus: [liberty-to-openshift-instanton/liberty-to-openshift-instanton-85fbc954dd-tx7jp/ee56edbf-3e8a-48d0-a0f6-8e7c8afc5731]: error waiting for pod: Unauthorized...

The error can be seen at the top of the event details.

FailedCreatePodSandBox   pod/liberty-to-openshift-instanton-85fbc954dd-tx7jp    Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_liberty-to-openshift-instanton-85fbc954dd-tx7jp_liberty-to-openshift-instanton_ee56edbf-3e8a-48d0-a0f6-8e7c8afc5731_0(3fc5e47b2d1ae51b8ba64231c3b1d8e389846102fbf7e769ff7ca7eb0df4ff4d): error adding pod

To solve this issue, I restarted OpenShift Local. I’m not sure whether the problem will re-appear, but in this instance a restart solved the issue.

crc stop
crc start

Tagged in :

, , , ,
Sean Boyd Avatar
Sean Boyd

One response to “How to solve problems when using Open Liberty InstantOn”

  1. Vijay Sundaresan Avatar
    Vijay Sundaresan

    Very interesting and practically useful article, Sean. I guess some of these problems are actually InstantOn related and others are part of the usage environment (OCP Local etc.). I think the most useful aspects are errors that we did not anticipate (and therefore did not document anywhere), e.g. setting the SE linux option as root and building the container image as non-root. When we discuss how to emit better error messages in the not distant future, we will definitely consider these issues you brought up.

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

More Articles & Posts