How to solve problems while using OpenShift Local

Sean Boyd Avatar
Sean Boyd

Share with

While OpenShift Local is fantastic to use when it runs, you may encounter many frustrating hours like me. This is especially true when new to OpenShift Local.

This post contains a collection of problems I’ve encountered with possible solutions.

ProblemProblem
Error starting machine: Error in driver during machine start: exit status 1The node was low on resource: ephemeral-storage
‘podman’ is not recognized as an internal or external commandError: unable to connect to Podman socket: failed to connect: ssh: handshake failed: open C:\Users\sean.ssh\known_hosts: The system cannot find the path specified.

Error starting machine: Error in driver during machine start: exit status 1

I used to encounter this error regularly when running OpenShift Local on my laptop, but never on my desktop PC.

Eventually I realised this was caused by lack of memory on my laptop. I have 64GB on my desktop PC, but only 16GB on my laptop.

While the official documentation states you need a minimum of 9GB free memory (for OpenShift Local), in reality I have found a 16GB laptop to be barely sufficient. At times I have been lucky to get OpenShift Local running after a clean reboot, but, once you start running other software such as a browser, development IDEs such as Eclipse or VS Code, and other software, you may be disappointed. To overcome this, I now have a laptop with 32GB of memory.

I don’t really know whether other reasons could cause this error, I only describe one below, plus a process you can follow to debug the error.

crc start

After issuing the crc start command, the error “Error starting machine: Error in driver during machine start: exit status 1” appeared late in the start process.

crc start
Command output. 
C:\Users\sean>crc start
WARN A new version (2.39.0) has been published on https://developers.redhat.com/content-gateway/file/pub/openshift-v4/clients/crc/2.39.0/crc-windows-installer.zip
INFO Using bundle path C:\Users\sean\.crc\cache\crc_hyperv_4.14.12_amd64.crcbundle
INFO Checking minimum RAM requirements
INFO Checking if running in a shell with administrator rights
INFO Checking Windows release
INFO Checking Windows edition
INFO Checking if Hyper-V is installed and operational
INFO Checking if Hyper-V service is enabled
INFO Checking if crc-users group exists
INFO Checking if current user is in crc-users and Hyper-V admins group
INFO Checking if vsock is correctly configured
INFO Checking if the win32 background launcher is installed
INFO Checking if the daemon task is installed
INFO Checking if the daemon task is running
INFO Checking admin helper service is running
INFO Checking SSH port availability
INFO Loading bundle: crc_hyperv_4.14.12_amd64...
INFO Starting CRC VM for openshift 4.14.12...
Error starting machine: Error in driver during machine start: exit status 1

At the end of the output, you’ll see the error.

C:\Users\sean>crc start
:
INFO Starting CRC VM for openshift 4.14.12...
Error starting machine: Error in driver during machine start: exit status 1

To try to resolve the below error, I followed the well-travelled path to reboot my laptop; but to no avail – no surprise there!

crc start in debug mode

The best and perhaps only way to diagnose this problem is to use the debug option in the start command.

crc start --log-level debug
Command output. 
C:\Users\sean>crc start --log-level debug
DEBU CRC version: 2.33.0+c43b17
DEBU OpenShift version: 4.14.12
DEBU Podman version: 4.4.4
DEBU Running 'crc start'
DEBU Total memory of system is 16952647680 bytes
WARN A new version (2.39.0) has been published on https://developers.redhat.com/content-gateway/file/pub/openshift-v4/clients/crc/2.39.0/crc-windows-installer.zip
DEBU Checking file: C:\Users\sean\.crc\machines\crc\.crc-exist
DEBU Running 'Hyper-V\Get-VM crc | Select-Object -ExpandProperty State'
INFO Using bundle path C:\Users\sean\.crc\cache\crc_hyperv_4.14.12_amd64.crcbundle
INFO Checking minimum RAM requirements
DEBU Total memory of system is 16952647680 bytes
INFO Checking if running in a shell with administrator rights
DEBU Running '$currentPrincipal = New-Object Security.Principal.WindowsPrincipal([Security.Principal.WindowsIdentity]::GetCurrent());$currentPrincipal.IsInRole([Security.Principal.WindowsBuiltInRole]::Administrator)'
INFO Checking Windows release
DEBU Running '(Get-ItemProperty -Path "HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion" -Name ReleaseId).ReleaseId'
INFO Checking Windows edition
DEBU Running '(Get-ItemProperty -Path "HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion").EditionID'
DEBU Running on Windows Professional edition
INFO Checking if Hyper-V is installed and operational
DEBU Running '@(Get-Wmiobject Win32_ComputerSystem).HypervisorPresent'
DEBU Running '@(Get-Service vmms).Status'
INFO Checking if Hyper-V service is enabled
DEBU Running '@(Get-Service vmms).Status'
INFO Checking if crc-users group exists
DEBU Running 'Get-LocalGroup -Name crc-users'
INFO Checking if current user is in crc-users and Hyper-V admins group
DEBU Running '(Get-LocalGroupMember -Group 'crc-users').Name'
DEBU Checking current user is in the 'crc-user' group
DEBU group members: MYLAPTOP22\sean
DEBU Running '(Get-LocalGroupMember -SID 'S-1-5-32-578').Name'
DEBU Checking current user is in the 'Hyper-v Administrators' group
DEBU group members: MYLAPTOP22\sean
INFO Checking if vsock is correctly configured
DEBU Running 'Get-Item -Path "HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Virtualization\GuestCommunicationServices\00000400-FACB-11E6-BD58-64006A7986D3"'
INFO Checking if the win32 background launcher is installed
DEBU Running '(Get-Item 'C:\Program Files\Red Hat OpenShift Local\crc-background-launcher.exe').VersionInfo.FileVersion'
DEBU Found crc-background-launcher.exe version 0.0.0.1
INFO Checking if the daemon task is installed
DEBU Running 'Get-ScheduledTask -TaskName crcDaemon'
DEBU Running '(Get-ScheduledTask -TaskName "crcDaemon").Version'
INFO Checking if the daemon task is running
DEBU Running '(Get-ScheduledTask -TaskName "crcDaemon").State'
INFO Checking admin helper service is running
DEBU Running '(Get-Service crcAdminHelper).Status'
INFO Checking SSH port availability
DEBU Checking file: C:\Users\sean\.crc\machines\crc\.crc-exist
DEBU Copying 'C:\Users\sean\.crc\cache\crc_hyperv_4.14.12_amd64\oc.exe' to 'C:\Users\sean\.crc\bin\oc\oc.exe'
DEBU Copying 'C:\Users\sean\.crc\cache\crc_hyperv_4.14.12_amd64\podman.exe' to 'C:\Users\sean\.crc\bin\oc\podman.exe'
INFO Loading bundle: crc_hyperv_4.14.12_amd64...
DEBU Checking file: C:\Users\sean\.crc\machines\crc\.crc-exist
DEBU Running 'Hyper-V\Get-VM crc | Select-Object -ExpandProperty State'
DEBU Copying 'C:\Users\sean\.crc\cache\crc_hyperv_4.14.12_amd64\oc.exe' to 'C:\Users\sean\.crc\bin\oc\oc.exe'
DEBU Copying 'C:\Users\sean\.crc\cache\crc_hyperv_4.14.12_amd64\podman.exe' to 'C:\Users\sean\.crc\bin\oc\podman.exe'
INFO Starting CRC VM for openshift 4.14.12...
DEBU Updating CRC VM configuration
DEBU Running 'Hyper-V\Start-VM crc'
DEBU Command failed: exit status 1
DEBU stdout:
DEBU stderr: Hyper-V\Start-VM : 'crc' failed to start.
Unable to allocate 9216 MB of RAM: Insufficient system resources exist to complete the requested service.
(0x800705AA).
'crc' failed to start. (Virtual machine ID 933DE899-E345-4C58-B6B2-E60802F6B9D6)
'crc' is unable to allocate 9216 MB of RAM: Insufficient system resources exist to complete the requested service.
(0x800705AA). (Virtual machine ID 933DE899-E345-4C58-B6B2-E60802F6B9D6)
At line:1 char:43
+ $ProgressPreference = 'SilentlyContinue'; Hyper-V\Start-VM crc
+                                           ~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Start-VM], VirtualizationException
    + FullyQualifiedErrorId : Unspecified,Microsoft.HyperV.PowerShell.Commands.StartVM

Error starting machine: Error in driver during machine start: exit status 1

Have a look towards the end of the output for the exact error.

It makes no sense to me why the real error is not displayed, rather than the meaningless error that is actually displayed.

It took me much time trying to identify this error before I came across a link showing the debug option. I hope this section saves you the frustrations I faced early on.

INFO Starting CRC VM for openshift 4.14.12...
DEBU Updating CRC VM configuration
DEBU Running 'Hyper-V\Start-VM crc'
DEBU Command failed: exit status 1
DEBU stdout:
DEBU stderr: Hyper-V\Start-VM : 'crc' failed to start.
Unable to allocate 9216 MB of RAM: Insufficient system resources exist to complete the requested service.
(0x800705AA).
'crc' failed to start. (Virtual machine ID 933DE899-E345-4C58-B6B2-E60802F6B9D6)
'crc' is unable to allocate 9216 MB of RAM: Insufficient system resources exist to complete the requested service.
(0x800705AA). (Virtual machine ID 933DE899-E345-4C58-B6B2-E60802F6B9D6)
At line:1 char:43
+ $ProgressPreference = 'SilentlyContinue'; Hyper-V\Start-VM crc
+                                           ~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Start-VM], VirtualizationException
    + FullyQualifiedErrorId : Unspecified,Microsoft.HyperV.PowerShell.Commands.StartVM

Error starting machine: Error in driver during machine start: exit status 1

In this case, the error was “Unable to allocate 9216 MB of RAM: Insufficient system resources exist to complete the requested service.”.

If you have plenty of memory, stop as many programs as possible and re-try. If you, like me, have limited laptop RAM you are out of luck. I only have 16GB RAM on my laptop; even after rebooting I faced the same error. On my desktop PC I upgraded to 64GB RAM so don’t have the problem.

The node was low on resource: ephemeral-storage

Perhaps the strangest error for me was this one. What is ephemeral storage? Being new to OpenShift Local it made no sense. This also took many hours of searching and reading and eventually resulted me in using the sledge-hammer approach of re-installing OpenShift Local from scratch. Many hours later, I discovered the “crc config” command – no need to reinstall, only change the config and restart.

This problem first appeared after I followed a guide to install the strimzi (kafka) operator.

The following error message kept appearing.

The node was low on resource: ephemeral-storage. Threshold quantity: 4902142351, available: 4506488Ki. Container docker-build was using 4Ki, request is 0, has larger consumption of ephemeral-storage.

The node was low on resource: ephemeral-storage. Threshold quantity: 4902142351, available: 4452140Ki.

Even after uninstalling the operator, I got the same error. Deleting old projects, same error.

As mentioned, my first solution was to uninstall OpenShift Local and reinstall it from scratch. The problem disappeared but this is not the proper option.

Many hours later, I found the following command to increase the ephemeral storage for OpenShift Local. The default is 9GB, in my case and for the purpose I am using OpenShift Local, I chose to set to 25GB. Rather excessive, yes, but for me it is fine. You can use a more suitable lower value. You can find more details on the post to install OpenShift Local on my PC.

First, ensure CRC is stopped. Considering the “crc start” command failed, it is a given that it will be stopped.

crc status

The command output should appear as below. Notice that the state is stopped.

c:\ocp\LibertyToOpenShift>crc status
CRC VM:          Stopped
OpenShift:       Stopped (v4.14.12)
RAM Usage:       0B of 0B
Disk Usage:      0B of 0B (Inside the CRC VM)
Cache Usage:     25.21GB
Cache Directory: C:\Users\sean\.crc\cache

Next, set the memory to a value of your choice.

crc config set memory 12288

Now start OpenShift Local and if all goes well, CRC will now start.

crc start

Check for output similar to the below.

INFO Using bundle path C:\Users\sean\.crc\cache\crc_hyperv_4.14.12_amd64.crcbundle
INFO Checking minimum RAM requirements
INFO Checking if running in a shell with administrator rights
INFO Checking Windows release
INFO Checking Windows edition
INFO Checking if Hyper-V is installed and operational
INFO Checking if Hyper-V service is enabled
INFO Checking if crc-users group exists
INFO Checking if current user is in crc-users and Hyper-V admins group
:
:
INFO Adding crc-admin and crc-developer contexts to kubeconfig...
Started the OpenShift cluster.

The server is accessible via web console at:
  https://console-openshift-console.apps-crc.testing

Log in as administrator:
  Username: kubeadmin
  Password: xxxxxxxxxxxxxxxxxxxxxxxxx

Log in as user:
  Username: developer
  Password: developer

Use the 'oc' command line interface:
  > @FOR /f "tokens=*" %i IN ('crc oc-env') DO @call %i
  > oc login -u developer https://api.crc.testing:6443

‘podman’ is not recognized as an internal or external command

The OpenShift Local “crc start” command mentions to use “crc oc-env”. To avoid problems, I used “crc podman-env”. This sets required variables to use podman that comes with Open Shift Local and not any other podman you have installed.

Issue the following commands to set the environment.

crc podman-env
@FOR /f "tokens=*" %i IN ('crc podman-env') DO @call %i

Error: unable to connect to Podman socket: failed to connect: ssh: handshake failed: open C:\Users\sean.ssh\known_hosts: The system cannot find the path specified.

If you receive the below error message.

c:\ocp\LibertyToOpenShiftInstantOn>podman ps
Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
Error: unable to connect to Podman socket: failed to connect: ssh: handshake failed: open C:\Users\sean\.ssh\known_hosts: The system cannot find the path specified.
unable_to_connect_to_podman

Try starting podman.

podman machine start

If that doesn’t work, you may need to initialize the podman environment. Take care if doing this so as not to cause problems.

podman machine init

Tagged in :

,
Sean Boyd Avatar
Sean Boyd

Leave a Reply

Your email address will not be published. Required fields are marked *

More Articles & Posts