Issue

Failed to perform any activity/task on Container (eg. Stop, Start, Inspect, Exec,…) as Hyper-v Host Compute Service service terminates unexpectedly

Symptoms:

• Failed to perform any activity/task on Container (eg. Start, Stop, Inspect, Exec,….). Container start and stop commands waits on blinking cursor, no output or response

• In the System Logs observed error: ‘The Hyper-V Host Compute Service service terminated unexpectedly.  It has done this 1196 time(s)’

• Failed to connect to running container with error: ‘failure in a Windows system call: The remote procedure call failed and did not execute. (0x6bf)‘

• Container runs after server is rebooted, however becomes unresponsive after 20-30 minutes

Summary:

Tried to connect to running container, failed with following error:

PS> docker exec -it 93a09d90f3b0 powershell

container 93a60053f3b0d56c7c97b2df0f94f863ab86eb60c2e571385feff864b7357ceb encountered an error during CreateProcess: failure in a Windows system call: The remote procedure call failed and did not execute. (0x6bf)

• Checked the Application logs and found following error :

Log Name: Application

Source: docker

Event ID: 11

Task Category: None

Level: Error

Keywords: Classic

User: N/A

Description:

failed to shutdown hccshim container [pid=10880 signal=15 namespace=moby error=container 93a60053f3b0d56c7c97b2df0f94f863ab86eb60c2e571385feff864b7357ceb encountered an error during Shutdown: failure in a Windows system call: The remote procedure call failed and did not execute. (0x6bf) module=libcontainerd container=93a60053f3b0d56c7c97b2df0f94f863ab86eb60c2e571385feff864b7357ceb process=init]

Log Name: Application

Source: docker

Event ID: 1

Task Category: None

Level: Error

Keywords: Classic

User: N/A

Description:

Error running exec b19fab1fa412fcc4395d3e2014e71aab21d38757b67b74697474b1e8992c7dcf in container: container 93a60053f3b0d56c7c97b2df0f94f863ab86eb60c2e571385feff864b7357ceb encountered an error during CreateProcess: failure in a Windows system call: The remote procedure call failed and did not execute. (0x6bf)

• Checked the System logs and found following error :

Information 7036 Service Control Manager  The Docker Engine service entered the stopped state.

Error  7034 Service Control Manager  The Hyper-V Host Compute Service service terminated unexpectedly.  It has done this 1196 time(s).

• To recover the Containers following action plan:

1. From Task Manager -> Selected “Dockerd.exe” -> End Task
2. Rebooted the Server 

• Once the Container Host started, Container started working 

• We were able to connect to the Containers and access .Net web-app 

• However, after 20-30 mins, again all access to the container is lost 

• In Application logs same error : Event ID: 11 and Event ID: 1

Cause :

In VMware Tools 11.0.0, a new plugin called ‘appInfo’ was bundled. This new plugin runs every 30 minutes (can be configured by the guest administrator) to capture the information about running processes. For this, the plugin creates a remote thread in the target 64-bit process to capture the ‘command line’ of the target process. If the 64-bit application of the process is not built with ‘/LARGEADDRESSAWARE:YES, ‘the remote thread’ crashes the application.

Solution

Upgrade VMware tool to 11.0.6 or above 

Workaround

Disable AppInfo plugin inside guest virtual machine

VMwareToolboxCmd.exe config set appinfo disabled true 

OR

VMwareToolboxCmd.exe config set appinfo poll-interval 0 

Reference

https://forums.docker.com/t/the-running-container-failed-after-a-few-minutes-solved/91195

https://kb.vmware.com/s/article/78550