Problem pulling images when running private docker registry inside of Kubernetes





.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box;
}







3















I migrated our docker registry that was running on an external dedicated server into our Kubernetes cluster



Now I can still push and pull images to the registry from every external machine but when I try to deploy images from the registry to the Kubernetes cluster itself it is not able to pull it. I get the following error log:



 Warning  Failed                 47s (x3 over 1m)  kubelet, gke-kube-1-default-pool-c5e11d0f-zxm8  Failed to pull image "myregistry.example.com/appimage:1": rpc error: code = Unknown desc = Error response from daemon: Get https://myregistry.example.com/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

 Warning  Failed                 47s (x3 over 1m)  kubelet, gke-kube-1-default-pool-c5e11d0f-zxm8  Error: ErrImagePull


The registry is configured to be accessible via https://myregistry.example.com by a traefik ingress controller and it looks like Kubernetes is internally trying to take a different route?






docker kubernetes







share|improve this question

























  • Did you add imagePullSecret in your manifest file?

    – Shudipta Sharma
    Nov 16 '18 at 11:43











  • Why not just use the ClusterIP or internal K8S DNS to reach the registry ? In your current way, you are exiting the cluster and retuning back in. Kind of sounds like a longer path to take.

    – Jason Stanley
    Nov 17 '18 at 0:59













  • the imagePullSecret is ok, using the ClusterIP would be strange because the configuration would have to be different than on any other server or cluster pulling the same image

    – Thomas Einwaller
    Nov 17 '18 at 21:04











  • Which version of k8s? Did you solve this? I have exactly same issue running k8s 1.11.6 on Azure. Moreover, some times I succeed to create pod with image from my in-cluster registry, but some times not (for the same image). I monitored logs of the pod running my registry (kubectl logs -n docker-registry docker-registry-5c6998f89f-hx96l -f) and found that it doesn't receive any requests when ImagePullBackOff happens. Found this issue: github.com/kubernetes/kubernetes/issues/63874 which recommends upgrading to the latest k8s, will give it a try...

    – Stanislav Poslavsky
    Jan 30 at 22:35











  • Also this seems to be relevant: digitalocean.com/community/questions/…

    – Stanislav Poslavsky
    Jan 30 at 22:40


















3















I migrated our docker registry that was running on an external dedicated server into our Kubernetes cluster



Now I can still push and pull images to the registry from every external machine but when I try to deploy images from the registry to the Kubernetes cluster itself it is not able to pull it. I get the following error log:



 Warning  Failed                 47s (x3 over 1m)  kubelet, gke-kube-1-default-pool-c5e11d0f-zxm8  Failed to pull image "myregistry.example.com/appimage:1": rpc error: code = Unknown desc = Error response from daemon: Get https://myregistry.example.com/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

 Warning  Failed                 47s (x3 over 1m)  kubelet, gke-kube-1-default-pool-c5e11d0f-zxm8  Error: ErrImagePull


The registry is configured to be accessible via https://myregistry.example.com by a traefik ingress controller and it looks like Kubernetes is internally trying to take a different route?






docker kubernetes







share|improve this question

























  • Did you add imagePullSecret in your manifest file?

    – Shudipta Sharma
    Nov 16 '18 at 11:43











  • Why not just use the ClusterIP or internal K8S DNS to reach the registry ? In your current way, you are exiting the cluster and retuning back in. Kind of sounds like a longer path to take.

    – Jason Stanley
    Nov 17 '18 at 0:59













  • the imagePullSecret is ok, using the ClusterIP would be strange because the configuration would have to be different than on any other server or cluster pulling the same image

    – Thomas Einwaller
    Nov 17 '18 at 21:04











  • Which version of k8s? Did you solve this? I have exactly same issue running k8s 1.11.6 on Azure. Moreover, some times I succeed to create pod with image from my in-cluster registry, but some times not (for the same image). I monitored logs of the pod running my registry (kubectl logs -n docker-registry docker-registry-5c6998f89f-hx96l -f) and found that it doesn't receive any requests when ImagePullBackOff happens. Found this issue: github.com/kubernetes/kubernetes/issues/63874 which recommends upgrading to the latest k8s, will give it a try...

    – Stanislav Poslavsky
    Jan 30 at 22:35











  • Also this seems to be relevant: digitalocean.com/community/questions/…

    – Stanislav Poslavsky
    Jan 30 at 22:40














3












3








3


1






I migrated our docker registry that was running on an external dedicated server into our Kubernetes cluster



Now I can still push and pull images to the registry from every external machine but when I try to deploy images from the registry to the Kubernetes cluster itself it is not able to pull it. I get the following error log:



 Warning  Failed                 47s (x3 over 1m)  kubelet, gke-kube-1-default-pool-c5e11d0f-zxm8  Failed to pull image "myregistry.example.com/appimage:1": rpc error: code = Unknown desc = Error response from daemon: Get https://myregistry.example.com/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

 Warning  Failed                 47s (x3 over 1m)  kubelet, gke-kube-1-default-pool-c5e11d0f-zxm8  Error: ErrImagePull


The registry is configured to be accessible via https://myregistry.example.com by a traefik ingress controller and it looks like Kubernetes is internally trying to take a different route?






docker kubernetes







share|improve this question
















I migrated our docker registry that was running on an external dedicated server into our Kubernetes cluster



Now I can still push and pull images to the registry from every external machine but when I try to deploy images from the registry to the Kubernetes cluster itself it is not able to pull it. I get the following error log:



 Warning  Failed                 47s (x3 over 1m)  kubelet, gke-kube-1-default-pool-c5e11d0f-zxm8  Failed to pull image "myregistry.example.com/appimage:1": rpc error: code = Unknown desc = Error response from daemon: Get https://myregistry.example.com/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

 Warning  Failed                 47s (x3 over 1m)  kubelet, gke-kube-1-default-pool-c5e11d0f-zxm8  Error: ErrImagePull


The registry is configured to be accessible via https://myregistry.example.com by a traefik ingress controller and it looks like Kubernetes is internally trying to take a different route?






docker kubernetes




docker kubernetes






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Nov 17 '18 at 0:53









Rico

29.3k95471




29.3k95471










asked Nov 16 '18 at 11:21









Thomas EinwallerThomas Einwaller

4,27443246




4,27443246













  • Did you add imagePullSecret in your manifest file?

    – Shudipta Sharma
    Nov 16 '18 at 11:43











  • Why not just use the ClusterIP or internal K8S DNS to reach the registry ? In your current way, you are exiting the cluster and retuning back in. Kind of sounds like a longer path to take.

    – Jason Stanley
    Nov 17 '18 at 0:59













  • the imagePullSecret is ok, using the ClusterIP would be strange because the configuration would have to be different than on any other server or cluster pulling the same image

    – Thomas Einwaller
    Nov 17 '18 at 21:04











  • Which version of k8s? Did you solve this? I have exactly same issue running k8s 1.11.6 on Azure. Moreover, some times I succeed to create pod with image from my in-cluster registry, but some times not (for the same image). I monitored logs of the pod running my registry (kubectl logs -n docker-registry docker-registry-5c6998f89f-hx96l -f) and found that it doesn't receive any requests when ImagePullBackOff happens. Found this issue: github.com/kubernetes/kubernetes/issues/63874 which recommends upgrading to the latest k8s, will give it a try...

    – Stanislav Poslavsky
    Jan 30 at 22:35











  • Also this seems to be relevant: digitalocean.com/community/questions/…

    – Stanislav Poslavsky
    Jan 30 at 22:40



















  • Did you add imagePullSecret in your manifest file?

    – Shudipta Sharma
    Nov 16 '18 at 11:43











  • Why not just use the ClusterIP or internal K8S DNS to reach the registry ? In your current way, you are exiting the cluster and retuning back in. Kind of sounds like a longer path to take.

    – Jason Stanley
    Nov 17 '18 at 0:59













  • the imagePullSecret is ok, using the ClusterIP would be strange because the configuration would have to be different than on any other server or cluster pulling the same image

    – Thomas Einwaller
    Nov 17 '18 at 21:04











  • Which version of k8s? Did you solve this? I have exactly same issue running k8s 1.11.6 on Azure. Moreover, some times I succeed to create pod with image from my in-cluster registry, but some times not (for the same image). I monitored logs of the pod running my registry (kubectl logs -n docker-registry docker-registry-5c6998f89f-hx96l -f) and found that it doesn't receive any requests when ImagePullBackOff happens. Found this issue: github.com/kubernetes/kubernetes/issues/63874 which recommends upgrading to the latest k8s, will give it a try...

    – Stanislav Poslavsky
    Jan 30 at 22:35











  • Also this seems to be relevant: digitalocean.com/community/questions/…

    – Stanislav Poslavsky
    Jan 30 at 22:40

















Did you add imagePullSecret in your manifest file?

– Shudipta Sharma
Nov 16 '18 at 11:43





Did you add imagePullSecret in your manifest file?

– Shudipta Sharma
Nov 16 '18 at 11:43













Why not just use the ClusterIP or internal K8S DNS to reach the registry ? In your current way, you are exiting the cluster and retuning back in. Kind of sounds like a longer path to take.

– Jason Stanley
Nov 17 '18 at 0:59







Why not just use the ClusterIP or internal K8S DNS to reach the registry ? In your current way, you are exiting the cluster and retuning back in. Kind of sounds like a longer path to take.

– Jason Stanley
Nov 17 '18 at 0:59















the imagePullSecret is ok, using the ClusterIP would be strange because the configuration would have to be different than on any other server or cluster pulling the same image

– Thomas Einwaller
Nov 17 '18 at 21:04





the imagePullSecret is ok, using the ClusterIP would be strange because the configuration would have to be different than on any other server or cluster pulling the same image

– Thomas Einwaller
Nov 17 '18 at 21:04













Which version of k8s? Did you solve this? I have exactly same issue running k8s 1.11.6 on Azure. Moreover, some times I succeed to create pod with image from my in-cluster registry, but some times not (for the same image). I monitored logs of the pod running my registry (kubectl logs -n docker-registry docker-registry-5c6998f89f-hx96l -f) and found that it doesn't receive any requests when ImagePullBackOff happens. Found this issue: github.com/kubernetes/kubernetes/issues/63874 which recommends upgrading to the latest k8s, will give it a try...

– Stanislav Poslavsky
Jan 30 at 22:35





Which version of k8s? Did you solve this? I have exactly same issue running k8s 1.11.6 on Azure. Moreover, some times I succeed to create pod with image from my in-cluster registry, but some times not (for the same image). I monitored logs of the pod running my registry (kubectl logs -n docker-registry docker-registry-5c6998f89f-hx96l -f) and found that it doesn't receive any requests when ImagePullBackOff happens. Found this issue: github.com/kubernetes/kubernetes/issues/63874 which recommends upgrading to the latest k8s, will give it a try...

– Stanislav Poslavsky
Jan 30 at 22:35













Also this seems to be relevant: digitalocean.com/community/questions/…

– Stanislav Poslavsky
Jan 30 at 22:40





Also this seems to be relevant: digitalocean.com/community/questions/…

– Stanislav Poslavsky
Jan 30 at 22:40












1 Answer
1






active

oldest

votes


















1














It's an odd path, but it should work, but taking a wild guess it looks like a DNS issue. (It works for me connecting to an externally facing service). Some things to look at:




  • Can you resolve myregistry.example.com from any other running pod?

  • What does the /etc/resolv.conf look like?

  • What about your K8s nodes /etc/resolv.conf

  • Can you resolve myregistry.example.com from your nodes?






share|improve this answer
























  • It seems like it is not a DNS issue because it gets a connection timeout

    – Thomas Einwaller
    Nov 17 '18 at 21:07











  • Did you confirm?

    – Rico
    Nov 17 '18 at 21:22











  • I can resolve the registry form any running pod, the resolv.conf on the pod has nameserver 10.31.240.10 in it, I can resolv the registry from the kube nodes, though the resolv.conf looks strange: nameserver 169.254.169.254

    – Thomas Einwaller
    Nov 18 '18 at 14:20













  • That looks odd...

    – Rico
    Nov 19 '18 at 5:11












Your Answer






StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});














draft saved

draft discarded


















StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53336855%2fproblem-pulling-images-when-running-private-docker-registry-inside-of-kubernetes%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown

























1 Answer
1






active

oldest

votes








1 Answer
1






active

oldest

votes









active

oldest

votes






active

oldest

votes









1














It's an odd path, but it should work, but taking a wild guess it looks like a DNS issue. (It works for me connecting to an externally facing service). Some things to look at:




  • Can you resolve myregistry.example.com from any other running pod?

  • What does the /etc/resolv.conf look like?

  • What about your K8s nodes /etc/resolv.conf

  • Can you resolve myregistry.example.com from your nodes?






share|improve this answer
























  • It seems like it is not a DNS issue because it gets a connection timeout

    – Thomas Einwaller
    Nov 17 '18 at 21:07











  • Did you confirm?

    – Rico
    Nov 17 '18 at 21:22











  • I can resolve the registry form any running pod, the resolv.conf on the pod has nameserver 10.31.240.10 in it, I can resolv the registry from the kube nodes, though the resolv.conf looks strange: nameserver 169.254.169.254

    – Thomas Einwaller
    Nov 18 '18 at 14:20













  • That looks odd...

    – Rico
    Nov 19 '18 at 5:11
















1














It's an odd path, but it should work, but taking a wild guess it looks like a DNS issue. (It works for me connecting to an externally facing service). Some things to look at:




  • Can you resolve myregistry.example.com from any other running pod?

  • What does the /etc/resolv.conf look like?

  • What about your K8s nodes /etc/resolv.conf

  • Can you resolve myregistry.example.com from your nodes?






share|improve this answer
























  • It seems like it is not a DNS issue because it gets a connection timeout

    – Thomas Einwaller
    Nov 17 '18 at 21:07











  • Did you confirm?

    – Rico
    Nov 17 '18 at 21:22











  • I can resolve the registry form any running pod, the resolv.conf on the pod has nameserver 10.31.240.10 in it, I can resolv the registry from the kube nodes, though the resolv.conf looks strange: nameserver 169.254.169.254

    – Thomas Einwaller
    Nov 18 '18 at 14:20













  • That looks odd...

    – Rico
    Nov 19 '18 at 5:11














1












1








1







It's an odd path, but it should work, but taking a wild guess it looks like a DNS issue. (It works for me connecting to an externally facing service). Some things to look at:




  • Can you resolve myregistry.example.com from any other running pod?

  • What does the /etc/resolv.conf look like?

  • What about your K8s nodes /etc/resolv.conf

  • Can you resolve myregistry.example.com from your nodes?






share|improve this answer













It's an odd path, but it should work, but taking a wild guess it looks like a DNS issue. (It works for me connecting to an externally facing service). Some things to look at:




  • Can you resolve myregistry.example.com from any other running pod?

  • What does the /etc/resolv.conf look like?

  • What about your K8s nodes /etc/resolv.conf

  • Can you resolve myregistry.example.com from your nodes?







share|improve this answer












share|improve this answer



share|improve this answer










answered Nov 17 '18 at 2:02









RicoRico

29.3k95471




29.3k95471













  • It seems like it is not a DNS issue because it gets a connection timeout

    – Thomas Einwaller
    Nov 17 '18 at 21:07











  • Did you confirm?

    – Rico
    Nov 17 '18 at 21:22











  • I can resolve the registry form any running pod, the resolv.conf on the pod has nameserver 10.31.240.10 in it, I can resolv the registry from the kube nodes, though the resolv.conf looks strange: nameserver 169.254.169.254

    – Thomas Einwaller
    Nov 18 '18 at 14:20













  • That looks odd...

    – Rico
    Nov 19 '18 at 5:11



















  • It seems like it is not a DNS issue because it gets a connection timeout

    – Thomas Einwaller
    Nov 17 '18 at 21:07











  • Did you confirm?

    – Rico
    Nov 17 '18 at 21:22











  • I can resolve the registry form any running pod, the resolv.conf on the pod has nameserver 10.31.240.10 in it, I can resolv the registry from the kube nodes, though the resolv.conf looks strange: nameserver 169.254.169.254

    – Thomas Einwaller
    Nov 18 '18 at 14:20













  • That looks odd...

    – Rico
    Nov 19 '18 at 5:11

















It seems like it is not a DNS issue because it gets a connection timeout

– Thomas Einwaller
Nov 17 '18 at 21:07





It seems like it is not a DNS issue because it gets a connection timeout

– Thomas Einwaller
Nov 17 '18 at 21:07













Did you confirm?

– Rico
Nov 17 '18 at 21:22





Did you confirm?

– Rico
Nov 17 '18 at 21:22













I can resolve the registry form any running pod, the resolv.conf on the pod has nameserver 10.31.240.10 in it, I can resolv the registry from the kube nodes, though the resolv.conf looks strange: nameserver 169.254.169.254

– Thomas Einwaller
Nov 18 '18 at 14:20







I can resolve the registry form any running pod, the resolv.conf on the pod has nameserver 10.31.240.10 in it, I can resolv the registry from the kube nodes, though the resolv.conf looks strange: nameserver 169.254.169.254

– Thomas Einwaller
Nov 18 '18 at 14:20















That looks odd...

– Rico
Nov 19 '18 at 5:11





That looks odd...

– Rico
Nov 19 '18 at 5:11




















draft saved

draft discarded




















































Thanks for contributing an answer to Stack Overflow!


  • Please be sure to answer the question. Provide details and share your research!

But avoid …



  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.


To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53336855%2fproblem-pulling-images-when-running-private-docker-registry-inside-of-kubernetes%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







-

Popular posts from this blog

Florida Star v. B. J. F.

Image
Florida Star v. B. J. F. From Wikipedia, the free encyclopedia Jump to navigation Jump to search United States Supreme Court case Florida Star v. B. J. F. Supreme Court of the United States Argued March 21, 1989 Decided June 21, 1989 Full case name The Florida Star v. B. J. F. Citations 491 U.S. 524 ( more ) 109 S. Ct. 2603; 105 L. Ed. 2d 443; 1989 U.S. LEXIS 3120; 57 U.S.L.W. 4816; 16 Media L. Rep. 1801 Prior history The Florida Star v. B.J.F., 530 So.2d 286 (1988) Supreme Court of Florida; Florida Star v. B.J.F., 499 So.2d 883 (1986) Fla. Dist. Court of Appeals Holding Florida Stat. § 794.03 is unconstitutional to the extent it makes the truthful reporting of information that was a matter of public record unlawful, as it violates the First Amendment. Court membership Chief Justice William Rehnquist Associate Justices William J. Brennan Jr.  · Byron White Thurgood Marshall  · Harry Blac
Read more

Danny Elfman

Image
Danny Elfman From Wikipedia, the free encyclopedia Jump to navigation Jump to search Danny Elfman Elfman at the 2010 San Diego Comic-Con Born Daniel Robert Elfman ( 1953-05-29 ) May 29, 1953 (age 65) Los Angeles, California, U.S. Spouse(s) Bridget Fonda ( m.  2003) Children 1 Musical career Genres Rock [1] ska [2] new wave film music video game music Occupation(s) Composer, singer, songwriter, record producer Instruments Trombone guitar percussion vocals keyboards [3] Years active 1972–present Associated acts Oingo Boingo James Newton Howard Daniel Robert Elfman (born May 29, 1953) is an American composer, singer, songwriter, and record producer. Elfman first became known for being the lead singer and songwriter for the band Oingo Boingo from 1974 to 1995. He is well known for scoring films and television shows, particularly his frequent collabora
Read more

Retrieve a Users Dashboard in Tumblr with R and TumblR. Oauth Issues

Image
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box; } 0 I am trying to use the TumblR package in R to set up the Oauth Authentication to Retrieve a user's dashboard using the second example in tumblR documentation However I get the following error, it seems that using twitter others have been able to use a different function to get around this, but I am not finding the same function available for Tumblr. See twitter package for R authentication: error 401 My code consumer_key <- OKey consumer_secret <- SKey appname <- App_name tokenURL <- 'http://www.tumblr.com/oauth/request_token' accessTokenURL <- 'http://www.tumblr.com/oauth/acces_token'
Read more