Hi, We are observing since 3 days instabilities while cherry-picking patches from coreboot.org: git fetch https://review.coreboot.org/coreboot refs/changes/40/52140/3 && git cherry-pick FETCH_HEAD fatal: The remote end hung up unexpectedly fatal: protocol error: bad pack header This is happening on several machines, our automation is also impacted. Any change/issue that could explain such behavior? Thanks, Selma.
On Wed, May 19, 2021 at 6:25 PM Bensaid, Selma selma.bensaid@intel.com wrote:
Hi,
We are observing since 3 days instabilities while cherry-picking patches from coreboot.org:
git fetch https://review.coreboot.org/coreboot refs/changes/40/52140/3 && git cherry-pick FETCH_HEAD
fatal: The remote end hung up unexpectedly
fatal: protocol error: bad pack header
This is happening on several machines, our automation is also impacted. Any change/issue that could explain such behavior?
I think it's on your end, Selma. Perhaps the git repo got corrupted? I just tried your instructions and the fetch was successful.
The root cause is likely the first error: remote end hung up unexpectedly. The following might be helpful. Also if there's a way to get more verbose logging out that might help point in the right direction.
https://stackoverflow.com/questions/6842687/the-remote-end-hung-up-unexpecte...
Thanks,
Selma. _______________________________________________ coreboot mailing list -- coreboot@coreboot.org To unsubscribe send an email to coreboot-leave@coreboot.org
Aaron Durbin via coreboot wrote:
We are observing since 3 days instabilities while cherry-picking patches from coreboot.org:
git fetch https://review.coreboot.org/coreboot refs/changes/40/52140/3 && git cherry-pick FETCH_HEAD
fatal: The remote end hung up unexpectedly
fatal: protocol error: bad pack header
This is happening on several machines, our automation is also impacted. Any change/issue that could explain such behavior?
I think it's on your end, Selma. Perhaps the git repo got corrupted?
..
The root cause is likely the first error: remote end hung up unexpectedly. The following might be helpful.
https://stackoverflow.com/questions/6842687/the-remote-end-hung-up-unexpecte...
My guess is either a change in an outgoing proxy at Intel or that all your repos changed in the same way at the same time to now interact badly with how outgoing proxying has always worked. (HTTP request larger than http.postBuffer git config setting leading to HTTP/1.1 chunked transfer.)
Also if there's a way to get more verbose logging out that might help point in the right direction.
Do try (from that stackoverflow):
# Linux export GIT_TRACE_PACKET=1 export GIT_TRACE=1 export GIT_CURL_VERBOSE=1
#Windows set GIT_TRACE_PACKET=1 set GIT_TRACE=1 set GIT_CURL_VERBOSE=1
Before the git command, and see what comes out.
//Peter
Thanks Peter and Aaaron! For more context we observe this failure on our local coreboot hosted in GitHub (Intel Org). Cherry-picking from coreboot.org worked well until few days ago, where the failure showed up. Looking into the logs (that I cannot attach) and if I understand well while fetching from a remote git looks for a common ancestor between the local repo and the remote. In our case this common ancestor is quite old (https://review.coreboot.org/c/coreboot/+/34368). It seems that the remote (coreboot.org) has a timeout mechanism that triggers before we find the common ancestor. I think I figured out a dirty WA ☹, any idea will be more than welcome! Thanks, Selma.
-----Original Message----- From: Peter Stuge peter@stuge.se Sent: Thursday, May 20, 2021 10:57 AM To: coreboot@coreboot.org Subject: [coreboot] Re: failure cherry-picking patches from https://review.coreboot.org/coreboot
Aaron Durbin via coreboot wrote:
We are observing since 3 days instabilities while cherry-picking patches from coreboot.org:
git fetch https://review.coreboot.org/coreboot refs/changes/40/52140/3 && git cherry-pick FETCH_HEAD
fatal: The remote end hung up unexpectedly
fatal: protocol error: bad pack header
This is happening on several machines, our automation is also impacted. Any change/issue that could explain such behavior?
I think it's on your end, Selma. Perhaps the git repo got corrupted?
..
The root cause is likely the first error: remote end hung up unexpectedly. The following might be helpful.
https://stackoverflow.com/questions/6842687/the-remote-end-hung-up-une xpectedly-while-git-cloning
My guess is either a change in an outgoing proxy at Intel or that all your repos changed in the same way at the same time to now interact badly with how outgoing proxying has always worked. (HTTP request larger than http.postBuffer git config setting leading to HTTP/1.1 chunked transfer.)
Also if there's a way to get more verbose logging out that might help point in the right direction.
Do try (from that stackoverflow):
# Linux export GIT_TRACE_PACKET=1 export GIT_TRACE=1 export GIT_CURL_VERBOSE=1
#Windows set GIT_TRACE_PACKET=1 set GIT_TRACE=1 set GIT_CURL_VERBOSE=1
Before the git command, and see what comes out.
//Peter _______________________________________________ coreboot mailing list -- coreboot@coreboot.org To unsubscribe send an email to coreboot-leave@coreboot.org
On Thu, May 20, 2021 at 11:05 PM Bensaid, Selma selma.bensaid@intel.com wrote:
Thanks Peter and Aaaron! For more context we observe this failure on our local coreboot hosted in GitHub (Intel Org). Cherry-picking from coreboot.org worked well until few days ago, where the failure showed up. Looking into the logs (that I cannot attach) and if I understand well while fetching from a remote git looks for a common ancestor between the local repo and the remote. In our case this common ancestor is quite old ( https://review.coreboot.org/c/coreboot/+/34368). It seems that the remote (coreboot.org) has a timeout mechanism that triggers before we find the common ancestor. I think I figured out a dirty WA ☹, any idea will be more than welcome!
This is not original thought, but since the ancestor is so old maybe it is the solution in the article: raise limits to get all the SHAs of the trees and objects. If it's too big then that might cause the hangup. I think it'd be good for someone on your end to do a proper rebase. Once that's complete I expect the fetchting to start working again.
Thanks, Selma.
-----Original Message----- From: Peter Stuge peter@stuge.se Sent: Thursday, May 20, 2021 10:57 AM To: coreboot@coreboot.org Subject: [coreboot] Re: failure cherry-picking patches from https://review.coreboot.org/coreboot
Aaron Durbin via coreboot wrote:
We are observing since 3 days instabilities while cherry-picking patches from coreboot.org:
git fetch https://review.coreboot.org/coreboot refs/changes/40/52140/3 && git cherry-pick FETCH_HEAD
fatal: The remote end hung up unexpectedly
fatal: protocol error: bad pack header
This is happening on several machines, our automation is also impacted. Any change/issue that could explain such behavior?
I think it's on your end, Selma. Perhaps the git repo got corrupted?
..
The root cause is likely the first error: remote end hung up
unexpectedly.
The following might be helpful.
https://stackoverflow.com/questions/6842687/the-remote-end-hung-up-une xpectedly-while-git-cloning
My guess is either a change in an outgoing proxy at Intel or that all your repos changed in the same way at the same time to now interact badly with how outgoing proxying has always worked. (HTTP request larger than http.postBuffer git config setting leading to HTTP/1.1 chunked transfer.)
Also if there's a way to get more verbose logging out that might help point in the right direction.
Do try (from that stackoverflow):
# Linux
export GIT_TRACE_PACKET=1
export GIT_TRACE=1
export GIT_CURL_VERBOSE=1
#Windows
set GIT_TRACE_PACKET=1
set GIT_TRACE=1
set GIT_CURL_VERBOSE=1
Before the git command, and see what comes out.
//Peter _______________________________________________ coreboot mailing list -- coreboot@coreboot.org To unsubscribe send an email to coreboot-leave@coreboot.org _______________________________________________ coreboot mailing list -- coreboot@coreboot.org To unsubscribe send an email to coreboot-leave@coreboot.org
Thanks Aaron, yes this might be the only clean solution (I wanted to avoid it). Any way thanks again for your time! Selma.
From: Aaron Durbin adurbin@google.com Sent: Thursday, May 20, 2021 11:04 PM To: Bensaid, Selma selma.bensaid@intel.com Cc: Peter Stuge peter@stuge.se; coreboot@coreboot.org Subject: Re: [coreboot] Re: failure cherry-picking patches from https://review.coreboot.org/coreboot
On Thu, May 20, 2021 at 11:05 PM Bensaid, Selma <selma.bensaid@intel.commailto:selma.bensaid@intel.com> wrote: Thanks Peter and Aaaron! For more context we observe this failure on our local coreboot hosted in GitHub (Intel Org). Cherry-picking from coreboot.orghttp://coreboot.org worked well until few days ago, where the failure showed up. Looking into the logs (that I cannot attach) and if I understand well while fetching from a remote git looks for a common ancestor between the local repo and the remote. In our case this common ancestor is quite old (https://review.coreboot.org/c/coreboot/+/34368). It seems that the remote (coreboot.orghttp://coreboot.org) has a timeout mechanism that triggers before we find the common ancestor. I think I figured out a dirty WA ☹, any idea will be more than welcome!
This is not original thought, but since the ancestor is so old maybe it is the solution in the article: raise limits to get all the SHAs of the trees and objects. If it's too big then that might cause the hangup. I think it'd be good for someone on your end to do a proper rebase. Once that's complete I expect the fetchting to start working again. Thanks, Selma.
-----Original Message----- From: Peter Stuge <peter@stuge.semailto:peter@stuge.se> Sent: Thursday, May 20, 2021 10:57 AM To: coreboot@coreboot.orgmailto:coreboot@coreboot.org Subject: [coreboot] Re: failure cherry-picking patches from https://review.coreboot.org/coreboot
Aaron Durbin via coreboot wrote:
We are observing since 3 days instabilities while cherry-picking patches from coreboot.orghttp://coreboot.org:
git fetch https://review.coreboot.org/coreboot refs/changes/40/52140/3 && git cherry-pick FETCH_HEAD
fatal: The remote end hung up unexpectedly
fatal: protocol error: bad pack header
This is happening on several machines, our automation is also impacted. Any change/issue that could explain such behavior?
I think it's on your end, Selma. Perhaps the git repo got corrupted?
..
The root cause is likely the first error: remote end hung up unexpectedly. The following might be helpful.
https://stackoverflow.com/questions/6842687/the-remote-end-hung-up-une xpectedly-while-git-cloning
My guess is either a change in an outgoing proxy at Intel or that all your repos changed in the same way at the same time to now interact badly with how outgoing proxying has always worked. (HTTP request larger than http.postBuffer git config setting leading to HTTP/1.1 chunked transfer.)
Also if there's a way to get more verbose logging out that might help point in the right direction.
Do try (from that stackoverflow):
# Linux export GIT_TRACE_PACKET=1 export GIT_TRACE=1 export GIT_CURL_VERBOSE=1
#Windows set GIT_TRACE_PACKET=1 set GIT_TRACE=1 set GIT_CURL_VERBOSE=1
Before the git command, and see what comes out.
//Peter _______________________________________________ coreboot mailing list -- coreboot@coreboot.orgmailto:coreboot@coreboot.org To unsubscribe send an email to coreboot-leave@coreboot.orgmailto:coreboot-leave@coreboot.org _______________________________________________ coreboot mailing list -- coreboot@coreboot.orgmailto:coreboot@coreboot.org To unsubscribe send an email to coreboot-leave@coreboot.orgmailto:coreboot-leave@coreboot.org
Hi Selma,
Bensaid, Selma wrote:
FYI, we are observing the same issue today on chromium coreboot repo https://chromium-review.googlesource.com/admin/repos/chromiumos/third_party/... I think we need to figure out a proper solution.
Thanks for providing the debug output!
So since the issue does stem from a large distance between branches and hitting network communication limits maybe an effective workaround would be to maintain a local clone of the upstream repo (in a separate repo) and using a file:// URL pointing to that repo for the origin remote in your working repo?
The upstream clone must be on the same machine, so it's more overhead, but file:// should be reliable.
HTTP, regardless of version, is really inappropriate for applications.
//Peter
Thanks Peter, yes we implemented something similar in out GitHub repo. I'll keep monitoring. Thanks again for your time! Selma.
-----Original Message----- From: Peter Stuge peter@stuge.se Sent: Tuesday, May 25, 2021 4:56 PM To: coreboot@coreboot.org Subject: [coreboot] Re: failure cherry-picking patches from https://review.coreboot.org/coreboot
Hi Selma,
Bensaid, Selma wrote:
FYI, we are observing the same issue today on chromium coreboot repo https://chromium-review.googlesource.com/admin/repos/chromiumos/third_ party/coreboot I think we need to figure out a proper solution.
Thanks for providing the debug output!
So since the issue does stem from a large distance between branches and hitting network communication limits maybe an effective workaround would be to maintain a local clone of the upstream repo (in a separate repo) and using a file:// URL pointing to that repo for the origin remote in your working repo?
The upstream clone must be on the same machine, so it's more overhead, but file:// should be reliable.
HTTP, regardless of version, is really inappropriate for applications.
//Peter _______________________________________________ coreboot mailing list -- coreboot@coreboot.org To unsubscribe send an email to coreboot-leave@coreboot.org
-
Aaron Durbin -
Bensaid, Selma -
Peter Stuge