Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

git submodule update fails #1654

Open
zimingH opened this issue Jan 13, 2025 · 20 comments
Open

git submodule update fails #1654

zimingH opened this issue Jan 13, 2025 · 20 comments

Comments

@zimingH
Copy link

zimingH commented Jan 13, 2025

brief description

Try to do git clone and make, and then some error shows indicating this repo is missing something under binutils. Try to update the submodules. It fails.

terminal log:

ziming.huang@sdc02:~/tools/toolchain$ git clone https://github.com/riscv-collab/riscv-gnu-toolchain.git
Cloning into 'riscv-gnu-toolchain'...
remote: Enumerating objects: 9508, done.
remote: Counting objects: 100% (418/418), done.
remote: Compressing objects: 100% (102/102), done.
remote: Total 9508 (delta 356), reused 317 (delta 316), pack-reused 9090 (from 2)
Receiving objects: 100% (9508/9508), 6.45 MiB | 2.71 MiB/s, done.
Resolving deltas: 100% (4735/4735), done.
Updating files: 100% (1009/1009), done.
ziming.huang@sdc02:~/tools/toolchain$ cd riscv-gnu-toolchain
ziming.huang@sdc02:~/tools/toolchain/riscv-gnu-toolchain$ git submodule update --init --recursive
Submodule 'binutils' (https://sourceware.org/git/binutils-gdb.git) registered for path 'binutils'
Submodule 'dejagnu' (https://git.savannah.gnu.org/git/dejagnu.git) registered for path 'dejagnu'
Submodule 'gcc' (https://gcc.gnu.org/git/gcc.git) registered for path 'gcc'
Submodule 'gdb' (https://sourceware.org/git/binutils-gdb.git) registered for path 'gdb'
Submodule 'glibc' (https://sourceware.org/git/glibc.git) registered for path 'glibc'
Submodule 'llvm' (https://github.com/llvm/llvm-project.git) registered for path 'llvm'
Submodule 'musl' (https://git.musl-libc.org/git/musl) registered for path 'musl'
Submodule 'newlib' (https://sourceware.org/git/newlib-cygwin.git) registered for path 'newlib'
Submodule 'pk' (https://github.com/riscv-software-src/riscv-pk.git) registered for path 'pk'
Submodule 'qemu' (https://gitlab.com/qemu-project/qemu.git) registered for path 'qemu'
Submodule 'spike' (https://github.com/riscv-software-src/riscv-isa-sim.git) registered for path 'spike'
Submodule 'uclibc-ng' (https://github.com/wbx-github/uclibc-ng.git) registered for path 'uclibc-ng'
Cloning into '/home/ziming.huang/tools/toolchain/riscv-gnu-toolchain/binutils'...
Cloning into '/home/ziming.huang/tools/toolchain/riscv-gnu-toolchain/dejagnu'...
Cloning into '/home/ziming.huang/tools/toolchain/riscv-gnu-toolchain/gcc'...
Cloning into '/home/ziming.huang/tools/toolchain/riscv-gnu-toolchain/gdb'...
Cloning into '/home/ziming.huang/tools/toolchain/riscv-gnu-toolchain/glibc'...
Cloning into '/home/ziming.huang/tools/toolchain/riscv-gnu-toolchain/llvm'...
Cloning into '/home/ziming.huang/tools/toolchain/riscv-gnu-toolchain/musl'...
Cloning into '/home/ziming.huang/tools/toolchain/riscv-gnu-toolchain/newlib'...
Cloning into '/home/ziming.huang/tools/toolchain/riscv-gnu-toolchain/pk'...
Cloning into '/home/ziming.huang/tools/toolchain/riscv-gnu-toolchain/qemu'...
Cloning into '/home/ziming.huang/tools/toolchain/riscv-gnu-toolchain/spike'...
Cloning into '/home/ziming.huang/tools/toolchain/riscv-gnu-toolchain/uclibc-ng'...
error: Server does not allow request for unadvertised object beb2cdbcda911764b2bed5e57921fe90493260bd
Fetched in submodule path 'binutils', but it did not contain beb2cdbcda911764b2bed5e57921fe90493260bd. Direct fetching of that commit failed.
@zimingH
Copy link
Author

zimingH commented Jan 13, 2025

I saw this
#1642
and just tried this:
git clone https://github.com/riscv-collab/riscv-gnu-toolchain issue-1642 --recursive
still throwing me some errors:

Cloning into 'issue-1642'...
remote: Enumerating objects: 9508, done.
remote: Counting objects: 100% (416/416), done.
remote: Compressing objects: 100% (102/102), done.
remote: Total 9508 (delta 354), reused 315 (delta 314), pack-reused 9092 (from 2)
Receiving objects: 100% (9508/9508), 6.47 MiB | 4.21 MiB/s, done.
Resolving deltas: 100% (4729/4729), done.
Updating files: 100% (1009/1009), done.
Submodule 'binutils' (https://sourceware.org/git/binutils-gdb.git) registered for path 'binutils'
Submodule 'dejagnu' (https://git.savannah.gnu.org/git/dejagnu.git) registered for path 'dejagnu'
Submodule 'gcc' (https://gcc.gnu.org/git/gcc.git) registered for path 'gcc'
Submodule 'gdb' (https://sourceware.org/git/binutils-gdb.git) registered for path 'gdb'
Submodule 'glibc' (https://sourceware.org/git/glibc.git) registered for path 'glibc'
Submodule 'llvm' (https://github.com/llvm/llvm-project.git) registered for path 'llvm'
Submodule 'musl' (https://git.musl-libc.org/git/musl) registered for path 'musl'
Submodule 'newlib' (https://sourceware.org/git/newlib-cygwin.git) registered for path 'newlib'
Submodule 'pk' (https://github.com/riscv-software-src/riscv-pk.git) registered for path 'pk'
Submodule 'qemu' (https://gitlab.com/qemu-project/qemu.git) registered for path 'qemu'
Submodule 'spike' (https://github.com/riscv-software-src/riscv-isa-sim.git) registered for path 'spike'
Submodule 'uclibc-ng' (https://github.com/wbx-github/uclibc-ng.git) registered for path 'uclibc-ng'
Cloning into '/home/ziming.huang/tools/toolchain/issue-1642/binutils'...
remote: Enumerating objects: 40585, done.        
remote: Counting objects: 100% (40585/40585), done.        
remote: Compressing objects: 100% (31685/31685), done.        
remote: Total 40585 (delta 10525), reused 23485 (delta 7520), pack-reused 0        
Receiving objects: 100% (40585/40585), 78.18 MiB | 3.84 MiB/s, done.
Resolving deltas: 100% (10525/10525), done.
Cloning into '/home/ziming.huang/tools/toolchain/issue-1642/dejagnu'...
remote: Counting objects: 224, done.        
remote: Compressing objects: 100% (203/203), done.        
remote: Total 224 (delta 65), reused 62 (delta 8)        B/s
Receiving objects: 100% (224/224), 634.39 KiB | 274.00 KiB/s, done.
Resolving deltas: 100% (65/65), done.
Cloning into '/home/ziming.huang/tools/toolchain/issue-1642/gcc'...
error: RPC failed; curl 16 Error in the HTTP2 framing layer
fatal: the remote end hung up unexpectedly
fatal: clone of 'https://gcc.gnu.org/git/gcc.git' into submodule path '/home/ziming.huang/tools/toolchain/issue-1642/gcc' failed
Failed to clone 'gcc'. Retry scheduled
Cloning into '/home/ziming.huang/tools/toolchain/issue-1642/gdb'...
remote: Enumerating objects: 40585, done.        
remote: Counting objects: 100% (40585/40585), done.        
remote: Compressing objects: 100% (31685/31685), done.        
remote: Total 40585 (delta 10526), reused 23485 (delta 7520), pack-reused 0        
Receiving objects: 100% (40585/40585), 78.18 MiB | 3.09 MiB/s, done.
Resolving deltas: 100% (10526/10526), done.
Cloning into '/home/ziming.huang/tools/toolchain/issue-1642/glibc'...
remote: Enumerating objects: 20097, done.        
remote: Counting objects: 100% (20097/20097), done.        
remote: Compressing objects: 100% (9934/9934), done.        
remote: Total 20097 (delta 9678), reused 13328 (delta 8953), pack-reused 0        
Receiving objects: 100% (20097/20097), 42.95 MiB | 2.84 MiB/s, done.
Resolving deltas: 100% (9678/9678), done.
Cloning into '/home/ziming.huang/tools/toolchain/issue-1642/llvm'...
remote: Enumerating objects: 160814, done.        
remote: Counting objects: 100% (160814/160814), done.        
remote: Compressing objects: 100% (138185/138185), done.        
remote: Total 160814 (delta 34641), reused 56838 (delta 18179), pack-reused 0 (from 0)        
Receiving objects: 100% (160814/160814), 232.65 MiB | 8.79 MiB/s, done.
Resolving deltas: 100% (34641/34641), done.
Checking connectivity: 160814, done.
Cloning into '/home/ziming.huang/tools/toolchain/issue-1642/musl'...
remote: Enumerating objects: 2700, done.        
remote: Counting objects: 100% (2700/2700), done.        
remote: Compressing objects: 100% (2510/2510), done.        
remote: Total 2700 (delta 321), reused 960 (delta 96), pack-reused 0
Receiving objects: 100% (2700/2700), 1.27 MiB | 382.00 KiB/s, done.
Resolving deltas: 100% (321/321), done.
Cloning into '/home/ziming.huang/tools/toolchain/issue-1642/newlib'...
remote: Enumerating objects: 6971, done.        
remote: Counting objects: 100% (6971/6971), done.        
remote: Compressing objects: 100% (5331/5331), done.        
remote: Total 6971 (delta 2585), reused 3244 (delta 1512), pack-reused 0        
Receiving objects: 100% (6971/6971), 14.31 MiB | 3.05 MiB/s, done.
Resolving deltas: 100% (2585/2585), done.
Cloning into '/home/ziming.huang/tools/toolchain/issue-1642/pk'...
remote: Enumerating objects: 443, done.        
remote: Counting objects: 100% (443/443), done.        
remote: Compressing objects: 100% (134/134), done.        
remote: Total 443 (delta 312), reused 374 (delta 308), pack-reused 0 (from 0)        
Receiving objects: 100% (443/443), 358.86 KiB | 918.00 KiB/s, done.
Resolving deltas: 100% (312/312), done.
Cloning into '/home/ziming.huang/tools/toolchain/issue-1642/qemu'...
remote: Enumerating objects: 10555, done.        
remote: Counting objects: 100% (10555/10555), done.        
remote: Compressing objects: 100% (9371/9371), done.        
remote: Total 10555 (delta 1604), reused 5735 (delta 858), pack-reused 0 (from 0)        
Receiving objects: 100% (10555/10555), 40.44 MiB | 6.69 MiB/s, done.
Resolving deltas: 100% (1604/1604), done.
Cloning into '/home/ziming.huang/tools/toolchain/issue-1642/spike'...
remote: Enumerating objects: 1364, done.        
remote: Counting objects: 100% (1364/1364), done.        
remote: Compressing objects: 100% (903/903), done.        
remote: Total 1364 (delta 469), reused 1012 (delta 349), pack-reused 0 (from 0)        
Receiving objects: 100% (1364/1364), 628.06 KiB | 916.00 KiB/s, done.
Resolving deltas: 100% (469/469), done.
Cloning into '/home/ziming.huang/tools/toolchain/issue-1642/uclibc-ng'...
remote: Enumerating objects: 4922, done.        
remote: Counting objects: 100% (4922/4922), done.        
remote: Compressing objects: 100% (3854/3854), done.        
remote: Total 4922 (delta 2045), reused 2142 (delta 926), pack-reused 0 (from 0)        
Receiving objects: 100% (4922/4922), 3.87 MiB | 1.87 MiB/s, done.
Resolving deltas: 100% (2045/2045), done.
Cloning into '/home/ziming.huang/tools/toolchain/issue-1642/gcc'...
remote: Enumerating objects: 145473, done.        
remote: Counting objects: 100% (145473/145473), done.        
remote: Compressing objects: 100% (111873/111873), done.        
remote: Total 145473 (delta 37545), reused 90496 (delta 31574), pack-reused 0        
Receiving objects: 100% (145473/145473), 175.10 MiB | 5.32 MiB/s, done.
Resolving deltas: 100% (37545/37545), done.
Checking connectivity: 145473, done.
error: Server does not allow request for unadvertised object beb2cdbcda911764b2bed5e57921fe90493260bd
Fetched in submodule path 'binutils', but it did not contain beb2cdbcda911764b2bed5e57921fe90493260bd. Direct fetching of that commit failed.

Did I miss anything? or did anything wrong? Appreciate any help. Thanks!

@caseyavila
Copy link

I also just ran into this issue. This fixed it for me.

@TommyMurphyTM1234
Copy link
Collaborator

TommyMurphyTM1234 commented Jan 13, 2025

Yes, encountered here too:

Disabling the shallow clone/checkout is a temporary workaround until the issue is fixed properly.

Edit: actually maybe not the same issue - see my following post...

@TommyMurphyTM1234
Copy link
Collaborator

TommyMurphyTM1234 commented Jan 13, 2025

brief description

Try to do git clone and make, and then some error shows indicating this repo is missing something under binutils. Try to update the submodules. It fails.

terminal log:

ziming.huang@sdc02:~/tools/toolchain$ git clone https://github.com/riscv-collab/riscv-gnu-toolchain.git
Cloning into 'riscv-gnu-toolchain'...
remote: Enumerating objects: 9508, done.
remote: Counting objects: 100% (418/418), done.
remote: Compressing objects: 100% (102/102), done.
remote: Total 9508 (delta 356), reused 317 (delta 316), pack-reused 9090 (from 2)
Receiving objects: 100% (9508/9508), 6.45 MiB | 2.71 MiB/s, done.
Resolving deltas: 100% (4735/4735), done.
Updating files: 100% (1009/1009), done.
ziming.huang@sdc02:~/tools/toolchain$ cd riscv-gnu-toolchain
ziming.huang@sdc02:~/tools/toolchain/riscv-gnu-toolchain$ git submodule update --init --recursive

Why are you doing the last command?
It's unnecessary if you let the Makefile do the cloning of submodules:

git clone https://github.com/riscv-collab/riscv-gnu-toolchain issue-1654
cd issue-1654
./configure --prefix=`pwd`/installed-tools
make 2>&1 | tee build.log
...
# Build completes successfully

@TommyMurphyTM1234
Copy link
Collaborator

Why are you doing the last command? It's unnecessary if you let the Makefile do the cloning of submodules:

git clone https://github.com/riscv-collab/riscv-gnu-toolchain issue-1654
cd issue-1654
./configure --prefix=`pwd`/installed-tools
make 2>&1 | tee build.log
...
# Build completes successfully

FWIW...

# build.log

...

cd /home/user/issue-1654 && \
flock `git rev-parse --git-dir`/config git submodule init /home/user/issue-1654/binutils/ && \
flock `git rev-parse --git-dir`/config git submodule update --progress --depth 1 /home/user/issue-1654/binutils/
Submodule 'binutils' (https://sourceware.org/git/binutils-gdb.git) registered for path 'binutils'
Cloning into '/home/user/issue-1654/binutils'...
remote: Enumerating objects: 40585, done.        
remote: Counting objects:   0% (1/40585)        
...
remote: Counting objects: 100% (40585/40585), done.        
remote: Compressing objects:   0% (1/31685)        
...
remote: Compressing objects: 100% (31685/31685), done.        
Receiving objects:   0% (1/40585)
Receiving objects:   0% (125/40585), 2.61 MiB | 2.60 MiB/s
Receiving objects:   1% (406/40585), 4.75 MiB | 3.15 MiB/s
...
Receiving objects:  99% (40180/40585), 74.14 MiB | 4.64 MiB/s
remote: Total 40585 (delta 10526), reused 23485 (delta 7520), pack-reused 0        
Receiving objects: 100% (40585/40585), 76.92 MiB | 4.62 MiB/s
Receiving objects: 100% (40585/40585), 78.17 MiB | 4.71 MiB/s, done.
Resolving deltas:   0% (0/10526)
Resolving deltas:   1% (106/10526)
...
Resolving deltas: 100% (10526/10526)
Resolving deltas: 100% (10526/10526), done.
From https://sourceware.org/git/binutils-gdb
 * branch              beb2cdbcda911764b2bed5e57921fe90493260bd -> FETCH_HEAD
Submodule path 'binutils': checked out 'beb2cdbcda911764b2bed5e57921fe90493260bd'
mkdir -p /home/user/issue-1654/installed-tools/.test || \
	(echo "Sorry, you don't have permission to write to" \
	 "'/home/user/issue-1654/installed-tools'. Please make sure that the location is " \
	 "writable or use --prefix to specify another destination.'" \
	 && exit 1)
rm -r /home/user/issue-1654/installed-tools/.test
mkdir -p stamps/ && touch stamps/check-write-permission
rm -rf stamps/build-binutils-newlib build-binutils-newlib
mkdir build-binutils-newlib
cd build-binutils-newlib && CC_FOR_TARGET=riscv64-unknown-elf-gcc /home/user/issue-1654/binutils/configure \
	--target=riscv64-unknown-elf \
	 \
	--prefix=/home/user/issue-1654/installed-tools \
	--enable-plugins \
	 \
	--disable-werror \
	--with-expat=yes  \
	--disable-gdb \
	--disable-sim \
	--disable-libdecnumber \
	--disable-readline \
	--with-isa-spec=20191213
checking build system type... x86_64-pc-linux-gnu
...

@zimingH
Copy link
Author

zimingH commented Jan 14, 2025

Thanks! I tried but still encounter the same issue.

ziming.huang@sdc02:~/tools/toolchain$ git clone https://github.com/riscv-collab/riscv-gnu-toolchain issue-1654
Cloning into 'issue-1654'...
remote: Enumerating objects: 9508, done.
remote: Counting objects: 100% (416/416), done.
remote: Compressing objects: 100% (102/102), done.
remote: Total 9508 (delta 354), reused 316 (delta 314), pack-reused 9092 (from 2)
Receiving objects: 100% (9508/9508), 6.47 MiB | 4.16 MiB/s, done.
Resolving deltas: 100% (4729/4729), done.
Updating files: 100% (1009/1009), done.
ziming.huang@sdc02:~/tools/toolchain$ git branch -r
fatal: not a git repository (or any parent up to mount point /home)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
ziming.huang@sdc02:~/tools/toolchain$ cd issue-1654/
ziming.huang@sdc02:~/tools/toolchain/issue-1654$ git branch
* master
ziming.huang@sdc02:~/tools/toolchain/issue-1654$ git branch -r
  origin/HEAD -> origin/master
  origin/__archive__
  origin/big-endian
  origin/fix-typo
  origin/gdb-test
  origin/improve-report
  origin/kitoc/fix-generate_target_board
  origin/kitoc/update-qemu-cpu-opt
  origin/master
ziming.huang@sdc02:~/tools/toolchain/issue-1654$ ./configure --prefix=`pwd`/installed-tools
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether the compiler supports GNU C... yes
checking whether gcc accepts -g... yes
checking for gcc option to enable C11 features... none needed
checking for grep that handles long lines and -e... /usr/bin/grep
checking for fgrep... /usr/bin/grep -F
checking for grep that handles long lines and -e... (cached) /usr/bin/grep
checking for bash... /bin/bash
checking for __gmpz_init in -lgmp... yes
checking for mpfr_init in -lmpfr... yes
checking for mpc_init2 in -lmpc... yes
checking for curl... /usr/bin/curl
checking for wget... /usr/bin/wget
checking for ftp... /usr/bin/ftp
configure: creating ./config.status
config.status: creating Makefile
config.status: creating scripts/wrapper/awk/awk
config.status: creating scripts/wrapper/sed/sed
ziming.huang@sdc02:~/tools/toolchain/issue-1654$ make 2>&1 | tee build.log
cd /home/ziming.huang/tools/toolchain/issue-1654 && \
flock `git rev-parse --git-dir`/config git submodule init /home/ziming.huang/tools/toolchain/issue-1654/gcc/ && \
flock `git rev-parse --git-dir`/config git submodule update --progress --depth 1 /home/ziming.huang/tools/toolchain/issue-1654/gcc/
Submodule 'gcc' (https://gcc.gnu.org/git/gcc.git) registered for path 'gcc'
Cloning into '/home/ziming.huang/tools/toolchain/issue-1654/gcc'...
remote: Enumerating objects: 145507, done.        
remote: Counting objects: 100% (145507/145507), done.        
remote: Compressing objects: 100% (111891/111891), done.        
remote: Total 145507 (delta 37560), reused 90570 (delta 31590), pack-reused 0        
Receiving objects: 100% (145507/145507), 175.10 MiB | 4.69 MiB/s, done.
Resolving deltas: 100% (37560/37560), done.
Checking connectivity: 145507, done.
error: Server does not allow request for unadvertised object 04696df09633baf97cdbbdd6e9929b9d472161d3
Fetched in submodule path 'gcc', but it did not contain 04696df09633baf97cdbbdd6e9929b9d472161d3. Direct fetching of that commit failed.
make: *** [Makefile:348: /home/ziming.huang/tools/toolchain/issue-1654/gcc/.git] Error 1
ziming.huang@sdc02:~/tools/toolchain/issue-1654$ ls
binutils  build.log  config.log  config.status  configure  configure.ac  contrib  dejagnu  gcc  gdb  glibc  LICENSE  linux-headers  llvm  Makefile  Makefile.in  musl  newlib  pk  qemu  README.md  regression  scripts  spike  test  uclibc-ng
ziming.huang@sdc02:~/tools/toolchain/issue-1654$ cat build.log 
cd /home/ziming.huang/tools/toolchain/issue-1654 && \
flock `git rev-parse --git-dir`/config git submodule init /home/ziming.huang/tools/toolchain/issue-1654/gcc/ && \
flock `git rev-parse --git-dir`/config git submodule update --progress --depth 1 /home/ziming.huang/tools/toolchain/issue-1654/gcc/
Submodule 'gcc' (https://gcc.gnu.org/git/gcc.git) registered for path 'gcc'
Cloning into '/home/ziming.huang/tools/toolchain/issue-1654/gcc'...
remote: Enumerating objects: 145507, done.        
remote: Counting objects: 100% (145507/145507), done.        
remote: Compressing objects: 100% (111891/111891), done.        
remote: Total 145507 (delta 37560), reused 90570 (delta 31590), pack-reused 0        
Receiving objects: 100% (145507/145507), 175.10 MiB | 4.69 MiB/s, done.
Resolving deltas: 100% (37560/37560), done.
Checking connectivity: 145507, done.
error: Server does not allow request for unadvertised object 04696df09633baf97cdbbdd6e9929b9d472161d3
Fetched in submodule path 'gcc', but it did not contain 04696df09633baf97cdbbdd6e9929b9d472161d3. Direct fetching of that commit failed.
make: *** [Makefile:348: /home/ziming.huang/tools/toolchain/issue-1654/gcc/.git] Error 1

@TommyMurphyTM1234
Copy link
Collaborator

TommyMurphyTM1234 commented Jan 14, 2025

Thanks! I tried but still encounter the same issue.

I don't know why you're doing git branch.
If you want to build the toolchain then just do this:

git clone https://github.com/riscv-collab/riscv-gnu-toolchain issue-1654
cd issue-1654
./configure --prefix=`pwd`/installed-tools ...
make 

@caseyavila
Copy link

@zimingH Did you try git config --global protocol.version 2? Like I said in my earlier comment, I was having the same issue and this fixed it.

@TommyMurphyTM1234
Copy link
Collaborator

@zimingH Did you try git config --global protocol.version 2? Like I said in my earlier comment, I was having the same issue and this fixed it.

You don't need to do anything extra if you just follow the instructions in my previous post.

@zimingH
Copy link
Author

zimingH commented Jan 14, 2025

Hi all, I tried again and this work if I just want to build the toolchain:

git clone https://github.com/riscv-collab/riscv-gnu-toolchain issue-1654
cd issue-1654
./configure --prefix= `pwd`/installed-tools ...
make 

but still having problem with submodule updates.

I have tried git config --global protocol.version 2. Similar issue persists.

@TommyMurphyTM1234
Copy link
Collaborator

Hi all, I tried again and this work if I just want to build the toolchain:

git clone https://github.com/riscv-collab/riscv-gnu-toolchain issue-1654
cd issue-1654
./configure --prefix=`pwd`/installed-tools ...
make 

I can't reproduce your problem using those commands.
Please do the following.

rm -rf issue-1654
git clone https://github.com/riscv-collab/riscv-gnu-toolchain issue-1654
cd issue-1654
./configure --prefix=`pwd`/installed-tools 
make 2>&1 | tee build.log

and upload the (zipped if large) build.log.

@TommyMurphyTM1234
Copy link
Collaborator

This is what I get:

git clone https://github.com/riscv-collab/riscv-gnu-toolchain issue-1654
Cloning into 'issue-1654'...
remote: Enumerating objects: 9508, done.
remote: Counting objects: 100% (415/415), done.
remote: Compressing objects: 100% (101/101), done.
remote: Total 9508 (delta 353), reused 316 (delta 314), pack-reused 9093 (from 2)
Receiving objects: 100% (9508/9508), 6.46 MiB | 7.64 MiB/s, done.
Resolving deltas: 100% (4729/4729), done.

cd issue-1654/

./configure --prefix=`pwd`/installed-tools
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether the compiler supports GNU C... yes
checking whether gcc accepts -g... yes
checking for gcc option to enable C11 features... none needed
checking for grep that handles long lines and -e... /usr/bin/grep
checking for fgrep... /usr/bin/grep -F
checking for grep that handles long lines and -e... (cached) /usr/bin/grep
checking for bash... /bin/bash
checking for __gmpz_init in -lgmp... yes
checking for mpfr_init in -lmpfr... yes
checking for mpc_init2 in -lmpc... yes
checking for curl... /usr/bin/curl
checking for wget... /usr/bin/wget
checking for ftp... no
configure: creating ./config.status
config.status: creating Makefile
config.status: creating scripts/wrapper/awk/awk
config.status: creating scripts/wrapper/sed/sed

make 2>&1 | tee build.log
...
cd /home/user/issue-1654 && \
flock `git rev-parse --git-dir`/config git submodule init /home/user/issue-1654/binutils/ && \
flock `git rev-parse --git-dir`/config git submodule update --progress --depth 1 /home/user/issue-1654/binutils/
Submodule 'binutils' (https://sourceware.org/git/binutils-gdb.git) registered for path 'binutils'
Cloning into '/home/user/issue-1654/binutils'...
remote: Enumerating objects: 40627, done.        
...
From https://sourceware.org/git/binutils-gdb
 * branch              beb2cdbcda911764b2bed5e57921fe90493260bd -> FETCH_HEAD
Submodule path 'binutils': checked out 'beb2cdbcda911764b2bed5e57921fe90493260bd'
mkdir -p /home/user/issue-1654/installed-tools/.test || \
	(echo "Sorry, you don't have permission to write to" \
	 "'/home/user/issue-1654/installed-tools'. Please make sure that the location is " \
	 "writable or use --prefix to specify another destination.'" \
	 && exit 1)
rm -r /home/user/issue-1654/installed-tools/.test
mkdir -p stamps/ && touch stamps/check-write-permission
rm -rf stamps/build-binutils-newlib build-binutils-newlib
mkdir build-binutils-newlib
cd build-binutils-newlib && CC_FOR_TARGET=riscv64-unknown-elf-gcc /home/user/issue-1654/binutils/configure \
	--target=riscv64-unknown-elf \
	 \
	--prefix=/home/user/issue-1654/installed-tools \
	--enable-plugins \
	 \
	--disable-werror \
	--with-expat=yes  \
	--disable-gdb \
	--disable-sim \
	--disable-libdecnumber \
	--disable-readline \
	--with-isa-spec=20191213
checking build system type... x86_64-pc-linux-gnu
checking host system type... x86_64-pc-linux-gnu
checking target system type... riscv64-unknown-elf
...

@zimingH
Copy link
Author

zimingH commented Jan 14, 2025

Hi, thanks. I can build. If I just want to build gcc and stuff key stuff, actually there is no problem anymore. That's what I was trying to say.

I still cannot do submodule update. I also need some submodules like pk and spike. Now I still cannot do git submodule update --init from this repo, and after build some submodule folders, like pk, are still empty.

@TommyMurphyTM1234
Copy link
Collaborator

I still cannot do submodule update. I also need some submodules like pk and spike. Now I still cannot do git submodule update --init from this repo, and after build some submodule folders, like pk, are still empty.

Do you mean this?

rm -rf issue-1654

git clone https://github.com/riscv-collab/riscv-gnu-toolchain issue-1654
Cloning into 'issue-1654'...
remote: Enumerating objects: 9508, done.
remote: Counting objects: 100% (418/418), done.
remote: Compressing objects: 100% (102/102), done.
remote: Total 9508 (delta 356), reused 318 (delta 316), pack-reused 9090 (from 2)
Receiving objects: 100% (9508/9508), 6.45 MiB | 6.58 MiB/s, done.
Resolving deltas: 100% (4735/4735), done.

cd issue-1654/

git submodule update --init
Submodule 'binutils' (https://sourceware.org/git/binutils-gdb.git) registered for path 'binutils'
Submodule 'dejagnu' (https://git.savannah.gnu.org/git/dejagnu.git) registered for path 'dejagnu'
Submodule 'gcc' (https://gcc.gnu.org/git/gcc.git) registered for path 'gcc'
Submodule 'gdb' (https://sourceware.org/git/binutils-gdb.git) registered for path 'gdb'
Submodule 'glibc' (https://sourceware.org/git/glibc.git) registered for path 'glibc'
Submodule 'llvm' (https://github.com/llvm/llvm-project.git) registered for path 'llvm'
Submodule 'musl' (https://git.musl-libc.org/git/musl) registered for path 'musl'
Submodule 'newlib' (https://sourceware.org/git/newlib-cygwin.git) registered for path 'newlib'
Submodule 'pk' (https://github.com/riscv-software-src/riscv-pk.git) registered for path 'pk'
Submodule 'qemu' (https://gitlab.com/qemu-project/qemu.git) registered for path 'qemu'
Submodule 'spike' (https://github.com/riscv-software-src/riscv-isa-sim.git) registered for path 'spike'
Submodule 'uclibc-ng' (https://github.com/wbx-github/uclibc-ng.git) registered for path 'uclibc-ng'
Cloning into '/home/user/issue-1654/binutils'...
Cloning into '/home/user/issue-1654/dejagnu'...
Cloning into '/home/user/issue-1654/gcc'...
Cloning into '/home/user/issue-1654/gdb'...
Cloning into '/home/user/issue-1654/glibc'...
Cloning into '/home/user/issue-1654/llvm'...
Cloning into '/home/user/issue-1654/musl'...
Cloning into '/home/user/issue-1654/newlib'...
Cloning into '/home/user/issue-1654/pk'...
Cloning into '/home/user/issue-1654/qemu'...
Cloning into '/home/user/issue-1654/spike'...
Cloning into '/home/user/issue-1654/uclibc-ng'...
remote: Enumerating objects: 1080286, done.
remote: Counting objects: 100% (1080243/1080243), done.
remote: Compressing objects: 100% (167358/167358), done.
remote: Total 1057495 (delta 910248), reused 998861 (delta 883083), pack-reused 0
Receiving objects: 100% (1057495/1057495), 330.38 MiB | 10.46 MiB/s, done.
Resolving deltas: 100% (910248/910248), completed with 16200 local objects.
From https://sourceware.org/git/binutils-gdb
 * branch                    beb2cdbcda911764b2bed5e57921fe90493260bd -> FETCH_HEAD
Submodule path 'binutils': checked out 'beb2cdbcda911764b2bed5e57921fe90493260bd'
error: Server does not allow request for unadvertised object ca371cf9c48186716d26c8e10d442affaf002c80
fatal: Fetched in submodule path 'dejagnu', but it did not contain ca371cf9c48186716d26c8e10d442affaf002c80. Direct fetching of that commit failed.

If you need pk and Spike specifically then you can do this:

rm -rf issue-1654

git clone https://github.com/riscv-collab/riscv-gnu-toolchain issue-1654
Cloning into 'issue-1654'...
remote: Enumerating objects: 9508, done.
remote: Counting objects: 100% (418/418), done.
remote: Compressing objects: 100% (102/102), done.
remote: Total 9508 (delta 356), reused 318 (delta 316), pack-reused 9090 (from 2)
Receiving objects: 100% (9508/9508), 6.45 MiB | 6.58 MiB/s, done.
Resolving deltas: 100% (4735/4735), done.

cd issue-1654/

git submodule update --init pk
Submodule 'pk' (https://github.com/riscv-software-src/riscv-pk.git) registered for path 'pk'
Cloning into '/home/user/issue-1654/pk'...
Submodule path 'pk': checked out 'abadfdc507d5a75b6272dc360e70a80a510c758a'

git submodule update --init  spike
Submodule 'spike' (https://github.com/riscv-software-src/riscv-isa-sim.git) registered for path 'spike'
Cloning into '/home/user/issue-1654/spike'...
remote: Enumerating objects: 22062, done.
remote: Counting objects: 100% (22061/22061), done.
remote: Compressing objects: 100% (6265/6265), done.
remote: Total 21209 (delta 14762), reused 20381 (delta 14050), pack-reused 0 (from 0)
Receiving objects: 100% (21209/21209), 5.27 MiB | 6.57 MiB/s, done.
Resolving deltas: 100% (14762/14762), completed with 410 local objects.
From https://github.com/riscv-software-src/riscv-isa-sim
 * branch              88fc84ded155a9e01987c4dfb7a77800e69b232b -> FETCH_HEAD
Submodule path 'spike': checked out '88fc84ded155a9e01987c4dfb7a77800e69b232b'

@TommyMurphyTM1234
Copy link
Collaborator

@mickflemm - any ideas about this?

git clone https://git.savannah.gnu.org/git/dejagnu.git --depth=1
Cloning into 'dejagnu'...
remote: Counting objects: 224, done.
remote: Compressing objects: 100% (203/203), done.
remote: Total 224 (delta 65), reused 62 (delta 8)
Receiving objects: 100% (224/224), 634.39 KiB | 859.00 KiB/s, done.
Resolving deltas: 100% (65/65), done.

cd dejagnu

git checkout ca371cf9c48186716d26c8e10d442affaf002c80
fatal: reference is not a tree: ca371cf9c48186716d26c8e10d442affaf002c80

@mickflemm
Copy link
Contributor

I can't reproduce this in my system...

mick@Gazofonias /tmp $ git clone https://git.savannah.gnu.org/git/dejagnu.git --depth=1
Cloning into 'dejagnu'...
remote: Counting objects: 224, done.
remote: Compressing objects: 100% (203/203), done.
remote: Total 224 (delta 65), reused 62 (delta 8)
Receiving objects: 100% (224/224), 634.39 KiB | 1.03 MiB/s, done.
Resolving deltas: 100% (65/65), done.
mick@Gazofonias /tmp $ cd dejagnu/
mick@Gazofonias /tmp/dejagnu $ ls
AUTHORS  COPYING  ChangeLog  ChangeLog-1992  INSTALL  MAINTAINERS  Makefile.am  Makefile.in  NEWS  README  TODO  aclocal.m4  autogen.sh  baseboards  commands  compile  config  config.guess  config.sub  configure  configure.ac  contrib  dejagnu  dejagnu.h  depcomp  doc  install-sh  lib  mdate-sh  missing  runtest  runtest.exp  stub-loader.c  testglue.c  testsuite

Have in mind that they support shallow clones as mentioned in their git guide:
https://savannah.gnu.org/maintenance/UsingGit/

@mickflemm
Copy link
Contributor

Ah I noticed the checkout, this will obviously fail since the commit you mention is not fetched, you only get the latest commit when doing shallow clone (unless you ask for a specific branch when cloning):

git log
commit 935a51f3c66ece357ce0d18f3aa3627a13cef7d5 (grafted, HEAD -> master, origin/master, origin/HEAD)
Author: Jacob Bachmeyer <[email protected]>
Date:   Sun Oct 20 20:52:06 2024 -0500

    Fix typo in reference manual

@TommyMurphyTM1234
Copy link
Collaborator

Ah I noticed the checkout, this will obviously fail since the commit you mention is not fetched, you only get the latest commit when doing shallow clone (unless you ask for a specific branch when cloning):

git log
commit 935a51f3c66ece357ce0d18f3aa3627a13cef7d5 (grafted, HEAD -> master, origin/master, origin/HEAD)
Author: Jacob Bachmeyer <[email protected]>
Date:   Sun Oct 20 20:52:06 2024 -0500

    Fix typo in reference manual

Hi @mickflemm - would you expect this to work?

rm -rf issue-1654

git clone https://github.com/riscv-collab/riscv-gnu-toolchain issue-1654
Cloning into 'issue-1654'...
remote: Enumerating objects: 9508, done.
remote: Counting objects: 100% (418/418), done.
remote: Compressing objects: 100% (102/102), done.
remote: Total 9508 (delta 356), reused 318 (delta 316), pack-reused 9090 (from 2)
Receiving objects: 100% (9508/9508), 6.45 MiB | 6.58 MiB/s, done.
Resolving deltas: 100% (4735/4735), done.

cd issue-1654/

git submodule update --init
Submodule 'binutils' (https://sourceware.org/git/binutils-gdb.git) registered for path 'binutils'
Submodule 'dejagnu' (https://git.savannah.gnu.org/git/dejagnu.git) registered for path 'dejagnu'
Submodule 'gcc' (https://gcc.gnu.org/git/gcc.git) registered for path 'gcc'
Submodule 'gdb' (https://sourceware.org/git/binutils-gdb.git) registered for path 'gdb'
Submodule 'glibc' (https://sourceware.org/git/glibc.git) registered for path 'glibc'
Submodule 'llvm' (https://github.com/llvm/llvm-project.git) registered for path 'llvm'
Submodule 'musl' (https://git.musl-libc.org/git/musl) registered for path 'musl'
Submodule 'newlib' (https://sourceware.org/git/newlib-cygwin.git) registered for path 'newlib'
Submodule 'pk' (https://github.com/riscv-software-src/riscv-pk.git) registered for path 'pk'
Submodule 'qemu' (https://gitlab.com/qemu-project/qemu.git) registered for path 'qemu'
Submodule 'spike' (https://github.com/riscv-software-src/riscv-isa-sim.git) registered for path 'spike'
Submodule 'uclibc-ng' (https://github.com/wbx-github/uclibc-ng.git) registered for path 'uclibc-ng'
Cloning into '/home/user/issue-1654/binutils'...
Cloning into '/home/user/issue-1654/dejagnu'...
Cloning into '/home/user/issue-1654/gcc'...
Cloning into '/home/user/issue-1654/gdb'...
Cloning into '/home/user/issue-1654/glibc'...
Cloning into '/home/user/issue-1654/llvm'...
Cloning into '/home/user/issue-1654/musl'...
Cloning into '/home/user/issue-1654/newlib'...
Cloning into '/home/user/issue-1654/pk'...
Cloning into '/home/user/issue-1654/qemu'...
Cloning into '/home/user/issue-1654/spike'...
Cloning into '/home/user/issue-1654/uclibc-ng'...
remote: Enumerating objects: 1080286, done.
remote: Counting objects: 100% (1080243/1080243), done.
remote: Compressing objects: 100% (167358/167358), done.
remote: Total 1057495 (delta 910248), reused 998861 (delta 883083), pack-reused 0
Receiving objects: 100% (1057495/1057495), 330.38 MiB | 10.46 MiB/s, done.
Resolving deltas: 100% (910248/910248), completed with 16200 local objects.
From https://sourceware.org/git/binutils-gdb
 * branch                    beb2cdbcda911764b2bed5e57921fe90493260bd -> FETCH_HEAD
Submodule path 'binutils': checked out 'beb2cdbcda911764b2bed5e57921fe90493260bd'
error: Server does not allow request for unadvertised object ca371cf9c48186716d26c8e10d442affaf002c80
fatal: Fetched in submodule path 'dejagnu', but it did not contain ca371cf9c48186716d26c8e10d442affaf002c80. Direct fetching of that commit failed.

@mickflemm
Copy link
Contributor

mickflemm commented Jan 14, 2025

When we clone a specific tag, that commit is advertised as a reference and we can fetch it directly, for example:

mick@Gazofonias ~/Workspace/issue-1654 $ git submodule | grep binutils
-beb2cdbcda911764b2bed5e57921fe90493260bd binutils
mick@Gazofonias ~/Workspace/issue-1654 $ git ls-remote https://sourceware.org/git/binutils-gdb.git | grep beb2cdbcda911764b2bed5e57921fe90493260bd
beb2cdbcda911764b2bed5e57921fe90493260bd	refs/tags/binutils-2_43_1^{}

which also matches what we have in .gitmodules

[submodule "binutils"]
	path = binutils
	url = https://sourceware.org/git/binutils-gdb.git
	branch = binutils-2_43-branch
	shallow = true

Here is a quick n dirty bash script I wrote to check if the sha and/or branch of each submodule is advertised:

#!/bin/bash

function get_url_of_submodule ()
{
	cat .gitmodules | grep -A3 "submodule \"${1}\"" | grep url | awk '{print $3}'
}

function get_branch_of_submodule ()
{
	cat .gitmodules | grep -A3 "submodule \"${1}\"" | grep branch | awk '{print $3}'
}

function get_sha_for_submodule ()
{
	local OUT=$(git submodule | grep $1 | awk '{print $1}')
	echo ${OUT:1}
}

function check_for_submodule_sha ()
{
	local URL=$(get_url_of_submodule ${1})
	local SHA=$(get_sha_for_submodule ${1})

	REF=$(git ls-remote ${URL} | grep ${SHA})
	if [[ $? == 0 ]]; then
		echo Commit id for submodule ${1} is advertised
		echo Ref: $(echo ${REF} | awk '{print $2}')
	else
		echo Commit id for submodule ${1} not found
	fi
}

function check_for_submodule_branch ()
{
	local URL=$(get_url_of_submodule ${1})
	local BRANCH=$(get_branch_of_submodule ${1})

	if [[ ${BRANCH} == "" ]]; then
		return
	fi

	REF=$(git ls-remote ${URL} | grep ${BRANCH})
	if [[ $? == 0 ]]; then
		echo Branch for submodule ${1} is advertised
		echo Ref: $(echo ${REF} | awk '{print $2}')
	else
		echo Branch for submodule ${1} not found
	fi
}

for i in $(git submodule | awk '{print $2}'); do
	check_for_submodule_sha ${i}
	check_for_submodule_branch ${i}
	echo -e '\n'
done

And its current output...

Commit id for submodule binutils is advertised
Ref: refs/tags/binutils-2_43_1^{}
Branch for submodule binutils is advertised
Ref: refs/heads/binutils-2_43-branch


Commit id for submodule dejagnu not found
Branch for submodule dejagnu is advertised
Ref: refs/heads/master


Commit id for submodule gcc is advertised
Ref: refs/tags/releases/gcc-14.2.0^{}
Branch for submodule gcc is advertised
Ref: refs/heads/releases/gcc-14


Commit id for submodule gdb is advertised
Ref: refs/tags/gdb-15.2-release^{}
Branch for submodule gdb is advertised
Ref: refs/heads/gdb-15-branch


Commit id for submodule glibc is advertised
Ref: refs/tags/glibc-2.40^{}


Commit id for submodule llvm is advertised
Ref: refs/heads/release/18.x
Branch for submodule llvm is advertised
Ref: refs/heads/release/18.x


Commit id for submodule musl is advertised
Ref: refs/tags/v1.2.5
Branch for submodule musl is advertised
Ref: refs/heads/master


Commit id for submodule newlib is advertised
Ref: refs/tags/newlib-4.4.0^{}
Branch for submodule newlib is advertised
Ref: refs/heads/master


Commit id for submodule pk is advertised
Ref: HEAD
Branch for submodule pk is advertised
Ref: refs/heads/master


Commit id for submodule qemu is advertised
Ref: refs/heads/stable-9.2


Commit id for submodule spike not found
Branch for submodule spike is advertised
Ref: refs/heads/master


Commit id for submodule uclibc-ng is advertised
Ref: HEAD

Notice that the commit ids for dejagnu and spike are not advertised, so a shallow clone may fail since it won't fetch the full list of commits, but the latest few, and as new commits come it, we'll loose the one we are after. We may increase depth to 10, 100 etc in order to get more of them, and increase the probability of finding the one we are looking for, but that's not the right approach. Instead we should make sure that when adding / updating a submodule, we use a specific branch / tag and not e.g. "master" (which basically means the latest HEAD that may change in the future).

The problem with dejagnu, spike and pk is that they don't do releases, so we can either go with some ancient tag they put at some point, get the latest HEAD which is probably the best option (although we risk getting bugs as well), or not do a shallow clone for those.

To use the latest one, in your command add --remote --force, e.g.:

mick@Gazofonias ~/Workspace/issue-1654 $ git submodule update --init --remote --force --depth=1 dejagnu
Submodule 'dejagnu' (https://git.savannah.gnu.org/git/dejagnu.git) registered for path 'dejagnu'
Cloning into '/home/mick/Workspace/issue-1654/dejagnu'...
remote: Total 0 (delta 0), reused 0 (delta 0)
Submodule path 'dejagnu': checked out '935a51f3c66ece357ce0d18f3aa3627a13cef7d5'
mick@Gazofonias ~/Workspace/issue-1654 $ ls dejagnu/
AUTHORS  COPYING  ChangeLog  ChangeLog-1992  INSTALL  MAINTAINERS  Makefile.am  Makefile.in  NEWS  README  TODO  aclocal.m4  autogen.sh  baseboards  commands  compile  config  config.guess  config.sub  configure  configure.ac  contrib  dejagnu  dejagnu.h  depcomp  doc  install-sh  lib  mdate-sh  missing  runtest  runtest.exp  stub-loader.c  testglue.c  testsuite

As you see we should also add correct branch values for uclibc-ng, qemu, newlib, musl, and glibc, and in general improve this. We have more options when we clone through Makefile or gitnub actions, we can do more things there, even periodically update dejagnu/spike/pk automatically so that we don't risk being left behind again. I'll try to come back to this when I get some time (probably after Jan. 25), and also deal with the night release script (so that it uses the submodule cache) etc.

@TommyMurphyTM1234
Copy link
Collaborator

Thanks a lot @mickflemm - that's very helpful. 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants