User Tools

Site Tools


64bit-migration

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision
Previous revision
64bit-migration [2021/01/22 09:33] snarg64bit-migration [2023/06/21 12:02] (current) snarg
Line 3: Line 3:
 This is my method - letting portage track every file and package except glibc and ld-linux.so.2 loader. System and all services can remain running. If this is done correctly only 2 reboots will be required. This is my method - letting portage track every file and package except glibc and ld-linux.so.2 loader. System and all services can remain running. If this is done correctly only 2 reboots will be required.
  
 +  * Keep in mind that pre profile 17.1 and post profile 17.1 will be slightly different as /lib and /usr/lib are not symlinks. These differences are noted in the relevant steps.
   * make sure system is completely up to date and has been depcleaned, preserved-rebuilt and revdep-rebuild - to clean up old 32bit libraries else lots of work later   * make sure system is completely up to date and has been depcleaned, preserved-rebuilt and revdep-rebuild - to clean up old 32bit libraries else lots of work later
   * a full up to date backup of the system is a good idea and has saved me in the past.   * a full up to date backup of the system is a good idea and has saved me in the past.
   * make sure you have an *identically* up to date 64bit multilib profile VM to create binary packages on. Ensure CFLAGS is going to be identical, or compatible ("-mtune=generic -O2 -pipe" is safe). Make sure USE flags are the same between the systems as this makes live easier when doing emerge -k binarypackage  on the other side. If changes are required to CFLAGS then emerge -e world.   * make sure you have an *identically* up to date 64bit multilib profile VM to create binary packages on. Ensure CFLAGS is going to be identical, or compatible ("-mtune=generic -O2 -pipe" is safe). Make sure USE flags are the same between the systems as this makes live easier when doing emerge -k binarypackage  on the other side. If changes are required to CFLAGS then emerge -e world.
 +  * You can use a portage tree from git (on both target system and migration VM) using something like:
 +<code>cd /usr
 +git clone https://github.com/gentoo/gentoo.git ./gentoo-git
 +cd gentoo-git
 +git checkout `git rev-list -n 1 --first-parent --before="2020-09-01" master`
 +#note backticks</code>
 +  * keep in mind if using a portage git snapshot like the above, you will need to read this for crossdev, particularly the bit regarding "masked by corruption" (search for that). The crossdev overlay will need thin-manifests = true : https://wiki.gentoo.org/wiki/Crossdev 
 +  * read ahead a few points (or the whole page) when starting a step - several problems may be solved in subsequent notes
 +  * if problems arise with libraries and mismatched ABI binaries, scp them over from the donor VM as needed. subsequent emerge will overwrite them later.
  
 == Steps == == Steps ==
Line 13: Line 22:
   cp /etc/portage/make.conf /migration-backup/   cp /etc/portage/make.conf /migration-backup/
  
-  see previous versions of this page which built a cross compiler for compiling a 64bit kernel. This is not needed if building the kernel and initramfs on the 64bit VM.+ 
 +  build cross compiler. This will be used during emerging the glibc BINARY packageit needs a working gcc-x86_64-pc-linux-gnu compiler (which will be still running as i686 ABI under /usr/i686-pc-linux-gnu/x86_64-pc-linux-gnu) 
 + 
 +<code>emerge crossdev -av</code> 
 + 
 +  * convert /etc/portage/package.* into directories 
 + 
 +<code>crossdev -S -s3 --target x86_64-pc-linux-gnu</code>
  
   * ON THE 64bit VM (which will build ia32 emulation by default):   * ON THE 64bit VM (which will build ia32 emulation by default):
Line 27: Line 43:
   ldconfig   ldconfig
  
-  mkdir /lib32+  mkdir /lib32        (not if donor VM profile is 17.1 and onwards which uses lib and lib64 with no symlink)
   mkdir /lib64   mkdir /lib64
-  mkdir /usr/lib32+  mkdir /usr/lib32    (not if donor VM profile is 17.1 and onwards which uses lib and lib64 with no symlink)
   mkdir /usr/lib64   mkdir /usr/lib64
  
Line 39: Line 55:
  
   cp -ar /usr/x86_64-pc-linux-gnu /migration-backup/usr/   cp -ar /usr/x86_64-pc-linux-gnu /migration-backup/usr/
-  +  cp -ar /usr/i686-pc-linux-gnu /migration-backup/usr/ 
 +  cp -ar /usr/libexec /migration-backup/usr/
   cp -ar /usr/include /migration-backup/usr   cp -ar /usr/include /migration-backup/usr
  
Line 47: Line 64:
 <code># cat /etc/portage/profile/use.mask <code># cat /etc/portage/profile/use.mask
 -abi_x86_32</code> -abi_x86_32</code>
-  * change make.profile symlink to point to a multilib amd64 profile +  * change make.profile symlink to point to a multilib amd64 profile. choose 17.0 or 17.1 based on Donor VM profile. be aware of the difference.
  
-  * on donor pc or VM:+  * on donor pc or VM (take note of wildcards, adjust if required):
  
-  quickpkg glibc gcc zlib ncurses gmp mpfr util-linux acl attr sandbox coreutils procps binutils libcap mpc cracklib flex =db-5* glib pam pambase =python-3.6* bash readline findutils install-xattr bzip2 tar xz-utils make grep patch gawk sed libpcre dev-libs/libffi gettext libxml2+  quickpkg glibc gcc zlib ncurses gmp mpfr util-linux acl attr sandbox coreutils procps binutils libcap libxcrypt mpc cracklib flex =db-5* glib pam pambase =python-3* bash readline findutils install-xattr bzip2 tar xz-utils make grep patch gawk sed libpcre dev-libs/libffi gettext libxml2
  
   * on source VM, scp -r /var/cache/binpkgs to target PC   * on source VM, scp -r /var/cache/binpkgs to target PC
Line 58: Line 75:
   ** make sure busybox is functional. run this from busybox   ** make sure busybox is functional. run this from busybox
   ** have a few shells open, in case you break the one you're working from   ** have a few shells open, in case you break the one you're working from
 +  ** remember.. if the source VM is profile 17.1 then this system will need to be built using 17.1, so think about making a /lib and /usr/lib folder (and no lib32 folders) rather than symlinking.
  
   mv /lib oldlib   mv /lib oldlib
-  ln -s lib64 lib+  ln -s lib64 lib   (not on 17.1 - mkdir lib)
   cd /usr   cd /usr
   mv lib to oldlib   mv lib to oldlib
-  ln -s lib64 /lib+  ln -s lib64 lib   (not on 17.1 - mkdir lib)
  
   * this will get overwritten by emerge at the appropriate time, but lets the 32bit loader continue to work for now   * this will get overwritten by emerge at the appropriate time, but lets the 32bit loader continue to work for now
  
   cp migration-backup/lib/ld-linux.so.2 /lib   cp migration-backup/lib/ld-linux.so.2 /lib
 +  ldconfig
  
-  * /usr/lib symlink creation will break python. The following will not create orphans as the files are in exactly the same location from portage's point of view.+  * /usr/lib symlink creation or directory move will break python. The following will not create orphans as the files are in exactly the same location from portage's point of view. Modify python version(s) as appropriate.
  
   cd /usr/oldlib   cd /usr/oldlib
-  mv python-exec python3.6 libpython* portage gcc ../lib64 +  mv python-exec python3.6 libpython* portage gcc ../lib/ 
-  cp -r gentoo /lib+  cp -r /oldlib/gentoo /lib/
  
-  * use quickpkg on source VM to grab and then PATH="/migration-backup/bin:/migration-backup/usr/bin:/migration-backup/sbin:/migration-backup/usr/sbin:${PATH}" emerge -1kav to deploy initial binary packages donated from prebuilt VM from /usr/portage/packages or /var/cache/binpkgs. Of these, glibc needs to come first, then everything else will come good. In the process of deploying glibc, /lib needs to be a symlink to /lib64 which will break 32bit programs. see ld-linux.so.2 trick above. ldconfig is your friend here with the above 99migration env file.+  * disable collision detection and emerge gcc AND binutils using this method: 
 +<code>LD_PRELOAD="/migration-backup/usr/lib/gcc/i686-pc-linux-gnu/10.3.0/libgcc_s.so.1:/migration-backup/usr/lib/libffi.so.7" PATH="/migration-backup/bin:/migration-backup/usr/bin:/migration-backup/sbin:/migration-backup/usr/sbin:${PATH}  FEATURES="-collision-detect -protect-owned" emerge -1kav sys-devel/gcc</code>
  
-  * backup these: /etc/env.d/99glibc /etc/gai.conf /etc/host.conf /etc/locale.gen /etc/nsswitch.conf /etc/rpc +  * check  /usr/libexec/gcc/x86_64-pc-linux-gnufor any links pointing at i686 versionsremove. 
-  * to get 64bit loader and glibc workingextract glibc tbz2 binary package directly into root of filesystem. subsequent remerge will track the files correctly later. +  * continue to deploy initial binary packages using this method (except with collision detection) donated from prebuilt VM from /usr/portage/packages or /var/cache/binpkgs. Of these, mpc, zlib, glibc needs to come first, then everything else will come good. In the process of deploying glibc, /lib needs to be a symlink to /lib64 which will break 32bit programssee ld-linux.so.2 trick above. ldconfig is your friend here with the above 99migration env file.
-  * restore etc files from above backup+
  
-  * remove crossdev packages (and crossdev) and check gcc sanity+  * 202112 notes: had to remove the following files to get glibc binary package to emerge: 
 +<code>/usr/bin/cal 
 +/bin/date 
 +/usr/bin/env 
 +/usr/bin/free 
 +/bin/ls 
 +/usr/bin/uname 
 +/usr/bin/uptime</code> 
 +These can be restored from these binary packages immediately after getting glibc working: 
 +<code>emerge -1kav util-linux coreutils procps</code> 
 +To emerge coreutils, working mv command is required. libattr.so.1 and libacl.so.1 (symlinks.. copy the actual librarieswere copied over from donor VM lib64 directories and appropriate symlinks made. These belong to packages acl and attr binary packages which can then be emerged to account for those files. 
 +  * If you get into trouble getting 64bit loader working, do this (only if needed): 
 +  ** backup these: /etc/env.d/99glibc /etc/gai.conf /etc/host.conf /etc/locale.gen /etc/nsswitch.conf /etc/rpc 
 +  ** extract glibc tbz2 binary package directly into root of filesystem. subsequent remerge will track the files correctly later. 
 +  ** restore etc files from above backup
  
-  emerge --unmerge cross-x86_64-pc-linux-gnu/* crossdev -av +  * Now is the right time to remove crossdev versions of the toolchain, as they are now conflicts with our native amd64 versions.  Take a look in the repo directory and unmerge any crossdev packages like gcc glibc binutils etc (fixme: put actual commands here)
-  ldconfig+
  
-  * fix 32bit toolchain and python in wrong directory by running:+  * This will be needed for fixing things after removing crossdev versions. Fix 32bit toolchain and python in wrong directory by running:
  
-  PKGDIR=/var/cache/binpkgs emerge -1kav sys-devel/gcc mpfr mpc binutils +<code>PKGDIR=/var/cache/binpkgs emerge -1kav sys-devel/gcc mpfr mpc binutils 
   gcc-config 1   gcc-config 1
   binutils-config 1   binutils-config 1
-  . /etc/profile+  . /etc/profile</code>
  
-Example final binary emerge. Note obvious package version changes required based on source VM: +  * Finally now that theres a working coreutils etc, If you manually extracted glibc in previous step, overwrite it with package copy.
- +
-  PKGDIR=/var/cache/binpkgs USE="fix any differences" emerge -1kav =zlib-1.2.11-r2  =ncurses-6.2-r1  =gmp-6.2.0-r1 =util-linux-2.35.1-r2  =acl-2.2.53  =attr-2.4.48-r3  =sandbox-2.13  =coreutils-8.31-r1  =procps-3.3.15-r1  =libcap-2.26-r2  =cracklib-2.9.7  =flex-2.6.4-r1  =db-5.3.28-r2  =glib-2.62.6  =pam-1.3.1-r2  =pambase-20190402  =python-3.6.10-r2 =bash-5.0_p17  =readline-8.0_p4  =findutils-4.7.0  =install-xattr-0.5  =bzip2-1.0.6-r11  =tar-1.32  =xz-utils-5.2.4-r2  =make-4.2.1-r4  =grep-3.3-r1  =patch-2.7.6-r4  =gawk-5.0.1  =sed-4.7  =libpcre-8.43 =dev-libs/libffi-3.3-r1 =gettext-0.20.1 =libxml2-2.9.9-r3 +
-   +
-Finally now that theres a working coreutils etc, binary glibc package can overwrite the manual tar version we extracted in previous step.+
  
   PKGDIR=/var/cache/binpkgs emerge -1kav glibc   PKGDIR=/var/cache/binpkgs emerge -1kav glibc
Line 103: Line 131:
 Double check /etc/nsswitch.conf and other files backed up above, this may have overwritten them again. Double check /etc/nsswitch.conf and other files backed up above, this may have overwritten them again.
          
-  mv /migration-backup/lib/modules /lib64 +<code>mv /migration-backup/lib/modules /lib64</code>  (or to /lib if 17.1 profile)
  
   * Emerging gcc needs to be done now, but also acts as a sanity check to see what we've missed. Repeat the quickpkg / emerge -1k steps for missing things   * Emerging gcc needs to be done now, but also acts as a sanity check to see what we've missed. Repeat the quickpkg / emerge -1k steps for missing things
  
-  emerge gcc -1av+<code>emerge gcc -1av</code>
  
   * leave /etc/env.d/99migration in place at least until libraries in /lib32 are populated   * leave /etc/env.d/99migration in place at least until libraries in /lib32 are populated
Line 114: Line 141:
   * @system rebuild - adjust USE as needed   * @system rebuild - adjust USE as needed
  
-  FEATURES="-sandbox -usersandbox" USE="-X -gtk -qt5 -introspection -cairo -gpm -consolekit -policykit -net -ipv6 -spell -sqlite -nls -berkdb -gdbm -openssl" emerge -1eav @system+<code>FEATURES="-sandbox -usersandbox" USE="-X -gtk -qt5 -introspection -cairo -gpm -consolekit -policykit -net -ipv6 -spell -sqlite -nls -berkdb -gdbm -openssl" emerge -1eav @system</code>
  
   * work through any problems caused by emerge ordering above. If a dependancy problem proves too hard to solve, cheat by grabbing a binary package from the donor VM prepared earier. The subsequent @world emerge will take over the package later.   * work through any problems caused by emerge ordering above. If a dependancy problem proves too hard to solve, cheat by grabbing a binary package from the donor VM prepared earier. The subsequent @world emerge will take over the package later.
Line 131: Line 158:
   * take out /etc/env.d/99migration   * take out /etc/env.d/99migration
   * remove /migration-backup   * remove /migration-backup
-  * erase /usr/portage/packages (whole dir)+  * erase /usr/portage/packages (whole dir) - modern version is /var/cache/binpkgs
   * remove abi_x86_32 from package.use and profiles/use.mask, then emerge --newuse   * remove abi_x86_32 from package.use and profiles/use.mask, then emerge --newuse
   * clean up /lib.old and /usr/lib.old   * clean up /lib.old and /usr/lib.old
 +  * check for packages that own files in 32bit directories:
 +<code>emerge -1av /lib32 /usr/lib32</code>
  
   * check for orphans in /usr/lib and all the other places. (bin sbin /usr/bin /usr/sbin). For finding orphan files after migration in /lib32, /usr/lib32, /usr/lib64   * check for orphans in /usr/lib and all the other places. (bin sbin /usr/bin /usr/sbin). For finding orphan files after migration in /lib32, /usr/lib32, /usr/lib64
 <code>cd /lib32 ; find  -xdev -type f -exec qfile -o {} + |less</code> <code>cd /lib32 ; find  -xdev -type f -exec qfile -o {} + |less</code>
  
64bit-migration.1611279201.txt.gz · Last modified: 2021/01/22 09:33 by snarg