diff --git a/CHANGELOG.md b/CHANGELOG.md index a796ecaf..0b39331a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,6 +13,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). - The `--exclude` flag for the `pit util package` command, to exclude certain packages when using the `--all` flag. - New and improved documentation. - The notice fields in the metadata to show a message before or after an installation. +- The local metadata storage, which stores the metadata of installed packages. ### Changes - The build tests are now turned off by default and can be turned on with `--execute-build-test` (`--skip-build-test` is removed). diff --git a/docs/README.md b/docs/README.md index f833da45..eaf819d0 100644 --- a/docs/README.md +++ b/docs/README.md @@ -4,11 +4,14 @@ The docs folder contains all documentation of Packit. ## Available files -| File | Short description | -| --------------------------------------------- | ------------------------------------------------------------------------------ | -| [Repository metadata](./metadata.md) | Describes the structure of metadata repositories and all available fields. | -| [Metadata checks](./meta-checks.md) | Describes how the metadata checks work. | -| [Tests](./tests.md) | Describes how Packit's tests work and how to run them. | -| [Structure and Configuration](./structure.md) | Describes Packit's file structure and how to configure Packit. | -| [Verifier](./verifier.md) | Describes how to use the verifier and repairer, and its structure in the code. | -| [Commands](./commands) | Contains explanations and examples of all Packit commands. | +| File | Short description | +| ------------------------------------------------- | -------------------------------------------------------------------------- | +| [Repository metadata](./metadata.md) | Describes the structure of metadata repositories and all available fields. | +| [Metadata checks](./meta-checks.md) | Describes how the metadata checks work. | +| [Tests](./tests.md) | Describes how Packit's tests work and how to run them. | +| [Structure and Configuration](./structure.md) | Describes Packit's file structure and how to configure Packit. | +| [Verifier](./verifier.md) | Describes how to use the verifier and repairer, and its structure in the code. | +| [Commands](./commands) | Contains explanations and examples of all Packit commands. | +| [Build Environment](./build-env.md) | Describes the environment that is used for building packages. | +| [Installation Process](./installation-process.md) | Describes the full process that Packit executes when installing a package. | +| [Register](./register.md) | Explains the use of the `Register.toml` file and `metadata` directory. | diff --git a/docs/commands/update.md b/docs/commands/update.md index 5ca032af..179d05dc 100644 --- a/docs/commands/update.md +++ b/docs/commands/update.md @@ -1,7 +1,7 @@ # Update The `update` command has the following command line syntax:
-`pit update [[@] ...] [--new-version ] [--all] [--exclude ...]` +`pit update [[@] ...] [--new-version ] [--all] [--exclude ...] [--refresh-only] [--skip-refresh]` ## Basic update The `update` command updates the specified packages, using the following syntax:
@@ -34,3 +34,9 @@ The `--all` flag can be used to update all latest installed versions to the late ### `--exclude ...` The `--exclude` flag can be used to exclude certain packages when using the `--all` flag. + +### `--refresh-only` +The `--refresh-only` flag can be used to only refresh the local metadata of the specified packages. It cannot be used together with the `--skip-refresh` flag. + +### `--skip-refresh` +The `--skip-refresh` flag can be used to skip the refreshing of the local metadata of the packages. It cannot be used together with the `--refresh-only` flag. diff --git a/docs/register.md b/docs/register.md new file mode 100644 index 00000000..2ed9db3b --- /dev/null +++ b/docs/register.md @@ -0,0 +1,15 @@ +# Register and Local Metadata + +This file explains the `Register.toml` file and the local metadata storage. + +# `Register.toml` +The `Register.toml` file stores all information that Packit needs to keep track of which packages are installed, which dependencies they have and where they were installed from. + +This file stores package information, for example the active version and the symlinked state. + +It also stores package version information, which consists of installation path, dependencies, source metadata repository and other fields that describe the package installation. + +# Local Metadata +The local metadata is stored in the `metadata` directory. This storage consists of the relevant package metadata that is required for all operations on packages that require the metadata. By storing this data locally, we do not rely on a remote repository. If the device is offline or the repository is not available anymore, the packages keep working as before. + +This storage is structured in the same way as the `packages` directory, it contains directories for each package, with directories for each version of the package inside. Inside these version directories, the metadata for that version is stored. Scripts and other files that are needed are downloaded to this directory. The `metadata.toml` file stores all relevant metadata fields of the package. diff --git a/docs/structure.md b/docs/structure.md index 96ce889d..3b7f3ded 100644 --- a/docs/structure.md +++ b/docs/structure.md @@ -5,6 +5,7 @@ This file explains the Packit structure and configuration. ## File Structure The Packit directory, which contains Packit data is called the [prefix](#prefix) directory. This directory contains the following files and directories: - [Register.toml](#registertoml) +- [metadata](#local-metadata) - [packages](#packages) - [bin, gnubin, lib, include and share](#symlinks) - [active](#active-packages) @@ -21,6 +22,9 @@ The prefix directory of Packit contains all data of installed packages.
### Register.toml The `Register.toml` file is located inside the prefix and stores information about all installed packages. This file is managed by Packit and should not be changed directly. +### Local Metadata +The `metadata` directory contains all of the local metadata of the installed packages. This directory is managed by Packit and should not be changed directly. + ### Packages All installed packages will go in `/packages///`. diff --git a/src/cli/commands/info.rs b/src/cli/commands/info.rs index edb608f7..5ce53a25 100644 --- a/src/cli/commands/info.rs +++ b/src/cli/commands/info.rs @@ -3,6 +3,7 @@ use crate::{ cli::display::{standard_print::DisplayJoined, styled::MapStyled}, installer::types::PackageName, platforms::{DEFAULT_CONFIG_DIR, OsVersion, Target}, + register::metadata::LocalMetaHandler, repositories::manager::RepositoryManager, utils::packit_version::{current_packit_version, packit_version_name}, }; @@ -100,11 +101,11 @@ impl HandleCommand for InfoArgs { // Show package version specific information if let Some(package_version) = package_version { let package_id = PackageId::new(package.name.clone(), package_version.clone()); - self.display_package_version_info(&package_id, ®ister, installed_package); + self.display_package_version_info(&package_id, ®ister, &config, installed_package); return; } - self.display_package_info(&package.name, installed_package); + self.display_package_info(&config, &package.name, installed_package); } } @@ -178,7 +179,7 @@ impl InfoArgs { } /// Displays package info. - fn display_package_info(&self, package_name: &PackageName, package: &InstalledPackage) { + fn display_package_info(&self, config: &Config, package_name: &PackageName, package: &InstalledPackage) { // Sort installed versions for display let mut installed_versions: Vec<_> = package.versions.keys().collect(); installed_versions.sort(); @@ -195,29 +196,43 @@ impl InfoArgs { println!(); if self.verbose { + let local_meta_handler = LocalMetaHandler::new(&config.prefix_directory); + let conflicts = local_meta_handler.read_package_conflicts(package).unwrap_or_exit_msg("Error while reading local metadata", 1); print!("Conflicts with: "); - standard_print::print_list_or_none(package.conflicts_with.iter()); + standard_print::print_list_or_none(conflicts.iter()); } } /// Displays the package version info, also checking for the verbose flag for some info. - fn display_package_version_info(&self, package_id: &PackageId, register: &PackageRegister, package: &InstalledPackage) { + fn display_package_version_info( + &self, + package_id: &PackageId, + register: &PackageRegister, + config: &Config, + package: &InstalledPackage, + ) { let package_version = match register.get_package_version(package_id) { Some(package) => package, None => not_found::register_package_version(package_id, register), }; + // Retrieve local metadata of package + let local_meta_handler = LocalMetaHandler::new(&config.prefix_directory).get_package(package_id); + let local_metadata = local_meta_handler.read_metadata().unwrap_or_exit_msg("Error while reading local metadata", 1); + println!("{}", package_id.style()); println!("{}", package.description.italic().cyan()); let mut pair_aligner = PairAligner::new(); pair_aligner.add("Homepage", package.homepage.display()); - pair_aligner.add("License", package_version.license.style()); + pair_aligner.add("License", local_metadata.license.style()); pair_aligner.add("Install path", package_version.install_path.display()); pair_aligner.add("Active", package.active_version == package_id.version); pair_aligner.add("Symlinked", package.symlinked); + pair_aligner.add("Last metadata refresh", package_version.last_metadata_refresh); if self.verbose { + pair_aligner.add("Last metadata change", package_version.last_metadata_change); pair_aligner.add("Metadata repository provider", &package_version.metadata_repository_provider); pair_aligner.add("Metadata repository url", &package_version.metadata_repository_url); } diff --git a/src/cli/commands/init.rs b/src/cli/commands/init.rs index 1dfd6769..7738e2f5 100644 --- a/src/cli/commands/init.rs +++ b/src/cli/commands/init.rs @@ -6,18 +6,19 @@ use std::{ str::FromStr, }; +use chrono::DateTime; use clap::Args; use crate::{ cli::{commands::HandleCommand, display::logging::error}, - config::{Config, EditableConfig}, + config::{Config, EditableConfig, Repository}, installer::{ Symlinker, types::{PackageId, PackageName, Version}, }, platforms::{DEFAULT_CONFIG_DIR, DEFAULT_PREFIX, permissions}, - register::{installed_package_version::InstalledPackageVersion, package_register::PackageRegister}, - repositories::types::Licenses, + register::{installed_package_version::InstalledPackageVersion, metadata::LocalMetaHandler, package_register::PackageRegister}, + repositories::provider, utils::{ constants::{DEFAULT_METADATA_REPOSITORY_PROVIDER, DEFAULT_METADATA_REPOSITORY_URL}, packit_version::packit_version, @@ -108,7 +109,6 @@ impl HandleCommand for InitArgs { let installed_package_version = InstalledPackageVersion { package_id: package_id.clone(), - license: Licenses::Single("GPL-3.0-only".into()), metadata_repository_provider: DEFAULT_METADATA_REPOSITORY_PROVIDER.into(), metadata_repository_url: DEFAULT_METADATA_REPOSITORY_URL.into(), prebuilds_repository_url: None, @@ -117,6 +117,8 @@ impl HandleCommand for InitArgs { dependents: HashSet::new(), install_path: packit_package_path, revisions: Vec::new(), + last_metadata_refresh: DateTime::default(), // Initialize to UNIX epoch + last_metadata_change: DateTime::default(), // Initialize to UNIX epoch }; let active = false; let symlinked = false; @@ -125,14 +127,7 @@ impl HandleCommand for InitArgs { let package_homepage = Some("https://github.com/pack-it/packit".into()); // Add Packit to register - register.add_package_raw( - installed_package_version, - active, - symlinked, - package_description, - package_homepage, - Vec::new(), - ); + register.add_package_raw(installed_package_version, active, symlinked, package_description, package_homepage); // Save register register @@ -150,5 +145,35 @@ impl HandleCommand for InitArgs { "Packit cannot be initialized: error while setting permissions of files in the prefix", 1, ); + + // Get the installed package from the register + let Some(installed_package_version) = register.get_package_version_mut(&package_id) else { + error!(msg: "Packit cannot be initialized: newly created register does not contain the Packit version"); + exit(1); + }; + + // Create the repository provider to fetch Packit metadata + let repository = Repository::new( + &installed_package_version.metadata_repository_url, + &installed_package_version.metadata_repository_provider, + ); + let Some(provider) = provider::create_metadata_provider(&repository) else { + error!(msg: "Packit cannot be initialized: cannot fetch Packit metadata from repository"); + exit(1); + }; + + // Fetch Packit metadata from the default repository + let updated = LocalMetaHandler::new(&prefix_directory) + .get_package(&package_id) + .refresh(&provider) + .unwrap_or_exit_msg("Packit cannot be initialized: error while retrieving Packit metadata", 1); + + // Update last refresh in the package version + installed_package_version.update_metadata_refresh(updated); + + // Save register + register + .save_to(&PackageRegister::get_path(&prefix_directory)) + .unwrap_or_exit_msg("Packit cannot be initialized: error while saving register", 1); } } diff --git a/src/cli/commands/link.rs b/src/cli/commands/link.rs index c98738a4..c7e86679 100644 --- a/src/cli/commands/link.rs +++ b/src/cli/commands/link.rs @@ -8,18 +8,17 @@ use crate::{ cli::{ commands::HandleCommand, display::{ - logging::{error, warning}, + logging::warning, not_found, standard_print, styled::{MapStyled, Styled}, }, }, - config::{Config, Repository}, - installer::{Symlinker, types::PackageName}, - platforms::Target, - register::{ - installed_package::InstalledPackage, installed_package_version::InstalledPackageVersion, package_register::PackageRegister, + config::Config, + installer::{ + Symlinker, + types::{PackageId, PackageName}, }, - repositories::provider, + register::{installed_package::InstalledPackage, metadata::LocalMetaHandler, package_register::PackageRegister}, utils::unwrap_or_exit::UnwrapOrExit, }; @@ -67,7 +66,7 @@ impl HandleCommand for LinkArgs { .unwrap_or_exit_msg("Unable to retrieve active version of package", 1); // Check if linking is allowed, exit if force is not enabled - if !self.linking_allowed(®ister, package, package_version) { + if !self.linking_allowed(®ister, &config, package, &package_version.package_id) { if !self.force { println!("Try '--force' if you are sure you want to link, note that this can result in issues"); exit(1); @@ -96,8 +95,15 @@ impl HandleCommand for LinkArgs { impl LinkArgs { /// Checks if linking is allowed and shows a message when it is not allowed or cannot be checked. /// Returns true if linking is allowed, false otherwise. - fn linking_allowed(&self, register: &PackageRegister, package: &InstalledPackage, package_version: &InstalledPackageVersion) -> bool { - let conflicts = register.get_conflicting_packages(&self.package_name, &package.conflicts_with); + fn linking_allowed(&self, register: &PackageRegister, config: &Config, package: &InstalledPackage, package_id: &PackageId) -> bool { + // Read and get conflicts + let local_meta_handler = LocalMetaHandler::new(&config.prefix_directory); + let package_conflicts = + local_meta_handler.read_package_conflicts(package).unwrap_or_exit_msg("Error while reading local metadata", 1); + let conflicts = local_meta_handler + .get_conflicting_packages(register, &self.package_name, &package_conflicts) + .unwrap_or_exit_msg("Error while reading local metadata", 1); + if !conflicts.is_empty() { warning!("The package has conflicts with other packages, cancelling linking"); println!("Conflicting packages:"); @@ -105,48 +111,11 @@ impl LinkArgs { return false; } - let repository = Repository::new( - &package_version.metadata_repository_url, - &package_version.metadata_repository_provider, - ); - - let Some(provider) = provider::create_metadata_provider(&repository) else { - error!(msg: "Cannot create provider for repository"); - return false; - }; - - let package_version_meta = match provider.read_package_version(&self.package_name, &package.active_version) { - Ok(package_version_meta) => package_version_meta, - Err(e) => { - error!(e, "Unable to read package metadata for package"); - return false; - }, - }; - - // Skip if the package version metadata defines skip_symlinking - if package_version_meta.skip_symlinking { - warning!("The package metadata defines we should not symlink this package"); - return false; - } - - let target_bounds = match package_version_meta.get_best_target(&Target::current()) { - Ok(target_bounds) => target_bounds, - Err(e) => { - error!(e, "The metadata does not contain the current target"); - return false; - }, - }; - - let target = match package_version_meta.get_target(&target_bounds) { - Ok(target) => target, - Err(e) => { - error!(e, "Cannot get current target from package metadata"); - return false; - }, - }; + let local_meta_handler = LocalMetaHandler::new(&config.prefix_directory).get_package(package_id); + let local_metadata = local_meta_handler.read_metadata().unwrap_or_exit_msg("Unable to read local metadata", 1); - // Skip if the package version target metadata defines skip_symlinking - if let Some(true) = target.skip_symlinking { + // Skip if the local metadata defines skip_symlinking + if local_metadata.skip_symlinking { warning!("The package metadata defines we should not symlink this package"); return false; } diff --git a/src/cli/commands/uninstall.rs b/src/cli/commands/uninstall.rs index 4aec571f..f3cb4aa6 100644 --- a/src/cli/commands/uninstall.rs +++ b/src/cli/commands/uninstall.rs @@ -93,7 +93,7 @@ impl UninstallArgs { for dependent in &missing_dependents { // If none of the given packages covers the dependent throw an error and exit - if !parameter_checks::contains_package_id(&self.packages, &dependent) { + if !parameter_checks::contains_package_id(&self.packages, dependent) { error!( msg: "{} cannot be uninstalled, because it is a dependency of the following packages:", package.style() @@ -119,13 +119,13 @@ impl UninstallArgs { return package.get_versions().iter().flat_map(|p| p.dependents.iter().cloned()).collect(); } - not_found::register_package(&optional_id.name, ®ister) + not_found::register_package(&optional_id.name, register) }; if let Some(package_version) = register.get_package_version(&package_id) { return package_version.dependents.iter().cloned().collect(); } - not_found::register_package_version(&package_id, ®ister) + not_found::register_package_version(&package_id, register) } } diff --git a/src/cli/commands/update.rs b/src/cli/commands/update.rs index 7afb678b..fab64fa4 100644 --- a/src/cli/commands/update.rs +++ b/src/cli/commands/update.rs @@ -14,14 +14,14 @@ use crate::{ }, parameter_checks, }, - config::Config, + config::{Config, Repository}, installer::{ Installer, InstallerOptions, types::{OptionalPackageId, PackageName, Version}, }, platforms::Target, - register::package_register::PackageRegister, - repositories::manager::RepositoryManager, + register::{metadata::LocalMetaHandler, package_register::PackageRegister}, + repositories::{manager::RepositoryManager, provider}, utils::unwrap_or_exit::UnwrapOrExit, }; @@ -46,6 +46,14 @@ pub struct UpdateArgs { /// Exclude packages when using the `--all` flag, specified with ... #[arg(long, requires = "all")] exclude: Vec, + + /// Only refresh the local metadata + #[arg(long, default_value = "false", conflicts_with = "skip_refresh")] + refresh_only: bool, + + /// Skips refreshing of the local metadata + #[arg(long, default_value = "false", conflicts_with = "refresh_only")] + skip_refresh: bool, } impl HandleCommand for UpdateArgs { @@ -55,10 +63,16 @@ impl HandleCommand for UpdateArgs { let register_dir = PackageRegister::get_path(&config.prefix_directory); let mut register = PackageRegister::from(®ister_dir).unwrap_or_exit(1); + // If refresh only is specified, only refresh the specified packages + if self.refresh_only { + self.refresh_metadata(&mut register, &config); + return; + } + let options = InstallerOptions::default(); let installer = Installer::new(&config, &mut register, &manager, options); - // If `--all` is specified use all the updatable pacakges + // If `--all` is specified use all the updatable packages let optional_ids = match self.all { true => &self.get_updatables(installer), false if self.packages.is_empty() => { @@ -81,6 +95,7 @@ impl HandleCommand for UpdateArgs { exit(1); } + // Update all given packages for optional_id in optional_ids { match optional_id.versioned() { Some(package_id) if register.get_package_version(&package_id).is_some() => {}, @@ -127,6 +142,11 @@ impl HandleCommand for UpdateArgs { None => println!("{} is up-to-date!", optional_id.name.style()), } } + + // Refresh metadata of all given packages + if !self.skip_refresh { + self.refresh_metadata(&mut register, &config); + } } } @@ -144,9 +164,10 @@ impl UpdateArgs { } } + // If there are no updatable packages, show a message and return an empty list if filtered_updatables.is_empty() { println!("All packages are up-to-date!"); - exit(0); + return Vec::new(); } println!("The following packages will be updated:"); @@ -161,4 +182,50 @@ impl UpdateArgs { filtered_updatables.into_iter().map(OptionalPackageId::from).collect() } + + /// Refreshes the metadata for the packages given in the command. + fn refresh_metadata(&self, register: &mut PackageRegister, config: &Config) { + // If `--all` is specified use all installed packages + let packages = match self.all { + // Only filter exclude packages when all is specified + true => ®ister.iterate_all().map(|x| x.package_id.clone()).filter(|x| !self.exclude.contains(&x.name)).collect(), + false if self.packages.is_empty() => { + error!(msg: "No packages specified to refresh"); + exit(1); + }, + false => ¶meter_checks::expand_optional_ids(register, config, &self.packages), + }; + + for package_id in packages { + let Some(package_version) = register.get_package_version_mut(package_id) else { + error!(msg: "Expected package version {} to exist, skipping refresh", package_id.style()); + continue; + }; + + // Create repository provider for package + let repository = Repository::new( + &package_version.metadata_repository_url, + &package_version.metadata_repository_provider, + ); + let Some(provider) = provider::create_metadata_provider(&repository) else { + error!(msg: "Cannot create provider for repository"); + continue; + }; + + // Refresh metadata + let local_meta = LocalMetaHandler::new(&config.prefix_directory).get_package(package_id); + let updated_metadata = local_meta.refresh(&provider).unwrap_or_exit_msg(&format!("Cannot refresh metadata of {package_id}"), 1); + package_version.update_metadata_refresh(updated_metadata); + + // Save register to store updated timestamps + register + .save_to(&PackageRegister::get_path(&config.prefix_directory)) + .unwrap_or_exit_msg("Error while saving register", 1); + + // Show message + if updated_metadata { + println!("Updated local metadata of {}", package_id.style()); + } + } + } } diff --git a/src/cli/commands/util/package.rs b/src/cli/commands/util/package.rs index 4e42cd97..4a5c7a39 100644 --- a/src/cli/commands/util/package.rs +++ b/src/cli/commands/util/package.rs @@ -12,12 +12,12 @@ use crate::{ display::{Spinner, logging::error, not_found, styled::Styled}, parameter_checks, }, - config::{Config, Repository}, + config::Config, installer::types::{OptionalPackageId, PackageId}, packager, platforms::Target, - register::package_register::PackageRegister, - repositories::{provider, types::PrebuildsList}, + register::{metadata::LocalMetaHandler, package_register::PackageRegister}, + repositories::types::PrebuildsList, utils::unwrap_or_exit::UnwrapOrExit, }; @@ -86,39 +86,14 @@ impl PackageArgs { None => not_found::register_package_version(package_id, register), }; - // Create metadata provider - let repository = Repository::new( - &package_version.metadata_repository_url, - &package_version.metadata_repository_provider, - ); - let Some(provider) = provider::create_metadata_provider(&repository) else { - error!(msg: "Cannot create provider for {}, skipping packaging", package_id.style()); - return; - }; - - // Request package metadata - let package_meta = match provider.read_package(&package_id.name) { - Ok(package_meta) => package_meta, - Err(e) => { - error!(e, "Cannot read package metadata of {}, skipping packaging", package_id.style()); - return; - }, - }; - - // Request prebuilds list - let prebuilds_list = match provider.read_prebuilds_list(&package_id.name, &package_id.version) { - Ok(Some(prebuilds_list)) => prebuilds_list, - Ok(None) => PrebuildsList::default(package_meta.supported_versions.keys()), - Err(e) => { - error!(e, "Cannot read prebuild list for {}, skipping packaging", package_id.style()); - return; - }, - }; + // Get local metadata to get prebuild information + let local_meta_handler = LocalMetaHandler::new(&config.prefix_directory).get_package(package_id); + let local_metadata = local_meta_handler.read_metadata().unwrap_or_exit_msg("Error while reading local metadata", 1); - // Retrieve `prebuild_id` to use - let Some((prebuild_id, prebuild_meta)) = prebuilds_list.get_best_prebuild(&Target::current()) else { - error!(msg: "Cannot find prebuild to create for {}, skipping packaging", package_id.style()); - return; + // Get prebuild information from local metadata, or use default + let (prebuild_id, prebuild_meta) = match local_metadata.prebuild { + Some(prebuild) => (prebuild.id, prebuild.info), + None => PrebuildsList::default_for_target(&Target::current()), }; // Automatically create the destination directory @@ -129,7 +104,7 @@ impl PackageArgs { let spinner = Spinner::new(spinner_message); spinner.show(); let revisions = package_version.revisions.len() as u64; - packager::package(config, package_id, destination, revisions, prebuild_id, prebuild_meta).unwrap_or_exit(1); + packager::package(config, package_id, destination, revisions, &prebuild_id, &prebuild_meta).unwrap_or_exit(1); spinner.finish(); } } diff --git a/src/installer/error.rs b/src/installer/error.rs index 2c5beea4..35068fff 100644 --- a/src/installer/error.rs +++ b/src/installer/error.rs @@ -11,7 +11,7 @@ use crate::{ unpack::UnpackError, }, platforms::{permissions::error::PermissionError, symlink::SymlinkError}, - register::error::RegisterError, + register::{error::RegisterError, metadata::error::LocalMetadataError}, repositories::error::RepositoryError, utils::{ioerror, tree::TreeError}, }; @@ -103,6 +103,9 @@ pub enum InstallerError { #[error("Error while setting or getting permissions")] PermissionError(#[from] PermissionError), + #[error("Error while using local metadata")] + LocalMetadataError(#[from] LocalMetadataError), + #[error("Error while interacting with filesystem")] IOError(#[from] ioerror::IOError), } diff --git a/src/installer/installer.rs b/src/installer/installer.rs index 751b9502..2775193d 100644 --- a/src/installer/installer.rs +++ b/src/installer/installer.rs @@ -13,22 +13,25 @@ use crate::{ standard_print, styled::{MapStyled, Styled}, }, - config::{Config, Repository}, + config::Config, installer::{ InstallLabel, error::{InstallerError, Result}, install_tree::{InstallMeta, InstallTree, InstallTreeBuilder, InstallType}, options::InstallerOptions, - scripts::{self, ScriptData, ScriptError}, + scripts::{self, SCRIPT_EXTENSION, ScriptData, ScriptError}, symlinker::Symlinker, types::{OptionalPackageId, PackageId, PackageName, Version}, unpack::unpack, }, platforms::{DEFAULT_PREFIX, Target, permissions, symlink}, - register::{installed_package_version::InstalledPackageVersion, package_register::PackageRegister}, + register::{ + installed_package_version::InstalledPackageVersion, + metadata::{LocalMetaHandler, error::LocalMetadataError}, + package_register::PackageRegister, + }, repositories::{ manager::RepositoryManager, - provider, types::{Checksum, PackageTarget}, }, utils::{io, ioerror::IOResultExt, reading::ReadExt}, @@ -240,6 +243,21 @@ impl<'a> Installer<'a> { ); self.register.save_to(&PackageRegister::get_path(&self.config.prefix_directory))?; + let installed_package_version = match self.register.get_package_version_mut(&package_id) { + Some(installed_package_version) => installed_package_version, + None => { + return Err(InstallerError::UnreachableError { + msg: "Package version cannot be found eventhough it was inserted right before".to_string(), + }); + }, + }; + + // Refresh the local metadata for the new package + let local_metadata = LocalMetaHandler::new(&self.config.prefix_directory).get_package(&package_id); + let updated_metadata = local_metadata.refresh(self.repository_manager.get_metadata_provider(&install_meta.repository_id)?)?; + installed_package_version.update_metadata_refresh(updated_metadata); + self.register.save_to(&PackageRegister::get_path(&self.config.prefix_directory))?; + self.execute_postinstall(&package_id, install_meta, &install_directory, &script_args)?; self.determine_active(install_meta, &package_id, target_meta)?; @@ -417,7 +435,9 @@ impl<'a> Installer<'a> { }; // Check if the package has conflicting packages - let conflicts = self.register.get_conflicting_packages(&package_id.name, &install_meta.package_metadata.conflicts_with); + let local_meta_handler = LocalMetaHandler::new(&self.config.prefix_directory); + let conflicts = + local_meta_handler.get_conflicting_packages(self.register, &package_id.name, &install_meta.package_metadata.conflicts_with)?; if !conflicts.is_empty() { warning!("Skipping symlinking because of conflicting packages:"); standard_print::print_list(conflicts.iter().map_styled()); @@ -625,8 +645,8 @@ impl<'a> Installer<'a> { } // Remove entire package directory if there is only one version, otherwise only remove the package version directory - let installed_versions = self.register.get_all_package_versions(&package_id.name); - let directory = match installed_versions.len() { + let installed_versions = self.register.get_all_package_versions(&package_id.name).len(); + let directory = match installed_versions { 1 => self.config.prefix_directory.join("packages").join(&package_id.name), _ => self.config.prefix_directory.join("packages").join(&package_id.name).join(package_id.version.to_string()), }; @@ -640,12 +660,9 @@ impl<'a> Installer<'a> { }, }; - // Load metadata repository - let repository = match installed_package.get_package_version(&package_id.version) { - Some(package_version) => Repository::new( - &package_version.metadata_repository_url, - &package_version.metadata_repository_provider, - ), + // Get installed package version + let installed_package_version = match installed_package.get_package_version(&package_id.version) { + Some(installed_package_version) => installed_package_version, None => { return Err(InstallerError::UnreachableError { msg: "Package version cannot be found eventhough it was found before".to_string(), @@ -654,7 +671,7 @@ impl<'a> Installer<'a> { }; // Run uninstall script - self.run_uninstall_script(&repository, &package_id, &directory)?; + self.run_uninstall_script(installed_package_version)?; // Remove the dependency symlinks if they exist let dependency_directory_path = self.config.prefix_directory.join("dependencies").join(package_id.to_string()); @@ -680,15 +697,21 @@ impl<'a> Installer<'a> { } // Delete the determined directory - if let Some(directory) = directory.to_str() { - debug!("Removing the package directory: {directory}"); - } + debug!("Removing the package directory: {}", directory.display()); fs::remove_dir_all(&directory).err_with_path("remove dirs", &directory)?; // Remove package from the register debug!("Removing {} from the package register", package_id.style()); self.register.remove_package_version(&package_id); + // Remove local metadata + let local_metadata_path = match installed_versions { + 1 => self.config.prefix_directory.join("metadata").join(&package_id.name), + _ => self.config.prefix_directory.join("metadata").join(&package_id.name).join(package_id.version.to_string()), + }; + debug!("Removing the local metadata directory: {}", local_metadata_path.display()); + fs::remove_dir_all(&local_metadata_path).err_with_path("remove dirs", &directory)?; + Ok(vec![package_id]) } @@ -736,12 +759,6 @@ impl<'a> Installer<'a> { // Run uninstall scripts for all versions for package_version in &installed_versions { - // Create repository - let repository = Repository::new( - &package_version.metadata_repository_url, - &package_version.metadata_repository_provider, - ); - // Remove the dependency symlinks let dependency_directory_path = self.config.prefix_directory.join("dependencies").join(package_version.package_id.to_string()); if fs::exists(&dependency_directory_path).err_with_path("check existence of", &dependency_directory_path)? { @@ -749,7 +766,7 @@ impl<'a> Installer<'a> { } // Run uninstall script - self.run_uninstall_script(&repository, &package_version.package_id, &directory)?; + self.run_uninstall_script(package_version)?; } if let Some(directory) = directory.to_str() { @@ -763,57 +780,38 @@ impl<'a> Installer<'a> { debug!("Removing {} from the package register", package_name.style()); self.register.remove_package(package_name); + // Remove local metadata + let local_metadata_path = self.config.prefix_directory.join("metadata").join(package_name); + debug!("Removing the local metadata directory: {}", local_metadata_path.display()); + fs::remove_dir_all(&local_metadata_path).err_with_path("remove dirs", &directory)?; + Ok(uninstalled) } /// Downloads and runs the uninstall script of a given package. /// Could return an `InstallerError`. - fn run_uninstall_script(&self, repository: &Repository, package_id: &PackageId, install_directory: &Path) -> Result<()> { - // Create metadata repository provider for source repository - let provider = match provider::create_metadata_provider(repository) { - Some(provider) => provider, - None => { - warning!("Unable to create repository provider, skipping uninstall script execution. This may cause stray files"); - return Ok(()); - }, - }; + fn run_uninstall_script(&self, installed_package_version: &InstalledPackageVersion) -> Result<()> { + let package_id = &installed_package_version.package_id; - // Load package version from metadata repository - let package_version = match provider.read_package_version(&package_id.name, &package_id.version) { - Ok(package_version) => package_version, - Err(e) => { - warning!( - "Unable to read package version from metadata repository, skipping uninstall script execution. This may cause stray files" - ); - warning!("{e}"); + let local_meta_handler = LocalMetaHandler::new(&self.config.prefix_directory).get_package(package_id); + let local_metadata = local_meta_handler.read_metadata()?; + + // Copy uninstall script to tempfile if it exists + let script_text = match local_meta_handler.read_file(&format!("uninstall.{SCRIPT_EXTENSION}")) { + Ok(script_text) => script_text, + Err(LocalMetadataError::LocalMetadataFileNotFound { .. }) => { + debug!("Skipping uninstall script execution since metadata does not define it"); return Ok(()); }, - }; - - let target_bounds = package_version.get_best_target(&Target::current())?; - - // Check if uninstall script should be used - let target_meta = package_version.get_target(&target_bounds)?; - let use_script = target_meta.use_uninstall.unwrap_or(package_version.use_uninstall.unwrap_or(false)); - if !use_script { - debug!("Skipping uninstall script execution since metadata does not define it"); - return Ok(()); - } - - // Get script path from package version metadata - let script_path = package_version.get_uninstall_script_path(&target_bounds)?; - - // Download uninstall script if it exists - let Some(script_text) = provider.read_file(&package_id.name, &script_path)? else { - return Err(ScriptError::ScriptNotFound(script_path).into()); + Err(e) => return Err(e.into()), }; let script_file = scripts::write_script_to_tempfile(&script_text)?; // Run script - let script_args = package_version.get_script_args(&target_bounds)?; + let script_args = local_metadata.script_args.iter().map(|(k, v)| (k.as_str(), v.as_str())).collect(); let script_data = ScriptData::new( &script_file, - &install_directory, + &installed_package_version.install_path, package_id, self.config, &script_args, diff --git a/src/integrity/error.rs b/src/integrity/error.rs index b184acb9..23d7205c 100644 --- a/src/integrity/error.rs +++ b/src/integrity/error.rs @@ -11,7 +11,7 @@ use crate::{ }, packager::PackagerError, platforms::{permissions::error::PermissionError, symlink::SymlinkError}, - register::error::RegisterError, + register::{error::RegisterError, metadata::error::LocalMetadataError}, repositories::error::RepositoryError, utils::ioerror, }; @@ -64,6 +64,9 @@ pub enum VerifierError { #[error("Cannot perform check, because of an error when executing a script")] ScriptError(#[from] ScriptError), + #[error("Error while using local metadata")] + LocalMetadataError(#[from] LocalMetadataError), + #[error("Cannot perform check or fix, because of an error while interacting with the filesystem")] IOError(#[from] ioerror::IOError), } diff --git a/src/integrity/verifier/package.rs b/src/integrity/verifier/package.rs index fa14ee86..372be797 100644 --- a/src/integrity/verifier/package.rs +++ b/src/integrity/verifier/package.rs @@ -9,13 +9,17 @@ use crate::{ config::{Config, Repository}, installer::{ self, - scripts::{self, ScriptData, ScriptError}, + scripts::{self, SCRIPT_EXTENSION, ScriptData, ScriptError}, types::{Dependency, PackageId, PackageName}, }, integrity::{Issue, error::Result, utils::get_storage_packages}, packager, platforms::Target, - register::{installed_package_version::InstalledPackageVersion, package_register::PackageRegister}, + register::{ + installed_package_version::InstalledPackageVersion, + metadata::{LocalMetaHandler, error::LocalMetadataError}, + package_register::PackageRegister, + }, repositories::{ provider::{self, create_metadata_provider}, types::{Checksum, PackageVersionMeta, PrebuildsList}, @@ -69,45 +73,23 @@ fn check_package_alterations(package_id: &PackageId, register: &PackageRegister, disable_prebuilds: false, }; - // Create providers - let Some(provider) = provider::create_metadata_provider(&repository) else { - warning!("Cannot create metadata provider for {}, skipping check", package_id.style()); - return Ok(false); - }; + let local_meta_handler = LocalMetaHandler::new(&config.prefix_directory).get_package(package_id); + let local_metadata = local_meta_handler.read_metadata()?; + + // Create prebuild provider let Some(prebuild_provider) = provider::create_prebuild_provider(&repository) else { warning!("Cannot create prebuild provider for {}, skipping check", package_id.style()); return Ok(false); }; - // Request package metadata - let package_meta = match provider.read_package(&package_id.name) { - Ok(package_meta) => package_meta, - Err(e) => { - warning!("Cannot read package metadata of {}, skipping check", package_id.style()); - debug!(err: e, "Retrieving package metadata failed"); - return Ok(false); - }, - }; - - // Request prebuilds list - let prebuilds_list = match provider.read_prebuilds_list(&package_id.name, &package_id.version) { - Ok(Some(prebuilds_list)) => prebuilds_list, - Ok(None) => PrebuildsList::default(package_meta.supported_versions.keys()), - Err(e) => { - warning!("Cannot read prebuild list for {}, skipping check", package_id.style()); - debug!(err: e, "Retrieving prebuilds list failed"); - return Ok(false); - }, - }; - - // Retrieve `prebuild_id` to use - let Some((prebuild_id, prebuild_meta)) = prebuilds_list.get_best_prebuild(&Target::current()) else { - warning!("Cannot find prebuild to create for {}, skipping packaging", package_id.style()); - return Ok(false); + // Get prebuild information from local metadata, or use default + let (prebuild_id, prebuild_meta) = match local_metadata.prebuild { + Some(prebuild) => (prebuild.id, prebuild.info), + None => PrebuildsList::default_for_target(&Target::current()), }; let revision = package_version.revisions.len() as u64; - let prebuild_file_meta = match prebuild_provider.get_prebuild_meta(package_id, revision, prebuild_id) { + let prebuild_file_meta = match prebuild_provider.get_prebuild_meta(package_id, revision, &prebuild_id) { Ok(prebuild_file_meta) => prebuild_file_meta, Err(e) => { warning!( @@ -120,7 +102,7 @@ fn check_package_alterations(package_id: &PackageId, register: &PackageRegister, }; let install_directory = config.prefix_directory.join("packages").join(&package_id.name).join(package_id.version.to_string()); - let compressed = packager::compress(&install_directory, prebuild_meta)?; + let compressed = packager::compress(&install_directory, &prebuild_meta)?; let checksum = Checksum::from_bytes(&compressed); Ok(checksum != prebuild_file_meta.checksum) @@ -627,30 +609,18 @@ fn check_package_test(package_id: &PackageId, register: &PackageRegister, config let package_version = register.get_package_version(package_id).expect("Expected package to exist"); - let repository = Repository::new( - &package_version.metadata_repository_url, - &package_version.metadata_repository_provider, - ); + let local_meta_handler = LocalMetaHandler::new(&config.prefix_directory).get_package(package_id); + let local_metadata = local_meta_handler.read_metadata()?; - let provider = match provider::create_metadata_provider(&repository) { - Some(provider) => provider, - None => { - warning!("Cannot create metadata provider for '{package_id}', skipping check"); - return Ok(false); - }, + // Copy test script to tempfile if it exists + let script_text = match local_meta_handler.read_file(&format!("test.{SCRIPT_EXTENSION}")) { + Ok(script_text) => script_text, + Err(LocalMetadataError::LocalMetadataFileNotFound { .. }) => return Ok(false), + Err(e) => return Err(e.into()), }; - - // Get the target - let version_meta = provider.read_package_version(&package_id.name, &package_id.version)?; - let target_bounds = version_meta.get_best_target(&Target::current())?; - let target = version_meta.get_target(&target_bounds)?; - - // Get the test script data - let script_path = version_meta.get_test_script_path(&target_bounds)?; - let script_args = version_meta.get_script_args(&target_bounds)?; - let script_text = provider.read_file(&package_id.name, &script_path)?; - let Some(script_text) = script_text else { return Ok(false) }; let downloaded_script = scripts::write_script_to_tempfile(&script_text)?; + + let script_args = local_metadata.script_args.iter().map(|(k, v)| (k.as_str(), v.as_str())).collect(); let script_data = ScriptData::new( &downloaded_script, &package_version.install_path, @@ -661,25 +631,14 @@ fn check_package_test(package_id: &PackageId, register: &PackageRegister, config ); // Get the external test files - let external_files = version_meta.external_test_files.iter().chain(&target.external_test_files); let mut read_files = Vec::new(); - for file in external_files { - let file_content = provider.read_file_bytes(&package_id.name, file)?; - - match file_content { - Some(content) => read_files.push((file, content)), - None => { - warning!( - "Skipping {} test, because the required files could not be downloaded", - package_id.style() - ); - return Ok(false); - }, - } + for file in &local_metadata.external_test_files { + let file_content = local_meta_handler.read_file_bytes(file)?; + read_files.push((file, file_content)); } // Run the test script and only return true if the test itself failed - match scripts::run_test_script(&script_data, &read_files, &target.test_requirements) { + match scripts::run_test_script(&script_data, &read_files, &local_metadata.test_requirements) { Ok(_) => Ok(false), Err(ScriptError::RequirementNotSatisfied(requirement)) => { warning!( diff --git a/src/register/installed_package.rs b/src/register/installed_package.rs index b673ea91..ea219117 100644 --- a/src/register/installed_package.rs +++ b/src/register/installed_package.rs @@ -3,10 +3,7 @@ use std::collections::HashMap; use serde::{Deserialize, Serialize}; -use crate::{ - installer::types::{PackageName, Version}, - register::installed_package_version::InstalledPackageVersion, -}; +use crate::{installer::types::Version, register::installed_package_version::InstalledPackageVersion}; /// Represents a package that is installed on the system, holding package specific info /// and a mapping from the installed versions to package versions. @@ -21,9 +18,6 @@ pub struct InstalledPackage { #[serde(skip_serializing_if = "Option::is_none")] pub homepage: Option, - - #[serde(default, skip_serializing_if = "Vec::is_empty")] - pub conflicts_with: Vec, } impl InstalledPackage { @@ -33,7 +27,6 @@ impl InstalledPackage { symlinked: bool, package_description: String, package_homepage: Option, - conflicts_with: Vec, ) -> Self { let version = package_version.package_id.version.clone(); Self { @@ -42,7 +35,6 @@ impl InstalledPackage { active_version: version, description: package_description, homepage: package_homepage, - conflicts_with, } } diff --git a/src/register/installed_package_version.rs b/src/register/installed_package_version.rs index 500fa6b4..aac2fe78 100644 --- a/src/register/installed_package_version.rs +++ b/src/register/installed_package_version.rs @@ -1,9 +1,10 @@ // SPDX-License-Identifier: GPL-3.0-only use std::{collections::HashSet, path::PathBuf}; +use chrono::{DateTime, Utc}; use serde::{Deserialize, Serialize}; -use crate::{config::Repository, installer::types::PackageId, repositories::types::Licenses}; +use crate::{config::Repository, installer::types::PackageId}; /// Represents a specific package version which is installed on the system. #[cfg_attr(test, derive(PartialEq))] @@ -11,9 +12,6 @@ use crate::{config::Repository, installer::types::PackageId, repositories::types pub struct InstalledPackageVersion { pub package_id: PackageId, - #[serde(skip_serializing_if = "Licenses::is_unknown", default)] - pub license: Licenses, - #[serde(default = "Repository::default_repository_provider")] #[serde(skip_serializing_if = "is_repository_provider_default")] pub metadata_repository_provider: String, @@ -37,8 +35,27 @@ pub struct InstalledPackageVersion { #[serde(default)] pub revisions: Vec, + + // The default on the `last_metadata_refresh` and `last_metadata_change` is required to ensure backwards compatibility + #[serde(default)] + pub last_metadata_refresh: DateTime, + + #[serde(default)] + pub last_metadata_change: DateTime, } fn is_repository_provider_default(value: &String) -> bool { *value == Repository::default_repository_provider() } + +impl InstalledPackageVersion { + // Updates the `last_metadata_refresh` and the `last_metadata_change` based on the `updated` paramter. + pub fn update_metadata_refresh(&mut self, updated_metadata: bool) { + let now = Utc::now(); + + self.last_metadata_refresh = now; + if updated_metadata { + self.last_metadata_change = now; + } + } +} diff --git a/src/register/metadata/error.rs b/src/register/metadata/error.rs new file mode 100644 index 00000000..c8fc2adb --- /dev/null +++ b/src/register/metadata/error.rs @@ -0,0 +1,33 @@ +// SPDX-License-Identifier: GPL-3.0-only +use std::path::PathBuf; + +use thiserror::Error; + +use crate::{repositories::error::RepositoryError, utils::ioerror}; + +#[derive(Error, Debug)] +pub enum LocalMetadataError { + #[error("Cannot find metadata file '{file_path}' in source repository")] + RepositoryMetadataFileNotFound { + file_path: String, + }, + + #[error("Cannot find local metadata file '{}'", file_path.display())] + LocalMetadataFileNotFound { + file_path: PathBuf, + }, + + #[error("Cannot fetch package metadata from repository")] + RepositoryError(#[from] RepositoryError), + + #[error("Error while interacting with filesystem")] + IOError(#[from] ioerror::IOError), + + #[error("Cannot parse local metadata file")] + ParseError(#[from] toml::de::Error), + + #[error("Cannot serialize local metadata file")] + SerializeError(#[from] toml::ser::Error), +} + +pub type Result = core::result::Result; diff --git a/src/register/metadata/handler.rs b/src/register/metadata/handler.rs new file mode 100644 index 00000000..9304d766 --- /dev/null +++ b/src/register/metadata/handler.rs @@ -0,0 +1,368 @@ +// SPDX-License-Identifier: GPL-3.0-only +use std::{ + cmp::max, + collections::{HashMap, HashSet}, + fs, + path::{Path, PathBuf}, +}; + +use bytes::Bytes; +use serde::{Deserialize, Serialize}; + +use crate::{ + installer::{ + scripts::SCRIPT_EXTENSION, + types::{Dependency, PackageId, PackageName, Version}, + }, + platforms::Target, + register::{ + installed_package::InstalledPackage, + metadata::error::{LocalMetadataError, Result}, + package_register::PackageRegister, + }, + repositories::{ + provider::MetadataProvider, + types::{DeprecationInfo, Licenses, PackageMeta, PackageVersionMeta, PrebuildMeta, PrebuildsList, Requirement, TargetBounds}, + }, + utils::{ioerror::IOResultExt, serialization}, +}; + +const METADATA_FILENAME: &str = "metadata.toml"; + +/// Represents the local metadata. +#[derive(Serialize, Deserialize, Clone, Debug)] +pub struct LocalMetadata { + pub required_packit_version: Option, + + #[serde(default, skip_serializing_if = "Licenses::is_unknown")] + pub license: Licenses, + + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub dependencies: Vec, + + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub test_requirements: Vec, + + #[serde(default, skip_serializing_if = "HashSet::is_empty")] + #[serde(serialize_with = "serialization::serialize_set_sorted")] + pub external_test_files: HashSet, + + #[serde(default, skip_serializing_if = "HashMap::is_empty")] + #[serde(serialize_with = "serialization::serialize_map_sorted")] + pub script_args: HashMap, + + pub deprecation: Option, + pub skip_symlinking: bool, + + #[serde(default, skip_serializing_if = "HashSet::is_empty")] + #[serde(serialize_with = "serialization::serialize_set_sorted")] + pub conflicts_with: HashSet, + + pub prebuild: Option, +} + +/// Represents a prebuild in local metadata. +#[derive(Serialize, Deserialize, Clone, Debug)] +pub struct LocalPrebuildMetadata { + pub id: String, + + #[serde(flatten)] + pub info: PrebuildMeta, +} + +/// Handler which handles the reading and refreshing of local metadata. +pub struct LocalMetaHandler<'a> { + prefix_dir: &'a Path, +} + +/// Handler which handles the reading and refreshing of local metadata for a specific package. +pub struct LocalMetaPackageHandler<'a> { + prefix_dir: &'a Path, + package_id: &'a PackageId, +} + +impl<'a> LocalMetaHandler<'a> { + /// Creates a new `LocalMetaHandler`. + pub fn new(prefix_dir: &'a Path) -> Self { + Self { prefix_dir } + } + + /// Gets the local metadata handler for the given package. + pub fn get_package(&self, package_id: &'a PackageId) -> LocalMetaPackageHandler<'a> { + LocalMetaPackageHandler { + prefix_dir: self.prefix_dir, + package_id, + } + } + + /// Gets the conflicts of the given package. + /// Returns a list of conflicts defined in the metadata, which can contain packages that are not installed. + pub fn read_package_conflicts(&self, package: &InstalledPackage) -> Result> { + let mut conflicts = HashSet::new(); + + for version in package.versions.values() { + let metadata = self.get_package(&version.package_id).read_metadata()?; + + for conflict in metadata.conflicts_with { + conflicts.insert(conflict); + } + } + + Ok(conflicts) + } + + /// Checks if a package has conflicts with any package that is installed. + /// Returns a list of the names of all conflicting packages. + pub fn get_conflicting_packages( + &self, + register: &PackageRegister, + package_name: &PackageName, + package_conflicts: &HashSet, + ) -> Result> { + let mut conflicting_packages = HashSet::new(); + + for (name, package) in register.iterate_packages() { + if !package.symlinked || name == package_name { + continue; + } + + // Check if the package specifies this package as conflict + if package_conflicts.contains(name) { + conflicting_packages.insert(name.clone()); + } + + // Check if this package specifies the package as conflict + let conflicts = self.read_package_conflicts(package)?; + if conflicts.contains(package_name) { + conflicting_packages.insert(name.clone()); + } + } + + Ok(conflicting_packages) + } +} + +impl<'a> LocalMetaPackageHandler<'a> { + /// Gets the base path of the local metadata storage for the current package. + fn get_base_path(&self) -> PathBuf { + self.prefix_dir.join("metadata").join(&self.package_id.name).join(self.package_id.version.to_string()) + } + + /// Reads the local metadata file from the storage of the given package. + /// Returns the `LocalMetadata` parsed from the storage. + pub fn read_metadata(&self) -> Result { + let path = self.get_base_path().join(METADATA_FILENAME); + if !path.exists() { + return Err(LocalMetadataError::LocalMetadataFileNotFound { file_path: path }); + } + + let content = fs::read_to_string(&path).err_with_path("read", &path)?; + Ok(toml::de::from_str(&content)?) + } + + /// Reads the specified local metadata file from the storage of the given package. + /// Returns the file as string. + pub fn read_file(&self, file: &str) -> Result { + let path = self.get_base_path().join(file); + if !path.exists() { + return Err(LocalMetadataError::LocalMetadataFileNotFound { file_path: path }); + } + + let content = fs::read_to_string(&path).err_with_path("read", &path)?; + Ok(content) + } + + /// Reads the specified local metadata file from the storage of the given package. + /// Returns the file as bytes. + pub fn read_file_bytes(&self, file: &str) -> Result { + let path = self.get_base_path().join(file); + if !path.exists() { + return Err(LocalMetadataError::LocalMetadataFileNotFound { file_path: path }); + } + + let content = fs::read(&path).err_with_path("read", &path)?; + Ok(content.into()) + } + + /// Refreshes the local metadata of the given package. + /// Returns true if the metadata was changed, false otherwise. + #[expect(clippy::borrowed_box)] + pub fn refresh(&self, provider: &Box) -> Result { + let metadata_dir = self.get_base_path(); + + let package_meta = provider.read_package(&self.package_id.name)?; + let package_version_meta = provider.read_package_version(&self.package_id.name, &self.package_id.version)?; + let target_bounds = package_version_meta.get_best_target(&Target::current())?; + let target_meta = package_version_meta.get_target(&target_bounds)?; + let prebuilds_list = provider.read_prebuilds_list(&self.package_id.name, &self.package_id.version)?; + + let local_metadata = self.create_local_metadata(&package_meta, &package_version_meta, &target_bounds, prebuilds_list)?; + let local_meta_str = toml::ser::to_string(&local_metadata)?; + + let mut updated = false; + + // Create metadata dir if it does not exist + if !metadata_dir.exists() { + fs::create_dir_all(&metadata_dir).err_with_path("create dirs", &metadata_dir)?; + updated = true; + } + + // Collect a list of all files in the metadata directory before refreshing + let mut before_files = Vec::new(); + for entry in fs::read_dir(&metadata_dir).err_with_path("read", &metadata_dir)? { + let entry = entry.err_with_path("iterate", &metadata_dir)?; + before_files.push(entry.path()); + } + let mut after_files = Vec::new(); + + // If the metadata is updated, write the metadata to the file + let local_meta_destination = metadata_dir.join(METADATA_FILENAME); + let local_meta_bytes = local_meta_str.into(); + if self.write_file_if_changed(&before_files, &mut after_files, local_meta_destination, Some(local_meta_bytes))? { + updated = true; + } + + // Download external test files + let external_test_files = package_version_meta.get_external_test_files(&target_bounds)?; + for external_file in external_test_files { + // Flatten external file directory names + let normalized_file_name = external_file.replace("/", "-"); + + let destination = metadata_dir.join(normalized_file_name); + let new_file = self.request_file(provider, external_file, true)?; + if self.write_file_if_changed(&before_files, &mut after_files, destination, new_file)? { + updated = true; + } + } + + // Download test script + let test_script_path = package_version_meta.get_test_script_path(&target_bounds)?; + let test_script_destination = metadata_dir.join(format!("test.{SCRIPT_EXTENSION}")); + let new_file = self.request_file(provider, &test_script_path, false)?; + if self.write_file_if_changed(&before_files, &mut after_files, test_script_destination, new_file)? { + updated = true; + } + + // Download uninstall script + if target_meta.use_uninstall.unwrap_or(package_version_meta.use_uninstall.unwrap_or(false)) { + let uninstall_script_path = package_version_meta.get_uninstall_script_path(&target_bounds)?; + let uninstall_script_destination = metadata_dir.join(format!("uninstall.{SCRIPT_EXTENSION}")); + let new_file = self.request_file(provider, &uninstall_script_path, true)?; + if self.write_file_if_changed(&before_files, &mut after_files, uninstall_script_destination, new_file)? { + updated = true; + } + } + + // Remove files that are not needed anymore + let removed_files = before_files.iter().filter(|x| !after_files.contains(x)); + for removed_file in removed_files { + fs::remove_file(removed_file).err_with_path("remove", removed_file)?; + updated = true; + } + + Ok(updated) + } + + /// Creates the local metadata from the given package, version and target metadata. + /// Returns the created `LocalMetadata`. + fn create_local_metadata( + &self, + package_meta: &PackageMeta, + package_version_meta: &PackageVersionMeta, + target_bounds: &TargetBounds, + prebuilds_list: Option, + ) -> Result { + let target_meta = package_version_meta.get_target(target_bounds)?; + + let required_packit_version = max(&package_meta.required_packit_version, &package_version_meta.required_packit_version); + let dependencies = package_version_meta.dependencies.iter().chain(target_meta.dependencies.iter()).cloned().collect(); + + // Normalize external test files to flatten directories into the name + let external_test_files = package_version_meta + .get_external_test_files(target_bounds)? + .iter() + .map(ToString::to_string) + .map(|x| x.replace("/", "-")) + .collect(); + + let script_args = + package_version_meta.get_script_args(target_bounds)?.iter().map(|(k, v)| (k.to_string(), v.to_string())).collect(); + + let deprecation = package_meta.deprecation.as_ref().or(package_version_meta.deprecation.as_ref()).cloned(); + + let mut prebuild = None; + if let Some(prebuilds_list) = prebuilds_list { + if let Some((prebuild_id, prebuild_meta)) = prebuilds_list.get_best_prebuild(&Target::current()) { + prebuild = Some(LocalPrebuildMetadata { + id: prebuild_id.clone(), + info: prebuild_meta.clone(), + }); + } + } + + Ok(LocalMetadata { + required_packit_version: required_packit_version.clone(), + license: package_version_meta.license.clone(), + dependencies, + test_requirements: target_meta.test_requirements.clone(), + external_test_files, + script_args, + deprecation, + skip_symlinking: target_meta.skip_symlinking.unwrap_or(package_version_meta.skip_symlinking), + conflicts_with: package_meta.conflicts_with.clone(), + prebuild, + }) + } + + /// Requests a file from the given provider. + /// If the file cannot be found, it returns an `LocalMetadataError::MetadataFileNotFound`, or None if the file is not required. + /// Returns the bytes of the file if it can be found. + #[expect(clippy::borrowed_box)] + fn request_file(&self, provider: &Box, file_path: &str, required: bool) -> Result> { + let Some(bytes) = provider.read_file_bytes(&self.package_id.name, file_path)? else { + if !required { + return Ok(None); + } + + return Err(LocalMetadataError::RepositoryMetadataFileNotFound { + file_path: file_path.into(), + }); + }; + + Ok(Some(bytes)) + } + + /// Writes a metadata file when it is changed. + /// Compares the new content with the old content of the file. + /// Updates the list `after_files` when a file should be kept. + /// Returns true if the file changed, false otherwise. + fn write_file_if_changed( + &self, + before_files: &[PathBuf], + after_files: &mut Vec, + destination: PathBuf, + new_content: Option, + ) -> Result { + // Get new content, return when the file is not available + // True is returned when the file was present before, false if the file never existed + let Some(new_content) = new_content else { + return Ok(before_files.contains(&destination)); + }; + + // If the file already existed, check content equality + if before_files.contains(&destination) { + let old_content = fs::read(&destination).err_with_path("read", &destination)?; + + // If the file did not change, skip writing and store it in `after_files` + if new_content == old_content { + after_files.push(destination); + return Ok(false); + } + } + + // Write new file data + fs::write(&destination, new_content).err_with_path("write", &destination)?; + after_files.push(destination); + Ok(true) + } +} diff --git a/src/register/metadata/mod.rs b/src/register/metadata/mod.rs new file mode 100644 index 00000000..009f03ae --- /dev/null +++ b/src/register/metadata/mod.rs @@ -0,0 +1,11 @@ +// SPDX-License-Identifier: GPL-3.0-only +pub mod error; +mod handler; + +pub use handler::LocalMetaHandler; + +#[expect(unused_imports)] +pub use handler::LocalMetadata; + +#[expect(unused_imports)] +pub use handler::LocalPrebuildMetadata; diff --git a/src/register/mod.rs b/src/register/mod.rs index 724dd8c1..1e9a5374 100644 --- a/src/register/mod.rs +++ b/src/register/mod.rs @@ -2,4 +2,5 @@ pub mod error; pub mod installed_package; pub mod installed_package_version; +pub mod metadata; pub mod package_register; diff --git a/src/register/package_register.rs b/src/register/package_register.rs index cf218059..13cc9d97 100644 --- a/src/register/package_register.rs +++ b/src/register/package_register.rs @@ -5,6 +5,7 @@ use std::{ path::{Path, PathBuf}, }; +use chrono::DateTime; use serde::{Deserialize, Serialize}; use crate::{ @@ -117,7 +118,6 @@ impl PackageRegister { let installed_package_version = InstalledPackageVersion { package_id: PackageId::new(package.name.clone(), package_version.version.clone()), - license: package_version.license.clone(), metadata_repository_url: source_repository.url.clone(), metadata_repository_provider: source_repository.provider.clone(), prebuilds_repository_url, @@ -126,6 +126,8 @@ impl PackageRegister { dependents: HashSet::new(), install_path: install_path.into(), revisions: package_version.revisions.clone(), + last_metadata_refresh: DateTime::default(), // Initialize to UNIX epoch + last_metadata_change: DateTime::default(), // Initialize to UNIX epoch }; self.add_package_raw( @@ -134,7 +136,6 @@ impl PackageRegister { symlinked, package.description.clone(), package.homepage.clone(), - package.conflicts_with.clone(), ) } @@ -147,7 +148,6 @@ impl PackageRegister { symlinked: bool, package_description: String, package_homepage: Option, - conflicts_with: Vec, ) { // Add the package as a dependent in its dependencies for dependency in &installed_package_version.dependencies { @@ -166,13 +166,7 @@ impl PackageRegister { None => { self.packages.insert( package_name.clone(), - InstalledPackage::new( - installed_package_version, - symlinked, - package_description, - package_homepage, - conflicts_with, - ), + InstalledPackage::new(installed_package_version, symlinked, package_description, package_homepage), ); }, }; @@ -312,30 +306,6 @@ impl PackageRegister { latest } - /// Checks if a package has conflicts with any package that is installed. - /// Returns a list of the names of all conflicting packages. - pub fn get_conflicting_packages(&self, package_name: &PackageName, package_conflicts: &[PackageName]) -> Vec<&PackageName> { - let mut conflicts = Vec::new(); - - for (name, package) in &self.packages { - if !package.symlinked || name == package_name { - continue; - } - - // Check if the package specifies this package as conflict - if package_conflicts.contains(name) { - conflicts.push(name); - } - - // Check if this package specifies the package as conflict - if package.conflicts_with.contains(package_name) { - conflicts.push(name); - } - } - - conflicts - } - /// Returns an iterator, which iterates over all nested installed package version values. pub fn iterate_all(&self) -> impl Iterator { self.packages.values().flat_map(|p| p.versions.values()) @@ -379,7 +349,6 @@ pub mod tests { ) -> InstalledPackageVersion { InstalledPackageVersion { package_id, - license: Licenses::Unknown, metadata_repository_provider: "-".to_string(), metadata_repository_url: "-".to_string(), prebuilds_repository_url: None, @@ -388,6 +357,8 @@ pub mod tests { dependents, install_path: "-".into(), revisions: Vec::new(), + last_metadata_refresh: DateTime::default(), + last_metadata_change: DateTime::default(), } } @@ -405,7 +376,6 @@ pub mod tests { active_version, description: "-".to_string(), homepage: None, - conflicts_with: Vec::new(), } } @@ -483,7 +453,7 @@ pub mod tests { homepage: None, versions: vec![package_id.version.clone()], required_packit_version: None, - conflicts_with: Vec::new(), + conflicts_with: HashSet::new(), supported_versions: HashMap::from([(current_target_bounds, version_intervals)]), deprecation: None, } @@ -680,70 +650,4 @@ pub mod tests { assert!(package_ids.contains(&create_package_id("F@5"))); assert!(package_ids.contains(&create_package_id("F@6"))); } - - #[test] - fn conflicts() { - let mut register = create_register(); - - let conflicting = register.get_conflicting_packages(&create_package_name("F"), &[create_package_name("E")]); - assert_eq!(conflicting, [&create_package_name("E")]); - - let package_id = create_package_id("new_package@2.90"); - let mut package_meta = create_package_meta(&package_id); - package_meta.conflicts_with.push(create_package_name("E")); - let package_version_meta = create_package_version_meta(&package_id); - let dependency_ids = HashSet::new(); - - // Add the package - register.add_package( - &package_meta, - &package_version_meta, - dependency_ids, - &Repository::new("-", "-"), - &PathBuf::from("-"), - true, - false, - false, - ); - - let conflicting = register.get_conflicting_packages(&create_package_name("E"), &[]); - assert_eq!(conflicting, [&create_package_name("new_package")]); - - let conflicting = register.get_conflicting_packages(&create_package_name("E"), &[create_package_name("E")]); - assert_eq!(conflicting, [&create_package_name("new_package")]); - } - - #[test] - fn conflicts_no_symlink() { - let mut register = create_register(); - - let package_id = create_package_id("new_package@2.90"); - let mut package_meta = create_package_meta(&package_id); - package_meta.conflicts_with.push(create_package_name("E")); - let package_version_meta = create_package_version_meta(&package_id); - let dependency_ids = HashSet::new(); - - // Add the package - register.add_package( - &package_meta, - &package_version_meta, - dependency_ids, - &Repository::new("-", "-"), - &PathBuf::from("-"), - false, - false, - false, - ); - - let conflicting = register.get_conflicting_packages(&create_package_name("E"), &[]); - assert_eq!(conflicting, &[] as &[&PackageName]); - } - - #[test] - fn no_conflicts() { - let register = create_register(); - - let conflicting: Vec<&PackageName> = register.get_conflicting_packages(&create_package_name("F"), &[]); - assert_eq!(conflicting, &[] as &[&PackageName]); - } } diff --git a/src/repositories/manager.rs b/src/repositories/manager.rs index ec3b8745..79a81aa9 100644 --- a/src/repositories/manager.rs +++ b/src/repositories/manager.rs @@ -495,7 +495,7 @@ impl<'a> RepositoryManager<'a> { /// A helper method to get the metadata provider. /// Returns a `RepositoryNotFoundError` if no repository with the given `repository_id` can be found. #[expect(clippy::borrowed_box)] - fn get_metadata_provider(&self, repository_id: &str) -> Result<&Box> { + pub fn get_metadata_provider(&self, repository_id: &str) -> Result<&Box> { // Check if repository is unsupported if self.unsupported_repositories.contains_key(repository_id) { return Err(RepositoryError::RepositoryNotSupported { diff --git a/src/repositories/types/common.rs b/src/repositories/types/common.rs index 4797733a..2370b36f 100644 --- a/src/repositories/types/common.rs +++ b/src/repositories/types/common.rs @@ -71,7 +71,7 @@ pub struct FileSize(pub u32); pub struct Date(chrono::NaiveDate); /// Represents information about a package deprecation and disabling. -#[derive(Serialize, Deserialize, Debug)] +#[derive(Serialize, Deserialize, Clone, Debug)] pub struct DeprecationInfo { pub deprecated_from: Date, diff --git a/src/repositories/types/package.rs b/src/repositories/types/package.rs index bb6670e1..32b63c58 100644 --- a/src/repositories/types/package.rs +++ b/src/repositories/types/package.rs @@ -1,5 +1,5 @@ // SPDX-License-Identifier: GPL-3.0-only -use std::collections::HashMap; +use std::collections::{HashMap, HashSet}; use serde::{Deserialize, Serialize}; @@ -21,8 +21,8 @@ pub struct PackageMeta { pub versions: Vec, pub required_packit_version: Option, - #[serde(default, skip_serializing_if = "Vec::is_empty")] - pub conflicts_with: Vec, + #[serde(default, skip_serializing_if = "HashSet::is_empty")] + pub conflicts_with: HashSet, pub supported_versions: HashMap, #[serde(skip_serializing_if = "Option::is_none")] diff --git a/src/repositories/types/prebuilds.rs b/src/repositories/types/prebuilds.rs index d53ce586..03432b69 100644 --- a/src/repositories/types/prebuilds.rs +++ b/src/repositories/types/prebuilds.rs @@ -28,7 +28,7 @@ pub struct PrebuildsList { } /// Represents the information about a prebuild in the `prebuilds.toml` file. -#[derive(Serialize, Deserialize, Debug)] +#[derive(Serialize, Deserialize, Clone, Debug)] pub struct PrebuildMeta { targets: Vec, @@ -95,4 +95,20 @@ impl PrebuildsList { Self { prebuilds } } + + /// Creates the default prebuild id and meta pair for the given target. + pub fn default_for_target(target: &Target) -> (String, PrebuildMeta) { + let target_bounds = TargetBounds { + name: TargetName::Architecture(target.architecture.clone()), + addition: None, + version_intervals: VersionIntervals::default(), + }; + + let prebuild_meta = PrebuildMeta { + targets: vec![target_bounds], + exclude_paths: Vec::new(), + }; + + (target.architecture.to_string(), prebuild_meta) + } } diff --git a/src/utils/mod.rs b/src/utils/mod.rs index 174a3785..0e143511 100644 --- a/src/utils/mod.rs +++ b/src/utils/mod.rs @@ -8,5 +8,6 @@ pub mod packit_version; pub mod patches; pub mod reading; pub mod requests; +pub mod serialization; pub mod tree; pub mod unwrap_or_exit; diff --git a/src/utils/serialization.rs b/src/utils/serialization.rs new file mode 100644 index 00000000..d87176c2 --- /dev/null +++ b/src/utils/serialization.rs @@ -0,0 +1,35 @@ +// SPDX-License-Identifier: GPL-3.0-only +use std::collections::{HashMap, HashSet}; + +use serde::{Serialize, ser::SerializeMap}; + +/// Serializes a `HashMap` and ensures a sorted result. +pub fn serialize_map_sorted(map: &HashMap, serializer: S) -> Result +where + S: serde::Serializer, + K: Serialize + Ord, + V: Serialize, +{ + let mut entries: Vec<_> = map.iter().collect(); + entries.sort_by_key(|(x, _)| *x); + + let mut map_serializer = serializer.serialize_map(Some(entries.len()))?; + + for (key, value) in entries { + map_serializer.serialize_entry(key, value)?; + } + + map_serializer.end() +} + +/// Serializes a `HashSet` and ensures a sorted result. +pub fn serialize_set_sorted(set: &HashSet, serializer: S) -> Result +where + S: serde::Serializer, + T: Serialize + Ord, +{ + let mut values: Vec<_> = set.iter().collect(); + values.sort(); + + values.serialize(serializer) +}