Add local metadata

CHANGELOG.md

@@ -13,6 +13,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 - The `--exclude` flag for the `pit util package` command, to exclude certain packages when using the `--all` flag.
 - New and improved documentation.
 - The notice fields in the metadata to show a message before or after an installation.
+- The local metadata storage, which stores the metadata of installed packages.
 
 ### Changes
 - The build tests are now turned off by default and can be turned on with `--execute-build-test` (`--skip-build-test` is removed).

docs/README.md

@@ -4,11 +4,14 @@ The docs folder contains all documentation of Packit.
 
 ## Available files
 
-| File                                          | Short description                                                              |
-| --------------------------------------------- | ------------------------------------------------------------------------------ |
-| [Repository metadata](./metadata.md)          | Describes the structure of metadata repositories and all available fields.     |
-| [Metadata checks](./meta-checks.md)           | Describes how the metadata checks work.                                        |
-| [Tests](./tests.md)                           | Describes how Packit's tests work and how to run them.                         |
-| [Structure and Configuration](./structure.md) | Describes Packit's file structure and how to configure Packit.                 |
-| [Verifier](./verifier.md)                     | Describes how to use the verifier and repairer, and its structure in the code. |
-| [Commands](./commands)                        | Contains explanations and examples of all Packit commands.                     |
+| File                                              | Short description                                                          |
+| ------------------------------------------------- | -------------------------------------------------------------------------- |
+| [Repository metadata](./metadata.md)              | Describes the structure of metadata repositories and all available fields. |
+| [Metadata checks](./meta-checks.md)               | Describes how the metadata checks work.                                    |
+| [Tests](./tests.md)                               | Describes how Packit's tests work and how to run them.                     |
+| [Structure and Configuration](./structure.md)     | Describes Packit's file structure and how to configure Packit.             |
+| [Verifier](./verifier.md)                         | Describes how to use the verifier and repairer, and its structure in the code. |
+| [Commands](./commands)                            | Contains explanations and examples of all Packit commands.                 |
+| [Build Environment](./build-env.md)               | Describes the environment that is used for building packages.              |
+| [Installation Process](./installation-process.md) | Describes the full process that Packit executes when installing a package. |
+| [Register](./register.md)                         | Explains the use of the `Register.toml` file and `metadata` directory.     |

docs/commands/update.md

@@ -1,7 +1,7 @@
 # Update
 
 The `update` command has the following command line syntax:<br>
-`pit update [<PACKAGE-NAME>[@<VERSION>] ...] [--new-version <NEW-VERSION>] [--all] [--exclude <PACKAGE-NAME> ...]`
+`pit update [<PACKAGE-NAME>[@<VERSION>] ...] [--new-version <NEW-VERSION>] [--all] [--exclude <PACKAGE-NAME> ...] [--refresh-only] [--skip-refresh]`
 
 ## Basic update
 The `update` command updates the specified packages, using the following syntax:<br>
@@ -34,3 +34,9 @@ The `--all` flag can be used to update all latest installed versions to the late
 
 ### `--exclude <PACKAGE-NAME> ...`
 The `--exclude` flag can be used to exclude certain packages when using the `--all` flag.
+
+### `--refresh-only`
+The `--refresh-only` flag can be used to only refresh the local metadata of the specified packages. It cannot be used together with the `--skip-refresh` flag.
+
+### `--skip-refresh`
+The `--skip-refresh` flag can be used to skip the refreshing of the local metadata of the packages. It cannot be used together with the `--refresh-only` flag.

docs/register.md

@@ -0,0 +1,15 @@
+# Register and Local Metadata
+
+This file explains the `Register.toml` file and the local metadata storage.
+
+# `Register.toml`
+The `Register.toml` file stores all information that Packit needs to keep track of which packages are installed, which dependencies they have and where they were installed from.
+
+This file stores package information, for example the active version and the symlinked state.
+
+It also stores package version information, which consists of installation path, dependencies, source metadata repository and other fields that describe the package installation.
+
+# Local Metadata
+The local metadata is stored in the `metadata` directory. This storage consists of the relevant package metadata that is required for all operations on packages that require the metadata. By storing this data locally, we do not rely on a remote repository. If the device is offline or the repository is not available anymore, the packages keep working as before.
+
+This storage is structured in the same way as the `packages` directory, it contains directories for each package, with directories for each version of the package inside. Inside these version directories, the metadata for that version is stored. Scripts and other files that are needed are downloaded to this directory. The `metadata.toml` file stores all relevant metadata fields of the package.

docs/structure.md

@@ -5,6 +5,7 @@ This file explains the Packit structure and configuration.
 ## File Structure
 The Packit directory, which contains Packit data is called the [prefix](#prefix) directory. This directory contains the following files and directories:
 - [Register.toml](#registertoml)
+- [metadata](#local-metadata)
 - [packages](#packages)
 - [bin, gnubin, lib, include and share](#symlinks)
 - [active](#active-packages)
@@ -21,6 +22,9 @@ The prefix directory of Packit contains all data of installed packages.<br>
 ### Register.toml
 The `Register.toml` file is located inside the prefix and stores information about all installed packages. This file is managed by Packit and should not be changed directly.
 
+### Local Metadata
+The `metadata` directory contains all of the local metadata of the installed packages. This directory is managed by Packit and should not be changed directly.
+
 ### Packages
 All installed packages will go in `<prefix>/packages/<PACKAGE-NAME>/<PACKAGE-VERSION>/`.
 

src/cli/commands/info.rs

@@ -3,6 +3,7 @@ use crate::{
     cli::display::{standard_print::DisplayJoined, styled::MapStyled},
     installer::types::PackageName,
     platforms::{DEFAULT_CONFIG_DIR, OsVersion, Target},
+    register::metadata::LocalMetaHandler,
     repositories::manager::RepositoryManager,
     utils::packit_version::{current_packit_version, packit_version_name},
 };
@@ -100,11 +101,11 @@ impl HandleCommand for InfoArgs {
         // Show package version specific information
         if let Some(package_version) = package_version {
             let package_id = PackageId::new(package.name.clone(), package_version.clone());
-            self.display_package_version_info(&package_id, &register, installed_package);
+            self.display_package_version_info(&package_id, &register, &config, installed_package);
             return;
         }
 
-        self.display_package_info(&package.name, installed_package);
+        self.display_package_info(&config, &package.name, installed_package);
     }
 }
 
@@ -178,7 +179,7 @@ impl InfoArgs {
     }
 
     /// Displays package info.
-    fn display_package_info(&self, package_name: &PackageName, package: &InstalledPackage) {
+    fn display_package_info(&self, config: &Config, package_name: &PackageName, package: &InstalledPackage) {
         // Sort installed versions for display
         let mut installed_versions: Vec<_> = package.versions.keys().collect();
         installed_versions.sort();
@@ -195,29 +196,43 @@ impl InfoArgs {
         println!();
 
         if self.verbose {
+            let local_meta_handler = LocalMetaHandler::new(&config.prefix_directory);
+            let conflicts = local_meta_handler.read_package_conflicts(package).unwrap_or_exit_msg("Error while reading local metadata", 1);
             print!("Conflicts with: ");
-            standard_print::print_list_or_none(package.conflicts_with.iter());
+            standard_print::print_list_or_none(conflicts.iter());
         }
     }
 
     /// Displays the package version info, also checking for the verbose flag for some info.
-    fn display_package_version_info(&self, package_id: &PackageId, register: &PackageRegister, package: &InstalledPackage) {
+    fn display_package_version_info(
+        &self,
+        package_id: &PackageId,
+        register: &PackageRegister,
+        config: &Config,
+        package: &InstalledPackage,
+    ) {
         let package_version = match register.get_package_version(package_id) {
             Some(package) => package,
             None => not_found::register_package_version(package_id, register),
         };
 
+        // Retrieve local metadata of package
+        let local_meta_handler = LocalMetaHandler::new(&config.prefix_directory).get_package(package_id);
+        let local_metadata = local_meta_handler.read_metadata().unwrap_or_exit_msg("Error while reading local metadata", 1);
+
         println!("{}", package_id.style());
         println!("{}", package.description.italic().cyan());
 
         let mut pair_aligner = PairAligner::new();
         pair_aligner.add("Homepage", package.homepage.display());
-        pair_aligner.add("License", package_version.license.style());
+        pair_aligner.add("License", local_metadata.license.style());
         pair_aligner.add("Install path", package_version.install_path.display());
         pair_aligner.add("Active", package.active_version == package_id.version);
         pair_aligner.add("Symlinked", package.symlinked);
+        pair_aligner.add("Last metadata refresh", package_version.last_metadata_refresh);
 
         if self.verbose {
+            pair_aligner.add("Last metadata change", package_version.last_metadata_change);
             pair_aligner.add("Metadata repository provider", &package_version.metadata_repository_provider);
             pair_aligner.add("Metadata repository url", &package_version.metadata_repository_url);
         }

src/cli/commands/init.rs

@@ -6,18 +6,19 @@ use std::{
     str::FromStr,
 };
 
+use chrono::DateTime;
 use clap::Args;
 
 use crate::{
     cli::{commands::HandleCommand, display::logging::error},
-    config::{Config, EditableConfig},
+    config::{Config, EditableConfig, Repository},
     installer::{
         Symlinker,
         types::{PackageId, PackageName, Version},
     },
     platforms::{DEFAULT_CONFIG_DIR, DEFAULT_PREFIX, permissions},
-    register::{installed_package_version::InstalledPackageVersion, package_register::PackageRegister},
-    repositories::types::Licenses,
+    register::{installed_package_version::InstalledPackageVersion, metadata::LocalMetaHandler, package_register::PackageRegister},
+    repositories::provider,
     utils::{
         constants::{DEFAULT_METADATA_REPOSITORY_PROVIDER, DEFAULT_METADATA_REPOSITORY_URL},
         packit_version::packit_version,
@@ -108,7 +109,6 @@ impl HandleCommand for InitArgs {
 
         let installed_package_version = InstalledPackageVersion {
             package_id: package_id.clone(),
-            license: Licenses::Single("GPL-3.0-only".into()),
             metadata_repository_provider: DEFAULT_METADATA_REPOSITORY_PROVIDER.into(),
             metadata_repository_url: DEFAULT_METADATA_REPOSITORY_URL.into(),
             prebuilds_repository_url: None,
@@ -117,6 +117,8 @@ impl HandleCommand for InitArgs {
             dependents: HashSet::new(),
             install_path: packit_package_path,
             revisions: Vec::new(),
+            last_metadata_refresh: DateTime::default(), // Initialize to UNIX epoch
+            last_metadata_change: DateTime::default(),  // Initialize to UNIX epoch
         };
         let active = false;
         let symlinked = false;
@@ -125,14 +127,7 @@ impl HandleCommand for InitArgs {
         let package_homepage = Some("https://github.com/pack-it/packit".into());
 
         // Add Packit to register
-        register.add_package_raw(
-            installed_package_version,
-            active,
-            symlinked,
-            package_description,
-            package_homepage,
-            Vec::new(),
-        );
+        register.add_package_raw(installed_package_version, active, symlinked, package_description, package_homepage);
 
         // Save register
         register
@@ -150,5 +145,35 @@ impl HandleCommand for InitArgs {
             "Packit cannot be initialized: error while setting permissions of files in the prefix",
             1,
         );
+
+        // Get the installed package from the register
+        let Some(installed_package_version) = register.get_package_version_mut(&package_id) else {
+            error!(msg: "Packit cannot be initialized: newly created register does not contain the Packit version");
+            exit(1);
+        };
+
+        // Create the repository provider to fetch Packit metadata
+        let repository = Repository::new(
+            &installed_package_version.metadata_repository_url,
+            &installed_package_version.metadata_repository_provider,
+        );
+        let Some(provider) = provider::create_metadata_provider(&repository) else {
+            error!(msg: "Packit cannot be initialized: cannot fetch Packit metadata from repository");
+            exit(1);
+        };
+
+        // Fetch Packit metadata from the default repository
+        let updated = LocalMetaHandler::new(&prefix_directory)
+            .get_package(&package_id)
+            .refresh(&provider)
+            .unwrap_or_exit_msg("Packit cannot be initialized: error while retrieving Packit metadata", 1);
+
+        // Update last refresh in the package version
+        installed_package_version.update_metadata_refresh(updated);
+
+        // Save register
+        register
+            .save_to(&PackageRegister::get_path(&prefix_directory))
+            .unwrap_or_exit_msg("Packit cannot be initialized: error while saving register", 1);
     }
 }

src/cli/commands/link.rs

@@ -8,18 +8,17 @@ use crate::{
     cli::{
         commands::HandleCommand,
         display::{
-            logging::{error, warning},
+            logging::warning,
             not_found, standard_print,
             styled::{MapStyled, Styled},
         },
     },
-    config::{Config, Repository},
-    installer::{Symlinker, types::PackageName},
-    platforms::Target,
-    register::{
-        installed_package::InstalledPackage, installed_package_version::InstalledPackageVersion, package_register::PackageRegister,
+    config::Config,
+    installer::{
+        Symlinker,
+        types::{PackageId, PackageName},
     },
-    repositories::provider,
+    register::{installed_package::InstalledPackage, metadata::LocalMetaHandler, package_register::PackageRegister},
     utils::unwrap_or_exit::UnwrapOrExit,
 };
 
@@ -67,7 +66,7 @@ impl HandleCommand for LinkArgs {
             .unwrap_or_exit_msg("Unable to retrieve active version of package", 1);
 
         // Check if linking is allowed, exit if force is not enabled
-        if !self.linking_allowed(&register, package, package_version) {
+        if !self.linking_allowed(&register, &config, package, &package_version.package_id) {
             if !self.force {
                 println!("Try '--force' if you are sure you want to link, note that this can result in issues");
                 exit(1);
@@ -96,57 +95,27 @@ impl HandleCommand for LinkArgs {
 impl LinkArgs {
     /// Checks if linking is allowed and shows a message when it is not allowed or cannot be checked.
     /// Returns true if linking is allowed, false otherwise.
-    fn linking_allowed(&self, register: &PackageRegister, package: &InstalledPackage, package_version: &InstalledPackageVersion) -> bool {
-        let conflicts = register.get_conflicting_packages(&self.package_name, &package.conflicts_with);
+    fn linking_allowed(&self, register: &PackageRegister, config: &Config, package: &InstalledPackage, package_id: &PackageId) -> bool {
+        // Read and get conflicts
+        let local_meta_handler = LocalMetaHandler::new(&config.prefix_directory);
+        let package_conflicts =
+            local_meta_handler.read_package_conflicts(package).unwrap_or_exit_msg("Error while reading local metadata", 1);
+        let conflicts = local_meta_handler
+            .get_conflicting_packages(register, &self.package_name, &package_conflicts)
+            .unwrap_or_exit_msg("Error while reading local metadata", 1);
+
         if !conflicts.is_empty() {
             warning!("The package has conflicts with other packages, cancelling linking");
             println!("Conflicting packages:");
             standard_print::print_list(conflicts.iter().map_styled());
             return false;
         }
 
-        let repository = Repository::new(
-            &package_version.metadata_repository_url,
-            &package_version.metadata_repository_provider,
-        );
-
-        let Some(provider) = provider::create_metadata_provider(&repository) else {
-            error!(msg: "Cannot create provider for repository");
-            return false;
-        };
-
-        let package_version_meta = match provider.read_package_version(&self.package_name, &package.active_version) {
-            Ok(package_version_meta) => package_version_meta,
-            Err(e) => {
-                error!(e, "Unable to read package metadata for package");
-                return false;
-            },
-        };
-
-        // Skip if the package version metadata defines skip_symlinking
-        if package_version_meta.skip_symlinking {
-            warning!("The package metadata defines we should not symlink this package");
-            return false;
-        }
-
-        let target_bounds = match package_version_meta.get_best_target(&Target::current()) {
-            Ok(target_bounds) => target_bounds,
-            Err(e) => {
-                error!(e, "The metadata does not contain the current target");
-                return false;
-            },
-        };
-
-        let target = match package_version_meta.get_target(&target_bounds) {
-            Ok(target) => target,
-            Err(e) => {
-                error!(e, "Cannot get current target from package metadata");
-                return false;
-            },
-        };
+        let local_meta_handler = LocalMetaHandler::new(&config.prefix_directory).get_package(package_id);
+        let local_metadata = local_meta_handler.read_metadata().unwrap_or_exit_msg("Unable to read local metadata", 1);
 
-        // Skip if the package version target metadata defines skip_symlinking
-        if let Some(true) = target.skip_symlinking {
+        // Skip if the local metadata defines skip_symlinking
+        if local_metadata.skip_symlinking {
             warning!("The package metadata defines we should not symlink this package");
             return false;
         }