diff --git a/docs/index.md b/docs/index.md index 00570e5c9ef..f267de7d88e 100644 --- a/docs/index.md +++ b/docs/index.md @@ -52,7 +52,7 @@ pre-requisites section under each of the topics! - [Installing Rucio Daemons](operator/installing_daemons.md) - [Monitoring](operator/monitoring.md) - [Database](operator/database) -- [Configuration parameters](operator/configuration_parameters.md) +- [Configuration parameters](operator/configuration_parameters.mdx) ## Developer Documentation diff --git a/docs/operator/configuration_parameters.md b/docs/operator/configuration_parameters.mdx similarity index 81% rename from docs/operator/configuration_parameters.md rename to docs/operator/configuration_parameters.mdx index e0fb21e2eb1..557c08046b7 100644 --- a/docs/operator/configuration_parameters.md +++ b/docs/operator/configuration_parameters.mdx @@ -1,7 +1,10 @@ +import RSETable from './rse-attributes.mdx' + --- id: configuration_parameters title: Configuration Parameters --- + Description of the configuration parameters split in three different places: Rucio configuration file (`rucio.cfg`), Rucio configuration table and RSE attributes. @@ -527,50 +530,4 @@ and only contains information about the specific RSE's for the Rucio instance. Read more about RSEs [here](started/concepts/rucio_storage_element.md) and how to set them up [here](operator/configuration.md#creating-new-rses). -- **associated_sites**: String. Separated by commas. Used for chaining of subscriptions so that transfers to one RSE will also be mirrored to associated_sites. No default. -- **archive_timeout**: Integer: Only used for transfers with a tape destination. Controls the number of seconds the FTS3 transfer manager will wait for the tape archival of the file to go `FAILED` or `FINISHED`. No default. -- **auto_approve_bytes**: String: Upper limit for the size in bytes of a DID for which rules will be automatically approved. Example: `500GB`. No default. -- **auto_approve_files**: Integer: Upper limit for the number of files covered by a rule which will be automatically approved. No default. -- **available_for_multihop**: Boolean. If True, allow to use this RSE as an intermediate hop in a multi-hop transfer. Default: `False`. -- **block_manual_approval**: Boolean. Disable manual rule approval for this RSE. Default: `False`. -- **bittorrent_tracker_addr**: String. Used to configure the URL of the bittorrent tracker API when using the torrent transfer manager. No Default. -- **checksum_key**: String. Used to specify an alternate RSE attribute to search for supported checksums beyond those with global support (ADLER32, MD5). Default: `supported_checksums`. -- **country**: String. Used for country level granularity of RSE selectors. No default. -- **decommission**: Boolean. Indicates to the RSE Decommissioning Daemon that this RSE is to be decommissioned. Default: `False`. -- **driver_name_rse_attribute**: String. Used to specify alternate drivers when using the Bittorrent transfer manager. Default: `bittorrent_driver`. -- **fts**: String. Specify the REST API URL of the FTS3 transfer manager. No default. -- **greedyDeletion**: Boolean. Allow files without a rule locking them to be deleted by a Reaper Daemon. Default behavior only marks a file for deletion when there is no space on an RSE for a new required file. Default: `False`. -- **group_by_rse_attribute**: String. Control the RSE attribute (such as `country`) which transfer source RSEs will be grouped by when determining an appropriate transfer source. Default: `UNKNOWN`. -- **globus_endpoint_id**: String. Specify the REST API URL of the Globus transfer manager. No default. -- **hop_penalty**: Integer. Usage cost of this RSE as an intermediate in [multihop transfers](operator_transfers/transfers_overview.md). Overrides the global `transfers/hop_penalty` configuration for this particular RSE. - Requires `available_for_multihop` attribute is True on the RSE. No default. -- **is_object_store**: Boolean. Control the auditor daemon's behavior. Instead of dumping all files, list them by date. Default: `False`. -- **istape**: Boolean. Default: `False`. -- **lfn2pfn_algorithm**: String. Name of the algorithm to be used for generating paths to files on the storage. Must be defined in the configured policy package. Default: `default`. -- **mock**: Boolean. Default: `False`. -- **multihop_tombstone_delay**: Integer. Delay before a multihop transfer intermediate rule is to be deleted. Defined in seconds. Default: `7200`. -- **naming_convention**: String. Name of the algorithm in the configured policy package which is to be used to validate DIDs on this RSE. Default: `None`. -- **oidc_support**: Boolean. Specifies that the RSE supports OIDC authentication for FTS3 transfers. Default: `False`. -- **overwrite_when_only_on_disk**: Boolean. On a `TAPE` RSE, controls if a file can be overwritten. A file may only be overwritten if it has not yet been written to the tape backend. Default: `False`. -- **overwrite**: Boolean. Controls if a file can be overwritten on the RSE. Default: `True` for `rse_type: DISK`. `False` for `rse_type: TAPE`. -- **physgroup**: String. Used for grouping of rules by CERN experiments. Default: ` `. -- **qbittorrent_management_address**: String. Used to configure the URL of the bittorrent management API when using the torrent transfer manager. No Default. -- **quota_approvers**: List[string]. List of Rucio users separated by commas. Only used in the permission layer of the policy package, and likely specific to the CERN experiments only. Default: `None`. -- **restricted_read**: Boolean. If True, only allow transfers FROM this RSE if started by an account with admin privileges. Default: `False` -- **restricted_write**: Boolean. If True, only allow transfers TO this RSE if started by an account with admin privileges. Default: `False` -- **rule_approvers**: List[string]. List of Rucio users separated by commas which will be notified by email to approve rules on this RSE. Default: `None`. -- **rule_deleters**: List[string]. List of Rucio users separated by commas. Only used in the permission layer of the policy package, and likely specific to the CERN experiments only. Default: `None`. -- **sign_url**: Optional[str]. Controls if URLs for uploads and transfers are to be cryptographically signed by an external service. No default. -- **simulate_multirange** Integer. Control the number of connections for multirange byte requests on commercial cloud storage. Multirange is not supported on S3. No default. -- **site**: String. Used to determine if downloads/transfers are local or wide area network. No default. -- **skip_upload_stat**: Boolean. Disables the use of a GFAL `stat` when calling `rucio upload` for this RSE. Default: `False`. -- **source_for_total_space**: String. Used to specify where Rucio should obtain storage capacity information from. Default: `storage`. -- **source_for_used_space**: String. Used to specify where Rucio should obtain storage usage information from. Default: `storage`. -- **staging_buffer**: String. Used with `TAPE` RSEs to specify to which RSE a file on tape should be transferred to as a disk buffer. Destination RSE should have `staging_area: True`. No default. -- **staging_required**: Boolean. Duplicates the `rse_type` RSE setting. Specifies that files on this RSE will require staging from high-latency storage. Default `False`. -- **strict_copy**: Boolean. Instructs the transfer manager to ONLY copy the file, disabling all validation checks such as `PROPFIND` and checksumming. Default `False`. -- **s3_url_style**: String. For S3 storage elements [specify](https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html) `path` or `host`. No default. -- **tier**: Datacenter TIER. Integer (from 1 to 4). No default. -- **type**: Values: `{LOCALGROUPDISK, LOCALGROUPTAPE, GROUPDISK, SCRATCHDISK, MOCK, TEST, DATADISK}`. Default: ` `. -- **use_ipv4**: Boolean. Force the transfer manager to use IPv4 for transfers to this RSE. Default `False`. -- **verify_checksum**: Boolean. Control if checksum is to be queried on transfer source and destination to confirm successful transfers. Default: `True` + \ No newline at end of file diff --git a/docs/operator/rse-attributes.mdx b/docs/operator/rse-attributes.mdx new file mode 100644 index 00000000000..f2683e1888d --- /dev/null +++ b/docs/operator/rse-attributes.mdx @@ -0,0 +1,52 @@ +- **archive_timeout: String. Only used for transfers with a tape destination. Controls the number of seconds the FTS3 transfer manager will wait for the tape archival of the file to go `FAILED` or `FINISHED`. Default: *None* (no default). +- **associated_sites: String. Separated by commas. Used for chaining of subscriptions so that transfers to one RSE will also be mirrored to associated_sites. Default: *None* (no default). +- **auto_approve_bytes: String. Upper limit for the size in bytes of a DID for which rules will be automatically approved. Example: `500GB`. Default: *None* (no default). +- **auto_approve_files: String. Upper limit for the number of files covered by a rule which will be automatically approved. Default: *None* (no default). +- **available_for_multihop: Boolean. If ``True``, allow this RSE to be used as an intermediate hop in a multi‑hop transfer. Default: ``False``. +- **block_manual_approval: Boolean. Disable manual rule approval for this RSE. Default: ``False``. +- **bittorrent_tracker_addr: String. Used to configure the URL of the bittorrent tracker API when using the torrent transfer manager. Default: *None* (no default). +- **checksum_key: String. Used to specify an alternate RSE attribute to search for supported checksums beyond those with global support (ADLER32, MD5). Default: ``supported_checksums``. +- **country: String. Used for country‑level granularity of RSE selectors. Default: *None* (no default). +- **decommission: Boolean. Indicates to the RSE Decommissioning Daemon that this RSE is to be decommissioned. Default: ``False``. +- **driver_name_rse_attribute: String. Used to specify alternate drivers when using the Bittorrent transfer manager. Default: ``bittorrent_driver``. +- **fts: String. Specify the REST API URL of the FTS3 transfer manager. Default: *None* (no default). +- **greedyDeletion: Boolean. Allow files without a rule locking them to be deleted by a Reaper daemon. Default behaviour only marks a file for deletion when there is no space on an RSE for a new required file. Default: ``False``. +- **group_by_rse_attribute: String. Control the RSE attribute (such as ``country``) which transfer source RSEs will be grouped by when determining an appropriate transfer source. Default: ``UNKNOWN``. +- **globus_endpoint_id: String. Specify the REST API URL of the Globus transfer manager. Default: *None* (no default). +- **hop_penalty: String. Usage cost of this RSE as an intermediate in multihop transfers. Overrides the global ``transfers/hop_penalty`` configuration for this particular RSE. Requires ``available_for_multihop`` to be ``True``. Default: *None* (no default). +- **is_object_store: Boolean. Control the auditor daemon's behavior. Instead of dumping all files, list them by date. Default: ``False``. +- **istape: Boolean. Designates the RSE as a tape storage element. Default: ``False``. +- **lfn2pfn_algorithm: String. Name of the algorithm to be used for generating paths to files on the storage. Must be defined in the configured policy package. Default: ``default``. +- **maximum_pin_lifetime: String. +- **mock: Boolean. Flag used for test/mock RSEs. Default: ``False``. +- **multihop_tombstone_delay: String. Delay before a multihop transfer intermediate rule is to be deleted (seconds). Default: ``7200``. +- **naming_convention: String. Name of the algorithm in the configured policy package which is to be used to validate DIDs on this RSE. Default: ``None``. +- **oidc_support: Boolean. Specifies that the RSE supports OIDC authentication for FTS3 transfers. Default: ``False``. +- **overwrite_when_only_on_disk: Boolean. On a ``TAPE`` RSE, controls if file can be overwritten. A file may only be overwritten if it has not yet been written to the tape backend. Default: ``False``. +- **overwrite: Boolean. Controls if a file can be overwritten on the RSE. Default: ``True`` for ``rse_type: DISK``; ``False`` for ``rse_type: TAPE``. +- **physgroup: String. Used for grouping of rules by CERN experiments. Default: ``.``. +- **qbittorrent_management_address: String. Used to configure the URL of the bittorrent management API when using the torrent transfer manager. Default: *None* (no default). +- **quota_approvers: String. List of Rucio users (comma‑separated) that can approve quota changes. Permission‑layer only; typically CERN‑specific. Default: ``None``. +- **restricted_read: Boolean. If ``True``, only allow transfers **FROM** this RSE if started by an account with admin privileges. Default: ``False``. +- **restricted_write: Boolean. If ``True``, only allow transfers **TO** this RSE if started by an account with admin privileges. Default: ``False``. +- **rule_approvers: String. List of Rucio users (comma‑separated) that will be notified by email to approve rules on this RSE. Default: ``None``. +- **rule_deleters: String. List of Rucio users (comma‑separated) that can delete rules. Permission‑layer only; typically CERN‑specific. Default: ``None``. +- **sign_url: String. Controls if URLs for uploads and transfers are to be cryptographically signed by an external service. Default: *None* (no default). +- **simulate_multirange: String. Control the number of connections for multirange byte requests on commercial cloud storage. Multirange is not supported on S3. Default: *None* (no default). +- **site: String. Used to determine if downloads/transfers are local or wide‑area network. Default: *None* (no default). +- **skip_upload_stat: Boolean. Disables the use of a GFAL ``stat`` when calling ``rucio upload`` for this RSE. Default: ``False``. +- **source_for_total_space: String. Used to specify where Rucio should obtain storage capacity information from. Default: ``storage``. +- **source_for_used_space: String. Used to specify where Rucio should obtain storage usage information from. Default: ``storage``. +- **staging_buffer: String. Used with ``TAPE`` RSEs to specify to which RSE a file on tape should be transferred to as a disk buffer. Destination RSE should have ``staging_area: True``. Default: *None* (no default). +- **staging_required: Boolean. Duplicates the ``rse_type`` RSE setting. Specifies that files on this RSE will require staging from high‑latency storage. Default: ``False``. +- **strict_copy: Boolean. Instructs the transfer manager to **ONLY** copy the file, disabling all validation checks such as ``PROPFIND`` and checksumming. Default: ``False``. +- **s3_url_style: String. For S3 storage elements [specify](https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html) `path` or `host`.Default: *None* (no default). +- **tier: String. Datacenter tier (1‑4). Default: *None* (no default). +- **tombstone_delay: String. Delay before a rule is to be deleted (seconds) +- **type: String. Values: ``{LOCALGROUPDISK, LOCALGROUPTAPE, GROUPDISK, SCRATCHDISK, MOCK, TEST, DATADISK}``. Default: ``.``. +- **use_ipv4: Boolean. Force the transfer manager to use IPv4 for transfers to this RSE. Default: ``False``. +- **verify_checksum: Boolean. Control if checksum is to be queried on transfer source and destination to confirm successful transfers. Default: ``True``. +- **block_manual_approve: Boolean. VO‑specific duplicate of ``block_manual_approval``. Default: ``False``. +- **cms_type: String. VO‑specific attribute used by CMS. Default: *None* (no default). +- **default_limit_files: String. VO‑specific default limit on number of files. Default: *None* (no default). + diff --git a/tools/build_documentation.sh b/tools/build_documentation.sh index 2434c15575b..add7a02fd23 100755 --- a/tools/build_documentation.sh +++ b/tools/build_documentation.sh @@ -173,6 +173,7 @@ for dir in rse_policies did_meta_plugins transfer_protocols transfer_tools clien done cp -r "${AUTO_GENERATED}/bin" "${DOCS}" +cp -r "${AUTO_GENERATED}/rse-attributes.mdx" "${DOCS}/operator" echo "[INFO] Generating Release Notes..." "${SCRIPT_DIR}/generate_release_notes.py" diff --git a/tools/run_in_docker/generate_configuration_pages.py b/tools/run_in_docker/generate_configuration_pages.py new file mode 100644 index 00000000000..f6bbe4b6270 --- /dev/null +++ b/tools/run_in_docker/generate_configuration_pages.py @@ -0,0 +1,16 @@ +from rucio.common.constants import RseAttr, RSEAttrObj + +if __name__ == "__main__": + mkd_file = "" + + all_attrs = [ + attr for attr in RseAttr.__dict__.values() if isinstance(attr, RSEAttrObj) + ] + for attr in all_attrs: + doc = attr.doc.replace("\n", " ") + mkd_file += ( + f"- **{attr.name}: {'Boolean' if attr.type_ is bool else 'String'}. {doc}" + ) + mkd_file += "\n" + + print(mkd_file) diff --git a/tools/run_in_docker/generate_configuration_pages.sh b/tools/run_in_docker/generate_configuration_pages.sh new file mode 100755 index 00000000000..16744a5a1c0 --- /dev/null +++ b/tools/run_in_docker/generate_configuration_pages.sh @@ -0,0 +1,7 @@ +#!/bin/bash + +set -e + +echo "Dynamically generating pages for configuration_parameters.mdx..." + +/usr/bin/env python3 generate_configuration_pages.py > /auto_generated/rse-attributes.mdx \ No newline at end of file diff --git a/tools/run_in_docker/generate_docs.sh b/tools/run_in_docker/generate_docs.sh index a8efd578029..348fb4a1cc7 100755 --- a/tools/run_in_docker/generate_docs.sh +++ b/tools/run_in_docker/generate_docs.sh @@ -17,3 +17,4 @@ export PYTHONPATH="/run_in_docker/rucio/lib:${PYTHONPATH}" "$SCRIPT_DIR"/generate_bin_help_docs.sh "$SCRIPT_DIR"/generate_client_api_docs.sh "$SCRIPT_DIR"/generate_policy_docs.sh +"$SCRIPT_DIR/"generate_configuration_pages.sh \ No newline at end of file