What does the key parameter do under the hood?¶
LaminDB is designed around associating biological metadata to artifacts and collections. This enables querying for them in storage by metadata and removes the requirement for semantic artifact and collection names.
Here, we will discuss trade-offs for using the key parameter, which allows for semantic keys, in various scenarios.
We’re simulating an artifact system with several nested folders and artifacts. Such structures are resembled in, for example, the RxRx: cell imaging guide.
# !pip install 'lamindb[jupyter]'
import random
import string
from pathlib import Path
def create_complex_biological_hierarchy(root_folder):
root_path = Path(root_folder)
if root_path.exists():
print("Folder structure already exists. Skipping...")
else:
root_path.mkdir()
raw_folder = root_path / "raw"
preprocessed_folder = root_path / "preprocessed"
raw_folder.mkdir()
preprocessed_folder.mkdir()
for i in range(1, 5):
artifact_name = f"raw_data_{i}.txt"
with (raw_folder / artifact_name).open("w") as f:
random_text = "".join(
random.choice(string.ascii_letters) for _ in range(10)
)
f.write(random_text)
for i in range(1, 3):
collection_folder = raw_folder / f"Collection_{i}"
collection_folder.mkdir()
for j in range(1, 5):
artifact_name = f"raw_data_{j}.txt"
with (collection_folder / artifact_name).open("w") as f:
random_text = "".join(
random.choice(string.ascii_letters) for _ in range(10)
)
f.write(random_text)
for i in range(1, 5):
artifact_name = f"result_{i}.txt"
with (preprocessed_folder / artifact_name).open("w") as f:
random_text = "".join(
random.choice(string.ascii_letters) for _ in range(10)
)
f.write(random_text)
root_folder = "complex_biological_project"
create_complex_biological_hierarchy(root_folder)
!lamin init --storage ./key-eval
→ connected lamindb: testuser1/key-eval
import lamindb as ln
ln.settings.verbosity = "hint"
→ connected lamindb: testuser1/key-eval
ln.UPath("complex_biological_project").view_tree()
4 sub-directories & 8 files with suffixes '.txt'
/home/runner/work/lamindb/lamindb/docs/faq/complex_biological_project
├── raw/
│ ├── Collection_1/
│ ├── raw_data_4.txt
│ ├── Collection_2/
│ ├── raw_data_3.txt
│ ├── raw_data_1.txt
│ └── raw_data_2.txt
└── preprocessed/
├── result_2.txt
├── result_4.txt
├── result_3.txt
└── result_1.txt
ln.track("WIwaNDvlEkwS0000")
→ notebook imports: lamindb==0.76.14
• tracked pip freeze > /home/runner/.cache/lamindb/run_env_pip_4aczEGr35kNwNYhp3R0t.txt
→ created Transform('WIwaNDvl'), started new Run('4aczEGr3') at 2024-10-22 13:15:18 UTC
Storing artifacts using Storage, File, and Collection¶
Lamin has three storage classes that manage different types of in-memory and on-disk objects:
Storage: Manages the default storage root that can be either local or in the cloud. For more details we refer to Storage FAQ.Artifact: Manages datasets with an optionalkeythat acts as a relative path within the current default storage root (seeStorage). An example is a single h5 artifact.Collection: Manages a collection of datasets with an optionalkeythat acts as a relative path within the current default storage root (seeStorage). An example is a collection of h5 artifacts.
For more details we refer to Tutorial: Artifacts.
The current storage root is:
ln.settings.storage
StorageSettings(root='/home/runner/work/lamindb/lamindb/docs/faq/key-eval', uid='2r43F7zPwlqp')
By default, Lamin uses virtual keys that are only reflected in the database but not in storage.
It is possible to turn this behavior off by setting ln.settings.creation._artifact_use_virtual_keys = False.
Generally, we discourage disabling this setting manually. For more details we refer to Storage FAQ.
ln.settings.creation._artifact_use_virtual_keys
True
We will now create File objects with and without semantic keys using key and also save them as Collections.
artifact_no_key_1 = ln.Artifact("complex_biological_project/raw/raw_data_1.txt")
artifact_no_key_2 = ln.Artifact("complex_biological_project/raw/raw_data_2.txt")
• path content will be copied to default storage upon `save()` with key `None` ('.lamindb/OaqXLgCLq8qEMuw70000.txt')
• path content will be copied to default storage upon `save()` with key `None` ('.lamindb/RyqlSgEzlUs5HwPt0000.txt')
The logging suggests that the artifacts will be saved to our current default storage with auto generated storage keys.
artifact_no_key_1.save()
artifact_no_key_2.save()
✓ storing artifact 'OaqXLgCLq8qEMuw70000' at '/home/runner/work/lamindb/lamindb/docs/faq/key-eval/.lamindb/OaqXLgCLq8qEMuw70000.txt'
✓ storing artifact 'RyqlSgEzlUs5HwPt0000' at '/home/runner/work/lamindb/lamindb/docs/faq/key-eval/.lamindb/RyqlSgEzlUs5HwPt0000.txt'
Artifact(uid='RyqlSgEzlUs5HwPt0000', is_latest=True, suffix='.txt', size=10, hash='YV2MYPIrXNmlf6Bzi1OR9w', _hash_type='md5', visibility=1, _key_is_virtual=True, storage_id=1, transform_id=1, run_id=1, created_by_id=1, created_at=2024-10-22 13:15:19 UTC)
artifact_key_3 = ln.Artifact(
"complex_biological_project/raw/raw_data_3.txt", key="raw/raw_data_3.txt"
)
artifact_key_4 = ln.Artifact(
"complex_biological_project/raw/raw_data_4.txt", key="raw/raw_data_4.txt"
)
artifact_key_3.save()
artifact_key_4.save()
• path content will be copied to default storage upon `save()` with key 'raw/raw_data_3.txt'
• path content will be copied to default storage upon `save()` with key 'raw/raw_data_4.txt'
✓ storing artifact '0nrr2e7pXfUXmvn80000' at '/home/runner/work/lamindb/lamindb/docs/faq/key-eval/.lamindb/0nrr2e7pXfUXmvn80000.txt'
✓ storing artifact '5GFr8OuOmJmuLi6M0000' at '/home/runner/work/lamindb/lamindb/docs/faq/key-eval/.lamindb/5GFr8OuOmJmuLi6M0000.txt'
Artifact(uid='5GFr8OuOmJmuLi6M0000', is_latest=True, key='raw/raw_data_4.txt', suffix='.txt', size=10, hash='2e9va6_OVlr5weA4XVG7Mw', _hash_type='md5', visibility=1, _key_is_virtual=True, storage_id=1, transform_id=1, run_id=1, created_by_id=1, created_at=2024-10-22 13:15:19 UTC)
Files with keys are not stored in different locations because of the usage of virtual keys.
However, they are still semantically queryable by key.
ln.Artifact.filter(key__contains="raw").df().head()
| uid | version | is_latest | description | key | suffix | type | size | hash | n_objects | n_observations | _hash_type | _accessor | visibility | _key_is_virtual | storage_id | transform_id | run_id | created_at | created_by_id | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| id | ||||||||||||||||||||
| 3 | 0nrr2e7pXfUXmvn80000 | None | True | None | raw/raw_data_3.txt | .txt | None | 10 | gf2OXdx3_ZcAoJzq9mqKrQ | None | None | md5 | None | 1 | True | 1 | 1 | 1 | 2024-10-22 13:15:19.081557+00:00 | 1 |
| 4 | 5GFr8OuOmJmuLi6M0000 | None | True | None | raw/raw_data_4.txt | .txt | None | 10 | 2e9va6_OVlr5weA4XVG7Mw | None | None | md5 | None | 1 | True | 1 | 1 | 1 | 2024-10-22 13:15:19.086768+00:00 | 1 |
Collection does not have a key parameter because it does not store any additional data in Storage.
In contrast, it has a name parameter that serves as a semantic identifier of the collection.
ds_1 = ln.Collection([artifact_no_key_1, artifact_no_key_2], name="no key collection")
ds_2 = ln.Collection([artifact_key_3, artifact_key_4], name="sample collection")
ds_1
Collection(uid='AoxpWRiDz2oZVRg20000', is_latest=True, name='no key collection', hash='UJGbFB9JEZVG2KbDE2cPfA', visibility=1, created_by_id=1, transform_id=1, run_id=1)
Advantages and disadvantages of semantic keys¶
Semantic keys have several advantages and disadvantages that we will discuss and demonstrate in the remaining notebook:
Advantages:¶
Simple: It can be easier to refer to specific collections in conversations
Familiarity: Most people are familiar with the concept of semantic names
Disadvantages¶
Length: Semantic names can be long with limited aesthetic appeal
Inconsistency: Lack of naming conventions can lead to confusion
Limited metadata: Semantic keys can contain some, but usually not all metadata
Inefficiency: Writing lengthy semantic names is a repetitive process and can be time-consuming
Ambiguity: Overly descriptive artifact names may introduce ambiguity and redundancy
Clashes: Several people may attempt to use the same semantic key. They are not unique
Renaming artifacts¶
Renaming Files that have associated keys can be done on several levels.
In storage¶
A artifact can be locally moved or renamed:
artifact_key_3.path
PosixUPath('/home/runner/work/lamindb/lamindb/docs/faq/key-eval/.lamindb/0nrr2e7pXfUXmvn80000.txt')
loaded_artifact = artifact_key_3.load()
!mkdir complex_biological_project/moved_artifacts
!mv complex_biological_project/raw/raw_data_3.txt complex_biological_project/moved_artifacts
artifact_key_3.path
PosixUPath('/home/runner/work/lamindb/lamindb/docs/faq/key-eval/.lamindb/0nrr2e7pXfUXmvn80000.txt')
After moving the artifact locally, the storage location (the path) has not changed and the artifact can still be loaded.
artifact_3 = artifact_key_3.load()
The same applies to the key which has not changed.
artifact_key_3.key
'raw/raw_data_3.txt'
By key¶
Besides moving the artifact in storage, the key can also be renamed.
artifact_key_4.key
'raw/raw_data_4.txt'
artifact_key_4.key = "bad_samples/sample_data_4.txt"
artifact_key_4.key
'bad_samples/sample_data_4.txt'
Due to the usage of virtual keys, modifying the key does not change the storage location and the artifact stays accessible.
artifact_key_4.path
PosixUPath('/home/runner/work/lamindb/lamindb/docs/faq/key-eval/.lamindb/5GFr8OuOmJmuLi6M0000.txt')
artifact_4 = artifact_key_4.load()
Modifying the path attribute¶
However, modifying the path directly is not allowed:
try:
artifact_key_4.path = f"{ln.settings.storage}/here_now/sample_data_4.txt"
except AttributeError as e:
print(e)
property of 'Artifact' object has no setter
Clashing semantic keys¶
Semantic keys should not clash. Let’s attempt to use the same semantic key twice
print(artifact_key_3.key)
print(artifact_key_4.key)
raw/raw_data_3.txt
bad_samples/sample_data_4.txt
artifact_key_4.key = "raw/raw_data_3.txt"
print(artifact_key_3.key)
print(artifact_key_4.key)
raw/raw_data_3.txt
raw/raw_data_3.txt
When filtering for this semantic key it is now unclear to which artifact we were referring to:
ln.Artifact.filter(key__icontains="sample_data_3").df()
! No records found
| uid | version | is_latest | description | key | suffix | type | size | hash | n_objects | n_observations | _hash_type | _accessor | visibility | _key_is_virtual | storage_id | transform_id | run_id | created_at | created_by_id | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| id |
When querying by key LaminDB cannot resolve which artifact we actually wanted.
In fact, we only get a single hit which does not paint a complete picture.
print(artifact_key_3.uid)
print(artifact_key_4.uid)
0nrr2e7pXfUXmvn80000
5GFr8OuOmJmuLi6M0000
Both artifacts still exist though with unique uids that can be used to get access to them.
Most importantly though, saving these artifacts to the database will result in an IntegrityError to prevent this issue.
try:
artifact_key_3.save()
artifact_key_4.save()
except Exception as e:
print(
"It is not possible to save artifacts to the same key. This results in an"
" Integrity Error!"
)
We refer to What happens if I save the same artifacts & records twice? for more detailed explanations of behavior when attempting to save artifacts multiple times.
Hierarchies¶
Another common use-case of keys are artifact hierarchies.
It can be useful to resemble the artifact structure in “complex_biological_project” from above also in LaminDB to allow for queries for artifacts that were stored in specific folders.
Common examples of this are folders specifying different processing stages such as raw, preprocessed, or curated.
Note that this use-case may also be overlapping with Collection which also allows for grouping Files.
However, Collection cannot model hierarchical groupings.
Key¶
import os
for root, _, artifacts in os.walk("complex_biological_project/raw"):
for artifactname in artifacts:
file_path = os.path.join(root, artifactname)
key_path = file_path.removeprefix("complex_biological_project")
ln_artifact = ln.Artifact(file_path, key=key_path)
ln_artifact.save()
→ returning existing artifact with same hash: Artifact(uid='5GFr8OuOmJmuLi6M0000', is_latest=True, key='raw/raw_data_3.txt', suffix='.txt', size=10, hash='2e9va6_OVlr5weA4XVG7Mw', _hash_type='md5', visibility=1, _key_is_virtual=True, storage_id=1, transform_id=1, run_id=1, created_by_id=1, created_at=2024-10-22 13:15:19 UTC)
! key raw/raw_data_3.txt on existing artifact differs from passed key /raw/raw_data_4.txt
→ returning existing artifact with same hash: Artifact(uid='OaqXLgCLq8qEMuw70000', is_latest=True, suffix='.txt', size=10, hash='4Y1QzyVulZIScd4xhw_AiQ', _hash_type='md5', visibility=1, _key_is_virtual=True, storage_id=1, transform_id=1, run_id=1, created_by_id=1, created_at=2024-10-22 13:15:19 UTC)
! key None on existing artifact differs from passed key /raw/raw_data_1.txt
→ returning existing artifact with same hash: Artifact(uid='RyqlSgEzlUs5HwPt0000', is_latest=True, suffix='.txt', size=10, hash='YV2MYPIrXNmlf6Bzi1OR9w', _hash_type='md5', visibility=1, _key_is_virtual=True, storage_id=1, transform_id=1, run_id=1, created_by_id=1, created_at=2024-10-22 13:15:19 UTC)
! key None on existing artifact differs from passed key /raw/raw_data_2.txt
• path content will be copied to default storage upon `save()` with key '/raw/Collection_1/raw_data_4.txt'
✓ storing artifact 'CvmS3OCWQm9SdumZ0000' at '/home/runner/work/lamindb/lamindb/docs/faq/key-eval/.lamindb/CvmS3OCWQm9SdumZ0000.txt'
• path content will be copied to default storage upon `save()` with key '/raw/Collection_1/raw_data_3.txt'
✓ storing artifact 'oUPjWZYp1ndD4Gca0000' at '/home/runner/work/lamindb/lamindb/docs/faq/key-eval/.lamindb/oUPjWZYp1ndD4Gca0000.txt'
• path content will be copied to default storage upon `save()` with key '/raw/Collection_1/raw_data_1.txt'
✓ storing artifact 'q6DPfk5wPM6DCOnS0000' at '/home/runner/work/lamindb/lamindb/docs/faq/key-eval/.lamindb/q6DPfk5wPM6DCOnS0000.txt'
• path content will be copied to default storage upon `save()` with key '/raw/Collection_1/raw_data_2.txt'
✓ storing artifact 'v4nhSZVFL8nWHv760000' at '/home/runner/work/lamindb/lamindb/docs/faq/key-eval/.lamindb/v4nhSZVFL8nWHv760000.txt'
• path content will be copied to default storage upon `save()` with key '/raw/Collection_2/raw_data_4.txt'
✓ storing artifact 'gddxE4e5pt3WdpN60000' at '/home/runner/work/lamindb/lamindb/docs/faq/key-eval/.lamindb/gddxE4e5pt3WdpN60000.txt'
• path content will be copied to default storage upon `save()` with key '/raw/Collection_2/raw_data_3.txt'
✓ storing artifact 'BFsN1QXc9CFadpP00000' at '/home/runner/work/lamindb/lamindb/docs/faq/key-eval/.lamindb/BFsN1QXc9CFadpP00000.txt'
• path content will be copied to default storage upon `save()` with key '/raw/Collection_2/raw_data_1.txt'
✓ storing artifact 'juGPsaxHR59YqtQg0000' at '/home/runner/work/lamindb/lamindb/docs/faq/key-eval/.lamindb/juGPsaxHR59YqtQg0000.txt'
• path content will be copied to default storage upon `save()` with key '/raw/Collection_2/raw_data_2.txt'
✓ storing artifact 'gHn7eVT05gCcAY6P0000' at '/home/runner/work/lamindb/lamindb/docs/faq/key-eval/.lamindb/gHn7eVT05gCcAY6P0000.txt'
ln.Artifact.filter(key__startswith="raw").df()
| uid | version | is_latest | description | key | suffix | type | size | hash | n_objects | n_observations | _hash_type | _accessor | visibility | _key_is_virtual | storage_id | transform_id | run_id | created_at | created_by_id | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| id | ||||||||||||||||||||
| 3 | 0nrr2e7pXfUXmvn80000 | None | True | None | raw/raw_data_3.txt | .txt | None | 10 | gf2OXdx3_ZcAoJzq9mqKrQ | None | None | md5 | None | 1 | True | 1 | 1 | 1 | 2024-10-22 13:15:19.081557+00:00 | 1 |
| 4 | 5GFr8OuOmJmuLi6M0000 | None | True | None | raw/raw_data_3.txt | .txt | None | 10 | 2e9va6_OVlr5weA4XVG7Mw | None | None | md5 | None | 1 | True | 1 | 1 | 1 | 2024-10-22 13:15:19.086768+00:00 | 1 |
Collection¶
Alternatively, it would have been possible to create a Collection with a corresponding name:
all_data_paths = []
for root, _, artifacts in os.walk("complex_biological_project/raw"):
for artifactname in artifacts:
file_path = os.path.join(root, artifactname)
all_data_paths.append(file_path)
all_data_artifacts = []
for path in all_data_paths:
all_data_artifacts.append(ln.Artifact(path))
data_ds = ln.Collection(all_data_artifacts, name="data")
data_ds.save()
→ returning existing artifact with same hash: Artifact(uid='5GFr8OuOmJmuLi6M0000', is_latest=True, key='raw/raw_data_3.txt', suffix='.txt', size=10, hash='2e9va6_OVlr5weA4XVG7Mw', _hash_type='md5', visibility=1, _key_is_virtual=True, storage_id=1, transform_id=1, run_id=1, created_by_id=1, created_at=2024-10-22 13:15:19 UTC)
→ returning existing artifact with same hash: Artifact(uid='OaqXLgCLq8qEMuw70000', is_latest=True, suffix='.txt', size=10, hash='4Y1QzyVulZIScd4xhw_AiQ', _hash_type='md5', visibility=1, _key_is_virtual=True, storage_id=1, transform_id=1, run_id=1, created_by_id=1, created_at=2024-10-22 13:15:19 UTC)
→ returning existing artifact with same hash: Artifact(uid='RyqlSgEzlUs5HwPt0000', is_latest=True, suffix='.txt', size=10, hash='YV2MYPIrXNmlf6Bzi1OR9w', _hash_type='md5', visibility=1, _key_is_virtual=True, storage_id=1, transform_id=1, run_id=1, created_by_id=1, created_at=2024-10-22 13:15:19 UTC)
→ returning existing artifact with same hash: Artifact(uid='CvmS3OCWQm9SdumZ0000', is_latest=True, key='/raw/Collection_1/raw_data_4.txt', suffix='.txt', size=10, hash='XaFGdCG0iBGKaGm3mVuL-Q', _hash_type='md5', visibility=1, _key_is_virtual=True, storage_id=1, transform_id=1, run_id=1, created_by_id=1, created_at=2024-10-22 13:15:19 UTC)
→ returning existing artifact with same hash: Artifact(uid='oUPjWZYp1ndD4Gca0000', is_latest=True, key='/raw/Collection_1/raw_data_3.txt', suffix='.txt', size=10, hash='RZwmo2-y8WOo-wQnlzGVdQ', _hash_type='md5', visibility=1, _key_is_virtual=True, storage_id=1, transform_id=1, run_id=1, created_by_id=1, created_at=2024-10-22 13:15:19 UTC)
→ returning existing artifact with same hash: Artifact(uid='q6DPfk5wPM6DCOnS0000', is_latest=True, key='/raw/Collection_1/raw_data_1.txt', suffix='.txt', size=10, hash='MgO0oue4KS29hxb5BUL3aA', _hash_type='md5', visibility=1, _key_is_virtual=True, storage_id=1, transform_id=1, run_id=1, created_by_id=1, created_at=2024-10-22 13:15:19 UTC)
→ returning existing artifact with same hash: Artifact(uid='v4nhSZVFL8nWHv760000', is_latest=True, key='/raw/Collection_1/raw_data_2.txt', suffix='.txt', size=10, hash='DYS03CNxpMvvoLsS1XmrzQ', _hash_type='md5', visibility=1, _key_is_virtual=True, storage_id=1, transform_id=1, run_id=1, created_by_id=1, created_at=2024-10-22 13:15:19 UTC)
→ returning existing artifact with same hash: Artifact(uid='gddxE4e5pt3WdpN60000', is_latest=True, key='/raw/Collection_2/raw_data_4.txt', suffix='.txt', size=10, hash='42ba92mlNPqHubZhC3Hrbg', _hash_type='md5', visibility=1, _key_is_virtual=True, storage_id=1, transform_id=1, run_id=1, created_by_id=1, created_at=2024-10-22 13:15:19 UTC)
→ returning existing artifact with same hash: Artifact(uid='BFsN1QXc9CFadpP00000', is_latest=True, key='/raw/Collection_2/raw_data_3.txt', suffix='.txt', size=10, hash='wjBoap6NbIupv342794McQ', _hash_type='md5', visibility=1, _key_is_virtual=True, storage_id=1, transform_id=1, run_id=1, created_by_id=1, created_at=2024-10-22 13:15:19 UTC)
→ returning existing artifact with same hash: Artifact(uid='juGPsaxHR59YqtQg0000', is_latest=True, key='/raw/Collection_2/raw_data_1.txt', suffix='.txt', size=10, hash='1eUHBGvHa02EBEjj1z7pdQ', _hash_type='md5', visibility=1, _key_is_virtual=True, storage_id=1, transform_id=1, run_id=1, created_by_id=1, created_at=2024-10-22 13:15:19 UTC)
→ returning existing artifact with same hash: Artifact(uid='gHn7eVT05gCcAY6P0000', is_latest=True, key='/raw/Collection_2/raw_data_2.txt', suffix='.txt', size=10, hash='tKib9Wa6KCStcBHrd2gp1Q', _hash_type='md5', visibility=1, _key_is_virtual=True, storage_id=1, transform_id=1, run_id=1, created_by_id=1, created_at=2024-10-22 13:15:19 UTC)
Collection(uid='c5lNPjrPqDHJysvI0000', is_latest=True, name='data', hash='NBe0V7uAmoYQlJ9Jt0OSIA', visibility=1, created_by_id=1, transform_id=1, run_id=1, created_at=2024-10-22 13:15:19 UTC)
ln.Collection.filter(name__icontains="data").df()
| uid | version | is_latest | name | description | hash | reference | reference_type | visibility | transform_id | meta_artifact_id | run_id | created_at | created_by_id | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| id | ||||||||||||||
| 1 | c5lNPjrPqDHJysvI0000 | None | True | data | None | NBe0V7uAmoYQlJ9Jt0OSIA | None | None | 1 | 1 | None | 1 | 2024-10-22 13:15:19.721177+00:00 | 1 |
This approach will likely lead to clashes. Alternatively, Ulabels can be added to Files to resemble hierarchies.
Ulabels¶
for root, _, artifacts in os.walk("complex_biological_project/raw"):
for artifactname in artifacts:
file_path = os.path.join(root, artifactname)
key_path = file_path.removeprefix("complex_biological_project")
ln_artifact = ln.Artifact(file_path, key=key_path)
ln_artifact.save()
data_label = ln.ULabel(name="data")
data_label.save()
ln_artifact.ulabels.add(data_label)
→ returning existing artifact with same hash: Artifact(uid='5GFr8OuOmJmuLi6M0000', is_latest=True, key='raw/raw_data_3.txt', suffix='.txt', size=10, hash='2e9va6_OVlr5weA4XVG7Mw', _hash_type='md5', visibility=1, _key_is_virtual=True, storage_id=1, transform_id=1, run_id=1, created_by_id=1, created_at=2024-10-22 13:15:19 UTC)
! key raw/raw_data_3.txt on existing artifact differs from passed key /raw/raw_data_4.txt
→ returning existing artifact with same hash: Artifact(uid='OaqXLgCLq8qEMuw70000', is_latest=True, suffix='.txt', size=10, hash='4Y1QzyVulZIScd4xhw_AiQ', _hash_type='md5', visibility=1, _key_is_virtual=True, storage_id=1, transform_id=1, run_id=1, created_by_id=1, created_at=2024-10-22 13:15:19 UTC)
! key None on existing artifact differs from passed key /raw/raw_data_1.txt
→ returning existing ULabel record with same name: 'data'
→ returning existing artifact with same hash: Artifact(uid='RyqlSgEzlUs5HwPt0000', is_latest=True, suffix='.txt', size=10, hash='YV2MYPIrXNmlf6Bzi1OR9w', _hash_type='md5', visibility=1, _key_is_virtual=True, storage_id=1, transform_id=1, run_id=1, created_by_id=1, created_at=2024-10-22 13:15:19 UTC)
! key None on existing artifact differs from passed key /raw/raw_data_2.txt
→ returning existing ULabel record with same name: 'data'
→ returning existing artifact with same hash: Artifact(uid='CvmS3OCWQm9SdumZ0000', is_latest=True, key='/raw/Collection_1/raw_data_4.txt', suffix='.txt', size=10, hash='XaFGdCG0iBGKaGm3mVuL-Q', _hash_type='md5', visibility=1, _key_is_virtual=True, storage_id=1, transform_id=1, run_id=1, created_by_id=1, created_at=2024-10-22 13:15:19 UTC)
→ returning existing ULabel record with same name: 'data'
→ returning existing artifact with same hash: Artifact(uid='oUPjWZYp1ndD4Gca0000', is_latest=True, key='/raw/Collection_1/raw_data_3.txt', suffix='.txt', size=10, hash='RZwmo2-y8WOo-wQnlzGVdQ', _hash_type='md5', visibility=1, _key_is_virtual=True, storage_id=1, transform_id=1, run_id=1, created_by_id=1, created_at=2024-10-22 13:15:19 UTC)
→ returning existing ULabel record with same name: 'data'
→ returning existing artifact with same hash: Artifact(uid='q6DPfk5wPM6DCOnS0000', is_latest=True, key='/raw/Collection_1/raw_data_1.txt', suffix='.txt', size=10, hash='MgO0oue4KS29hxb5BUL3aA', _hash_type='md5', visibility=1, _key_is_virtual=True, storage_id=1, transform_id=1, run_id=1, created_by_id=1, created_at=2024-10-22 13:15:19 UTC)
→ returning existing ULabel record with same name: 'data'
→ returning existing artifact with same hash: Artifact(uid='v4nhSZVFL8nWHv760000', is_latest=True, key='/raw/Collection_1/raw_data_2.txt', suffix='.txt', size=10, hash='DYS03CNxpMvvoLsS1XmrzQ', _hash_type='md5', visibility=1, _key_is_virtual=True, storage_id=1, transform_id=1, run_id=1, created_by_id=1, created_at=2024-10-22 13:15:19 UTC)
→ returning existing ULabel record with same name: 'data'
→ returning existing artifact with same hash: Artifact(uid='gddxE4e5pt3WdpN60000', is_latest=True, key='/raw/Collection_2/raw_data_4.txt', suffix='.txt', size=10, hash='42ba92mlNPqHubZhC3Hrbg', _hash_type='md5', visibility=1, _key_is_virtual=True, storage_id=1, transform_id=1, run_id=1, created_by_id=1, created_at=2024-10-22 13:15:19 UTC)
→ returning existing ULabel record with same name: 'data'
→ returning existing artifact with same hash: Artifact(uid='BFsN1QXc9CFadpP00000', is_latest=True, key='/raw/Collection_2/raw_data_3.txt', suffix='.txt', size=10, hash='wjBoap6NbIupv342794McQ', _hash_type='md5', visibility=1, _key_is_virtual=True, storage_id=1, transform_id=1, run_id=1, created_by_id=1, created_at=2024-10-22 13:15:19 UTC)
→ returning existing ULabel record with same name: 'data'
→ returning existing artifact with same hash: Artifact(uid='juGPsaxHR59YqtQg0000', is_latest=True, key='/raw/Collection_2/raw_data_1.txt', suffix='.txt', size=10, hash='1eUHBGvHa02EBEjj1z7pdQ', _hash_type='md5', visibility=1, _key_is_virtual=True, storage_id=1, transform_id=1, run_id=1, created_by_id=1, created_at=2024-10-22 13:15:19 UTC)
→ returning existing ULabel record with same name: 'data'
→ returning existing artifact with same hash: Artifact(uid='gHn7eVT05gCcAY6P0000', is_latest=True, key='/raw/Collection_2/raw_data_2.txt', suffix='.txt', size=10, hash='tKib9Wa6KCStcBHrd2gp1Q', _hash_type='md5', visibility=1, _key_is_virtual=True, storage_id=1, transform_id=1, run_id=1, created_by_id=1, created_at=2024-10-22 13:15:19 UTC)
→ returning existing ULabel record with same name: 'data'
labels = ln.ULabel.lookup()
ln.Artifact.filter(ulabels__in=[labels.data]).df()
| uid | version | is_latest | description | key | suffix | type | size | hash | n_objects | n_observations | _hash_type | _accessor | visibility | _key_is_virtual | storage_id | transform_id | run_id | created_at | created_by_id | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| id | ||||||||||||||||||||
| 4 | 5GFr8OuOmJmuLi6M0000 | None | True | None | raw/raw_data_3.txt | .txt | None | 10 | 2e9va6_OVlr5weA4XVG7Mw | None | None | md5 | None | 1 | True | 1 | 1 | 1 | 2024-10-22 13:15:19.086768+00:00 | 1 |
| 1 | OaqXLgCLq8qEMuw70000 | None | True | None | None | .txt | None | 10 | 4Y1QzyVulZIScd4xhw_AiQ | None | None | md5 | None | 1 | True | 1 | 1 | 1 | 2024-10-22 13:15:19.043213+00:00 | 1 |
| 2 | RyqlSgEzlUs5HwPt0000 | None | True | None | None | .txt | None | 10 | YV2MYPIrXNmlf6Bzi1OR9w | None | None | md5 | None | 1 | True | 1 | 1 | 1 | 2024-10-22 13:15:19.048928+00:00 | 1 |
| 5 | CvmS3OCWQm9SdumZ0000 | None | True | None | /raw/Collection_1/raw_data_4.txt | .txt | None | 10 | XaFGdCG0iBGKaGm3mVuL-Q | None | None | md5 | None | 1 | True | 1 | 1 | 1 | 2024-10-22 13:15:19.566350+00:00 | 1 |
| 6 | oUPjWZYp1ndD4Gca0000 | None | True | None | /raw/Collection_1/raw_data_3.txt | .txt | None | 10 | RZwmo2-y8WOo-wQnlzGVdQ | None | None | md5 | None | 1 | True | 1 | 1 | 1 | 2024-10-22 13:15:19.576080+00:00 | 1 |
| 7 | q6DPfk5wPM6DCOnS0000 | None | True | None | /raw/Collection_1/raw_data_1.txt | .txt | None | 10 | MgO0oue4KS29hxb5BUL3aA | None | None | md5 | None | 1 | True | 1 | 1 | 1 | 2024-10-22 13:15:19.585426+00:00 | 1 |
| 8 | v4nhSZVFL8nWHv760000 | None | True | None | /raw/Collection_1/raw_data_2.txt | .txt | None | 10 | DYS03CNxpMvvoLsS1XmrzQ | None | None | md5 | None | 1 | True | 1 | 1 | 1 | 2024-10-22 13:15:19.595229+00:00 | 1 |
| 9 | gddxE4e5pt3WdpN60000 | None | True | None | /raw/Collection_2/raw_data_4.txt | .txt | None | 10 | 42ba92mlNPqHubZhC3Hrbg | None | None | md5 | None | 1 | True | 1 | 1 | 1 | 2024-10-22 13:15:19.604502+00:00 | 1 |
| 10 | BFsN1QXc9CFadpP00000 | None | True | None | /raw/Collection_2/raw_data_3.txt | .txt | None | 10 | wjBoap6NbIupv342794McQ | None | None | md5 | None | 1 | True | 1 | 1 | 1 | 2024-10-22 13:15:19.614344+00:00 | 1 |
| 11 | juGPsaxHR59YqtQg0000 | None | True | None | /raw/Collection_2/raw_data_1.txt | .txt | None | 10 | 1eUHBGvHa02EBEjj1z7pdQ | None | None | md5 | None | 1 | True | 1 | 1 | 1 | 2024-10-22 13:15:19.623878+00:00 | 1 |
| 12 | gHn7eVT05gCcAY6P0000 | None | True | None | /raw/Collection_2/raw_data_2.txt | .txt | None | 10 | tKib9Wa6KCStcBHrd2gp1Q | None | None | md5 | None | 1 | True | 1 | 1 | 1 | 2024-10-22 13:15:19.633096+00:00 | 1 |
However, Ulabels are too versatile for such an approach and clashes are also to be expected here.
Metadata¶
Due to the chance of clashes for the aforementioned approaches being rather high, we generally recommend not to store hierarchical data with solely semantic keys.
Biological metadata makes Files and Collections unambiguous and easily queryable.
Legacy data and multiple storage roots¶
Distributed Collections¶
LaminDB can ingest legacy data that already had a structure in their storage.
In such cases, it disables _artifact_use_virtual_keys and the artifacts are ingested with their actual storage location.
It might be therefore be possible that Files stored in different storage roots may be associated with a single Collection.
To simulate this, we are disabling _artifact_use_virtual_keys and ingest artifacts stored in a different path (the “legacy data”).
ln.settings.creation._artifact_use_virtual_keys = False
for root, _, artifacts in os.walk("complex_biological_project/preprocessed"):
for artifactname in artifacts:
file_path = os.path.join(root, artifactname)
key_path = file_path.removeprefix("complex_biological_project")
print(file_path)
print()
ln_artifact = ln.Artifact(file_path, key=f"./{key_path}")
ln_artifact.save()
complex_biological_project/preprocessed/result_2.txt
• path content will be copied to default storage upon `save()` with key './/preprocessed/result_2.txt'
✓ storing artifact 'waM4KZ1nQSfJLw0U0000' at '/home/runner/work/lamindb/lamindb/docs/faq/key-eval/preprocessed/result_2.txt'
complex_biological_project/preprocessed/result_4.txt
• path content will be copied to default storage upon `save()` with key './/preprocessed/result_4.txt'
✓ storing artifact 'pfjJ0eleJ8D3YKZ10000' at '/home/runner/work/lamindb/lamindb/docs/faq/key-eval/preprocessed/result_4.txt'
complex_biological_project/preprocessed/result_3.txt
• path content will be copied to default storage upon `save()` with key './/preprocessed/result_3.txt'
✓ storing artifact 'NQtnuCQPar4o9g4b0000' at '/home/runner/work/lamindb/lamindb/docs/faq/key-eval/preprocessed/result_3.txt'
complex_biological_project/preprocessed/result_1.txt
• path content will be copied to default storage upon `save()` with key './/preprocessed/result_1.txt'
✓ storing artifact 'h6dvvG38l8yYuq2m0000' at '/home/runner/work/lamindb/lamindb/docs/faq/key-eval/preprocessed/result_1.txt'
ln.Artifact.df()
| uid | version | is_latest | description | key | suffix | type | size | hash | n_objects | n_observations | _hash_type | _accessor | visibility | _key_is_virtual | storage_id | transform_id | run_id | created_at | created_by_id | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| id | ||||||||||||||||||||
| 16 | h6dvvG38l8yYuq2m0000 | None | True | None | .//preprocessed/result_1.txt | .txt | None | 10 | tamSYccxJkaBSGeo4xd_NA | None | None | md5 | None | 1 | False | 1 | 1 | 1 | 2024-10-22 13:15:20.088928+00:00 | 1 |
| 15 | NQtnuCQPar4o9g4b0000 | None | True | None | .//preprocessed/result_3.txt | .txt | None | 10 | U9YQvEI4dyPJuaO83fdzFA | None | None | md5 | None | 1 | False | 1 | 1 | 1 | 2024-10-22 13:15:20.078693+00:00 | 1 |
| 14 | pfjJ0eleJ8D3YKZ10000 | None | True | None | .//preprocessed/result_4.txt | .txt | None | 10 | RDQLqqUBKi7iNC9-DR0vQw | None | None | md5 | None | 1 | False | 1 | 1 | 1 | 2024-10-22 13:15:20.068425+00:00 | 1 |
| 13 | waM4KZ1nQSfJLw0U0000 | None | True | None | .//preprocessed/result_2.txt | .txt | None | 10 | XdR8ytsda4FmToNMd1HlfA | None | None | md5 | None | 1 | False | 1 | 1 | 1 | 2024-10-22 13:15:20.058109+00:00 | 1 |
| 12 | gHn7eVT05gCcAY6P0000 | None | True | None | /raw/Collection_2/raw_data_2.txt | .txt | None | 10 | tKib9Wa6KCStcBHrd2gp1Q | None | None | md5 | None | 1 | True | 1 | 1 | 1 | 2024-10-22 13:15:19.633096+00:00 | 1 |
| 11 | juGPsaxHR59YqtQg0000 | None | True | None | /raw/Collection_2/raw_data_1.txt | .txt | None | 10 | 1eUHBGvHa02EBEjj1z7pdQ | None | None | md5 | None | 1 | True | 1 | 1 | 1 | 2024-10-22 13:15:19.623878+00:00 | 1 |
| 10 | BFsN1QXc9CFadpP00000 | None | True | None | /raw/Collection_2/raw_data_3.txt | .txt | None | 10 | wjBoap6NbIupv342794McQ | None | None | md5 | None | 1 | True | 1 | 1 | 1 | 2024-10-22 13:15:19.614344+00:00 | 1 |
| 9 | gddxE4e5pt3WdpN60000 | None | True | None | /raw/Collection_2/raw_data_4.txt | .txt | None | 10 | 42ba92mlNPqHubZhC3Hrbg | None | None | md5 | None | 1 | True | 1 | 1 | 1 | 2024-10-22 13:15:19.604502+00:00 | 1 |
| 8 | v4nhSZVFL8nWHv760000 | None | True | None | /raw/Collection_1/raw_data_2.txt | .txt | None | 10 | DYS03CNxpMvvoLsS1XmrzQ | None | None | md5 | None | 1 | True | 1 | 1 | 1 | 2024-10-22 13:15:19.595229+00:00 | 1 |
| 7 | q6DPfk5wPM6DCOnS0000 | None | True | None | /raw/Collection_1/raw_data_1.txt | .txt | None | 10 | MgO0oue4KS29hxb5BUL3aA | None | None | md5 | None | 1 | True | 1 | 1 | 1 | 2024-10-22 13:15:19.585426+00:00 | 1 |
| 6 | oUPjWZYp1ndD4Gca0000 | None | True | None | /raw/Collection_1/raw_data_3.txt | .txt | None | 10 | RZwmo2-y8WOo-wQnlzGVdQ | None | None | md5 | None | 1 | True | 1 | 1 | 1 | 2024-10-22 13:15:19.576080+00:00 | 1 |
| 5 | CvmS3OCWQm9SdumZ0000 | None | True | None | /raw/Collection_1/raw_data_4.txt | .txt | None | 10 | XaFGdCG0iBGKaGm3mVuL-Q | None | None | md5 | None | 1 | True | 1 | 1 | 1 | 2024-10-22 13:15:19.566350+00:00 | 1 |
| 2 | RyqlSgEzlUs5HwPt0000 | None | True | None | None | .txt | None | 10 | YV2MYPIrXNmlf6Bzi1OR9w | None | None | md5 | None | 1 | True | 1 | 1 | 1 | 2024-10-22 13:15:19.048928+00:00 | 1 |
| 1 | OaqXLgCLq8qEMuw70000 | None | True | None | None | .txt | None | 10 | 4Y1QzyVulZIScd4xhw_AiQ | None | None | md5 | None | 1 | True | 1 | 1 | 1 | 2024-10-22 13:15:19.043213+00:00 | 1 |
| 4 | 5GFr8OuOmJmuLi6M0000 | None | True | None | raw/raw_data_3.txt | .txt | None | 10 | 2e9va6_OVlr5weA4XVG7Mw | None | None | md5 | None | 1 | True | 1 | 1 | 1 | 2024-10-22 13:15:19.086768+00:00 | 1 |
| 3 | 0nrr2e7pXfUXmvn80000 | None | True | None | raw/raw_data_3.txt | .txt | None | 10 | gf2OXdx3_ZcAoJzq9mqKrQ | None | None | md5 | None | 1 | True | 1 | 1 | 1 | 2024-10-22 13:15:19.081557+00:00 | 1 |
artifact_from_raw = ln.Artifact.filter(key__icontains="Collection_2/raw_data_1").first()
artifact_from_preprocessed = ln.Artifact.filter(
key__icontains="preprocessed/result_1"
).first()
print(artifact_from_raw.path)
print(artifact_from_preprocessed.path)
/home/runner/work/lamindb/lamindb/docs/faq/key-eval/.lamindb/juGPsaxHR59YqtQg0000.txt
/home/runner/work/lamindb/lamindb/docs/faq/key-eval/preprocessed/result_1.txt
Let’s create our Collection:
ds = ln.Collection(
[artifact_from_raw, artifact_from_preprocessed], name="raw_and_processed_collection_2"
)
ds.save()
Collection(uid='5dsrse15HAsHkXPW0000', is_latest=True, name='raw_and_processed_collection_2', hash='63kEl_oFEVn_wnflFWN5LA', visibility=1, created_by_id=1, transform_id=1, run_id=1, created_at=2024-10-22 13:15:20 UTC)
ds.artifacts.df()
| uid | version | is_latest | description | key | suffix | type | size | hash | n_objects | n_observations | _hash_type | _accessor | visibility | _key_is_virtual | storage_id | transform_id | run_id | created_at | created_by_id | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| id | ||||||||||||||||||||
| 11 | juGPsaxHR59YqtQg0000 | None | True | None | /raw/Collection_2/raw_data_1.txt | .txt | None | 10 | 1eUHBGvHa02EBEjj1z7pdQ | None | None | md5 | None | 1 | True | 1 | 1 | 1 | 2024-10-22 13:15:19.623878+00:00 | 1 |
| 16 | h6dvvG38l8yYuq2m0000 | None | True | None | .//preprocessed/result_1.txt | .txt | None | 10 | tamSYccxJkaBSGeo4xd_NA | None | None | md5 | None | 1 | False | 1 | 1 | 1 | 2024-10-22 13:15:20.088928+00:00 | 1 |
Modeling directories¶
ln.settings.creation._artifact_use_virtual_keys = True
dir_path = ln.core.datasets.dir_scrnaseq_cellranger("sample_001")
ln.UPath(dir_path).view_tree()
• file has more than one suffix (path.suffixes), using only last suffix: '.bai' - if you want your composite suffix to be recognized add it to lamindb.core.storage.VALID_SIMPLE_SUFFIXES.add()
3 sub-directories & 15 files with suffixes '.cloupe', '.mtx.gz', '.bai', '.h5', '.html', '.tsv.gz', '.bam', '.csv'
/home/runner/work/lamindb/lamindb/docs/faq/sample_001
├── cloupe.cloupe
├── web_summary.html
├── filtered_feature_bc_matrix/
│ ├── features.tsv.gz
│ ├── matrix.mtx.gz
│ └── barcodes.tsv.gz
├── raw_feature_bc_matrix/
│ ├── features.tsv.gz
│ ├── matrix.mtx.gz
│ └── barcodes.tsv.gz
├── metrics_summary.csv
├── filtered_feature_bc_matrix.h5
├── raw_feature_bc_matrix.h5
├── possorted_genome_bam.bam.bai
├── possorted_genome_bam.bam
├── analysis/
│ └── analysis.csv
└── molecule_info.h5
There are two ways to create Artifact objects from directories: from_dir() and Artifact.
cellranger_raw_artifact = ln.Artifact.from_dir("sample_001/raw_feature_bc_matrix/")
! folder is outside existing storage location, will copy files from sample_001/raw_feature_bc_matrix/ to /home/runner/work/lamindb/lamindb/docs/faq/key-eval/raw_feature_bc_matrix
✓ created 3 artifacts from directory using storage /home/runner/work/lamindb/lamindb/docs/faq/key-eval and key = raw_feature_bc_matrix/
for artifact in cellranger_raw_artifact:
artifact.save()
✓ storing artifact '59EhohFT59L7bZD50000' at '/home/runner/work/lamindb/lamindb/docs/faq/key-eval/.lamindb/59EhohFT59L7bZD50000.tsv.gz'
✓ storing artifact 'F1jQdyzR1suKFakX0000' at '/home/runner/work/lamindb/lamindb/docs/faq/key-eval/.lamindb/F1jQdyzR1suKFakX0000.mtx.gz'
✓ storing artifact 'jZHvJbuZfouOfkg20000' at '/home/runner/work/lamindb/lamindb/docs/faq/key-eval/.lamindb/jZHvJbuZfouOfkg20000.tsv.gz'
cellranger_raw_folder = ln.Artifact(
"sample_001/raw_feature_bc_matrix/", description="cellranger raw"
)
cellranger_raw_folder.save()
• path content will be copied to default storage upon `save()` with key `None` ('.lamindb/kNY2czOJV8RZEpCa')
✓ storing artifact 'kNY2czOJV8RZEpCa0000' at '/home/runner/work/lamindb/lamindb/docs/faq/key-eval/.lamindb/kNY2czOJV8RZEpCa'
Artifact(uid='kNY2czOJV8RZEpCa0000', is_latest=True, description='cellranger raw', suffix='', size=18, hash='oiNcLTu0SrCG62fO6JEdaA', n_objects=3, _hash_type='md5-d', visibility=1, _key_is_virtual=True, storage_id=1, transform_id=1, run_id=1, created_by_id=1, created_at=2024-10-22 13:15:20 UTC)
ln.Artifact.filter(key__icontains="raw_feature_bc_matrix").df()
| uid | version | is_latest | description | key | suffix | type | size | hash | n_objects | n_observations | _hash_type | _accessor | visibility | _key_is_virtual | storage_id | transform_id | run_id | created_at | created_by_id | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| id | ||||||||||||||||||||
| 17 | 59EhohFT59L7bZD50000 | None | True | None | raw_feature_bc_matrix/features.tsv.gz | .tsv.gz | None | 6 | 7q9-lrfMxBDyTsItFaPpww | None | None | md5 | None | 1 | True | 1 | 1 | 1 | 2024-10-22 13:15:20.223046+00:00 | 1 |
| 18 | F1jQdyzR1suKFakX0000 | None | True | None | raw_feature_bc_matrix/matrix.mtx.gz | .mtx.gz | None | 6 | dkwPvYVXya0zm2cUvl1f8g | None | None | md5 | None | 1 | True | 1 | 1 | 1 | 2024-10-22 13:15:20.228226+00:00 | 1 |
| 19 | jZHvJbuZfouOfkg20000 | None | True | None | raw_feature_bc_matrix/barcodes.tsv.gz | .tsv.gz | None | 6 | WsUIGwb92J0mhx9aYATMjA | None | None | md5 | None | 1 | True | 1 | 1 | 1 | 2024-10-22 13:15:20.234492+00:00 | 1 |
ln.Artifact.get(key__icontains="raw_feature_bc_matrix/matrix.mtx.gz").path
PosixUPath('/home/runner/work/lamindb/lamindb/docs/faq/key-eval/.lamindb/F1jQdyzR1suKFakX0000.mtx.gz')
artifact = ln.Artifact.get(description="cellranger raw")
artifact.path.glob("*")
<generator object Path.glob at 0x7fb3c65cf780>