Hi, thank you for releasing the HiLo code and checkpoints.
I am trying to reproduce the DomainNet protocol and found a class-split discrepancy that appears to be separate from the dataset-length issue reported in #4.
Observations
Table 1 of the paper reports the following DomainNet statistics:
- 172 labelled/old classes
- 39.1K labelled images
- 345 total classes
However, in the released code at commit 34f63a4:
-
With the default pre_splits=False, get_class_splits() uses:
args.train_classes = range(173)
args.unlabeled_classes = range(173, 345)
This gives 173 old classes and 172 new classes.
-
Both the training and evaluation entrypoints default to pre_splits=False, and scripts/mi_pmtrans/domainnet.sh does not override it.
-
With pre_splits=True, the code instead loads:
os.path.join(osr_split_dir, "domainnet_splits.pkl")
However, domainnet_splits.pkl does not appear to be included in the public repository.
-
Using the official cleaned DomainNet train/test lists, merged per domain to reproduce the ImageFolder support expected by the released loader, classes 0..172 contain 87,864 Real-domain images. With prop_train_labels=0.5, this produces exactly 43,932 labelled samples.
-
The released DomainNet curriculum-weight files are also consistent with this number: for every auxiliary domain, the pre array starts with exactly 43,932 entries equal to 1.0, matching the labelled prefix of MergedTriDataset.
For context, using the cleaned DomainNet lists and merging train/test per domain also makes the total dataset lengths match all five released sample_weights files. Therefore, the remaining question here is specifically about the old/new class split and labelled count, rather than the raw-vs-cleaned dataset-length mismatch in #4.
Questions
Could you please clarify:
- Were the DomainNet results in Table 2 produced with
pre_splits=True using the external domainnet_splits.pkl, or with the released default pre_splits=False?
- If
pre_splits=True was used, could you release domainnet_splits.pkl, or at least the exact known_classes and unknown_classes lists?
- Which split were the released DomainNet checkpoints and
sample_weights files generated with?
- Should public reproduction use:
- the paper protocol: 172 old classes / approximately 39.1K labelled samples, or
- the released-code protocol: 173 old classes / 43,932 labelled samples?
- If 172/39.1K reflects an earlier protocol or a typo in Table 1, could this be documented in the repository?
Providing the exact class split and its corresponding sample-weight/checkpoint protocol would help establish a matched reproduction of the DomainNet results.
Thank you!
Hi, thank you for releasing the HiLo code and checkpoints.
I am trying to reproduce the DomainNet protocol and found a class-split discrepancy that appears to be separate from the dataset-length issue reported in #4.
Observations
Table 1 of the paper reports the following DomainNet statistics:
However, in the released code at commit
34f63a4:With the default
pre_splits=False,get_class_splits()uses:This gives 173 old classes and 172 new classes.
Both the training and evaluation entrypoints default to
pre_splits=False, andscripts/mi_pmtrans/domainnet.shdoes not override it.With
pre_splits=True, the code instead loads:However,
domainnet_splits.pkldoes not appear to be included in the public repository.Using the official cleaned DomainNet train/test lists, merged per domain to reproduce the
ImageFoldersupport expected by the released loader, classes0..172contain 87,864 Real-domain images. Withprop_train_labels=0.5, this produces exactly 43,932 labelled samples.The released DomainNet curriculum-weight files are also consistent with this number: for every auxiliary domain, the
prearray starts with exactly 43,932 entries equal to1.0, matching the labelled prefix ofMergedTriDataset.For context, using the cleaned DomainNet lists and merging train/test per domain also makes the total dataset lengths match all five released
sample_weightsfiles. Therefore, the remaining question here is specifically about the old/new class split and labelled count, rather than the raw-vs-cleaned dataset-length mismatch in #4.Questions
Could you please clarify:
pre_splits=Trueusing the externaldomainnet_splits.pkl, or with the released defaultpre_splits=False?pre_splits=Truewas used, could you releasedomainnet_splits.pkl, or at least the exactknown_classesandunknown_classeslists?sample_weightsfiles generated with?Providing the exact class split and its corresponding sample-weight/checkpoint protocol would help establish a matched reproduction of the DomainNet results.
Thank you!