Error in instalation

[rafasupremoyo]

Sorry in the installation of graphbench I get an error importing graphbench-lib, and if I access the package I get the following output

import graphbench

graphbench.__
graphbench.annotations graphbench.le(
graphbench.class( graphbench.loader
graphbench.delattr( graphbench.lt(
graphbench.dict graphbench.name
graphbench.dir( graphbench.ne(
graphbench.doc graphbench.new(
graphbench.eq( graphbench.package
graphbench.file graphbench.path
graphbench.format( graphbench.reduce()
graphbench.ge( graphbench.reduce_ex(
graphbench.getattribute( graphbench.repr()
graphbench.getstate() graphbench.setattr(
graphbench.gt( graphbench.sizeof()
graphbench.hash() graphbench.spec
graphbench.init( graphbench.str()
graphbench.init_subclass( graphbench.subclasshook(

which can access the datasets through import instead of using it as an object.

[Timo-SH]

This issue could already be fixed in the current version 0.1.2.2 However, we changed the import instructions and the usage instructions from earlier versions. These changes are reflected in the current readme file.

I'll leave this issue open for now. Feel free to try the new version and reach out if the issue still persists.

[rafasupremoyo]

I have tested it, and the import has been fixed. I will trying it try it further and give you a proper overview.

[rafasupremoyo]

Using metric: ['RSE'] for task: regression
Traceback (most recent call last):
File "/home/rafael/Chaostest.py", line 59, in
test(model,dataset,eval)
File "/home/rafael/.conda/envs/h/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/rafael/Chaostest.py", line 42, in test
train_sc = eval.evaluate(torch.stack(h_train,dim=0),torch.stack(y_train,dim=0))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rafael/.conda/envs/h/lib/python3.12/site-packages/graphbench/evaluator.py", line 118, in evaluate
metric = self._get_metric()
^^^^^^^^^^^^^^^^^^
File "/home/rafael/.conda/envs/h/lib/python3.12/site-packages/graphbench/evaluator.py", line 109, in _get_metric
return self._get_metric_from_name(self.metric[0])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/rafael/.conda/envs/h/lib/python3.12/site-packages/graphbench/evaluator.py", line 79, in _get_metric_from_name
'ACC': self._get_acc(),
^^^^^^^^^^^^^^^
File "/home/rafael/.conda/envs/h/lib/python3.12/site-packages/graphbench/evaluator.py", line 133, in _get_acc
acc = torchmetrics.Accuracy()
^^^^^^^^^^^^^^^^^^^^^^^
TypeError: Accuracy.new() missing 1 required positional argument: 'task'

I get this error in electronic circuits when I call the evaluator.

[Timo-SH]

Hi, thanks for letting us know about the issues in the evaluation method. This should be fixed in the newest version (0.1.2.3). Please let us know any further bugs you find during testing.

[rafasupremoyo]

Thank you so much, it works now, but can tell more precisely the expected outputs the evaluator requires for each dataset. It is in my opinion not mirroring the paper description

[Timo-SH]

Generally we expect the target values (y_true ) and the predicted values (y_pred ). Further, the evaluator expects tensors to be concatenated along the first axis if batched tensors are used (e.g. of shape [b x n] with b denoting the batch dimension would be a tensor of shape [b*n]). 
Target values are usually given by the data object via .y unless otherwise noted below.

However, shape and content may differ between each of the datasets. Therefore, we provide a list of what the evaluator expects for each dataset below: 


Bluesky: A regression target for a single graph, with likes, replies, and reposts predicted for each node. Therefore, the expected shape of a tensor is [n x 3] with n denoting the number of nodes.
Different from the other datasets, bluesky also uses the spearman correlation coefficient as an evaluation metric. This metric however, is measured using the same tensors as above.

CO: For combinatorial optimization there exist two different types of targets. In the unsupervised case the evaluator expects a tensor of shape [n] denoting the node features computed and a torch_geoemetric batch object, denoting which nodes belong to a graph. This object should be given from the dataloader method in torch_geometric.
For supervised tasks a tensor of shape [n] is expected with n denoting the number of nodes.


SAT: For SAT solving we either have a classification target (to select the correct solver out of 11 available) or a regression target. The expected shapes of the tensors are [n x 11] for classification and [n] for regression tasks with n as the number of graphs.

Electronic circuits: A regression target for a graph level task. Therefore y_pred should simply be a tensor of shape [n] , where n denotes the batch size. The RSE is then computed automatically.

Chip design: Since chip design is a generation task, we do not expect a tensor as with the other tasks but rather a torch_geoemetric data object holding the following information:
.x : one-hot encoded gate types: [AND, INPUT, OUTPUT].
.edge_index : the edge index of the generated graph
.edge_attr : inversion flag: 0 = direct connection, 1 = negated/inverted connection.

The corresponding data object is then evaluated against the underlying true data object.

Weather forecasting: The target for weather forecasting is given by .grid_x in the data object
Since each data object refers to a set point in time with a 6 hour time difference between the, selecting the next object (or any other following one) provides the data for the forecast targets.
The expected shape is the same as the input values for the grid data (.grid_x ) and refers to the regression values for different measurements across pressure levels.

Example: If you want to generate a forecast and compare it to the 12hr forecast values you would need to consider .grid_x from data[i+2] if your input data comes from data[i]

Important: Since this data is time dependent, it should not be shuffled by default.

Algorithmic reasoning: A regression or classification target depending on the dataset. It can be either a node, edge or graph level task. For both of them the expected shape of tensors is given by [n] or [n x 1] with n being the number of nodes, edges or graphs and should be the same shape as .y in the corresponding data object.
Note that some of the tasks are edge level tasks, therefore there is a target for each edge instead of each node. 




We also put some of the information on our website graphbench.io already. But we will work on incorporating this information in the repository and/or paper.

[rafasupremoyo]

Thank you for your response, the chip design should be each of those 3 gates I assume is 0 for and, 1 for input and 2 for output. Can the design include other types of gates out of NAND (inverse and) and the AND gate. Also can this be extended for quantum gates circuits. I think that problem might be if combined with classical chip circuits more interesting as a problem.

[asiraudin]

Hi @rafasupremoyo,

Right now, GraphBench only includes AIGs since they are functionally complete. We haven't thought about including quantum circuits yet. Chip/circuit design using graph generative models remains widely unexplored, and has no well established baseline so far. I think it's probably better to keep the focus on one fundamental task before expanding to different variants.

This is not definitive though, GraphBench is meant to be a living benchmark and we could add new tasks in future versions.

[rafasupremoyo]

I understand thank you so much for the help